Can TensorFlow, PyTorch, and other mainstream ML frameworks be used for research-grade work in AI?

Your statement that researchers build their network from the ground-up using C++ or some other low level library couldn't be further from the truth.
You could take a look at this analysis showing the popularity of these two frameworks in the top ML conferences. The following Figure is taken from there.

In CVPR-2020, for example, TensorFlow and pytorch combined for over 500 papers! Furthermore, because the two most active research entities (Google and Facebook) are backing these two frameworks, they are used in some of the most impactful research studies.

I want to give some reasons that support the popularity of these frameworks, but first I'm going to rephrase your question a bit:

Why use TensorFlow/Pytorch in python rather than build your model on your own using C++?

Note: The reason I rephrased the question is because TensorFlow and PyTorch both have a C++ APIs.
Why are these frameworks so popular in contrast to lower-level programming languages?
Some reasons are the following

Rapid prototyping. Languages link C++, have bloated syntaxes, require low-level operations (e.g. memory management) and cannot be run interactively. This means it takes someone much less time to create and test a model in python than it does in C++.

No need to re-invent the wheel. Some operations are common in most networks (e.g. backpropagation), why re-implement them? Other functionalities are hard to implement on your own (e.g. parallel processing, GPU computation). Do data scientists need to have such a strong technical background to research neural networks?

Open-source. They benefit from being opensource and can offer a great deal of tools at your disposal for building neural networks. You want to add batchnorm to your network? No worries, just import it and add it in a single line! Also, they offer the perfect opportunity for sharing pretrained models.

They are optimized. These frameworks are optimized to run as fast as possible on GPUs (if available) or CPUs. It would be virtually impossible for someone to write code that runs as fast on his own.


Could machine learning be used to measure the distance between two objects from a picture or live camera?

The short answer is: yes, it could. In what you are describing, there's nothing very new or specific conceptually; it sounds like a standard regression task. Now the problem that you're actually facing is: do you have the data?

Algorithms won't be able to learn the distance between eyes if you don't have the data that it takes. It could be supervised labels (1 distance per image which would be your regression target), reconstruction from depth maps, multi-view estimation etc. There's a number of ways you could do that given the appropriate data.

People focus on algorithms a lot, and that's good. But taking a good look at your data is often as important (if not more).

Now a good example would be in the self-driving car literature. You could start with this blog-post and go through the papers they reference: https://towardsdatascience.com/vehicle-detection-and-distance-estimation-7acde48256e1

There also seems to be some litterature about your eyes examples (https://arxiv.org/pdf/1806.10890.pdf, https://www.sciencedirect.com/science/article/pii/S0165027019301578) so skimming through these papers & the datasets they use could guide to towards answering my question: is there data for this task?


Does it exist a human-like artificial intelligence?

This depends on your definition of human-like.

If you mean a robot that looks and acts like a human, arguably, yes. Here's one of many examples: http://www.hansonrobotics.com/robot/sophia/

If you are looking for something that performs work and tasks, or works and thinks and talks like-or better than a human, the answer is mostly no, not yet.

I recommend you look at 'ANI, AGI, ASI"

ANI: artificial narrow intelligence. This is what you see around you right now.

AGI: artificial general intelligence. A theoretic AI that can "think" like a human. It does not yet exist. Estimates are between 20-60 years before we will successfully create AGI.

ASI: artificial super intelligence. In a nutshell, it is theorized to be everything we wish we could be or hope never to be. It does not exist yet. It is generally believed that, IF we create an AGI, ASI will evolve seconds or less than a decade after AGI is created.


What is local consistency in constraint satisfaction problems?

If we can do some reduction in the search space using CSP (constraint propagation) we can drastically reduce the search space or sometimes completely avoid the need for a search by directly reaching the solutions (for e.g. with variables having their domains reduced to size one). It could also happen that we come to a point when a variable domain size becomes zero, in that case no solution exists, given the constraints, so no need for a search.

Constraint propagation basically involves the concept of enforcing local consistency (this is done by enforcing node-consistency, arc-consistency, path-consistency and also global constraints using Alldiif or Atmost).

The terms: nodes, arc, path, etc. basically reflects a CSP problem represented as a graph with nodes as the variables and the arcs/edges as constraints. The process is simply to remove values from the domains of the variables that are inconsistent. Algorithms such as AC-3, PC-2, etc. precisely are for these purposes.


Do scientists know what is happening inside artificial neural networks?

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned.

Visualizing the activations and layer weights

Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse (most values are zero) and localized. This sometimes shows what exactly a particular layer is focused on when it sees an image.

Another great work on activations that I'd like to mention is deepvis that shows reaction of every neuron at each layer, including pooling and normalization layers. Here's how they describe it:

  In short, we’ve gathered a few different methods that allow you to
  “triangulate” what feature a neuron has learned, which can help you
  better understand how DNNs work.

The second common strategy is to visualize the weights (filters). These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. For example, the first layer usually learns gabor-like filters that basically detect edges and blobs.

Occlusion experiments

Here's the idea. Suppose that a ConvNet classifies an image as a dog. How can we be certain that it’s actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object?

One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object.
If we iterate over regions of the image, replace it with all zeros and check the classification result, we can build a 2-dimensional heat map of what's most important for the network on a particular image. This approach has been used in Matthew Zeiler’s Visualizing and Understanding Convolutional Networks (that you refer to in your question):

Deconvolution

Another approach is to synthesize an image that causes a particular neuron to fire, basically what the neuron is looking for. The idea is to compute the gradient with respect to the image, instead of the usual gradient with respect to the weights. So you pick a layer, set the gradient there to be all zero except for one for one neuron and backprop to the image.

Deconv actually does something called guided backpropagation to make a nicer looking image, but it's just a detail.

Similar approaches to other neural networks

Highly recommend this post by Andrej Karpathy, in which he plays a lot with Recurrent Neural Networks (RNN). In the end, he applies a similar technique to see what the neurons actually learn:

  The neuron highlighted in this image seems to get very excited about
  URLs and turns off outside of the URLs. The LSTM is likely using this
  neuron to remember if it is inside a URL or not.

Conclusion

I've mentioned only a small fraction of results in this area of research. It's pretty active and new methods that shed light to the neural network inner workings appear each year.

To answer your question, there's always something that scientists don't know yet, but in many cases they have a good picture (literary) of what's going on inside and can answer many particular questions.

To me the quote from your question simply highlights the importance of research of not only accuracy improvement, but the inner structure of the network as well. As Matt Zieler tells in this talk, sometimes a good visualization can lead, in turn, to better accuracy.


How can genetic programming be used in the context of auto-encoders?

The first classic paper your present uses a genetic algorithm (GA), while you mention genetic programming (GP). In short, a GA uses an evolutionary process to evolve a fixed-length vector, normally a bit string, while a GP evolves variable-length decision trees, which could be a computer program or mathematical function.
An autoencoder is something that automatically encodes or decodes something. For example, a noisy image is decoded to the original image (very simplified I know), the point here is automatically, so, without any defined process, an autoencoder can "learn" how to remove the noise from the image.
Generally, an autoencoder is made using a neural network (NN). The noisy image is read in as input and the output is a clear image. To train the NN, backpropagation is used. For backpropagation to work, an error is needed to calculate the weight changes. Alternatively, the weights could be encoded as a vector. A fixed-length vector is then evolved using a GA to evolve the weights of the NN. This is what the "classic paper" mentioned does.
Using a GA to evolve the weights of a NN is not really efficient, because weights affect each other which means each time crossover happens the weights are totally messed up and the GA can easily get stuck in a local optimum. Gradient descent is probably more efficient.
The structure of a multi-layered network is important. Here, gradient descent is still used to update the weights, but a GP could be used to evolve the structure of the NN.
Alternatively, the GP itself could be used to evolve an autoencoder without using a NN. Here, the chromosome will contain nodes such as mathematical operations or computer code that would alter the input to produce the output.


What is the difference between Attention Gate and CNN filters?

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel.
Note, when I say "at every pixel" this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: https://arxiv.org/pdf/1709.01507.pdf
While this is one of the strengths of CNNs, since it drastically reduces the number of parameters of a network, you can imagine that it may not make sense to treat every part of the image the same regardless of the content. This is what the attention gate is for.
By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image.
CNNs without AG can typically learn the same things but may need more channels and layers. Attention gates allow to treat filters differently depending on the content and hence make it easier to learn with less filters.
What you use as the attention tensor can vary. In the paper you referenced, the tensor was one of the input tensors of the convolutional layer. In this case, it is called self-attention. However, with attention in general this gate tensor may also come from other information such as another network.


Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for the details.
However, compared to $\epsilon$-greedy (widely used in RL), UCB1 is more computationally expensive, given that, for each action, you need to recompute this upper confidence bound for every time step (or, equivalently, action taken during learning).
To see why, let's take a look at the UCB1 formula
$$
\underbrace{\bar{x}_{j}}_{\text{value estimate}}+\underbrace{\sqrt{\frac{2 \ln n}{n_{j}}}}_{\text{UCB}},
$$
where

$\bar{x}_{j}$ is the value estimate for action $j$
$n_{j}$ is the number of times action $j$ has been taken
$n$ is the total number of actions taken so far

So, at each time step (or new action taken), we need to recompute that square root for each action, which depends on other factors that evolve during learning.
So, the higher time complexity than $\epsilon$-greedy is probably the first reason why UCB1 is not so much used in RL, where interaction with the environment can be the bottleneck. You could argue that this recomputation (for each action) also needs to be done in bandits. Yes, it's true, but, in the RL problem, you have multiple states, so you need to compute value estimates for each action in all states (i.e. the full RL problem is more complex than bandits or contextual bandits).
Moreover, $\epsilon$-greedy is so conceptually simple that everyone can easily implement it in less than $5$ minutes (though this is not really a problem, given that both are simple to implement).
I am currently not familiar with Thompson sampling, but I guess (from some implementations I have seen) it's also not as cheap as $\epsilon$-greedy, where you just need to perform an argmax (can be done in constant time if you keep track of the highest value) or sample a random integer (it's also relatively cheap). There's a tutorial on Thompson sampling here, which also includes a section dedicated to RL, so you may want to read it.


What is the smoothness assumption in SVMs?

Smoothness here is the mathematical definition, so as you implied smoothness is ruled out by output data with sharp spikes or discontinuous jumps (and possibly the data of the gradient, the gradient's gradient, ad infinitum, depending on who defines smoothness).
By any definition a lot of activation functions are not smooth, for example RELU. This means neural networks in general are not smooth.


What are the differences between network analysis and geometric deep learning on graphs?

Network analysis does not necessarily use deep learning techniques, while geometric deep learning (GDL) on graphs uses only deep learning techniques (that is, you train a neural network using gradient descent or other optimization methods). You can do some network analysis using GDL.


Wind speed forecasting using ARIMA model in Python3

First things first, time series is probably the most difficult of all things to predict. If it wasn't I'd be sitting on nice warm beach or on yacht cruising the Mediterranean.
Besides that here's a few hints. First, always visualise your data first. Tables of numbers are fine for computers by the MK1 eyeball is still your best weapon for time series. Is there any obvious trends or seasonal behaviour? If not then ARIMA is unlikely to be of much utility. Secondly, visualise your data after forecasting. Don't just rely on error measures like AIC, use your eyes. Most time series forecasts are useful only in the short term. So you may see the earlier part of your forecast make sense while the latter doesn't.
Finally ARIMA is paramterised by by p, d, and q. The value of d is the minimum number of differencing needed to make the series stationary. If the time series is already stationary, then d = 0. ‘p’ is the order of the ‘Auto Regressive’ (AR) term. It refers to the number of lags of Y to be used as predictors. And ‘q’ is the order of the ‘Moving Average’ (MA) term. It refers to the number of lagged forecast errors that should go into the ARIMA Model.
You have only shown results for p,d,q = 1,0,1. Much of ML is done by trial and error, trying different parameters. So I'd suggest you try varying your values of p, d and q. AIC can then be used to compare models, with smaller values indicating better models.


Looking for NLP technique and drawing a blank

You might be referring to Semantic role labeling. SRL is the task of assigning labels to words or phrases in a sentence that shows their semantic role in that sentence.

In your example CV was hit by IV, the task is to identify the verb "hit" carried out by the actor "CV" affected "IV" the recipient.

Note: If you're only interested in the syntactic relationship among words or phrases in a sentence, not the semantic relationship between them, simple dependency parsing would do the job.


Semantic segmentation failing in small instance detection

You may find it useful to categorize these smaller defects as a separate class, then introduce a class weights matrix to penalize incorrect classification of the smaller defects more heavily. If these small defects represent a very small portion of the total number of pixels in your training data, then the model may be stuck at a local minimum, where it just predicts them as zero because the loss is not heavily penalized for this. So you need to add extra penalty for the incorrect classification.
Assuming you create a new class for the small defects, the class weights matrix is implemented as a keras callback function.
If you do not want to create a multiclass problem, there is also this paper : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977596/#!po=32.4561
The multilevel unet architecture will not solve class imbalance if that is truly the underlying issue, but if may make the network more aware of smaller scale changes in the images you're working with.


What is a "codon" in grammatical evolution?

Grammatical evolution
To understand what a codon is, we need to understand what GE is, so let me first provide a brief description of this approach.
Grammatical evolution (GE) is an approach to genetic programming where the genotypes are binary (or integer) arrays, which are mapped to the phenotypes (i.e. the actual solutions, which can be represented as trees, which, in turn, represent programs or functions), using a grammar (for example, expressed in Backus-Naur form). So, the genotypes (i.e. what is mutated, combined, or searched) and the phenotypes (the actual solutions, which are programs) are different in GE, and the genotype needs to be mapped to the phenotype to get the actual solution (or program), but this is not the case in all GP approaches (for example, in tree-based GP, the genotype and the phenotype are the same, i.e. trees, which represent functions).
Codons
In GE, a codon is a subsequence of $m$ bits of the genotype (assuming that genotypes are binary arrays). For example, let's say that we have only two symbols in our grammar, i.e. a (a number) and b (another number). In this case, we only need 2 bits to differentiate the two. If we had 3 symbols,  a, b and + (the addition operator), we would need at least a sequence of 2 bits to define each symbol. So, in this case, we could have the following mapping

a is represented by 00 (or the integer 0),
b is represented by 01 (or 1), and
+ is represented by 10 (or 2)

The operation a+b could then be represented by the binary sequence 001001 (or the integer sequence 021). The 2-bit subsequences 00, 01 and 10 (or their integer counterparts) are the codons.
What do we need codons for?
In GE, codons are used to index the specific choice of a production rule. To understand this, let's define a simple grammar, which is composed of

a set of non-terminals (e.g. functions) $N = \{ \langle \text{expr} \rangle, \langle \text{op} \rangle, \langle \text{operand} \rangle, \langle \text{var} \rangle \}$
,
a set of terminals (e.g. specific numbers or letters) $\mathrm{T}=\{1,2,3,4,+,-, /, *, \mathrm{x}, \mathrm{y}\}$,
a set of production rules $P$, and
an initial production rule $S = \langle \text{expr} \rangle $.

In this case, the set of production rules $P$ is defined as follows
\begin{align}
\langle \text{expr} \rangle & ::=
\langle \text{expr} \rangle \langle \text{op} \rangle \langle \text{expr} \rangle \; | \;  \langle \text{operand} \rangle \\
\langle \text{op} \rangle & ::= + \;  | \;  - \;  | \; * \; | \; / \\
\langle \text{operand} \rangle & ::= 1 \;  | \; 2 \; | \;  3 \; | \;  4 \;  | \;  \langle \text{var} \rangle \\
\langle \text{var} \rangle & ::= \mathrm{x} \; | \;  \mathrm{y}
\end{align}
So, there are four productions rules. To be clear, $\langle \text{var} \rangle  ::= \mathrm{x} \; | \;  \mathrm{y}$ is a production rule. The symbol $|$ means "or", so the left-hand side of each production is a non-terminal (and note that all non-terminals are denoted with angle brackets $\langle \cdot \rangle$), which is defined as (or can be replaced with) one of the right-hand choices, which can be a non-terminal or terminal. The first choice of each production rule is at index $0$. The second choice at index $1$, and so on. So, for example, in the case of the production $\langle \text{var} \rangle  ::= \mathrm{x} \; | \;  \mathrm{y}$, $\mathrm{x}$ is the choice at index $0$ and $\mathrm{y}$ is the choice at index $1$.
The codons are the indices that we use to select the production rule's choice while transforming (or mapping) the genotype into a phenotype (an actual program).
So, we start with the first production, in this case, $S = \langle \text{expr} \rangle$. If it's a non-terminal, then we replace it with one of its right-hand side choices. In this case, there are two choices

$\langle \text{expr} \rangle \langle \text{op} \rangle \langle \text{expr} \rangle$ (choice at index $0$)
$\langle \text{operand} \rangle$ (choice at index $1$)

If our genotype (integer representation) is, for example, $01$ (note that this is a sequence of integers), we would replace $\langle \text{expr} \rangle$ with $\langle \text{expr} \rangle \langle \text{op} \rangle \langle \text{expr} \rangle$, then we would replace the first $\langle \text{expr} \rangle$ in $\langle \text{expr} \rangle \langle \text{op} \rangle \langle \text{expr} \rangle$ with $\langle \text{operand} \rangle$, so we would get $\langle \text{operand} \rangle \langle \text{op} \rangle \langle \text{expr} \rangle$, and so on and so forth, until we get an expression that only contains terminals, or the genotype is terminated. There can be other ways of mapping the genotype to the phenotype, but this is the purpose of codons.
Codons in biology
The term codon has its origins in biology: a subsequence of 3 nucleotides is known as a codon, which is mapped to an amino acid in order to produce the proteins. The set of all mappings from codons to amino acids is known as genetic code. Take a look at this article for a gentle introduction to the subject. So, in GE, codons also have a similar role to the role of codons in biology, i.e. they are used to build the actual phenotypes (in biology, the phenotypes would be the proteins or, ultimately, the organism).
Notes
Codons do not have to be 2-bit subsequences, but they can be $m$-bit subsequences, for some arbitrary $m$. The term codon is similar to the term gene, which is also often used to refer to specific subsequences of the genotype (for example, in genetic algorithms), although they may not be synonymous (at least in biology, genes are made up of sequences of codons, so they are not synonymous). Moreover, the binary $m$-bit codons can first be mapped to integers, so codons can also just be integers, as e.g. used here or illustrated in figure 2.2 of this chapter.
Further reading
You can find more info about codons in the book Genetic Programming: An Introduction by Wolfgang Banzhaf et al., specifically sections 9.2.2. (p. 255), 9.2.3. (an example) and 2.3 (p. 39), or in chapter 2 of the book Foundations in Grammatical Evolution by Dempsey et al.


Can machine learning algorithms be used to differentiate between small differences in details between images?

Attentive Recurrent Comparators (2017) by Pranav Shyam et al. is an interesting paper that helps to answer the question you're wondering, along with a blog post that helps to describe it in easier terms.

The way it's implemented is actually rather intuitive. If you have ever played a "what is different" game with two images usually what you'd do is look back and forth between the images to see what the difference is.  The network that the researchers created does just that!  It looks at one image and then remembers important features about that images and looks at the other image and goes back and forth.


Are there any conferences dedicated to artificial general intelligence?

There are several conferences dedicated to AGI or human-level intelligence, such as

AGI Conferences (organized by AGI society)
Biologically Inspired Cognitive Architectures (organized by BICA Society)
Advances in Cognitive Systems
IEEE Task Force on Towards Human-like Intelligence

The conferences focus on topics such as cognitive architectures, autonomy, creativity, lifelong learning, and formal models of general intelligence. There are also journals associated with these conferences (see [1], [2], [3]). If you want to know more about them, I suggest that you go to their websites.


Pseudocode for CNN with Bounding Box and Classifier

The minimal algorithm for convolution in $\mathbb{R}^2$ is a four dimensional iteration.

for all vertical kernel positions
  for all horizontal kernel positions
    initialize the value at the output position to the bias
    for all vertical positions in the kernel
      for all horizontal positions in the kernel
        add the product of the input value to that of the output position

In $\mathbb{R}^n$ it is a $2n$ dimensional iteration following this pattern.

The minimal algorithm for regression of bounding boxes orthogonal with respect to the image grid (no tilting) is this.

until number of boxes reaches max
  make first guess of two coordinates
  until number of guesses reaches max or matching criteria is met
    evaluate guess
    remember guess and guess results
    improve on guess based on evaluation results and
          possibly injected randomness,
          excluding locations already covered
    if some intermediate criteria is met
      change the nature of the guessing, evaluation, and improving
            as is appropriate for the criteria match
            (this covers approaches that have multiple phases)
  if no guess matched criteria
    break

That's approaching concepts from the top down. When approaching from the other direction, reverse engineer the best code. In the case of RCNN, it is unadvisable to find implementations following the first paper expressing the approach. Reading the first paper may be helpful to get the gist of the approach, but reverse engineer the best one, which, in this case, may be Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, 2016. Study the implementation they pushed to git at https://github.com/rbgirshick/py-faster-rcnn/tree/master/. The algorithm is in lib/fast_rcnn.

The reason this algorithm isn't spelled out in their paper or any paper from the first on down through the lineage to their paper is simple.

The pseudo-code above is universal across all convolutions and all bounding box regressions, so that doesn't need to be restated with each approach.
The main features of an approach like RCNN, SSD, or YOLO are not algorithmic. They are algebraic expressions of the guess, the evaluation, the improvement upon the guess, and the test for the criteria.
The use of objects and functional programming makes the implementation more readable, so it can be easier to read the implementation than read a huge chunk of the above pseudo-code with all the algebra and test branches plugged in.
For the above reasons, it is rare that pseudo-code would be used prior to the implementation when the paper is written.
The return on investment of reverse engineering from code to pseudo-code is only sufficient motivation if one is going to improve the algorithm and write another paper, and on the way to finishing the prior paper's pseudo-code, the new paper and the new code gets finished first.

Since the author of this question seems interested in writing their own code, it may be reasonable to assume the same author may be interested in thinking their own thoughts, so I'll add this.

None of these algorithms are object recognition. Recognition has to do with cognition, and these approaches do not even touch upon cognitive processing, another branch of AI not related to convolution and probably not closely related to formal regression either. Additionally, bounding boxes are not the way animal vision systems work. Early gestalt experiments in vision indicate a complete independence of human vision from rectilinear formalities. In lay terms, humans and other organisms with vision systems don't have any conception of Cartesian coordinates. We can still read books if tilted slightly relative to the plane passing through our eyes. We don't zoom or tilt in Cartesian coordinates.

These facts may not be necessary to comprehend to create an automated vehicle driving system that produces a better safety record than average human drivers, but that is only because humans don't set that bar very high and because cars roll in the plane of the road. These facts are indeed necessary in aeronautic system used in military applications, where nothing is particularly Cartesian and the meaning of horizontal and vertical is ambiguous. For that reason, it is unlikely that bounding boxes will be the edge of vision technology for very long.

If one wishes to transcend current mediocrity, consider bounding circles with fuzzy boundaries, which would be more like the systems that evolved over millions of biological iterations. If the computer hardware is poorly fit to radial processing, design new hardware in which radial processing is native and in which Cartesian coordinates may be foreign and cumbersome.

Regarding the classifier, the classifier papers do generally include the algorithm, so those can be found by doing an academic search for the original paper describing the classifier being used.


How do you efficiently choose the hyper-parameters of a neural network?

In deep learning era, there are two possible choices. Caveat approach and Panda approach.

Caviar Approach

In this approach, it is supposed that you have a very powerful cluster system that enables you to run different models simultaneously on different nodes. In this manner, you construct a d-dimensional space which each corresponds to a special hyperparameter. Then you can have a grid approach to partition the space and for each intersection of the grids, you can have a possible initialisation although there are some better ways in order to find better initialisations.

Panda Approach

This approach is widely used among students due to using desktop computers. In this approach, you usually try to find an initialisation of hyperparameters based on your experience or the other architectures' which are available and you try to refine them step by step by using cross-validation.

To answer your question, the efficiency depends on your computing power. For more details, you can take a look at the contents of the third week.


What does the symbol $\mathbb E$ mean in these equations?

That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that expression involves some random variables, which means that there may be a wide range of different values the expression may take in any concrete, single event. Taking the expectation basically means that you "average" over all the values the expression could potentially take, appropriately weighted by the probabilities with which certain events occur.

You'll often find assumptions under which the expectation is taken after a vertical line ($\mid$) inside the brackets, and/or in the subscript to the right of the $\mathbb{E}$ symbol. Sometimes, some assumptions may also be left implicit.

For example:

$$\mathbb{E} \left[ R_t \mid s_t = s, a_t = a, \pi \right]$$

may be read in english as "the expected value of the discounted returns from time $t$ onwards ($R_t$), given that the state at time $t$ is $s$ ($s_t = s$), given that our action at time $t$ is $a$ ($a_t = a$), given that we continue behaving according to policy $\pi$ after time $t$".

I would say that, in this case, the expected value also relies on the transition dynamics of the Markov decision process (i.e. the probabilities with which we transition between states, given our actions). This is left implicit.

Second example:

$$V^{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} \left[ Q^{\pi}(s, a) \right]$$

may be read as "$V^{\pi}(s)$ is equal to the expected value of $Q^{\pi}(s, a)$, under the assumption that $a$ is sampled according to $\pi(s)$".

In theory, something like this could be computed by enumerating over all possible $a$, computing $Q^{\pi}(s, a)$ for every such $a$, and multiplying it by the probability that $\pi(s)$ assigns to $a$. In practice, you could also approximate it by running a large number of experiments in which you sample $a$ from $\pi(s)$, and then evaluate a single concrete case of $Q^{\pi}(s, a)$, and average over all the evaluations.


How to organize artificial intelligence efforts at work?

I know what you mean. It can be difficult to parse between the hype and the application of AI. Although AI (specifically deep learning) can do a lot of things, this doesn't negate previous methods that work just as well or better in certain domains. Some times managers will hear 'AI' and think of giant networks when they actually just needed a simple linear regression for their problem.

That being said, here are some suggestions to help decide if AI is right for your project and some things to consider if so :

Get some advice from a researcher. There are many AI consulting firms popping up in large cities that give companies hours of consulting time. If there isn't one in your city, go to AI meetups and meet the community, find educated people and ask them about your general questions. For more in depth advice, perhaps a local university can lend a hand.
Understand your problem. Taking the time to know exactly what problems you are trying to solve is invaluable. Not only will it help new employees but if you do end up using AI and get consultation hours from an AI expert, it will save time and money explaining what you want to do.
Know your data. First this is a check to see if you have the right data for AI and if you have enough of it. For example many problems are approached using supervised learning which requires having a lot of labeled data (Eg, think of 1000s of pictures labeled Cat or Dog depending on the image content). If you don't have or can't easily collect a large amount of labeled data, perhaps you can use an existing data set with similar data to help get you started. If that's not an option, then you probably won't like the alternative where you hand label the data or hire Amazon MechanicalTurks.
Be prepared to fail at first. AI is not easy and is even harder when you can't just Google your questions because you are doing something no one has done before. It takes some time to understand your problem, your data, and what kind of models you should try.
Do you have the infrastructure? If everything is going well and you have a working model on your local GPU, if your model is part of your product, look at how you might deploy it. Do you have your own GPU servers? Can you afford the cloud GPS that some companies offer? Do you have to learn a model for each user?


Why do I need an initial arbitrary policy to implement value iteration algorithm

It seems to me that you're thinking about the parameters a and b as being characteristic of the agent that's moving in the environment (therefore determining the final policy), but they are actually a characteristic of the environment.
Think of a frozen lake. You want to pass the lake but there is a hole five meters in front of you. Let's say you have boots with a rubber sole, so there is no risk to slip while walking (i.e. all transition probabilities =1). What is the optimal policy? Simply moving forward till the hole, goes around it and then forward again. But what if you were wearing wooden clogs instead? Now if you walk forward there is the risk to not be able to stop when you want to (i.e. when you decide to move forward of one step you might fail and keep going forward for several steps, with probability a). How many steps would you wait before moving to the left or to the right to avoid risking to fall into the hole? The optimal policy is obviously different in this second scenario. Think about it, I will not answer in detail about how the parameters affect the final policy because stack exchange is not meant for homework.
Also, an arbitrary policy is necessary simply because in value iteration you first update the values and only when they converge you infer from then the optimal policy. So you need some arbitrary prior belief about the environment to start update the values.
Edit
So here's a quick visualisation of the impact of different transition probabilities (the parameters a and b in your assignment) on the optimal policy. I used a different grid to make it easier and more explicit (i.e. you don't even need to calculate the values to see the optimal policy).
As you can see in the picture below, there are some terminal states in the third column and a goal state which gives a huge reward on top of it. From the starting point on the bottom we want to reach the goal state.

We have two cases:

Non stochastic environment

On the left we have the case in which all transition probabilities are 1. In this case the optimal policy is simply going forward till the second row, where the goal state is located, and then move right or left to reach it. The terminal states are not scary at all, in this case we could consider them as walls that we can't cross, but the optimal policy do not reflect any risk of falling in them. This is precisely because the transition probabilities are all 1.

Stochastic environment

On the right we have a different situation. In this case only 3 action have a transition probability of 1: going south, east or west. The transition probability for nord is 0.2 meaning that 80% of the time we will slip and go east instead (I wrote right in the picture meaning east, my bad). What is the impact of this simple change in the environment?
This time the second column become really dangerous, because if we try to go nord and fail we will end on a terminal state. On the other hand, if we choose another action we will end on a legitimate state, without any risk because the other transition probabilities are 1! Therefore the agent learn that on the left side of the grid the best strategy is to go down and circumnavigate the terminal states from the right, because only in the fourth column we can move north without risk, in fact in those cells even if we try go north and fail we would just hit a wall and we could try again going north the next turn till we will eventually succeed.
Hope this make sense!


In NEAT, is it a good idea to give the same ID to node genes created from the same connection gene?

In NEAT, the innovation of a node does not affect the evolution directly. Only the connection genes and their innovation will matter. So you can simply have whole numbers as IDs under each Genome / Network.
--EDIT-- (Complete reasoning)
In the original paper, it is clearly stated that the nodes from the better genome is taken during crossover and then only the connections are cross-over'ed (in some method) and hence the innovation numbers for the connections. NEAT is connection-centric and does not care much about evolving nodes.
Adding to that from basic neural networks theory, the nodes will never matter in a neural network because all the calculation and the learning happens in the connections. Think about a regular feed-forward network. You only care about the weight matrix which is a property of the connections instead of the nodes though both are present. Similarly in a NEAT generated network, the nodes will not matter, as all the learning takes place in the way these nodes are connected and the weights of the network.
Further, the nodes list can easily be derived from the connection list and hence marking the connections, are enough.


How does the number of stacked LSTM layers or units in each layer affect the model complexity?

In computational learning theory, the VC dimension is a formal measure of the capacity of a model. The VC dimension is defined in terms of the concept of shattering, so have a look at the related Wikipedia article, which briefly describes the fundamental concept of shattering. See also my answer to the question How to estimate the capacity of a neural network? for more details.

The paper Vapnik-Chervonenkis dimension of recurrent neural networks (1998), by Pascal Koirana and Eduardo D. Sontag, partially (because they do not take into account more advanced recurrent neural network architectures, such as the LSTM) answers your question.

In the paper, the authors show and prove different theorems that state the VC dimension of (standard) recurrent neural networks (RNNs), with different activation functions, such as non-linear polynomials, piecewise polynomials and the sigmoid function.

For example, Theorem 5 (page 70) states

  Let $\sigma$ be an arbitrary sigmoid. The VC dimension of recurrent architectures with activation $\sigma$, with $w$ weights and receiving inputs of length $k$, is $\Omega(wk)$.

The proof of this theorem is given on page 75.

What does this theorem intuitively tell you? If you are familiar with big-O notation, then you are also familiar with the notation $\Omega(wk)$, which means that $wk$ is, asymptotically, a lower bound on the capacity of the RNN. In other words, asymptotically, the capacity of an RNN with $w$ weights receiving inputs of length $k$ is at least $wk$. How does the capacity of the RNN increase as a function of $w$?

Of course, this is a specific result, which only holds for RNNs with the sigmoid activation function. However, this at least gives you an idea of the potential capacity of an RNN. This theorem will hopefully stimulate your appetite to know more computational learning theory!

The paper On Generalization Bounds of a Family of Recurrent Neural Networks may also be useful, although it has been rejected for ICLR 2019.


Are information processing rules from Gestalt psychology still used in computer vision today?

Rules based on Gestalt psychology could be seen as a 'local minima' in terms of optimal image processing. Some of them could be surprisingly effective, but difficult to extend and improve upon, since they assume certain high level attributes that might not generalize well.
"If it's circular, then it's a fruit 90% of the time"
Modern methods like neural networks come at the problem from another direction, and try to build towards those high level attributes incrementally. You could say that a goal of these modern methods is to recreate a version of those 'assumptions' or 'Gestalt rules' empirically. That way, we can feel more confident that we're not using a locally optimal approach.
"This is the smallest algorithm that will accurately identify fruit 90% of the time"
Are such methods being used or worked on today?
Yes, and not just in image processing. If you look at any industry where fast processing of ambiguous domain-specific data is needed, there are usually systems that were designed by experts in that field using 'hand-crafted' heuristics. A benefit of these approaches is that the 'Gestalt rules' are understandable apart from the system that uses it, and is therefore easier to trust.
Was any progress made on this? Or was this research program dropped?
Aside from trivial cases, employing the gestalt approach in a particular domain requires expertise. "Image Expert" is an expensive and hard to fill role. Even then, you're bounded by that expert's ability to effectively generalize an extremely complex problem. Over the past decades, machine learning has become a lot easier and less expensive to employ, and has been slowly replacing all those 'hand-crafted' approaches.
However, one of the interesting things we're finding when we let the machine learning approaches loose is that they sometimes don't agree with our previous rules. Why they don't agree, and what this means about automation and our psychological biases, is one of the big open questions in AI.
You might also be interested in Saliency Maps, and similar techniques intended to visualize the general rules used by neural networks: https://en.wikipedia.org/wiki/Saliency_map
References:
http://wayback.archive-it.org/219/20121229060421/http://www.cs.indiana.edu/~jwmills/ANALOG.NOTEBOOK/klm/klm.html
https://journals.sagepub.com/doi/pdf/10.1177/002029400403701001
https://maritime.org/doc/op1140/index.htm
https://airandspace.si.edu/stories/editorial/inventing-apollo-spaceflight-biomedical-sensors


What exactly do people mean when they say that the computer "learns" to do something?

Your confusion is a result of Moravec’s Paradox; the just tell it to... step is enormously more difficult than it sounds. So what researchers do is attempt to find a general approach to that problem. The problem is expressed as a matrix of numbers (a grid that may be a million on each side these days) and “learning” is the process of computing the values for each of those numbers. Obviously no human can solve an equation like that, so a machine solver is used. Therefore the work of the AI researcher is to find better ways to generate these large matrices i.e. to express go to X and to create better solvers. The holy grail is a matrix and a solver that will work for any X.


Does stochasticity of an environment necessarily mean non-stationarity in MDPs?

Is a stochastic environment necessarily also non-stationary?

No.
A stochastic environment (i.e. an MDP with a transition model $p(s', r \mid s, a)$) can be stationary (i.e. $p$ does not change over time) or non-stationary ($p$ changes over time). Similarly, a deterministic environment, i.e. the probabilities are $1$ or $0$, can also be either stationary or not. To emphasize that an MDP may be non-stationary, you could write $p$ as a function of time, i.e. $p_t$ (you also do the same thing for the reward function if it's separate from the transition function).
The same idea applies to a stochastic/deterministic policy, which can either be stationary or not.
A non-stationary environment may lead to a non-stationary policy (or may require you to relearn a model of the environment, if you need to learn a model of the environment) [1]. However, note that a stochastic environment (i.e. an MDP) does not necessarily imply a stochastic policy (actually, under some conditions, stationary and stochastic MDPs are known to have a deterministic optimal policy [1]).
In general, if something (e.g. environment, policy, value function or reward function) is non-stationary, it means that it changes over time. This can either be a function or a probability distribution. So, a probability distribution (the stochastic part of an MDP) can change or not over time. If it changes over time, then it makes the MDP non-stationary.

But it is non-stationary in the sense that the reward obtained from taking a certain action in a given state may change at a given time

Informally, you could say that the empirical reward obtained is non-stationary because it changes over time, due to the stochasticity of the reward function, behaviour policy, etc., but the dynamics (transition function and reward function) would still be fixed, so the environment would still be stationary. So, there's a difference between the environment and the experience that you collected so far (with some behaviour policy).


How to add a pretrained model to my layers to get embeddings?

I think you should use Keras embedding layer. It will be too easier than what you are doing.

Steps

Create Embedding Matrix
add matrix to embedding layer while building model.

You will find detailed article

https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html


Strong AI vs Singularity - which should happen first?

The definition of "technological singularity" answers the question:

  The technological singularity (also, simply, the singularity) is
  the hypothesis that the invention of artificial superintelligence will
  abruptly trigger runaway technological growth, resulting in
  unfathomable changes to human civilization.

(wiki)

note the order of facts is "invention of artificial superintelligence" (AGI) followed by the "unfathomable changes".


At what rate could AI theoretically self-improve?

The processing power might still follow the Moore's law, but it is basically an economical law, although it has worked surprisingly well considering the physical limitations of technology. An Artificially Intelligent agent could focus on manipulating the market, research possible hardware improvements, allocate human resources more effectively to the problem, etc.

I would expect some improvement in the AI algorithm design itself, which will not influence the number of operations available, but the number of operations required to complete a task. By using a better heuristics, the AI algorithm will consume less brute force to complete the task, thus being able to complete more tasks given the same number of operations. Sorry if I misunderstood the question. Thanks for the improvement.


Is there a reason to use TensorFlow over PyTorch for research purposes?

In the past, I have used TensorFlow (1 and 2), Keras and PyTorch, so I will give an answer based on my experience. Currently, I use TF 2 and Keras (the version shipped with TF 2).

In my (but not only) opinion, TF 1 is really ugly and painful, given that it involves sessions, placeholders and, in general, you need to define the computational graph before executing anything (even the simplest programs). With TF 2, you don't need sessions and placeholders anymore, and this is a very big improvement because you don't need to think that you first need to define a computational graph and then feed it with the input. Essentially, in TF 2, you can write TF code and you can almost be sure that your TF 2 code will look like any other Python code.

TF and Keras have some bugs, but I think that also PyTorch must have some bugs (even though I don't remember having encountered them).

TF comes with TensorBoard. You can also easily use TensorBoard from Keras. About one year ago, I had written a blog post about visualization tools for PyTorch. At the time, the tools didn't seem as developed as TensorBoard, but there were already a few options.

Debugging in TF 2 isn't still perfect, but I would say that it has improved with respect to TF 1 (mainly because there is less boilerplate code). To debug in TF 2.0, I mainly use tf.print (which behaves similarly to Python's print function), but there may be more advanced tools (e.g. I think there's a TF debugging plugin for PyCharm), even though I've never used them. TensorBoard could also be used as a debugging tool (for example, to understand how your models are learning, etc).

The simplicity of PyTorch is comparable to that of Keras. TF now comes with an integrated version of Keras, which is optimized for TF, so TF code can look as simple as Keras code.

In general, my impression is that, if you need to, TF provides more flexibility, but it's been a while since I used PyTorch. Furthermore, with TF 2.0, the code looks simpler, more consistent and the framework is better organized. On the other hand, PyTorch may still be easier to learn (because it doesn't have the past of TF, i.e. there are still many TF code that uses sessions, etc.).

Both TF and PyTorch are supported by big companies (respectively, Google and Facebook), but TF is more mature and there are functionalities/libraries that exist for TF that still do not exist for PyTorch (AFAIK). I doubt that there is a functionality for PyTorch that doesn't also exist for TF or Keras.


In reinforcement learning, is the value of terminal/goal state always zero?

Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.
The value of a state $v(s)$ is the expected sum (perhaps discounted) of rewards from all future time steps. There are no future time steps when in a terminal state, so this sum must be zero.
For the sake of consistent maths notation, you can consider a terminal state to be "absorbing", i.e. any transition out of it results in zero reward and returning to the same terminal state. Then you can use the definition of value function to show the same thing:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_{t} = s]$$
If $s = s_T$, the terminal state, then all the "future rewards" from $k=0$ onwards starting with reward $R_{t+1}$ must be zero. This is consistent with the reward $R_{t}$, i.e. the reward when transitioning to the terminal state, being any value.
You can show similar using action value functions, if you accept a "null" action in the terminal state.


What exactly does "channel" refer to in tensorflow/lucid?

The channel they are talking about is the depth of the layer L.

In this image, the channel is 5. There are 5 $3*2$ filters

They optimize an image from noise, to better respond to the filter F, in the layer L. Then you obtain this kind of image, and you can try to interpret the prupose of that filter (it learns to detect eyes, faces, wheels, and so on)

EDIT: To be more precise, you visualise the channel only if you optimise images for every filters, otherwise, you obtain a filter visualisation (if you do the optimisation one time, for one filter)

They call it "channel", because in a colored image, you have 3 dimensions : witdh, height, and channel (for color), as layers which have also 3 dimensions.


What are examples of good free books that cover the back-propagation algorithm?

Deep Learning by Goodfellow et. al is a good book for anything related neural networks, and it's freely available online. Backpropagation is covered in Chapter 6.5.


Can I develop a chatbot to carry on a natural conversation with a human using NLP and neural networks?

I would not recommend using neural networks and NLP together to create a system sufficiently capable of conversation/dialogue that it would pass that current crop of Turing-like tests.

Conversations follow certain rules and regularities (which we have only partially discovered so far), and training an ANN with dialogues in order to pick up those regularities is simply not feasible. In conversations you have a memory of what has been mentioned previously, you build up assumptions about the intentions of your dialogue partner, and keep track of the current topic and sub-topics. This is far too complex to be reduced to a machine learning approach.

As a starting point I would suggest looking at ELIZA, developed by Weizenbaum in the mid-1960s. There are plenty of implementations in various programming languages available. Use that as a starting point to extend the capabilities according to topics you want to talk about, and store in memory what the user has said before, trying to refer back to it, etc. This is a lot easier to do with 'symbolic' AI rather than subsymbolic processing.

A lot of current tech companies offer chatbot variants based on machine learning, but they rarely go beyond intent recognition or simple question-answer dialogues. For more sophisticated dialogues they are simply not suitable.

(Disclaimer: I work for a company producing conversational software)


How can supervised learning be viewed as a conditional probability of the labels given the inputs?

This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input $x$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.
People often use this notation to indicate that, in the case of classification, there is a categorical distribution over the labels given the inputs, but this can be misleading, as the softmax (the function often used to model this categorical distribution) only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network, although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a point estimate for each parameter of the network is learned and no uncertainty is properly modeled.
Nevertheless, certain supervised learning problems have a formal probabilistic interpretation. For example, the minimization of the mean squared error function is equivalent to the maximization of a log probability, assuming your probability distribution is a Gaussian with a mean equal to the output of your model. In this probabilistic interpretation, you typically attempt to learn a probability (e.g. of the labels in the training dataset) and not a probability distribution. Watch Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning) by G. Hinton or read the paper Bayesian Learning via Stochastic Dynamics or Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method by R. Neal for more details.
Moreover, there are Bayesian neural networks (BNNs), which actually maintain a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each parameter of the neural network, then you will learn the mean and variance of these Gaussians.


Choosing the right neural network settings

Easy ones first:

Activations are going to be RELU all the way down, until your final softmax layer (win probability?) (because empirically, RELU does great on most problems, except when your model is an RNN, and it makes the gradient explode, or in the final layer of a regression model - numerical stability etc.).
You probably to structure this with some layers of convolutions with max pooling between them, then 1 or 2 fully connected layers (FC) near the end (because if you only have FC layers, you probably won't have enough data to train them)
Well worth trying some 1D convolutions (which cleverly combine channels from convolutions created by previous layers).
Learning rate: take the SGD default to begin with, then tune later. Problem dependent, I'm afraid!  The returns to tuning can be large though (my kids twigged to this quite quickly when playing with a toy problem).

Now the hard bit - encoding your input:

Categorical encoding using a single integer per board position could cause your model some grief (it is an input that "looks like" a real number, but of course it isn't, and values that seem numerically close may represent pieces with radically different abilities (perhaps the code for King is 1 and Queen is 2 and Bishop is 3, but all those pieces have such different attributes).
I would strongly consider thinking of each piece/player combo as a "colour channel" - a 64 cell grid where each value is either zero or one (so, one channel for White's pawns, another for White's knights, and so on).

Finally, those labels of yours: do think about what you'll do with drawn games - perhaps you have 3 possible outcomes, not 2?

We would all be fascinated to hear how you get on - I hope you'll write your work up in some form (and that you'll come back and complain/praise our advice, as appropriate!).


Is there a simple proof of the convergence of TD(0)?

As far as I know, there is no very simple proof of the convergence of temporal-difference algorithms. The proofs of convergence of TD algorithms are often based on stochastic approximation theory (given that e.g. Q-learning can be viewed as a stochastic process) and the work by Robbins and Monro (in fact, the Robbins-Monro conditions are usually assumed in the theorems and proofs).

The proofs of convergence of Q-learning (a TD(0) algorithm) and SARSA (another TD(0) algorithm), when the value functions are represented in tabular form (as opposed to being approximated with e.g. a neural network), can be found in different research papers.

For example, the proof of convergence of tabular Q-learning can be found in the paper Convergence of Stochastic Iterative Dynamic Programming Algorithms (1994) by Tommi Jaakkola et al. The proof of convergence of tabular SARSA can be found in the paper Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (2000) by Satinder Singh et al.

See also How to show temporal difference methods converge to MLE?.


Terminology for the use of datasets as data points

I don't think that this is anything new. Let's use your example of classifying an entire time series, say predicting word 1 vs word 2 for speech recognition. We can write out the data as a data frame like we would do with any other multivariate data: observations at time 1, time 2, etc as the predictors and the classification label as the response variable.

Each observation is a vector of the values at particular times for your subject, plus the label--no different than any other multivariate data. Sure, there might be special dependence structure because of the time series nature of your data, but you can still write it as a multivariate problem.

Okay, let's say that you hit the speech signal with a wavelet transform, resulting in an image-looking spectrogram of 2D data. Then just consider each "pixel" (time-frequency pair) to be a variable in your multivariate problem, along with the classification label. This is some kind of bijection between an $m\times n$ matrix and $\mathbb{R}^{n\times m}$.

You can extend this idea to 3D or 4D data (or higher), too. Just unwrap the high-dimension tensor in some kind of map $T^{m\times n \times \dots} \rightarrow \mathbb{R}^{m\times n \times \dots}$.


Further Normalization of Standardized data - ANN

Data scaling or normalization is a process of making model data in a standard format so that the training is improved, accurate, and faster.

So you just have to scale the data once. Doesn't matter what scaler you are using. Just make sure to initialize the scaler with the training data and then use the same parameters to scale the test data.

The z-score normalized data (MAX-ZS) can be used directly to train the network.


Does optimal policy implies deterministic?

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic optimal policy can not exist at the same time.
Suppose you have an MDP with one state and two actions $a_1$ and $a_2$, both yielding the reward 0 in expectation (as in your example). Then consider a policy that takes action $a_1$ with probability $\alpha \in [0,1]$ and $a_2$ with probability $1-\alpha$. Either of the two deterministic policies with $\alpha=0$ or $\alpha=1$ are optimal, but so is any stochastic policy with $\alpha \in (0,1)$. All of these policies yield the expected return of 0.
This is all assuming your optimality criterion is the expected cumulative (discounted) reward. If you have a different optimality criterion, such as something that accounts for risk, you might distinguish between rewards that have the same expected value but a different variance - but I think that is beyond the scope of the question.


Are neural networks really used apart from specific hi-tech organisations?

Before putting up my experience I'd like to provide some facts about AI startups and established Hi-Tech companies.

Most of companies claiming to use AI doesn't actually use ML/DL
And there are products by some companies that are classified to be using AI since they just use linear regression

Coming to my experience I can say that AI is passively as well as actively utilized in product development and services. Though Neural Networks are the driving force behind the current AI tech, the thing that we study in the text books are just basic building blocks that are used to designing complex network architectures. Though most of High-Tech companies as you have mentioned might not be willing to share their secrets, the publications they make in conferences are good enough. I would suggest one to go through the papers that are published by companies like NVIDIA, Microsoft, and Google to get understanding on how complex networks are designed to address specific problems.


What is the use of softmax function in a CNN?

The main purpose of the softmax function is to transform the (unnormalised) output of $K$ units (which is e.g. represented as a vector of $K$ elements) of a fully-connected layer to a probability distribution (a normalised output), which is often represented as a vector of $K$ elements, each of which is between $0$ and $1$ (a probability) and the sum of all these elements is $1$ (a probability distribution).

In the case of a classification task, the $i$th element of the vector produced by the softmax function corresponds to the probability of the input of the network of belonging to the $i$th class (e.g. a dog).


How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.


What is a RAM state in the gym's breakout-ram environment?

Yes, it is the state of the memory; this would mainly involve variables, since the code would be in ROM. Since it is only 128 bytes in size, the screen memory would also not be included in this.

The idea is that all information relevant to the game is captured in these 128 bytes; they represent the state of the game world at any given time. Movements of the ball, the game controller position, etc are all encoded there.

For machine learning it is not actually relevant which bytes are representing which value, as any optimisation of outcomes will treat all these 128 bytes as parameters describing the state. A machine learning system will pick up the optimal configuration through the learning process, eg that the 'racket' position should always near the ball x-coordinate. That will just be a correlation of two bytes, no matter which ones they actually are.


Is there a "better" (signal-based) language for artificial intelligence

What You need are other ways of knowledge representation, such as semantic networks or conceptual graphs. there you can define any possible relation between your entities. the knowledge of "x related to 4" exactly fits into "frames" and "semantic networks".
Jaynes in his book,discusses thoroughly what "plausibility" means and why we need to take into account weak syllogisms and start using probability theory as a platform for developing a (general) AI. this might also help with your "reasoning" phase (after you've developed your knowledge base)


In speech recognition, what kind of signal is used?

Not sure I understand what you mean by signal. However, incoming audio must be cleaned and processed for a machine to understand. Some people speak faster and others slower. First thing is to sample in the incoming audio. In other words use analog electic signals and store them as binary values at a set rate of sampling frequency.

See Nyquist Shannon theorem.

Now that it is sampled we need to clean it up and extract only the voice data. Effectively the sample is a set of values (frequencies) recorded over time. Or a signal :).
We can cut up this signal into many smaller components of a few milliseconds say 10ms. Each of the millisecond samples is further processed
A Fourier transform is generally used to break a signal into components, or frequencies that make up the signal. This is basic signal processing and we have not even entered machine learning yet.

Now we have a fingerprint of each 10ms sample these samples are then fed into a neural network. Samples are broken between frequency drops to create blocks of sound which may or may not be a word. These blocks are passed to the neural network.

A traditional feed forward neural network will not work as it forgets the data. So a recurrent neural network is used.
To train the neural network known words are tagged and encoded as a number. The neural network will output a number and the difference between this number and the expected number is the error. The error is then used to update the weights

Once we have the words, those can be further processed by other networks that specialise in natural language understanding


I will first recapitulate the key concepts which you need to know in order to understand the answer to your question (which will be very simple, because I will just try to clarify what is given as a "definition").
In logic, a formula is e.g. $f$, $\lnot f$, $f \land g$, where $f$ can be e.g. the proposition (or variable) "today it will rain".  So, in a (propositional) formula, you have propositions, i.e. sentences like "today it will rain", and logical connectives, i.e. symbols like $\land$ (i.e. logical AND), which logically connect these sentences. The propositions like "today it will rain" can often be denoted by a single (capital) letter like $P$. $f \land g$ is the combination of two formulae (where formulae is the plural of formula). So, for example, suppose that $f$ is composed of the propositions "today it will rain" (denoted by $P$) or "my friend will visit me" (denoted by $Q$) and $f$ is defined as "I will play with my friend" (denoted by $S$). Then the formula $f \land g = (P \lor Q) \land S$. In general, you can combine formulae in any logically appropriate way.
In this context, a model is an assignment to each variable in a formula. For example, suppose $f = P \lor Q$, then $w = \{ P=0, Q = 1\}$ is a model for $f$, that is, each variable (e.g. $P$) is assigned either "true" ($1$) or "false" ($0$) but not both. (Note that the word model may be used to refer to different concepts depending on the context; again, in this context, you can simply think of a model as an assignment of values to the variables in a formula.)
Suppose now we define $I(f, w)$ to be a function that receives the formula $f$ and the model $w$ as input, and $I$ returns either "true" ($1$) or "false" ($0$). In other words, $I$ is a function that automatically tells us if $f$ is evaluated to true or false given the assignment $w$.
You can now define $M(f)$ to be a set of assignments (or models) to the formula $f$ such that $f$ is true. So, $M$ is a set and not just an assignment (or model). This set can be empty, it can contain one assignment or it can contain any number of assignments: it depends on the formula $f$: in some cases, $M$ is empty and, in other cases, it may contain say $n$ valid assignments to $f$, where by "valid" I mean that these assignments make $f$ evaluate to "true". For example, suppose we have formula $f = A \land \lnot A$. Then you can try to assign any value to $A$, but $f$ will never evaluate to true. In that case, $M(f)$ is an empty set, because there is no assignment to the variables (or propositions) of $f$ which make $f$ evaluate to true.
A knowledge base is a set of formulae $\text{KB} = \{ f_1, f_2, \dots, f_n \}$. So, for example, $f_2 = $ "today it will rain" and $f_3 = $ "I will go to school AND I will have lunch".
We can now define $M(\text{KB})$ to be the set of assignments to the formulae in the knowledge base $\text{KB}$ such that all formulae are true. If you think of the formulae in $KB$ as "facts", $M(\text{KB})$ is an assignment to these formulae in $KB$ such that these facts hold or are true.
In this context, we then say that a particular knowledge base (i.e., a set of formulae as defined above), denoted by $\text{KB}$, is consistent with formula $f$ if $M(\text{KB} \cup \{ f \})$ is a non-empty set, where $\cup$ means the union operation between sets: note that (as we defined it above) $\text{KB}$ is a set, and $\{ f \}$ means that we are making a set out of the formula $f$, so we are indeed performing an union operation on sets.
So, what does it mean for a knowledge base to be consistent? First of all, the consistency of a knowledge base $\text{KB}$ is defined with respect to another formula $f$. Recall that a knowledge base is a set of formulae, so we are defining the consistency of a set of formulae with respect to another formula.
When is then a knowledge base $\text{KB}$ consistent with a formula $f$? When $M(\text{KB} \cup \{ f \})$ is a non-empty set. Recall that $M$ is an assignment to the variables in its input such that its inputs evaluate to true. So, $\text{KB}$ is consistent with $f$ when there is a set of assignments of values to the formulae in $\text{KB}$ and an assignment of values to the variables in $f$ such that both $\text{KB}$ and $f$ are true. In other words, $\text{KB}$ is consistent with $f$ when both all formulae in $\text{KB}$ and $f$ can be true at the same time.


Are tabular reinforcement learning methods obsolete (or getting obsolete)?

Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?

Individual problems don't get any more complex, but the scope of solvable environments increases due to research and discovery of better or more apt methods.
Using deep RL methods with large neural nets can be a lot less efficient for solving simple problems. So tabular methods still have their place there.
Practically, if your state/action space (number of states times number of actions) is small enough to fit a Q table in memory, and it is possible to visit all relevant state/action pairs multiple times in a relatively short time, then tabular methods offer guarantees of convergence that approximate methods cannot. So tabular approaches are often preferred if they are appropriate.
Many interesting, cutting edge problems that are relevant to AI, such as autonomous robots acting in the real world, do not fit the tabular approach. In that sense, the approach is "obsolete" in that it no longer provides challenging research topics for practical AI (there are still unanswered theoretical questions, such as proof of convergence for Monte Carlo control).
It is still worth understanding tabular value-based methods in detail, because they form the foundations of the more complex deep learning methods. In some sense they represent ideal solutions that deep RL tries to approximate, and the design of tabular solutions can be the inspiration for changes and adjustments to neural-network methods.


Snake game: snake converges to going in the same direction every time

There are two problems here.

The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 & 54. Using SKLearn's _fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.
Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays perfectly, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.

In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.

If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.


Why isn't my implementation of A2C for the the atari pong game converging?

Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4

In [3]: histories
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)

In [5]: histories
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]

So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]

In [8]: histories
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)

In [10]: histories
Out[10]: [deque([1]), deque([]), deque([]), deque([])]

The curve representing the mean reward looks like


Can machine learning help me digest asymmetrical order descriptions?

Yes a variant of NLP processing could help find the correct number to extract and type of object in this data.

Compared to the spreadsheet, the raw text data is ambiguous without understanding language to a reasonable depth, and without knowing the business context in order to extract the relevant information.

For instance, you are expecting to extract "soup pan" and "covered pan", but not "capacity pot". Also that parts of phrases such as "30 mm" or "10-quart" are lower importance qualifiers, and not specifying quantity of something.

The current state of the art for extracting this kind of data would be a bidirectional LSTM (a type of Recurrent Neural Network). You would likely get it to flag the parts of each entry that were relevant to the tabulated data you wanted to extract, then feed those into a simpler stage that put them into the spreadsheet. However, there are two caveats:

You need a lot of correctly-labelled training data to get reasonable performance. Using a word embedding layer, such as word2vec or GloVe, should significantly reduce the amount of training data required, but may require a careful pre-processing stage, and may be less useful when you have a lot of jargon in your data.
Performance is never perfect, and the system can still make stupid mistakes, because it does not truly understand the text is is dealing with. That applies to all ML approaches to this problem, and likely also to coding up an "expert system", although it may be easier to write the expert system to recognise when it had failed and ask for help.


I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.

An Emphatic Approach to the Problem
of Off-policy Temporal-Difference Learning gives a proof of stability.
At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book.

On Convergence of Emphatic Temporal-Difference Learning gives a proof of convergence.

I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.


What is a time-step in a Markov Decision Process?

In a Markov Decision Process (MDP) model, we define a set of states ($S$), a set of actions ($A$), the rewards ($R$), and the transition probabilities $P(s' \mid s, a)$. The goal is to figure out the best action to take in each of the states, i.e. the policy $\pi$.
Policy
To calculate the policy we make use of the Bellman equation:
$$V_{i+1}(s)=R(s)+\gamma \max _{a \in A}\left(\sum_{s^{\prime} \in S} P\left(s^{\prime} \mid s, a\right) V_{i}\left(s^{\prime}\right)\right)$$
When starting to calculate the values we can simply start with:
$$V_{1}(s)=R(s)$$
To improve this value, we should take into account the next action, which can be taken by the system and will result in a new reward:
$$V_{2}(s)=R(s)+\gamma \max _{a \in A}\left(\sum_{s^{\prime} \in S} P\left(s^{\prime} \mid s, a\right) V_{1}\left(s^{\prime}\right)\right)$$
Here you take into account the reward of the current state $s$: $R(s)$, and the weighted sum of possible future rewards. We use $P(s' \mid s, a)$ to give the probability of reaching state $s'$ from $s$ with action $a$. $\gamma$ is a value between $0$ and $1$ and is called the discount factor because it reduces the importance of future rewards since these are uncertain. An often-used value is $\gamma = 0.95$.
When using value iteration this process is continued until the value function has converged, which means that the value function does not change significantly when doing new iterations:
$$\left\|V_{i+1}(s)-V_{i}(s)\right\|<\epsilon, \; \forall_{s \in S},$$
where $\epsilon$ is a really small value.
Discounted sum of future rewards
If you look at the Bellman equation and execute it iteratively you'll see:
$$ {\scriptstyle  V(s)=R(s) + \gamma \max _{a \in A}\left(\sum_{a \in A} P\left(s^{\prime} \mid s, a\right)\left[R\left(s^{\prime}\right) +  \gamma \max _{a \in A}\left(\sum_{s^{\prime \prime} \in S} P\left(s^{\prime \prime} \mid s^{\prime}, a\right)\left(R\left(s^{\prime \prime}\right) + \gamma \max _{a \in A}\left(\sum_{s^{\prime \prime \prime} \in S} P\left(s^{\prime \prime \prime} \mid s^{\prime \prime}, a\right) V\left(s^{\prime \prime \prime}\right)\right)\right]\right)\right.\right. }$$
This is like (without transition functions):
$$R(s)+\gamma R\left(s^{\prime}\right)+\gamma^{2} R\left(s^{\prime \prime}\right)+\gamma^{3} R\left(s^{\prime \prime \prime}\right)+\ldots$$
To conclude
So when we start in state s we want to take the action that gives us the best total reward taking into account not only the current, or next state, but all possible next states until we reach the goal. These are the time steps you refer to, i.e. each action taken is done in a time step. And when we learn the policy we try to take into account as many time steps as possible to choose the best action.

You can find quite a large number of examples if you search on the internet, for example, in the slides of the CMU, the UC Berkeley or the UW.


What are the common myths associated with Artificial Intelligence?

As Artificial Intelligence is rapidly invading in our lives the myths around AI is also fabricating rapidly. Before getting into details one need to get clear off from this myths.

Myth 1: AI will take away our jobs:

Reality: AI is not completely different from other technologies and AI will not take away jobs but AI will change the way we work and helps us to increase the productivity by removing monotonous works.

Myth 2: Artificial intelligence will take over the world:

Reality: AI controlling the world. According to me it will not possible unless we give it that power. AI or robots will assist in our work and helps us to solve some tedious works that are difficult for human to solve easily.

Myth 3: Intelligent machines can learn on their own

Reality: It seems that a Intelligent machine can learn by it own. But the fact is that a AI Engineer or AI specialist should develop the algorithm and feed the machine with datasets and instructions and continuous monitoring should be done and most importantly regular update of software should be done.

Myth 4: Artificial Intelligence, Machine learning and Deep learning all three are same:

Reality: No not at all. To be clear machine learning is a part of AI and deep learning is the subset of ML. All three- AL, ML and DL are different but they are inter related with each other.


How to avoid running out of solutions in genetic algorithm due to selection?

There are multiple ways to interpret those steps. The most common standard approaches are

select two parents and produce two offspring; repeat until child population is the same size as parent population, and let the children replace their parents unconditionally (generational GA)

same as the above, but allow a few parents to live on instead of a few children if the parents have higher fitness (elitism)

each iteration, select two parents, produce one child, let the child replace a member of the parent population if it is better (steady state GA)

But there are other ways to go. There's an algorithm called CHC that lets the child population get smaller over time, and when it reaches zero, the algorithm triggers a smart restart. The point is there's no single definition for what makes an evolutionary algorithm. It's up to you to decide how to make something that works well for your problem. When you're a beginner though, it's handy to start from known points, like the three I mentioned above.


Yes, it has been demonstrated that the main factor for CNNs to work is its architecture, which exploits locality during the feature extraction. A CNN with random weights will do a random partition of the feature space, but still with that spatial prior that works so well, so those random features are OK for classification (and sometimes even better than trained ones, as they don't introduce additional bias).

You can read more in these papers:

D. Ulyanov et al. Deep Image Prior
A. Rosenfeld and J. K. Tsotsos. Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing.


Why don't integrated gradients explain samples correctly?

After several tests, I've opted for the following configuration:
Approximation mode: riemann trapezoidal
Steps: 1000
Baseline: average mean of each column (feature)
Although the global score remains good, there're important variations in the number of correctly explained local samples. For example, an execution returns 200 out of 390 samples explained as expected, but the following execution only 17.
Is there a reason for that? And, is there a way to make the algorithm predictable?


Could AI understand what the number one is and detect it in our real world same as AI can recognize cars and other things?

A understanding for this level of abstraction is technically possible.
It is not hard to create an AI able to count.
Unfortunately, this does not imply that the AI knows what 1 is.
It knows: This is 1 piece of cake and this is 1 sheet of paper.
But the idea of the number is not grasped yet.
Also, animals can count, but they do not understand why it works.

I would say that not even every human knows what 1 is:
The number, who is the neutral element of the multiplication.
The natural successor of 0.

I can say with certainty, that there is no AI, who reached this level of abstraction.


Is possible to train a robot or AI to prune fruit trees?

I am not an expert in robotics (definitely not in pear trees pruning either) but I will try to give some hints to partially answer and also to help reframe the problem a bit. On overall I'll give already an answer: it is most likely possible, but also most likely not convenient.
Problem statement
First things first: in general the rule is that machine learning should be applied when a task can't be automatised otherwise. Before asking if an AI can be trained to solve a task one should always think how to automatise a task using a rule based system. This is important especially because during the process of thinking how to automatise the task you will realise that some steps can be perform without an expert, while others can't be performed without any supervision. Let's brake down your task in subtasks: a system that prune trees should be (at least) capable of:

Moving between trees
Selecting branches to cut
Cut the selected branches

Selecting the branch to cut is probably the step that requires most of the know how and expert supervision and for which a machine learning component might be suitable. Moving instead is a perfect example of a subtasks that could be tackled at different levels. Creating a machine able to anticipate other objects movements and avoid them in real time definitely require to train an AI component, but when you say that the environment is highly structured (trees disposed in a grid) this make me think that maybe some hand coded rules would do the trick without bothering machine learning.
Theoretical tools
Once you have understood which subtasks your machine should be capable of solving, you can start dig on the theoretical feasibility of them. Following the same order as in the previous paragraph:

Self-driving is a widely studied topics, the algorithms used to train robotic vacuum cleaners to move automatically in a house could be applied straight away to the problem of training an agent to move between trees.

Selecting the right branches involves mostly computer vision. This sub-task should actually be also dived again into sub tasks: detecting branches from other objects and select which ones should be cut. Nevertheless, the field is quite huge and training two models able to perform both actions is, in my opinion, doable.

Cutting a branch is probably harder than driving in this situation, because of the small movements that might be required to reach branches positioned in difficult spots. Anyway, it is possible to train robots to perform fine-grained movements (for a funny example see: Robot learn to flip pancakes).

Again it depends also on how high your expectation are about the final machine/system. Should the system have a surgical precision or could it risks to brake few extra branches in hard situations? Obviously the higher the expectations the harder it would be to make everything work harmoniously.
Resources required
Last but not least you also need to understand if what you're trying to do is feasible in reality and not just in theory. A big problem when it comes to use machine leaning is that these models can be trained only with huge amount of data.

Train an artificial agent to move in an environment can be done by reproducing the environment in an artificial simulator, which is good news. A single guy with a laptop could potentially do the job.

Collecting data to train a model able to detect branches on photo and then select which one should be cut will be highly tedious, and also expensive because data need also to be labelled by experts. Which means that some experienced people will have to take photos, in the order of tens of thousands at least, and write for each photo (using a software) which are the branches they will cut if they were working for real on that tree. I strongly doubt that a dataset like this already exists.

Training a robotic arm to cut branches will also be challenging. Despite the fact that also in this case simulators could be leveraged, the task is inherently harder and this comes with bigger difficulties (for example in designing a proper reward function if using reinforcement leaning). Concretely this mean more time to spend in research and testing.

Consider also that the success of the final model trained for each subtask would be not guaranteed at all, reason why I said at the beginning that training a system with AI modules would be probably not convenient, and that the best thing is always to try to create a rule based system first.


How does L2 regularization make weights smaller?

Here is my take.
The larger the $\lambda$, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example:
\begin{align*} \theta_j := \theta_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \right] &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align*}
\begin{align*} \theta_j := \theta_j (1- \alpha \frac{\lambda}{m})\left( \frac{\alpha}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align*}
From this derivation, it is clear that at every update the coefficients get reduced by a factor that is usually a little less than 1 and directly proportional to $\lambda$, so, as $\lambda$ gets bigger, the weights get reduced more and more; eventually, for very big values of $\lambda$, we risk to totally underfit the data, since only the regularization term will remain in the cost function and all the weights will go to zero.
This is for linear regression, but it is essentially the same logic also for logistic regression.
This is taken from Andrew Ng's course on Coursera. A more mathematical precise (and complex) traction of the problem can be found in the Bloomberg machine learning course material.
PS: in the derivation of the update rule for gradient descent, the $\lambda$ should be divided by the number of training examples, this is important in choosing the right $\lambda$ because we want this relationship to be inversely proportional, otherwise the coefficient won't be decreased.


Last semester I, along with my team, made a project on OCR.
Note: I am assuming the data set for your pictures has white
background with black (or some other dark text) on it.
These are the overview steps we followed:

Pre-processing includes grayscale conversion, noise reduction,
binarization and skew detection.
Next step was segmentation. This process extracts the individual
characters from the image. Histogram taken along the y-axis divided
the image into lines. This is followed by histogram along the x-axis
which divided them into words and further into characters.
At the end of the step, we used Savgol filter to smooth the curves of the histogram.
Next step was feature extraction. This is the most important step.
The accuracy of your code depends on how well your features are.
We used the following features:

Crossing: Counting number of transitions between foreground and background. We used
two diagonal lines, two horizontal and one vertical line. You can used any number you want.
Zoning: Whole character region is divided into 16 zones, and density of each zone is measured.
Projection Histogram: Each character has unique (almost) vertical and horizontal histogram signature.
Other features include number of endpoints in the character, number of loops and horizontal/vertical line count.

We used three different classification algorithms for our project. They were KNN (K-Nearest Neighbours), Artificial Neural Network (ANN) and Extra Tree classification. Their F1 score was 0.84, 0.82 and 0.77 respectively.
For training, you will need to find datasets. Many data sets for OCR was available online. Make sure you are using good ones.


Wouldn't an AI that specializes in making other AI be an AGI if they can cooperate?

If the AI can indeed assess arbitrary scenarios and come up with solutions to handle them, then it would indeed be an AGI.

  What’s the constraint ?

It doesn't exist.  Current programmers are very good at developing AI that can handle specific tasks ("narrow AIs"), but it is currently impossible to build an AI that can assess and handle "general" situations (unlike your proposed algorithm, which possess that capacity).

Theoretically, we can have a program that can build other programs (genetic algorithms are arguably one such example), but handling arbitrary scenarios and problems requires a form of "general intelligence", which we don't know how to program. Therefore, we can't build this machine.

It's possible that we can built this machine, but we must first figure out  the hard problem of "general intelligence". We're nowhere near reaching that level.

If we figure out how to program "general intelligence", then it should be fairly simple to use your approach (building an AGI to assess scenarios and then build "narrow AIs" that can handle Those scenarios). Only then we can understand the AGI's limitations and weaknesses, and be able to identify probable constraints to its power. For example, it's possible that such an AGI may be slow in handling arbitrary scenarios and developing the "narrow AIs"...in which case, it may take an absurdly long period of time to develop "a suite of AIs powerful/specialized for their tasks".

But until we build the AGI itself, we won't be able to identify its faults or weaknesses. Going beyond that would be science-fiction speculation.


How can I merge outputs of two separate layers so that the overall performance improves?

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options.

I see a little thing that might be making your model a little worse, though:

You're concatenating "relu" with "sigmoid".

Placing two different nature values in the same array may make it more difficult for updating weights properly.

A few independent suggestions:

Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).
Instead of concatenating, you can try Multiply()([img_output, numeric_output]), in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".

Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch.