At which point we have to stop post pruning in decision tree?

There are a variety of conditions we can use when deciding whether to prune a sub-tree or not after generating a decision tree model. There are three common approaches.

We can prune branches with less support than a specific threshold. These are branches which were constructed using very few points from the training data.
We can prune branches where the information gain from a split (or any other splitting measure we are using, like GINI), is smaller than a threshold.
You can do what is done in Quinlan's 4.5 & C5.0 learners (which are the standard approaches; J48 is another implementation of the same algorithm). Quinlan performs a Chi-squared-like test for the relationship between the attribute we split upon and the target attribute. If the relationship is statistically significant, then the split is preserved. If not, it is not. The "confidence factor" parameter found in most implementations of these algorithms corresponds to the $\alpha$ value used in determining whether the relationship is considered significant. This approach captures the idea that we should prefer to keep branches that have few datapoints, but a very strong signal, or that have a weak signal, but very many datapoints supporting the pattern, since both cases are less likely to be overfitting that cases where we have weak signals and small numbers of points.


How to prove that gradient descent doesn't necessarily find the global optimum?

You can find by yourself a counterexample that, in general, GD is not guaranteed to find the global optimum!

I first advise you to choose a simpler function (than the one you are showing), with 2-3 optima, where one is the global and the other(s) are local. You don't need neural networks or any other ML concept to show this, but only basic calculus (derivatives) and numerical methods (i.e. gradient descent). Just choose a very simple function with more than one optimum and apply the basic gradient descent algorithm. Then you can see that, if you start gradient descent close to one local optimum (i.e. you choose an initial value for $x$ or $\theta$, depending on your notation for the variable of the function) and then you apply gradient descent (for some iterations), you will end up in that close local optimum, from which you cannot escape, after having applied the gradient descent steps.

See also the question Does gradient descent always converge to an optimum? and For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?


One hot encoding vs dummy variables best practices for explainable AI (XAI)

Personally I would chose one hot encoding as it is statistically correct/model friendly. Moreover, you can always provide additional help/tools to aid explainability. Lastly even if you add the nth column, you still need some idea about the working of model(and the boundaries it created while training) to interpret the result.


Aren't all discrete convolutions (not just 2D) linear transforms?

Convolution is a pretty misused term in recent times with the advent of CNN.

Short answer, Convolution is a linear operator (check here) but what you are defining in context of CNN is not convolution, it is cross-correlation which is also linear in case of images (dot product).

Convolution:

Computers work only on discrete values, so the discrete time convolution is defined similarly as:
$$
y[n] = \sum_{k=\infty}^{\infty} x[k] h[n-k]
$$

which has the nice property of

$$Y(e^{j\omega}) = X(e^{j \omega})H(e^{j \omega})$$

where each $A(e^{j\omega})$ is the Fourier Transform of $a(t)$
So this is the basic idea of discrete convolution. The point to illustrate here is that convolution operation has some nice properties and very helpful.

Now there are 2 main types of convolution (as far as  I know). One is linear convolution another is circular convolution. In terms of $2D$ signal they are defined in the following ways:
$$y[m,n] = \sum_{j=-\infty}^{\infty}\sum_{k=-\infty}^{\infty} x[i,j]h[m-i,n-j]$$

Circular convolution is the same except that the input sequence is finite from $0$ to $N-1$ giving rise to periodicity in frequency domain. The convolution operation is a little bit different due to this periodicity.

So these are the actual definitions of convolution. It has linearity property (clearly obvious since it is used to calculate output of LTI system), and it also has the linearity in terms of matrix multiplications, since we want to do the computer to do these calculations for us. I don't know a lot but there are many clever manipulations e.g the FFT algorithm an indispensable tool in signal processing (used to convert signals to frequency domain for a certain sampling rate). Like this you can define convolutions in terms of Hermitian matrices and stuff (only if you want to process in the $n$ domain it is much easier to process in the frequency domain).

For example a $1D$ circular convolution between 2 signals in $n$ domain defined in matrix form, can be written as follows $y_n=h_n*x_n$:
where
$$y_t = \begin{bmatrix} y_0 \\
                      y_1 \\
                      y_2 \\
                      y_3
       \end{bmatrix} = \begin{bmatrix}
h_0 & h_3 & h_2 & h_1\\
h_1 & h_0 & h_3 & h_2\\
h_2 & h_1 & h_0 & h_3 \\
h_3 & h_2 & h_1 & h_0
\end{bmatrix}  \begin{bmatrix} x_0 \\
                      x_1 \\
                      x_2 \\
                      x_3
       \end{bmatrix}$$

The same can be done quite easily by converting the functions into frequency domain, multiplying and converting back to time domain.

When we move to $2D$ domain similar equation is formed except (in $n$ domain)instead of $h_0$ we will have a Hermitian matrix $H_0$ and we will perform the Kronecker product (I don't know the justification or proofs of these equations, it probably satisfies the convolution theorem and is fast when ran on computers). Again much easier to do in frequency domain.

When we move to multiple dimensions, it is called Multidimensional discrete convolution as per this Wikipedia article. As the article suggests the property where everything is in frequency domain.
$$Y(k_1,k_2,...,k_M)=H(k_1,k_2,...,k_M)X(k_1,k_2,...,k_M)$$
still holds good. When we do convolutions in the $n$ domain things get tricky and we have to do clever matrix manipulations as shown in the example above. Whereas, as stated above things get much easier in the frequency domain.

Its counter-intuitive that a picture has a frequency domain, and in general its actually not frequency. BUT in DSP we use a lot of filters whose math properties are similar to those of filters in the traditional sense of frequency and hence has the same calculations as in a frequency setting as shown in the 1st example of $1D$ signal.

The point convolution by definition is a linear operation. Check here if the explanations are unconvincing. There are Linear Time Varying systems and its output maybe determined by convolution, but then it uses the equation at a certain point of time i.e:
$$Y(e^{j\omega}, t) = H(e^{j\omega}, t)X(e^{j\omega} )$$.

Now whether it can be represented by matrix products in $n$ domain or not I cannot say. But generalising, it should be possible, only it will involve increasingly complex matrix properties and operations.

Cross Correlation:

Cross correlation is nothing similar to convolution, it has a bunch of interesting properties on its own, hence I do not like the two being mixed together. In signal processing it is mainly related to finding energy (auto-correlation) and has other applications in communication channels.

While in statistics, it is related to finding correlation (misused term since it can take any value, but correlation coefficient takes value between -1 to 1) between 2 stochastic processes, at different time instants, where we have multiple samplings of the signal or ensembles of the signal based on which the expectation is taken (See here). In CNNs maybe a way to see this would be, the image dataset is a ensemble of stochastic process, while the filter is another process which is fixed. And moving the filter over the image is time delay.

In the context of digital image processing cross correlation is just a dot product. And hence it can be represented by matrices. Whether cross correlation is linear or not is difficult to tell (at least I don't know), but in the context of image processing it seems linear. Bottom-line is that it can be definitely implemented by matrices as it is a dot product, whether it is linear in true mathematical sense is doubtful.

The more intuitive way in context of CNNs would be to view filters as just a bunch of shared weights for better regularisation instead of cross correlation or convolution.


Why do I get a straight line as an output from a neural network?

You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.


How much power might the creator of the first Artificial General Intelligence obtain?

"Artificial Intelligence" has existed for many, many years.  Decades even.  Go back to Arthur Samuel and his Checkers playing program (if not before).  That's AI.

The thing is, as soon as computers can do something, people want to quit treating it as AI.  AI is almost, by definition, something that can't really exist, because it has to be about something humans can do, that computers can't.   So all progress made in the field is reclassified as "not AI" as soon as it does anything useful.

A more salient question might be "who will be the first to announce that Artificial General Intelligence" has been created.   AGI will, if it ever happens, be fundamentally different in terms of being able to learn to do anything, as opposed to, say, a Checkers program which can only play Checkers, or a Go program that can only play Go, etc.

And the answer to that question is -- nobody knows.  Nobody even know for sure that AGI will ever happen (though a lot of us presume that it will).  Anyway, there isn't a ton of research on AGI going on, relative to the amount of research in Machine Learning / Narrow AI.  But somebody working in their garage could make a huge breakthrough tomorrow for all we know.  I think a lot of people assume that creating this will take Nation-State level resources, but that hasn't really been proven.  So it might be some kid in a garage in Omaha tomorrow, or it might be the Chinese government in 2074.  There's just no way to really know.


Are there neural networks with 3-dimensional topologies?

Yes. Convolutional neural networks are usually 3-dimensional. In fact, they usually deal with images (e.g. RGB images), which can already be 3-dimensional.


What are the advantages of RL with actor-critic methods over actor-only methods?

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?

One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners, such as genetic algorithms, require episodic problems. The smallest unit those can learn from is an entire episode. That is because without a critic providing value estimates, the only way to estimate return is to sample an actual return from the end of an episode.
A TD-based critic may also have lower variance, which can aid in fast learning and stability, although this is not always a benefit. TD-based critics also have bias, which can cause instability. See Why do temporal difference (TD) methods have lower variance than Monte Carlo methods? on Cross Validated for more details on this.
In practice, RL algorithm choice is a hyperparameter. As well as impacting difficulty of implementation, CPU and other resource costs, it can impact how well learning occurs depending on the problem being attempted. Usually, the only way you can tell a method is better for your problem is to try all the valid ones and measure their performance.

I think it's effective to use only actors, especially for sparse rewards. Is that correct?

The sparsity of reward is not a major factor here. The hard credit assignment problem that this brings into play means that the agent has to either assign a value, or pick an action, in states which have no direct feedback. All else being equal, both the value function and optimal action choice are hard to resolve when they depend on a large number of future policy decisions and state transitions which may vary.
Which approach is best will depend on whether it is easier for a statistics-based learner to approximate the mapping from state to expected future reward, or to the action. The two functions can have different degrees of complexity.
For an example where a policy function (for an actor) is simpler than a value function (for a critic or value-based method), you could consider a simple chase environment where a wolf tries to catch a rabbit on a simple continuous plane surface. The agent is the wolf, and gets a reward of +1 for catching the rabbit (I won't bother with other details, there are plenty of variations you could make).
In the example environment, a simple strategy for the wolf is to turn to face the rabbit and move forwards. This is easy to map from current locations and facings of the wolf and the rabbit - for ultimate ease you can express the state as the rabbit's position and velocity in polar coordinates from the wolf's perspective. Compare that to the value function - it has to predict the time it will take to reach the rabbit given a current action choice. This is a far harder function to express based on the state, and as a result may also be harder to learn.


Your interpretation is quite correct. I could not understand how it would speed up the convergence though. What they are doing is basically re-assigning the magnitude of the weight vector (also called norm of the weight vector).

To put things in the perspective, the conventional approach to any Machine Learning cost function is to not only check the variations of the error with respect to a weight variable (gradient) but also add a normalization term which is $\lambda (w_0^2 + w_1^2 + \dots)$. This has got a few advantages:

The weights will not get exponentially high, even though if you make some mistake (generally bouncing off to exponential costs due to the wrong choice of learning rate).
Also, the convergence is quicker somehow (maybe because you have now 2 ways to control how much weight should be given to a feature. Unimportant features weights are not only getting reduced by normal gradient, but also the gradient of the normalization term $\lambda (w_0^2 + w_1^2 + \dots)$).

In this paper, they have proposed to fix the magnitude of the weight vector. This is a good way, although I am not sure if it is better than feature normalization. By limiting the magnitude of weight to $g$, they are fixing the resource available. The intuition is that, if you have 24 hours, you have to distribute this time among subjects. You'll distribute it in a way such that your grade/knowledge is maximized. So this might be helping in the faster convergence.

Also, another intuition would be, when you are subtracting the gradient from a weight vector you use a learning rate $\alpha$. This decides by how much weight-age of error you want to give which will be subsequently be subtracted from the weights. In this approach, you are not only subtracting the weights but also using another learning rate $g$ to scale the weight. I call this $g$ a learning rate because you can customize it which in turn customizes the value of weights which in turn affects the future reductions of weight gradient descent.

I am sure someone will post a better mathematical explanation of this stuff but this is all the intuition I could think of. I would be grateful if other intuitions and mathematical subtleties are pointed out. Hope this helps!


How to implement AI strategy for Mastermind

You could possibly apply neural networks, reinforcement learning to summarise results of previous choices (what you are calling context) and use score predictions to suggest the next turn's guess. However, the game of Mastermind has a small search space and it is possible to process this "context" more directly by refining a set of guesses. This will be much more efficient and simpler to understand than a neural network approach. It would be very hard to make a neural network variant which was as efficient - either in terms of CPU time, or in terms of number of turns it takes to find a solution.

In practice, a Mastermind solver is much like a Hangman solver, or a Guess Who? solver. You have an initial large set of all possible answers, and need to narrow it down to a single correct answer. You do this by processing after each guess to reduce the set of answers that meet all the constraints that the game has given you so far.

The agent needs to know the score function that compares a target value with the guess and returns the score. Let's call that score(guess, target)

The algorithm looks like this:

(Opponent sets unknown_target)
Initialise possible_answers as list of all valid targets

For each turn:
  Select one of possible_answers as this turn's guess
  Ask opponent for gscore = score(guess, unknown_target)
  If gscore is [0,4] then exit(win)
  If it was last guess then exit(lose)
  For each possible_answer in possible_answers:
    pscore = score(guess, possible_answer)
    If pscore != gscore, then remove possible_answer from possible_answers

You can finesse this for the stage Select one of possible_answers as this turns guess by trying to optimise either by psychological model of opponent or trying to find choices that are likely to cause the best reduction in size of possible_answers. However, a simple random choice should do quite well.

Also worth noting is that the algorithm does not depend on the exact nature of the scoring function, so it is applicable for many variations of guessing games. It does rely on the score providing information that will reduce the remaining set of guesses. In some games that may mean taking more care about the precise nature of a guess, in order to maximise this effect.

Out of interest, I implemented this algorithm and tested it when there were 10 choices at each position (i.e. digits 0 to 9), maximum of 10 guesses allowed, and the target to guess was set randomly. Using random guesses and the algorithm exactly as written, the above approach guessed correctly 9,996 times out of 10,000, and on average the guesser won the game in 6.2 turns.


What will happen when you place a fake speedsign on a highway?

https://www.technologyreview.com/s/530276/hidden-obstacles-for-googles-self-driving-cars/

  Google’s cars can detect and respond to stop signs that aren't on its map, a feature that was introduced to deal with temporary signs used at construction sites. But in a complex situation like at an unmapped four-way stop the car might fall back to slow, extra cautious driving to avoid making a mistake.

It's highly probable they would slow down with current technology, as they can detect temporary signs and are designed to use slow speed in complex cases.

If it was a true temporary sign (e.g. road repair), how can it make the distinction? It probably would be worse to ignore a slow down sign than slow down at a fake one.

IMHO, the problem there is with the joke in the first place, as some humans might slow down too.


Is a GPU always faster than a CPU for training neural networks?

This changes according to your data and complexity of your models. See following article by microsoft. Their conclusion is

  The results suggest that the throughput from GPU clusters is always
  better than CPU throughput for all models and frameworks proving that
  GPU is the economical choice for inference of deep learning models.
  ...

  It is important to note that, for standard machine learning models
  where number of parameters are not as high as deep learning models,
  CPUs should still be considered as more effective and cost efficient.

Since you are training MLP, it can not be thought as standard machine learning model. See my preprint, The impact of using large training data set KDD99 on classification accuracy. I compare different machine learning algorithms using weka.

As you can see from above image, MLP takes 722 minutes to train while Naive Bayes ~2 minutes. If your data is small and your models parameters are not high, you will see better performance on CPU.


How does the regression layer in the localization network of a spatial transformer work?

Localization network is not trained separately on special transform of input. It's just a part of feed forward network which is trained as a whole, with normal backpropagation.
It's just part of the network which affect the loss function. As in any backpropagation loss function propagate back gradient, which backpropagate through final part of the network, backpropagate through differentiable sampler(that is non-trivial part, which use transfromation produced by localization subnetwork) and after that backpropagate into localization part
Whole approach of spatial transformers could be in doubt now. There were some anecdotal evidences that it was not working on less trivial tasks (private communications, was not working for me either)


Can artificial intelligence also make mistakes?

Since the question was tagged with philosophy, I'll provide a philosophical perspective.

There is no definition of perfection. Just think about the silly example that is always brought up when we talk about autonomous cars: You are about to crash into a group of school kids and the AI's only chance to avoid them is by crashing into an old person. What should the AI do in order to react "perfectly"?

All actions that we take are based on our personal value system, so what is perfect to one person might be completely nonsense to another person. It might be possible to create an AI that always acts perfectly in terms of the value system you fitted it out with, but even in this case there are some caveats like:

reaction time: is the system fast enough to process all available information fast enough in order to take a fast action?
available information: not all information might be available with the system. Do you consider it acting imperfect, if the action of the system turns out to be wrong once you posses all information about the situation, even though some information will never be available to the system (e.g. Heisenberg's principle of uncertainty)


Effect of batch size and number of GPUs on model accuracy

This should make a difference, but how big is the difference heavily depends on your task. However generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but have a higher speed in batch/minutes. If the batch size is too small the batch/minute will be very low and therefore decreasing training speed severely. However a batch size too small(for example 1) will make the model hard to generalize and also slower to converge.

Source
This slide is a great demonstration of how batch size affects training. As you can see from the diagram, when you have a small batch size the route to convergence will be ragged and not direct. This is because the model may train on an outlier and have it's performance decrease before fitting again. Of course this is an edge case and you would never train a model with 1 batch size. However on the other hand with a batch size too large, your model will take too long per iteration. With at least a decent batch size (like 16+) the amount of iterations needed to train the model is similar, so a larger batch size is not going to help a lot. The performance is not going to vary a lot.
In your case, the accuracy will make a difference but only minimally. Whilst writing this answer, I have ran a few test on batch size effect on performance and time, and here is the results. (Results to be added for 1 batch size)
Batch size 256 Time required 98.50849771499634s : 0.9414
Batch size 128 Time required 108.53689193725586s : 0.9668
Batch size 64 Time required 129.92272853851318s : 0.9776
Batch size 32 Time required 162.13709354400635s : 0.9844
Batch size 16 Time required 224.82269191741943s : 0.9854
Batch size 8 Time required 351.2729814052582s : 0.9861
Batch size 4 Time required 514.2667407989502s : 0.9862
Batch size 2 Time required 829.1623721122742s : 0.9869

You can test out yourself here.

As you can see, accuracy increase with the batch size decreasing and this is because a higher batch size means it will be trained on less iterations. 2x batch size = half the iterations so this is expected. The time required have risen exponentially, but batch size of 32 or below don't seems to make a large difference in time taken. The accuracy seems to be normal as half the iterations are trained with double the batch size.
In your case I would actually recommend you stick with 64 batch size even for 4 GPU. In the case of multi-gpu, the rule of thumb will be using at least 16 or so batch size per GPU, as if you are using 8 or 4 batch size the GPU cannot be completely utilized to train the model.
For multiple GPU accuracy impact, there is virtually no impact on accuracy as far as I know. Multiple GPU just take it's own slice of batches and computes it and feed the output to one GPU to calculate the loss, and the operations done across the GPUs are the same operations done on 1 GPU, so it shouldn't matter.
EDIT:
For multiple GPU, there might me a slight difference due to percision error as told by @Posi2 . Please see here. Thanks @Posi2
Conclusion
Batch size don't matter to performance too much as long as you set a reasonable batch size (16+) and keep the iterations not epochs the same. However training time will be affected and for multi-gpu you should use the minimum batch size for each GPU that will utilize 100% of the GPU to train. 16 per GPU is quite good.


Is color information only extracted in the first input layer of a convolutional neural network?

Neural networks are all about taking raw input data (RGB values and pixel location) and learning useful features that are relevant to some downstream task. This process of aggregating raw inputs into higher-level features can start at the first layer past the inputs.

So yes, only the first layer of the network is using the actual raw color information from the image. Beyond that, the network has already started to put together nearby pixels and disparate color channels in order to find more complex patterns. Deeper layers in the neural network typically do further aggregation on features learned in earlier layers, rather than taking raw color information as input.


Are these two TRPO objective functions equivalent?

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.

First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with that time index). This is definitely something undesirable. If you choose to add the importance sampling term, then you will make really small most of the terms if $\gamma$ is not so close to 1.

Second, while the original objective is very nice to obtain theoretical results, you may ask yourself if that objective is really what you want to maximize. Do you really care more about the performance at the beginning of the trajectory than at the end?

While I have no rigorous proofs, I like to think that a better definition for the expected discounted reward is a time average:

$$\eta(\pi) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\mathbb E_{s_k,a_k,s_{k+1},\cdots}\left[\sum_{t=k}^\infty\gamma^{t-k}r(s_t)\right].$$

Since each term of the average satisfies the original proofs, the only difference with the equation 14 is that the density is $\tilde\rho_{\theta_{old}}(s) = \lim_{N\to\infty}\frac{1}{N}\sum_{k=1}^N\rho_{k,\theta_{old}}(s)$, where $$\rho_{k,\theta_{old}} = P(s_k = s)+\gamma P(s_{k+1}=s) + \gamma^2 P(s_{k+2}=s) + \dots.$$
You can notice that for a large $N$ you basically give the same importance to all of the time-steps. So, the usual implementations actually optimize some complicated function similar to the one I defined, where you give a similar weight to all samples, irrespective of their corresponding time-step.


Should I apply ReLU to non negative output?

Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.

What you may wish to do is apply $\ln(|\vec{x}|)$, $\arctan(x[0], x[1])$, ..., $\arctan(x[0], x[N-1])$, where $N$ is the number of dimensions to $\vec{x}$, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.


PPO agent for vehicle control does not learn to stop at traffic lights

The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.
It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.


What kind of hardware and software are required to build an AI system?

Many different components may go into building an AI system. AI is a broad topic encompassing, machine learning, statistical learning so an AI system could involve anything from a raspberry pi to a spoon. Your title refers specifically to processors of an AI system. The mathematics behind AI can be run on any processor. GPU, CPU, FPGA, and new Neural network chips. A neural network can be run on a Window PC a linux PC AI packages are not OS dependent. I can train a model on my laptop now.

2019-11-14 20:10:57.503229:
Epoch 1/5
60000/60000 [==============================] - 8s 137us/sample - loss: 0.2925 - acc: 0.9141
Epoch 2/5
60000/60000 [==============================] - 8s 140us/sample - loss: 0.1426 - acc: 0.9574
Epoch 3/5
60000/60000 [==============================] - 9s 142us/sample - loss: 0.1059 - acc: 0.9674
Epoch 4/5
60000/60000 [==============================] - 8s 135us/sample - loss: 0.0862 - acc: 0.9732
Epoch 5/5
60000/60000 [==============================] - 7s 112us/sample - loss: 0.0744 - acc: 0.9763

Just like any computer application an AI needs a processor and memory to store information. Beyond that it is difficult what specifically you are looking for.

A configurable cloud-scale DNN processor for real-time AI


Converging to a wrong optimal policy if the agent is given more choices

Reinforcement Learning is really fun because the agent will find any bug in your implementation and will exploit it.
>>> take_left(0)
0
>>> take_left(1)
-4

The agent figured out your bug with negative values and exploits negative indexing to get to the target faster.


Why is having low variance important in offline policy evaluation of reinforcement learning?

Having low variance is important in general as it reduces the number of samples needed to obtain accurate estimates. This is the case for all statistical machine learning, not just reinforcement learning.

In general, if you are estimating a mean or expected quantity by taking many samples, the variation in the error is proportional to $\frac{\sigma}{\sqrt{N}}$ for a direct arithemtic mean of all samples, and behaves similarly for other averaging approaches (such as recency-weighted means using a learning rate). The bounds on accuracy can be made better by either increasing $N$ i.e. taking more samples, or by decreasing the variance $\sigma^2$.

So anything you can do to reduce variance in your measurements has a direct consequence of reducing the number of samples required to achieve the same degree of accuracy.

In the case of off-policy reinforcement learning, there is added variance - compared to on-policy learning - due to different probabilities of taking an action in behaviour and target policies. This is due to the need to adjust reward signals using importance sampling - multiplying by the importance sampling ratio will make the reward signal vary more (in fact it can become unbounded). This is not really any more of a challenge than any other source of variance, but as it interferes with the goal of speedy learning, a lot of research effort has been put into methods that reduce the variance.


Can the normalization factor for the belief state update be zero?

I think that the normalisation factor is assumed to be non-zero. So, in practice, I guess, you must eventually check that $P(z \mid b, a)$ is non-zero (even though, I guess, it will likely never be zero because of round-off errors in computers).

The formula to calculate $b'(s')$ comes from its definition, which is based on Bayes' theorem, where the denominator is assumed to be non-zero (in general).

The definition of $b'(s')$ is $P(s' \mid z, a, b)$, that is, the new belief $b'$ of being in state $s'$ is defined as the probability of landing in the next state $s'$, given that we have observed $z$, have taken action $a$ from the previous state $s$ and we had the previous belief $b$. We will expand this definition, but first let us recall a few probability definitions.

Recall that $P(A, B) = P(A \mid B) P(B) = P(B \mid A) P(A)$, where $A$ and $B$ can actually be multiple events (that is, $A$ could actually be the intersection of multiple events). In other words, suppose we want to calculate $P(A, B, C)$, we can actually consider e.g. $B$ and $C$ as one event. Let $(B \cap C) = (A, B) = D$ (note that the notation $(A, B)$ means the "intersection" of events $A$ and $B$, in the case $A$ and $B$ are events). Then

\begin{align}
P(A, B, C)
& = P(A, (B, C)) \\
&= P(A, D) \\
&= P(A|D)P(D) \\
&= P(A|B, C)P(B, C) \\
&= P(A|B, C)P(B|C)P(C) \\
\end{align}

In general, this idea generalises to more variables/events.

Note also that $\frac{P(A, B)}{P(B)} = P(A \mid B)$.

At this point, we are prepared to expand $P(s' \mid z, a, b)$ and understand its expansion.

We can expand $P(s' \mid z, a, b)$ as follows

\begin{align}
P(s' \mid z, a, b)
&= \frac{P(s', z, a, b)}{P(z, a, b)}\\[0.7em]
&= \frac{P(s', z, a, b)}{P(z \mid b, a)P(a|b)P(b)}\\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b) P(a | b) P(b)}{P(z \mid b, a)P(a|b)P(b)} \\[0.7em]
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)}
\end{align}

It then turns out that (I will maybe explain this more in detail later)

\begin{align}
P(s' \mid z, a, b)
&= \frac{P(z \mid s', a, b) P(s' \mid a, b)}{P(z \mid b, a)} \\[0.7em]
&=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)} \\[0.7em]
&=b'(s')
\end{align}

So, by assumption, $P(z \mid b, a) = \sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z)$ must be different from zero for the equality above to hold.

You can see this from a very simply example of the Bayes' theorem. Let $P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}$. Now, intuitively, $P(A \mid B)$ (which is what we want to calculate using Bayes' theorem) means the probability of $A$ occurring given that $B$ has occurred, which means that $B$ couldn't have had a probability of $0$ of happening if we wanted to calculate $P(A \mid B)$, so $P(B)$ couldn't have been zero if we wanted to calculate $P(A \mid B)$ using Bayes' theorem. We can also apply this reasoning to the definition of $b'(s')$ above.

For completeness, note also that, in the normalisation factor $$\sum_{s \in S}b(s)\sum_{s' \in S} T(s, a, s')O(a, s', z),$$ $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ are probability distributions, which means that not all terms of $b(s)$, $T(s, a, s')$ and $O(a, s', z)$ can be zero, for all $s$, $s'$ and $a$ (given they must sum up to $1$).

Note also that $\sum_{s' \in S} T(s, a, s')O(a, s', z)$ is a convex combination of all $O(a, s', z)$ (for all $s'$), where the coefficients are $T(s, a, s')$. The normalisation factor is also a convex combination where the coefficients are $b(s)$ (for all $s$).


In graph clustering, we want to cluster the nodes of a given graph, such that nodes in the same cluster are highly connected (by edges) and nodes in different clusters are poorly or not connected at all.

A simple (hierarchical and divisive) algorithm to perform clustering on a graph is based on first finding the minimum spanning tree of the graph (using e.g. Kruskal's algorithm), $T$. It then proceeds in iterations. At each iteration, we remove from $T$ the edge with the highest weight. Given that $T$ is a tree, the removal of an edge from $T$ will create a forest (with connected components). So, after the removal of the edge of highest weight from $T$, we will have two connected components. These two connected components will represent two clusters. So, after one iteration, we will have two clusters. At the next iteration, we remove the edge with the second highest weight, and this will create other connected components, and so on, until, possibly, all nodes are in their own cluster (that is, all edges have been removed from $T$).

There are several limitations of this algorithm. For example, it only considers the edges of the initial graph that are shared with $T$ (that is, it only considers the edges of the minimum spanning tree). It also requires the edges of the graph to be weighted. It does not require the number of clusters to be known in advance (like any other hierarchical clustering algorithm), but we still need to choose the optimal number of clusters (after the algorithm has terminated). We can do that in several ways. A way to do it would be to have a threshold value $t$ that is used to decide when we should stop removing edges from $T$: more specifically, we will keep removing edges from $T$ until the next highest weight is higher than this threshold $t$.

There are numerous applications of this type of clustering. For example, we might want to discover certain groups of people in social networks.

The paper Graph clustering (by Satu Elisa Schaeffer, 2007) provides a readable and quite detailed overview of this field.

There are algorithms based on k-means that can also work on graphs. See e.g. Graph-based k-means Clustering: A Comparison of the Set Median versus the Generalized Median Graph (by Ferrer et al.).


Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?

You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still "correct", regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a "policy").
There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).

Or I should go with the behavior cloning technique to train the actor network only, so that it converges rapidly?

I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.


How do the current input and the output of the previous time step get combined in an LSTM?

(1) $X_t$ and $H_{t-1}$ are concatenated. The blog you cited explained its notation "Lines merging denote concatenation". For example, if $X_t=[1,2,3]$ and $H_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$

(2) When you say "input weights" or "weights of the input of the previous time step", are you referring to the $W_i$ in your cited blog? If so they are not the weights of the connections between the time-steps/units. They are part of the input gate only. The connections between the time-steps/units do not have weights applied to them.


An intuitive explanation of Adagrad, its purpose and its formula

I found a somewhat more accessible introduction here:

https://medium.com/konvergen/an-introduction-to-adagrad-f130ae871827

Let me start from the last part of your question. The frequency of a parameter is in G_t, which is the accumulated sum of squared gradients from all the time steps up to step t. If the gradient vanishes in many of the previous steps, then you divide the learning rate with a smaller number for that parameter.

And for the first part, you want the parameter that is more frequent to have a smaller learning rate as it is updated on more iterations compared to a parameter which is updated only a small number of times.


What are the conditions of convergence of temporal-difference learning?

There are different TD algorithms, e.g. Q-learning and SARSA, whose convergence properties have been studied separately (in many cases).

In some convergence proofs, e.g. in the paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo), the required conditions for Q-learning to converge (in probability) are the Robbins-Monro conditions

$\sum_{t} \alpha_t(s, a) = \infty$
$\sum_{t} \alpha_t^2(s, a) < \infty,$

where $\alpha_t(s, a)$ is the learning rate at time step $t$ (that can depend on the state $s$ and action $a$), and that each state is visited infinitely often.

(The Robbins-Monro conditions (1 and 2) are due to Herbert Robbins and Sutton Monro, who started the field of stochastic approximation in the 1950s, with the paper A Stochastic Approximation Method. The fields of RL and stochastic approximation are related. See this answer for more details.)

However, note again that the specific required conditions for TD methods to converge may vary depending on the proof and the specific TD algorithm. For example, the Robbins-Monro conditions are not assumed in Learning to Predict by the Methods of Temporal Differences by Richard S. Sutton (because this is not a proof of convergence in probability but in expectation).

Moreover, note that the proofs mentioned above are only applicable to the tabular versions of Q-learning. If you use function approximation, Q-learning (and other TD algorithms) may not converge. Nevertheless, there are cases when Q-learning combined with function approximation converges. See An Analysis of Reinforcement Learning with Function Approximation by Francisco S. Melo et al. and SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation by Bo Dai et al.


How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the policy $\pi$ after that state. If you repeat this process many times, you'll get many samples of trajectories starting at $s$ with some total return associated with them. If you average them, you'll have a sample-based estimate of $V^\pi(s)$. In a similar way, you can do a sample-based approach to estimate $Q^\pi(s, a)$, where you start at $s$, take some arbitrary action $a$, and then continue following $\pi$. After you collect many samples, you can also average the returns to get a sample-based estimate of $Q^\pi(s, a)$. Once you have a good enough estimate of $Q^\pi(s, a)$, you can see that it's easy to improve the current policy $\pi$ to get a new policy $\pi'$ that always chooses the best action on any given state: $\pi'(s) = \underset{a}{argmax}\ Q^\pi(s, a)$.

The procedure I described in the last paragraph where you sample an entire trajectory and wait until the end of the episode to estimate a return is the Monte Carlo approach. In contrast, TD exploits the recursive nature of the Bellman equation to learn as you go, even before the episode ends. For example, you can estimate the value function like this: $V^\pi(s_0) = r_0 + \gamma V^\pi(s_1)$, where $r_0$ comes from experience and instead of collecting more $r_t$ until the end of the episode, you rely on your estimate of the value of the next state (which you also get from experience). At the beginning, your estimates might be random, but after many iterations the process converges to good estimates. The same idea of exploiting this recursive structure can be used to estimate $Q^\pi(s, a)$. I guess this approach is known as temporal difference because you sample $r_0$ which is in some sense the temporal difference between $s_0$ and $s_1$.


Emulating human brain - with analogous NN chips

I'm not sure about "emulating the brain" per-se, but in a more general sense there has been some thought given to using analog computing for AI/ML.  It seems clear that analog computers do have certain advantages over digital computers.  For one, they can (depending on the application) be faster, albeit at the cost of some loss of precision.  But that's OK, because I don't think anybody believes the human brain is calculating floating point math using digital computing techniques either.  The human brain appears, at least superficially, to be largely probabilistic and able to tolerate some "slop" numerically.

The downside to analog computers, as I understand it, is that they're not as flexible... you basically hardwire a circuit to do one specific "thing" and that's really all it can do.  To change the "programming" you have to literally solder in a new component! Or, I suppose, adjust a potentiometer or adjustable capacitor, etc.  Anyway, the point is that digital computers are supremely flexible, which is one big reason they came to dominate the world.  But I can see where there could be room for going analog for discrete functions that make up some or all of an intelligent system.

As for research in the area, you might look into whatever DARPA was / is doing. There was an article in Wired a while back, talking about some DARPA initiatives related to analog computing.


How to quickly change hand-drawn shapes to symmetrical polished shapes?

From how it looks, the most reliable method to try out is using Hough transform.
The Hough transform can be used to detect e.g. lines and circles in images (depending on which variant you are using; in this case it would amount to a combination of variants for both lines and circles obviously).
So, given some input image, the Hough transform tells you what the line/circle parameters are that have created a line in the input image. For example, given a line, it would tell you the intersection with the $y$-axis and the $slope$ of the line detected in the image.
Then, you could use these parameter information to reconstruct lines and circles detected in the image.
The last remaining problem to be solved then is to check where a detected line starts and stops (since this is not obvious from line parameters like $m$ (=$slope$) and $b$ (=intersection with $y$-axis) in the equation $y = mx+b$ describing some line).
But for that, you could "walk" along a line in the image space and check where the line is present or not. Then, you can draw line segments in the reconstruction image when the corresponding elements are also present in the original image.
The problem with (C)NNs would be that they are sensitive to rotation and scale etc. You could of course take a tremendously large number of filters to account for shapes of different rotation and scale, but that would increase the demand for labeled training data again (which could of course be automatically be generated a priori in this simple case).
Anyway, I'd suggest checking out Hough transform. To get some feeling for it, there are lots of libraries available implementing it for Python or MatLab, for example.
For further information, check out Wikipedia or YouTube.


How to train a sequence labeling model with annotations from three annotators?

Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.
Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))


Why are model-based methods more sample efficient than model-free methods?

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.

Main difference between those two is like you said, that the model helps in finding the correct path more efficiently because of the existence of the model. Maybe you cannot find new samples (point 1) but you know better the whole inner logic about the system and instead of just knowing what to do with the specific sample, you can relate it to the whole picture (sort of like in point 2: you can play with the choices) and make more profound calculations.

The article had a comparison, which told that you are writing a map in a city about every possible direction you can take when you are model-based and while in model-free you can enter specific places and remember which direction was best based on last visits but you still never know where you are coming nor going exactly.

In other words, if you think you're teaching a taxi driver on a big city with many signs and rules, model based guy would drive more precisely sooner because the sign language (the inner logic and model of the city) helps them on understanding the map sooner than just reacting crossing by crossing somewhat by chance all the time.

Sample efficiency tells what is the amount of information fetched from one sample [2]. Model-based machine can adjust model, maybe make some calculations about expected rewards AND after that the same as model-free, adjust the common policy. Model-free does only have the policy. Again the taxi guys: model-free guys know that last time and second last I stopped in crossing, model-based guy knows also it was due to red lights in pole. Third time model-free guy is the first in row and BANG - hits the crossing car. Next time rule is there comes sometimes cars and model-based guy knew that from the first place.

My sources:

[1] https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698

[2] What is sample efficiency, and how can importance sampling be used to achieve it?


Obtain the most important input data for binary classification on a neural network

Two popular methods I’ve seen done:

1) For each feature, remove it and run the model and see the impact it has on the result. The idea is that the larger the impact, the more pertinent it was to the result.

2) Look at the gradients magnitude $|\nabla_f {y} |$. You can either look at the raw gradient or look at the guided back-propagation which is just the back props  product rule, but you only look at when the nodes positively help trigger a neuron by taking only the positive gradients at each step.

There’s probably also more methods. Hope this helped.


Unable to replicate Figure 2.1 from "Reinforcement Learning: An Introduction"

You are calculating the average reward for each action (i.e. bandit arm) incorrectly. You cannot calculate this simply with a list comprehension, and you need to keep a second list storing the number of times each action was taken.

The correct calculation is to divide the total reward obtained from each action by the number of times that action was taken. You are dividing all actions by the total number of actions taken. Adding:

...
counts = [0 for i in range(10)]
...
else:
    averages = []
    for i in range(0, 10):
        averages.append(Q[i]/counts[i] if counts[i] > 0 else 0)
    ...
...
counts[action] += 1

results in working code, and I can generate this graph using 200 samples per method, which looks like a noisier version of the one in the figure you referenced:


Is there any open source counterpart to the IBM Watson?

I am not aware of any open-source products that can readily replace IBM Watson, but the following projects could be of interest to you in that regard:

DARPA DeepDive
Snorkel
Apache UIMA
YodaQA
OpenCog
OAQA (Open Advancement of Question Answering Systems)
Stanford CoreNLP – Natural language software
openQA


A neural network to learn the connection between two totally different type of images

I would try a pair of separate deep image embeddings with a contrastive loss. The idea is similar to the Siamese network architecture. In Siamese networks the pairs of images are of the same type - so both input images are fed through the copy of the same network. In your case the images are of different kinds, so I would just have separate nets for person images and fingerprints.


How can I perform lane detection with reinforcement learning?

I will agree with malioboro, maybe RL is an overkill for such a task. Even though with the trend of autonomous driving research lately, papers dealing with lane changing almost certainly exist, you should check them out for more details.

As stated in Lane Change Decision-making through DeepReinforcement Learning with Rule-basedConstraints , "When the states are discrete and finite, the Q-function canbe easily formulated in a tabular form. But in many practical applications, for example, lane change decision-making task,the state space of them is very large or even continuous,using the Q-learning algorithm will lead to dimension disaster. Therefore, tabular Q-learning algorithm does not applicable to
the learning problem of continuous state space and continuousaction space"

So, i would recommend using image processing techniques, or Deep-Q-Learning. Q-Learning is not capable of dealing with continuous problems like lane changing or lane tracking.


Neural network does not give out the required out put?

The numbers you are seeing as output are a probability vector. This is a common output format for multi-class classification models.

In this case, you can interpret the vector as saying:

22% chance of class 1
20% chance of class 2
37% chance of class 3
11% chance of class 4
4% chance of class 5
2% chance of class 6
2% chance of class 7

If you want to get a concrete label out of this, the easiest choice is to compute and return the index of the maximum element.


Can transformer be better than RNN for online speech recognition?

Are there examples that transformer have better accuracy than RNN end-to-end model like RNN-transducer for speech recognition?
  Can transformer be used for online speech recognition which require low speech-end-to-result latency?
  Does transformer have the potential to replace RNN end-to-end models for speech recognition in most cases in the future? This may mainly depends on accuracy and deploy cost, not training cost.

You can check facebook results on wav2letter on all this:

https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/

https://research.fb.com/publications/scaling-up-online-speech-recognition-using-convnets/

Transformers definitely have a potential in speech especially when combined with faster computatoin methods (hashing) just like in NLP.

The problem with transformers is that you need a lot of GPUs to train them.


Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

TL;DR: This has to do with the way A. Ng has defined back propagation for the course.

Left Column

This is only with respect to one input example and so the $\frac{1}{m}$ factor reduces to 1 and can be omitted. He uses lower case to represent one input example (eg a vector $dz$) and upper case with respect to a (mini-)batch (eg a matrix $dZ$).

The $\frac{1}{m}$ factors in $dW,db$

In this definition of backprop, he "defers" multiplying by the $\frac{1}{m}$ factor until $dW,db$ rather than "absorbing" it into $dZ^{[2]}$. That is, the $dZ^{[2]}$ term is defined in a way that it does not have $\frac{1}{m}$.

Observe, if you move the $\frac{1}{m}$ factor to be in the definition of $dZ^{[2]}$ and remove it from the definitions of $dW,db$ you will still come out with the same values for all $dW,db$.

Speculation

This "deferred" multiplication might have to do with numerical stability. Or simply a stylistic choice made by A. Ng. This might also prevent one from "accidentally" multiplying by $\frac{1}{m}$ more than once.


Difficulty in understanding identifiability in the "Dueling Network Architectures for Deep Reinforcement Learning" paper

I believe that is explained on the prior page:

"Intuitively, the value function $V$ measures the how good it is to be in a particular state $s$. The $Q$ function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the $Q$
function to obtain a relative measure of the importance of
each action."

Then two paragraphs above were you started your quote:

"However, we need to keep in mind that $Q(s, a; \theta, \alpha, \beta)$ is only a parameterized estimate of the true $Q$-function. Moreover, it would be wrong to conclude that $V (s; \theta, \beta)$ is a good estimator of the state-value function, or likewise that $A(s, a; \theta, \alpha)$ provides a reasonable estimate of the advantage function.
Equation (7) is unidentifiable in the sense that given $Q$ we cannot recover $V$ and $A$ uniquely. To see this, add a constant to $V (s; \theta, \beta)$ and subtract the same constant from $A(s, a; \theta, \alpha)$. This constant cancels out resulting in the same $Q$ value. This lack of identifiability is mirrored by poor practical performance when this equation is used directly."

Another way of looking at it would be:

You receive answers to your question

Answers receive votes

Answerers have reputation

In a perfect world people could vote based on reputation, with a weighing based upon the correctness of the answer.
You could simply look at which answer received the most votes and choose it as correct.
In the real world things don't work that way, things are correct or incorrect whether they are measured or not (think quantum mechanics) and measurement doesn't always reveal the true answer.
See: Parameter Estimation.
The estimate of the advantage is only so good, sometimes it's useful to consider it and in other instances it's useful to reject it - intelligently doing both maximizes it's usefulness.


How to extract parameters from a text using AI/NLP

I think U are looking for PLSA
for PLSA either U find out those topics(catogeries) with EM or NNMF
Personally I recommend NNMF
or u can use LDA which is Bayesian version of PLSA
here is code for PLSA:
https://github.com/Man-ash/Probabilistic-Latent-semantic-analysis
which use NNMF
for EM method I code it by myself but i am not sure if it is right
https://stackoverflow.com/questions/65783880/problem-about-m-step-in-probabilistic-latent-semantic-analysis-with-code


Is it possible for 'unsupervised learning' model to recognize features on unlabelled images?

I am assuming each image contains a single object.

It is possible, however, it is not as easy as you might think. Firstly, you need extract as many features as possible: original image, LBP, SIFT, moments, contour descriptors to name a few. Than concatenate these features into a single feature vector. After this step, use clustering. You will need a lot of samples to compensate for the number of features. After clustering, use a correlation method to find which features are related to each cluster.

If you need features to classify within a cluster, you could do a second clustering with full set of features and apply the same method. The features that are selected for a cluster will not be suitable to classify within the cluster.


Classifying generated samples with Wasserstein-GAN as real or fake

My suggestion is to compare the critics rating for the real data and the generated data. Compute the mean of the ratings on real data to get some kind of threshold. If your critic is designed so that a high rating indicates a real sample, then generated data with ratings greater than the mean of the ratings of the real data could be viewed as good enough to fool the critic at that point in training.
This might be a too harsh criteria, but you will at least not accept poorly generated data.


In a single neuron output layer should the output be a scalar?

Look, your code says your network has many outputs.
Look at the two lines below. This two lines says the output depends on the dimension of np.dot(w, inputs). In your case it's 4 diminutional vector. And in the last line you are assigning them as output. You can write self.output = sigmoid(np.dot(new_weihts, inputs)) instead of self.output = inputs. Must ensure that new_weights is a vector and has same shape as previous layer's output.

inputs = sigmoid(np.dot(w, inputs) + b)
self.output = inputs

Note: Your network don't look like a 4 layers network. It looks like a one layer network with 4 unit of neutrons.


There is no strong evidence that a single mechanism of learning exists in the neural networks of the human brain.  The scientifically collected evidence does not support that simplistic conception.

Learning is an umbrella term under which many things reside.

Accumulation of rules triggered by corresponding sets of conditions
Pattern recognition, such as the recognition of the probable source of a knock or a saxophone note
Connection between intention, motor output, and sensory feedback, backed by pattern recognition, as with hand-eye coordination
Construction and refinement of internal models of subjective reality that can be called upon to make choices or determine quantities
Construction and refinement of external models of objective reality, such as Newton's recognition that Kepler's description of elliptical orbits, Aristotle's attraction of objects to the ground, and the constant rate of velocity increase in falling objects were manifestations of the same phenomenon, thus the simultaneous invention of the foundations of both calculus and physics
Dreams during which insights can be gained, often of a seemingly illogical nature, yet reportedly useful in life goal attainment
Sleep in between dreams where it is possibly that the day's queued observations are integrated with existing learned material or otherwise assimilated
Largely creative or intuitive, where models and rules are combined to create hypotheses to be tested through experimentation or through long periods of in situ observation (observation in natural environment)
Subconscious and related to archetypes

There are several approach to simulating these partially understood learning mechanisms in computers. It is also important to note that not all machine learning is a simulation of a mechanism of the human nervous system.  What arrangements of architectures, algorithms, parallelism, and interconnections optimally learn depend on many things.

The nature of the things to be learned (tennis, calculus, how to recognize a zip code in a mail sorter, someone's voice, someone's retina or fingerprint, how to identify social or story structure in a drama, self-improving real time trajectory prediction and antiaircraft aiming)
The mechanism of input and output (tabular, natural language, sound, visual, tactile, stepper motor signals)
The balance of the dimensions of the quality of intelligence to which the term optimal can be applied (accuracy, reliability, range of conditions for which learning is functional at all, relationship between complexity and speed of response capacity for complexity, depth of abstraction, wit in presentation of conclusions, listing of reference, auditability of thinking)
Hardware running the algorithms, their processes, and interconnecting them

How Learning Presumably Occurs

The current scientific theory (which is subject to change as more discovery occurs and paradigms shift) is that the human brain is programmed by the portions of DNA that define neural structures.  It is theorized that structural changes inside of and between neurons persist memories and establish pathways that lead to adaptations beyond instinctive neural activity and organism behavior, which is part of the DNA programming.

Organic or Digital Circuit Based Learning

For learning to occur in either a meat or silicon based system, there are a few requirements, the listing of which will give a general view into the answer to the question.

Way for the system to understand what is expected of it (real time acceptability/unacceptability/quality feedback, built in goal achievement detection, crisis detection, or multiple cases of conditions and expected results)
Way for the system to persist information needed to hold a representation of what had thus far been learned
Way for the system to acquire input or sense conditions
Way for the system to output results, execute decisions made, or impact conditions
Way to load algorithms into processes, interconnect them, and manage the overall system throughput
Hardware to support the above
Optionally, typical enterprise type features such as crash/disaster recovery, the ability to independently handle concurrent learning processes on a single system, or the ability to scale the system to handle larger problems or more concurrent learning processes

All of the above is dismissing entirely the possibility of concepts such as synchronicity, the metaphysical, or deity, none of which have been scientifically disproven.

Code

There are a few code bases that may help begin an investigation into learning using some of the most common and publicly available approaches.  Some of these have evolved to learn for specific problem sets.

https://github.com/mnielsen/neural-networks-and-deep-learning
https://github.com/kootenpv/neural_complete
https://github.com/rougier/neural-networks
https://github.com/jcjohnson/neural-style
https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
https://github.com/uncomplicate/bayadera
https://github.com/timnugent/naive-bayes
https://sourceforge.net/projects/cpp-fuzzy-logic/
https://www.codeproject.com/Articles/316668/Cplusplus-Fuzzy-Logic-API-plus-Simple-DSL
https://github.com/abudnik/tcalc
https://github.com/metagol/metagol
https://sourceforge.net/projects/clipsrules/
http://openrules.com/
https://www.drools.org/

Truly Learning About Learning

Rather than running example code, it may be best to commit to actual learning about learning.  Downloading, compiling, configuring and running some existing POC is not your proof of concept.  It may be your starting point, but until you change those POCs to perform some other task you've devised and for which you've located sufficiently realistic data, you haven't learned anything about learning yet.

This is an example of a self-constructed lab project that wades right into the center of learning about learning software.

Develop an object oriented model of a tic-tac-toe game
Develop classes to represent the possible moves available for two opponents
Add an object oriented model of HOW the next move could be decided based on some arbitrary logic that can be mutated or permuted
Create some mechanism to try various HOWs in actual play
Create an algorithm that processes as input win-loose statistics and the
corresponding HOWs that converges on the optimal HOW for winning

Representing the tic-tac-toe three by three grid and the possible moves that players can make may only require a few dozen lines of code, but the simulation of human learning of the game (or some other form of learning the game that humans might not actually do) is not simple.

Even for the limited problem set of tic-tac-toe games, to write a program that actually learns the always-win-if-winning-is-possible algorithm by actually learning tic-tac-toe by playing it is complex and raises questions about random versus educated guessing worth investigation and about which many articles and books may be found and studied.

Developing such a program that learn successful tic-tac-toe play without knowing how to win in advance may take some brilliant thinking and days of programming, testing, experimenting, and re-thinking.  However, that will reveal more than reading articles, posts, books, or library documentation.

Writing a program that would deduce how to play tic-tac-toe from the rules of the game represented as predicate logic is a longer term project but may be rewarding and quite educational.


Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Removing the learning rate will likely yield poor convergence to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.
Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and gradually until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $\alpha = 0.1$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their book, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.


How can a neural network learn to play sudoku?

I think it is the wrong way to frame sudoku as a regression problem in neural networks.
Firstly, you have to understand what regression is. "Regression" is when you predict a value given certain parameters, where the parameters are related to the value you have to predict. This happens because at the core neural networks are "function approximators", they model the function by adjusting their weights using lots of data. They tend to form a highly non-linear boundary to separate classes internally in a high dimensional data-space.

The sudoku doesn't fit in this scenario, the combinatorial complexity of sudoku is way too high for a neural network even if you add many layers to it, it is a totally different problem in its own right. You simple can't "regress" the right values of a perfect sudoku here, they are not numbers like "pixel " intensities in images.

However, you could apply reinforcement learning techniques to learn an optimal policy to solve sudoku.

And you have mentioned an "approximate" solution for the sudoku, what do you mean by "approximate"?  If you mean by this that only a few squares are out of place, then it is a wrong assumption, because neural networks are proven to be good image classifiers, as they are robust to translational invariance in this case, that is not what you need.

You could, however, do a small experiment to see what the neural network actually learns, replace the numbers by pixel value intensities and train a generative adversarial network on the sudoku images and see the images of sudokus produced by it, to see what actually the network can't learn.


Python Network for Simple Image Classification

Using neural networks

If it is as simple as that, probably a neural network consisting of just a simple convolutional network with a few filters, a fully connected layer with 2 neurons, and a softmax layer, might do it.

To develop that, you can use Keras wrapper over tensorflow or theano, but I think scikit-learn also provides some basic neural network functionalities.

Depending on the simplicity of the images on your dataset, you might do it even with no convolutional layer if there are not many non-linearities, or you might need more complexity (eg. adding convolutional and max pooling layers).

I made personally for teaching a workshop a very simple example of image classification, with 3 groups (trees, people, vehicles), after building a dataset collecting and selectioning images from the CIFAR-100 dataset.  You can see this ultra simple example here. However, it is in spanish. Let me translate the most important piece of code from it:

model = Sequential()

# CONVOLUTION. Final size: 29x29x32
model.add(Conv2D(   32,
                    (4, 4),
                    padding='same',
                    input_shape=data_x_train.shape[1:]
        ))

model.add(Activation('relu'))

# CONVOLUTION. Final size: 26x26x32
model.add(Conv2D(   32,
                    (4, 4),
                    padding='same',
                    input_shape=data_x_train.shape[1:]
        ))

model.add(Activation('relu'))

# MAX POOLING. Final size: 13x13x32
model.add(MaxPooling2D(pool_size=(2,2)))

# Dropout to prevent overfitting
model.add(Dropout(0.25))

# CONVOLUTION. Final size: 10x10x64
model.add(Conv2D(   64,
                    (4, 4),
                    padding='same',
                    input_shape=data_x_train.shape[1:]
        ))

model.add(Activation('relu'))

# MAX POOLING. Final size: 5x5x64
model.add(MaxPooling2D(pool_size=(2,2)))

# Dropout to prevent overfitting
model.add(Dropout(0.25))

# Fully connected layer. Neuron number: 1600
model.add(Flatten())

# Output layer: 3 neurons
model.add(Dense(3))
model.add(Activation('softmax'))

So the input for this model is an array of shape [num_rows, 32, 32]. 32x32 is the image size, and I assume 1 color channel. As I say, this is ultra simple. The output, some kind of probability for each of the 3 classes.

Using other aproaches

However, there are many other supervised learning models that you can use for that, most of them supported by scikit-learn, such as kNN, random forests decision trees, RBF SVM, or even just a kernelized multiclass logistic regression. See here.


Is there an AI system that automatically generates classes and methods by giving it voice commands?

While you can use NLTK for analyzing and parsing the text obtained from the speech to text interface (e.g. Siri), there are higher level APIs available for this. The class of problem you are trying to solve in NLP is "intent detection".

There are several open source and commerical APIs available for this including Amazon Alexa, Google Cloud Natural Language, Azure, as well as libraries like RASA NLU, etc.

The high level flow of your program will be:

Record/receive spoken audio
Convert audio speech to text
Detect intent of the text command using an intent detection library
Use the intent to feed a script/automation that generates the code in your IDE


How do you choose the number of neurons, synapses, and hidden layers of an ANN (in NEAT)?

To find the number of neurons and layers that you will use is not that straightforward. The best way to do this is through experimentation however you will be able to better estimate the number of layers and neurons needed through experience. One of the common rules is that more neurons are better for more complex datasets. However, you do not want too many or you will get an overfitted model.

As for NEAT, that stands for neuron evolution of augmenting topologies. This is a genetic algorithm that works fairly well, however, I would recommend that you use a different algorithm like Q-learning. If you wish to learn more about Q-learning, then I would definitely recommend that you check out Google's DeepMind research on training deep neural networks to play the game of Go using Q-learning.


Why can't DQN be used for self-driving cars?

I'm not familiar with the ins and outs of self-driving cars, but I imagine that the action space is not discrete. For instance, the car may want to decide what angle it needs to turn (rather than left or right). The update in Q-Learning involves taking $\max_aQ(s, a)$; this is theoretically possible for a continuous action space, but it would itself require some expensive optimisation at each time step to find the maximum. It is more likely that if RL were to be applied to self-driving cars it would be through a method that easily allows for a continuous action space, like the methods detailed in this paper.
I found this survey of Deep RL for autonomous driving that you may want to look at.


Is Monte Carlo Tree Search appropriate for problems with large state and action spaces?

MCTS is often said to be a good choice for problems with large branching factors... but the context where that sentiment comes from is that it originally became popular for playing the game of Go, as an alternative to older game-playing approaches such as alpha-beta pruning. The branching factor of Go is more like 250-300 though, which is often viewed as a large branching factor for board games. It's not such an impressive branching factor anymore when compared to your branching factor of $6,000,000$...

I don't see MCTS working well out of the box when you have 6 million choices at every step. Maybe it could do well if you have an extremely efficient implementation of your MDP (e.g. if you can simulate millions of roll-outs per second), and if you have a large amount of "thinking time" (probably in the order of hours or days) available.

To have any chance of doing better with such a massive branching factor, you really need generalization across actions. Are your 6 million actions really all entirely different actions? Or are many of them somehow related to each other? If you gather some experience (a simulation in MCTS, or just a trajectory with Reinforcement Learning approaches), can you generalize the outcome to other actions for which you did not yet collect experience?

If there is some way of treating different actions as being "similar" (in a given state), you can use a single observation to update statistics for multiple different actions at once. The most obvious way would be if you can define meaningful features for actions (or state-action pairs). Standard Reinforcement Learning approaches (with function approximation, maybe linear or maybe Deep Neural Networks) can then relatively "easily" generalize in a meaningful way across lots of actions. They can also be combined with MCTS in various ways (see for example AlphaGo Zero / Alpha Zero).

Even with all that, a branching factor of 6 million still remains massive... but generalization across actions is probably your best bet (which may be done inside MCTS, but really does need a significant number of bells and whistles on top of the standard approach).


Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$
Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices, where each entry is a some random distribution with $0$ mean and $1$ variance. Every entry is independent from each other.
Since each entry of $Q$ and $K$ have identical distribution, we can focus only on the top-left-most element of $QK$ without loss of generality. The same applies to every other element.
The top-left-most element of $QK$ is $\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1}$.
Since $Q$ and $K$ are independent:
$\mathbb{E}[Q_{1, i} K_{i, 1}] = \mathbb{E}[Q_{1, i}] \mathbb{E}[K_{i, 1}] = 0 \\
Var(Q_{1, i} K_{i, 1}) = (Var(Q_{1, i}) + \mathbb{E}[Q_{1, i}]^2)(Var(K_{i, 1}) + \mathbb{E}[K_{i, 1}]^2) - \mathbb{E}[Q_{1, i}]^2\mathbb{E}[K_{i, 1}]^2 = 1$
And so summing up $d_k$ of them:
$\mathbb{E} \left[\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right] = \displaystyle \sum_{i=0}^{d_k} \mathbb{E} \left[ Q_{1,i} K_{i, 1}  \right] = 0 \\
Var\left(\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right) = \displaystyle \sum_{i=0}^{d_k} Var\left( Q_{1,i} K_{i, 1}  \right) = d_k$
For your code block, you are computing the dot product of matrices $a$ and $b$, when you should be doing a matrix multiplication (the attention function multiplies $Q$ by $K$ after all, which is the vectorized form of dot-product -- it doesn't actually do dot product). It should work out to unit variance.
edit: the last paragraph is incorrect, as dot is the same as matrix multiply in the above case


Which AI algorithm to use to identify a subject from many unknown factors

You have a set of different types of data available for each of your subjects, and given one set you'd like to classify which subject it belongs to. This looks like a supervised classificiation problem.

The most popular classifiers for supervised learning are neural networks. Now given the heterogenous nature of your data types, a simple approach would be to use separate classifiers for each type of data. For example, a convolution neural net for the image data, and a simple feed forward net for the biosensor data. Another thing you try is a multi channel approach, where towards the input side you have multiple channels for the different types of data, and the final few layers are fully connected.

The image has only CNNs for the multi channel part but you could have one channel as a simple feed forward net while another one as having conv layers.

Also, if you wish to classify the data as belonging to a subject on the basis of just one of the data types from the set, then you should have separate classifiers for all types. In that case it might be worthwhile to look into classifiers other than neural nets like multi class logistic regression which might be simpler to work with for a particular data type.


Can we use NLP to understand/parse/compile programming code?

Yes, and that is the ambition of the Decoder project (H2020 funded in Europe). Analyzing with NLP techniques the comments in e.g. C or C++ source code (of open source projects, and perhaps their git logs).
I even happen to be on the photo.
My dream is to try similar things, in a few years, in RefPerSys. You could join that project, BTW.


Why do we need multiple LSTM units in a layer?

Let's write down Fibonacci!

  K = 0 1 1 2 3 5 ...

And another series that is derived from Fibo;

  X = 1 4 7 12 30 $X_{5}$

Guessing $X_{5}$ is our task and both series are available to you (Fibonacci as an additional feature).

One unit that you feed with X will try to capture the relation of $X_{t}$ and $X_{t-1}$ only;

$X_{t}$ = $X_{t-1}$ + $X_{t-2}$

However, an additional unit that you insert&feed with K will not only try to capture $K_{t}$ and $K_{t-1}$ relation but also the relation between $K_{t}$ and $X_{t}$.

$K_{t}$ = $K_{t-1}$ + $2*$$X_{t}$ + $1$

In the example above, there is a clear correlation between $K_{t}$ and $X_{t}$ (which is not always the case) and it will support the network to capture the sequential relation. Even the CORRELATION is not crystal clear, almost every additional feature data correlates with the other features and will support the network to grab the relation.


Would a new human-like general artificial intelligence be more similar, in terms of eduction, to a toddler or an adult human?

A human-level AI should be able to learn and behave in the same way that humans learn and behave, otherwise, we shouldn't be calling it a human-level AI. So, either if it starts with more or less knowledge of a baby human, it should be able to learn similarly to a human and acquire more knowledge with experience.

  What is the apparent level of education of an artificial intelligence that could be called human-like?

To answer you question more directly, the level of education of a human-like AI (even newly created ones) can potentially be variable (i.e. different human-like AIs could potentially have different levels of education), but the human-like AI will necessarily need to be able to increase its level of education and, in general, knowledge (because humans can also do this).

Moreover, note that AGI is not necessarily restricted to human-level AI. So, other AGIs may not follow the same principles of humans.


The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$.
Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}$. This makes the update quite intuitive - the numerator shifts the parameters in the direction that gives the highest increase in probability that the action will be repeated, given the state, proportional to the returns - this is easy to see because it is essentially a gradient ascent step. The denominator controls for actions that would have an advantage over other actions because they would be chosen more frequently, by inversely scaling with respect to the probability of the action being taken; imagine if there had been high rewards but the action at time $t$ has low probability of being selected (e.g. 0.1) then this will multiply the returns by 10 leading to a larger update step in the direction that would increase the probability of this action being selected the most (which is what the numerator controls for, as mentioned).
That is for the intuition -- to see why it does work, then think about what we've done. We defined an objective function, $v_\pi(s)$, that we are interested in maximising with respected to our parameters $\theta$. We find the derivative of this objective with respect to our parameters, and then we perform gradient ascent on our parameters to maximise our objective, i.e. to maximise $v_\pi(s)$, thus if we keep performing gradient ascent then our policy parameters will converge (eventually) to values that maximise $v$ and thus our policy would be optimal.


What is the relation between back-propagation and reinforcement learning?

Backpropagation is a subroutine often used when training Artificial Neural Networks with a Gradient Descent learning algorithm. Gradient Descent requires computation of the error gradient, i.e. derivatives, of a cost function with respect to the network parameters. BP allows you to find this gradient a lot faster than using naive methods. Reinforcement Learning refers to inferring "optimal" behavior, i.e. a strategy, of an agent maximizing some goal in an environment. Depending on the specific RL algorithm, BP may be employed to adjust parameters of a function approximator used to represent aspects of the environment and or the agent.


Levenshtein Distance between each word in a given string

well... simply put a .split() at the end of your first two lines:

a = "This is a dog.".split()
b = "This is a cat.".split()

Your algorithm works with the iterables, and the string is broken into it's characters. You do the split, and a,b would be a list of words, then your algorithm works on the word-level

Output on your example:

[[0. 1. 2. 3. 4.]
 [1. 0. 1. 2. 3.]
 [2. 1. 0. 1. 2.]
 [3. 2. 1. 0. 1.]
 [4. 3. 2. 1. 1.]]

1.0


How can the $\lambda$-return be defined recursively?

To rewrite $G_t^\lambda$ recursively, our goal is to define it in terms of
$$G_{t+1}^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t+1:t+n+1}.\tag{0}$$

The $\lambda$-return is a weighted average of all $n$-step returns. We will split up the summation by  pulling out the one-step return $G_{t:t+1}$ and the first step's reward $R_{t+1}$.

$$
\begin{align*}
G_t^\lambda &= (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n} \tag{1}\\
&\\
&= (1-\lambda)\lambda^0G_{t:t+1} + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}G_{t:t+n}\tag{2}\\
&\\
&= (1-\lambda)\left(R_{t+1}+\gamma\hat{v}(S_{t+1})\right)\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{3}\\
&\\
&= (1-\lambda)\left(R_{t+1}+\gamma\hat{v}(S_{t+1})\right) + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1} R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{4}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1} R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{5}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{6}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + \gamma\lambda(1-\lambda)\sum_{n=2}^\infty \lambda^{n-2}(R_{t+2}+\dots +\gamma^{n-2}R_{t+n} + \gamma^{n-1}\hat{v}(S_{t+n}))\tag{7}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + \gamma\lambda(1-\lambda)\sum_{m=1}^\infty \lambda^{m-1}(R_{t+2}+\dots +\gamma^{m-1}R_{t+m+1} + \gamma^{m}\hat{v}(S_{t+m+1}))\tag{8}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1} + \gamma\lambda(1-\lambda)\sum_{m=1}^\infty \lambda^{m-1}G_{t+1:t+m+1}\tag{9}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1} + \gamma\lambda G_{t+1}^\lambda \tag{10}\\
\end{align*}
$$

$ $
$(2)$ pulls out the one-step return from the summation.
$(3)$ expands the $n$-step returns.
$(4)$ pulls out the remaining first step rewards.
$(5)$ combines first step rewards.
$(6)$ simplifies the geometric series.
$(7)$ pulls a factor of $\gamma\lambda$ out of the summation.
$(8)$ makes the substitution $m=n-1$.
$(9)$ uses the definition of the $n$-step return.
$(10)$ uses the definition of the $\lambda$-return

The result can be verified in equation $(12.18)$ of Sutton and Barto's RL book.


How to deal with a small amount of labeled samples?

In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.


How do I determine whether a truck is inside its lane?

One possible approach will be to use an algorithm which detects lines (Ex. Hough lines or any deep neural net trained to detect lanes) and use some threshold range so that we can get the lane and the edges of truck, then after extracting the lines, you can easily find the distance between them.

Then you need to experiment out on few images to get the threshold distance that you are expecting the truck to maintain as the real distance and the distance calculated using images are not same

If you want to classify using deep learning, you may need to preprocess the images and send them. As it will become very difficult to directly learn to classify
based on image, you may need to first detect the lanes, then apply a mask and then send the masked image to your network to make the network to converge.


Alternative to sliding window neural network (was: Object detect (or) image classification at specific locations in the frame)

There are many different problems in computer vision. Four of them are well described by the top image here:

Classification: Given an image, say what is on it (a single thing)
Classification+Localization: Given an image, say what is on it and draw an axis-aligned bounding box (AABB) around it
Object detection: Given an image, draw AABBs around every object and classify those objects
Semantic segmentation: See Survey of semantic segmentation
Instance segmentatation: Like semantic segmentation, but if there are multiple cats then they should be recognized as different objects.

Your question seems to be about object detection. The relevant papers here are:

SSD: Single Shot MultiBox Detector: Code
You Only Look Once: Unified, Real-Time Object Detection (YOLO): Website with video
YOLOv3

If you actually already have the regions, then you can simply perform classification on them. When you pad / scale / crop them, you can batch-predict them.


How to compute dominant colors in an image?

You can find an explanation here (github of the googleapi):

My current understanding of a color's score is a combination of two things:

What is the focus of the image?
What is the color of that focus?

For example, given the following image:

The focus is clearly the cat, and therefore the color annotation for this image with the highest score (0.15) will be RGB = (232, 183, 135) which is the beige color:

The green of the grass (despite having more pixels in the image dedicated to it) has a much lower score by virtue of the algorithm's detection that it's the background and not the focus of the image.

In other words, higher "scores" means higher confidence that the color in question is prominent in the central focus of the image.

It is analogous to your case. Therefore, using a background removal this can help to find the focus of the image, and then make the histogram of the remaining objects' color. An example of the background removal using deep learning can be found in this post.


Are there any results on how difficult certain languages are to learn for chatbots?

It's not the language itself but rather the structure and for the language's ambiguity, for example in English: person a says "John and Bob (his fish)" person B says "He died!", posed question, to whom does person B refer to by he died.  More than the language, but the application. You can write a chat bot in Assembly, C, C++, C#, Java or Python. The all work a bit differently but can accomplish the same result, but one language might have more pros or cons to the other. So it will boil down to not language but the understanding of what is being said, research of language in the brain has come to confirm we associate a meaning/feeling/and other inputs with a given language.

So to conclude: English is by far the most chaotic for a chat bot but Japanese is actually the best due to the way the language itself is written/spoken. There is more structure to it and less ambiguity.

I'm a Software Engineer and An AI Researcher for the past 7 years.


What algorithms are used for segmentation and classification of non solid regions in an image?

If you have enough data, you can train a segmentation net example for a specific group of data (i.e. trees). Single DNN with multiple output branches can easily segment & classify the data that you are searching for.


Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.

If you're going to use someone else's ready-made code for your prioritized experience replay, then you'll probably end up using proportional sampling. Every implementation I've been able to find, including OpenAI's baselines implementation, uses proportional sampling. If you were going to write your own, proportional sampling might be preferred for being simpler to implement (so less error-prone).

Comparing these two sampling methods is complicated by the fact that rank-based sampling involves a hyperparameter that determines how often you sort your replay buffer. The authors of the original PER paper only sorted every $n$ timesteps, giving the nice amortized time of $O(\log n)$, where $n$ is the size of the replay buffer. Sampling takes constant time, so sampling a minibatch of size $k$ takes $O(k)$ time.

Proportional sampling doesn't involve sorting, but it does need to maintain a sum tree structure, which takes $O(\log n)$ time each time we add to the buffer. Sampling also takes $O(\log n)$ time, so sampling a minibatch of size $k$ takes $O(k\log n)$ time.

If we only sort our replay buffer every $n$ timesteps, then rank-based sampling is faster. However, because the buffer is almost always only approximately sorted, the distribution we sample from is only a rough estimate of the distribution we wanted. It's not clear that this estimation would be accurate enough to be performant when the replay buffer is scaled up in size past the $n=10^6$ transitions used in the paper. I haven't seen a study that compares performance for different frequencies of sorting and different buffer sizes.

So, rank-based sampling might be faster, but it also might not work as well. Adjustment of the sorting frequency hyperparameter might be necessary. The simpler and surer approach would be to use proportional sampling with clipped TD-errors.


Should I choose a model with the smallest loss or highest accuracy?

You should choose the model A. The loss is just a differentiable proxy for accuracy.

That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine the data which produce high loss and check for presence of overfitting or incorrect labels.

If the higher loss is due to a regularizer then reducing the regularization factor may further improve the results.


How to define a "don't care" class in time series classification in Pytorch?

Don't care classes are not advisable since don't care doesn't have any inherent pattern which would hurt your model inadvertently.
You can assign class weights for your classes. CrossEntropyLoss has an argument 'weights'.
BatchNormalization can be left alone since the normalization should happen on the basis of all data and not just some of the data.