You can cluster all your features in one matrix X, in which each line would be one element of the data set you want to construct, and each column would be a different feature of this element.

You construct then a Y vector containing the different target classes, where the i-th element will be the target class of the i-th X element.

For the following I suppose you use python

Create train and test sets using sklearn wrapper train_test_split on X and Y just built, in which you can shuffle your X and Y (non-avoidable in this case) :

from sklearn.model_selection import train_test_split

BTW, I recommend you to rescale your data with sklearn with for example one of :

from sklearn.preprocessing import MinMaxScaler, StandardScaler

It allows better learning and so better generalization


I'd describe this as a Ranking.

However, it's not always possible to distil a set of outcome preferences to a transitive metric, this is one of the central axioms of game theory. The Transitivity Axiom so the problem is relatively well understood (that is, at least, if I've understood your problem)


In number classification using neural network, is training with edge image better than gray image?

Theoretically you will have no gain in the error ratio if the system preprocess the images with a linear high-pass filter before to send the image to the NN.

Let see a simple 1-dimension case that supports this statement:

Assume inputs are "a", "b" and "c". A node at first hidden layer will receive an input to the activation function equal to s=w1*a+w2*b*w3*c+... being w1, w2, w3, ... the weights for this node.

Now, assume a simple differential case, where inputs will be ... , b'=b-a, c'=c-b, ... being the input to the hidden node s=w1'a'+w2'b'+w3'c'+...=...+w2'(b-a)+w3'(c-b)+...=...+(w2'-w3')b+...

Note both inputs to hidden node are the same if w1=w1'-w2', w2=w2'-w3', ... . So, the NN can itself perform the equivalent of the linear high-pass filtering adjusting the weights of the first hidden layer.

However, changes in learning speed (convergence time) can be expected.

(ps: please, activate tex/latex in this site, urgently ! )


Natural language recommendation system: to pre-classify inputs or not?

Likely yes!  When you split out the inputs like this, you are adding information.  How much this helps is an open question till you build your system, get some data and start training.

Of course, it would be marvelous to have a machine do all the work straight unstructured text - but you want a functional, easy-to-use website, not a research project of your own.  To that end, do everything that you can to constrain the scope of the problem, and maximize the information available to your model.  For example, you might want to see if you can add researchers using Google Scholar (so you can link to their profiles, and perhaps mine some information that way).

You'll be somewhat 'data constrained' till you've got a decent number of research proposals and user interactions to learn from.  Tools like our NLP architect may help you get more out of your text (there are some other really cool new-generation ML-for-NLP packages you should also evaluate).


What is the scope of real-world deep learning applications in 2020?

Deep learning is used to perform language translation in Google Translate [1]. Specifically, Google Translate now uses transformers and RNNs rather than the original GNMT system (proposed in 2016), which was also based on neural networks. Deep learning is also used in DeepL (though I cannot find a good resource to cite apart from Wikipedia [2] given that the system is closed-source), a strong alternative to Google Translate. However, note that, in general, machine translation is still far from perfect and it is probably not adopted to perform serious translations.
Tesla's autopilot also uses neural networks [3].

Built on a deep neural network, Tesla Vision deconstructs the car's environment at greater levels of reliability than those achievable with classical vision processing techniques.

DeepFakes are also developed using deep learning techniques [4].


Are there deep neural networks that have inputs connected with deeper hidden layers?

This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: ResNet, SkipRNN. In addition here you can find a paper that empirically explores the skip connections for sequential tagging, or this one for speech enhancement.


How to have zero value or a value between 200 and 400 in the output of a deep learning model?

generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq b_1$ where $b_0$ and $b_1$ are your bounds.
In thios setup, during inference you would check $h_k$ and if its greater than some threshold (usually .5), youd evaluate/output $f_k$.
In this case your loss would look something like $L_i = L_i^{(1)}(h_i, y_i)+y_i*L_i^{(2)}(f_i,v_i)$ where $(y,v)$ are your labels, and the $L$'s are your loss of choice.


Are ReLUs incapable of solving certain problems?

There are a variety of possible things that could be wrong, but let me give you some potentially useful information.
Neural networks with ReLU activation functions are Turing complete for a computation with on order as many steps as the network contains nodes - for a recurrent network (an RNN), that means the same level of turing completeness as any finite computer. In other words, for any function/algorithm that you want to compute, you can devise a neural network, potentially recurrent, that will approximate/compute it.
As an example, suppose that we want to compute the NOR function, which can be used to implement a Turing machine.  We can do it with the following neural network with a ReLU activation function.
Let the input be
$$ W = \begin{bmatrix}x_1 \ \ x_2 \end{bmatrix}$$
the weight matrix be
$$ W = \begin{bmatrix} -20 \\ -20 \end{bmatrix}$$
and the bias be
$$ b = \begin{bmatrix} 1 \end{bmatrix}$$
Then the ReLU unit (or neuron) performs the following operation
$$o = \max(Wx + b, 0)$$
So, $o = 1$ only when both $x_1$ and $x_2$ are $0$, otherwise, it's always $0$.
However, gradient descent is a finicky way to search for RNNs. There are a wide variety of ways that it might have been failing. In general, once you have very thoroughly checked your gradient, I'd make sure to use Adam as the optimizer and then play with the hyperparameters endlessly until I find an incantation that works.
For further reading on general understanding of this level of deep learning's capability limitations, I'd recommend this blog post by Ilya Sutskever, now an OpenAI researcher: http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html?m=1


Is it possible to manage the evolution of super-intelligent AI?

Assuming super-intelligence is possible, the answer is probably yes and no.

Yes in Kurzweil-like scenarios, where super-intelligence is an extension of human beings by technology (we are already in to some extent). Then control follows, as super-intelligence depends on us. It would extend our capabilities, such as speed of processing, extent of processing, etc. Even then control is debatable, as a remote-controlled killing machine would be part of a super-intelligent organism, partially human "controlled", partially autonomous.

No in "Future of Life Institute"-like scenarios, where super-intelligence is independent from humans. The thinking is simple: What can we hope to do facing someone way more intelligent? The usual parallel is to compare this scenario with the arrival of the "developed" conquistadors in early America. Gunpowder vs. mere raw strength and arrows.


How can a specific connectivity pattern be stored in an optimally compact representation?

Solution

In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.

The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.

Encoding of ordinals
Lossless conversion of IEEE floats
Serialization of the structure
Lossless compression of the stream

Upon retrieval, the elements are reversed.

Decompression of the stream
Parsing to reassemble structure
Expansion of IEEE float representations
Decoding of ordinals

The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.

If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.

Notions of General Intelligence

Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.

The complexity of networks in the brain
That there is a significant varieties of neurons in brains
The impact of neuroplasticity
Complexity of chemical signaling independent from nerves
Whether intelligence is in fact multi-dimensional
Large number of genes that correlate with intelligence
The academic slant of concepts regarding intelligence
Difference between brain signal topology and computing models
Data-centricity

There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.

Why did Artificial Intelligence projects fail?
Can the IQ of an AI program be measured?
Can a brain be intelligent without a body?
Is the singularity concept mathematically flawed?
Is there a ceiling to intelligence itself?
Self-conscious artifical intelligence
Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?

There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.


Why are inhibitory connections often used in virtual neural networks when they don't seem to exist in real life neural networks?

I think you are confused. The reason why some neural neural networks have neurons with an output in range (-1, 1) all depends on the activation function used. Some networks even have neurons with an output range of (-Infinity, +Infinity) ( aka the identity function ).

I advise you to take a look at this list: activation functions w/ ranges.

Nowadays a lot of neural networks have a mixture of different activation functions. E.g. combining ReLU with Sigmoid or TanH.

This neural network library even shows that for most simple problems, neural networks prefer a variety of activation functions.

The red lines just mean that that connection has a low value in comparison with other connections.

There is no rule of thumb for activation functions, some problems benefit from a certain function, some don't.


Machine Learning hardware usage in embedded applications

Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.

So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess"  what each animal is.

The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data.

What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was.


Reinforcement learning with uniformly random dynamics

Convergence guarantees for basic RL algorithms like policy gradient / actor-critic methods make no assumptions about the dynamics of the MDP. So, theoretically, you don't need to change much.

Practically, when the number of possible trajectories from any given state is so high, the return from each state will have high variance. This means you'll have to collect much more experience for your estimates of expected return to converge to their true values. Intuitively, an environment with high uncertainty requires the agent to do more knowledge-gathering to behave optimally.

My real advice to you depends on what exactly you're trying to do. If you want to have the kind of agent that could learn to behave well in an extremely random environment, then all you need to worry about is giving it enough experience to learn from.

(Your agent should also take a little longer before deciding it's "confident" in its evaluation of different states. That is, don't behave greedily before you're sure your estimates are accurate. Explore adequately. This advice is only relevant if your MDP dynamics aren't actually completely uniform.)

If, however, you want to train an RL agent specifically to solve a problem formulated as an MDP with uniformly random dynamics, then I would tell you to not waste your time. We know before spending the computation that all policies would be equally good/bad in this setting. Since actions are irrelevant to the environment, it would be inefficient to deploy an RL agent that will only learn that which action it takes doesn't matter.

As noted in the comments, the last paragraph is only true when reward from each state-action pair $(s,a)$ is also uniformly random. If it is not, just being aware of the high variance and giving your agent a lot of experience should do the trick.


Which algorithm is used in the robot Sophia to understand and answers the questions?

Sophia ,first , has all the questions and corresponding answers preprogrammed. It is a system which is a hybrid of Bayes Text classification and decision trees.

It may consist of a speak recognizer which converts the question into a string.
This string then travels into a algorithm which gets a suitable answer for it.
The answer is then spoken by the speech synthesizer along with some face actions.
It might also have a sentiment analyser which identifies sentiments in a answer to show face expressions accordingly.
Along with this, it identifies the emotions on the face of the person sitting in front of it, so as to provide a greater expression.

Hence, Sophia understands or thinks nothing like the human brain. It is just a system which gives predefined answers corresponding to the question asked.


A way to leverage machine learning to reduce DFS/BFS search time on a tree graph?

It sounds like you may be looking for the A* search algorithm. It is a search algorithm, like DFS and BFS, but it will explore only the most promising branches based on a heuristic function you supply. The difficult part of implementing this is deciding on a low-cost, admissible heuristic.
Excited to reflect nbro's suggestion from comments.


Q-values represent expected return after taking action $a$ in state $s$, so they do tell you how good it is to take an action in the specific state. Better actions will have larger Q-values. Q-values can be used to compares actions but they are not very meaningful in representing performance of the agent since you have nothing to compare them with. You don't know the actual Q-values so you can't conclude if your agent is approximating well those Q-values or not.

Better performance metric would be the average reward per episode/epoch, or average reward in last $N$ timesteps for continuing tasks. If your agent is improving its performance then it's average reward should be increasing. You said that you have rewards per state and that some of them represent more important criteria then others. You could plot the average reward per episode by doing some kind of weighted linear combination of criteria rewards
\begin{equation}
\bar R = \bar R_1 w_1 + \bar R_2 w_2 + \bar R_3 w_3 + \bar R_4 w_4
\end{equation}
where $\bar R_i$ is the average episode reward for criteria $i$.That way you can provide more importance to some specific criteria in your evaluation.


How can max-pooling be applied to find features in words?

In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.

For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others.

MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.

When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is searching for a feature, and so you use an exorbitant number of the kernels at various sizes to search for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used.

A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).


Are sentences from the same document independent and identically distributed?

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the occurrence of another sentence $s_j$. If they are not independent, they can also not be independent and identically distributed (which is a stronger condition). The same can be said for words in the same sentence.


Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

What normally happens in DQN is the following:

First, the NN is used to estimate an approximation of Q for each state-action pair by using a vector of weights:
$$Q(s,a,w) \simeq Q(s,a)$$

There are two options: You can either feed it a certain state, and get as output the value of each possible action, or, you can input a state-action pair and get as output the value of that pair. (basically the NN gives you the policy)

After having this output you can now use your environment to simulate the next state. So basically you apply the action given by the NN (maybe using $\epsilon -greedy$), and you get the next state, and your next reward from your environment by observing it.

Where does the Q function comes in all of this? The Q-function
$$Q(s,a) = Q(s,a) + \alpha [r(s,a) + \gamma  max_a{Q(s',a)} - Q(s,a)]$$
that in standard Q-learning is used to update the values after each simulation step, is now used as the Target of the neural network, to make the back propagation.

There are some considerations you have to make to help this converge, mainly experience replay and fixed Q-targets.

So when you say " I don't have yet the next state", is because you need to simulate the action that is selected to get this next state.

EDIT: The main thing you might be missing is that the right part of the equation you present in the title is called TD-Target, and it is not used to make the updates, you might see it as the final result you want to achieve $Q^*(s,a)$ (the optimal values, after convergence). But to make the update you need to use the weighted function with the previous value of $Q(s,a)$ as I showed above, using a learning rate $0< \alpha < 1$


How to know if the hyperparameters of a neural network relate to each other?

This is one of the most difficult and unsolved problems in machine learning and deep learning!

There are many different ways to estimate the most appropriate hyper-parameters, such as grid search, random search, Bayesian optimization, meta-learning, reinforcement learning, and evolutionary algorithms (e.g. NEAT).

However, the problem is that most if not all these approaches are typically not very computationally efficient if your model isn't anything but very small. The number of possible configurations of the hyper-parameters is very large. If you don't have good computational resources (e.g. GPUs and powerful servers), you probably out of luck or require some days to get some insight.

In certain cases, it is obvious that certain hyper-parameters are dependent on each other, e.g. in the case of the batch size and the learning rate, because we have some decent understanding of gradient descent, but, in other cases, the situation isn't so nice.

As far as I know, there isn't a very good general rule of thumb or method to solve this issue (i.e. find the dependence of hyper-parameters on each other). Maybe, as our knowledge of our models (especially, neural networks) increases, we'll get some more insights and we'll develop more efficient approaches to understand the dependence of the hyper-parameters.

Nowadays, there's automatical machine learning (AutoML), which is a fancy name to denote services that provide hyper-parameter optimization (plus some other stuff).


The problem was the dimensions of the logit_length argument to tf.nn.ctc_loss was incorrect.
It was this:
tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])

But it should have been
tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])


Why has cross entropy become the classification standard loss function and not Kullbeck Leibler divergence?

When it comes to classification problem in machine learning, the cross entropy and KL divergence are equal. As already stated in the question, the general formula is this:

$$H(p, q) = H(p) + D_{KL}(p||q)$$

Where $p$ a “true” distribution and $q$ is an estimated distribution, $H(p, q)$ is the cross-entropy, $H(p)$ is the entropy and $D$ is the Kullback-Leibler divergence.

Note that in machine learning, $p$ is a one-hot representation of the ground-truth class, i.e.,

$$p = [0,..., 1, ..., 0]$$

which is basically a delta-function distribution. But the entropy of the delta function is zero, hence KL divergence simply equals the cross-entropy.

In fact, even if $H(p)$ wasn't $0$ (e.g., soft labels), it is fixed and has no contribution to the gradient. In terms of optimization, it's safe to simply remove it and optimize the Kullback-Leibler divergence.


What is the purpose of the arrow $\leftarrow$ in this formula?

I assume this is an iterative function. It means the current $V(S_t)$ is the sum of the previous plus some adjustment. The arrow is like an assignment.

In code, you would do

vst = vst + alpha * (gt - vst)

So vst will be overwritten.


Why is it called back-propagation?

I don't think there is anything special here!

It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).

It's also called BP because it is just the application of the chain rule. Why is this interesting?

Let me answer this question with an example. Consider the function $y=e^{\sin(x^{2})}$. This is a composite function, i.e. a function composed of multiple simpler functions, which, in this case, are $e^x$, $\sin(x)$, $x^2$ and $x$. To compute the derivative of $y$ with respect to $x$, let's define the following variables

\begin{align}
y &= f(u) = e^u,\\
u &= g(v) = \sin v = \sin(x^2),\\
v &= h(x) = x^2
\end{align}

The derivative of $y$ with respect to the variable $x$ is (according to the chain rule)

$$
\underset{\color{red}{\LARGE \rightarrow}}{
\frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv}  \color{green}{\cdot}
\frac{dv}{dx}}
$$

If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $y$ to the function $v$). This is the same thing with BP!

Why is it called "chain rule"? Because you are chaining different partial derivatives. More specifically, you are multiplying them.

BP is also known as the reverse mode of automatic differentiation. Why? The automatic differentiation should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this automatically, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $e^x$) to the inner function (which, in the example above, is $x$). The Wikipedia article related to automatic differentiation provides more details.

  What exactly are you back-propagating?

The partial derivative of the loss function $\mathcal{L}$ with respect to a parameter $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$, intuitively, represents the "contribution" of the parameter $w_i$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $w_i$ as follows

$$
w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i}
$$

where $\frac{\partial \mathcal{L}}{\partial w_i}$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.

This gradient descent step will hopefully make your network produce a smaller error next time.

The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, although some people claim this is not the case because they want to give credits to others (because Seppo Linnainmaa didn't reference neural networks, but back-propagation isn't only applicable to neural networks) or they don't even know the history. This Jürgen Schmidhuber's article goes into the details of the history of BP.


Should the exploration rate be reset after each trial in Q-learning?

The exploration rate, typically parameterized as epsilon / ε, can be changed on every trial. It depends on the complexity of the model and the goals.

The simplest thing to do is keep exploration rate high and fixed. That means the model will continue to explore new options, even at the cost of not "exploiting" the best available option.

Another option is setting the exploration rate high at the beginning of learning so the model will search the space for possible successful solutions. Then as the model creates a set of policies that are successful for given states, the exploration rate can be lowered or decay. Exploration rate decay can be fixed (i.e., over time there is consistently less exploration and more exploitation). Exploration rate decay can be dynamic and learned. This last option is often the best but also the most complex to implement.

"Dare to Discover: The Effect of the Exploration Strategy on an Agent’s Performance" goes into greater detail on this topic.


Deep audio fingerprinting for word search

You would need to perform some kind of speech-to-text to get the audio transcription with the corresponding synchronization wrt the audio. Then search in the transcription.

You could use DSAlign by mozilla


Why does the bias need to be a vector in a neural network?

In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a single scalar bias value per layer will most of the time reduce the effectiveness of a neural network through lost flexibility in how it fits to the target function.

Once you have $N$ output neurons in a layer leading to needing $N$ values for bias, then it is fairly straightforward to model this collection of bias values as a vector.

Often you will see a neural network layer function written in this form or similar:

$$\mathbf{y} = f(\mathbf{W}\mathbf{x} + \mathbf{b})$$

Where $f()$ is the activation function (applied element-wise), $\mathbf{W}$ the weights matrix for the layer and $\mathbf{b}$ is the bias. When written in this form, it is easy to see that $\mathbf{y}$, $\mathbf{W}\mathbf{x}$ and $\mathbf{b}$ must all be vectors of the same size.

This layer design has become so standard that it is possible to forget that other designs and implementations are possible for neural network parameters, and can sometimes be useful. Frameworks like TensorFlow also make it easier to take the standard approach, which is why you need a vector for bias on the example you are using. Whilst you are learning, and probably 99% of the time after that, it will be best to go with what the framework is doing here.


Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition
$$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$
where $a_t$ is the learning rate at iteration $t$, does not hold (given that a number bigger than $1$ squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a minimum [1] (although the condition \ref{1} is a sum from $t=0$ to $t=\infty$, but, of course, we only iterate for a finite number of iterations). Moreover, note that, if the learning rate is bigger than $1$, you are essentially giving more weight to the gradient of the loss function than to the current value of the parameters (you give weight $1$ to the parameters).
This is probably the main reason why the learning rate is usually in the range $(0, 1)$ and there are methods to decay the learning rate, which can be beneficial (and there are several explanations of why this is the case [2]).


Why is embedding important in NLP, and how does autoencoder work?

The subword-based embedding is rather visual and easily understandable. However, the autoencoder embedding is what machines understand the componential meaning of words.

1) An autoencoder embedding layer can be trained together with other layers to fit with the relation of data in dataset.

2) Or the embedding layer can be kept unchanged as used as a function, required that the embedding layer must have been trained on a similar task like said in 1)

And as stated in the question, embedding is important in NLP as word indices don't show the full meanings of words, they must be separated into embedding values for better efficiency.

TensorFlow random uniform trainable embedding: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding

TensorFlow utility class for creating subword-based encoder: https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder


How do you calculate KL divergence on a three-dimensional space for a Variational Autoencoder?

Your three dimensional latent representation consists of two images of mean pixels and covariance pixels as shown in Fig. 3. Which represents a Gaussian distribution with the mean and covariance for each pixel in the latent representation. Each pixel value is a random variable.
Now, have a close look at KL-loss Eq. 3 and it's corresponding description in the paper:
$$\mathcal{L}_{KL} = \frac{1}{2 \times (\frac{W}{16} \times \frac{H}{16}) } \sum^M_{m = 1}[\mu^2_m + \sigma^2_m - \log(\sigma^2_m) - 1]$$

Finally, $M$ is  the  dimensionality  of  the  latent  features $\theta \in \mathbb{R}^M$ with  mean $\mu  = [\mu_1,...,\mu_M]$ and covariance matrix $\Sigma = \text{diag}(\sigma_1^2,...,\sigma_M^2)$, [...].

The covariance matrix is diagonal, thus all pixel values are independent of each other. That is the reason why we have this nice analytical form for the KL-divergence given by Eq. 3. Therefore you can treat your 2D random matrix simply as a random vector of size $M = \frac{W}{16} \times \frac{H}{16}$ ($\times 3$ if you like to include color dimension). The third dimension (RGB channel) can be considered independent as well, therefore it can be also flattened to a vector and appended. Indeed this is what is done in the paper indicated by the second half of the sentence from above:

that are reparameterized by via sampling from a standard multivariate Gaussian $\epsilon \sim \mathcal{N}(0,I_M)$, i.e. $\theta = \mu + \Sigma^{\frac{1}{2}}\epsilon$.


How to design an ANN to give an answer that includes two different components?

Nim is a simple game and it's really simple to build a bot that gives the optimal solution.

  The correct move is to leave an odd number of piles of size 1

So when it comes to training an ANN to play a game, there are some this to keep in mind.

Fixed play area (This is taken care of as there are only 3 stacks)
Providing input to the model so it knows it's current state (You can use a simple array of integers [3, 4, 1] denoting the current state of the stacks)
Providing feedback. An important step to guide the network in the right direction (You already have an optimal bot to do this)

Now that you can easily cover all the requirements, it's pretty simple to teach the model.

Input - Current state of the model [3, 4, 5]
Output - Move that model will make [2, 0, 0]
Final output - Here you will have to add another layer with a custom function.

This is the important part, to direct the model. Check if the move made is GOOD, BAD or ILLEGAL and assign a custom output to the same. For example

GOOD return1 if it matches one of the optimal moves.
BAD return 0 if it's not the optimal move and
ILLEGAL return -1 if the move is not allowed.

Another thing to keep in mind is that this might require a slightly larger network as it as to learn complex functions that are not linearly separable.

You don't need to train that model by playing it against itself as you already have the optimal solution. Playing against itself is required only when you are trying to achieve something that better than the current best agent.

Check this out, it's pretty interesting Neural network to play snake


What could an oscillating training loss curve represent?

Try lowering the learning rate.

Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss the local minima. Then it will try to come back to the minima in the next step and overshoot it again.

You may also try switching to a momentum-based GD algorithm. Such a training loss curve can be indicative of a loss contour like in this example, for which momentum-based GD methods are helpful.

I noticed that you have a very small training set. You may have better luck with a larger training data (~1000 examples) or using a pre-trained Conv network as a starting point.


How to compare the training performance of a model on different data input?

Well, this is more of a subjective question but I will give my best shot.

Regarding your 1st question, the nature of methods in deep learning states that you should experiment otherwise you will only have weak intuitions. So which dataset should you choose? I would say, select your primary concerns since you cannot try everything. If it's time, make a little random or grid search over hyperparameters and observe speed of convergence in the datasets then choose the best one. If it's accuracy, ideally you should make an input analysis on distribution of data, etc. If you can analyze it well, you are expected to obtain best result from the chosen dataset. So training 30 epochs each dataset and choosing the one with lowest loss is not a safe approach. Maybe your model will converge at 40th epoch for the worst dataset in the 30th epoch but it will be much more robust. My final advice is to set a threshold for your evaluation metrics, once you reach them in any of your dataset choose that one -assuming datasets are more or less equal in number of instances. In this way you are at least know that the dataset you choose satisfies your expectations.

For your 2nd question, seeding during comparisons is a good approach though in the long run it won't matter much. Hence there is no harm in seeding unless you are very very very unlucky.


Are there any machine learning techniques to detect coding standard violations?

If the system claims that a piece of code has violated standards, then to be useful to the programmer, it really needs to provide more information than just a 'yes/no' classifier: you need some form of explanation about why it is claimed to be wrong.

Clearly ANNs aren't much use for that.

If I were tackling such a problem (and my suspicion is that a lot of effort could be spent trying and failing to reproduce coding standards which are already well-understood), then my inclination would be to use a more explicitly rule-based representation.

Possibilities include:

Genetic Programming
Learning Classifier Systems
Decision Trees

The ever-useful "Essentials of Metaheuristics" has a whole section on the evolution of rulesets. Obviously, nothing prevents you from initializing the evolutionary process with rules known to be useful.

As I point out here, with our current AI algorithms, the success of an  approach is very sensitive to human expertise/effort in feature selection/preprocessing, choice of training set etc, so creative experiment with this is vital.

Training set: how about two sets of negative and positive examples, consisting of (features extracted from) bad code and from a refactored version (respectively)?

One elementary choice of features would be to apply a bunch of code complexity metrics and have the learning algorithm combine those. The plus side of working in such a numeric domain is that the learning algorithm might readily find a gradient to exploit. The downside is that the rules (which are then likely of the form if mcabe > 2.8 etc) are still not as informative as might be desired.

For more complex rules (e.g. requiring if elseif else) you may want to extract your features from the abstract syntax tree. You could in principle use the entire tree but to my knowledge ML on graph and tree structures is still in relative infancy.


Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?

The main difference between an MDP and contextual bandit setting is time steps and state progression. If those are important to the problem you want to solve, then it is not possible to convert.
Essentially MDPs are a strict generalisation of contextual bandits. You can model a CB as an MDP but not vice-versa.
In some very specific cases you can convert MDP to CB - any one of these situations means that the MDP can be simplified to CB, and then you could use bandit solving algorithms to optimise it:

When there is only one time step for an episodic problem.

When the discount factor is zero.

When state transition rules are completely independent of action choice, but reward is not.


How to forecast multiple target attributes in Python?

In general, multi output models is not that different.
I.e.

As Raghu mentioned in commentary, you could train separate model for each output. There is even helper module in sklearn for that (MultiOutputRegressor)
DecisionTreeRegressor from sklearn allows multiple outputs out-of-the box
Any neural network framework allows any number of outputs

In your particular case there are two much bigger factors.

It's timeseries. Timeseries requires whole another set of models then other data, i.e ANOVA, RNN or transformers.
You data is not well scaled. The reason is that the change between points is insignificant in comparison with their absolute values. You need to deal with it -- but exact solution would depend on your particular task. You could try to substract the mean value on train set or predict the difference, not absolute value.

Edit: here is simple example
import numpy as np
raw = np.array([
[414049364,21773560],
[414049656,21773926],
[414049938,21774287],
[414050204,21774638],
[414050453,21774975],
[414050682,21775296],
[414050895,21775597],
[414051093,21775874],
[414051278,21776125],
[414051453,21776344],
[414051620,21776530],
[414051780,21776678],
[414051935,21776785],
[414052089,21776849],
[414052242,21776865]
])

X = raw[:-1]
y = raw[1:]
from sklearn.linear_model import LinearRegression
model = LinearRegression()
# Train the model
model.fit(X, y)
# Try to predict next step
model.predict([[414051780,21776678]])
# The predicted value [414051933,  21776806] and the true value would be [414051935,21776785], not so far

Next you could normalize the values, use few previous steps as a features, use more complex algorithm then linear regression, and make train/eval split for better model evaluation.


Is there an AI that can extract proper nouns from free text?

This is a hard problem, unless you have a list of proper nouns you want to recognise. If John Allen is in this list, then you can easily use a longest match to prefer it over John or Allen. The same applies to the other examples you give.
Capitalisation on its own is not very reliable, as words at the beginnings of sentences are also capitalised, and sometimes technical terms or emphasis are expressed By Capitalising Them In Mid-Sentence. It's not what I would do, but you really have to expect anything in free text.
You could go some way by looking for sequences of proper nouns with of, de or van etc between them. There is probably a reasobaly small list of those connectors.
You can set up a grammar to capture complex names, but by far the most reliable way is to have a list. Unfortunately there is no easy solution for this. I would approach it iteratively, ie process a segment of text, tidy up the results, generate a list from them, and repeat with more text. Everything that is in the list you accept, other candidates you vet before adding them to the list. Eventually you should get fewer candidates that are not in the list.
You can probably kick-start your list with a gazetteer, ie a list of names and places. There are several of those on the web.
In general this procedure is referred to as Named Entity Recognition (NER). There are many ways to solve it, and my suggestion here is a pragmatic approach, which works reasonably well with not too much effort.


Your first approach doesn't make any sense to me. After all, you are not just interested in the centre pixel are you? And if you have two nodes for every pixel, what are those two nodes encoding? For the probability of water you just need one.

So clearly approach two. I would just use 0.5 as cutoff. Using a higher or lower cutoff only makes sense, if either false positives or false negatives are for some reason more problematic. Using some additional heuristic to adjust the given probabilities would just do what the ConvNet should already have done.

If 4096 output nodes is too expensive you can always make it a bit fuzzier, for example by predicting the probability that at least one of four pixels shows water, reducing the number of nodes to 1024.


What is the main difference between additive rewards and discounted rewards?

I will disagree slightly with @mico. There is a usage of "additive rewards" that refers to decomposable reward functions (e.g. my reward in selling an item I do not want to own is composed of the reward of not having an unwanted item anymore, plus the monetary gain in selling the item). But, there is indeed a fundamental relation between additive and discounted rewards. Additive rewards are formulated simply as

$$
R([s_0, s_1, s_2, ...]) = R(s_0) + R(s_1) + R(s_2) + \cdots
$$

whereas discounted rewards include a discount factor $\gamma \in [0,1]$ such that

$$
R([s_0, s_1, s_2, ...]) = R(s_0) + \gamma R(s_1) + \gamma^2R(s_2) + \cdots
$$

Intuitively, the additive reward for a sequence of states is simply the sum of the rewards acquired at each state, while discounted rewards include a multiplicative discount factor that reduces the influence of rewards as time goes on. You will typically see additive rewards for finite-horizon problems, i.e. you have a discrete number of timesteps to optimize over, and discounted rewards are more relevant for infinite-horizon problems, i.e. you may need to optimize over an infinite number of timesteps (or at least a very large number). The discount factor governs the agent’s greediness in achieving immediate reward, where very small discount factors (closer to 0) encourage the agent to only seek rewards in proximal states, and very large discount factors (closer to 1) encourage the agent to think further into the future about what rewards it can expect to achieve in states it will visit later.

The most direct reference I found for this distinction is in these course slides, which are somewhat authoritative since they from Andrew Barto, a co-author of the de facto text on reinforcement learning.


How can these 7 AI problem characteristics help me decide on an approach to a problem?

The List

This list originates from Bruce Maxim, Professor of Engineering, Computer and Information Science at the University of Michigan.  In his lecture Spring 1998 notes for CIS 4791, the following list was called,

  "Good Problems For Artificial Intelligence."

  Decomposable to easier problems
  Solution steps can be ignored or undone
  Predictable Problem Universe
  Good Solutions are obvious
  Internally consistent knowledge base (KB)
  Requires lots of knowledge or uses knowledge to constrain solutions
  Interactive

It has since evolved into this.

  Decomposable to smaller or easier problems
  Solution steps can be ignored or undone
  Predictable problem universe
  Good solutions are obvious
  Uses internally consistent knowledge base
  Requires lots of knowledge or uses knowledge to constrain solutions
  Requires periodic interaction between human and computer

What it is

His list was never intended to be a list of AI problem categories as an initial branch point for solution approaches or a, "heuristic technique designed to speed up the process of finding a satisfactory solution."

Maxim never added this list into any of his academic publications, and there are reasons why.

The list is heterogeneous.  It contains methods, global characteristics, challenges, and conceptual approaches mixed into one list as if they were like elements.  This is not a shortcoming for a list of, "Good problems for AI," but as a formal statement of AI problem characteristics or categories, it lacks the necessary rigor.  Maxim certainly did not represent it as a, "7 AI problem characteristics," list.

  It is certainly not a, "7 AI problem characteristics," list.

Are There Any Category or Characteristics Lists?

There is no good category list for AI problems because if one created one, it would be easy to think of one of the millions of problems that human brains have solved that don't fit into any of the categories or sit on the boundaries of two or more categories.

It is conceivable to develop a problem characteristics list, and it may be inspired by Maxim's Good Problems for AI list.  It is also conceivable to develop an initial approaches list.  Then one might draw arrows from the characteristics in the first list to the best prospects for approaches in the second list.  That would make for a good article for publication if dealt with comprehensively and rigorously.

An Initial High Level Characteristics to Approaches List

Here is a list of questions that an experienced AI architect may ask to elucidate high level system requirements prior to selecting an approaches.

Is the task essentially static in that once it operates it is likely to require no significant adjustments?  If this is the case, then AI may be most useful in the design, fabrication, and configuration of the system (potentially including the training of its parameters).
If not, is the task essentially variable in a way that control theory developed in the early 20th century can adapt to the variance?  If so, then AI may also be similarly useful in procurement.
If not, then the system may possess sufficient nonlinear and temporal complexity that intelligence may be required.  Then the question becomes whether the phenomenon is controllable at all.  If so, then AI techniques must be employed in real time after deployment.

Effective Approach to Architecture

If one frames the design, fabrication, and configuration steps in isolation, the same process can be followed to determine what role AI might play, and this can be done recursively as one decomposes the overall productization of ideas down to things like the design of an A-to-D converter, or the convolution kernel size to use in a particular stage of computer vision.

As with other control system design, with AI, determine your available inputs and your desired output and apply basic engineering concepts.  Thinking that engineering discipline has changed because of expert systems or artificial nets is a mistake, at least for now.

Nothing has significantly changed in control system engineering because AI and control system engineering share a common origin.  We just have additional components from which we can select and additional theory to employ in design, construction, and quality control.

Rank, Dimensionality, and Topology

Regarding the rank and dimensions of signals, tensors, and messages within an AI systems, Cartesian dimensionality is not always the correct concept to characterize the discrete qualities of internals as we approach simulations of various mental qualities of the human brain.  Topology is often the key area of mathematics that most correctly models the kinds of variety we see in human intelligence we wish to develop artificially in systems.

More interestingly, topology may be the key to developing new types of intelligence for which neither computers nor human brains are well equipt.

References

http://groups.umd.umich.edu/cis/course.des/cis479/lectures/htm.zip


Any papers regarding different/inconsistent action space in Reinforcement Learning?

Does anyone know any paper regarding this subject?

I'm not familiar with any off the top of my head... I do know that the vast majority of Reinforcement Learning literature focuses on settings with a fixed action space (like robotics where your actions determine how you attempt to move / rotate a particular part of the robot, or simple games where you always have the same set of actions to move and maybe ''shoot'' or ''use'' etc.). Another common class of settings is where the action space can easily be treated as if it always were the same (by enumerating all actions that every could be legal in some state), and filtering out illegal actions in some sort of post-processing steps (e.g. RL work in board games).

So... there might be something out there, but it's certainly not common. Most RL people like to involve as little domain knowledge as possible, and I suppose that a function that generates a legal set of actions given a particular state can very much be considered to be domain knowledge.

  Is the terminology wrong? "Inconsistant", "Irregular", "Different"... ?

I wouldn't use inconsistent, because that word can be interpreted as implying that something would be "wrong" or "ill-defined". I would say that you have a variable action set (the action set varies per state). When I search for that, there aren't a lot of results either though... but I think that term would be more promising.

  Anyone having another approach worth digging into?

The problem you describe is mostly a problem in Reinforcement Learning with function approximation, in particular function approximation using Neural Networks. If you can get away with using tabular RL algorithms, the problem instantly disappears. For example, a table of $Q(s, a)$ values as commonly used in the tabular, value-based algorithms does not need to contain entries for all possible $(s, a)$ pairs; it's fine if it only contains entries for $(s, a)$ pairs such that $a$ is legal in $s$.

Variable action spaces primarily turn into a problem in Deep RL approaches, because we normally work with a fixed neural network architecture. A DQN-style algorithm involves neural networks that take feature vectors describing states $s$ as inputs, and provide $Q(s, a)$ estimates as outputs. This immediately implies that we need one output node for every action, which means you have to enumerate all the actions... which is where your problem comes in. Similarly, policy gradient methods traditionally also require one output node per action, which again means you have to be able to enumerate all the actions in advance (when determining the network architecture).

If you still want to use Neural Networks (or other kinds of function approximators with similar kinds of inputs and outputs), the key to addressing your problem (if none of the suggestions you've already listed in the question are to your liking) is to realize that you'll have to find a different way to formulate your inputs and outputs, such that you are no longer required to enumerate all actions in advance.

The only way I can think of doing that really is if you are able to compute meaningful input features for complete state-action pairs $(s, a)$. If you can do that, then you could, for example, build neural networks which:

Take a feature vector $x(s, a)$ as input, which describes (hopefully in some meaningful way) the full pair of the state $s$ and the action $a$
Produce a single $\hat{Q}(s, a)$ estimate as output, for the specific pair of state + action given as input, rather than producing multiple outputs.

If you can do that, then in any given state $s$ you can simply loop through all the legal actions $A(s)$, compute $\hat{Q}(s, a)$ estimates for them all (note: we now require $\lvert A(s) \rvert$ passes through the network rather than just a single pass as would normally be required in DQN-style algorithms), and otherwise proceed similarly to standard DQN-style algorithms.

Obviously the requirement of having good input features for actions is not always going to be satisfied... but I doubt there are many good ways to get around that. It's very similar to the situation with states really. In tabular RL, we enumerate all states (and all actions). With function approximation, we usually still enumerate all actions, but avoid the enumeration of all states by replacing them with meaningful feature vectors (which enables generalization across states). If you want to avoid enumerating actions, you'll in a very similar way have to have some way of generalizing across actions, which again means you need features to describe actions.


What’s the difference between LSTM and RNN?

You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more recurrent (or cyclic) connections. Or you could say that layer $l$ of neural network $N$ is a recurrent layer, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).
In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name). On the other hand, an LSTM can refer to an LSTM unit (or neuron), an LSTM layer (many LSTM units), or an LSTM neural network (an NN with LSTM units or layers), depending on the context.
An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so an LSTM network is a recurrent network. The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is more sophisticated. More precisely, it is composed of the so-called gates that supposedly regulate better the flow of information through the unit.
Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called peephole connection).

This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices). You can easily see from this diagram that an LSTM unit (or layer) is composed of gates and recurrent connections. It's also composed of a cell. To understand the details (i.e. the purpose of all these components, such as the gates), you should e.g. read the paper that originally proposed the LSTM by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers.
Given the presence of cyclic connections, any recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard RNN or an LSTM network (or maybe a variant of it, e.g. the GRU).

RNNs are particularly suited for tasks that involve sequences (because of the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it is more computationally effective. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the vanishing gradient problem.


Shortening the development time of a neural network

Your scenario is common.

The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be comparable. The accuracy might be lower, but the purpose is to do quick sanity check.


What's the difference between hyperbolic tangent and sigmoid neurons?

Sigmoid > Hyperbolic tangent:

As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.

Hyperbolic tangent > Sigmoid:

Hyperbolic tangent has a property called "approximates identity near the origin" which means $\tanh(0) = 0$, $\tanh'(0) = 1$, and $\tanh'(z)$ is continuous around $z=0$ (as opposed to  $\sigma(0)=0.5$ and $\sigma'(0)=0.25$). This feature (which also exists in many other activation functions such as identity, arctan, and sinusoid) lets the network learn efficiently even when its weights are initialized with small values. In other cases (eg Sigmoid and ReLU) these small initial values can be problematic.

Further Reading:

Random Walk Initialization for Training Very Deep Feedforward Networks


What happens if 2 genes have the same connection but a different innovation number?

When do mutations occur and between which nodes?

There are two types of mutations in the NEAT model, each of them appears randomly during one epoch on different individuals; the number of structures affected by mutations may vary depending upon the nature of the problem.

A new gene/node is added to the structure and properly linked.
A new connection between two nodes is added.

  When mating, what happens if 2 genes have the same connection but a different innovation number?

During a single epoch/generation every mutation is tracked and if the same mutation appears it can't have the same global innovation number. Same mutation, same innovation number. In that way, during the mating phase, there is no decisional problem which leads to prefer one structure to the other.


Additive Attention in Convolutional Networks

It seems that in the reference you provided they use attention to compute weights for the encoder representations based on the context provided by the decoder (Fig. 1). As far as I can tell, attention is applied after convolution (in fact, after the GLU step), so it does not affect the feature maps directly. Rather, attention is used to select the target words in the decoder. In other papers (e.g., this one), attention is applied directly to the feature maps in a way that is more similar to what you described.

Regarding your second question, the paper you reference actually provides a link to the source code they used. It is written in Lua (using Torch) instead of PyTorch, probably because PyTorch development was just starting when the paper was published. At any rate, you should be able to follow the Lua code and translate it into PyTorch.


Keywords to describe people counting from a camera?

I usually start with some papers and look at the references:

Counting people using video cameras
Sheng-Fuu Lin, Jaw-Yeh Chen, Hung-Xin Chao, Estimation of Number of People in Crowded Scenes Using Perspective Transformation,  IEEE Transactions on Systems, Man and Cybernetics, November 2001, Part A, Vol. 31, Issue 6, pp. 645-654.
A.  C.  Davies,  J.  H.  Yin,  and S.  A.  Velastin,  Crowd monitoring using image processing,  Electronics  & Communication Engineering Journal, February  1995, Vol. 7, Issue 1, pp. 37-47.

By this technique you only find older work.

To find more recent articles, use Google scholar and find who cites the articles you investigated.

Always write them down (e.g. in your bibtex file) to keep track of what you looked at already.

Good luck!


What does it mean to parameterise a policy in policy gradient methods?

In the context of RL, for a policy to be parameterised it typically means we explicitly model the policy and is common in policy gradient methods.

Consider value based methods such as Q-learning where our policy is usually something like $\epsilon$-greedy where we choose our action using the following policy

\begin{align}
\pi(a|s) = \left\{ \begin{array}{ll}
\arg \max_a Q(s,a) & \text{with probability } 1-\epsilon\;; \\ \text{random action} & \text{with probability } \epsilon\;.
\end{array}\right.
\end{align}
Here we have parameterised the policy with $\epsilon$ but the learning is done by learning the Q-functions. When we parameterise a policy we will explicitly model $\pi$ by the following:
$$\pi(s|a,\boldsymbol{\theta}) = \mathbb{P}(A_t = a | S_t=s, \boldsymbol{\theta}_t = \boldsymbol{\theta})\;.$$
Learning is now done by learning the parameter $\boldsymbol{\theta}$ that maximise some performance measure $J(\boldsymbol{\theta})$ by doing approximate gradient ascent updates of the form
$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t + \alpha \hat{\Delta J(\boldsymbol{\theta}_t)}. $$

Note that, as per the Sutton and Barto textbook, $\hat{\Delta J(\boldsymbol{\theta}_t)}$ is a noisy, stochastic estimate of $\Delta J(\boldsymbol{\theta}_t)$ where the former approximates the latter in expectation.

The policy can be parameterised in any way as long as it is differentiable with respect to the parameters. Commonly in Deep RL the policy is parameterised as a neural network so $\boldsymbol{\theta}$ would be the weights of the network.


What could be the cause of the drop in the reward in A3C?

I'll share my understanding so far.
This kind of behavior is actually normal when using on-policy algorithms with a sparse final reward.
Issue stems from the fact that once you get stuck in a behavior policy which does nothing (uses a "do nothing" action, for instance, until timeout), it's quite hard to get out of it, because you keep getting experiences that teach you nothing (no reward signal at all) and keeps you in the same policy.
Possible mitigations:

Encourage more exploration (in A3C, make the entropy loss coefficient bigger) in order to recover more quickly from this type of stationary behavior.
Use an off-policy algorithm with a big enough replay buffer, so that even if you start behaving this way you still use experiences from a "healthy" older policy.

If you stick to a totally on-policy algorithm, making the batch_size bigger might help a little.


How can we compute the ratio between the distributions if we don't know one of the distributions?

The rationale behind importance sampling is that $q(x)$ is difficult to sample from but easy to evaluate. Or at least you can easily evaluate some $\tilde{q}$ such that:
$$
\tilde{q}(z) = Zq(z)
$$
where $Z$ (scalar) might be unknown. The geometrical example would be here e.g. sampling uniformly from an area under the curve $q(x)$ (in general it's not easy).

  Because if we know $q(x)$, then we can compute the expectation directly.

That's the task we're trying to solve to begin with. And calculating expectation might be hard if we can't sample efficiently from $q$.

Say you want to compute an expectation of $x$, $E[x]$. For this you need to calculate the following integral:
$$
E[x] = \int{xq(x)dx}
$$
where $q$ is a probability distribution of $x$ for which you have an expression - so you can evaluate $q(x)$ (up to the constant of proportionality). This integral might be hard to evaluate analytically so we need to use other methods such as Monte Carlo. Let's say it is hard to generate samples from $q$ (as per example above, e.g. generating samples from the area under the curve $q(x)$ uniformly).

What you can do is to calculate an expectation under a simple distribution $p$ (proposal distribution) which is a distribution of your choice that needs to allow you to easily sample from it (say Gaussian). Then you can rewrite your integral as:
$$
E_q[x] = \int{xq(x)dx} = \int{xq(x) \color{blue}{\frac{p(x)}{p(x)}} dx}
= \int{x \frac{q(x)}{\color{blue}{p(x)}} \color{blue}{p(x)} dx}
= E_p \Big[{x\frac{q(x)}{p(x)}}\Big]
$$
(added index $p$ and $q$ to expectation to denote the sampling distribution). Now you can approximate the last expectation by Monte Carlo:
$$
E_p \Big[{x\frac{q(x)}{p(x)}}\Big] = \frac{1}{S} \sum_{s}{x^{(s)} \frac{q(x^{(s)})}{p(x^{(s)})} }, \ x^{(s)} \sim q(x)
$$


Based on the Turing test, what would be the criteria for an agent to be considered smart?

If an artificial agent (AA) passes the (standard) Turing test (i.e. where you have to imitate a human that speaks), then, on average, the AA should be able to imitate any human in any situation that mainly requires the conversation abilities and common-sense knowledge of a human, without being ever recognized as an AA.
For example, if you want to talk about football, you don't expect the AA only to say "I don't know" or to clearly avoid a topic (e.g. by redirecting you to a search engine) when it doesn't know something (although some humans behave in this way), but you expect it to have common-sense knowledge, such as that Messi, Cristiano Ronaldo, Pelé, Maradona, etc., are among the best players of all time, and this should be precious information that the AA should have in any case, even if it doesn't know much about football.
You also expect it to be emotional and have a personality, given that humans are emotional and have personalities. So, in the example above, maybe the AA could say that Maradona is its favorite player, and then it could explain why in an emotional way (e.g. by changing the tone of the voice).
The AA should also be able to keep track of (almost) everything you and it said, and it should be able to contextualize very well, as humans do. Some personal assistants already take context into account, but they don't do this very well or just do it to a little extent.
The AA should also be able to reason given the current situation. For example, if you explain something to the AA, you expect it to infer or predict something based on the information it has acquired and the common-sense knowledge.
Moreover, when you speak or write to the AA, you don't expect it to regularly hear badly or not understand what you say or ask (and ask you to repeat), but you expect it to understand well what you say almost always, provided you don't talk or write trash. You also don't expect big delays and you expect the AA to at least say something while it searches for a more appropriate answer, although not all humans behave in this way, but I think that interjections or words such as "hm", "well", "let me think...", etc., will be very important to make the AA look or sound like a human. In general, the AA should be as interactive as a human.
These are some traits that the AA absolutely needs to have in order to pass the Turing test (and be considered intelligent according to that test), but there are probably many others.


What is $ \nabla_{\theta_{k-1}} \theta_{k}$ in the context of MAML?

$\nabla_{\theta_{k-1}} \theta_k$ is gradient of $\theta_k$ with respect to $\theta_{k-1}$, it follows chain rule as noted in the side comment in the image. $\nabla_{\theta} \mathcal L(\theta_k)$ is also not a Hessian but a gradient vector.


How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?

I think DuttaA is right in his comment. VC Dimension of Neural Networks establishes VC bounds depending on the number of weights, whereas the UAT refers to a class of neural networks in which the number of weights a particular network can have is not bounded, although it needs to be finite.
I think that we can show, from theorem 2 and the observations below theorem 3 in Approximation by Superpositions of a Sigmoidal Function, that the VC dimension of
$$S=\left\{\sum_{i=1}^N \alpha_i\sigma(y_i^T x + \theta_i) : N\in\mathbb N, \alpha_i, \theta_i \in\mathbb R, y_i\in\mathbb{R}^n \right\}$$
is infinite.
Let $\{(x_i, y_i)\}_{i=1}^k$ be a sample of arbitrary size $k\in\mathbb N$, and let us see that there is a function in $S$ which can correctly classify it, i.e., $S$ shatters $\{x_i\}_{i=1}^k$.
We note $B(x, \varepsilon) := \{ y\in\mathbb{R}^n : d(x,y) < \varepsilon \}$ (this is just standard notation to denote a ball).
First, let $\varepsilon > 0$ be such that $B(x_i, \varepsilon)\cap B(x_j, \varepsilon) = \emptyset$ every time that $i \ne j$.
Now define $D = \cup_{y_i=1} B(x_i, \varepsilon)$. Define $f_{\varepsilon}(x)$ as in the observations below theorem 3 of Cybenko's paper, and use theorem 2 to find a function $G(x)$ in $S$ that classifies correctly all points at least $\varepsilon$ away from the boundary of $D$, i.e., all points in the sample.


How are Artificial Neural Networks and the Biological Neural Networks similar and different?

We all know that artificial neural networks (ANNs) are inspired by but most of them are only loosely based on the biological neural networks (BNNs).
We can analyze the differences and similarities between ANNs and BNNs in terms of the following components.
Neurons
The following diagram illustrates a biological neuron (screenshot of an image from this book).

The following one illustrates a typical artificial neuron of an ANN (screenshot of figure 1.14 of this book).

Initialization
In the case of an ANN, the initial state and weights are assigned randomly. While for BNNs, the strengths of connections between neurons and the structure of connections don't start as random. The initial state is genetically derived and is the byproduct of evolution.
Learning
In BNN, learning comes from the interconnections between myriad neurons in the brain. These interconnections change configuration when the brain experiences new stimuli. The changes result in new connections, strengthening of existing connections, and removal of old and unused ones.
ANNs are trained from scratch usually using a fixed topology (remember topology changes in case of BNNs), although the topology of ANN can also change (for example, take a look at NEAT or some continual learning techniques), which depends on the problem being solved. The weights of an ANN are randomly initialized and adjusted via an optimization algorithm.
Number of neurons
Another difference (although this difference is always smaller) is in the number of neurons in the network. A typical ANN consists of hundreds, thousands, millions, and, in some exceptional (e.g. GPT-3), billions of neurons. The BNN of the human brain consists of billions. This number varies from animal to animal.
Further reading
You can find more information here or here.


Is intersection of labels acceptable in computer vision?

In my opinion, the second option will be more general. You can refer to some famous datasets for object detection task such as COCO or Pascal VOC, they usually accept the intersect annotations. As the image below, image from this link where they process the annotation of COCO dataset.
I think the reason is that the model will be easier to separate the intersect patterns in the bounding box than interpolate the missing patterns of the object to understand it


When using neural networks to detect features in an image, how can locate that specific feature in the original image?

This problem is called object detection.

If you have a trainings set of images with boxed objects you can just train a neural network to directly predict the box. I.e. the output has the same dimension as the input and the NN learns to assign each pixel the probability of belonging to a certain object.

If you don't have such a convenient dataset you could just recursively narrow the location down by feeding parts of the image to the network until you find the smallest part that still fully activates a certain classification.

In this paper they try a mixture of these two approaches.


How do I derive the gradient with respect to the parameters of the softmax policy?

Softmax policy $\pi_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space.
Taking log, this becomes
$$
\log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\
= \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})
$$

Taking derivative wrt $\theta$, this becomes
$$
\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \nabla_\theta log({\Sigma e^{\phi(s,a) ^T \theta }})
$$

We can rewrite $\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$ as follows.
$$
\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})
= \frac{\nabla_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
= \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
= \Sigma \phi(s,a) \pi_\theta(s,a)
$$

The final equation then becomes
$$
\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi_\theta(s,a)
$$


How can I design an AI that knows when its being spoken to?

Phrase detection instead of text-to-speech

It's worth noting that detection of particular phrases or commands is considered a distinct problem, different from text to speech / text transcription.

While you can simply convert everything it hears to text and then look up keywords there, a specialized detector that directly tries to match incoming audio to a small subset of commands can be done with better accuracy and less processing power required. For this reason, this would generally be the preferred approach in commercial products.

However, for beginner experiments with home automation, you should probably start with choosing an existing speech analysis API where all the audio and natural language parts are appropriately implemented by someone else. Building a good speech command analysis system from scratch is a major undertaking by itself, and you will have your hands full with developing an "artificial intelligent agent"; as a rule, you don't want a project where you have to tackle two major open-ended problems, pick one of them and then you'll have a chance to achieve something interesting there.


Is this neural network architecture appropriate for CIFAR-10?

To be honest, your model is not very clear. But basically after the convolution, you need to add non-linear layers. Otherwise, there is no point of Neural Networks.

You can add a Relu layer for sure.


Can anyone find the source code for the chatbot Luna?

You can't find the source code because it doesn't exist. The whole thing was a 2015 kickstarter scam, and now it's a patreon scam. All the tech "demos" are obviously pre-scripted videos.

There was actually a question on Quora about it, in which one of the answerers managed to find, among other things, the question and answer that his fake AI gave in one of the videos.


What would happen if someone jumped in the front of a Google car?

Given this YouTube video which is being given by Sebastian Thrun who had a TED talk which had nowhere near the same level of detail but had similar conclusions, it looks like the lidar system used by google's automated car system has decent resolution out to at least 30m picking out mobile bodies in the static background and then identifying it. So it should have plenty of time to brake and stop long before there was any risk to the pedestrian attempting to cross the street.

Skip to about 6:40 in the video to see a visual representation of the detection system.


What is the difference between multi-agent and multi-modal systems?

An agent is a concept, which can have slightly different meanings, abilities or instantiations depending on the context. However, given the purpose of this website, I will use and refer to the definition of agent commonly used in artificial intelligence.

  An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

For more details regarding the definition of an agent in AI, see my answer to the question What is an agent in Artificial Intelligence?.

A multi-agent system is a system composed of multiple agents that interact with an environment. See Multi-Agent Systems: A survey (2018) fore a more exhaustive overview of the field.

Multimodal interaction (MI) refers to the interaction with a system (e.g. a computer) using multiple modalities (e.g. speech or gestures). For example, we usually can interact with a laptop using a keyboard and a touchpad (or mouse), so the keyboard and the touchpad are the two different modalities that are used to interact with the computer. MI could thus be considered a sub-field of human-computer interaction.

Conceptually, an agent could be associated with each modality provided by a multimodal system, so a system that provides multimodal interaction could indeed be a multi-agent system. See, for example, A Multi-Agent based Multimodal System Adaptive to the User’s Interaction Context (2011).


Is there a database somewhere of common lists?

So here’s a couple quick resources that i could think of.  First of all, you could look at this,

https://en.m.wikipedia.org/wiki/List_of_lists_of_lists

It has classics such accidents,  hospitals in Asia, or even a list of famous resignations. It’s easentially a list of random lists of things.  It may not owner cover your requirement for sequences but it’ll help with small subsets of lists.

As for sequences, you could always check out,

https://oeis.org

It’s pretty much a list of official mathematical sequences.  It’s got everything from the Fibonacci sequence to esoteric sequences you’ve never heard of.


What is the difference between search and planning?

What is planning?
Planning (aka automated planning or AI planning) is the process of searching for a plan, which is a sequence of actions that bring the agent/world from an initial state to one or more goal states, or a policy, which a function from states to actions (or probability distributions over actions). Planning is not just the use of a search algorithm (e.g. a state-space search algorithm, such as A*) to find the plan, but planning also involves the modelling of a problem by describing it in some way, e.g. with an action language, such as PDDL or ADL, or even with propositional logic.
There are different planning problems, such as

classical planning
planning in Markov Decision Processes (which are mathematical models that describe stochastic environments/problems, i.e. environments where an action $a$ may not always lead to the same outcome)
temporal planning (i.e. when multiple actions can be taken at the same time and/or their duration may be different/variable)
hierarchical vs non-hierarchical planning

There are different planning approaches, such as

planning by state-space search (with or without heuristics), i.e. formulate the planning problem so that you can apply a state-space search algorithm  (examples of state-space search algorithms are A*, IDA*, WA*, DFS or BFS); this is probably the approach to planning that makes people confuse planning with search (which typically implicitly refers to state-space search)

planning by SAT, i.e. formulate the planning problem so that you can you a SAT solver to solve it (e.g. SATPLAN)

planning by solving constraint satisfaction problems (similar to SAT)

planning by "symbolic" search methods (such as Binary Decision Diagrams)

There are many different planning algorithms or planners (of course, each of these corresponds to some approach or is used to solve a particular planning problem), such as

policy iteration in the context of Markov decision processes
GraphPlan
STRIPS (which stands for Stanford Research Institute Problem Solver, but note that STRIPS also and maybe more commonly refers to the corresponding description language used to describe the planning problem)

What is search?
In the context of AI, when people use the word search, they can either implicitly refer to state-space search (with algorithms like A*) or, more vaguely/generally, to any type of search, e.g. gradient descent can also be viewed as a search algorithm, typically, in a continuous state space (rather than discrete state space), e.g. the space of real-valued vectors of parameters for a certain model (e.g. a neural network).
What is the difference between planning and search?
The difference should more or less be clear from my first paragraph above that describes planning: search is used for planning, but the planning process/study also includes other things, such as describing the problem with an action language, such as PDDL. You can also view planning as an application of search algorithms of different types. In any case, planning is about searching for plans or policies. You should probably check this course for more info. This definition of (automated) planning is consistent with our usual notion of planning.


What parameters can be tweaked to avoid a generator or discriminator loss collapsing to zero when training a DC-GAN?

GANs are notably hard to train and it is not uncommon to have large bumps in the losses. The learning rate is a good start but the instability may come from a wide variety of reasons. I'm assuming that you have no bug in your code or data.

For one, gradient descent is not well suited to the 2-player game we're playing. I've personally found ExtraAdam to yield much more stable training (code, paper).

It could also come from the loss and many tricks have been developed and one of the most popular one is enforcing smoothness in the gradient (see W-GAN, W-GAN-GP etc.). SpectralNorm (code paper) is a very popular and recent normalization technique for the discriminator.

There are a number of additional tricks to make GANs work like label smoothing and flipping, different update rates for the discriminator and generators (as in BigGAN for instance). I suggest you have a look at this nice repo of (somewhat seasoned) tricks: ganhacks.


CNN for detecting not just the nature of the object, but position within image as well

You could use another type of CNN that instead of classification is performing regression so it will also give you as output the position(it's not really like that but this is the core idea) .
Some algorithms are SSD or YOLO.


When is max pooling exactly applied in convolutional neural networks?

The pooling operation is applied to the output of the convolution layer. More precisely, it is applied separately for each of the input channels (or slices). So, if the pooling layer receives an input volume of $H_i \times W_i \times D$, then it will produce an output volume $H_o \times W_o \times D$, so the depth of the output volume is equal to the depth of the input volume. There is no sum involved in the pooling operation. For example, in the case of max pooling, you will choose the maximum number of a certain 2D window of values. You do this for each of the input channels.

In the article (which is part of some course notes) Convolutional Neural Networks: Architectures, Convolution / Pooling Layers, Andrej Karpathy says

  Pooling layer downsamples the volume spatially, independently in each depth slice of the input volume.

The following diagram (a screenshot of one of the figures from the mentioned article) should provide some intuition behind the pooling operation.


Decision tree: more than 2 classes, how to represent elements that are in a class vs ones that aren't?

I don't think that is possible with a decision tree, unless there is some measure of confidence that you can use as a threshold.

I ran into the same problem with the ID3 algorithm. It assigns classes, but you only have the resulting class without any confidence or probability attached.

One possible solution could be to add a number of counter examples as a second (dummy, catch-all) class; if the elements of C = 0 are reasonably tightly clustered and your counter examples for C != 0 cover the remaining problem space, then that might work.


What is the difference between the $\epsilon$-greedy and softmax policies?

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $\epsilon $. The problem with $\epsilon$-greedy is that, when it chooses the random actions (i.e. with probability $\epsilon$), it chooses them uniformly (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others. Of course, this approach is not ideal in the case certain actions are extremely worse than others. Therefore, a natural solution to this problem is to select the random actions with probabilities proportional to their current values. These policies are called softmax policies.

Q-learning is an off-policy algorithm, which means that, while learning a so-called target policy, it uses a so-called behaviour policy to select actions. The behaviour policy can either be an $\epsilon$-greedy, a softmax policy or any other policy that can sufficiently explore the environment while learning.

The figure below shows the pseudocode of the Q-learning algorithm. In this case, the $\epsilon$-greedy policy is actually derived from the current estimate of the $Q$ function. The target policy, in this context, is represented by the $\operatorname{max}$ operator, which is used to select the highest $Q$ value of the future state $s'$, which is the state the RL agent ends up in after having taken the action $a$ selected by the $\epsilon$-greedy behaviour policy, with respect to another action $a'$ from state $s'$. This may sound complicated, but if you read the pseudocode several times, you will understand that there are two different actions (and states). The target policy (i.e. the policy that the RL agent wants to learn) is represented by the $\operatorname{max}$ operator in the sense that the so-called target of the Q-learning update step, i.e. $r + \gamma \operatorname{max}_{a'} Q(s', a')$, assumes that the greedy action is taken from the next state $s'$. For this reason, Q-learning is said to learn the greedy policy (as a target policy), while using an exploratory policy, usually, the $\epsilon$-greedy, but it can also be the softmax. Note that, in both cases, the policies are derived from the current estimate of the Q function.

On the other hand, SARSA is often considered an on-policy algorithm, given that there aren't necessarily two distinct policies, i.e. the target policy is not necessarily different than the behaviour policy, like in Q-learning (where the target policy is the greedy policy and the behaviour policy is e.g. the softmax policy derived from the current estimate of the Q function). This can more easily be seen from the pseudocode.

.

In this case, no $\operatorname{max}$ operator is used and the $\epsilon$-greedy policy is mentioned twice: in the first case, it is used to choose the action $a$ and indirectly $s'$, and, in the second case, to select the action $a'$ from $s'$. In Q-learning, $a'$ is the action that corresponds to the highest Q value from $s'$ (i.e. the greedy action). Clearly, you are free to choose a different policy than the $\epsilon$-greedy (in both cases), but this will possibly have a different effect.

To conclude, to understand the difference between Q-learning and SARSA and the places where the $\epsilon$-greedy or softmax policies can be used, it is better to look at the pseudocode.


How do I generate a feature representation of a saliency map (or mask)?

I found a method to do it in the paper Cross-Modality Personalization for Retrieval (2020, accessed: 20-Feb-2020).

  Representation. For images, we extract Inception-v4 CNN features [36]. We then mask the image convolution feature with the BubbleView saliency map, by resizing the saliency map to the convolution feature size and multiplying them together. Finally, average pooling is performed to obtain a 1536-dimensional feature vector. We represent textual descriptions as their average 200-dimensional Glove embedding [31]. For personality, we use a 10-dimensional feature vector containing the scores for the personality questions in [32]. Below, we describe how we learn projections of these representations that place them in the same feature space.


How do updates in SARSA and Q-learning differ in code?

Picking actions and making updates should be treated as separate things. For Q-learning you also need to explore by using some exploration strategy (e.g. $\epsilon$-greedy).

Steps for Q-learning:
1) initialize state $S$
For every step of the episode:
2) choose action $A$ by some exploratory policy (e.g. $\epsilon$-greedy) from state $S$
3) take action $A$ and observe $R$ and $S'$
4) do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma*\max_aQ(S', a) - Q(S, A))$
5) update the state $S = S'$ and keep looping from step 2 until the end of episode

Steps for Sarsa:
1) initialize state $S$
2) initialize first action $A$ from state $S$ by some exploratory policy (e.g. $\epsilon$-greedy)
For every step of the episode:
3) take action $A$ and observe $R$ and $S'$
4) choose action $A'$ from state $S'$ by some exploratory policy (e.g. $\epsilon$-greedy)
5) do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma * Q(S', A') - Q(S, A))$
6) update state and action $S = S'$, $A = A'$ and keep looping from step 3 until end of the episode


What are the main differences between a perceptron and a naive Bayes classifier?

Naive Bayes is a generative algorithm while Perceptron is a discriminative algorithm. That is the main difference.


What effect does a negative output of a neuron have on neighbouring neurons?

In the case of artificial neural networks, your question can be (partially) answered by looking at the definition of the operation that an artificial neuron performs. An artificial neuron is usually defined as a linear combination of its inputs, followed by the application of a non-linear activation function (e.g. the hyperbolic tangent or ReLU). More formally, a neuron $i$ in layer $l$ performs the following operation

\begin{align}
o_i^l = \sigma \left(\sum_{j=1}^N w_j o_j^{l-1} \right) \tag{1}\label{1},
\end{align}

where $o_j^{l-1}$ is the output from neuron $j$ in layer $l-1$ (the previous layer), $w_j$ the corresponding weight, $\sigma$ an activation function and $N$ the number of neurons from layer $l-1$ connected to neuron $i$ in layer $l$.

Let's assume that $\sigma$ is the ReLU, which is defined as follows

$$
\sigma(x)=\max(0, x)
$$

which means that all negative numbers become $0$ and all non-negative numbers become themselves.

In equation \ref{1}, if $w_j$ and $o_j^{l-1}$ have the same sign, then the product $w_j o_j^{l-1}$ is non-negative (positive or zero), else it is negative (or zero). Therefore, the sign of the output of neuron $j$ in layer $l-1$ alone does not fully determine the effect on $o_i^l$, but the sign of the $w_j$ is also required.

Let's suppose that the product $w_j o_j^{l-1}$ is negative, then, of course, this will negatively contribute to the sum in equation \ref{1}. In any case, even if the sum $\sum_{j=1}^N w_j o_j^{l-1}$ is negative, if $\sigma$ is the ReLU, no matter the magnitude of the negative number, $o_i^l$ will always be zero. However, if the activation function is hyperbolic tangent, the magnitude of a negative $\sum_{j=1}^N w_j o_j^{l-1}$ affects the magnitude of $o_i^l$. More precisely, the more negative the sum is, the closest $o_i^l$ is to $-1$.

To conclude, in general, the effect of the sign of an output of an artificial neuron on neighboring neurons depends on the activation function and the learned weights, which depend on the error the neural network is making (assuming the neural network is trained with gradient descent combined with back-propagation), which in turn depends on the training dataset, the loss function, the architecture of the neural network, etc.

Biological neurons and synapses are more complex than artificial ones. Nevertheless, biological synapses are usually classified as either excitatory or inhibitory, so they can have an excitatory or inhibitory effect on connected neurons.


Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch:

If you have a small training set, use batch gradient descent (m < 200)

In practice:

Batch mode: long iteration times
Mini-batch mode: faster learning
Stochastic mode: lose speed up from vectorization

The typically mini-batch sizes are 64, 128, 256 or 512.

And, in the end, make sure the minibatch fits in the CPU/GPU.

Have also a look at the paper Practical Recommendations for Gradient-Based Training of Deep Architectures (2012) by Yoshua Bengio.


If the transition model is available, why would we use sample-based algorithms?

A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if your state space is large this will also be intractable.


How does an activation function's derivative measure error rate in a neural network?

This derivative is used when calculating the error of your machine learning algorithm during gradient based minimization methods.
Read below for more info.

When performing supervised classification (with X, Y data vectors of inputs and outcome data to train with) you begin with the error function

E(X, Y; θ)= ∑i (ƒ(xi; θ)-yi)2

for total error over all data instances i, where f is your neural network, linear regression,...method of interest and θ is the set of weights. The goal here is to find weights that minimize your error when predicting training data (y) (which ideally generalizes to new data as well). To be explicit, ƒ(xi; θ); outputs value of interest which should be yi. And E measures how far off it is in prediction.

So to train your classifier, you optimize E with something like gradient descent. Thus when ∂E/∂θ = 0 (for a particular θ), that means you hit a local minimum for the error function, or a point where the error in the current state of the predictor is low, meaning it is (hopefully) a good predictor.

Note the ƒ here is not the same as an activation function, as a neural network is defined differently than in linear regression, etc. and must perform a special kind of gradient descent called backpropagation.

So when you take ∂E/∂θ, what does it equal for a neural net? You should note the activation functions derivative is involved which is how it’s used to measure error so to say.