Is back-propagation applied for each data point or for a batch of data points?

Short answers

Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?

You can perform back-propagation using (or after) only one training input (also known as data point, example, sample or observation) or multiple ones (a batch). However, the loss function to train the neural network is slightly different in both cases.

Is back-propagation applied $n$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?

If we use only one example, we usually do not wait until the neural network gives satisfactory results for a single input-label pair $(x_i, y_i)$, but we keep feeding it with several input-label pairs, one after the other (and each time updating the parameters of the neural network using back-propagation), without usually caring whether the neural network already produces a satisfactory output for an input-label pair before feeding it with the next (although you could also do that).
Long answer (to the 1st question)
In case you want to know more about the first question, keep reading!
What is back-propagation?
Back-propagation is the (automatic) process of differentiating the loss function, which we use to train the neural network, $\mathcal{L}$, with respect to all of the parameters (or weights) of the same neural network. If you collect the $N$ parameters of the neural network in a vector
$$\boldsymbol{\theta} =
\begin{bmatrix}
\theta_1\\
\vdots \\
\theta_N
\end{bmatrix},
$$
then the derivative of the loss function $\mathcal{L}$ with respect to $\boldsymbol{\theta}$ is called the gradient, which is a vector that contains the partial derivatives of the loss function with respect to each single scalar parameter of the network, $\theta_i$, for $i=1, \dots, N$, that is, the gradient looks something like this
$$
\nabla \mathcal{L} =
\begin{bmatrix}
\frac{\partial \mathcal{L}}{ \partial \theta_1}\\
\vdots \\
\frac{\partial \mathcal{L}}{ \partial \theta_N}
\end{bmatrix},
$$
where the symbol $\nabla$ denotes the gradient of the function $\mathcal{L}$.
Loss functions
The specific loss function $\mathcal{L}$ that is used to train the neural network depends on the problem that we need to solve. For example, if we need to solve a regression problem (i.e. predict a real number), then the mean squared error (MSE) can be used. If we need to solve a classification problem (i.e. predict a class), the cross-entropy (CE) (aka negative log-likelihood) may be used instead.
Example
Let us assume that we need to solve a regression problem, so we choose the mean squared error (MSE) as the loss function. For simplicity, let's also assume that the neural network, denoted by $f_{\boldsymbol{\theta}}$, contains only one output neuron and contains no biases.
Stochastic gradient descent
Given an input-label pair $(x_i, y_i) \in D$ (where $D$ is a dataset of input-label pairs), the squared error function (not the mean squared error yet!) is defined as
$$\mathcal{L}_i(\boldsymbol{\theta}) = \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i) - y_i)^2,$$
where $f_{\boldsymbol{\theta}}(x_i) = \hat{y}_i$ is the output of the neural network for the the data point $x_i$ (which depends on the specific values of $\boldsymbol{\theta}$) and $y_i$ is the corresponding ground-truth label.
As the name suggests, $\mathcal{L}_i$ (where the subscript $_i$ is only used to refer to the specific input-label pair $(x_i, y_i)$) measures the squared error (i.e. some notion of distance) between the current prediction (or output) of the neural network, $\hat{y}_i$, and the expected output for the given input $x_i$, i.e. $y_i$.
We can differentiate $\mathcal{L}_i(\boldsymbol{\theta})$ with respect to the parameters of the neural network, $\boldsymbol{\theta}$. However, given that the details of back-propagation can easily become tedious, I will not describe them here. You can find more details here.
So, let me assume that we have a computer program that is able to compute $\nabla \mathcal{L}_i$. At that point, we can perform one step of the gradient descent algorithm
$$
\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_i, \label{1}\tag{1}
$$
where $\gamma$ is the learning rate and $\leftarrow$ is the assignment operator. Note that $\boldsymbol{\theta}$ and $\nabla \mathcal{L}_i$ have the same dimensions, $N$.
So, I have just shown you that you can update the parameters of the neural network using only one input-label pair, $(x_i, y_i)$. This way of performing GD with only one input-label pair is known as stochastic gradient descent (SGD).
Batch (or mini-batch) gradient descent
In practice, for several reasons (including learning instability and inefficiency), it is rarely the case that you update the parameters using only one input-label pair $(x_i, y_i)$. Instead, you use multiple input-label pairs, which are collected in a so-called batch
$$B = \{(x_1, y_i), \dots, (x_M, y_M) \},$$
where $M$ is the size of batch $B$, which is also known as mini-batch when $M$ is smaller than the total number of input-label pairs in the training dataset, i.e. when $|B| = M < |D|$. If you use mini-batches, typical values for $M$ are $32$, $64$, $128$, and $256$ (yes, powers of 2: can you guess why?). See this question for other details.
In this case, the loss function $\mathcal{L}_M(\boldsymbol{\theta})$ is defined as as the mean (or average) of the squared errors for single input-label pairs, $\mathcal{L}_i(\boldsymbol{\theta})$, i.e.
\begin{align}
\mathcal{L}_M(\boldsymbol{\theta})
&= \frac{1}{M} \sum_{i=1}^M \mathcal{L}_i(\boldsymbol{\theta}) \\
&= \frac{1}{M} \sum_{i=1}^M \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i)-y_i)^2 \\
&= \frac{1}{M} \frac{1}{2} \sum_{i=1}^M (f_{\boldsymbol{\theta}}(x_i)-y_i)^2.
\end{align}
The normalisation factor, $\frac{1}{M}$, can be thought of as averaging out the losses of all input-label pairs. Note also that we can take the $\frac{1}{2}$ out of the summation because it is a constant with respect to the variable of the summation, $i$.
In this case, let me also assume that we are able to compute (using back-propagation) the gradient of $\mathcal{L}_M$, so that we can perform a gradient descent (GD) update (using a batch of examples)
$$
\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_M \label{2} \tag{2}
$$
The only thing that changes with respect to the GD update in equation \ref{1} is the loss function, which is now $\mathcal{L}_M$ rather than $\mathcal{L}_i$.
This is known as mini-batch (or batch) gradient descent. In the case $M = |D|$, this is simply known as gradient descent, which is a term that can also be used to refer to any of its variants (including SGD or mini-batch GD).
Further reading
You may also be interested in this answer, which provides more details about mini-batch gradient descent.


Why does Monte Carlo work when a real opponent's behavior may not be random

First, we need to distinguish plain Monte-Carlo from Monte-Carlo Tree Search. They're different things.

Monte-Carlo search, in the context of game AI search algorithms, is typically understood to mean that we search randomly many times, and average the results, and nothing else. If this is all we're doing, then yes, your understanding is correct. This is also sometimes referred to as "plain Monte-Carlo (search)" or "pure Monte-Carlo (search)", to make it explicitly clear that we're not doing any tree search as in Monte-Carlo Tree Search (sometimes when we just say "Monte-Carlo" in the context of game AI, people will automatically assume Monte-Carlo Tree Search, due to how popular it is).

Monte-Carlo Tree Search does a lot more than just that though. It gradually builds up a search tree (through the Expansion step), and within that search tree (which is growing over time), it uses a much more sophsticated strategy for traversal than pure random (the Selection step).

  For example suppose i have a node with 100 children, 99 of which leading to an instant WIN, and the last one leading to an instant LOSS.

Suppose that this node you're talking about, the one with those 100 children, is relatively close to the root node. Then, it is likely that it and all 100 of its children will end up "growing into the search tree" that is slowly built up by the Expansion step. Once they have been added to the search tree, the Selection step will make sure that the vast majority of further iterations visiting this part of the tree will select the instant loss (assuming the opponent is to move in this node). In the limit (after an infinite amount of search time), this bias towards selecting the loss node will be so large that the average evaluation tends to the (correct) minimax evaluation.

Another way to view the idea of evaluation through many random sequences of play is the following; the idea is that if we're in a very strong position, a good game state, then we're more likely to win than to lose if both players start playing at random. Consider, for example, a game of chess where Player 1 has many pieces left, and Player 2 only has a few pieces left. Imagine both players were to play completely at random from that game state onwards. On average, which player would you expect to win more often? Probably Player 1.

When we're considering game states that are still far away from terminal states, this basic idea tends to work relatively well. Obviously not always correct, it's still a heuristic, but it can kind of work. When we're already very close to a terminal state that can be reached through one highly specific sequence of play, yeah, we might miss that through random actions; this is where we need the more informed policy of the Selection step.


How are generative adversarial networks trained?

Compare generated and real data

All the results produced by G are always considered "wrong" by definition, even for a very good generator.

You provide the discriminative neural network $D$ with a mix of results generated by the generator network $G$ and real results from an outside source, and then you train it to distinguish if the result was produced by the generator or not - you're not comparing "good" and "bad" results, you're comparing real versus generated results.

This will result in a "mutual evolution" as $D$ will learn to find features that separate real results from generated ones, and $G$ will learn how to generate results that are hard to distinguish from real data.


Does this hyperparameter optimisation approach yield the optimal hyperparameters?

After you've computed $h^{1}_{optimal}$ the only thing you can be sure is that this is the best (assuming constrained case) value of $h^1$ (with respect to some model performance metric) given your initial values for $h^2, ..., h^n$. If you change a bit any of $h^2, ..., h^n$ you're no longer certain that the value $h^1$ you found is the optimal one. So yes, the key here is the assumption about the independence.


What does Brooks mean by Representation?

First, Pascal is not a logic programming language. Logic programming refers to languages like Prolog where you have a declarative style of programming compared to a imperative style like you have in Pascal. Maybe you mean if-statements which are typical for imperative languages.

Second, representation means a certain level of abstraction. For instance, a model represents a certain part of the reality. Imagine a cup on a table. If the agent has a representation of this situation, it has a symbol table and a symbol cup which represents the things in the real world. Now it can have a relation on(cup, table) which represents the situation that the cup is on the table. This type of abstraction can be easily represented in a logic language like first order logic. Therefore, one uses logic programming languages like Prolog or other types of languages like OWL to represent knowledge and perform reasoning. So the important term to which Brooks refers is Knowledge Representation and Reasoning.

Third, if your agent only have sensor data like video or sonar data, then it knows only distances or pixels from the real world. That is not meant with representation. Brooks' Creatures have only this information and calculate with this data directly to perform an action without reasoning. In that sense also artificial neural networks have no representation.

Finally, for an open source project to understand representation I would recommend the above mentioned OWL. You can look at the Protégé editor for working with OWL. In an OWL ontology you can represent relations between things and reason about them.


Can I use an autoencoder with high latent representational space?

i feel , i can answer this question, based the web source that i found and i read. that can use an autoencoder with hight latent representational space. example use LSTM autoencoder, LTSM autoencoder used for sequence or time series data. on the source, they use CNN autoencoder to denoise some syntetic noised data they have generated. but they have asked what is the meaning of this latent representation space? for what they have done before. on that answer. they said"your input data is noisy sinewave data. your are not supposed to use Convolutional autoencoder for sequence data".

https://stackoverflow.com/questions/59438488/cnn-autoencoder-latent-space-representation-meaning

addition other source for answer this question
https://towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d
https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f

hopefully help answer :))


Which machine learning algorithm is used in self-driving cars?

It will not be single DNN architecture, rather it will be a collection of different DNN architectures that are used together to make the final decision. Convolutions are using the images/videos from the camera. Other architectures use other sensory sources. These DNNs will be trained to compute the high-level features from their sensory sources and then those high-level features will probably be fed into an LSTM (or some other form of RNN) that is trained with some form of Reinforcement learning algorithm to compute the action (like slowing down, applying breaks etc).


How large should the corpus be to optimally retrain the GPT-2 model?

After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my application the sweet spot is around 30-35k tokens.


What are the differences between 1-step SARSA and SARSA?

N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

Off-policy n-step SARSA: Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$

where

$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$

You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.

Off-policy one-step SARSA: You can think of Q learning as one-step off policy SARSA.


Is there a programming training dataset for robot program itself

On self improving AI

The way it's been done in the past is one AI turns a picture into text an the other turns text into a picture.

The AI generated material is added to known good material. The AI's job then is to guess what is real an what is fake (generated by AI).

At the point when the AI cannot tell in a false positive way both AI are said to be at their limit. If the limit is good enough you're done otherwise go back and code (more nodes or better function or more real data)

The thing to remember is you still need real data. The act of putting the AI together is good only to create more data then you could ever provide.

If that data is bad then you are going to rely on the cost function more then the data. But this allows you to label data real and AI so if the function rewards identifying the AI the AI's find each other's flaws and the better at avoiding flaws the better the AI get. (You still need real data! just less)

For example if two people cannot play ping pong if they play each other they learn slowly and only the cost function enforces their learning.
But have them play a pro (lots of good data) and they get better much faster.


What are the ethical implications of creating (possibly sentient) AI systems?

I believe if an AI achieves sentience, it should be treated the same way we are required to treat any other sentient animal. This is belief though, there is no established ethics for AI. But there were no ethics for animals a couple of centuries ago.


Can I do oversampling by copying the same image multiple times? Will it effect my neural network accuracy?

This would be a pretty fun opportunity to experiment a bit. I know there's been breakthroughs in OCR technology by training the network on images that have been altered.

Long story short, there was a pre-process that took the source images and did minor manipulations to them. This involved skewing, rotating, changing resolution & adding artifacts into the image to introduce "Noise". This allowed the network to learn to recognize characters more accurately. It may be worthwhile to create an image pre-processing pipeline that will do a variation of modifications to the source images. When training you may want to lower your learning rate & split up your data into 70% train & 30% test. Then you can loop through all the images repeatedly until you hit a maximum for the learning rate.

Some examples
https://matthewearl.github.io/2016/05/06/cnn-anpr/
https://medium.com/@shreyas.s/image-data-generation-for-optical-character-recognition-ocr-9b19300649c8


How do AIs like Siri and Alexa respond to their names being called?

Is it because their listening function reloads in milliseconds or even nanoseconds

Yes, it expects the keyword to start every moment of time and it ignores the rest.
Overall, the algorithm is described here, you can read for details:
https://machinelearning.apple.com/research/hey-siri


Is there any way to draw a neural network's connections in a nice way?

I'm the developer of Neataptic, a Javascript neural network library. I don't know if it is exactly what you're looking for, but it has a built-in graph creator using D3 and webcola.

Basically, each connection gets an arrow. It also supports self-connections and gates.

There are some examples of images:

Play around yourself here.


Prerequisites for Andrew Ng's Machine Learning Course

This course is focused on machine learning using MATLAB, which is not practical nowadays as it is a programming language used specifically for computing, and cannot display GUI or communicate through the network. The language is powerful but limited in some ways. Nowadays most people use python for machine learning, as it is versatile and can connect to other backend like C++, java, JavaScript easily. The language is also a general language, and unlike MATLAB can do many things not limited to computing.

If you really want to join this course, I would recommend first learning MATLAB language and also learn basic calculus like derivatives. This would greatly help on your learning of the course.

However if you want to here serious about machine learning, I would encourage you to enroll in the Deep Learning specialization also by Andrew Ng on Coursera.
https://www.coursera.org/specializations/deep-learning
This course uses python as the programming language and teaches more modern approaches to deep learning like recurrent neural networks, convolutional neural networks and more. It also talks more about application of neural network. There is also theory, but it also talks about application of a specific algorithm and how it works.

Hope I can help you.


Rollout algorithm like Monte Carlo search suggest model based reinforcement learning?

From what I understand, Monte Carlo Tree Search Algorithm is a solution algorithm for model free reinforcement learning (RL).

Monte Carlo Tree Search is a planning algorithm. It can be considered part of RL, in a similar way to e.g. Dyna-Q.

As a planning algorithm MCTS does need access to a model of the environment. Specifically it requires a sampling model, i.e. a model that can accept a state and action, then return a single next state and reward with the same probabilities as the target system. The alternative model, used by other RL techniques such as Value Iteration, is a distribution model which provides the full distribution of probabilities for rewards and next states.

  if that is the case, then how come the agent knows which state it will observe during the rollout, since rollout is just a simulation, and the agent never actually perform that action ?

It is not the case. The agent knows what it will observe during the simulation because the simulation is a sampling model.

  In this case, doesnt rollout in Monte Carlo Tree Search algorithm suggests that the agent knows the transition model and reward model and thus a solution algorithm for model based reinforcement learning and not model free reinforcement learning ?

Yes. This is how most planning algorithms work.

The simulation can be driven purely by sampling from previous experience, which is how Dyna-Q works*. I would still consider that a model-based approach, and its success depends a lot on how well such a model can be learned. In many cases, variance due to errors in the learned model adversely affects learning. So MCTS works best in environments that can be accurately sampled, because they are strongly rules-driven. For example, board games.

* Functionally DynaQ is almost identical to experience replay. So much so, that whether you consider it a planning algorithm or experience replay added to basic Q learning is more a matter of how you present the design of the learning agent - e.g. perhaps a designer wants to focus on the model-learning side more, so has code that explicitly represents the learned model.


Creating videos of AI generated photographs

I wouldn't really consider what they are doing as AI - they are using a script that intelligently overlaps various images of existing people in order to create a new face.

Animating those images isn't impossible - essentially extrapolation + additional "real" images will be used to know what the face would like from all angles and in all states (happy, sad, intrigued, etc...).


Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks'

we start with

$\frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)]$

using definition of expectation for continuous case:

$\mathbb E[X] = \int xp(x) dx$

for the first equation we get:

$\frac{\partial}{\partial \theta} \int f(\mathbf w, \theta)q(\mathbf w \mid \theta) d\mathbf w$

we swap $q(\mathbf w \mid \theta) d\mathbf w$ for $q(\epsilon)d\epsilon$ and we get:

$\frac{\partial}{\partial \theta} \int f(\mathbf w, \theta)q(\epsilon)d\epsilon$

using definition of expectation again we have:

$\frac{\partial}{\partial \theta} \mathbb E_{q(\epsilon)}[f(\mathbf w,\theta)]$

i believe dominant convergence theorem let's us interchange expectation and derivative and we have:

$\mathbb E_{q(\epsilon)}[\frac{\partial}{\partial \theta}f(\mathbf w,\theta)]$

we have a function $f$ that depends on two variables $\mathbf w$ and $\theta$. In order to get a full derivative of such function we need to take derivative of $f$ with respect to both variables. We have:

$\mathbb E_{q(\epsilon)}[\frac{\partial f(\mathbf w,\theta)}{\partial \theta} + \frac{\partial f(\mathbf w,\theta)}{\partial \mathbf w} \frac{\partial \mathbf w}{\partial \theta}]$

First term inside expectation is derivative of function $f$ with respect to $\theta$ directly, and in the second term $f$ is derivated with respect to $\mathbf w$, but we need to apply chain rule to get full derivative with respect to $\theta$


Why is the state-action value function used more than the state value function?

We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return from specific state onward but they don't tell you which action to take. In order to derive an optimal action in a specific state you would have to simulate all possible actions one step ahead and then pick the action that leads you to the state with highest state value. That is often inconvenient or impossible. State action values connect the expected return with actions, not states, so you don't need to simulate all actions one step ahead and see where you end up, you only need to pick an action that has the highest value because you know that is the best action to take in that state.


How to define a loss function for a classifier where the confusion between some classes is more important than the confusion between others?

Classic Question

Not only can the reliability or accuracy needs be asymmetrically distributed between category boundaries, but the asymmetry might not be describable in terms of a first degree polynomial translation of the error surface.

The question data set are images belonging to class $c \in \Big( A: \{A_1, \, ..., \, A_n\} \land  B: \{B_1, \, ..., \, B_m\} \Big)$.  The convolutional network must be trained with a sample of labeled images to categorize with a reliability of $1 - \delta$ and accuracy of $1 - \epsilon$, based on the PAC (probably approximately correct) framework.

How can the loss function reflect the conceptualization of optimal training result that the reliability of categorization between $A$ and $B$ be of greater value than categorization between $A_i$ and $A_j$ where $i \ne j$, and similarly with B.

Criteria X would state that the value of differentiation $D$ is not sub-category index dependent for differentiation between super-categories.  That is, $\forall \; (i, j) \; \text{where} \; i \ne j$, $D(A_i, B_i) = D(A_i, B_j)$.

Criteria Y, the categorization WITHIN a super-category is not unlike the typical symmetric case $\forall \; (i, j, k) \; \text{where} \; i \ne j \land i \ne k \land j \ne k$, $D(A_i, A_j) = D(A_i, B_k)$.)

(a) Are there any loss functions that take this requirement into account?

Let $1_p$ be the bit value that is 1 if and only if the label matches the super-category and let $1_b$ be the bit value that is 1 if and only if the label matches the super-category and the sub-category.  Because of Criteria X, a conical and normalized loss function could then be defined as thus.

$\epsilon = \dfrac {(1 - 1_p) + \alpha (1 - 1_b)} {1 + \alpha}$

The proportional value of sub-category reliability in relation to super-category reliability $\alpha < 1.0$.

Because of criteria Y, there is no meaning to concavity in this context.  Applying mean squaring to the loss simply changes the curvature and meaning of $\alpha$ in relation to the loss as can be seen here.

$\epsilon = \dfrac {(1 - 1_p)^2 + \alpha^2 (1 - 1_b)^2} {1 + \alpha^2} = \dfrac {(1 - 1_p) + \tau (1 - 1_b)} {1 + \tau}\text{, where }\tau = \alpha^2$

(b) Could a weighted cross entropy work in this case?

Cross entropy between the two levels of categorization would be baseless unless there were correlative effects between super classification feature and sub classification feature inherent in the target concept of the learning.

How this loss would change if the classes were unbalanced?

Although there may be schemes for modifying the loss function to compensate for skewed distribution of categories represented in the sample, this is primarily a data input problem best solved by improving data collection or choosing training algorithms, designs, architectures, treating input, and properly controlling hyper-parameters.  The loss function represents the inverse of what is considered optimal, not how one achieves it.


Problem extracting features from convolutional layer where the dimensions are big for feature maps

I think it's okay to let CNN extract a big number of feature maps, then you can reduce the features from it. To reduce features from CNN result, you can do feature selection or feature extraction. Some people prefer to use feature selection and select the most significant feature maps, for examples:

Action unit selective feature maps in deep networks for facial expression recognition
Convolutional Neural Network Simplification Based on Feature Maps Selection

Hope it helps.


What happens when the output length in the brevity penalty is zero?

Division by zero is not mathematically defined.

A usual or standard way of dealing with this issue is to raise an exception. For example, in Python, the exception ZeroDivisionError is raised at runtime if you happen to divide by zero.

If you execute the following program

zero = 0
numerator = 10
numerator / zero

You will get

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    numerator / zero
ZeroDivisionError: division by zero

However, if you want to avoid this runtime exception, you can check for division by zero and deal with this issue in a way that is appropriate for your program (without needing to terminate it).

In the paper BLEU: a Method for Automatic Evaluation of Machine Translation that introduced the BLEU (and brevity penalty) metric, the authors defined the brevity penalty as

\begin{align}
BP =
\begin{cases}
1, & \text{if } c > r\\
e^{(1- r/c)} & \text{if } c \leq r\\
\end{cases} \label{1} \tag{1}
\end{align}

This definition does not explicitly take into account the division by zero.

The Python package nltk does not raise an exception, but it (apparently, arbitrarily) returns zero when c == 0. Note that the BLEU metric ranges from 0 to 1. For example, if you execute the following program

from nltk.translate.bleu_score import brevity_penalty, closest_ref_length

reference1 = list("hello") # A human reference translation.
references = [reference1] # You could have more than one human reference translation.

# references = [] Without a reference, you will get a ValueError.

candidate = list() # The machine translation.
c = len(candidate)

r =  closest_ref_length(references, c)
print("brevity_penalty =", brevity_penalty(r, c))

You will get

brevity_penalty = 0

In the example above, the only human reference (translation) is reference1 = list("hello") and the only candidate (the machine translation) is an empty list. However, if references = [] (you have no references), then you will get the error ValueError: min() arg is an empty sequence, where references are used to look for the closest reference (the closest human translation) to the  candidate (the machine translation), given that there could be more than one human reference translation, and one needs to be chosen to compute the brevity penalty, with respect to your given candidate.

In fact, in the documentation of the brevity_penalty function, the following comment is written

# If hypothesis is empty, brevity penalty = 0 should result in BLEU = 0.0.

where hypothesis is a synonym for candidate (the machine translation) and the length of the candidate is $c$ in the formula \ref{1} (and c in the example above).

To answer your second question more directly, I don't think there's a standard way of dealing with the issue, but I've not fully read the BLEU paper yet.


What are evolutionary algorithms for topology and weights evolving of ANN (TWEANN) other than NEAT?

The Wikipedia article on neuroevolution contains a list of neuroevolution techniques (e.g. NEAT). I will list below the examples that evolve both the parameters and the topology of the neural network.

Cellular encoding (1994)
GNARL (1994)
EPNet (1997)
Evolutionary acquisition of neural topologies (2007)
Interactively constrained neuro-evolution (2012)
Deus ex neural network (2012)
Spectrum-diverse unified neuroevolution architecture (2017)
Covariance matrix adaptation with hypervolume sorted adaptive grid algorithm (2017)


What is the difference between artificial neural network (ANN) and deep learning?

A few years ago, deep learning was a buzz word, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!).
From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, deep learning typically refers to the subset of machine learning techniques (specifically, gradient descent and back-propagation) applied to neural networks, and, in particular, neural networks with multiple layers. However, there is no consensus on what multiple refers to. In fact, Schmidhuber (co-author of the LSTM), in [1], writes

At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. Instead of committing myself to a precise answer, let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.

In other words, although there is typically no agreement on the definition of deep or multiple, there are several problems (such as the vanishing or exploding gradient problems) that arise as the number of layers in a neural network increases, so there are reasons behind the distinction of deep and non-deep learning.
Note that some research papers (e.g. [1]) distinguish between deep learning applied to neural networks and other types of deep learning, so that not to exclude the layered composition of other models (e.g. SVMs) other than neural networks and the use of machine learning algorithms (e.g. gradient descent) to train them.
To answer your question more directly, neural networks are models or function approximators, while deep learning is the set of machine learning techniques applied to the layered composition of models (typically, neural networks).


Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi gradient methods work well in Reinforcement Learning, but what is the reason of not using the true gradient if it can be computed?

Just complexity and extra computation, in many cases for a marginal benefit.

  I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?

It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:

size of minibatch to train from experience replay on each step
number of training steps between taking frozen copies for estimation*
whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)

There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.

* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.


How should we interpret all the different metrics in reinforcement learning?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it should increase).
Policy loss (I assume you mean the "actor loss"?) is generally harder to interpret. You should think of this more as a source of gradients and not necessarily a good indicator of how well your agent is performing.
I'm not really sure why you'd be monitoring the value during training. Value loss, however, is basically equivalent to value loss in value based methods like Q-learning, for example. So this one should be decreasing overall. Otherwise the baselines you compute to reduce variance in the policy gradient will either be less effective, or even harmful.
Entropy is a nice quantity to measure, because it's a good indicator of how much your agent is exploring. If you see that your agent is not achieving high returns and the entropy is really low, this means that your policy has converged to a suboptimal one. If the entropy is really high, this means the agent is acting fairly randomly (so it's basically exploring a lot). Ideally the entropy should decrease over time, so your policy becomes more deterministic (less exploration) as it reaches an optimum.


Yes. A computational linguist is someone who (among other things) uses computers to process/model/analyse/... natural language. Coding might be one aspect of it, but is about the least important: you can always get a non-linguist programmer to do coding for you.

I studied "Computational Linguistics" at university, and while programming was taught as part of the course, coding was only a minor aspect in the actual subject matter. The senior professor (and head of the department) wasn't able to do any coding himself; he came from the linguistics side of it.

Being able to program is useful, as it speeds things up and makes you more independent, but it is by no means an important part of being a computational linguist.

UPDATE: I have been accused of misrepresenting the field of CL. However, it is a broad, interdisciplinary field, and comprises many elements. Sure, on the academic/research side you might do more programming than in the applied/commercial side, but I maintain that you can easily work as a computational linguist without actually doing any programming. For most tasks, readily available software exists by now, so you don't actually need to program anything new.


Can we optimize an optimization algorithm?

First, you need to consider what are the "parameters" of this "optimization algorithm" that you want to "optimize". Let's take the most simple case, a SGD without momentum. The update rule for this optimizer is:

$$
w_{t+1} \leftarrow w_{t} - a \cdot \nabla_{w_{t}} J(w_t) = w_{t} - a \cdot g_t
$$

where $w_t$ are the weights at iteration $t$, $J$ is the cost function, $g_t = \nabla_{w_{t}} J(w_t)$ are the gradients of the cost function w.r.t $w_t$ and $a$ is the learning rate.

An optimization algorithm accepts as its input the weights and their gradients and returns the update. So we could write the above equation as:

$$
w_{t+1} \leftarrow w_{t} - SGD(w_t, g_t)
$$

The same is true for all optimization algorithms (e.g. Adam, RMSprop, etc.). Now our initial question was what are the parameters of the optimizer, which we want to optimize. In the simple case of the SGD, the sole parameter of the optimizer is the learning rate.

The question that arises at this point is can we optimize the learning rate of the optimizer during training? Or more practically, can we compute this derivative?

$$
\frac{\partial J(w_t)}{\partial a}
$$

This idea was explored in this paper, where they coin this technique "hypergradient descent". I suggest you take a look.


How to add more features to the input of a machine learning algorithm?

Data pre-processing and feature extraction are by far the most important part of any machine learning algorithm. It's even more important that the model you choose to do the classification.

Unfortunately, pre-processing and feature extraction are completely different for each type of data. You need to play around with the data yourself to find out what works best with the nature of your data. With experience you start noticing some patterns with different data types. For example, as you are doing, building a word vector is an effective means of feature extraction with text based data.

"I could just add tweet location at the end of the vector I suppose, but that would give it a very small weighting."

This is entirely untrue for any machine learning algorithm I would choose. Your model should not associate weights to the inputs based on their array location. They should be associated based on the explained variance (information gain) it provides.

After you do your pre-processing and feature extraction you can then further refine your feature set with some common methods which can be found in libraries such as: principle component analysis (PCA) and linear discriminant analysis (LDA).


Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control?

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.

Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.


How can I use deep neural networks to recognize characters on vehicle license plate?

Before jumping into any machine learning task it is good to consider your dataset and the features you wish to use. License plates are unique thus feeding the entire plate into a machine learning algorithm would not yield very accurate results. First, you will want to make sure that the extracted license plate images are similar in size. Then you will want to separate the license plate such that you can separate the different digits and letters into their own respective images. Thus your dataset that will be used to train your network will consists of many single character images which contain either a letter or a digit. When you have finished training your model, future license plates will also have to be broken down in this way, classified and then the outputs be combined to get the license plate number.

What is a neural network?

A neural network (NN) is a series of nodes which contain a simple, continuous, differentiable function such as logistic regression. Alone these nodes are not very powerful, however, the beauty comes in when they are connected in networks. The tuning of such a network can learn highly complex functions. Typically, a NN is set up with three layers. The input layer, where you will feed in your features, the hidden layer and the output layer. Naturally, if you want a deep NN then you just need to add more hidden layers. But, there is a trap. The deeper your network is the harder it will be to train, thus you will need MORE data.

The input-layer

This is where you will be feeding in your images. If you have a 16*16 pixel image then you will have 256 input nodes. This layer typically does not perform any function on the input, it is simply taking in the input. It will then feed the input to the hidden layer. You can use some dimensionality reduction techniques to try and eliminate some useless information from your dataset such as the white spaces in the corner of the image. The less input nodes you have, the more informative your samples become, thus with higher information entropy your network will be able to learn faster, or more rigidly with the same amount of data.

The hidden layers

You are free here to choose how many layers you want and how many nodes you want in each layer. I suggest using grid search and cross-validation as a means to determine which configuration is ideal for your dataset. Too many layers and nodes and you will not end up with a fully trained networks once you run out of training data (too much bias). Too little nodes, your network will not be able to learn the subtleties which are embedded within the data (too much variance). You want to find yourself at the crossroad where these two are minimized. Choose wisely.

The output layer

This layer can be configured in a few different ways. Typically people like to do one output node for each class. In your case there are 26 letters in the alphabet and 10 digits. This gives 36 possible output classes. So you will need 36 nodes in your output layer.

How do I train this?

The training of the NN is by tuning the weights that will be applied to the inputs of each of your nodes. Try to draw your dense network, for each line connecting nodes you will have a weight. I am sure you can see how this becomes quite huge. You will initialize your weights randomly to start. Then you will take each of your examples and pass them through your network. This is called a feed-forward pass through. This will provide you with an output. You will compare your output with your ground truth value using a cost function. This is often the root mean squared error (RMSE) but there are many other options. Then using the derivative you will back-propagate the error through your network to see which weights caused the error and to what extent. The derivative will then allow you to update those weights proportionately.

WOW do I need to program all that?

Nope not anymore!!! You can use Keras in python (my recommendation) or other similar frameworks which have made it a real tea sipping pleasure to program a variety of NN configurations.

My recommendation

I would start with a simple NN and see what kind of accuracy I can achieve. From there I would try to use convolutional neural networks which have been shown to garner much better results on datasets comprised of images. These can also be implemented in Keras. CNN require more data than typical NN thus you might need to get access to more images.

It is always best to train your dataset only on images you have in your original data. However, when it comes to images, adding to your data by artificially shifting images by 1-5 pixels or rotating by 1-5 degrees can add variability which can supplement your original data.

Make sure that the license plate pictures you are using to train your network are taken with a similar system as the images you will be using in the future. You will not get very good results if you train your network on images taken directly behind cars and then you expect to implement your solution in traffic cameras monitoring traffic from above.


What determines the values of weights in a neural network?

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network.

By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with stochastic gradient descent (SGD). Essentially, the algorithm determines how much each individual weight contributed to the network's loss. It then updates that weight in the direction that would reduce the loss.

I recommend going through Michael Nielsen's online book to learn the basics.


How do we know that the neurons of an artificial neural network start by learning small features?

We do it experimentally; you're able to look at what each layer is learning by tweaking various values throughout the network and doing gradient ascent. For more detail, watch this lecture: https://www.youtube.com/watch?v=6wcs6szJWMY&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=12 it provides many methods used for understanding exactly what your model is doing at a certain layer, and what features it has learnt.


Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.


Why does shifting all the rewards have a different impact on the performance of the agent?

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem.
Episodic problems
An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal behaviour can therefore depend quite critically on balance between positive and negative rewards.

If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner.

If an environment contains repeatable positive rewards, and these outweigh total negative rewards, then the agent will be motivated to loop through the postive rewards and not end the episode.

Scaling all rewards by the same positive factor makes no difference to the goals of an agent in an episodic problem. Adding a positive or negative offset to all rewards can make a difference though. It is likely to be most notable when such a change moves rewards from positive to negative or vice versa. In the MountainCar example, adding +2 to all rewards would mean the agent would gain +1 for each time step. As it would stop gaining any reward for reaching the goal, even though reaching that goal would score the highest possible +2.5 reward, the fact that this ends the episode means that it now becomes a poor choice. The best action for the car in this modified MountainCar is to stay at the bottom of the valley collecting the +1 reward per time step forever.
Continuing problems
In a continuing problem, there is no way for the agent to avoid the stream of new reward data. That means any positive scaling of all reward values or positive or negative offset, by the same amount, has no impact on what counts as the optimal policy. The calculated value of any state under the same policy, but with rewards all transformed with the same multiplier and offset, will be different, but the optimal policy in that environment will be the same.
If you scale or offset rewards differently to each other, then that can change the goals of the agent and what the optimal policy is. The balance does not really depend on whether rewards are positive or negative in a continuing environment.
There may be some exceptions to this for continuing problems when using a discount factor, and setting it relatively low (compared to the typical state "cycling" length in the problem). That can cause changes in behaviour due to offsets, similar to those seen in episodic problems. If you use an average reward setting this tends to be less relevant. Often in DQN, you will choose a high discount factor e.g. 0.99 or 0.999, and this will tend to behave close to an average reward setting provided rewards are not very sparse.
In general
In either case, if you change a reward system, and that results in an agent that consistently learns a different policy, that will usually mean one of two things:

The original reward system was incorrect. It described a goal that you did not intend, or had "loopholes" that the agent could exploit to gain more reward in a way that you did not intend.

The implementation of the agent was sensitive in some way to absolute values of total reward. That could be due to a hyperparameter choice in something like a neural network for example, or maybe a bug.

Another possibility, that you may see if you only run a few experiments, is that the agent is not learning 100% consistently, and you are accidentally correlating your changes to reward scheme with the noise/randomness in the results. A DQN-based agent will usually have some variability in how well it solves a problem. After training, DQN is usually only approximately optimal, and by chance some approximations are closer than others.


We know what Lee's strategy was during the game, and it seems like the sort of thing that should work. Here's an article explaining it. Short version: yes, we know what went wrong, but probably not how to fix it yet.

Basically, AlphaGo is good at making lots of small decisions well, and managing risk and uncertainty better than humans can. One of the things that's surprising about it relative to previous bots that play Go is how good it was at tactical fights; in previous games, Lee had built a position that AlphaGo needed to attack, and then AlphaGo successfully attacked it.

So in this game, Lee played the reverse strategy. Instead of trying to win many different influence battles, where AlphaGo had already shown it was stronger than him, he would set up one critical battle (incurring minor losses along the way), and then defeat it there, with ripple events that would settle the match in his favor.

So what's the weakness of AlphaGo that allowed that to work? As I understand it, this is a fundamental limitation of Monte Carlo Tree Search (MCTS). MCTS works by randomly sampling game trees and averaging them; if 70% of games from a particular position go well and 30% of games from another position go well, then you should probably play the first move instead of the second move.

But when there's a specific sequence of plays that go well--if, say, W has a path that requires them playing exactly the right stone each time, but B has no possible response to this path--then MCTS breaks down, because you can only find that narrow path through minimax reasoning, and moving from the slower minimax reasoning to the faster MCTS is one of the big reasons why bots are better now than they were in the past.

It's unclear how to get around this. There may be a way to notice this sort of threat, and then temporarily switch from MCTS reasoning to minimax reasoning, or to keep around particular trajectories in memory for consideration in future plays.


How to replicate legacy systems with machine learning?

Let's assume from the outset that the space of inputs is too large to allow exhaustive tabulation.

The essential issues when applying ML are that the program being modelled is likely in general to:

Be discrete, i.e. operate (at least in part) on integer, boolean or categorical variables.
Contain various conditional/looping constructs (if/while/for etc).
Have side-effects that affect other parts of the program (e.g. non-local variables) or world state (e.g. writing to a file).

These pose obstacles for ML methods such as ANN. The ML approach which is most immediately compatible with these issues is Genetic Programming (GP).
A recent specialisation of GP that is specifically concerned with the transformation of existing software systems is Genetic Improvement (GI).

However neither GP/GI (nor any other current ML technique) is a 'silver bullet' here:

Despite decades of research, GP still works best at synthesizing relatively small functions, certainly not entire legacy programs.
Because it is only possible to train on a very small subset of a program's input space, there is little guarantee that the program will generalize to inputs it hasn't been trained on.
How will the success of side effects be formally determined for training purposes?

Some of these issues could be addressed to some degree if the program has a comprehensive test suite, but replacing an entire nontrivial program is not likely anytime soon. Replacing smaller parts of the program that have good unit tests is more realistic goal.

Here is a case study showing how GI was successfully used to fix errors in the implementation of Apache Hadoop, by operating only on the program binary.


Are there deep networks that can differentiate object class from individual object?

In this case, you have an ontology and want to learn the ontology. There are many researches in this topic that you can find. However, the data could be the most challenging part. Some of the researches:

Ontology learning for the Semantic Web
Ontology Learning

Also, as these are some frameworks to ontology learning, you can use deep networks such as RNN to learn the task.


Why is image recognition a key function of AI?

Image recognition is an important application of AI techniques, as images usually act as sensory input for further problems to be solved.

For example, a self-driving car needs to take into account its environment; it needs to recognise the path/road it is driving on, obstacles, other traffic, traffic signs, etc. All this is visual input which needs to be recognised. Without image recognition a self-driving car would not be possible.

A machine that operates in a warehouse picking goods: in order to handle the goods, eg with a robotic arm, it needs to recognise where the goods are and what they look like, so that the arm can be controlled.

A system controlling a CCTV camera should be able to distinguish an intruder from an animal that happens to just walk past. Again, with image recognition and classification this could be achieved.

An AI system will react to sensory input from the 'outside' world, and act on it. The type of sensory input depends on the application: it could be a numerical value (like a bank balance), language (text or speech), tactile (eg a robotic hand), or visual.  If it is a stock brokering system, image recognition is not needed. Machine translation would also not need it. But for many real-world interactions, having 'eyes' is either necessary or at the very least useful.


How can we encourage data analysts and decision-makers to adopt AI?

Not all of the mistrust aimed at AI systems is unjustified, particularly when it comes to neural networks and other such systems that rely on large training data sets. There are a number of high profile cases, facial recognition being one that has often (understandably) received a lot of flak, where improperly configured training data has resulted in skewed and questionable results.

If you want to foster more trust in the systems, it will require better tools for analyzing how they are approaching problems and reaching decisions, as well as determining if there are holes in the training data. It will require a community working in the field that gives a lot more thought and care to how they approach their training data and what unintended biases they may be introducing by forgetting something than has often been displayed presently.

Of course, some people are just distrustful of new technology, but I think it's more interesting to address the more legitimate concerns.


Is artificial intelligence and, in particular, neural networks being used in real-world critical applications?

This is as much an ethical concern as a practical one.

AI systems are already reaching or exceeding human performance in many critical areas. Consider the detection of common cancers, where AI systems match or exceed humans. Another good example is Tesla's Autopilot, which is actually safer than human drivers, but gets a lot of bad press when it makes a mistake. Both of these systems are likely using Neural Networks, possibly alongside other heuristic or rule-drive approaches.

The issue isn't whether these systems can be "safe enough" for everyday use. They are safe enough in a societal sense already. The concern is that the people who die when these systems make mistakes are randomly selected, whereas the people who die when a human performs the work die because a human makes a mistake (usually). This is difficult to accept for the same reason that some people are scared of flying, even though it is many times safer than driving the same distance: there is a loss of control, and "good" people may die through no fault of their own, or perhaps through events that are no one's fault.

Whether we use this systems will thus probably depend on the application. In Medicine it's easy to see a case: we can do more tests than before for the same price. People who can afford to have a doctor review the machine's decisions are probably no worse off. People who couldn't afford this already are better off (they get a diagnosis with some positive predictive power value now, instead of none at all). In driving, it's more complicated, and will probably require further development. No one knows for sure how good self-driving cars can get, but they'll probably get somewhat better over time. Maybe they'll get good enough that they more or less never kill people, or maybe they won't.

I actually think your friend is way off about the use cases he thinks AI is not suitable for though. Check out this article in Military Embedded Systems. In applications where decisions are made on a pure cost/benefit basis (and not based on people's gut feelings about morality), AI systems are actually easier to adopt, and are already often better than human operators. This trend seems likely to increase in the future, so "technical devices and sensors", which are often black-boxes to the lay public anyway, seem like they are among the first things to go.


The confidence equation you are referring to is the definition of precision in the Classification/pattern-recongition/information-retrieval contexts. You can visually understand the equation with the help of the following figure from the wikipedia page:

 P     : Refers to the set of samples in your dataset. (Selected elements)
|P|    : Refers to the number of samples in your dataset.
 D     : Refers to the set of correct class labels.(aka. Ground Truth).
|P & D|: Refers to the number of samples in the dataset that the classifier correctly labeled.

I hope with this understanding you can implement the fitness function for your Genetic Algorithm. If you want more help in defining the fitness function, then you should probably add details about your approach or links to the research paper you are trying to follow.


Should the mutation be applied with the hill climbing algorithm?

In general, hill climbing algorithms select a random initial solution, then takes the best move available after evaluating all possible operations available. The possible operations are determined by the search operators (which you determine and is dependent on the problem setting). So by your words, you should 'replace it with the fittest solution among its neighbours'.

Mutation can be applied as a search operator, but this varies by degree and context. For example, in structure learning for bayesian networks, a random sequence of search operators are sometimes applied to escape local optima during the search process.

See https://www.cs.helsinki.fi/u/bmmalone/probabilistic-models-spring-2014/StructureLearning.pdf - slide 9 for random restarts


Can this be a possible deep q learning pseudocode?

It looks generally valid to me. There are a couple of things missing/implied that I'd like to give feedback on though:

  I am not using replay here

Then it won't work, except for the most simple and trivial of problems (where you probably would not need a neural network anyway).

  get s,a,r,n_s

I would make it more explicit where you get these values from, and split up the assignments.

# before step()
s = env.start state()
a = behavior_policy(q_net[n_s,:])

  # inside step(), first action
  r, n_s = env.step(s, a)

  # ....rest of loop

  # inside step(), last actions
  s = n_s
  a = behavior_policy(q_net[n_s,:])

In the above code, env is the environment which takes state, action pairs as input and returns reward plus next state. It would be equally valid to have current state as a property of the environment, and query that when needed. The behaviour_policy is a function that selects an action based on current action values - typically this might use an $\epsilon$-greedy selection.

  while(!terminal) {

You appear to run just one episode in your pseudo-code. You will want an outer loop to run many episodes. Also it is not clear how you are deciding the value of terminal - in practice you will need a handle to current environment state via some variable.

Without the outer loop to start new episodes, and some variables defined to communicate between sections of code, it is difficult to follow the code and decide where to add details (such as where env.start_state() call should go, or whether something like s = env.reset() would be more appropriate).

  totalReturn+=step();

In your pseudocode you do not return anything from step and it is not clear what you hope to do with this totalReturn variable. Technically it won't equal the definition of return in RL for any state, not even the starting state if gamma < 1.0.

However, the sum of all rewards seen in an episode is a useful metric. In Deep RL it is OK to treat gamma as a solution hyperparameter, and your target metric can be expected undiscounted return from the start state.


Ah hah!

The way I had defined the disks made d5 the LARGEST disk, not the smallest. So, the last few lines of the file should be:

   (clear d4) (clear d2) (clear d5)
    (on d4 peg1) (on d1 peg2) (on d2 d1) (on d3 peg3) (on d5 d3))

  (:goal (and (on d1 d2) (on d2 d3) (on d3 d4) (on d4 d5)))
)

However

If I wanted the opposite to be true, d1 to be the largest, the changes would have been as following:

; Swapped ?disc & ?to in (smaller ...) statement
:precondition (and (smaller ?disc ?to) (on ?disc ?from)

And

; Swapped all d* and peg* objs within (smaller ...) staements
(smaller d1 peg1) (smaller d2 peg1) (smaller d3 peg1)
(smaller d4 peg1) (smaller d5 peg1)
(smaller d1 peg2) (smaller d2 peg2) (smaller d3 peg2)
(smaller d4 peg2) (smaller d5 peg2)
(smaller d1 peg3) (smaller d2 peg3) (smaller d3 peg3)
(smaller d4 peg3) (smaller d5 peg3)

:)


When to choose Stochastic Hill Climbing over Steepest Hill Climbing?

The steepest hill climbing algorithms works well for convex optimization. However, real world problems are typically of the non-convex optimization type: there are multiple peaks. In such cases, when this algorithm starts at a random solution, the likelihood of it reaching one of the local peaks, instead of the global peak, is high. Improvements like Simulated Annealing ameliorate this issue by allowing the algorithm to move away from a local peak, and thereby increasing the likelihood that it will find the global peak.

Obviously, for a simple problem with only one peak, the steepest hill climbing is always better. It can also use early stopping if a global peak is found. In comparison, a simulated annealing algorithm would actually jump away from a global peak, return back and jump away again. This would repeat until its cooled down enough or a certain preset number of iterations have completed.

Real world problems deal with noisy and missing data. A stochastic hill climbing approach, while slower, is more robust to these issues, and the optimization routine has a higher likelihood of reaching the global peak in comparison to the steepest hill climbing algorithm.

Epilogue: This is a good question which raises a persistent question when designing a solution or choosing between various algorithms: the performance-computational cost trade-off. As you might have suspected, the answer is always: it depends on your algorithm's priorities. If it is part of some online learning system that is operating on a batch of data, then there is a strong time constraint, but weak performance constraint (next batches of data will correct for erroneous bias introduced by first batch of data). On the other hand, if it is an offline learning task with the entire available data in hand, then performance is the main constraint, and the stochastic approaches are advisable.


How should I handle invalid actions in a grid world?

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options:

The move simply cannot be made and count as playing the game as intended. In which case, do not allow the agent to make that choice. You can achieve that by filtering the list of choices that the agent is allowed to make. In DQN that will mean supplying an action mask to the agent based on the state so it does not include the action at the stage it makes a choice. This "available actions" function is usually coded as part of the environment.

The move can be attempted, but results in no change to state (e.g. the agent bumps into a wall). If the goal is to reach a certain state in shortest possible time, then you will typically have 0 reward and a discount factor, or negative reward for each attempted action. Either way, the agent should learn that the move was wasted and avoid it after a few iterations.

The move can be attempted, but results in disaster (e.g. the agent falls off a cliff). This is the case where a large negative reward plus ending the episode is appropriate. However, don't use infinite rewards, positive or negative, because that will cause significant problems with numeric stability. Simply large enough to offset any interim positive rewards associated with that direction should be adequate. For a simple goal-seeking environment with no other positive rewards than reaching the goal, ending the episode early is already enough.

When you don't have a toy environment where you get to decide, then the three basic scenarios above can still help. For instance, in most board games we are not interested in having the agent learn the rules for valid moves when they are already supplied by the environment, so the first scenario applies - only select actions from the valid ones as provided by the environment.


I can not seem to grasp which specs make an encoder better than another one

In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent variables that explain the variations observed in the training set.

The autoencoder family (Variational, Denoising, Contrastive, Sparse) try to approximate p(x) so we have a performance metric to tell us how our model is doing.  e.g. (negative log likelihood of p(x))

  lets say, unsupervised learning. For example, I am trying to teach my neural network to classify cats,

If you use some autoencoder model to learn the distribution of cats, you could use the encoder part to further augmented with a linear classifier to discriminate between cats and other categories Therefore you have an intrinsic task (learn a good representation of the data distribution) and an extrinsic task (learn to classify cat vs not a cat). So you could do a hyper-parameter search for the model that best suits your problem by measuring its accuracy on the extrinsic task.

Side note:
GAN (Generative Adversarial Network) is a generative model, it provides some way of interacting less directly with this p(x) by drawing samples from it starting without any input. thus the situation here is different.


Why is it easier to construct adversarial examples relative to training neural networks?

input to output is linear refers to the input X i.e image and output is the output logits/softmax from the network.
So how does linearity help in constructing adversarial examples? Imagine a simple logistic regressor and a simple 2D space. There is a definite boundary beyond which the label that the model(i.e logisitc regressor in this case) changes. So if we move perpendicular to the boundary (i.e the line represented by the model in this case) we can get to another class's space. So if we perturb the input in this direction, the model outputs wrong class. { Refer the slide with the title Adversarial Examples from Excessive Linearity for diagram }
Now imagine the neural network trained on imagenet, there are so many boundaries and a small change can just shift change the class the model would predict. Now it is important to note that these subspaces of the image remain nearly the same if we train VGG or ResNet etc. So this explains how an adversarial example on one network effects another.
You may ask how such small change can effect. this is because the vectors we deal it is not 2d or 3d, it is very large and small changes add up.


Can you let specific data impact a neural network more than other data?

Yes, you can weight the loss function for each example, so that instead of your cost function being

$$J = \sum_i \mathcal{L}(y_i, \hat{y}_i)$$

It will be

$$J = \sum_i w_i\mathcal{L}(y_i, \hat{y}_i)$$

Where $i$ iterates over your data set, $\mathcal{L}$ is the loss function you are using, $y_i$ is ground truth for each example and $\hat{y}_i$ is prediction for each example.

You can set your relevance weighting according to any criteria you like based on the example, and your dataset/goals. So for instance you could set it to $1.0$ for complete examples and $0.1$ for incomplete examples. Depending on your NN framework, it may already offer per-example weighting, and even if it does not, auto-differentiation means that typically you only need to implement the forward logic into the cost function for each minibatch, and the weighting will be applied correctly to gradients with no more work required. If you do need to implement gradient calculations yourself, you simply multiply the initial gradient of each example by the example's weight $w_i$.

Once you make a change like this, you need to take care on how you set up and interpret your test set. When your model is used in production, if there are still inputs with missing details, you probably don't want to weight the metrics (e.g. accuracy rating) in the same way, but to report it as-is against a correctly sampled dataset of unseen examples from the population of expected inputs.


So in a sense you are correct. Using your jargon: linear regression will only "work" if the true function is approximately $y=h(x)=\beta^{T}x+\beta_0$. Advantages to using this is that its light, its convex, and all-around easy.

but for alot of larger problems, this wont work. As you said you want the machine to do the work, so this is (kinda) where deeper models come into play: You allow a learn-able featurization and classification/regression. Think about it this way, the result of your regression is most likely linearly associated with some set of features, they just may not be the ones you are interested in (you can prove this actually with any infinitely wide network :: Universal approx Thm). Unfortunately we cant use an infinitely dimensional model, so we run with these giant over-parametrized models where we hope the a good function can be described by a sub-structure (only recently are we starting to pay attention, to how these sub-structures form -- look at this paper)

But the way you bring about thinking about it is a large pit fall for many trying to move forward. Alot of ML people now of days gain success by throwing a function without alot of parameters on a big data problem, but youll see the largest advancements in the field come from a theoretical understanding of the featurization and optimization.

I hope this helped


Can a fully convolutional network always return an image of the same size as the original?

I ended up using a work around.
I set up the network so that an C x C (i.e. 320 x 320) input would output a C x C mask for some constant C (in my case it was 320).
I then resized the image I wanted to pass in to C x C, and then resized the output back to the original size of the Image.


How can I manually calculate the output a specific neural network given some input?

Given that the neurons are linear in the hidden layer of the neural network, so the output is just the dot product of the weights and the input. To put things in perspective, generally, we use an activation function (sigmoid, signum, etc.), which is applied to the dot product.

Hence, for an input of $3$ inputs to node 1 of hidden layer is $-3 * 3 = -9$, to node 2 is $2 * 3= 6$ to node 3 is $-1 * 3 = -3$ to node 4 is $0.5 *3 = 1.5$ (basically, I have performed dot product). Since neurons are linear, no activation function is applied and is directly propagated to the next layer, the output layer.

Contribution of all the hidden nodes to the output layer is the dot product of the weights with the output of the hidden layer that is $9*2 + -0.5*6 + -3*-3 + 1*1.5 = -10.5$ (check for any calculation errors). And, finally, the output layer is bipolar hence an activation function (probably sigmoid) is applied to it. Since for sigmoid function is smaller than $0.5$ if $z<0$, hence the output will be $0$, as $z = -4.5$, as per convention.


Why would giving my AI more data make it perform worse?

A lot of possible reasons:

In forecasting, you could have a seasonality. If you have it exactly 3 times, then it is good. If you have it 3.5 times, it becomes worse because it overfits on the months which occur more often than the others.
Data quality: In practice, improving the quality of data often yields better results (see Analysis and Optimization of Convolutional Neural Network Architectures, page 15 (Analysis Techniques) for 7 approaches to improve your results)
Instable training / bad luck: some architectures / problems are super instable. You can execute the randomly initialized training 5 times and get vastly different results.


Periodic Pattern in Validation Loss Curve

your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.


Training happens once you have a result. If the result is good (maybe you won in pong, or you improved your highscore in breakout) all the actions in the game are "supported" by backpropagation, if the result is bad, all the actions in the game are suppressed.

This sounds weird because in each game regardless of the end result you'll have many good and bad actions, but it works if you keep it up for many thousands of games.


You can follow the below steps :

LSTMs are slower in terms of convergence. They take much time to train amd thereby give better results. Try training the network for a more number of epochs like 50 or 65.
Use smaller batch size. Try using a RMSProp optimizer. It takes a long time to converge but gives excellent results in case of recurrent nets.
Also, try word  level predictions. Means, train your LSTM in such a way that it can predict words and not characters.
Use dropout layers in between two LSTM layers. Use a rate like 0.2 or 0.35.

As a working example, you can refer this
project which generates poem lines using LSTM in tensorflow.


Why is there an expectation sign in the Bellman equation?

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-

The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a_t$ is distributed as-
$$a_t \sim \pi(s_t)$$
Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a_t$, in general, only provides us with a distribution over the possible next states of the agent. That is,
$$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$
Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.

As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in Monte-Carlo estimation of value functions.

EDIT:
As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.


Is reinforcement learning needed to create Strong AI?

Motivational Theory Versus Learning Theory

Rewards and punishment are based on human motivational theory and are broad generalizations of the neurological factors in mental growth.  At a neurological level, there are chemically based signals, such as seratonin, dopamine, and oxytocin.  Some of these have their own pathways in the brain.

There are also pain signals arising from the body and internally generated stimulatory and inhibitory signals in chemical and electro-chemical forms.  There is neuroplasticity, where the brain circuits form new topologies.  The hippocampus is a structure known to be important in learning.1  The study of learning in the brain is fervent, but the object of study is complex.  Even so, every year, progress is made.

Multiple Dimensions or Singly Quantifiable Intelligence

Conjecture about strength versus weakness of intelligence may be of limited use.  There are two valid criticisms discussed in other Q&A here in more detail.

Twenty-two independent genes have been identified that correlate to typical academic intelligence quantification in humans, rational and linguistic, eleven of which were identified in 2017.2  There are likely more.  Creative, athletic, emotional, and leadership qualities have not yet been captured in standardized intelligence testing, so it is likely that additional genes related to those abilities exist.  The notion that intelligence can be accurately aggregated into a single g-factor is weakly supported, if it were true, implies that all twenty-two genes affect only one magical function.  Statistically, such alignment is close to impossible.
Intelligence is only meaningful in context.  Fast learning in one context appears like mental brilliance, but in another context the same learning capability may be dysfunctional.

Reinforcement in the Larger Pool of AI Research

Reinforcement Learning is not a requirement for AI as far as is currently known.  Reinforcement is a mathematical strategy for driving a set of behavioral parameters toward a defined optimum.  Although its variants work well for many applications, it is not closely aligned with any of the features of the hippocampus, the limbic system, the cerebral cortex, or any other learning systems in the human brain.

Even though there is still much to learn about brain workings, the success of reinforcement learning, rule based systems, convolution, attention based networks, and generative networks provides information about what is possible in computers and what may or may not be a mechanism of the brain.

What Reward and Punishment Imply

The existence of rewards and punishment imply the existence of favorable and unfavorable family, workplace, or community states.  If the term world in the question is referring to the immediately related environment, then the sentence is probably true, but it may be misleading without clarification.

World states are based on economics which involve the minds of many people in community, aggressive actions, resource acquisition, consumption, cultural perspectives, and many other factors.  A single individual's perception of world states is subjective.

Defying Motivational Mechanisms and Intelligence

The same person on two different days may have different ideas of what is favorable or unfavorable, which may or may not correlate to reward and punishment motivations.  The words rebellion and indifference describe states where external reward and punishment produce opposing or independent results in learned behavior respectively.  Some of the most intelligent behavior in humans were of these kinds.

What drives the classification of external states within a single individual is something not well understood.  The neurological factors involved in the emergence of opposition to pain, pleasure, reward, and punishment is an interesting area of study and certainly related to AI.

Genetic factors involved in such contrary learning are likely to fall outside the twenty-two genes, since those genes correlated with standardized academic testing.  Academia is about conformance.  If we don't agree with the author of the textbook for a course, we pretend we do when we take the test if we want a high grade.

World-changing intelligence, such as that of Ghandi's nonviolent opposition, Isaac Newton's invention of physics, the furtherance of logical disproof by Socrates, or whoever dared to learn how to light a fire. is always the exhibition of classifying in a way that contradicts that of the surrounding culture.

Discovery and Acceptance in Group Learning

This is even true of project-changing intelligence.  A software engineer might say, "That software will never work because the database schema indicates a one to many relationship between companies and office addresses, but it is a many to many relationship."  Such a statement doesn't change the world and may completely lack group acceptance when spoken.  Later in the project, after the statement is fully accepted, the reclassification changes the design so that the project is not caught in a well of wasted coding.

Interestingly, convincing the others often falls to someone other than the originator of the idea — someone else on the team that has a different set of mental abilities.

It is true that classification of what is favorable or unfavorable is key, but how that classification exhibits intelligence is not well understood.

What Benefits AI Development

Although not a requirement, a solid approach to AI development is discovery, from the directions of multiple fields of study, of what intelligence is.  Some say we know what it is now, but most definitions are largely expressions of historical inaccuracies.

Clarity about what intelligence is and what it is expected to be in such a way that it can be represented mathematically is needed.  Only then can we better achieve it in humans and machines.  Sources of understanding may come from discovery in any area of study, such as these.

Evidence based psychology
Evidence based genetics
Evidence based neurological research
Evidence based artificial systems design

There is a growing trend to hack AI, but playing with AI as if it were a game might be a formula for creating Frankenstein.  A useful and collaborative collection of AI devices in robots, autonomous vehicles, and data centers will require interdisciplinary inquiry, experimentation, and engineering.  The description of this site implies that ethics should be involved, and that's a smart conception.

With a purview that crosses divisions in study and practice, central themes and models of intelligence are likely to develop.  This question is on the right track in this respect. It asks questions without accepting too quickly conjecture  — ideas and hypotheses presented but not yet shown to be true.

What is Known

That computer simulation or replication of human abilities have already been achieved and that such achievements indicate AI to be a reasonable long term objective few would deny.  Certainly in the areas of arithmetic, high speed machine control, sorting of addressed parcels, and many other roles, computers far outperform humans.  Whether world-changing intelligence will emerge in artificial systems is hoped for by some, feared by others, and asserted as inevitable by those who seek funding to prove it.

References

[1] Training your brain: Do mental and physical (MAP) training enhance cognition through the process of neurogenesis in the hippocampus?, D.M. Curlik and T.J. Shors, 2012

[2] Genome-wide association meta-analysis of 78,308 individuals identifies new loci and genes influencing human intelligence, Suzanne Sniekers et. al., 2017


Are there any DeepQA-based computers other than Watson?

I cannot say for certain, but I know of no such other uses (I work at the building where Watson is developing but do not directly work with it).

The DeepQA team's page (https://www.research.ibm.com/deepqa/deepqa.shtml) only ever references Watson as the implementation, and based on the structure of the FAQ there I would imagine they'd be eager to list any interesting other uses it would have, but no such entries exist there.

I would, however, also note that while DeepQA is IBM's proprietary implementation of QA using deep-learning methods, that approach is becoming more popular and is almost certainly being worked on by other companies and could soon be on other computers, if not already so in some form.


Should I train different models for detecting subsets of objects?

This is called decomposition of multi-class classifier. Your proposed method is called one vs all.

  One vs. all provides a way to leverage binary classification. Given a classification problem with $N$ possible solutions, a one-vs.-all solution consists of $N$ separate binary classifiers—one binary classifier for each possible outcome. During training, the model runs through a sequence of binary classifiers, training each to answer a separate classification question.

Source: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/one-vs-all.

According to this article. The author of the article did experiments on SVM on 8 different benchmark problems. According to the results, this method is sometimes as good as others, but usually not the best. It is also never substantially better than any other method. The article also stated that the best method is usually problem dependent.

Also, this method will decrease inference speed a lot, and used substantial amount of GPU memory. According to the source, it does not improve performance a lot, so you best bet for getting a higher performance is probably to use a different model architecture, for example the FPN FRCN, which is stated in the YOLO v3 paper having the best performance, but not fast inference speed. YOLOv3 is designed to have a fast inference speed, to provide real time object detection system, so for performance you should probably use other model architecture instead.


Can CNNs be made robust to tricks where small changes cause misclassification?

These are known as adversarial attacks, and the specific examples that are misclassified are known as adversarial examples.
There is a reasonably large body of work on finding adversarial examples, and on making CNNs more robust (i.e. less prone to these attacks). An example is the DeepFool algorithm, which can be used to find perturbations of data which would cause the label to change.
There are several techniques in the literature which are used to fight against adversarial attacks. Here is a selection:

Augmenting the training data with various random perturbations. This is intended to make the model more robust to the typical adversarial attack where random noise is added to an image. An example of this approach is discussed in [1].

Constructing some sort of model to "denoise" input before feeding into the CNN. An example of this is Defense-GAN [2], which uses a generative adversarial model which models the "true" image distribution and finds an approximation of the input closer to the real distribution.

References
[1] Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. Explaining and Harnessing Adversarial Examples. ICLR (2015). URL.
[2] Pouya Samangouei, Maya Kabkab, Rama Chellappa. Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models. ICLR (2018). URL.


Encourage Deep Q to seek short-term reward

Thanks for the responses - after taking into account the 'risk' the network experiences when working with unstable financial data I modified the activation layers from RelU to Sigmoid - this has lead to a considerable improvement to both results (profit!) and additionally the rate at which actions are carried out.

I will also edit the above question to provide further details.


What's the right way of building a deep Q-network?

Do I have to associate to each output an action?

You are absolutely correct! In DQN, each output node of the neural network will be associated with one of your possible actions (assuming a finite discrete action space). After passing an input through the network, the value of each output node is the estimated q-value of the corresponding action. One benefit of this architecture is that you only need to pass the input through the neural network once to compute the q-value of each action, which is constant in the number of actions. If you were to include an action as an input to the neural network along with an observation, then you would need to pass an input for each action, which scales linearly in the number of actions. This is mentioned in paragraph 2 of Section 4.1 in the original DQN paper (https://arxiv.org/abs/1312.5602)

  Is it really the correct way? If so, how do you compute the argmax or the max?

It's one possible way that is used in many popular algorithms such as DQN. To find the argmax, you simply take the action corresponding to the output node with highest q-value after passing an input through the network.


How can I train a neural network to give probability of a random event?

The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.
You said both
and

there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.

It seems that this problem has 2 solutions:

Don't use a neural net and create a 'std' statistical model.
It may be possible since you said:

I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)

Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.


LSTM network doesn't converge, what should be changed?

writing here my suggestion, because i haven't earned the right to comment yet.

Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer.

Alternatively you could change the loss function to fit your need. For example you could use:

loss  = tf.reduce_mean(tf.square(exps-outs))

You will get a smaller loss value which decreases clearly after every output.

I hope this helps :)


In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?

The most important word for answering your question from that quote from the paper is probably the word "usually": These search probabilities usually select much stronger moves than the raw move probabilities $p$ of the neural network. It's not always going to be true, but more often than not / most of the time / "on average", we intuitively expect it to be true. In theory there could be pathological cases, especially with low MCTS iteration counts and/or a poorly-trained network, where it may not be true. But even if we just find it to be true most of the time, that can be good enough for the algorithm to work well in practice.
Recall that in the Selection phase of MCTS, Alpha(Go) Zero always selects the action $a$ that maximises the following expression for a state $s$ when traversing the tree (see supplementary material of Alpha Zero paper):
$$Q(s, a) + C(s) \frac{P(s, a) \sqrt{N(s)}}{1 + N(s, a)}$$
The $Q(s, a)$ values here are the value estimates resulting from the MCTS search; these are the results produced by our tree search, and these generally become more and more accurate the longer our tree search runs. If we run our tree search for an infinite amount of time, we expect these to converge to the true minimax values.
These same $Q(s, a)$ values are the things that can cause the final distribution of visit counts $\pi$ to shift away from the prior neural network distribution $P$. Suppose that we have a hypothetical case where for all the different actions $a$, the $Q(s, a)$ values end up staying identical. In this (unlikely) hypothetical case, the visit counts would continue getting distributed proportionally to $P(s, a)$, causing the $\pi$ distribution to stay equal to the $P$ one (barring minor potential differences due to $\pi$ being derived from discrete, integer visit counts which may not be able to exactly replicate the real-valued probabilities in $P$). Only if the $Q(s, a)$ values resulting from a (hopefully smart!) tree search algorithm give us reason to shift away from the prior distribution $P$ do we actually really start shifting away from it.