Constructing a dataset that scores well only for a specific set of hyper parameter values

I'm not familiar with any existing, robust methods to generate such a dataset. Here are some thoughts though.

You propose using an MLP with a single hidden layer. That means we have two weight matrices, and two activation functions (one for hidden layer, one for output layer). Some notation:

$d$: dimensionality of input vectors
$n$: dimensionality of hidden layer
You mentioned binary output, so I'll assume that dimensionality is $2$
$W_n^{(1)} \in \mathbb{R}^{d \times n}$: the first weight matrix in the scenario where we're using $n$ hidden nodes
$W_n^{(2)} \in \mathbb{R}^{n \times 2}$: the second weight matrix in the scenario where we're using $n$ hidden nodes
$g$: first activation function
$h$: second activation function

Then, given an input vector $x \in \mathbb{R}^d$, our neural network will generate output $f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$.

Now, in general we of course expect Neural Networks to only find a local optimum, but if you want a robust solution you'll want it to be able to handle the worst case, and the worst case for your "adversarial" task is when the Neural Network manages to find the global optimum. So, we'll assume it can find the global optimum.

Essentially, what you're looking for is a dataset $D_n$ containing a number of input vectors $x$, such that:

$f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$ provides good / optimal results (after training to the global optimum)
$f_{n - 1}(x) = h(W_{n - 1}^{(2)} (g(W_{n - 1}^{(1)}x)))$ provides poor results (even after training to the global optimum of this setup).

In other words, you want to find a collection of vectors $x$ such that it becomes impossible to find a collection of weights in the $n - 1$ case where $f_{n-1}(x)$ is a good approximation of $f_n(x)$. You want to make it impossible that $f_{n - 1}(x) \approx f_n(x)$.

Now this has turned into a clear mathematical problem. I'm not familiar with any established methods in mathematics to solve a problem like this, maybe there are though. My best guess at this point in time would be a procedure like the following:

Generate random "ground truth" versions of the weight matrices $W^{(1)*}_{n}$ and $W^{(2)*}_{n}$ (just completely random matrices).
Generate completely random input vectors $x$. Compute the corresponding ground truth labels as $f^*_n(x) = h(W_n^{(2)*} (g(W_n^{(1)*}x)))$.
Hope that the neural network with $n$ hidden nodes can recover the ground truth weight matrices that were previously generated randomly.
Hope that the neural network with $n - 1$ hidden nodes cannot find an accurate approximation.

In theory, the MLP with $n$ hidden nodes should be able to learn the exact ground truth function. In theory, under certain conditions, the MLP with $n$ hidden nodes should not be able to learn the exact ground truth function. I suspect those "certain conditions" would be that the rows/columns of the weight matrices should be linearly independent, which is likely with randomly generated matrices, but I'm not 100% sure on this. Even if it can't learn the exact ground truth, it may still be capable of learning an approximation... there may be ways to find upper bounds on how close such an approximation could get, but I'm not sure.


What is the difference between backpropagation and predictive coding?

I would say that these concepts are quite different, even though they might have a few things in common (or might be vaguely related).

Back-propagation is an algorithm used (in machine learning) to compute the gradient of a function with respect to its parameters. This gradient is then used by algorithms, like gradient descent, to update the parameters of the model (e.g. a neural network).

Roughly, predictive coding is a general (neuroscience) theory of how the brain builds an internal model of the external world, of how it continuously predicts the sensory inputs (from the world) using its current model of the external world, and of how it updates this internal model once the sensory inputs are actually received.

You could think of the output of an ML model, while it is being trained (using gradient descent with back-propagation), as a prediction associated with the inputs. However, note that the output of the model is often not a prediction of the actual input, but e.g. a label (which is, nonetheless, associated with the input). Furthermore, it is the model that produces an output and not the back-propagation algorithm (even though the back-propagation algorithm is often used to train such models, e.g. neural networks). We could think of back-propagation as the way of updating these predictions associated with the inputs, but, anyway, it is well known (and accepted) that our brain does not perform back-propagation, but we learn in an associative fashion (Hebbian learning).


Is there an advantage in decaying $\epsilon$ during Q-Learning?

Yes Q-learning benefits from decaying epsilon in at least two ways:

Early exploration. It makes little sense to follow whatever policy is implied by the initialised network closely, and more will be learned about variation in the environment by starting with a random policy. It is fairly common in DQN to initially fill the experience replay table whilst using $\epsilon = 1.0$ or otherwise an effectively random policy.
Later refinement. Q-learning can only learn from experiences it has. A behaviour policy that is too random may not experience enough states close to optimal in order to gain enough statistics on them to overcome variance. In more difficult cases, it may never experience the whole of an optimal trajectory even when combining all the different transitions that it ever observes.

In addition, when using a function approximator such as a neural network, predictions of the target policy will be influenced by the distribution of states and actions in the experience replay memory. If that is is biased towards the distribution of states of a very different behaviour policy, then basic Q-learning has no good way to adjust for that - it adjusts for differences in expected return between behaviour and target policies, but not in distribution of observed states. In fact this is still a somewhat open problem, you want the agent to learn from mistakes and imperfect behaviour so as to avoid it, but you don't want those mistakes to skew predictions of what should happen under an optimal policy. Function approximation such as neural networks is influenced soemwhat by the distribution of input data, so it typically performs better in Q learning when the behaviour policy and target policy are close e.g. $\epsilon$ should be relatively low if using $\epsilon$-greedy.

A typical implementation might start with $\epsilon=1.0$, set a decay factor per time step or per episode e.g. $0.999$ per episode, and a minimum $\epsilon = 0.01$. These are all hyper-parameters that you can adjust depending on the problem.


Understanding how to calculate $P(x|c_k)$ for the Bernoulli naïve Bayes classifier

Bernoulli naïve Bayes

$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$

Let's examine the example of document classification.
Let K different text classes and n different terms that our vocabulary contains. $x_i$  are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document x. x is a vector of dimension n.

$P(x|c_k)$ is the probability that given the class k, document x to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x_i = 1$, then $1 - x_i = 0$ and inversely. In other words, for each term it takes the probability that the document does contain this term or it does not.

$p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, that is it could be the prior probability a document that belongs to k class contains this term of the vocabulary.


What are the counterparts of non-linearities and dropout in fully convolutional networks?

All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to replace the pooling and fully-connected layer with particular convolutional layers that do the same. Note that activation functions and dropout are not affected by this transformation. You can read more about it here.

The same will be true for the deconvolutional layers. I didn't find it stated explicitly in the original paper, but here's another work that applies fully-convolutional network for image segmentation. On the figure 2, you can see ReLu activations: they are placed after (simple) convolutional layers, in both upsampling and downsampling parts. They don't use dropout, but batchnorm instead, but if there were a dropout, just as before, it should be placed after the convolutional layer. The keep probability, though, should probably be higher - this relates to hyperparameters tuning and may be different depending on the problem and available data.


How to determine if an Amazon review is likely to be fake using text classification

This will not be that hard of a problem once you have a lot of training data. But, before you have a lot of training data, you will need to get some training data one way or another. You will need a lot of training data for quite a few of the models that will give you a high accuracy. Then, you will probably want to use a Long short term memory recurrent neural network along with a word2vec model, or maybe even a sentence/ paraphraph2vec model. This will be able to give you some fairly good results with a little bit of tweaking, but if you want really accurate results you will want to try an ensemble. An ensemble is when you use multiple neural networks that provide an output that gets fed through another classifier(usually XGboost) in order to achieve better classification results. You might also want to try a little bit of feature engineering. Finally, in order to implement this model, you can take one of several approaches. First off, you can use tensorflow serving. Tensorflow was created by google and used by much of the machine learning community. However, tensorflow was written in python, so you will need to use tensorflow serving in order to use it within your Asp.net app. Or, you could use microsofts CNTK. Or finally, you could use a custom built implementation, but this would take a lot of time and not be worthwhile unless you are doing research. Also, if you really do not want to use deep learning for one reason or another, you can probably use a simpler model, but you will need to understand your data. You will need also need to understand your data if you are doing deep learning but maybe not as much.


What is it called in AI when a program is designed to make "x in the style of y"?

This problem is known as "style transfer".
The field was started by Gatsys et al. in 2016, and has seen a lot of work over the last few years, including conditional translation from Isola et al. in 2017 and unpaired translation from Zhu et al. 2017.

This picture (Figure 2 from Gatsys et al.) shows the idea behind style transfer, and illustrates the dramatic results that are possible with these models.


Why do we need Upsampling and Downsampling in Progressive Growing of Gans

Use of Transposed Convolution can lead to checkerboard artifacts.  So we prefer to up-sample and then apply convolution. You can check this article for more information https://distill.pub/2016/deconv-checkerboard/.


If you mean high level assessment of self-learned evaluation functions in chess, then no, the advantage of a better evaluation function lies in the ability to prune the search tree more aggressively. So you would on the contrary try to search narrowly but deeply.

(In reality neural network based evaluation functions are so slow, that you would search narrowly and still not get very deep. Nor very strong.)

If you mean chess programming in general, than the answer is also no. In chess you have to go deep, at least selectively, because tactical possibilities that occur deep in some variations are important.


How exactly does adversarial training help in handling mode-collapse in generative networks?

I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then $min_{\phi}max_{\theta}L(G_\phi, D_\theta)$ starts to look a lot like $max_{\theta}min_{\phi}L(G_\phi, D_\theta)$.

This is problematic because in the latter case the generator has an obvious minimum of transforming all generated output into a single-mode that the discriminator has considered acceptable.

Since then a lot of work has been done to deal with this point of failure. Examples include Unrolled GANs (he mentions this one in the talk), where you essentially make the generator optimize what the discriminator will think $K$ steps in the future to ensure the ordering of the $min \ max$ game, and Wasserstein GANs, where you focus on a different metric that still has the same global minimum but allows for side by side training completely eliminating the ordering and failure mode, to begin with. On top of this, other work has been done as well, these are just two important examples.

Regarding how they fare against other generative models, like VAEs, there is no one is better than the other. The recent empirical success of GANs is why they are so popularly used, but we still see others being used in practice as well.


Is it a valid assumption that a purely MLP based tic-tac-toe player will learn lookahead strategies?

A MLP only does pattern recognition, it will not learn search.

Tictactoe, (Oughts and Crosses), is such a simple game that your network should learn the moves from the training data by heart, no generalisation required. If it still loses games, maybe your training data doesn't consist of particularly good moves.


What are the strategies for computationally heavy environments or long-time waiting environments?

You could use Ray RLlib. It has support for parallel environments, even over multiple GPUs and compute nodes.


Is there a neural network method for time-varying directed graphs?

I'm seeing recent trend of combining RNN/CNN with GNN(graph neural networks) so that both time dependency and topology are captured. I would suggest you to start by looking at DCRNN (Yaguang Li et al.), it's a strong baseline that everyone uses nowadays. Other good resources:

Graph wavenet for deep spatial-temporal graph modeling (Zonghan Wu et al.)
Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting (Bing Yu et al.)
STANN: A Spatio–Temporal Attentive Neural Network for Traffic Prediction (Zhixiang He et al.)


Does MMD-VAE solve the problem of blurred images of vanilla VAEs?

[Answering my own question after 5 months of studying VAE models]

The point of the MMD-VAE or InfoVAE is not exactly to emphasise on the visual quality of generated samples. It is to preserve greater amount of information through the encoding process. The MMD formulation stems from introducing a mutual coefficient factor into the Evidence Lower BOund (ELBO) loss of VAEs. Refer to the paper appendices for full derivation. This formulation improves information content in latent space and provides for more accurate approximation of the true posterior - these results have also been empirically proven in the paper.

However, the InfoVAE uses pixel-wise or element-wise reconstruction loss. An element-wise reconstruction loss is likely to lead to some extent of blurriness inrespective of the prior loss term. On Github, several developers have implemented the InfoVAE model and shown their results. Here is a link to one such implementation whose results I could personally verify.

From my own experimentations, I can say that even though InfoVAE may give better reconstructions for some data, there is still considerable blurriness.

Perceptual similarity metrics may be learned or computed as a static function of the input image. With a learned perceptual loss, VAEs can produce much sharper images. PixelVAE and VAEGAN are well-known models with such implementations. For a static function of the image itself, reconstruction quality will depend on the nature of that function and such a model may not be very useful for all kinds of datasets. Using measures like SSIM, FSIM, we may still end up getting blurred images.


If the training data are linearly separable, which of the following $L(w)$ has less optimum answer for $w$, when $y = w^Tx$?

First things first. What is an optimal $w$?. In this case it is supposed to be not only the one that minimizes the emprical /sample loss, but also non-trivial as we shall sonn see. Now inspect the loss functions, we see a term $-y_iw^Tx_i$ coming up. What exactly is this term? It can be anything. The correct term would have been $-y_i(w^Tx_i + b)$, or atleast I will assume this term, but the reasoning will exactly be the same without this term.
Now this term is well defined and since the Eucledian distance of a point, along with sign, $x_i$ from the hyperplane $w^Tx + b=0$ (the proof for this can be found easily), is given by $w^Tx_i + b$. So if the datapoint is supposed to have $y_i=-1$ but according to the classifier it gives $w^Tx_i + b = l_i > 0$, then one cane see $-y_il_i > 0$, hence $\max (0, -y_i l_i) = -y_i l_i$ a positive loss, and the same will apply to case when $y_i = 1$ and $w^Tx_i + b = l_i <0 $
We see another assumption, the datapoints are linearly seperable. This means, there exists a classifier (or a hyperplane) which can completely seperate the 2 classes. Now if one inspects the loss function $1,3,4$ all of them have one goal to make $\max (0, -y_i l_i) = 0, \forall i$. A point to note is that $1$ does this by directly reducing $\sum_{i=1}^n \max (0, -y_i l_i)$, $3$ does this using the $0-1$ loss i.e if the instance is wrongly classified (or $-y_il_i > 0$), then incur $\frac{1}{n}$ loss else $0$, while $4$ incurs a loss of $\frac{1}{n}(w^Tx_i + b)^2$ for each wrong classification. It is very important to note that the solution of all these 3 problems are all the same $w^*,b^*$ which classifies the dataset correctly, which makes the problem equivalent. (This may not hold under some very similar looking loss functions, so it is a point to take care about, a problem is equivalent only if it results in the same solution).
In $2$, first $C \rightarrow \infty$ is an incorrect statement, since then not only does it become mathematically unsound (atleast for me), it also means $2$ is the same as $1,3,4$. Why? $C$ is a weighting factor, it decides how much priority the algorithm minimizing loss will give to the term $C \sum_{i=1}^n \max(0,-y_il_i)$. If say $C=0$ then you are basically optimizing $||w||_2^2$ whose optimal value is at $w=0$. hence, we get nothing useful as optimizing $||w||_2^2$ has no connection with our objective of reducing misclassifications. But if,  $0.5||w||_2^2 + C \sum_{i=1}^n \max(0,-y_il_i)$ is used the algorithm is now giving some weightage to minimizing both the $||w||_2^2$ as well as the misclassifications. If $C$ is very small the algorithm will prefer to minimize $||w||_2^2$ rather than $C \sum_{i=1}^n \max(0,-y_il_i)$. Thus giving a solution which may not be optimal, as per the dataset. But, if $C$ is very large the algorithm will mostly try to minimize  $C \sum_{i=1}^n \max(0,-y_il_i)$, but still it will give some weightage in minimizng $||w||_2^2$, hence may not find an optimal hyperplane. If $C \rightarrow \infty$ it is matheatically ill defined (as per my knowledge) so I won't go into it.
But, all these aforementioned explanation (i.e the tradeoff between optimizaing two different objectives) will be useful only when there are some additional constraints, like instead of $\max(0,-y_il_i)$ it is $\max(0,1-y_il_i)$ where making $w$ bigger actually makes sense as it will also increase $l_i$ (i.e the same hyperplane denoted by a larger $w$ say $kw$ will make l_i large hence $-l_iy_i$ small if correctly classified, but this doesn't make sense if we are talking about the same hyperplane), hence the $||w||_2^2$ term opposes such an increase. This is a very famous formulation of loss used in SVM.
But, the challenging part in this question is the assumption 'linearly seperable' and also the additional constraint is missing (which is present in SVMs). So if $w^*,b^*$ incurs $0$ classification error so will $kw^*, b^*$ where $k$ is any scaling factor. And since $C$ is very large the second term in $2$ must be $0$. Thus, any $kw^*, b^*$ will work to make the second term 0. But, the first term now becomes $0.5k^{2}||w^*||_2^2$, where $w^*$ denotes the optimal hyperplane, and hence fixed. Thus the first term will choose  $|k| \rightarrow 0$ or basically $k=0$ to minmize the 1st term and the second term now simply becomes $C\sum_{i=1}^n \max(0,-y_ib)$ which can be made $0$ by choosing $b=0$. Thus, a $0$ loss when $kw^*=0,b^*=0$ or a trivial solution and definitely not optimal as this will be true for all linearly seperable datasets.
I assumed and extra $b$ term to leverage linear seperability, without it the reasoning becomes easier and you can follow the exact same line of reasoning, albeit the problem might not be linearly seperable anymore. The solution may seem not very elegant but this is a standard line of reasoning in optimization problems where the optimization variable is changed from $w \rightarrow k$, where $w \in \mathbb{R^n}$ and $k \in \mathbb{R}$.
In short for $2$ the solution produced will be $w^*=0,b^*=0$ which will be the best solution but easily seen as trivial and non optimal at all.


Deep learning for bitmap tracing: extract simple svg paths from bitmaps

If we seek proven working source code to plug into a GPLv2-licence compatible solution, we should at least consider autotrace. Its source code is open for review. It can be tested against the example images we have and, if it works fine, called by our GPLv2 software. We can even use the calling code in Inkscape's plug-in image tracing implementation as a good starting point for design and implementation of our calling program, whether it be C, C++, Java, Python, or ECMA (JS).

The trace algorithm in Adobe Illustrator is comparable but is not open source.

If we seek theory, there are several academic publications discussing some of the theory, the last being most aligned with machine learning ideology. I would not dismiss earlier work simply because it doesn't connect with the current machine learning idioms. Investigating what is fully implemented and successfully used by many follows a wise old business proverb: The bird in the hand is worth two in the bush.

Potrace: a polygon-based tracing algorithm, Peter Selinger, 2003
Vector Representation of Binary Images Containing Halftone Dots, Kei Kawamura, Hiroshi Watanabe, Hideyoshi Tominaga, 2004
Testing AutoTrace:
A Machine-learning Approach to Automated
Tongue Contour Data Extraction, Jae-Hyun Sung, Jeff Berry, Marissa Cooper, Gustave Hahn-Powell, and Diana Archangeli, 2013

Many of the online drawing programs collect data.  It would not be surprising if, behind the gracious give-away of online bandwidth, they are establishing a continuously improving data set for training a new breed of autotracers.  None have published AI designs admitting as much, but they would not be legally obligated to do so because a single input example is indeterminable from the autotrace service that could resulting from the training.


Let me try to explain how recommender systems work in production, as intuitively as possible:

Let's say we want to build a rec sys. for a restaurant discovery product, where users can rate restaurants, add reviews, photos, etc and also order food from there.

So, the user's feed will have a list of restaurants in his/her area. But, as I gain money from restaurants in the $-per-click model, I need to maximize the number of times a user clicks on a restaurant.

Also, it is obvious that a user is more likely to click on a restaurant if he tends to like it the most. [Restaurant features being Cost-for-two, cuisines, ratings, etc]

So here, a user is a data point and so is the restaurant. So, let's say the distance between the user vectors and the restaurant is the "likeability" of the user for the restaurant.

Let's say the vector is in the form [Japanese Spanish Mexican Chinese Indian Thai Turkish Lebanese]
Let's say a restaurant's vector is: A = [0 0 0 1 1 0 0 0]

and the user's is: B = [2 23 4 53 43 21 2 45]

Each number being a particular cuisine. For restaurant, it is yes or no (1 or 0), a.k.a whether the cuisine is served or not.

For user, it can be the number of clicks he/she did on restaurants with that cuisine. (I am over-simplifying here. But, this can be as complex as a weighted score for clicks+transactions+content_generated like review, ratings, etc)

Now, the the cosine similarity measure between the 2 vectors in an 8-dimensional space (length of the vector) is the likeability score.

Now, the system uses these scores while doing it's feed ranking for each user.

This can be near-realtime to being updated hourly/daily, depending on the servers the company can afford.


What is this algorithm? Is it a variant of Monte-Carlo Tree Search?

In short it looks like you have constructed a valid reinforcement learning method, but it does not have much in common with Monte Carlo Tree Search. It may have some weaknesses compared to more established methods, that means it will work better in some environments rather than others.
Your approach may be novel, in that you have combined ideas into a method which has not been used in this exact form before. However, it is following the principles of general policy improvement (GPI):

Estimate action values of a current policy.

Create a new policy by maximising action choices with respect to latest estimates.

Set current policy to new policy and repeat.

Your method covers exploration with a deterministic policy by sweeping through a list of state and action pairs. This resembles policy iteration, or perhaps exploring starts in Monte Carlo control. It evaluates each state/action pair by following the current policy up to a time step horizon. This has some weakness, but it may work well enough in some cases.
The depth iteration is more complex to analyse. It is not doing what you suggest - i.e. it does not make the whole algorithm equivalent somehow to a tree search. Technically the value estimates with a short horizon will be poor, biased estimates of true value in the general case, but on the plus side they may still help differentiate between good and bad action choices in some situations. It may even help as a form of curriculum learning by gathering and training on data about immediately obvious decisions first (again I expect that would be situational, not something you could rely on).
Overall, although you seem to have found a nice solution for your racing game, I think your method will work unreliably in a general case. Environments with stochastic state transitions and rewards would be a major problem, and this is not something you could fix without major changes to the algorithm.
I could suggest that you to try one or more standard published methods, such as DQN, A3C etc, and compare them with your approach on the same environments in different ways. E.g. how much time and computation it takes to train to an acceptable level, or how close to optimal each method can get to.
The main comparison with Monte Carlo Tree Search is that you evaluate each state, action pair with a kind of rollout. There is lots more going on in MCTS that you are not doing though, and the iteration with longer rollouts each time is not like anything in MCTS and does not compensate for missing parts of MCTS in your algorithm.


What is experience replay in laymen's terms?

In reinforcement learning (RL), an agent interacts with an environment in time steps. At each time step $t$, the agent and the environment are in some state $s_t$. From that state $s_t$, the agent chooses and executes an action $a_t$ and the environment emits a reward $r_t$ (which values the just taken action $a_t$). Finally, the agent and the environment move to the next state, $s_{t+1}$. This interaction proceeds until either the agent dies or some other termination criterion is met.

The goal of the agent is to obtain the highest amount of reward in the long run (that is, not just in the next time step, but in all successive time steps). To do that, ideally, the agent needs to find a way of behaving "optimally". In RL, the behaviour of the agent is called a "policy". An optimal policy is a policy that allows the agent to obtain the (expected) highest amount of reward in the long run.

In this context, we can describe a full and finite (in terms of time steps) interaction between an agent and an environment as a sequence (sometimes called a "rollout" or a "trajectory") of states, actions, rewards and next states. So, a rollout might look like this $$(s_t, a_t, r_t, s_{t+1}, a_{t+1}, r_{t+1}, s_{t+2}, \dots, s_{T-1}, a_{T-1}, r_{T-1}, a_{T}),$$ where $T$ is the last time step of the interaction between the agent and the environment. During the interaction between the agent and the environment, the agent might decide to store this "experience" in a "buffer" (e.g. an array), so that it can use it later (you will see below a use case).

The elements of this type of sequences are often temporally correlated. What does this mean? For example, suppose that states are frames of a video game (that is, each frame is a different state). In this context, successive frames (or states) are similar to each other, which mathematically means that they are correlated.

It turns out that neural networks (NNs) are able to approximate (almost) any function. In RL, a policy is also a function: it is a function from a state to an action (or probability distribution over actions). So, we can represent a policy using a NN. (Deep RL is essentially a combination of traditional RL algorithms, like Q-learning, with NNs).

Moreover, it also turns out that training a NN using back-propagation with data that is temporally correlated might lead the NN not to capture the essential characteristics of the data, which, in practice (during training), means that we are not able to find the NN that represents the optimal policy (or another function, e.g. $Q(s, a)$, used later to retrieve the policy). In such cases, we often say that the training of the NN is not stable.

In the case of RL, the data used to train such types of neural networks (which represent the policy, or other functions that are used in RL) are the "rollouts", which contain elements that are often temporally correlated. Hence, we can't just feed the NN with a rollout, in the same order that the elements (states, actions, rewards and next states) are collected. So, in order to use uncorrelated data to train an NN, we can randomly take tuples of the form $\langle s_h, a_h, r_h, s_{h+1} \rangle$ from the rollout (where $h$ is some time step between $t$ and $T$). For example, suppose we take (or "sample") $3$ tuples $\langle s_7, a_7, r_7, s_{8} \rangle$, $\langle s_{97}, a_{97}, r_{97}, s_{98} \rangle$ and $\langle s_{2}, a_{2}, r_{2}, s_{3} \rangle$. Given that these elements have been observed at quite different points in time, they are likely to be less correlated than e.g. $\langle s_7, a_7, r_7, s_{8} \rangle$, $\langle s_{8}, a_{8}, r_{8}, s_{9} \rangle$ and $\langle s_{9}, a_{9}, r_{9}, s_{10} \rangle$ (which are successive "tuples of experience").

In this context, "experience replay" (or "replay buffer", or "experience replay buffer") refers to this technique of feeding a neural network using tuples (of "experience") which are less likely to be correlated (given that "random sampling" procedure). The "buffer" part refers to a data structure (e.g. an array or list) that stores the trajectory (or rollout), that is, it stores the "experience" of the agent (hence the name "experience"). The "replay" refers to the fact that this "experience" is reused (or "replayed") by randomly sampling from it to train the NN.

See also this question Why exactly do neural networks require i.i.d. data? regarding the fact that NNs often require i.i.d. data.


What is the difference between one-shot learning, transfer learning and fine tuning?

They are all related terms.

From top to bottom:

One-shot learning aims to achieve results with one or very few examples. Imagine an image classification task. You may show an apple and a knife to a human and no further examples are needed to continue classifying. That would be the ideal outcome, but for algorithms.

In order to achieve one-shot learning (or close) we can rely on knowledge transfer, just like the human in the example would do (we are trained to be amazing at image processing, but here we would also exploit other knowledge like abstract reasoning abilities, and so on).

This brings us to transfer learning. Generally speaking, transfer learning is a machine learning paradigm where we train a model on one problem and then try to apply it to a different one (after some adjustments, as we'll see in a second).

In the example above, classifying apples and knives is not at all trivial. However, if we are given a neural network that already excels at image classification, with super-human results in over 1000 categories... perhaps it is easy to adapt this model to our specific apples vs knives situation.

This "adapting", those "adjustments", are essentially what we call fine-tuning. We could say that fine-tuning is the training required to adapt an already trained model to the new task. This is normally much less intensive than training from scratch, and many of the characteristics of the given model are retained.

Fine-tuning usually covers more steps. A typical pipeline in deep learning for computer vision would be this:

Get trained model (image classifier champion)
Note the head of our model does not match our needs (there's probably one output per category, and we only need two categories now!)
Swap the very last layer(s) of the model, so that the output matches our needs, but keeping the rest of the architecture and already trained parameters intact.
Train (fine-tune!) our model on images that are specific to our problem (only a few apples and knives in our silly example). We often only allow the last layers to learn at first, so they "catch up" with the rest of the model (in this case we talk about freezing and unfreezing and discriminative learning rates, but that's a bit beyond the question).

Note that some people may sometimes use fine-tuning as a synonym for transfer learning, so be careful about that!


What is the basic purpose of local search methods?

Hill climbing, Simulated annealing, genetic algorithms are different variants of local search algorithm. Hill Climbing use to find good solution for NP-hard problems. Simulated annealing was first used extensively to solve VLSI layout problems. It has been applied widely to factory scheduling and other large-scale optimization tasks. Genetic algorithm have had a wide-spread impact on optimization problems, such as circuit layout and job-shop scheduling.


Which model can I use for this problem with multiple inputs and outputs?

This depend on type of data you use.

Time sequence data

If the data advanced in time, a LSTM or similar RNN should be used. RNN calculate output through time. It works very good on time series data as it have a real sense of time. While CNN and MLP could work for time series data, it often don't work that well as different timestep of data is not defined.

Non- Time sequence data

According to you previous comment, the data seems to be of this kind. In the case of your data, a normal Multi-layer perceptron works well for this. The data is a direct mapping between the input and the output. If the input data is an image, use a CNN.

For example code in keras, see here:
You need the pandas module for this to work. Run pip install pandas to install pandas.
    from keras.models import Sequential
    from keras.utils import np_utils
    from keras.layers.core import Dense, Activation, Dropout

import pandas as pd
import numpy as np

# Read data
train = pd.read_csv('../input/train.csv')
labels = train.ix[:,0].values.astype('int32')
X_train = (train.ix[:,1:].values).astype('float32')
X_test = (pd.read_csv('../input/test.csv').values).astype('float32')

# convert list of labels to binary class matrix
y_train = np_utils.to_categorical(labels)

# pre-processing: divide by max and substract mean
scale = np.max(X_train)
X_train /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_test -= mean

input_dim = X_train.shape[1]
nb_classes = y_train.shape[1]

# Here's a Deep Dumb MLP (DDMLP)
model = Sequential()
model.add(Dense(128, input_dim=input_dim))
model.add(Activation('relu'))
model.add(Dropout(0.15))
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.15))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

# we'll use categorical xent for the loss, and RMSprop as the optimizer
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

print("Training...")
model.fit(X_train, y_train, nb_epoch=10, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=2)

print("Generating test predictions...")
preds = model.predict_classes(X_test, verbose=0)

def write_preds(preds, fname):
    pd.DataFrame({"ImageId": list(range(1,len(preds)+1)), "Label": preds}).to_csv(fname, index=False, header=True)

write_preds(preds, "keras-mlp.csv")

Code source: https://www.kaggle.com/fchollet/simple-deep-mlp-with-keras

In conclusion, in the case of our data, use a multi layer perceptron  should work. Hope kit can help you and have a nice day!


There are a few different ways to improve on your simple heuristic approach, but they mostly resolve to these three things:

Find a better heuristic. This could be done by calculating probabilities of results, or running loads of training simulations and somehow tuning the heuristic function.

Look-ahead search/planning. There are many possible search algorithms. Most rely on you being able to simulate the impact of future decisions before taking them.

Take account of more player knowledge. So far your simple heuristic does not take account of which cards have already been played (thus which values remain to be drawn).

Currently my AI is playing piecemeal with a very simple heuristic. I do not see how to improve this heuristic so I am wondering if it is possible to improve the performance by having a view over several turns for example. But I don't see how to simulate the next rounds since they will depend on the cards drawn.

I think the main conceptual barrier you have to improvements is how to account for the complex behaviour of probabilities for drawing specific useful cards. There are a few ways to do this, but I think the simplest would be some kind of rollout (simulated look ahead), which might lead to more sophisticated algorithm such as Monte Carlo Tree Search (MCTS).
Here's how a really simple variant might work:

For each possible choice of play in the game that you are currently looking at:

Simulate the remaining deck (shuffle a copy of the known remaining cards)

Play a simulation (a "rollout") to the end of game against the simulated deck using a simple heuristic (your current greedy choice version should be good as long as it is fast enough, but even random choices can work). Take note of the final score.

Repeat 1.1 and 1.2 as many times as you can afford to (given allowed decision time). Average the result and save it as a score for the choice of play being considered.

Instead of choosing the next play by your heuristic, choose the one that scores best out of all the simulations.

This statistical mean of samples works in a lot of cases because it avoids the complexity and time-consuming calculations that would be required to make a perfect decision analytically from probability theory. The important things it does in your case is look-ahead planning plus taking account of additional knowledge that the player has about the state of the game.
MCTS is like the above but nested so that the simulations are made from multiple starting points.
In terms of robustness, provided you run enough rollouts per decision to be confident about the mean scores, then it should be OK.


How do you perform a gradient based adversarial attack on an SVM based model?

A way to avoid computing the SVM loss by hand is to use a differentiable programming framework, such as JAX. These frameworks will automatically calculate gradients using automatic differentiation.

If you can write down the SVM loss using numpy operations then you can use the framework's tools to get a function which evaluates the gradient with respect to any argument.

In JAX this would look like:

import jax.numpy as np
def hinge_loss(x, y, theta):
    # x is an nxd matrix, y is an nx1 matrix
    y_hat = model(x, theta) # returns nx1 matrix, model parameters theta
    return np.maximum(0, 1 - y_hat * y)

hinge_loss_grad = grad(hinge_loss)
# hinge_loss_grad takes an x, y, theta and returns gradient of hinge loss wrt x


How to choosing the random value for parameter w in deep learning network?

Weights initialisation is strictly related to the vanishing/exploding gradient problem. For a complete explanation, please check this awesome page (also from deeplearning.ai). Here I'll summarise the main concepts:

initialising weight all to zero will cause all weights to have the same derivative value with respect to the loss function, hence the network would be incapable of learning anything.
initialising to zero the biases has no drawback since they are constants (no effect at all when computing the derivative anyway).
initialising weights with too high or too small values will lead to an exploding gradient (oscillating gradient values without convergence) or a vanishing gradient (small gradient values variation that converge before reaching the loss global minimum).

In order to avoid these problems, a method called Xavier initialisation has been proposed: the weights should be initialised in such a way that they will generate activations scores with a distribution that has:

Mean 0
Constant variance across layers (i.e. no vanishing/exploding)

The value "np.sqrt(layer_dims[l - 1])" pup up when imposing the second constrain.  For a formal prove check the page I linked. To grasp the concept just focus on the fact that the variance of the weights of a layer depends on the amount of nodes of the previous layer. This mean that for layers preceded by layers with a big amount of hidden nodes, the weights will be initialised with a small variance, and this is ok cause we don't want a small bunch nodes to have stronger influence on the subsequent activations. But in layers which instead are preceded by layers with a small amount of nodes, it's ok to allow the weights to vary more.


What's the purpose of layers without biases?

Bias is one of the hyperparameters in neural networks, which let you shift activation function. Disabling bias means setting bias to be zero.
Even though, in many cases, bias is a big help for successful learning, in some cases, you may want to add an extra constraint to your neural network in finding the objective function. For example, in the paper below, a zero-bias layer as the last layer helps increase the output's interpretability.
https://ieeexplore.ieee.org/abstract/document/9173537


Fitness Function altenatives in Genetic Algorithms for game AI

Is this a viable replacement to the Fitness function?

Sure, the fitness is 1 for the winner and 0 for the loser. You're using some kind of the Tournament selection.

It might be better to use more chromosomes and let A play against B, C, D... and define the fitness as the number of wins. Or not, as such an evaluation is more precise but also more time-consuming.

  ... Infinity ... Do I also need to add this score to the chromosome?

Why should you? The exact value doesn't matter (as it only needs to be big enough), so there's nothing to evolve there. You don't represent the number  of players either, right? Just use common sense.

  ... Would it still be okay to generate chromosomes randomly?

I guess so, but a distribution providing values closer to the expected outcome should be better. This depends on how you mutate them (adding a small random value won't bring you far, multiplying by 1 + small_random_value would).

Alternatively, you can generate values from some fixed interval and scale them up.

Your values are IMHO far too big. Whatever your 5 and 40000 mean, I guess, that 5 and 400 would work the same.


Why isn't conditional probability sufficient to describe causality?

Perhaps the shortest answer to this question is that Bayes' Theorem itself allows us to easily change the direction of a conditional probability:

$$
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
$$

So if you have $P(B|A)$, $P(A)$, and $P(B)$, we can determine $P(A|B)$, and similarly you can determine $P(B|A)$ from $P(A|B)$, $P(B)$ and $P(A)$. Just by looking at $P(B|A)$ and $P(A|B)$, it is therefore impossible to tell what the causal direction is (if any).

In fact, probabilistic inference usually works the other way round: When there is a known causal relation, say from diseases $A$ to symptoms $B$, we usually have $P(B|A)$, and are interested in the diagnostic reasoning task of determining $P(A|B)$ from that. (The only other thing we need for that is the prior probability $P(A)$ since $P(B)$ is just a normalization factor.)


What is the difference between exhaustive nearest neighbor search and k-nearest neighbour search?

The exhaustive nearest neighbor search performs an exhaustive search of the nearest neighbor (i.e. the closest image or matches, depending on your application). Here an exhaustive search means that you will compare your query image with any other image, or, in the case of feature matching, you will compare every feature of the query image with every other feature of the target/reference image.

The $k$-nearest neighbor search finds the $k$ closest images. You typically do that by first building a KD-tree with your query (or target) image, so that to speed up the search for the $k$-nearest neighbors of your query image.

(Btw, in case you are interested in the concept of feature matching, in OpenCV, the class that you can use to perform an exhaustive search for the matches between two images is the BFMatcher (which stands for Brute Force Matcher), while the class to perform the search with a KD-tree for the k-nearest neighbhours is FlannBasedMatcher).


Why are autonomous cars categorized as AI?

There is a neat definition of artificial intelligence, which circumvents the problem of defining "intelligence" and which I would ascribe to McCarthy, the founder of the field, although I can only find it now in this book by H. Simon:

"… having to do with finding ways to do intelligent tasks, to do tasks which, if they were done by human beings, would call for our human intelligence."

So, at its core we call the automation of every task AI, that can only be done by the human mind. At the time people thought that a computer able to play chess would also be intelligent in other ways. When this turned out to be false, the term AI was split into "narrow or weak AI", i.e. a program able to do one task of the human mind, and "general or strong AI", a program that can do all the tasks of the human mind.

Self-driving cars are narrow AI.

Note, that all these definitions don't specify whether these programs copy the way the human mind works or whether they come to the same result via completely different algorithms.


Can a model, retrained on images classified previously by itself, increase its accuracy?

The most likely outcome of this approach is wasted time and very little effect on accuracy.

There will be changes to the model. Some will be beneficial and improve the model, but some will backfire making it worse.

For instance:

The model predicts with probability 0.4 that an image is in a certain class. It is the highest prediction, and actually true. It will be added to the training dataset with a "ground truth" of probability 1.0, so on balance more and better data has been added to the data set. This will improve generalisation, as whatever caused the relatively low 0.4 value initially - e.g. a pose or lighting variation - will now be covered correctly in the training set.
The model predicts with probability 0.4 that an image is in a certain class. It is the highest prediction, and actually false. It will be added to the dataset with "ground truth" of probability 1.0 for the wrong class. This will weaken associations to the correct class for similar input images, meaning for exaple that a certain pose or lighting difference that is already causing problems for the model will be used to incorrectly classify images in future.

These two scenarios will occur, on average, at a rate determined by the model's current test accuracy. So if your current model is 90% accurate, 1 in 10 images in your new training data will be mislabelled. This will  "lock in" the current errors at the same rate on average as they already occur.

The effect may be a drift up or down in accuracy as the model will definitely change due to the new training data. However you have little to no control over how this drift effect goes if you are not willing or able to oversee the automatic classifications generated on new data by the model.

There are a few ways to get some improvement unsupervised from new data. For instance:

Build an autoencoder from the early convolutional layers of your model and train it to re-generate all inputs as outputs. This should help it learn important features of the variations in data that you are using. Once this training is done, discard the decoder part of the auto-encoder and add your classifier back in to fine tune it. This may help if you have only a small amount of labelled data, but a lot of unlabelled data.
Use a model that has better accuracy than yours to auto-label the data. This might seem a little chicken-and-egg, but you may be able to create such a model using ensemble techniques. The ensemble model could be too awkward to use in production, but may still be used in an auto-labeling pipeline to improve your training data.

Note you may get even better results simply ignoring the extra unlabeled data, and instead fine tuning a high quality ImageNet-trained model on the labeled data you already have - saving yourself a lot of effort. Depends on the nature of the images, and how much labeled data you are already working with.


Is it possible to generate "Karel the robot" programs with genetic programming?

A genetic algorithm is used to find an optimal solution to a problem. The parameters of the problem are encoded in the population, where each individual contains a solution to the problem in itself. Each individual is evaluated using a fitness function, and then the highest scoring individuals are used to create a new set of individuals by mixing up their parameters, and this new set replaces the worst scoring individuals from the previous generation. Over time, the average fitness of the population will (hopefully) increase, and a good solution be found. Essentially, the search space is explored in parallel through the many individuals in the population, which means that local maxima can probably be avoided.

The question now is: What is the problem you want to optimise?

The Karel language is fairly simple, and has few tokens. A program should be easily encodable in the right format, though genetic algorithms usually require individuals to have the same size (though shorter programs could be padded with no-ops). Recombining the best programs could mean swapping tokens. Maybe syntactic constraints could be factored into this to avoid crass syntactic errors. That should not be an insurmountable problem.

However, that still leaves the question of the fitness function. A basic criterion would be syntactic: if the program doesn't parse, it's not 'fit'. This could potentially wipe out a lot of offspring, and doesn't really help you much: it would produce more or less random programs which are syntactically well-formed.

What you really want is to evaluate the outcome of the programs. Like, which program can best navigate a maze. So you need to execute them and see what they produce. Then you run straight away into the halting problem: your little program might contain an endless loop and will never terminate. So you need to add a kind of time-out, or a limit on the steps the program can execute. This should be easy to do in a virtual machine.

So, to answer your question: Yes, it is possible to turn creating Karel programs into a problem that a genetic algorithm can be applied to. However, you need to be clear about what problem you want to solve. I don't quite understand what you mean by "learning from demonstration" or how you can achieve the aim of replicating human actions, but if you can somehow encode that in a fitness function, GAs are a tool you can use.


In what ways can financial markets be hacked? [Algorithmic Trading]

I don't have a proper source for this, as i've only read this from an online forum: everything i say is just hearsay and I am very uneducated on the subject. With that being said...

As you may know algorithmic trading relies on strategies, i.e. I trade a certain way once I see certain indexes move in a certain way. If your procedure is known by other people for some reason, then other adverse agents can release information or influence the market to manipulate your bot into making trades that the adverse agent then takes advantage over.

I'm not sure how this works in detail, and frankly I find it quite unlikely that a single adverse agent can target a single known strategy with the entire market in-between the two. But I've heard that a process known as the alpha algorithm can be used to discover hidden trading strategies. But again, I wasn't able to find any articles that directly linked this method towards financial trading. So my usual warning of me talking out my butt again applies.

Just a fun thought. A friend of mine told me (unfortunately, again I wasn't able to find a proper source for this) that Jane Street found that the most effective trading strategies were based on relatively simple ML techniques (I recall the specific example mentioned was just a common form of regression). The important follow question would be "What are their input metrics", I don't think I was told this, but I would believe that it would also just be something computed by relatively well-founded stock market indices. If this is the case, than again it would be very difficult for a single adverse agent to influence that particular bot since the force of the entire market is in between them.

On the opposite end, we have this redditor(finally, a source!), who amazingly analyzed about 130+ trading strategies, each of which were "big brained" in kind. This redditor concluded that all papers were essentially p-hacked, and none of the results were significant. Of course there is the possibility that authors with profitable results would not publish them, but the conclusion remains: keep your strategies simple. Again, this implies that it would be harder to "hack" a specific strategy used in the market by a particular individual.

P.S. There are other "big brained" methods that are kept well secret and thus difficult to analyze: I heard a rumour that a Chinese company paid people to take pictures inside shopping malls, to use to analyze shopping behavior, and how vibrant the economy is; I also hear that some people use satellites photos to look at, say, car dealerships, to see how much of their stuff is being purchased. Again its unclear if these techniques are profitable or not, but certainly a lot of money is being thrown into it.


Complete deep learning text classification with Python example

You may look at these github repositories :

Text classification in NLP with deep learning
Convolutional Neural Networks for Sentence Classification


Is A* with an admissible but inconsistent heuristic optimal?

It depends on what you mean by optimal.

A* will always find the optimal solution (that is, the algorithm is admissible) as long as the heuristic is admissible. (Note that the definition of admissible is overloaded and means something slightly different for an algorithm and a heuristic.)

If you talk about the set of nodes expanded by A*, then it expands the minimal set of nodes up to tie-breaking even with an inconsistent heuristic.

If you talk about the number of expansions, then A* is not optimal. A* can perform up to $2^N$ expansions of $N$ states with an inconsistent heuristic. This result comes from Martelli, 1977.

Algorithm B has worst-case $N^2$ expansions and budgeted graph search (BGS) has worst-case $N \log C$, where $C$ is the optimal solution cost.

You can see a demo that shows the worst-case performance of A* as well as the performance of BGS here:
https://www.movingai.com/SAS/INC/


Does reinforcing correct predictions increase model accuracy further?

Your suggestion is risky. It might make improvements to your classifier, but it may also reduce generalisation.

The two conflicting factors in play are:

Adding data points to a model which has capacity to learn more detail can improve its performance.
Training from a different distribution of data points than your target population can reduce its performance.

From the first point, you need your model to be able to accept the new data. This can be harder to achieve than you might think at first - if you have tuned some regularisation parameters by using cross-validation, then you may have at least in part saturated the model's capacity in order to prevent over-fitting. That means fitting to new data could require re-tuning your hyper-parameters as well - but you probably won't need to start from scratch, just search nearby.

It is hard to tell the impact from the second point. There is not going to be a general answer here, too much depends on how the data is arranged for your problem. My gut feeling is that you will notice an improvement to the model initially, but that there will be diminishing returns due to the self-selection of already-correct data points.

You should definitely keep back a cross-validation set of data distributed as it is in production, that has been properly and fully labelled in order to assess this work. This will be your way to assess whether generalisation is improving using your approach.

Worst case: You may need to go back and re-label the mistakes in order to significantly improve performance of your model. So I would suggest part of your auto-labelling process should store the misclassified items somewhere so you can re-visit them and spend the extra effort.

Sadly, even collecting more properly labelled data is not a guaranteed fix - some models already have enough data. You can check whether that might applies in your case by training with different sizes of training set (from the available training data you have so far), and seeing what the trend in performance is when you increase the training set size. In fact, you should probably do this first, before you invest significant effort in collecting more data.


Are the dialogs at Sophia's (the robot) appearings scripted?

Sophia uses ChatScript.

You can read about what ChatScript can do here.

  ChatScript keeps track of conversations with each user; can record
  where it is in a conversational flow and what facts it has learned
  about a user (you have to tell it what facts to try to learn). You can
  optionally keep logs of the conversations (either on a ChatScript
  server or locally on a freestanding device).


What are the scientific journals dedicated to artificial general intelligence?

There's also the journal Advances in Cognitive Systems. According to their website

Advances in Cognitive Systems (ISSN 2324-8416) publishes research articles, review papers, and essays on the computational study of human-level intelligence, integrated intelligent systems, cognitive architectures, and related topics. Research on cognitive systems is distinguished by a focus on high-level cognition, reliance on rich, structured representations, a systems-level perspective, use of heuristics to handle complexity, and incorporation of insights about human thinking. Advances in Cognitive Systems reviews submissions within approximately three months and publishes accepted papers on the journal Web site immediately upon receipt of final versions. Articles are distributed freely over the internet by the Cognitive Systems Foundation.


Why isn't my decision tree classifier able to solve the XOR problem properly?

I can reproduce this problem for an even more easily separable dataset:

The ideal tree for it should be as follows:

However, when I run DecisionTreeClassifier with the maximal depth = 2 in scikit-learn many times, it splits the dataset randomly and never gets it right.
This is an example of 4 different runs:

The problem is that scikit-learn has only two measures of the quality of a split: gini, and entropy. Both of them estimate mutual information between the target and only one predictor.
However, in XOR problem, mutual information of each predictor with the target is zero. You can read more about it here: link from which you can see that this problem exists not only for XOR but for any task where interaction between features is important.
In order to solve it, the tree should be built based neither on the Gini impurity, nor on the information gain but on measures that estimate how the target depends on multiple features, e.g. multivariate mutual information,  distance correlation, etc which might solve simple problems like XOR but might fail in the case of real tasks. It is easy to find simple cases when they fail (just try them for a regression for simple non-linear functions of a few variables). There is no such a measure that would estimate the dependence of a target on multiple interacting predictors very well and would work for all problems.
EDIT to answer Asher's comment: I did several runs for max_depth=3. It is better than for max_depth=2 but still misses the correct classification from time to time. Taking max_depth=4 almost always gets XOR correctly with the occasional misses. Below are pictures of some runs for max_depth=3 and max_depth=4.

However, the trees for max_depth=3 and max_depth=4 become ugly. They are ugly not only because they are bigger than the ideal tree shown above but they totally obscure the XOR function. For example, can you decipher an XOR from this tree?

It is probably possible with some pruning technique but still, an extra work.


Are winning and imitating the primary indicators of intelligence?

Winning

My feeling is that utility is the standard of determining intelligence, and merely connotes the strength of a decision making algorithm.

Winning is a metric involving strength in relation to other agents, but may not always result in optimal global utility.  (Here I'm thinking of a game AI that always wins.  In a game like Chess, a subset of serious players may appreciate this, but it's unclear that this would be any kind of fun for the average player.  Similarly, an unbeatable AI in a game like Civilization or Stellaris would be very poorly received by most players.)

In the paper Depth In Strategic Games they formalize the term "strategy ladder". A simple definition of this concept is a series of AIs that are "meaningfully weak".  Specifically, players learn to beat them and in so doing learn different levels of strategy, ascending up ladder.

In this example, user engagement is the ultimate goal, not the AI winning every game!

Imitation

Turing makes a compelling case that imitation is an indication of intelligence (self-awareness and "true comprehension" as strictly corollary.)

I like to reduce this idea to Captcha, because it involves utility.  At some point Captcha may no longer work because the average bot is a "smart" in solving them as the lowest-common denominator human.

The same goes for chat bot--it they can successfully imitate humans, they can be considered as smart as those humans in regard to chatting.

Conclusion: Both winning and imitation are indicators of intelligence, whether machine-based of biological.


What's the commercial usage of "image captioning"?

If "image captioning" is utilized to make a commercial product, what application fields will need this technique?

There are several important use case categories for image captioning, but most are components in larger systems, web traffic control strategies, SaaS, IaaS, IoT, and virtual reality systems, not as much for inclusion in downloadable applications or software sold as a product. These are a few examples.

Provision of captions to reduce the number of user operations (keying and selection) required to post a movie or image to improve posting volume of sticky media and therefore improve the position of a web site or page in terms of human interest
Provision of captions to provide HTML header and alt attribute content to improve search engine scoring of page for search terms related to the content of the movie or image
Testing to see if the content of an image submitted for posting matches the criteria intended by the owner or stake holder for the posting space on the basis of caption rather than directly through CNN categorization

The first two are usually monetized in ways such as these.

Improvements in online purchase daily volume

Acquisition of contact lists with consumer or business interests as fields in the list for marketing purposes

Additional draw of web traffic to enhance ad impression revenue

What is the level of required performance can this technique be usable?

Performance is in terms of these system qualities.

Cost of computing resources per caption generated
Accuracy of caption generated (which may be quantified in more than one way)
Reliability of generation (since the AI component should produce a rating of confidence with the caption string or produce a null or empty string based on a hyper-parameter confidence threshold

The quantification of cost, accuracy, and reliability is business dependent. Some may have a nasty negative effect on the business if the caption is wrong or missing. Others use cases may not.
In some cases the average revenue generated by the caption's presence is already known and known to be small, which requires that the CNN run time and computing resource requirements must be kept below that. In other cases, a Fortune 500 company CTO said, "Do it and send me the budget needed." In such cases the budget may be, for all practical purposes, unconstrained, not that a system that wastes time and resources is ever desirable, even if only for energy conservation reasons.


Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings?

Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.

One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model.

The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.


Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

Yes, OpenAI will release an API for GPT-3, so any developer can integrate it into their application. I don't believe the document for their API is public yet, so we don't know what the final interface will look like, but it's likely to be a simple REST API. In the future, I imagine your developers can take advantage of their API, or alternatively there will be community-made scripts for you to use/copy.
The pricing for using their API is explained here. Note that they charge per token, which might be important in case your game plans to make live calls to GPT-3 during gameplay (as opposed to mining a huge corpus of answers to build an offline database).
The use cases of GPT-3 suggests that you can legally use them for commercial products, although I couldn't find a definitive license or user agreement document.


It is correct that climate and economic models are distinct from waste models.

A memory based model is the correct approach because the time domain is key in prediction based on existing trend data.  However, the RNN is not a practical production model with others that have proven more productive.  Although this article is trendy and not accurate in every technical respect, it introduces the more effective models and their relationship to RNNs.  If you study the article and the terms it introduces on the web, the right solution and possibly an open source project that can get you started can be found.

Basis for Prediction of Urban Waste

Whether modelling solid or liquid waste, the following model is recommended by two consultants to several large European municipal clients.  All the variables and distributions are time dependent.

$v(t) = n(t) \; d\big(I(t), A(t), t\big) \; s(t) \; r(\vec{x(t)}, t)$

$v$ is the waste rate in metric tons per week
$n$ is the population of the collection region
$d$ is the average demographically based disposal rate in Kg per person – Income and age affect consumption and the waste products of consumption, affecting disposal.
$I$ is the distribution of income
$A$ is the distribution of age
$s$ is the dimensionless seasonal factor
$f$ is the dimensionless regional factor – The granularity of regions depends on variance considerations.
$\vec{x}$ is a vector of non-demographic factors that are likely to correlate with regional waste rates, such as number of university students, size of recycling receptacles, or number of fast food restaurants

Although some of the above factors can be combined in a single training process, there are advantages of leaving them as separate factors.

Demographics is measured at the municipal level and has its own model that can be trained based on immigration, emigration, births, deaths, and aging.  Proven models exist for this already.
Separation of training models allows for component wise system development and the understanding of relationships in data that would remain opaque if training was aggregated.

Each of the distributions can be obtained from municipal data. Labeled data for $n, d, s,$ and $r$ can be measured by implementing the appropriate municipal data collection processes.  If it is already available, that would be unusual, and cleaning the data (data sanitation) would be an important consideration.

Once labeled data is available for training and the artificial network design is selected, training can occur, followed by testing, followed by validation.  As the system is deployed, care would need to be taken to continuously refine the artificial network hyper-parameters for monthly, quarterly, or annual training and continue to improve data collection accuracy and reliability.

High accuracy and reliability in such prediction could be expected for short term.  Longer term prediction may have considerably lower accuracy and reliability because of the higher probably of the occurrence of new types of social and economic shift around which the model could not have been advanced trained.


Pixel-Level Detection of Each Object of the Same Class In an Image

What you want to do is instance segmentation on a pixel level.
I can point you two different way:

MaskRCNN, which in a nutshell, uses Region Proposal Networks then segment in the proposed regions.
Some other works which don't use Region Proposals rather, regress the pixels of the image in an embeding space to be able to apply some clustering algorithm. Here is a link to a good blog post about it: https://towardsdatascience.com/instance-embedding-instance-segmentation-without-proposals-31946a7c53e1


How do I perform object detection if there is only one type of object?

The hardest part is image annotation: here the difference between object recognition and object detection becomes important. If you just want to answer the question "does this image contain object X?", then you just need to provide as many images that contain object X as possible, together with as many images  that don't contain object X (but are otherwise similar). However if you want to answer the question "Where exactly object X is located in this image?" then you will need to manually provide a bounding box for each instance of object X in each image. Obviously, the second scenario is a lot more labor intensive.

After you've done this part, train either a binary image classifier (typically this will be a convolutional neural network) on your annotated images (split them into train and test partitions), or an object detector (googling "custom object detection" produces lots of code examples how to train it (e.g. https://towardsdatascience.com/tutorial-build-an-object-detection-system-using-yolo-9a930513643a start with Step 2B).


How can we compute the gradient of max pooling with overlapping regions?

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and $\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$.

That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.

So in the 1d case, if you have a max pool over the input params $[a_0, a_1, a_2, a_3, a_4]$ a max function $m_0 = max(a_0, a_1, a_2)$, $m_1 = max(a_2, a_3, a_4)$ which overlap at $a_2$, and gradients $\nabla_{\mathbf{m}} J = [\frac{\partial J}{\partial m_0}, \frac{\partial J}{\partial m_1}]$, then you would allocate those gradients to vector $\mathbf{a}$ according to which items in each group was the max of that group, adding them when they overlapped.

Examples:

If $\mathbf{a} = [3,0,1,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$

If $\mathbf{a} = [3,0,4,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$


Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input.  This is a trivial fact that applies to a single layer perceptron.  This is true for for the single layer case for any activation function, also a trivial fact.

When the layer number is between one and infinity (two or more), the theory bifurcates.  The identity function becomes a special case:  Any number of layers that conform to a first degree polynomial, $ax + b$, can be replaced with a single identity layers.  The alternative case, where there are multiple layers that functionally do not conform to a first degree polynomial, $ax + b$, cannot be replaced by a single layer of some equally simple function.  The complexity increases geometrically, which is the entire point of multilayer perceptrons.

Under particular constraints, the multilayer perceptron can produce a wide variety of functional behaviors that do not resemble the activation functions of the layers.

For instance, a properly trained network using sigmoid activation functions, with sufficient layer depth and sufficient massive allocation of computing resources, could theoretically approximate the topography of the Himalayas.


Is there a way of representing the minimax algorithm mathematically?

A simple google search gives plenty of results.

If you have a look at the entry in wikipedia for Minimax it has mathematical representations as well as some basic pseudocode and tree representations to help grasp the concept.

Proving it would be a matter of going through the regular methods of a mathematical proof and would probably be a bit complicated.  It is far easier to discuss this by way of graphs.


Is there an alternative to RNNs that doesn't require knowing input history?

You don't necessarily need to roll out the inputs to an RNN; doing so makes it easier to optimize computation (if the sequence length is the same each batch), but it's not a necessity. Furthermore, RNNs (and, incidentally, the brain) doesn't necessarily remember the input history as is; rather, the history is encoded via the RNN's cell state (or states, in the case of LSTMs and other RNN cell architectures with multiple states). Neural Turing Machines (NMTs) and Differential neural computers expand on that concept by also using a larger "memory" storage (in the form of a matrix).


What is the intuition behind the entropy formula used in the ID3 algorithm?

Please, take a look at Understanding Shannon's Entropy metric for Information.

The answer for the minus sign is in section 6. The probability logs are less than or equal to $0$, so the minus sign guarantees that information (entropy) is always greater than or equal to $0$.


Is it possible to use AI to denoise noised or 'dirty' documents?

It is possible remove noise from signal if what is considered noise is distinguishable from what is considered signal. This is true of documents, images, audio, or any other type of information.

If dirty means wrong content, then probably not. If dirty means mispellings, then probably yes. If dirty means tilted by a few degrees and containing edge artifacts around the border from scanning, then definitely yes.

Most OCR software, such as the open source program Tesseract, can produce fairly accurate text even with some of the above issues in the document.

If the documents have noise gradients, they may be difficult to OCR. That is different than a value or intensity gradient, which can be normalized. Using AI to remove noise gradients from documents is a reasonable plan, but there is a limit to any kind of intelligence. A scrambled egg cannot be reconstituted.

A simple test of feasibility is whether a human that has no domain knowledge about the subject of the document can make out the text. If they cannot, then a computer will not be able to either, unless the vocabulary is labeled with the probability of each word appearing in that subject matter and the OCR system is trained that way.

However, if the noise is on a gray scale so that the darkness of the type is greater than one or two standard deviations of the noise distribution, then the document can be divided by a parameterizable grid with parameterizable brightness and contrast. This can become the model used in conjunction with a basic feed forward network (mini-batched multilayer perceptron) and the OCR software. The idea in this architecture would be to converge on a network behavior that, for the documents in the example set that have been hand adjusted so that the OCR produces good output, could then be used for many more documents with similar noise gradients.

The inputs for the artificial network would be statistics about the value distribution for specific sections on a specific page of the document and the labels will be the manually created instructions for gradient based contract and value correction of the document so the OCR proceeds with few errors. Once the network is trained, its output could ideally drive the gradient based corrections. It would probably work if three things are in order.

The noise doesn't completely destroy the availability of the text so that an average human could not make out the words.
The training of the network converges to a degree that the OCR software works across the pages for most of the documents.
The characteristics of the noise gradients remain consistent, meaning that the training examples are representative of the documents that will be processed later.


The idea behind this kind of reasoning is that there is a "true" distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution.   Let's see a simple example to illustrate the point.
Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set $\{1, 2,\dots6\}$. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set $\{1, 2,\dots6\}$ (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.
In your case what happened is the following:
$x_1,\dots, x_m$ is your sample (in the example above, the $60$ numbers that you see as your friend throws the die). This sample defines a distribution, $\hat{p}_{data}$. In the example above, $\hat{p}_{data}$ would likely be close to the uniform distribution over $\{1, \dots, 6\}$. Now you can think about the sum over $x_1\dots, x_m$ of $\log(p_{model}(x^{(i)};\theta))$ as the average of $\log(p_{model}(x;\theta))$, where $x$ is drawn according to $\hat{p}_{data}$.
Let me make another example with some actual numbers. Let's say you toss a fair coin $7$ times and you see $$\{H, H, H, T, H, T, H\}.$$
The empirical distribution is $\mathbb{P}(H)=5/7$, $\mathbb{P}(T)=2/7$. So if you compute the expectation $$\mathbb{E}_{x \sim \hat{p}_{data}}[\log(p_{model}(x;\theta)]$$ you get
$$\log(p_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p_{model}(H;\theta) +\frac{2}{7}\log(p_{model}(T;\theta)),$$
which is what you will get if you compute the first sum you wrote.


How is Monte Carlo different from model-based methods?

Model-based methods (such as value or policy iteration) use a model of the environment, which is usually represented as a Markov decision process. More specifically, the model consists of the transition and reward functions of the Markov decision process, which should represent the dynamics of the environment. For example, in policy iteration, the rewards (used to estimate the policy or value functions) are not the result of the interaction with the environment but given by the MDP (the model of the environment), so the decisions are made according to the reward function (and the transition function) of the MDP that represents the dynamics of the environment. Model-based methods are not (usually) dependent on past actions. For example, policy iteration converges to the optimal policy independently of the initial values of the states, the initial policy or the order of iteration through the states.

Monte Carlo methods do not use such a model (the MDP), even though the assumption that the environment can be represented as an MDP is (often implicitly) made (and the MDP might actually be available). In the case of Monte Carlo methods, all estimates are solely based on the interaction with the environment. In general, Monte Carlo methods are based on sampling (or random) operations. In the case of reinforcement learning, they sample the environment. The samples are the rewards that are obtained when certain actions are taken from certain states.


Video Analysis: Providing a success score for a of a student carrying out a specific task

You have not described exactly what the tasks will be, but there are some open source libraries for real time pose tracking. For example, OpenPose is one that can be configured to track the body, the hands and the face. However, this is only going to give you predicted pose information for each frame. If the subjects are meant to be doing specific tasks, e.g. picking up objects and moving them, OpenPose won't help and you may need to do further image recognition/analysis or object tracking in order to interpret what is going on.


What is the difference between a non-stationary policy and a state that stores time?

So, here's is the question: Is it true that a non-stationary policy must satisfy this condition?

  $$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$

With your custom notation (which certainly isn't common, but seems reasonable)... I assume you meant to say that a stationary policy must satisfy that condition, rather than that a non-stationary policy must satisfy that condition. In that case, yeah, that seems correct to me. A stationary policy would satisfy that condition, and a non-stationary one wouldn't.

Wrapping back to the more usual notation, where $\pi(S, A)$ denotes a probability of selecting $A$ in $S$ (which also still covers the case of a deterministic policy, which would simply assign a probability of $1$ to a single action, and $0$ to all others)... I think it's still interesting to consider the case where we decide to "bake" a time-step counter into the state representation.

With this notation, for two different time steps $t$ and $t'$, such that $t \neq t'$, I'd say that a policy $\pi$ is stationary if and only if:

$$\pi(S_t, A_t) = \pi(S_{t'}, A_{t'}) \text{ if } S_t = S_{t'} \wedge A_t = A_{t'}.$$

Note that if we decide to include $t$ in the state-representation, the case that $S_t = S_{t'}$ with $t \neq t'$ will actually never hold within the same episode, states at different time steps will always automatically be different from each other if the time step is one of the "features" encoded in the state. So, within a single episode a policy with a "time-aware" state representation will always automatically be stationary because there cannot be any repeating states. Of course, if you start looking across multiple different episodes, this can change; this is what you're doing when you write:

  Once we reset the environment to evaluate the policy again, the turn, which we store in the state, will go back to 0, but the background number of evaluations won't reset and it will be 3.

If you chose to also embed that "episode counter" into the state representation, you would also no longer have any state repetitions at all anymore across different episodes (I don't think doing this would ever be a good idea though).


What are the available exploration strategies for continuous action space scenarios in RL?

I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):

The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:

Asynchronous Methods for Deep Reinforcement Learning mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).
Parameter Space Noise for Exploration describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.
DeepMind Control Suite is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.

Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. Here is the list of accepted papers at this workshop. Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.


Label arrangement for custom Keras image generator

I am not allowed to comment, so I am adding it here. Not sure I understand, are you asking if the training data must be ordered?

If so, training data must be random, and not AaBbCc or ABCabc.
Here the input to the NN will be the source image, and a category 90 degrees or 270 degrees as too separate inputs. i.e. inputs [0...k][k+1][k+2], where 0 - k are normalised pixel data, k+1 is 1 for 90 degree rotation else 0, and k+2 is 1 for 270 degree rotation else 0. Alternatively for more rotation options lets make a = input [k+1] and b = input [k+2]. If a = 0 and b = 0 then 0 degrees, if a = 0 and b = 1 then 90 degrees, if a = 1 and b = 0 then 180 degrees if a = 1 and b = 1 then 270 degrees.

From your image set, draw 3 samples. Sample 1 [training set] must be say 50% of the images, sample 2 [validation set] say 30% of the images and sample 3 [out of sample set] 20% of the images.

Use the training images from sample 1 to update the weights. Read in images from sample 1 at random, the rotation must be random, use your image rotator to rotate the image, and compare it to the output of the neural network, calculate the sum of the mean error and perform gradient descent to update the weights. Repeat the process. After every x weight updates randomly select a few images from the validation set and calculate an error value. If the average error value of the validation images are above some threshold then stop, or if the error begins to get worse. Do not use the validation set to update the weights

At the end use the out-of-sample set of images to verify the performance of the neural network.

The process above does not include hyper-parameter optimisation, or any other optimisation techniques, but does describe the basic way of training a NN using supervised learning.

Edit: I assumed that you wanted to NN to learn to rotate the image. After your updates, I assume you want rotated images to increase generalisation of your NN. In which case randomising or shuffling the input batch is preferred. i.e. send your images to Keras to rotate them, get the batch back any order is fine ABab or AaBb and then shuffle them. Pass the shuffled training and validation set to the NN and compare it to the known label for weight updates.


How to improve neural network training against a large data set of points with varying magnitude

I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:

1) Use the convolutional neural network for this dataset.

2) You can use huber loss instead of squared loss and see what happens.

3) See if you have enough small magnitude training data.

Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).


Does Gödel's second incompleteness theorem put a limitation on artificial intelligence systems?

I think the colloquial understanding of Gödel's incompleteness theorems allows them to be too broadly applied. Gödel's second incompleteness regards the consistency of a formal system, which is a technical concept of formal systems that means the system cannot prove every formula. It is commonly framed as a system not being able to prove both a formula and its negation (e.g. $2+2=4$ and $2+2 \neq 4$), since many logical systems allow anything to be proven from a contradiction.
The second incompleteness theorem states that if a consistent formal system is expressive enough to encode basic arithmetic (Peano arithmetic), then that system cannot prove its own consistency. This implies that we must use a stronger system B to prove the consistency of A. The system needs to be able to represent arithmetic because that is what is used to define the representability conditions of Gödel's proof that allowed him to formally construct the self-referential formulae central to the incompleteness theorems.
Here I diverge with my own opinion on this, I feel the concept of consistency in formal systems has no obvious bearing on the limits of artificial intelligence. An intelligent agent need not know anything about formal consistency to reach its level of intelligence -- the vast majority of humans have never encountered this concept, and yet they are still intelligent. Even many mathematicians don't give it a second thought unless they are in the trenches of mathematical logic. One would have to take an overly narrow view of artificial intelligence to allow Gödel's second incompleteness to serve as a limitation to it.
I caution against the popular informal restatements of Gödel's incompleteness theorems. These theorems were undoubtedly earth-shattering in the study of foundational mathematics and still have grand implications today, but projecting those results too far away from their rigorous origins is going to lead to many stray conclusions.


Why is the state value function sufficient to determine the policy if a model is available?

With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

As soon as I think about continuous state and action spaces with stochastic environment dynamics, computing the $\text{argmax}$ seems to be become very complicated and impractical.

For stochastic dynamics the calculations would be more complex, but will often be quite tractable. It depends on size of the distribution, and ease of calculating probabilities to make the correct weighted sums. Instead of $|\mathcal{A}(s)|$ calls to $Q(s,*)$ you will need to make roughly $|\mathcal{S'}(s)| \times |\mathcal{A}(s)|$ calls to $V(s')$ where $\mathcal{S'}(s) \in \mathcal{S}$ are all possible states that might result from the starting state. In the worst case this is $|\mathcal{S}| \times |\mathcal{A}|$
Despite the additional work here, you have been more efficient earlier. By only evaluating $V(s)$, you have removed the action dimension, which otherwise splits up your estimates. Your value function estimates will, all else being equal, converge faster because of this. So this is sometimes a compromise you might be willing to make.
For continuous states, this might still be practical, if both action choices and possible transitions are still discrete.
Once either action space is continuous, or transitions are continuous over some probability density function, then finding the maximising action via the model becomes impractical.


What can be deduced about the "algorithm" of backpropagation/gradient descent?

There are 3 separate issues that are often confounded in Deep Learning and Neuroscience:

Deep Learning is inspired by the way the biological brain works.
Deep Learning is how the biological brain works.
Deep Learning can model how the biological brain works.

Number 1 is accurate. The brain has many layers and many connections. Those principles have informed Deep Learning models.

Number 2 has little evidence to support that claim. The biological brain learns at the cellular level in very different ways than how any Deep Learnings system learns.

Number 3 is a current topic of research. Deep Learning is very good at learning patterns. There are good reasons to believe that Deep Learning can learn patterns in the brain. However, those Deep Learning models will not automatically give insight into the biological processes of the brain.

The video is an example of #1. Inspired by our current understanding of neurobiology, let's build better Deep Learning algorithms. These new algorithms might perform better on machine learning benchmarks. However, these algorithms are not better models of the biological brain. In order to understand the algorithms, the language of biology might not be helpful. It might be better to describe them mathematically.