In style transfer, why does the comparison between channels give a good sense of style?

When the original Neural Transfer paper was published, your question stayed unanswered for a while.  The reason why the Gram matrix represents artistic style was not entirely clear. A satisfactory (in my opinion) explanation came with the "Demystifying Neural Style Transfer" paper.
The basic idea is that you cannot just directly compare activations for two images. The spatial positions of various features are different for the source and target image, so you should somehow get rid of positional information of the activations and compare their distributions across the whole image. The goal of the style transfer task is thus to make the two distributions as close as possible. One of the possible measures of distance between two distributions $P$ and $Q$ is the Maximum Mean Distance (MMD):
$$\text{MMD}^2(P,Q) = \left\Vert \mathbb{E}_{P}[\phi(X)] - \mathbb{E}_{Q}[\phi(Y)]\right\Vert^2$$
With $\phi(\cdot)$ being a feature function - in our case it would be the NN activations in a particular layer. The next step would be to apply a "kernel trick" to the MMD, representing it through a kernel $k(x,y)$
$$\text{MMD}^2(P,Q) = \mathbb{E}[k(x_i,x_j)] + \mathbb{E}[k(y_i,y_j)]  - 2 \mathbb{E}[k(x_i,y_j)]$$
The Gram matrix of the original style transfer corresponds to the squared dot-product kernel $k(x,y) = (x^Ty)^2$:
$$\text{MMD}^2(P,Q) =  \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \left\Vert G^x - G^y\right\Vert_F^2 $$
Where $G^x$ and $G^y$ are the Gram matrices for activations and $\Vert\cdot\Vert_F$ is the Frobenius norm. The last equality gives a bit of trouble to those not used to the kernel trick, so I'll expand it (assuming Einstein summation convention):
$$\begin{array}{l}
 \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \\
 = x_{ik}x_{jk}x_{im}x_{jm} + y_{ik}y_{jk}y_{im}y_{jm} - 2  x_{ik}x_{jk}y_{im}y_{jm} \\
 = (x_{ik}x_{im} - y_{ik}y_{im})(x_{jk}x_{jm} - y_{jk}y_{jm}) \\
 = (G^x_{km} - G^y_{km})(G^x_{km} - G^y_{km}) \\
 = \left\Vert G^x - G^y\right\Vert_F^2
\end{array}
$$
The authors of the 1701.01036 also tried different kernel functions $k(x,y)$ getting more interesting style transfer results.


Can some one help me understand this paragraph from Nvidia's progressive gan paper?

A discriminator overshooting may result in a dataset that has not been thoroughly clean and probably has too many identical feature, as a result there will be an early convergence from the discriminator as there is little variation.
The drawback from this is that the model will not bee able to generalize well.


An image of a natural domain refers to any kind of image that has some variance in its structure, so it does not always present the same peculiarities.

For example, in a digits classification task, you can train your network on the MNIST dataset, that is a simple black & white set of images that are always well defined: the digit is in darker pixels, and the background is always white, they just have different shapes.

For the same task but in the natural image domain, the digits may vary in color, noise, and even position inside of a single image. As you may imagine this does not apply to digits only, but it includes any image that was not specifically created or modified to be fed into a neural network.

So, to put it simply, a neural network trained for natural images is able to perform classification in any real-life situation or scenario, without the need to edit the image to adapt it to what the network was trained for (for example, changing the color or cropping so the object you want to recognize is exactly in the center).

Here a similar question that may fill some doubts you may have.


What are some references that describe known filters (or kernels) and how we can create new ones?

I'd suggest you better understand edge detectors such as Robert or Sobel operators first to understand better how convolution operation on images extract features by constant value kernels.

Would personally recommend Gonzales and Woods for this,  as it gives a pure mathematical explanation to how and why these features are extracted.

Essentially the convolution kernels used in CNN's are ones with a learned set of values for the kernel.

For a better understanding of learned convolution kernels and, quite frankly, any idea under deep learning would easily recommend Deep Learning by Goodfellow et al


With gradient descent w/MSE on a regression, must/should every Epoch use the exact same training samples?

Your goal in regression should be to obtain the factors which result in the best fit model without over-fitting.  The more data you have in the training set, the better your regression will be.  Thus you would want to train on the most data, but you also want to have some data held out to validate that your model is not over-fit.  So this is where you should have your data split into say a 80/20 training and validation set.  And if data is scarce or you want that 20% to contribute to the model then you could do a 5 fold cross validation.

In the spirit of research perhaps you should try both of these routes, and report your findings.


Multi Armed Bandits with large number of arms

Without any knowledge on the references you came across, I am assuming that the authors were considering common applications of MAB (planning, online learning, etc.) for which the time horizon is usually small. In such applications, we usually cannot afford a large average regret which is inevitable for standard MAB algorithms due to the $\sqrt{|K|}$ factor.
Depending on the application or additional constraints you can impose on your problem, there are several works that consider structured stochastic MABs in which there are much better guarantees than the pessimistic $\sqrt{K}$ bounds. One variant of structured MABs is learning on graphs [1], where one can obtain regret bounds growing with $\sqrt{\beta(G)}$, where $\beta(G)$ is the independence number of a graph $G$.
[1] F. Liu, Z. Zheng, N. Shroff, "Analysis of Thompson Sampling for Graphical Bandits Without the Graphs", ArXiv. 2018.


Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

Derivative of the sigmoid curve is 0 when the output is 0 or 1 as you can see from the image above. The technique you are referring to is called label-smoothing which is used in various applications (e.g. GANs) but I can see how it would be applicable here also, by helping to avoid 0-gradients saturating learning.

To answer your third question, sigmoids are rarely used in the intermediate layers of networks these days. They have been replaced by ReLUs (or variants of ReLU) for this exact reason - sigmoids are prone to causing 'vanishing gradients' where the gradient becomes 0 and therefore does not get backpropagated any further. ReLUs alleviate this problem by always providing a gradient of 1 for positive input values.


A neural network with 2 or more hidden layers is a DNN?

I don't think there is a fixed threshold that differentiates between Shallow and Deep Learning, but I would say that a 2 layer NN should not be considered deep. But now-a-days, almost all NN architectures are studied under the umbrella of Deep Learning.
And yes, training a 2 hidden layers NN using Q-learning would technically mean doing deep RL.
I guess it is conceptually simple but making NN perform optimally is an art. Tuning hyperparameters or debugging NN can be tough and one learns with experience. I guess others in the community would be much more suited to answer this question. But these were my 2 cents.


Is creating dataset only by augmentation a bad practice?

Data augmentation is usually rotating, cropping and translating images. And this makes sense if your network could meet these kind of images.
If I take a digit recognition like LeNet, it is useless to complicate the task of the network by forcing it to learn rotated digits, which could lead to a more complex architecture and training and less accuracy in the task. Another example I could think of is human pose recognition (openpose project). As we humans usually stands with our feet on the bottom of the image and the head on top, openpose project didn't use rotation of images on the dataset.
So I would say data augmentation is a great tool (especially when we lack data), but I would only use it when the augmented data could be met when performing the task. If the digits are always oriented and placed on the middle of the image, it doesn't make much sense to use translations and rotations on the data set to augment it unless we really lack data.
In your example, it does make sense to me to create the dataset using data augmentation.


Is it possible to implement Asimov's Three Laws of Robotics?

The most challenging part is this section of the first law:

  or through inaction allow a human being to be harmed

Humans manage to injure themselves unintentionally in all kinds of ways all the time. A robot strictly following that law would have to spend all its time saving people from their own clumsiness and would probably never get any useful work done. An AI unable to physically move wouldn't have to run around, but it would still have to think of ways to stop all accidents it could imagine.

Anyway, fully implementing those laws would require very advanced recognition and cognition. (How do you know that industrial machine over there is about to let off a cloud of burning hot steam onto that child who wandered into the factory?) Figuring out whether a human would end up harmed after a given action through some sequence of events becomes an exceptionally challenging problem very quickly.


What are the mathematical prerequisites to be able to study artificial general intelligence?

I always recommend starting with game theory, combinatorial game theory, and algorithmic combinatorial game theory, (but I'm potentially biased;)

Combinatorics is a given--discrete mathematics is heavily utilized in computer science, and, with the advent of Combinatorial Game Theory (CGT), ability to determine if a given choice can be deemed optimal ("perfect play").  CGT arises out of traditional Game Theory, which we sometimes term "economic game theory" to make the distinction.  Out of Game Theory also arises subfields such as Evolutionary Game Theory, which is important in AI.

These fields relate to rationality, which is the basis for optimized decision making. Decision making algorithms seems to be the fundamental distinction of what constitutes an Artificial Intelligence.

From minimax to gametrees, it's probably a good idea to have a basic grounding in these fields, even if the problem you're AI is trying to solve isn't formally defined as a game.

All problems, from a fundamental standpoint, can be regarded either as puzzles--non-competitive context--or games--competitive context.  This distinction is based on whether there is a single agent (puzzles) or multiple agents (games.)


Are there neural networks with very few nodes that decently solve non-trivial problems?

Even if it’s impossible to answer this question properly, as non trivial is not well defined (maybe the author will edit this questions later, to specify it better), I take the opportunity to point out this paper which looks interesting to me

Smallest Neural Network to Learn the Ising Criticality

Assuming you have a general idea of the Ising Model I think the problem of identifying the critical temperature from a data driven perspective can be considered as non trivial and the paper shows how the authors have improved the performance related to solve this task with NN passing from 100 Hidden Neurons, as performed in this paper Machine learning phases of matter from 2017, to only 2 Hidden Neurons

Just my cents:

reducing the neurons, while keeping good performance, should help in terms of neural processing interpretability which is notoriously obscure and its complexity grows (exponentially) with the number of neurons


Integration of Sentiment analysis in CRM

I got one paper on the integration of opinion mining in CRM, which helps to get an idea about the process involved.


Are there emergent models of time in robots?

To my knowledge, this is very  much an open research issue.

Here is a paper by Prof Leslie Smith, an acknowledged expert on neuromorphic perceptual coding, which explains the importance of the notion of perceptual time for Artificial General Intelligence and sketches an architecture from which a notion of 'now' might emerge: Perceptual Time.


Does using the softmax function in Q learning not defeat the purpose of Q learning?

However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be chosen per game state, and as such, only one neuron should be active.

You are describing a network that approximates as policy function, $\pi(a|s)$, for a discrete set of actions.

  It also seems to me that would work well with the gradient update, and the network would learn appropriately.

Yes there are ways to do this, based on the Policy Gradient Theorem. If you read it you will probably discover this is more complex to understand than you first thought, the problem being that the agent is never directly told what the "best" action is in order to simply learn in a supervised manner. Instead, it has to be inferred from rewards observed whilst acting. This is a bit harder to figure out than the Q learning update rules which are just sampling from the Bellman optimality equation.

You can split Reinforcement Learning methods broadly into value-based methods and policy gradient methods. Q learning is a value-based method, whilst REINFORCE is a basic policy gradient method. It is also common to use a value based method within a policy gradient method in order to help estimate likely future return used to drive the polcy gradient updates - this combination is called Actor-Critic where the actor learns a policy function $\pi(a|s)$ and the critic learns a value function e.g. $V(s)$.

  But the big problem here is, this is no longer Q learning. The network no longer predicts the reward for each possible move, it now predicts which move is likely to give the greatest reward.

This is true, but it is not a big problem. The main issue is that policy gradient methods are more complex than value based methods. They may or may not be more effective, it depends on the environment you are tryng to create an optimal agent for.

  Is the softmax function used in Q learning at all?

I cannot think of any non-contrived environment in which this function would be useful for an action value approximation.

However, it is possible to use a variant of softmax to create a behaviour policy for Q learning. This uses a temperature hyperparameter $T$ to weight the Q values, and provide a probability of selecting an action, as follows

$$\pi(a_i|s) = \frac{e^{Q(s,a_i)/T}}{\sum_j e^{Q(s,a_j)/T}}$$

when $T$ is high all the probabilities of actions will be similar, when it is low even a small difference in $Q(s,a_i)$ will make a big difference to probability of selecting action $a_i$. This is quite a nice distribution for exploring whilst avoiding previously bad decisions. It will tend to focus the agent on exploring differences between similarly high rated actions. The main issue with it is that it introduces hyperparameters for deciding starting $T$, ending $T$ and how to move between them.


How does implementation of VAE's objective function equate to ELBO?

I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by applying its definition, you will end up with a different but equivalent version of the ELBO). I will use the most common (and definitely most intuitive, at least to me) one, which you can find in the paper that originally introduced the VAE, which you should use as a reference, when you are confused (although that paper may require at least 2 readings before you fully understand it).
Here's the most common form of the ELBO (for the VAE), which immediately explains the common VAE implementations (including yours):
$$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=
\underbrace{
\color{red}{
-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)
}
}_{\text{KL divergence}}
+
\underbrace{
\color{green}{
\mathbb{E}_{q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\color{blue}{ \log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]
}
}_{\text{Expected log-likelihood}},
$$
where

$p_{\boldsymbol{\theta}}(\mathbf{z})$ is the prior over $\mathbf{z}$ (i.e. the latent variable); in practice, the prior is not actually parametrized by $\boldsymbol{\theta}$ (i.e. it's just a Gaussian with mean zero and variance one, or whatever, depending on your assumptions about $\mathbf{z}$!), but in the paper they assume that $\mathbf{z}$ depends on $\boldsymbol{\theta}$ (see figure 1).

$q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$ is the encoder parametrized by $\boldsymbol{\phi}$

$p_{\boldsymbol{\theta}}$ is the decoder, parametrized by $\boldsymbol{\theta}$

If you assume that $p_{\boldsymbol{\theta}}(\mathbf{z})$ and $q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) $ are Gaussian distributions, then it turns out that the KL divergence has an analytical form, which is also derived in the VAE paper (appendix B)
$$
\color{red}{
-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)}
=
\color{red}{
\frac{1}{2} \sum_{j=1}^{J}\left(1+\log \left(\left(\sigma_{j}\right)^{2}\right)-\left(\mu_{j}\right)^{2}-\left(\sigma_{j}\right)^{2}\right)
}
$$
Hence this implementation, which should be equivalent to your term $0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma))$
The expected log-likelihood is actually an expectation, so you cannot compute it exactly, in general. Hence, you can approximate it with Monte Carlo sampling (aka sampling averages: remember the law of large numbers?). More concretely, if you assume that you have a Bernoulli likelihood, i.e. $p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)$ is a Bernoulli, then its definitions is (again from the VAE paper, Appendix C.1)
$$
\color{blue}{ \log p(\mathbf{x} \mid \mathbf{z})}= \color{blue}{ \sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) }\tag{1}\label{1},
$$
where $\mathbf{y}$ is the output of the decoder (i.e. the reconstruction/generation of the original input).
This formula should be very familiar to you if you are familiar with the cross-entropy. In fact, minimizing the cross-entropy is equivalent to maximizing the log-likelihood (this may still be confusing because of the flipped signs in the ELBO above, but just remember that maximizing a function is equivalent to minimizing its negative!). Hence this loss.
To answer/address some of your questions/doubts directly

and how $(0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma)))$ equates to the entropy?

I answered this above. That's just the analytical expression for the KL divergence (by the way, the KL divergence is also known as relative entropy: lol?): yes, but resolve me the logs and exponentials, though!

Still seems a bit off cause you're getting the cross-entropy of $Y$ with its reconstruction, not the Gaussian distribution for learning the latent $Z$.

As you can see from the definition of the ELBO above (from the VAE paper), the expected log-likelihood is, as the name suggests, an expectation, with respect to the encoder (i.e. the Gaussian, in case you choose a Gaussian). However, the equivalence is between the log-likelihood (which is the term inside the expectation) and the cross-entropy, i.e. once you have sampled from the encoder, you just need to compute the term inside the expectation (i.e. the cross-entropy). Your term $CrossEntropy(\hat{Y}, Y)$ represents the CE but after you have sampled a latent variable from the encoder (or Gaussian), otherwise, you could not have obtained the reconstruction $\hat{Y}$ (i.e. the reconstruction depends on this latent variable, see figure 1).
In equation \ref{1}, note that there is no expectation. In fact, in the implementations, you may just sample once from the encoder, and then immediately compute the ELBO. I have seen this also in the implementations of Bayesian neural networks (basically, normal neural networks with the same principles of the VAE). However, in principle, you could sample multiple times from the Gaussian encoder, compute \ref{1} multiple times, then average it, to compute a better approximation of the expected log-likelihood.
Hopefully, some of the information in this answer is clear. Honestly, I don't have much time to write a better answer now (maybe I will come back later). In any case, I think you can find all answers to your questions/doubts in the VAE paper (although, as I said, you may need to read it at least twice to understand it).
By the way, the simplest/cleanest implementation of the VAE that I have found so far is this one.


How to improve testing accuracy when training accuracy is high?

I think sometimes it can also help to examine your test and training sets.
Fundamentally, your data was produced by an underlying process/system that has certain properties. The system can have many "states" and all the possible states form the state space. If you have really tried things like dropout and regularization, my guess would be that the test set is somehow different from your train set. It is possible that your training set only takes samples from one part of the state space (AKA, your samples might all be similar in the training set and the test set has different samples - imagine you are classifying humans and all of your training samples have a class label of 1 meaning all the training samples have humans in them -> and all your test samples have no humans in them Good luck with that!).  Some questions to ask:

Are you combining datasets from different sources? If so:
   If you have "n" sources of data, you need to make sure that your training set has many samples from each of the "n" sources of data and your test set has samples from each of the "n" sources.
Are you shuffling your data enough and randomly putting samples in both the training and test sets? This relates to the human example I gave, make sure your training set has a little bit of everything (different combinations of inputs and/or outputs) and your testing set has a little bit of everything (different combinations of inputs and/or outputs).


Are probabilistic models dead ends in AI?

When considering effective approaches to AGI, one must extrapolate outwards to the types of modelling (and therefore inputs) that would be necessary to achieve any general utility. One consideration might be the fundamental "building blocks" of our physical world, and understanding the movements of these, can lead to accurate predictions of (all) occurrences. These fundamental elements are called (generally) subatomic particles and are anything but discrete values. In quantum field theory, the more accurate you are able to measure position, the less accurate you may know a quarks momentum (and vice versa). Our world, at the most fundamental layer, is probabilistic when observed. This is all not to say that an understanding of quantum mechanical kinematic descriptions is the only methodology to achieve true AGI, but to say probabilistic models, and therefore uncertainty, is a dead-end seems radically inaccurate.

That said, Dr.Minsky didn't really feel probabilistic models were dead ends. The emerging view in the field, one that Dr.Minsky urged for years, is that connectionism alone couldn't exclusively lead to AGI due to its uniformed structure. If you are unaware, connectionism is the concept of creating models around discrete units of representation (neurones in a neural net for example). You see the problem we have identified isn't that probabilistic models are inaccurate, it's that our current approach doesn't express the biological realism necessary for AGI (although sufficient for specific intelligence).

[I briefly worked with Dr.Minsky at the AI Lab before his passing last year, a hilarious man and brilliant scientist.]


Detecting abnormalities in x-rays while taking into account demographics of a patient -automated

First of all you don't need "text" input as in Christopher Bonnett's blog. Your case is more easy - demographic is table data, which can be expressed as vector of numeric values. This data should be processed - pushed through one or two fully connected layers. The trick is where, to what part of yolo to concatenate results of processing of this vector. Because it's vector data it should concatenated to fully connected layer. Where exactly should be found by experiments, but as starting point it could be concatenated to before-last (before output) fully connected layer (I think for yolo it's 4096-size layer).

Overall I'd say that is not a trivial task. It require some experience with deep learning, good understanding of yolo design, yolo algorithm and a lot of experimentation, both with architecture and hyperparameters. Probably worth solid paper. Good luck.


Are we supposed to "decide" ethics for intelligent systems?

Nonhuman ethics can be totally arbitrary

Regarding your citation, perhaps it's useful to consider why should anything that philosophers have said about human ethics apply for arbitrary nonhuman systems? We all share a common biological background, our behavior, emotions and especially positive/negative feelings arise from the particular way how our brains are built. Man is by nature a social animal, and much of our behavior and feelings of empathy, compassion, fairness, greed, social status, etc are formed and expressed in society. Our minds are adjusted for this, as are the minds of other mammals. A human that forms an ethical system when joining a civil society does so in the basis of all this shared context, and within the limitations of it.

However, all this is orthogonal to intelligence. All the variety of mankind encompasses a tiny island in the whole space of possible intelligences and their goals. Any system of ethics is feasible, a vast majority of them having no relationship whatsoever to what we could consider ethical. A human can be expected to join a society and learn ethics because our brain is hardwired to respond to social stimuli - an AI would learn ethics from a society only if its previous ethics already happen to value that highly, and that isn't likely to happen without careful design.

There is a thought experiment that is commonly used as an example - the "paperclip maximizer", an artificial agent with an ethical system that defines 'good' proportionally to the number of paperclips that exist in the universe and only that - and that is a perfectly reasonable example, because a completely random ethical system is overwhelmingly likely to reduce to something stupid like that. The space of "reasonable" (to us) ethical systems is very tiny compared to all ethical systems possible, and expecting that a randomly selected ethical system will just happen to land there is like relying on winning a high-stakes lottery.

We want their ethics to contain certain things

So the main reason why we are studying AGI ethics is because we (or some of us) really, really, want the AGI ethical system to include certain things. For example, I'd like that ethical system to include my survival. I'd prefer that ethical system to be consistent with my happiness and wellbeing - and the very existence of a very powerful entity that simply happens to not care about me at all is not compatible with my survival and wellbeing, so I'd really prefer for it to care about me, or alternatively never become very powerful.

One could certainly argue that we have no right to constitute ethics, rules and morale for another society which could be much more intelligent and better than us. That another society might be "better" in some manner - but better for whom ? If it's better for "them" but worse for me (and humanity), then I have strong motivation to simply claim that right and try my hardest to ensure that this hypothetical future society includes me, my desires and my ethics.


A parallel situation might be that of spam/not spam. The detection of spam by AI has been pretty successful, so there is an existing algorithm - classification. However while you have a possible approach you are still missing the key ingredient which is sufficient data to train the model on.

AI depends on a large amount of data to train the model. Ideally you will have a team of researchers available to read a (large?) number of sources and classify by hand whether the source is the original reference or just a repeater, and whether the topic is relevant. The more labelled and balanced data you have, the better the model and the better the results. Then, like the spam/not spam situation, you just apply your model and the best references pop out. You already have the key word of music, so you just use as candidates those sources that reference "music" and any other defining keywords.


If the Turing test is passed, does this imply that computers exhibit intelligence?

We don't know.

However, an important line will have been crossed - it will be impossible to tell the difference between an intelligent agent and the machine by use of a text interface. Which is the main point of the test - "if it quacks like a duck".

It is also an important philosophical point. Whether intelligence is defined purely by behaviour in an environment, or by the mechanisms that arrive at that behaviour. A suitably large database of conversational openers and "correct" responses can in theory mimic a lot of real world conversations. Some chatbots take advantage of this and use modern computer capacity to store a lot of responses, and that approach has gained competitive scores in the Loebner prize competition (although not to the stage of actually passing the test). This leads us to the Chinese Room issue, and wondering which part of the system is actually intelligent, or even how much of human conversation is actually intelligent or meaningful (and it what ways).


Different methods of calculating gradients of cost function(loss function)

I'm pretty sure they're the same thing. Both backpropagation and gradient descent's essential ideas are to calculate the partial derivative of the cost with respect to each weight, and subtract the partial derivative times the learning rate.


Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?

Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?

Yes.

An expected reward function from $s,a$ is all you need to construct valid Bellman equations for value functions. For example

$$q_{\pi}(s,a) = r(s,a) + \gamma\sum_{s'}p(s'|s,a)\sum_{a'}\pi(a'|s')q(s',a')$$

is a valid way of writing the Bellman equation for action values. You can derive this from $r(s,a) = \sum_{r,s'}rp(r,s'|s,a)$ and $q_{\pi}(s,a) = \sum_{r,s'}p(r,s'|s,a)(r + \gamma\sum_{a'}\pi(a'|s')q(s',a'))$ if you have the equations in that form.

However, in general there is no such thing as an "expected state" when there is more than one possible outcome (i.e. in environments with stochastic state transitions). You can take a mean of the state vector representations over the samples you see for $s'$ but that is not the same thing at all and could easily be a representation of an unreachable/nonsense state.

In some cases, the expectation $\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$ where $x(s)$ creates a feature vector from any given state $s$, $x(s): \mathcal{S} \rightarrow \mathbb{R}^d$, can be meaningful. The broadest and most trivial example of this is for deterministic environments. You may be able to construct stochastic environments where there is a good interpretation of such a vector, even if it does not represent any reachable state.

Simple one-hot encoded states could maybe made to work like this by representing a probability distribution over states (this would also require re-interpretations of expected reward function and value functions). That is effectively a kernel density function over discrete state space.

In general knowing this $\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$ expected value does not help resolve future rewards, as they can depend arbitrarily on specific state transitions.


For some reasons, a reward becomes a penalty if

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward.

To me this implies that your profit per transaction is not the true reward function that you should be using. You don't say directly in the question, but presumably there is some per-transaction cost, tax or other issue which means that these low gain transactions are not desirable.

The answer is to find a more accurate reward function. You have suggested a quick fix of subtracting a fixed offset from the reward. This should have a desirable effect of limiting borderline buy/sell arrangements and meeting your constraint, so I would suggest trying it. The main issue I see with it is that you may have changed the reward function so that it doesn't truly reflect your goals.

A better approach is to look more carefully at the problem and your goals at a higher level. There must be a reason for wanting to avoid these smaller profit transactions. What is it, and can it be expressed itself as a reward? For instance, if there will be a transaction fee, include that. If the fee structure or reasoning is complicated or appears delayed or aggregated over many actions, then this makes optimisation harder, but RL is actually designed to cope with this. You could for instance only reward the agent with profit/loss after a group of actions covering a whole day's trade. You would then rely on the learning algorithm to figure out which states and actions combined to generate the observed reward.

If there is an unknown or random difference between predicted and actual financial gain then no amount of juggling the reward function gets around that. For RL, learning about and predicting expected gains is "built in", but that does not mean that this task becomes easier - in fact if prediction in the problem is hard, you may be better off focusing on that and forgetting RL at least initially. Your question is not clear on that, but if you want to avoid risk whilst learning, you should bear in mind that a simple hard rule would likely interfere with the ability of RL to work. At its heart RL is a trial-and-error learning system. Errors should be expected, and are required for the system to learn where best the balance point is between risk and reward. Of course that doesn't help you if the learning system would deliberately make you bankrupt exploring what happens when you sell things at a loss - there are likely to be ways to avoid that and still achieve your goals, but you will need to explain more about your system in a different question.


Preparing data set for the YOLO algorithm

Ok, let go step by step.
What you are working on is YOLOv1, in this version of the YOLO algorithm, the maximum bounding boxes that the model can return is 7x7 = 49 boxes as 49 cells since the output shape is 7x7x30.
For each box, the depth of output is 30 because the number of labels of PASCAL VOCS 2012 is 20 (the author of YOLOv1 trained on this dataset) so from index 0 to index 19 will represent the label of that bounding box. From 20 to 23 are the position and size of that box.
The 24th represents 2 things, first is the confidence of that box, since this is ground truth so the confidence should be 1. Second, you know that YOLOv1 can only return 49 boxes at maximum (actually, you can edit the number of boxes by yourself) so the ground truth should only handle one per cell, hence the 24th is the binary value to make sure there are no duplicates bounding boxes in one cell. (25 to 29 is because the author predict 2 bounding boxes per cell)
In your case, the output should be 7x7x(2 + 5 x 2) = 7x7x12 with 2 classes and 2 boxes per cell.


How many training example text classifier needs to be trained?

As a general rule of thumb I typically use 10*(# of features) for shallow machine learning models such as Naive Bayes with only 2 classes.

So it all depends on the number of features you will be using. However, the more output classes the more data you will need for proper discrimination. The addition of more classes is not linear but I think you can get away with: 10*(# of features)*(# of output classes)


How is the Jacobian a generalisation of the gradient?

In short, the Jacobian matrix is a generalization of the gradient for vector-valued functions.

Recall that the gradient is a vector of partial derivatives of a multi-variable function. So, consider a multi-variable function of the form $f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}$. The output of this function is $f(x_1, x_2, \dots, x_N) = y$, where $x_i \in \mathcal{X}_i$, for $i=1, \dots, N$ and $y \in \mathcal{Y}$. And the gradient is $\nabla f = \left[ \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_N} \right] \in \mathbb{R}$.

A vector-valued function is a function whose output is a vector, i.e. a function of the form $f: \mathcal{X} \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M$ (I am not sure if this notation is rigorous enough!), so the output of this function is a vector $f(x) = [y_1, y_2, \dots, y_M]$, where $x \in \mathcal{X}$ and $y_i \in \mathcal{Y}_i$, for $i = 1, \dots, M$. You can also view a vector-valued function $f$ as a vector of scalar-valued functions $[f_1, f_2, \dots, f_M]$, where $f_i: \mathcal{X} \rightarrow \mathcal{Y}_i$, for all $i$.

You can also have multi-variable vector-valued functions, i.e. functions of the form

$$f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M.$$

The Jacobian matrix is an $N \times M$ matrix with one partial derivative for each combination of inputs and outputs (i.e. $f_i$).

If you want to optimize a multi-variable vector-valued function, you can make use of the Jacobian, in a similar way that you make use of the gradient in the case of multi-variable functions, but, although I've seen it in the past, I can't provide now a concrete example of an application of the Jacobian (but the linked slides probably do that).


What are the differences between uniform-cost search and greedy best-first search?

In the case of UCS, the evaluation function (that is, the function that is used to select the next node to expand) is $f(n) = g(n)$, where $g(n)$ is the cost of the path from the initial node to $n$, while in the case of the greedy BFS it is $f(n) = h(n)$, where $h(n)$ is the heuristic function that estimates the cost of the path from $n$ to the goal node. In other words, in the case of UCS, nodes are expanded only using the experience (in the form of $g(n)$), while, in the case of GBFS, nodes are expanded only using the estimate of the cost to the goal. Note that, in both cases, the node that is chosen to be expanded is the one with the smallest $f(n)$ value.


What is a cascaded convolutional neural network?

The paper you are citing is the paper that introduced the cascaded convolution neural network. In fact, in this paper, the authors say

  To realize 3DDFA, we propose to combine two achievements in recent years, namely, Cascaded Regression and the Convolutional Neural Network (CNN). This combination requires the introduction of a new input feature which fulfills the "cascade manner" and "convolution manner" simultaneously (see Sec. 3.2) and a new cost function which can model the priority of 3DMM parameters (see Sec. 3.4)

where 3DDFA stands for 3D Dense Face Alignment, the framework proposed in this paper for face alignment, in which a dense 3D Morphable Model (3DMM) is fitted to the image via cascaded CNNs (the regressor), where the term dense refers to the number of points of the face that will be modeled. See figure 1 of this paper, which should provide some intuition behind the purpose of this framework.

In section 3 (page 3), they also say

  In this section, we introduce how to combine Cascaded Regression and CNNs to realize 3DDFA. By applying a CNN as the regressor in Eqn. 1, Cascaded CNN can be formulated as:

\begin{align}
\mathbf{p}^{k+1} = \mathbf{p}^{k} + \text{Net}^{k}
(\text{Fea}(\mathbf{I}, \mathbf{p}^k)) \tag{1}\label{1}
\end{align}

where

$k$ is the iteration number
$\mathbf{p}$ is the regression objective
$\text{Net}$ is the CNN structure
$\text{Fea}$ contains the two constructed image features

Pose Adaptive Feature (PAF) (section 3.2.1)
Projected Normalized Coordinate Code (PNCC) (section 3.2.2)

$\mathbf{I}$ is the image

The expression cascaded CNN apparently refers to the fact that equation \ref{1} is used iteratively, so there will be multiple CNNs, one for each iteration $k$. In fact, in the paper, they say

  Unlike existing CNN methods that apply different network structures for different fitting stages, 3DDFA employs a unified network structure across the cascade. In general, at iteration $k$ ($k = 0, 1, \dots, K$), given an initial parameter $\mathbf{p}^k$, we construct PNCC and PAF with $\mathbf{p}^k$ and train a two-stream CNN $\text{Net}^k$ to conduct fitting. The output features from two streams are merged to predict the parameter update $\Delta \mathbf{p}^k$

  $$
\Delta \mathbf{p}^k = \text{Net}^k(\text{PAF}(\mathbf{p}^k, \mathbf{I}), \text{PNCC}(\mathbf{p}^k, \mathbf{I}))
$$

  Afterwards, a better intermediate parameter $\mathbf{p}^{k+1} = \mathbf{p}^k + \Delta \mathbf{p}^k$ becomes the input of the next network $\text{Net}^k$ which has the same structure but different weights with $\text{Net}^k$.

In figure 2 of the paper (page 4), the structure of this two-stream CNN, $\text{Net}^k$, at iteration $k$, is shown.


Is there a limit of minimum error for a particular training dataset in artificial Neural Network?

In some sense yes, in some sense no. The "error" word you are using is not defined in the question. The network can have two type of errors, quite different - generalization error and loss (training error). The lower bound of loss on the dataset obviously depend on the network architecture, because loss is value produced by network as a whole, and some network will be able to produce exactly zero loss (which is likely be overfitting, wich is bad).

The generalization error is how well network generalize form training dataset to potentially infinite (or very big) amount of data samples "in the wild", so generalization error can not be calculated precisely, only estimated. The lower bound on generalization error is studied in Statistical Learning. It always depend on the complexity of the classifier (in our case architecture of network). Statistical Learning theory produce estimations like VC generalization bound which connect generalization error with size of the dataset and complexity of the classifer (netwroks in our case) space. However for specific dataset it probably don't make sense to talk about generalization bound outside of context of complexity of classifier space, because if there is no restriction we can always construct bigger network which would have lesser generalization error (for example by pretraning bigger network on bigger dataset)
To put it short you have to put restriction on complexity of network space to be able estimate minimum generalization error (and that probably wouldn't be useful anyway, because it wouldn't say anything about if gradient descent would reach that error)


Can multiple activation functions be replaced with a single activation function?

TL;DR: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.

Mathematical Explanation

The outputs of your intermediate neurons (in the absence of activation functions) are now:

$\text{(1)}\quad H_i(x) = \sum_{j=1}^nw_{ij}\cdot x_j+b_i$

You are then summing each $H_i$:

$\text{(2)}\quad\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$

You then pass $\mathcal{H(x)}$ to some activation say $g$:

$\text{(3)}\quad\hat y = g(\mathcal{H(x)})$

The trouble is that the inner term $\mathcal{H(x)}$ mathematically reduces to a single linear operation on $x$. Proof:

$\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$. Substituting in (1):
$\mathcal{H(x)}=\sum_{i=1}^n((\sum_{j=1}^nw_{ij}\cdot x_j)+b_i)$. This can be re-aranged :
$\mathcal{H(x)}=(\sum_{j=1}^n(\sum_{i=1}^nw_{ij})\cdot x_j)+\sum_{i=1}^nb_i$. But this reduces to:
$\mathcal{H(x)}=\sum_{j=1}^n\tilde w_{j}\cdot x_j+\tilde b$. Where $\tilde w_{j},\tilde b$ are scalars.

Thus, without the non-linear activations (3) mathematically reduces to a single neuron.


Why are the Bellman operators contractions?

The inequality
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \label{1}\tag{1},
\end{align}
where $U$ and $V$ are two value functions, follows from the definition of Bellman policy operator (at slide  16)
\begin{align}
T^{\pi} V(s)
&\triangleq
R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right) \\
&=R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) V\left(s^{\prime}\right), \; \forall s \in S
\tag{2}\label{2},
\end{align}
where $\triangleq$ means "defined as". Note the $\pi$ in the definition, hence the name Bellman policy operator (BPO), and note that the BPO holds for all $s$.
To prove (\ref{1}), first recall that
$$\left\|\mathbf {x} \right\|_{\infty } \triangleq \max _{i}\left|x_{i}\right|.$$
In the case of value functions $V$ and $U$, we have
$$\left\|V - U \right\|_{\infty } \triangleq \max_{s \in S}\left|V(s) - U(s) \right|.$$
Note also that $Pr$ is always non-negative (specifically, between $0$ and $1$).
Successively, we expand the left-hand side of (\ref{1}) by applying the definition (\ref{2}) and using the properties just mentioned
\begin{align}
{\scriptsize
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty}
=
\left\|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) - \left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right\|_{\infty}
}%
=\\
{\scriptsize
\max_{s \in S}
\left|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) - \left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right|
}%
=
\\
\max_{s \in S}
\left|
\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
\gamma
\max_{s \in S}
\left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
\gamma \max_{s \in S} \left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left ( V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right)
\right|
=
\\
\gamma \max_{s \in S}
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|
\end{align}
Now, note that

$\max _{s^{\prime}}\left|V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right| = \| V - U \|_{_{\infty}}$
$\alpha x + (1- \alpha) x = x$ for $\alpha \in [0, 1]$
$\left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right| \leq \max_{s' \in S} \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|$

NOTE: I am not completely sure about all these steps. I need to think about them a little bit more (but later). I encourage you to try to spot wrong derivations!!


Which field to study to learn & create a.i generated simulations?

You need to define "simulation" more specific. Playing Mario, Swapping face on image/video, or generating simulation of objects that are orbiting use different techniques.

Playing Mario or "AI that playing game": the AI agent trained on available environment (Mario game, so the environment is not generated) and learn the best sequential actions to achieve the goal. It runs the game thousand times, when it did a wrong action then it gets "penalties" that improve its knowledge. The algorithm that can be used is Reinforcement Learning, but some earlier paper use Genetic Algorithm to generate the best action
Face swap: It's close to computer vision area, some methods that I know use Style Transfer principle (Convolutional Neural Network) to make transformation of face of one image to another image. You can read the basic of style transfer here.
Generating physical movement: I don't know too much about this topic but I know there are some papers talk about this, Fluid Net from Google workers and this paper from TU Munchen. At a glance they also use CNN to improve the result but the main simulation came from Euler Fluid Equation. So if you need to generate object that orbiting, I think you need to find equations that models that movement.

Hope it helps!


RL: Encoding action conditioned on previous action

It is hard to say for certain without knowing full details and results of experiments.
However, if the game allows for splitting decisions up, it will likely be better for the agent to take advantage of extra knowledge of the value of any previously hidden card just taken from the draw pile.
In general, if each player decision is taken sequentially, resulting in changes to state, then it is a separate action on a separate time step according to the MDP theoretical model used in reinforcement learning (RL). You might want to describe/notate the time steps differently so that they match how the game play proceeds. However, for the purposes of RL, each decision point should be on a new time step, and should result in a new state, new value estimates etc.
Similarly, whether or not the current choice is the player's first card or second card to be drawn needs to be part of the state. This detail of the state might already be covered by the number of cards in the player's hand, if logically the number of cards is always the same at each stage. However, if hand size can vary for other reasons, it is worth adding an explicit flag for "first draw choice" or similar so that the agent can use the information.
You have some freedom for encoding the action space. If drawing cards is the only possible action in this game at all stages, then a binary output vector of 1 + (number_of_players - 1) dimensions would be suitable. Other encodings may work well too, it depends if there is any logical structure to the choices or some derived data that encodes useful game information.
It may be useful to arrange the action choices so that the index for drawing from each player's discard pile is considered relatively to the current player's turn. That is, instead of actions being arranged $[draw, discard P1, discard P3, discard P4, discard P5]$ for P2, they would be arranged $[draw, discard P3, discard P4, discard P5, discard P1]$ and for P3 would be different: $[draw, discard P4, discard P5, discard P1, discard P2]$ . . . that would inherently allow for the cyclical nature of turns. State representation would need to similarly rotate knowledge about each player to match this. You might not need to do this, but I would recommend it for games where there is a lot of common logic regarding action choices relative to turn position that you could take advantage of. The opposite would apply (and you would use absolute player positions) if there were important differences throughout the game between being P1, P2, P3 etc.


Trying to make snake learn different directions

I observed the scoring system you used and compared it to the field size and starving rate. The worms either died or did not try enough hard with the prize of 10 of eating, so I added the amount to 1000.

This did still not introduce the 180 degrees turn yet. I had to adjust these settings on Neat:

  mutationRate: 0.7,
  mutationAmount: 6,

which introduced  where the worms optimize their path as diagonal and when forced to change 90 direction, actually a tiny 180 happens on screen.

On same mutationRate and mutationAmount, but 10 points I once observed a horizontal 180, when the food appeared back and diagonal movement was not invented. Also I observed with this last setup a motion of constant 180s flow where consecutive vertical or horizontal 180s formed a two pixel thick line on screen.

These observations do not 100% do the same as your animation, but do instead introduce various ways that are quite near, or can be in certain conditions an initiative to such phenomenom.

White worm introduces two pixel line, red 90 degree turn with miniature 180.

EDIT

I changed the nice curve ratio like this:

     var ratio = 0.75 + ((mindir / maxdir) * 0.25* difstepsratio);

and around generation 200 there lived one worm to be able to move to all directions and make 180 if needed.

Unfortunately it died before I found screen shot buttons from my phone and (s)he happened to become second in the race and thus was left as a unicorn who existed only once.

Still remains mystery whether that mutation could survive and become the main species some day.

Further remarks:

I noticed that when mutation takes place in snake code the same pattern is replicated many times. If snake gets idea of consecutive turn lefts or turn rights when it is long, a suicide with tail is quite often taking place.

I tried to taggle this with a higher score on snake collision than starvation and around generation #180 the dna of suicidal 360 loop was born and remained some generations.

However, this generation did not evolve to the desired intelligent 180 worm I previously had seen. Instead, the suicide was so rewarding that it killed that branch quite soon away.


What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation as per @nbro's suggestion, and it seems like the loss function is the cross entropy between the predicted ($\|sentence length\|$ x $\|vocab\|$) probability matrix (right before taking the argmax to find the token to output), and the $\|sentence length\|$-length vector of token IDs as the true label.


How do I show that uniform-cost search is a special case of A*?

Yes, UCS is a special case of A*.

UCS uses the evaluation function $f(n) = g(n)$, where $g(n)$ is the length of the path from the starting node to $n$, whereas A* uses the evaluation function $f(n) = g(n) + h(n)$, where $g(n)$ means the same thing as in UCS and $h(n)$, called the "heuristic" function, is an estimate of the distance from $n$ to the goal node. In the A* algorithm, $h(n)$ must be admissible.

UCS is a special case of A* which corresponds to having $h(n) = 0, \forall n$. A heuristic function $h$ which has $h(n) = 0$, $\forall n$, is clearly admissible, because it always "underestimates" the distance to the goal, which cannot be smaller than $0$, unless you have negative edges (but I assume that all edges are non-negative). So, indeed, UCS is a special case of A*, and its heuristic function is even admissible!

To see this with an example, just draw a simple graph, and apply the A* algorithm using $h(n) = 0$, for all $n$, and then apply UCS to the same graph. You will obtain the same results.


Interpolating image to increase resolution before feeding it to a neural network

One of the used is to use a neural interpolation, this is using a pretrained model to zoom our images, in some sense training with this kind of images is like combining this pretrainined model with our new data, and can be seen as transfer learning technique


Why does not the deepAR model of Amazon require the time series being stationary, as opposed to ARMA model?

I have have been part of a project where we implemented Amazon Forecast service in production. As per my practical experience, we need not worry about stationarity while applying DeepAR.
As per my understanding as in ARIMA the prediction function is a function around the time series moving average, but in DeepAR we are more relying on the backtest window and leg-horizon length.
DeepAR uses recurrent neural Networks (RNN) and gated recurrent unit (GRU) without any assumption on their probability distribution to learn the sequence for prediction.
For a detailed explanation, you can refer. https://aws.amazon.com/blogs/machine-learning/forecasting-time-series-with-dynamic-deep-learning-on-aws/


Why do you have to register a bot with Azure?

These bots depend on heavy NLP services that are provided via Azure. Implementing/deploying your own production-quality versions of these is nontrivial, if not impossible (since you don't have all the training data, internal algorithms, etc. that Microsoft/Amazon/Google/IBM et al have), and is generally non-feasible on home-grade devices.

And yes, that makes the relevant Azure services a dependency that your chatbot will have, and if you use that service you'll need to be aware of their SLAs.

Plus, they'd like to charge you money. :-)


Are there ways to learn and practice Deep Learning without downloading and installing anything?

As I understand, I think you wish to directly try out some deep learning stuff and things like library downloading, tools downloading, and managing all these really stop you from even starting to try out deep learning experiments.If this is what you asked for:

Google Colab

I think this is the best place for you.
Anyone with a Google Drive account can sign up for Colab by heading to
 and following the listed instructions.

Since you mentioned that you just wanted to try out and practice stuffs, this would be
ideal for you.

All major Python libraries like TensorFlow, Scikit-learn, Matplotlib among many
others
are pre-installed and ready to be imported.
Built on top of Jupyter Notebook

Please have a quick look at:

https://medium.com/lean-in-women-in-tech-india/google-colab-the-beginners-guide-5ad3b417dfa

2.Microsoft Azure

The Azure free account is available to all new customers of Azure. If you have never
  tried or paid for Azure before, you’re eligible.
  Try out a student account:
   https://azure.microsoft.com/en-in/free/students/

Hope this helps, go ahead with practicing deep learning
if not please feel free to raise questions, always ready to help


What is the optimal value function of the shifted version of the reward function?

Theorem
The optimal state-action value function of $r'(s, a) \triangleq r(s, a) + c$, for $c \in \mathbb{R}$, would be
\begin{align}
q_*(s, a) + c + c\gamma + c \gamma^2 + c \gamma^3 + \dots
&=q_*(s, a) + c \left( 1 + \gamma + \gamma^2 + \gamma^3 + \dots \right) \\
&= q_*(s, a) + c \left( \sum_{k=0}^{\infty} \gamma^{k} \right) \\
&=q_*(s, a) + c\left(\frac{1}{1 - \gamma}\right) \\
&=q_*(s, a) + \frac{c}{1 - \gamma},
\end{align}
where $\gamma < 1$ is the discount factor (and $\sum_{k=0}^{\infty} \gamma^{k}$ is a geometric series).
Therefore, the optimal policy does not change if we shift the reward function by some constant.
Proof
The proof follows the same reasoning of the proof in this answer, i.e. we show that the Bellman optimality equation also holds in this case.
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \max_{a' \in\mathcal{A}(s')} \left( q_*(s',a') + \frac{c}{1 - \gamma}  \right) \right) \tag{1}\label{1}
\end{align}
Given that $\frac{c}{1 - \gamma}$ is a constant, it does not affect the max, given that we add this constant to all pairs $(s', a')$, for all $a' \in\mathcal{A}(s')$. This holds even if $c$ is negative. So, we can take that addition out of the max, but note that we sum (rather than multiply, like the proof in the other answer).
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \left (\frac{c}{1 - \gamma} +  \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) +  \frac{c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c(1 - \gamma) + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c - c\gamma + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} \left ( p(s',r \mid s,a)\frac{c}{1 - \gamma} \right) + \\ & \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}  \left( p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \tag{2}\label{2} \\
\end{align}
Given that $p(s',r \mid s,a)$ is a probability distribution, then the affine combination of $\frac{c}{1 - \gamma}$, where the affine weights are $p(s',r \mid s,a)$, is $\frac{c}{1 - \gamma}$. To see that this is true, suppose that we have the probability vector $[0.2, 0.5, 0.3]$ and let $c$ be a constant, then $0.2c + 0.5c + 0.3c = c$.
So, equation \ref{2} becomes
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma}
&= \frac{c}{1 - \gamma}  + \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)\\
q_*(s,a)
&=\sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)
\end{align}
which is the Bellman optimality equation (see equation 3.20 of Sutton & Barto book, p. 64).


Is AGI likely to be developed in the next decade?

As a riff on my answer to this question, which is about the broader concern of the development of the singularity, rather than the narrower concern of the development of AGI:

I can say that among AI researchers I interact with, it far more common to view the development of AGI in the next decade as speculation (or even wild speculation) than as settled fact.

This is borne out by surveys of AI researchers, with 80% thinking "The earliest that machines will be able to simulate learning and every other
aspect of human intelligence" is in "more than 50 years" or "never", and just a few percent thinking that such forms of AI are "near". It's possible to quibble over what exactly is meant by AGI, but it seems likely that for us to reach AGI, we'd need to simulate human-level intelligence in at least most of its aspects. The fact that AI researchers think this is very far off suggests that they also think AGI is not right around the corner.

I suspect that the reasons AI researchers are less optimistic about AGI than Kurzweil or others in tech (but not in AI), are rooted in the fact that we still don't have a good understanding of what human intelligence is. It's difficult to simulate something that we can't pin down. Another factor is that most AI researchers have been working in AI for a long time. There are countless past proposals for AGI frameworks, and all of them have been not just wrong, but in the end, more or less hopelessly wrong. I think this creates an innate skepticism of AGI, which may perhaps be unfair. Nonetheless, expert opinion on this one is pretty well settled: no AGI this decade, and maybe not ever!


How to use validation dataset in my logistic regression model?

So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.
Now its important you do the split before entering the fit, and not do it for each epoch or something alike.
Once you do that and the fit method should be something like:
def fit(self, x_train, y_train, x_val, y_val, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):

Then you should, at the end of each epoch, test the performance of the model on the validation set entirely and calculate the desired metric for evaluation. If you improved it's better to save the current model. This is done repeatedly for each epoch until the end of the training and you will guarantee to have the model who gave you the best results on the validation set rather than on the training set, which might be overfitting it.
I will do it in a separate method with the following flow:

iterate on each sample in the validation set
for each one calculate the loss/metric
append it to some list
return the mean of that list

And if the average result is better from some previously saved one, save the new model


How do you handle multiple categorical values in a single column for wide_deep model in tensorflow?

I am also stuck with this problem and after some searchs I end up here.

Maybe you can take a look at indicator column. I think embedding column behaves in a similar way.


Is my fine-tuned model learning anything at all?

It's difficult to say without knowing what your data looks like but from the numbers it seems too less and the images might be too similar to one another or very different. In any case, I'd have checked using other networks like Inception and decreasing learning rate even further (say 0.0001) to not mess with the Imagenet weights if your data is not very different from Imagenet classes.


Could a Boltzmann machine store more patterns than a Hopfield net?

Indeed you're intuition is correct, a Boltzmann machine is able to hold more than a Hopfield network in its memory because of its stochastic nature as explored in this paper. In the paper they note that the capacity is around 0.6. After this ratio it starts to break down and adds much more noise to the recalled patterns.


Recommend item from set based on features

This problem is usually approached with "Singular Value Decomposition".

Search also for the "Netflix Challenge".


To what extent can artificially intelligent agents reliably predict trends in financial markets?

This is a highly relevant question as market trends have become more emphasized over the fundamentals of individual companies, and algorithmic trading has proven to be quite effective, particularly in areas such as high-frequency micro-trading.

This 2013 Forbes article estimated nearly 80% of stock trading volume in the U.S. is conducted by automated trading systems.  More recently, Bloomberg published an article on the subject: The U.S. Stock Market Belongs to Bots.

The massive adoption of algorithmic trading in a fairly short span is a pretty good indicator that these systems are effective.  However,

My feeling with any predictive system is that it works until it doesn't

Imperfect and Incomplete information are persistent in real-world scenarios.  The implication is that there are always unknown factors that can diminish quality of analysis, and even lead to disastrous outcomes.

No matter how complex an algorithm becomes, reality is more complex, (at least for the foreseeable, pre-singularity future;)

Adding to that, systems in general tend to be imperfect, and when engaged in massively repetitive functions, small errors can grow exponentially and produce catastrophic effects.

"Flash Crashes" are a recent phenomena, and a result of algorithmic, high-frequency, black-box trading.


Meaning of the statement $\forall x \exists y \forall z (z \neq y \iff f(x) \neq z)$

The statement is

"for all $x$, there exists a value of $y$ such that for all $z$,
$z\neq y$ if and only if $z \neq f(x)$".

This can be simplified:
$$\begin{align}
& & \forall x \exists y \forall z (z\neq y \iff z \neq f(x))\\
&\implies & \forall x \exists y \forall z (z=y \iff z = f(x))\\
&\implies & \forall x \exists y \forall z (y = f(x))\\
&\implies & \forall x \exists y (y = f(x))\\
\end{align}$$

If we denote the set of all values of $x$ by $X$ and the set of all values of $y$ by $Y$, then this tells us that the function $f$ maps every $x$ in $X$ to a $y$ in $Y$.  That is, $f: X \to Y$.


Why do we update all layers simultaneously while training a neural network?

We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm used to update the parameters and back-propagation is the algorithm used to compute the gradient of the loss function with respect to each of these parameters.

Let's denote a vector that contains all learnable parameters of a neural network $M$ by $\mathbf{w} = \left[w_1, \dots, w_n \right] \in \mathbb{R}^n$ (so $M$ contains $n$ learnable parameters), the loss function of $M$ by $\mathcal{L}$,  the gradient of the loss function with respect to each parameter $w_i$ of $M$ by $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] \in \mathbb{R}^n$, then the gradient descent step to update all parameters is

\begin{align}
\mathbf{w} \leftarrow \mathbf{w} - \gamma * \nabla \mathcal{L} \tag{1} \label{1}
\end{align}

where $\gamma \in \mathbb{R}$ is the learning rate.

In equation \ref{1}, we are assigning to $\mathbf{w}$ the value $\mathbf{w} - \gamma * \nabla \mathcal{L}$, so we are updating all parameters $\mathbf{w}$ simultaneously, so we are also updating all layers simultaneously.

In principle, you could update each parameter $w_i$ individually. To be more precise, you would have the following update rule

\begin{align}
w_i \leftarrow w_i - \gamma * \frac{\partial \mathcal{L}}{\partial w_i}, \; \forall i\tag{2} \label{2}
\end{align}

So, you could update first $w_1$, then $w_2$, and so on.

Actually, you don't need to update the parameters sequentially (also because there is no real order of the parameters). You can actually update them in any other way. You can update the parameters in any way because, although the computation of the gradient highly depends on the structure of the neural network (so if you change the structure, the computation of the gradient also changes), once the gradient is computed, you already have all information to update each parameter independently of each other.

You typically update all parameters (or layers) simultaneously because, in practice, you work with vectors and matrices rather than scalars in order to take benefit from efficient matrix multiplication algorithms and hardware (i.e. GPUs).


Is there any paper, article or book that analyzes the feasibility of acheiving AGI through brain-simulation?

The Title Question

Is there any paper, article or book that analyzes the feasibility of achieving AGI through brain-simulation?

Yes. There are various analyses that have been published. We have some early work like Some Philosophical Problems From the Standpoint of Artificial Intelligence, John McCarthy and Patrick J. Hayes, Stanford University, 1969. And there is more recent and optimistic work like Essentials of General Intelligence: The Direct Path to Artificial General Intelligence, Peter Voss, 2007. There are refutations of the concept of General Intelligence, artificial or not, in some posts here (listed below) and quite a bit of academic scrutiny of the assumptions behind the AGI concept and the offspring of it: The Singularity. Concern about The Singularity is a social phenomenon, not a scientific one. It began with the depiction of the emergence of a super-intelligence as an inevitability in science fiction and in sensational media.
But mathematically terse proofs of AGI feasibility are not found in the media or in the Terminator franchise started by the wonderful imagination of James Cameron. This is not to say that world domination cannot fall into the hands of a new species created by humans. In fact, Jaques Ellul, in his Techological Society points out that in many ways humans are already subservient to technology — that the creation of humankind is already autonomous and dominant. His 449 page heap of evidence is witty and quite convincing.
Academic publications either

(a) Assume feasibility and discuss approaches to its design or

(b) Illuminate caveats in the idea of a universally intelligent system

Where is the Mind?
Whether the entirety of the mind arises from a physical system is a question that dates back to René Descartes and Gottfried Leibniz. Current conceptions of intelligence tend to deny the interaction and interdependence of the human brain, the rest of the human, and the biosphere into which the human fits without extreme expenditure to produce a floating, underwater, or underground simulation of the biosphere. The mind may center in the brain, but any brain disconnected from the biosphere and the circulatory system for even a short time is completely useless. A think tank is not a bunch of smart people in a sensory deprivation tank. Quite the opposite. Early experimentation with think tanks revealed the importance of mental health of the participants through diversion and physical exercise so that the various forms of neurological stasis could be maintained.
It is quite questionable whether computers without bodies will ever learn as much of general use (that can be put to use in the biosphere) without moving about and interaction with the biosphere. It is highly likely that the mind is more like Carl Jung sees it, as in the brain and elsewhere. Ludwik Fleck, in a somewhat Jungian fashion, speaks of science as the product of a collective in his Genesis and Development of a Scientific Fact, a good read that throws considerable light on the folly of most people's naive conception of facts.
Quantum Computing Buzz
When Intel's founder, Robert Noyce, brought in Gordon Moore in 1975, it was because of various statistical observations he had made, including what is now known as Moore's Law, a curve that on semi-log paper in 1965 looked linear. Since then, the business assumption in Silicon Valley has been that compactness and speed would double frequently and consistently. In the last few years, speed has stopped doubling and compactness seems to have reached its limit around 7 nm. The transistors in this VLSI technology have only a handful of atoms to represent the semiconductor topologies that permit gating, the fundamental activity of digital circuitry.
In essence, to continue the Silicon Valley boon, it has to become Particle Valley, using carbon allotropes or other nano-tech to build computers where Moore's Law is not overturned by the limitations of the structure of the universe. Whether this can be done requires its own feasibility study. Already, SSD (solid state drive) technology relies heavily on probability. Errors are occurring in great proportions and error correcting techniques are used to keep them manageable. That's because, as we reach for designs closer to the quantum level, Brownian motion or the light electromagnetic interference can upset the digital operation intended.
Challenges Versus Show Stoppers
As with many trends in the history of human endeavors, determining what is a challenge and what is a show stopper is not clear until viewing in hindsight. That is why people are asking the various questions in this Stack Exchange listed in the question.
Stating that mentioning our partial knowledge of brain operation is off topic for this question might have been appropriate if we were trying to invent something that we had not already seen in operation in nature. However, that is not the case. We call it artificial intelligence because we think that we are intelligent and want to extend that facility we find in ourselves. The following fields, in addition to quantum physics and machine learning, are thus necessarily relevant to the feasibility of a more generally usable fabricated intelligent system.

Neuro-chemistry in general
Genetics as it applies to neurological network structuring and cell activity
Correlation between DNA coding and human mental performance and stability
Applied psychology
Addiction science
Cognitive science
Mathematics in general and probability and statistics specifically
Control theory
Philosophy, especially in regard to whether intelligence actually exists and the ideas of culturally determined knowledge and belief
Language and linguistics
Symbology
Conceptions of a thought collective

Not only do we not know enough in these areas, but we know very little. There is also considerable evidence that humanity tends to forget some of the things it once knew. Take away our petroleum and we may find out just how much we forgot.
Whether any of the things we have lost from our general knowledge are prerequisites to solving the challenges of universally intelligent system design is unknown. A simple example is that many computing science graduates do not know anything about Kurt Gödel's incompletness theorems and how they led to Alan Turing's completeness theorem. Many use back-propagation in their Python code but don't have any conception that it is a corrective feedback distribution scheme.
On the flip side, we have no strong argument to draw the conclusion that there is some theoretical limitation as solid as the second law of thermodynamics, one of the apparently insurmountable hard stops built into our physical reality on and off planet. We have no proof that intelligence does not exist and is rather an anthrocentric fantasy. We have no proof that whatever limitations humans have cannot be overcome by copying the intellectual life of human beings into human creations or using intellect to create greater intellect in machines.
What we know, as smart as we think we are, is greatly outweighed by what we don't know. Anyone in real research work will agree.

Refutations of AGI Philosophy
Some of the many refutations of AGI that appear throughout academia. Some articles include these.

What is AGI?, Luke Muehlhauser, 2013

Philosophy will be the key that unlocks artificial intelligence, David Deutsch, 2012, The Guardian

On Artificial Intelligence, Rodney Brooks, 2018, The Library of Economics and Liberty

These are some posts here that question various assumptions prevalent in popular AI media.

Who was the first person to recognize the distinction between human-like general intelligence and domain-specific intelligence?


Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

In any reinforcement learning problem, not just Deep RL, then there is an upper bound for the cumulative reward, provided that the problem is episodic and not continuing.
If the problem is episodic and the rewards are designed such that the problem has a natural ending, i.e. the episode will end regardless of how well the agent does in the environment, then then you could work it out by calculating the max possible reward in each step of the episode; however this is potentially non-trivial depending on your environment.
For an example in a trivial setting, however, imagine the problem of cartpole -- I could define the MDP to have a reward of +1 for every time step that the agent is able to balance the pole upright, and 0 when the pole falls. If I also defined that the problem terminates after 200 time steps then the upper bound on cumulative rewards for this problem would be 200.
In general, if the problem is continuing then in theory the problem goes on infinitely and so there is no upper bound, as the episode never ends -- this is partly why we use the discount factor, to ensure that $\sum_{k=0} \gamma^k R_{t+k}$ converges.


How to achieve recognition of postures and gestures?

Just to add some discourse; this is actually an incredibly complex task, as gestures (aka kinematics) function as an auxiliary language that can completely change the meaning of a sentence or even a single word. I recently did a dissertation on the converse (generating the correct gesture from a specific social context & linguistic cues). The factors that go into the production of a particular gesture include the relationship between the two communicators (especially romantic connotations), the social scenario, the physical context, the linguistic context (the ongoing conversation, if any), a whole lot of personal factors (our gesture use is essentially a hybrid of important individuals around us e.g. friends & family, and this is layered under the individual's psychological state). Then the whole thing is flipped around again when you look at how gestures are used completely differently in different cultures (look up gestures that swear words in other cultures for example!). There are a number of models for gesture production but none of them captures the complexity of the topic.
Now, that may seem like a whole lot of fluff that is not wholly relevant to your question, but my point is that ASIMO isn't actually very 'clever at this. AFAIK (I have heard from a visualization guy that this is how he thinks they do it) they use conventional (but optimized) image recognition techniques trained on a corpus of data to achieve recognition of particular movements. One would assume that the dataset consists of a series of videos/images of gestures labelled with that particular gesture (as interpreted by a human), which can then be treated as a machine learning problem. The issue with this is that it does not capture ANY of the issues I mentioned above. Now if we return to the current best interpretation of gesture that we have (that it is essentially auxiliary language in its own right), ASIMO isn't recognizing any element of language beyond the immediately recognizable type, 'Emblems'.
'Emblems' are gestures that have a direct verbal translation, for example in English-based cultures, forming a circle with your thumb and index finger translates directly to 'OK'. ASIMO is therefore missing out on a huge part of the non-verbal dictionary (illustrators, affect displays, regulators and adapters are not considered!), and even the part that it is accessing is based on particular individuals' interpretations of said emblems (e.g. someone has sat down and said that this particular movement is this gesture which means this), which as we discussed before is highly personal and contextual. I do not mean this in criticism of Honda; truth be told, gesture recognition and production are in my opinion one of the most interesting problems in AI (even if it's not the most useful) as it is a compound of incredibly complex NLP, visualization and social modelling problems!
Hopefully, I've provided some information on how ASIMO works in this context, but also on why ASIMO's current process is flawed when we look at the wider picture.


Were AI strategies identified at go or starcraft games and how?

Truth be told, I have no idea how to play Go, but luckily this is a AI forum and not a Go forum. Addressing your questions about the specific strategies that AI discovered, there's a paper released by OpenAI titled "Mastering the game of Go without human knowledge" (https://deepmind.com/research/publications/mastering-game-go-without-human-knowledge). Here, there's a section titled "Knowledge learned by AlphaGo Zero." In it, they mention that AlphaGo Zero discovered some variants of common corner sequences. I think it may be worth checking out. The section also lists some other common Go strategies that AlphaGo Zero learned. I believe during the training process, AlphaGo Zero was trained using self-play, where the AI played itself over and over again to get better.
To address your questions about chess. I don't think there's anything limiting AI from learning strategy in chess or anything about chess that limits AI from learning an effective strategy. Check out this paper for more: https://deepmind.com/research/publications/Mastering-Atari-Go-Chess-and-Shogi-by-Planning-with-a-Learned-Model
Here the authors present a general reinforcement learning model capable of mastering multiple common board games. I'm not too familiar with strategies in Star Craft, however, here's a link containing some other projects where strategies that emerge by the AI are shown: https://openai.com/blog/competitive-self-play/


What is the effect of mislabeled training data?

I think the crucial point here is what you precisely mean by mislabelled. Google's image classifier will likely do a 'pretty good' job of retrieving images with the given subject included, but how strict or lenient your class requisites are is quite important. For example, if one of your classes is 'dog' there may be hundreds of images procured from scraping that could display (examples off the top of my head, but you can get even more creative):

Ancient canine fossils
Wolves
Stuffed animal dogs
Hard mode dogs (i.e. partially occluded, variable lighting schemas, background color variation, intraclass/species variation)

Additionally, your computational tool will impact this. If you're using a neural network, the above issues can to some extent be accounted for, but a linear classifier would likely have difficulty adopting a broader/flexible view of your class.


Can a single neural network handle recognizing two types of objects, or should it be split into two smaller networks?

Well, I do not know what type of features you are giving to your neural network. However, in general, I would go with a single neural network. It seems that you have no limitation in resources for training your network and the only problem is resources while you apply your network.

The thing is that probably the two problems have things in common (e.g. both types of plates are rectangular). This means that if you use two networks, each has to solve the same sub-problem (the common part) again. If you use only one network the common part of the problem takes fewer cells/weights to be solved and the remaining weights/cells can be employed for better recognition.

In the end, if I was in your place I would try both of them. I think that is the only way to be really sure what is the best solution. When speaking theoretically it is possible that we do not include some factors.


How can I prevent the CNN from classifying a new input into one of the existing labels (it was trained with) when the input has a new different label?

You can introduce another class to your network - "not a book". After that, you will need to add new data to your dataset, random images that do not contain books to classify and train your network on that data. So when your network won't see a book it will output high probability for "not a book" class, if an image with a book will be shown to the network probability of the "not a book" class should be low.


Adversarial Q Learning should use the same Q Table?

should I let both 'players' use, and update, the same Q Table?

Yes this works well for zero-sum games, when player 1 wants to maximise a result (often just +1 for "player 1 wins") and player 2 wants to minimise a result (score -1 for "player 2 wins"). That alters algorithms such as Q-learning because the greedy choice switches beween min and max functions over action values - player 1's TD target becomes $R_{t+1} + \gamma \text{min}_{a'} [Q(S_{t+1}, a')]$ because the greedy choice in the next state is taken by player 2.

Alternatively, if the states never overlap between players, then you could learn a Q function that always returns the future expected return according the current player. This can be forced if necessary by making part of the state record whose turn it is. With this you need a way to convert between player 1 and player 2 scores in order to use Q learning updates. For zero-sum games, then the Q value from a player 2 state is negative of player 1's value for that same state, and vice versa, so the TD target changes for both players to $R_{t+1} - \gamma \text{max}_{a'} [Q(S_{t+1}, a')]$

The first option can result in slightly less complexity in the learned function, that might be an issue if you are using function approximation and learning a Q function e.g. using neural networks instead of a Q table. That may result in faster learning and generalisation, although it will depend on details of the game.

  Or would this lead to issues?

No major issues. I am performing this kind of training - a single Q function estimating a global score which P1 maximises and P2 minimises - for the Kaggle Connect X competition, and it works well.

I can think of a couple of minor things:

You may still want to have the ability for each player to be using a different version of the table or learned Q function. This would allow you to have different versions of your agent (e.g. at different stages of learning) compete, to evaluate different agents against each other. To do this, you have to write code that allows for multiple tables or functions in any case.
You need to keep track of how both players express and achieve their opposing goals when using the table, as you can already see by the modified TD targets above. This becomes more important when adding look-ahead planning, which is a common addition and can significantly improve the performance of an agent - during look-ahead you must switch between player views on what the best action choice is. It is possible to make off-by-one errors in some part of the code but not others and have the agent learn inconsistently.


How to deal with episode termination in Advantage Actor-Critic algorithm?

First a small note: I don't think your expression for $A(s_{T-1}, a_{T - 1})$ looks correct. If we assume that $V(s_T) = 0$ (i.e., assume that we cannot possibly reach the goal in one single step from $s_{T - 1}$), we have:

\begin{align}
A(s_{T-1}, a_{T-1}) &= r(s_{T-1}, a_{T-1}) + \gamma V(s_{T}) (1 - done_{T})  - V(s_{T-1}) \\
&= r(s_{T-1}, a_{T-1}) - V(s_{T-1}).
\end{align}

In this expression, we'd normally have that $r(s_{T-1}, a_{T-1}) = -0.1$, whereas you seem to have mistakenly taken $+0.1$ in your post.

Those details aside; yes, there can be sudden bumps of positive advantage estimates as you described. This is not a problem though, this is exactly what we'd expect to happen given your description of the environment.

You describe an environment in which the agent is likely going to be wandering around randomly (at least when it hasn't learned a good policy yet), and incurring negative rewards over and over again. This naturally leads to negative value estimates for all encountered states. Suddenly, it does something and the episode terminates; it receives a nice reward of $0$ rather than yet another negative reward (this actually "feels" like a bonus, a positive reward, something more than was expected). When your agent has not yet learned a good policy that can reach the better reward of $1$, this is indeed a good result, a good action, and it rightfully should get reinforced.

Because this event of the episode terminating is mostly uncorrelated with the state (I say "mostly", because in theory it probably ends up being a slightly rarer event in states close to the goal than in states far away from the goal), it will eventually (after sufficient training time) end up occurring approximately equally often in all states. From the perspective of an agent that is oblivious to the current time step, this will be perceived as an event that can simply occur by pure chance in a nondeterministic environment.

This is not necessarily a problem. It can slow down learning due to increased variance in your reward observations (which can be addressed by using low learning rates / large batch sizes), but Reinforcement Learning algorithms are almost always naturally built to handle nondeterministic environments, it can work this out, it can average out all the different outcomes observed for the same state+action pairs. This is not a problem that requires dealing with.

  My ideas:

  Differentiate between actual episode terminations and ones caused by the time limit, e.g. for them we will not replace next step value estimate with $0$.
  Somehow add $t$ to the state such that the value function can learn to anticipate the termination of the episode.

The first idea fundamentally changes the quantity that your algorithm is learning, it will essentially make your learning algorithm incorrect. There's always a chance that it might still appear to learn something useful (many Machine Learning/Reinforcement Learning algorithms can still appear to be okay even when there are bugs/technically incorrect parts), but it'll very likely perform worse.

The second idea, while not necessary as explained in my answer above, may still be beneficial to learning speed provided it is done well. It may help because it can add to the power of your algorithm to "explain" its observations, and more importantly "explain" the variance in its observations.

The main problem I see with adding $t$ to your input is that it is not naturally a binary variable. Very often you'll find that we're just using a bunch of binary inputs in (Deep) Reinforcement Learning algorithms. When all inputs are of the same magnitude like that, it tends to be easier to get the learning process to run well, tune hyperparameters like learning rate, etc. If you suddenly plug in an additional input which can take significantly larger values (like straight up adding $t$ as an input), this will be more difficult. Adding $\frac{t}{T}$ as an input may be better, since that will always still be bounded between $0$ and $1$.


When does backward propagation occur in n-step SARSA?

The important part, where you can see a single reward value is used for $n$ different updates, is the part where a sum of $R_i$ values with $i$ ranging from $\tau + 1$ to $\tau + n$ is assigned to $G$.

So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed $R_i$ values. Each of those $R_i$ values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).


Can AI 'fix' heavily compessed videos/photos?

Check out source code to DeepImagePrior it does a remarkable job guessing what's missing to repair images with a variety of damage.


With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long as the discriminator returns "real" (vs. "fake") the generator "wins".

The hope is that as the generator and discriminator are trained simultaneously, each will attempt to capture the faults of the other. As the discriminator gets better for one mode in telling real from fake, the generator will focus more on the other modes where it will have an easier time fooling the discriminator on.

But that isn't the only way things can happen. When the discriminator gets better on distinguishing one mode, the generator can instead concentrate on that mode. Instead of spreading its training over different modes, it can instead attempt to get really good at producing samples within a particular mode. As long as the discriminator can be fooled into reporting "real" for the samples being produced, the generator gets the same reward as if it were to spread itself across multiple modes.

Think about the forger/detective analogy. Instead of a forger trying to learn how to be good at making a range of paintings (Caravaggios, van Goghs, Warhols, etc.) the forger can concentrate on being really good at making a particular style of painting (e.g. Dutch Masters) or even a particular artist. Someone who spends all their time perfecting their Rembrandt forgeries will do a much better job evading detection than someone who tries to be good at both Picasso and Rembrandt (and Klee and Miró and ...)

Variational Autoencoders are different, though. With a VAE, the latent space is built from all of the input examples. There's no way for a VAE to concentrate on a particular mode. All of the training examples are being mapped to the same latent space. If, for some reason, the decoder network decided to concentrate on a single mode, mapping the latent space to outputs of just that mode, then all the training examples from the other modes would have poor recovery. This means there would be a strong loss and gradient correcting that for the decoder training. The decoder training is balanced across all the input examples/modes, so the decoder needs to be similarly balanced when decoding the latent space. GANs don't have this similar balance, as the generator doesn't necessarily need to be balanced in its output - just so long as it can reliably fool the discriminator.

That doesn't mean that VAEs don't have similar issues, though they're not identical to mode collapse. The problem with VAEs is that each individual example is (in extremis) being mapped to the same random distribution in the latent space. This can encourage the decoder to ignore the latent variable input, generating output more-or-less arbitrarily. There's a number of approaches recently (in particular Adversarial Autoencoders as well as InfoVAE and the closely related Wasserstein Autoencoders) which attempt to remedy this by constraining the latent space distribution on an ensemble level, rather than on an individual training example basis.