According to Wikipedia:

  A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.

Answer to your question:

To build any neural network model we assume the train, test and validation data are coming from a probability distribution. So, if you produce a neural network model based on statistical data then the network is a statistical model.

Moreover, neural networks' cost function is generally a parametric model and parametric modes are statistical models.

Please look at Goodfellow's Deep Learning book chapter Deep Feedforward Networks page 174 and 175.

From Goodfellow's book

  Fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. In most cases, our parametric model deﬁnes a distribution $p(y \mid x; \theta)$ and we simply use the principle of maximum likelihood.

In conclusion, ANNs (e. g. MLP, CNN, etc.) are statistical models


Are my steps correct for a proper classification of a sick brain?

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging

There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK

There is a DataCamp course that walks you through most of the process but instead of a classifier they use deep learning to reconstruct brain images.  They provide a link to their MNIST classifier example which should be easy to adapt for your purpose.  See: Reconstructing Brain MRI Images Using Deep Learning (Convolutional Autoencoder)

ResearchGate has a thread that may help:
What is the appropriate way to use Nifti files in deep learning?


What are mono-variable and multi-variable neural networks?

I am not sure about this, because most (if not all useful) neural networks are multivariable neural networks (i.e. they contain multiple parameters). Even the perceptron usually contains more than one parameter, so that terminology isn't clear even to me. Maybe they are referring to the number of inputs (sometimes called variables), but I don't see why this distinction in this context would make sense.

What are static and dynamic neural networks?

To answer this question, I will first quote an excerpt from this document (written in Spanish) to provide some context to Spanish speakers (I am not a Spanish speaker, but I understand 99% of it).

Un primer intento de clasificación puede separ estáticos y dinámicos o recurrentes (fig.2.1)
Los modelos estáticos realizan un mapeo entre entrada y salida. Despreciando el tiempo de procesamiento interno, la salida se obtiene en forma inmediata en función de la entrada, no existe memoria ni dinámica de estados en el sistema neuronal.
Por el contrario los sistemas recurrentes si la poseen, son sistemas realimentados que ante un estimulo de entrada evolucionan hasta converger a una salida estable.
Casos tipicos de ambos sistemas son el Perceptrón (Rosemblatt, 1960a) (de una o múltiples capas) y la memoria asociativa de Hopfield, respectivamente
(Tank, 1987).

So, in this document, the word "dynamic" and "recurrent" are being used interchangeably. An example of a static (i.e. non-recurrent) neural network is the perceptron. An example of recurrent (or dynamic) neural network is the Hopfield network.
Anyway, I recommend you contact the author of that article to ask for clarification (especially, about the mono-variable NNs)!


Is the expected value we sample in TD-learning action-value Q or state-value V?

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that?

MC and TD are methods for associating value estimates to time step based on experienced gained in later time steps. It does not matter what kind of value estimate is being associated across time, because all value functions are expressing the same thing in general, which is the expected return conditioned on a "current position" within the MDP. In MC the association is directly with a sampled return, in TD with a sampled combination of immediate reward and a later value estimate - most commonly in TD the same kind of value estimate (e.g. matching later state value estimates to state values).
Both approaches can be analysed and used from the perspective of both state value (V) and action value (Q) functions. They also apply to other value functions - e.g. afterstate values.
It is quite common for textbooks and tutorials to use the slightly simpler state value function to explain how MC or TD learning work in general, outside of being used for any purpose. You can also use the state value function for model-free policy evaluation in MC and TD.
However, without a model, you cannot use state value function for control (i.e. to learn an optimal policy). To pick the best action using state values, you need to do something like this:
$$\pi(s) = \text{argmax}_a [ \sum_{r,s'} p(r,s'|s,a)(r + \gamma v(s'))]$$
The problem here is that $p(r,s'|s,a)$ is a model of the environment. So, if it is needed, the control method would not be model-free.
Hence when you learn about MC or TD in a control scenario, model-free methods to learn optimal policies, then you generally need to use an action value (sometimes you can use an afterstate value, if the action involves choosing the next state directly).
With an action value function, the greedy policy becomes:
$$\pi(s) = \text{argmax}_a q(s, a)$$
This does not refer to any model of the environment. So it can be used when you have none.


How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).
You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they propagate normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.
However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.
There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the universal approximation theorem, whether it preserves normalisation, and so on. You may or may not care about some or any of those.
The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially tanh) because they are nicely bounded and very fast to compute, but most importantly because they work for my use cases. Others recommend leaky ReLU for the input and hidden layers as a go-to function if your NN fails to learn. You can even mix and match activation functions to evolve NNs for fancy applications.
At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.


How could decision tree learning algorithms cope with imbalanced classes?

Decision Tree learners, on their own, are not a good way to deal with imbalanced data. The most commonly used algorithms, by default, make no attempt to address this problem.

If you look carefully at the three sources you post, you will find that they actually all agree on this point.

Two of the sources actually propose methods of addressing this shortcoming, by making adjustments to the decision tree learning algorithms. The proposed adjustments are essentially standard solutions to these problems, being applied to decision trees.

An example technique, discussed in the first paper you reference, is changing the weightings of the classes. An inefficient/approximate way to do this is to increase the number of examples from the minority class. For example, if you had an 80/20 split, you could add 3 new copies of each minority class example to move to an 80/80 = 50/50 split. Of course, if you add new data points, your algorithm may take longer to run. Instead, you can just modify the weightings of the classes in your optimization function. This approach is algorithm-specific, and will depend on your loss function, but achieves the same effect, just without needing to increase the number of points you use.


From your matrix definitions, the issue is S always singular so it can never be inverted

I reimplemented the computation with numpy and here are the numbers

H = np.array([[1.0, 0.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0]])

P = np.identity(5) * 1000

R = np.array([[100, 0, 0, 0], [0, 100, 0, 0], [0,0,0,0], [0,0,0,0]])

S=H.dot(P.dot(np.transpose(H)))+R

and S is

array([[1100.,    0.,    0.,    0.],
       [   0., 1100.,    0.,    0.],
       [   0.,    0.,    0.,    0.],
       [   0.,    0.,    0.,    0.]])

Basically the definition of H with its all zeros lines cancels part of the information (make it unobservable) and you also do not have any observation noise component for them


What is the difference between actor-critic and advantage actor-critic?

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "bootstrapping" by using some sort of prediction of value.

Advantage Actor-Critic specifically uses estimates of the advantage function $A(s, a) = V(s) - Q(s, a)$ for its bootstrapping, whereas "actor-critic" without the "advantage" qualifier is not specific; it could be a trained $V(s)$ function, it could be some sort of estimate of $Q(s, a)$, it could be a variety of things.

In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict $V(s)$. Combined with an empirically observed reward $r$, they can then compute the advantage estimate $A(s, a) = r + \gamma V(s') - V(s)$.


Understanding the "unroling" step in the proof of the policy gradient theorem

The unrolling step is due to the fact you end up with an equation that you can keep expanding indefinitely.
Note that we start with calculating $\nabla v_\pi(s)$ and arrive at
$$\nabla v_\pi(s) = \sum_a\left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \sum_{s'}p(s'|s,a) \nabla v_\pi (s') \right]\;,$$
which contains a term for $\nabla v_\pi(s')$. This is a recursive relationship, similar to the bellman equation, so we can substitute in a term for $\nabla v_\pi(s')$ which will be a term similar just with $\nabla v_\pi(s'')$. As I mentioned, we can do this indefinitely which leads us to
$$\nabla v_\pi(s) = \sum_{x \in \mathcal{S}} \sum_{k=0}^\infty \mathbb{P}(s\rightarrow x, k, \pi) \sum_a \nabla \pi(a|x) q_\pi(x,a)\;.$$
We need the term $\sum_{x \in \mathcal{S}} \sum_{k=0}^\infty \mathbb{P}(s\rightarrow x, k, \pi)$ because we want to take an average over the state space, however due to unrolling there are many different $s_t$'s that we need to average over (this comes from the $s',s'',s''',...$ in the unrolling) so we also need to add the probability state of transitioning from state $s$ to $x$ in $k$ time steps, where we sum over an infinite horizon due to the repeated unrolling.
If you are wondering what happens to the terms $\pi(a|s)$ and $p(s'|s,a)$ terms and why they are not explicitly shown in this final form, it is because this is exactly what the $\mathbb{P}(s\rightarrow x, k, \pi)$ represents. The average over all possible states accounts for the $p(s'|s,a)$ and the fact that we follow policy $\pi$ in the probability statement accounts for the $\pi(a|s)$.


Monte Carlo epsilon-greedy Policy Iteration: monotonic improvement for all cases or for the expected value?

I think this equation answer your question:
$$ q_{\pi^{i}}(s,\pi^{i+1}(s)) = \mathbf{E}[q_{\pi^{i}}(s,\pi^{i+1}(s))] = \sum_{a \in A}\pi^{i+1}(a|s)q_{\pi^{i}}(s,a)$$

value of the Q while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$ is equal to the expected q value while taking action from policy $\pi^{i+1}$ and thereafter following the policy $\pi^{i}$. And for the second part of your question the answer is:

$$ V_{\pi^{i}}(s) = q_{\pi^{i}}(s,\pi^{i}(s))$$

state Value function following the policy $\pi^{i}$ is the same as action-value function while taking action from policy $\pi^{i}$ and thereafter following the policy $\pi^{i}$.


What are the dangers of AI applications for nuclear industry?

Any technology in the nuclear industry represents variance--it may be an improvement in safety or efficiency, or it may contain some unseen defect that allows a catastrophe to happen.

But the simple possibility of harm isn't enough to swing the decision one way or the other. The application of AI methods--whether to the real-time control of plant variables, or the early detection of problems, or to the design of plants and their components--seems likely to be as beneficial as in other realms.

For example, check out the publication list of a lab active in this area. Their paper I'm most familiar with is one in which they build a fault detector paired with a fault library classifier, so that the operators can be alerted not just that something is abnormal but what fault has probably occurred. This is done in such a way that standardized plants (such as, say, the French nuclear system) can share records with each other, meaning that any plant has the experience of every plant at their fingertips.


How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.
I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by changing state spaces?
One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.
Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.
Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.
In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots.
If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.
I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller.
See this paper (Section 3.2) for an example of the aforementioned technique.
Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. This paper shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.
Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.
When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.


How do I statistically evaluate a ML model?

There are a lot of ways to evaluate the performance of an ML model.  You mentioned AUROC and AUPRC.  Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc.  You can see a good outline of them here.

It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data.  You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives.  I recommend you read Neil Slater's answer at https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228.  He gives some good advice on the subjectivity of sentiment analysis classification and points out a labeled data set of Tweets which you might be able to use to test your classifier.

I also found this Kaggle competition which has a test set that might be of help to you: Angry Tweets


What are most commons methods to measure improvement rate in a meta-heuristic?

You can use one of your suggested methods to calculate the relative improvement, but you need also to define a threshold value $\epsilon$ that determines when a relative improvement is negligible, and so you can terminate the algorithm. To be more concrete, you could terminate the genetic algorithm when, for example, the following condition is met
$$
|f(x_{t+1}) - f(x_{t})| < \epsilon, \tag{1}\label{1}
$$
where

$f$ is the fitness function
$t$ is the iteration number of your iterative algorithm
$x_t$ is a solution at iteration $t$
$|\cdot|$ is the absolute value
$\epsilon$ is the threshold value (a hyper-parameter of your algorithm), which is typically a small number (e.g. $10^{-6}$), but this also depends on the magnitude of the fitness of the solutions

This stopping criterion in \ref{1} (sometimes known as the absolute error) is not the only possible one. For example, you can stop the genetic algorithm when it has run for a certain maximum number of iterations. The stopping criterion (or criteria, i.e. you can use more than one stopping criteria) that you choose probably depends on what you want to achieve and this is not just specific to genetic algorithms, but, in general, this applies to any iterative numerical algorithm.
The MathWorks article How the Genetic Algorithm Works enumerates several stopping criteria for genetic algorithms.
If you want to know more about the topic, maybe take a look at this paper On Stopping Criteria for Genetic Algorithms (2004) by Martín Safe et al.


Is it mostly the case to train with available models

Yes, it is recommended to start with pre-trained model if you don't have high-end hardware. You can use a pre-trained model for fine-tuning (their trained weight as your initial weight) or use it as feature extractor (you remove few last layers, and then train it).
Why we need a pre-trained network?

Because training a good deep model takes a lot of time and needs a lot of hardware. Some good models, like DenseNet, ResNet, even VGG16 need days of training. You can read from VGG original paper:

On a system equipped with four NVIDIA Titan Black GPUs, training a single net took 2–3 weeks depending on the architecture.

Sometimes our case is similar to the dataset of pre-trained models out there. For example, if we need to classify some images of flower type, even our case has different class, we can use some first Convolutional layers of pre-trained model and use it as a feature extractor.

You can read more about transfer learning from this paper or this page.


What is the relation between the context in contextual bandits and the state in reinforcement learning?

The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in state $s_t$ not only affects the reward $r_r$ that the agent will get but it will also affect the next state $s_{t+1}$ the agent will end up in, while, in contextual bandits (aka associative search problems), an action $a_t$ in the state $s_t$ only affects the reward $r_r$ that you will get, but it doesn't affect the next state the agent will end up in. The typical problem that can be formulated as a contextual bandit problem is a recommender system.
In CBs, like in RL, the agent also needs to learn a policy, i.e. a function from states to actions, but actions that you take in a certain state are independent of the actions you take in other states.
So, as Sutton and Barto put it (2nd edition, section 2.9, page 41), contextual bandits are an intermediate problem between (context-free) bandits (where there is only one state or, equivalently, no state at all) and the full reinforcement learning problem.
Another important characteristic of many RL algorithms, such as Q-learning, is that they assume that the state is Markov, i.e. it contains all necessary info to take the optimal action, but, of course, RL is not just applicable to fully observable MDPs. In fact, even Q-learning has been applied to POMDPs, with some approximations and tricks.
Regarding the use of neural networks to approximate $q(s, a)$ or a policy in CBs, in principle, this is possible. However, given that the optimal action in a state $s$ is independent of the optimal action in another state $s'$, this is probably not useful, but I cannot guarantee you that this has not been successfully done, because I've not yet read the relevant literature (maybe someone else will provide another answer to address this aspect).


Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

I believe the claim is true. Here is my attempt at a proof.
Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$.
The value $V_\alpha^*(s)$ is the expected sum of discounted rewards under an optimal policy $\pi_\alpha^*$, i.e.,
\begin{equation}
      V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[\sum\limits_{t=0}^{\infty}\gamma^t\left( \alpha R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)R_1(s_t, \pi_\alpha^*(s_t)) \right)\middle| s_0 = s, \right],
\end{equation}
with the expectation taken with respect to the steady state distribution $\rho_\alpha$ of states under $\pi_\alpha^*$.
In the following, I drop the condition $s_0=s$ for conciseness, but you can assume it's in each expectation.
Now, break up the sum:
\begin{equation}
     V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[ \alpha\sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)\sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Then, by linearity of expectation:
\begin{equation}
      V_\alpha^*(s) = \alpha\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Note that the first expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_0$, and the second expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_1$.
We already know that $\pi_0^*(s)$ is optimal in $\mathcal{M}_0$ with reward function $R_0$, and $\pi_1^*(s)$ is likewise optimal in $\mathcal{M}_1$ with $R_1$. Further, as per your assumption, $\pi_0^*(s) = \pi_1^*(s)$.
So $\pi_\alpha^*$ can be at most as good as $\pi_0^*$ with reward function $R_0$ (resp., with $R_1$):
\begin{equation}
      V_\alpha^*(s) \leq \alpha\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_0^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_0^*(s_t)) \right].
\end{equation}
Note that we know take the expectation under the steady state distribution $\rho_0$ of $\pi_0^*$ instead.
Thus, we have shown that $V_\alpha^*(s) \leq \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
Now it remains to argue that the case with a strict less than relation is not possible.
Suppose this were the case, and we would have $V_\alpha^*(s) < \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
But then $\pi_0^*$ would attain a higher value than $\pi_\alpha^*$ in $\mathcal{M}_\alpha$, which is a contradiction (because we assumed that $\pi_\alpha^*$ is an optimal policy for $\mathcal{M}_\alpha$).
Thus, $V_\alpha^*(s) = \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$ and furthermore, acting according to $\pi_0^*$ is optimal also in $\mathcal{M}_\alpha$.


Chatbot using Machine Learning vs Programming

In my view ML does not work very well for conversational AI systems. It is generally alright for intent recognition, so getting what the user wants if they ask a question ("I want to book a flight?", "What is the weather in London?"), but anything after that quickly becomes difficult to handle, especially multi-step conversations that go beyond simple question/answer pairs.

My suggestion would be to plan possible dialogues out as flow charts (more like trees/graphs, as there can be multiple branches at any point), and then write a program that interprets the graph based on user input and gives appropriate replies. You will also want to have some conversational memory to keep track of any information the user has mentioned. That is also tricky to do in a ML system.

For a very simple framework to start off with, have a look at ELIZA. It's half a century old, but you can still use it as a starting point.

(Disclaimer: I work for a company that makes conversational AI systems)


What is the difference between pixel-based object recognition and feature-based object recognition?

Pixel based object recognition, like the name says, works by analyzing the individual pixels of an image. For example: You analyze an image with a lot of different shades of blue and some grey pixels - you might assume that this is the picture of a plane in the sky or a ship in the water.

You could also look for similar pictures by calculating the difference between each individual pixel of 2 images and sum up the differences. The smaller the sum of the difference, the more the pictures look alike.

It depends on your field of work, but those methods will most likely not be good enough for solid image recognition.

You also mentioned neural networks. Of course you can feed the raw pixel data to a neural network, but that's not state of the art. You will almost always use a CNN, which brings me to your next point.

Feature based object recognition, like the name says, tries to extract certain features from an image to perform classification. You use several convolutional layers for analysis before you feed the output to a neural network. This architecture is called CNN (Convolutional Neural Network).

What those features actually look like is hard to explain without showing examples. I recommend to watch the following two lectures from the Stanford course CS231n:

Lecture 2 | Image Classification
Lecture 5 | Convolutional Neural Networks

The first one explains image classification in general and shows some implementations for pixel based algorithms. The second one dives into CNNs and explains feature based approaches. I highly recommend to watch the full series, but if you are in a hurry, those 2 lectures will tell you what you want to know.


How to keep track of the same person detected in different frames of a camera?

One way to solve the issue may be a tracker (like Kalman). It would be faster and easier approach than neural nets.

If you insist about solving this issue through neural nets, then some magic and creativity is needed. Based on the nature of tracking, you need to feed multiple frames to predict next location of the object and check if there is any object close by. So, you may consider combining RNNs with CNNs to predict next bounding box to track and replace Kalman filter's prediction with RNNs. (check)


Is it possible to train the neural network to solve math equations?

Yes, it has been done!

However, the applications aren't to replace calculators or anything like that. The lab I'm associated with develops neural network models of equational reasoning to better understand how humans might solve these problems. This is a part of the field known as Mathematical Cognition. Unfortunately, our website isn't terribly informative, but here's a link to an example of such work.

Apart from that, recent work on extending neural networks to include external memory stores (e.g. Neural Turing Machines) tend to use solving math problems as a good proof of concept. This is because many arithmetic problems involve long procedures with stored intermediate results. See the sections of this paper on long binary addition and multiplication.


Why different images of the same person, under some restrictions, are in a 50 dimension manifold?

The number 50 is essentially just a guess based on results when compressing and/or generating data of a certain type. The variables such as "the three translations of the body, the three rotations of the head and the independent movements of the face's muscles" are examples only. There is no known formal map with well-defined parameters that defines a well understood manifold of "clear images of this person" in natural images. The lecturer has not constructed such a map as far as I can tell, but has done some related experiments.
Experimentally, it is possible to establish parameter vectors that work, with models like Variational Autoencoders and Generative Adversarial Networks. Depending on the size of the target image, and amount of variation in subject matter that you want to allow for (pose, lighting, clothing, hair style, makeup, camera properties etc), you will end up with different sizes of embedding vectors that appear to capture the important variations. When dealing with multiple people, it is common to see vector sizes of 64, 128, 256.
The lecture suggests compressing images with a clear background, consistent lighting, same person with only changes being in pose. Around 50 dimensions for this relatively simple image space seems reasonable, given facial recognition engines that work well in a more complex domain using 128 dimensional embeddings.
I expect that the lecturer has seen experimental evidence that a vector of 50 dimensions performs well at representing all variations in these images, plus smaller vectors perform measurably worse and larger vectors do not perform better. This experiment is possible by constructing something like a VAE with a specific size of embedding vector, training it, then measuring loss when reconstructing a set of test images.


Why object detection algorithms are poor in optical character recognition?

Good question! Using Yolo to recognise characters would be a good experiment to try. It may be because of the density of characters on a page -- systems like Yolo are very good at detecting a small number e.g. 2,3 or 10, objects, but don't work so well when the number of objects  is the hundreds as you might have with OCR. A better approach might be to try face detection methods that work well with large crowds.


What are the pros and cons of sparse and dense rewards in reinforcement learning?

It is unusual to refer to this difference as "pros and cons" because that term is often used to make comparisons between difference choices. Assuming you have a specific problem to solve, then whether or not the rewards are naturally sparse or dense is not a choice. You cannot say "I want to solve MountainCar, I will use a dense reward setting", because MountainCar has (relatively, for a starting problem) sparse rewards. You can only say "I won't attempt MountainCar, it is too difficult".
In short however, your assessment is correct:

if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal

There is not really any other difference at the top level. Essentially, sparser rewards make for a harder problem to solve. All RL algorithms can cope with sparse rewards to some degree, the whole concept of returns and value backup is designed to deal with sparseness at a theoretical level. In practical terms however, it may take some algorithms an unreasonable amount of time to determine a good policy beyond certain levels of sparseness.

On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.

Imitation learning is one of many techniques available to work around or deal with problems that have sparse reward structure. Others include:

Reward shaping, which attempts to convert a sparse reward scheme to a dense one using domain knowledge of the researcher.

Eligibility traces, which back up individual TD errors across multiple time steps.

Prioritised sweeping, which focuses updates on "surprising" reward data.

Action selection planning algorithms that look ahead from the current state.

"Curiousity" driven reinforcement learning that guides exploration to new state spaces independently of any reward signal.


How does batch normalisation actually work?

A batch normalisation layer is like a standard FC layer but instead of learning weights and bias', you learn means and variances and scale the whole layer by said means and variances.
Fact 1:
Because it behaves just like a normal layer, and can learn, with the right structure it will learn to get a high enough accuracy.
Fact 2
Disabling learning on a batch norm layer is just like disabling learning on any other layer. It will not updated any of its parameters, and in this case the parameters are the means and variances, and so these will not be updated.


Determining minimal state representation for maze game

I don't get why the (a).i. uses (,) coordinates whereas (b).i. uses boolean list. I guess they can be used interchangeablly right?

No these cannot be used interchangeably. The situations really are different enough that completely different state representations are necessary.
In the case of (a) you know the current bug's location, and nothing else about the problem changes over time. So any state representation that captures a single bug's location is valid. The (x,y) coordinates are a natural choice given the framing of the question, but also you could enumerate all the possible grid positions for the bug, giving the same size of state space.
In case of (b) you do not know where either bug is, but you don't need to care about the difference between each bug to track it separately (this is becasue your goal state does not rely on the identity of either bug). Therefore the state representation has to capture where any bug could possibly be.
The proposed solution is simple, but you could also do it by tracking possible locations of each bug. That could work with $17^2$ $(x_1,y_1,x_2,y_2)$ tuples initialised to all possible position combinations that the two bugs start in. The state space in that case is large (it is not just $17^2$, but includes all valid reachable combinations of the combined $17^2$ $(x_1,y_1,x_2,y_2)$ tuples) and difficult to assess because many state combinations are not reachable. Also note this alternative proposal does differentiate between the bugs - you could potentially improve upon that by removing start states that simply swap between bug locations, although this is still not as efficient as the binary representation.
Technically the proposed solution for (b) also works the same for any number of bugs that are spread out in the maze and want to all be in the same location at the end.


How do I get multiple loss per sample in keras evaluate?

You cannot achieve that from a single model.evaluate(...) call. But you could always do evals = [model.eval(X[i:i+1], Y[i:i+1]) for i in range(len(X))].

Note if you get the tensors yourself, you can get the backend session from keras.backend.get_session() and get anything you want by making your own sess.run() calls


DQN rgb input channels problem using pytorch

You can reshape it to (12, H, W) using NumPy reshape function. By the way, this will only increase the complexity of this problem. If you want to practice RL then just get the idea from their code and try implementing on some other problem/game.


Handling emotion in informal text (Hi vs HIIIIII!!!!)?

These kinds of repetitions in text can place recurrence demands on learning algorithms that may or may not be handled without special encoding.

Hi.
Hiiii!
HIIIIIIIII
Hi!!!!!!!!!!!!!!

These have the same meaning on one level, but different emotional content and therefore different correlations to categories when detecting the value of an email, which in the simplest case is the placement of a message in one of two categories.

Pass to a recipient
Archive only

This is colloquially called spam detection, although not all useless emails are spam and some messages sent by organizations that broadcast spam may be useful, so technically the term spam is not particularly useful. The determinant should usually be the return on investment to the recipient or the organization receiving and categorizing the message.

  Is reading the message and potentially responding likely of greater value than the cost of reading it?

That is a high level paraphrase of what the value or cost function must represent when AI components are employed to learn about or track close to (in continuous learning) some business or personal optimality.

The question proposes a normalization scheme that truncates long repetitions of short patterns in characters, but truncation is necessarily destructive. Compression of some type that will both preserve nuance and work with the author's use of Word2Vec is a more flexible and comprehensive approach.

In the case of playful sequences of characters it is anthropomorphic to imagine that an artificial network will understand playfulness or anger, however existing learning devices can certainly learn to use character sequences that humans would call playful or angry in the function that emerges to categorize the message containing them. Just remember that model free learning is not at all like cognition, so the term understanding is placing an expectation on the mental capacities of the AI component that the AI component may not possess.

Since no indication that a recurrent or recursive network will be used but rather the entire message is represented in a fixed width vector, so the question becomes which of these two approaches will produce the best outcomes after learning.

Leaving the text uncompressed so that an 'H' character followed by ten 'i' characters is distinct as a word from an 'H' character followed by five 'i' characters
Compressing the text to "Hi [9xi]" and "Hi [4xi]" respectively or some such word bifurcation.

This second approach produces reasonable behavior with other cases mentioned, such as "" pre-processed into " [2x]". What the algorithm in Word2Vec will do with each of these two choices and how its handling of them will affect outcomes is difficult to predict. Experiments must be run. Three things are advisable courses of action.

Build a test fixture to allow quick evaluation of outcomes for various trials.
Experiment with dilligence. Don't leave any potentially interesting case untried.
Label as much production data as reasonably possible and use that as well as the canned data so that the above options can be evaluated in permuted combinations with the differences in pattern distribution between the canned and live data.


Is it possible to know the distance objects are from camera based on only knowing one object's height?

Monocular depth estimation basically does this and it implicitly brings in knowledge about object size. But you need to have prior information.


Is there a neural network that accepts both the current input and previous output?

You want to look at recurrent neural networks.


Is it a good idea to use different width and height of the kernel in a CNN?

It depends on your application. In case of text recognition, non-uniform kernels are used since the information about text is less on the horizontal axis and more on the vertical axis.
If in your case it is applicable then, it will be good idea. But, if it is not you are better off using a smaller uniform kernel (2x2, maybe). You can also zero-pad your image to make it uniform before putting it through convolutions. Also, check if you are doing 'valid' or 'same' padding in your convolutions since 'valid' convolutions chip away at your image dimensions.


Why is GLIE Monte-Carlo control an on-policy control?

In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are calculated. The last line in the "pseudocode" tells you that the policy $\pi$ will be a new $\epsilon$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.
If the last line was $\mu \leftarrow \epsilon\text{-greedy}(Q)$, it would be an off-policy method.


Are humans superior to machines in chess?

Losing games to computers because of mistakes made under time pressure was probably a thing about 20 years ago, when Kasparov lost to DeepBlue after such a mistake(correction: it was Kramnik with the blunder, not Kasparov (see edit 2)). But after Kramnik's loss in early 2000s, no world champion ever tried to play against a computer (to my knowledge). Nowadays, there are computer only tournaments among programs with ratings well above 3300 (for comparison, Carlsen's peak rating was around 2880), and it is not uncommon for computers to make moves with no apparent meaning to humans.
No time limit for humans also mean no time limit for computers so I doubt any human can win a single game against a computer. Older models like Stockfish 8 depend on their computational power as it can look at several millions of position per second, Google AlphaZero  managed to beat Stockfish with 80000 positions per second: they don't seem to depend on brute force calculations any more. Keep in mind that this is without any prior knowledge of openings etc, they are trained using reinforcement learning, starting from the rules of the game only. From there, they can develop their own strategies and implement them without making any mistakes. They create their own openings from scratch, so existing libraries is not going to be very useful.
I am not aware of any research on this but lack of a challenge from humans is probably enough evidence. Also, Grand Masters regularly use chess engines in their training routine to analyze positions, so there is that.
A few years ago there was a game between Stockfish against GM Nakamura + Rybka, which Stockfish won. It is possible that human GM + Stockfish might have better chances against AlphaZero in correspondence without any time limits, but we probably will never know.
Here is an interview with Carlsen after a game, very interesting to show what he thinks about AlphaZero.
Both Kramnik and Kasparov made serious mistakes in their matches against computers. Kasparov resigned in a drawn position and missed a knight sacrifice, and Kramnik missed a mate in 1.


First thing, you need to give more credit to more reliable users. You can establish the credit from amount of data they send, and a feature, where other users can review other's feed and classify it. From there, you will have a measure of certainty to what data is good and what is not.

You will need to implement a centralized server, unless you're trying to do some kind of a peer-to-peer trust systems, but I don't think smartphones are powerful enough to do training themselves.

You will need big machines for training NNets. Don't trust users to have them. You would end up with tons of badly trained NNets, which don't make for a good CoM.


Tensforflow schedule - does not change boundaries

After 100 steps learning rate will switch from 1e-2 to 1e-24.
What exactly is the problem here? Are you confusing steps with epochs?


Why are we using all hyperparameters in RL?

In RL, there are episodic and non-episodic tasks (or problems). In episodic tasks, each episode proceeds in time steps.

For example, most games are episodic tasks. For instance, in a football championship (e.g. Premier League), each football match during the whole season is an episode. In this example, each minute (or second) of a football match can be considered a time step of the the episode (that is, the football match).

The parameter total_episodes thus specifies the number of episodes of your RL episodic problem. Similarly, max_steps specifies the maximum number of time steps per episode.

Why do we also need total_test_episodes? In machine learning, when building a model, there are usually two phases: the training phase and the test phases. In the training phase, you use the training dataset to learn the parameters of the model. In the test phase, you test the performance (e.g. the total return, in the case of RL) of the model. Hence, total_test_episodes is used to specify this hyper-parameter.

In deep RL, a (deep) neural network (NN) is used to represent either a value function or a policy (which is also a function). In machine learning, neural networks are usually trained using an optimisation method, like gradient descent (GD) and the back-propagation (BP), which is used to compute the gradient of the objective function with respect to the parameters of the model. In the case of deep RL, the parameters of the NN that represents either the value function or policy are also learned using a similar approach. In this context, the learning rate is a hyper-parameter that determines the "strength" of the update step of the optimisation algorithm. More concretely, in the case of GD, the update step is

$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \alpha \nabla f(\mathbf{\theta} _{n})$$

where $\mathbf{\theta}_{n+1}$ is a vector containing the parameter of your model (in this case, a NN), $\alpha$ is the learning rate and $\nabla f(\mathbf{\theta} _{n})$ is the gradient of the objective function $f$. Hence, learning_rate specifies the value of $\alpha$.

In your case, learning_rate can actually specify the value of the learning rate of your RL algorithm. For example, in the case $Q$-learning, the learning rate, which we can denote by $\alpha$, also specifies the "strength" of the update.

Similarly, the gamma is a hyper-parameter of your RL algorithm. For example, in the case $Q$-learning, the parameter $\gamma$ (the discount factor) which determines the contribution of the estimate of the $Q$ value of the next state (while taking the greedy action from that state) to the $Q$ value that is being currently updated.

$Q$-learning is an off-policy RL algorithm, which means that it uses a behaviour policy that is possibly different than the policy it tries to estimate. The usual behaviour policy is the $\epsilon$-greedy (with probability $\epsilon$ a random action is taken in a certain state and with probability $1 - \epsilon$ the greedy action is taken). The parameter epsilon specifies this hyper-parameter. In this context, an initial $\epsilon$ that specifies the exploration rate is provided. As the training progresses, the estimate of the optimal value function should become more accurate. In that case, you want to explore less and follow more your estimate of the optimal value function, so min_epsilon is used to specify the lowest $\epsilon$.

The parameter decay_rate is used to specify the value of the decay rate. Have a look at https://stats.stackexchange.com/a/31334/82135 for more info.


Confused with backprop in pytorch with BCE loss

BCELoss ( Binary Cross Entropy Loss) is used for binary classifier, which is a neural network that have a binary output, 0 or 1. It is not used for multi-output neural network like your case. For that kind of networks, you can use MSELoss or CrossEntropyLoss as your loss for the network.
For the calculation of BCE, it is shown on pytorch documentation.
https://pytorch.org/docs/stable/nn.html#BCELoss
For across batch, it either sum the loss or take the mean. you can set that through the reduction="" argument.
As i said, the loss is either summed or taken the mean. Then there will be a single value so pytorch can do backdrop.

Hope I can help you


Apart from the state and state-action value functions, what are other examples of value functions used in RL?

Advantage function: $A(s,a) = Q(s,a) - V(s)$
More interesting is the General Value Function (GVF), the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $V(s)$. The GVF is defined on page 459 of the 2nd edition of Sutton and Barto's RL book as
$$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$
where $C_t \in \mathbb{R}$ is the signal being summed over time.
$\gamma(S_t)$ is a function $\gamma: \cal{S}\to[0,1]$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. Some call it the continuation function.
Also of note are the differential value functions. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $R_{t+1}-r(\pi)$, where $r(\pi)$ is the average reward under policy $\pi$.
$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$
$$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$
The differential value functions assume that a single fixed value of $r(\pi)$ exists. That is, they assume the MDP is "ergodic." See section 10.3 of Sutton and Barto for details.


Convergence of semi-gradient TD(0) with non-linear function approximation

Apparently there is an example of non-convergence for semi-gradient sarsa, according to Rich Sutton (check slide 35). I guess TD(0) is not so different. So, probably your approximator will need to satisfy certain conditions to proof convergence.

Maybe this paper will be useful for you. It seems that they show that constraining your network to have relu activation functions allow you to show some convergence properties.


How to represent Tic-Tac-Toe vs Checkers or Chess for a Neural Network

When you are working with neural networks, as long as the data is there, the network is usually able to learn how to process it into a useful result, you usually also want to keep the amount of weights to a minimum. When you you use extra weights, it will take longer to train the network because you need to tune even more values for an optimal network. So, for tic-tac-toe, any of your solutions involving 9 inputs should work just fine. Also, it helps if you keep the inputs between 0 and 1 if you are using log sigmoid, and -1 and 1 if you are using hyperbolic tangent for your activation function. You can probably easily figure out what to use for other activation functions. You can take your data and transform it into another dataset with values within a specific range through a process called range normalization. For chess, you can simply encode every piece in several different ways, and it will probably not make that much of a difference. The general rule of thumb is you want to minimize the amount of weights while still giving the most possible variables to the network.


Anomaly Detection in distributed system using generated log file

In the paper "Unsupervised real-time anomaly detection for streaming data" (by Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha), 2017, an algorithm for anomaly detection (particularly suited for cases where a stream of data is continuously provided) is described. This algorithm is based on Numenta's Hierarchical Temporal Memory model.

I've actually never used it, but I know that Numenta's work is particularly suited for anomaly detection. You can have a look at it and see if it fits your needs. Have also a look at the Numenta Anomaly Benchmark (NAB).


Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a much more useful introduction to this area)

Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.

Liu et al's survey, in my opinion, doesn't do much more than list and briefly describe the MORL algorithms which existed at that point. There's no deeper analysis of the field. The original version of their paper was also retracted due to blatant plagiarism of several other authors, including myself as can be confirmed here: https://ieeexplore.ieee.org/document/6509978
Our survey provides arguments for the need for multiobjective methods by describing 3 scenarios where agents using single-objective RL may be unable to provide a satisfactory solution that matches the needs of the user. Briefly, these are

the unknown weights scenario where the required trade-off between the objectives isn't known in advance, and so to be effective the agent must learn multiple policies corresponding to different trade-offs and then at run-time select the one which matches the current preferences (e.g. this can arise when the objectives correspond to different costs which vary in relative price over time);

the decision support scenario where scalarization of a reward vector is not viable (for example, in the case of subjective preferences, which defy explicit quantification), so the agent needs to learn a set of policies, and then present these to a user who will select their preferred option, and

the known weights scenario where the desired trade-off between objectives is known, but its nature is such that the returns are non-additive (i.e. if the user's utility function is non-linear), and therefore standard single-objective methods based on the Bellman equation can't be directly applied.

We propose a taxonomy of MORL problems in terms of the number of policies they require (single or multi-policy), the form of utility/scalarization function supported (linear or non-linear), and whether deterministic or stochastic policies are allowed, and relate this to the nature of the set of solutions which the MO algorithm needs to output. This taxonomy is then used to categorise existing MO planning and MORL methods.
One final important contribution is identifying the distinction between maximising Expected Scalarised Return (ESR) or Scalarised Expected Return (SER). The former is appropriate in cases where we are concerned about the results within each individual episode (for example, when treating a patient - that patient will only care about their own individual experience), while SER is appropriate if we care about the average return over multiple episodes. This has turned out to be a much more important issue than I anticipated at the time of the survey, and Diederik Roijers and his colleagues have examined it more closely since then (e.g. http://roijers.info/pub/esr_paper.pdf)


What is the impact of using multiple BMUs for self-organizing maps?

The following is my intuition about the behavior of the algorithm considering your question, based on my knowledge and experience with single BMU SOMs. I didn't verify it experimentally.
At early stages of training:
It should disturb the SOM topology preserving properties, as you're assigning the same pattern to different (and probably distant) locations in the lattice, making the distribution multimodal. This is not terrible on its own, as different positions could encode different relationships to neighbors, overcoming the limitations of a single 2D/3D neighborhood. But if your SOM is not large enough, the multiple modes may collapse, resulting in unimodal distributions covering large portions of the map, wasting space. It all depends on your training schedule.
At later stages of training:
If you started with a single BMU at early stages, it wouldn't provide any meaningful impact, as a SOM, by definition, clusters similar patterns at close positions of its lattice. In other words, the neurons closest to your input are already inside the BMU update radius. If you trained with multiple BMUs since the beginning, you would keep finding / activating the multiple maxima of your multimodal distributions for each pattern. The dimensionality reduction would become even more nonlinear than it already is with single BMUs. This may or may not be a problem, depending on your application.


JavaScript Client Side GAN implementation

While training a GAN, 2 models are used. A generator and a discriminator. This training process usually takes hours (or days) to complete. This is an offline process and is not happening in the browser.
The pict file is the pre-trained model that has been imported to deeplearn.js for inference. The example you have linked to above accepts a sketch drawn in the Canvas and uses the (previously-created) model with deeplearn.js to generate the result.

The full process used by the Tensorflow port of pix2pix is described here.
https://github.com/affinelayer/pix2pix-tensorflow#getting-started


How is local minima possible in gradient descent?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum.

However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\boldsymbol{y}$ the corresponding labels and the function $f$ is the model (e.g. a neural network), is not necessarily a parabola. In general, it is only a parabola if $f$ is a constant function and the sum is over one element.

For example, suppose that $f(x_i) = c, \forall i$, where $c \in \mathbb{R}$. Then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - c)^2$ will only change as a function of one variable, $\boldsymbol{y}$, as in the case of $g(x) = x^2$, where $g$ is a function of one variable, $x$. In that case, $(y_i - c)^2$ will just be a shifted version (either to the right or left depending on the sign of $c$) of $y_i^2$, so, for simplicity, let's ignore $c$. So, in the case $f$ is a constant function, then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2$, which is a sum of parabolas $y_i^2$, which is called a paraboloid. In this case, the paraboloid corresponding to $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2$ will only have one optimum, just like a parabola. Furthermore, if the sum is just over one $y_i$, that is, $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i y_i^2 = y^2$ (where $\boldsymbol{y} = y$), then the MSE becomes a parabola.

In other cases, the MSE might not be a parabola or have just one optimum. For example, suppose that $f(x) = x^2$, $y_i = 1$ ($\forall i$), then $h(x) = (1 - x^2)^2$  looks as follows

which has two minima (and one maximum): at $x=0$ and $x=1$. We can find the two minima of this function $h$ using calculus: $h'(x) = -4x(1 - x^2)$, which becomes zero when $x=0$ and $x=1$.

In this case, we only considered one term of the sum. If we considered the sum of terms of the form of $h$, then we could even have more "complicated" functions.

To conclude, given that $f$ can be arbitrarily complex, then also $\text{MSE}(\boldsymbol{x}, \boldsymbol{y})$, which is a function of $f$, can also become arbitrarily complex and have multiple minima. Given that neural networks can implement arbitrarily complex functions, then $\text{MSE}(\boldsymbol{x}, \boldsymbol{y})$ can easily have multiple minima. Moreover, the function $f$ (e.g. the neural network) changes during the training phase, which might introduce more complexity, in terms of which functions the MSE can be and thus which (and how many) optima it can have.


What is the advantage of having a stochastic classification procedure?

There are multiple potential reasons for having stochastic predictions (instead of categorical/binary).
First, it often simplifies training and improves the training outcome when training a classifier on producing probabilities per class. For example, it allows for the usage of many nice loss functions like the Mean Squared Error (MSE), which is compatible with the famous back-prop algorithm. Moreover, improving classification based on the loss computed from stochastic probabilities per class allows for improving the classifier even further when the counting loss computed on predicted class labels has reached 0 already. So, if you only have binary decisions and your classifier gets all classes correct, training can stop immediately since the loss becomes 0. However, when predicting probabilities per class and iteratively driving the probability of the correct class towards 1, training can continue for much longer, even after all classes have been classified correctly already. So, the classifier trained on making probabilistic predictions can in the end perform much better since training can progress for much longer, allowing for increasingly pronounced discrimination between classes with every update. A nice introduction is given in this Stanford lecture recording.
Second, it is often nice for a user to know how much certainty versus uncertainty there is in a classification. If one class has 80% probability, this is a pretty clear decision in favor of the 80%-probability-class. However, if you only get categorical class labels returned by your classifier, you cannot possibly know whether the evidence in favor of the winning class was only marginally higher than that of the other class (as measured by the classifier) or whether there was a clear difference.


Can neural networks learn to ignore an input datum?

This is a learnable behavior, given enough data.
We would expect the an error to backpropagate to $w$ whenever its use harmed classification accuracy. In this case, that would be whenever $|w|>0$. In general, I'm not sure how long this would take.

However, the speed of $w$'s convergence to zero would benefit from regularization, which is often basically a penalty on the magnitude of your network weights added to the loss function you're optimizing. If $w$ truly doesn't matter to classification, then regularization will definitely drive it to zero.


How to put commands into local database in c# visual studio for voice recognition

The easiest way is by using System. Speech. Recognition; here is described how to use it

Just change this code:

colors.Add(new string[] { "red", "green", "blue" });

And add results of a database query, for example, with Entity Framework and LINQ instead of new string [] {"red", "green", "blue"}. Something like this:

WordsToRecognize.Add(
context.MyTable
       .Where(x => x.Type == "type")
       .Select(x => new { x.Text }).ToList());

I'm not tested the code above, it just pseudocode


I am not absolutely sure, but I guess it is due to the domain gap. As far as I have seen in my project where YOLOv3 was trained on synthetic images, it performed better when the model was trained and tested on the same domain (synthetic), while the performance drops when we introduce real images for testing. So when you include real images along with synthetic images, you might have to use some domain adaptation methods to improve the performance.


You can maximize the loss by doing 1-loss (if the range of the loss is between 1 and 0. However, I am not sure if that would help. The network can just propagate the weights of the source classifier to always output the wrong answer. a better approach is to hand craft features of the document to make it less identifiable as the source. If the document have stated the source, remove it. This may help. Also, another way is to use documents from more sources so the network cannot use the source to classify the label. Hope I can help you.


Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Has this been done?

Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), it has not been done to the level of suggesting a global function worth approximating.

The problem is that learning rate tuning, like other hyperparameter tuning, is highly dependent on the problem at hand, plus the other hyperparamater values currently in use, such as size of layers, which optimiser is in use, what regularisation is being used, activation functions.

Although you may be hoping for $\Psi(\epsilon, i | P)$ to exist where P is the problem domain, it likely does not except as a mean value over all $\Psi(\epsilon, i | D, H)$ for the problem domain, where D is the dataset and H all the other hyperparameters.

It is likely that such a function exists, of ideal learning rate for best expected convergence per epoch. However, it would be incredibly expensive to sample it with enough detail to make approximating it useful. Coupled with limited applicability (not domain-specific, but linked to data and other hyperparameters), a search through all possible learning rate trajectories looks like it would give poor return on investment.

Instead, the usual pragmatic approaches are:

Include learning rate in hyperparameter searches, such as grid search, random search, genetic algorithms and other global optimisers.
Decay learning rate using one of a few approaches that have been successfully guessed and experiments have shown working. These have typically been validated by plotting learning curves of loss functions or other metrics, and the same tracking is usually required in new experiments to check that the approach is still beneficial.
Some optimisers use a dynamic learning rate parameter, which is similar to your idea but based on reacting to measurements during learning as opposed to changes based on an ideal function. They have a starting learning rate, then adjust it based on  heuristics derived from measuring learning progress. These heuristics can be based on per-epoch measurements, such as whether a validation set result is improving or not. One such approach is to increase learning rate whilst results per epoch are improving, and reduce learning rate if results are not improving, or have got worse.

I have tried this last option, on a Kaggle competition, and it worked to some extent for me, but did not really improve results overall - I think it is one of many promising ideas in ML that can be made to work, but that has not stayed as a "must have", unlike say dropout, or using CNNs for images.

Some optimisers store a multiplier per layer or even per weight - RMSProp and Adam for example track rate of change of each parameter, and adjust the rate for each weight during updates. These can work very well in large networks, where the issue is not so much needing a specific learning rate at any time, but that a single learning rate is too crude to cover the large range of gradients and differences in gradients across the index space of all the connections. With RMSProp and Adam, the need to pick specific learning rates or explore them is much reduced, and often a library's default is fine.


How can I encode states where the environment consists of multiple identical elements, but each is characterised by different features?

For general advice about state representation, you could check my answer to How to define states in reinforcement learning? - this does not cover your specific issue, but may help with other details such as whether to one-hot-encode and/or scale your discrete features. Also you will want to assess whether any state vector you construct actually contains enough data for reinforcement learning to work.

Assuming that is all good, then your approach here:

  Can I simply vectorize such state by concatenating the different features of each element obtaining $[f_{0e_0}, f_{1e_0}, f_{0e_1}, f_{1e_1}, f_{0e_2}, f_{1e_2}]$

should work with a generic feed-forward neural network with two or more layers. As this is simple to construct, then it is a reasonable place to start just considering development time. If nothing else it can be a benchmark against other representations and model architectures that you may want to try.

  should I use a convolutional architecture of some sort?

That will depend on any structure inherent in your environment. The interesting thing about your description of the environment is the repetition of meaning in the values. Each entity $e_i$ appears multiple times, and the units of feature $f_{0e_0}$ will be the same as $f_{0e_1}$.

An unfortunate consequence of using a flattened vector is that the information about these repeated entities and their units is lost to the system. Any important details due to the entities being essentially the same of this must be "rediscovered" by the approximator training with many examples. Although neural networks can generalise function results, this does not apply to generalising across different parts of the input unless you choose an architecture designed to help with that.

Typically you would use a RNN architecture (such as LSTM) if key information is in the sequence order of entities, or a CNN architecture if key information is in local patterns between neighbouring entities. By "key information" I mean whether useful relationship to a value function or optimal policy depends on those factors. This might be difficult to assess for some environments with mixed data types and complex meanings for the entity ids - so you may have to experiment with multiple architectures to figure out what factor is more important. However, it is also possible you have some insight based on known good policies, so can head more directly to a promising feature representation and architecture.

Learning may benefit if instead of ordering your entities arbitrarily by identity in the vector, that you sort them according to some factor that is relevant to the goals of your agent. I have used this is in a colour matching environment where an agent had to hit a target with a "current" colour selection whilst avoiding incorrect colours - sorting target entities by whether they matched colour and then by angular distance to the agent's aim enabled the agent to learn its task far more efficiently. This sorting may help independently of architecture choice - i.e. you don't need to be using a RNN or CNN to see a benefit with this approach.

Another thing worth considering, but probably not possible here because of the continuous feature $f_0$, is whether you can invert the representation so that instead of being based on entities with properties, it is a view over property space populated with entities in certain positions. A CNN processed over a small enough "property space" instead of "entity space" may well perform better - this is why board game RL tends to treat the board positions as defining feature vector indexes which represent which entities are present on them, which is a very sparse structure that can represent far more states than ever occur in real games. The over-specified space works because it can be matched to architectures that generalise efficiently over it.

In general, you need to consider different types of symmetry within the system - especially what is meaningful about the id number you have assigned to each entity. If the entity id is not meaningful, or is secondary to some other factor, you have some flexibility in changing the representation to make it meaningful and use architectures that take advantage of how you have injected that domain knowledge into the q function approximator. If the id is meaningful, think how could you best represent the difference between entities with different ids in your environment?


Is normalizing the data a way to improve generalization?

Batch Normalization is usually known to speed up the learning process as it makes the weights in the deeper layers more robust. It restricts the distribution of the weights in a particular layer - this video might tend to be useful to what BatchNorm does. This said, batch normalization does have a regularizing effect which does to tend to increase generalization.

Talking about generalization - A focus on regularization would probably be more helpful


What qualifies a Voice Assistant to be an AI?

Usually, this sort of encoded 1to1 response is not considered to be AI or ML. This is because a chatbot that simply responds to a designated string with a response string is hard programmed(for the most part). A lot of companies/individuals have tried to package this as true AI but it is not.

One of the simpler examples of an AI chatbot system is a model that does some sort of tree-search or similar for the input. This is more often than not, extremely naive but would be an example of a true AI system.

So to more concisely answer your question, the categorization as an AI chatbot would have less to do with the responses, and more to do with how the model selects/comes up with those responses.


Autoencoder: predictions missing for nodes in the bottleneck layer

My closest guess is because you are using the activation function ReLU, which pushes data to be greater than zero. However, because of your data's nature, the autoencoder is highly dependent on negative calculations to reconstruct your data, but the best it can achieve is zero.

In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument:
$$f(x) = x^+ = max(0, x)$$
https://en.wikipedia.org/wiki/Rectifier_(neural_networks)


What is the solution to the exercise 3.11 in the RL book by Sutton and Barto

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$.

If you are currently in state $S_t$ and we condition on taking action $a$ then the expected reward at time $t+1$ is given as follows:

\begin{align} \mathbb{E}[R_{t+1} | S_t = s, A_t=a] & = \sum_{s',r}rp(s'r|s,a)\;.
\end{align}

However, action $A_t$ is taken according to some stochastic policy $\pi$ so we need to marginalise this out of our expectation by using the tower law - i.e. we take

$$\mathbb{E}_{A_t \sim \pi}[\mathbb{E}[R_{t+1} | S_t = s, A_t=a]|S_t = s] = \sum_a \pi(a|s)\sum_{s',r}rp(s'r|s,a) = \mathbb{E}[R_{t+1} | S_t = s]\;.$$

To see why this holds we can re-write using some arbitrary mass functions $f(\cdot),h(\cdot),g(\cdot),m(\cdot)$ as

$$\pi(a|s)p(s'r|s,a) = \frac{f(a,s)}{g(s)} \times \frac{h(s',r,a,s)}{f(a,s)} = m(s',r,a|s)\;,$$
so when we would end up with (after re-arranging the summations)

$$\sum_{s',r}r \sum_{a}m(s',r,a|s) = \sum_{s',r}r m(s',r|s) = \mathbb{E}[R_{t+1}|S_t = s]\;;$$
as required.

NB: what you have done is mostly correct except be careful when going from (2) to (3). They are exactly the same equation except for the time stamps, which means you would have to change the time stamps in your $r(s,a)$. Note that when you are at time step $t$ you take your action $A_t$ from your current $S_t$ to transition into state $S_{t+1}$ and then receive reward $R_{t+1}$ (and the next state).


What is the best way to detect and recognize traffic signs in a picture?

Maybe this is helpful:
Recognising Traffic Signs With 98% Accuracy Using Deep Learning, by Eddie Forson.

Greetings
Mario


How to quantify the reflectance in an image?

You should first know about the layer-wise working of a convolutional neural network. Read this https://distill.pub/2018/building-blocks/

Each layer of a cnn forms representations that are increasingly complex , which are  combinations of simpler representations formed by its previous layers.
"shine" is not a pattern (from image processing perspective). shine occurs due to change in intensity values.

And for quantifying "shine" you dont have to go for convolutional networks only , there are many more simpler methods for such tasks you could try fitting simple linear model and progress towards more complex ones.

and for classifying textures , there is interesting work going with "invariant scattering convolutional networks" you should check that.


Why does the "reward to go" trick in policy gradient methods work?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$):
$$\mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) \right] = 0.$$
Taking the analytical expression of the gradient (from, for example, slide 9) as a starting point:
$$\begin{aligned}
\nabla_{\theta} J(\theta) &= \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \right) \left( \sum_{t=1}^T r(s_t, a_t) \right) \right] \\
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^T r(s_{t'}, a_{t'}) \right] \\
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) + \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \\
%
&= \sum_{t=1}^T \left( \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) \right] \\
+ \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \right) \\
\end{aligned}$$
At the $t^{th}$ "iteration" of the outer sum, the random variables
$ \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) $
and
$ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) $
are independent (we assume, by definition, the action only depends on the most recent state), which means we are allowed to split the expectation:
$$\nabla_{\theta} J(\theta) = \sum_{t=1}^T \left( \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \sum_{t'=1}^{t-1} r(s_{t'}, a_{t'}) \right] \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \right] \\
+ \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right] \right)$$
The first expectation can now be replaced by $0$ due to the lemma mentioned at the top of the post:
$$
\begin{aligned}
\nabla_{\theta} J(\theta)
%
&= \sum_{t=1}^T \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right]  \\
%
&= \mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_t \mid s_t) \left( \sum_{t'=t}^T r(s_{t'}, a_{t'}) \right). \\
\end{aligned}
$$
The expression on slide 18 of the linked slides is an unbiased, sample-based estimator of this gradient:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (a_{i, t} \mid s_{i, t}) \left( \sum_{t'=t}^T r(s_{i, t'}, a_{i, t'}) \right)$$

For a more formal treatment of the claim that we can pull $\sum_{t'=1}^{t-1} r(s_{t'}, a_{t'})$ out of an expectation due to the Markov property, see this page: https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof1.html


What would be the state of the art image captioning deep learning model?

Here are a couple of Kaggle Kernels, Notebooks and Tutorials for Image Captioning.

Kaggle Kernel | Neural Image Captioning:  ->

Kaggle Kernel | Show Attend and Tell

Kaggle Kernel | Flickr Image Captioning : TPU, TF2 & Glove

Tensorflow Tutorial | Image captioning with visual attention

Show and Tell: A Neural Image Caption Generator by Vinyals et al.


How do I combine models trained on different data to increase classification accuracy?

You can try using a multi-input model. Here is a recent post with a similar discussion, with the required architecture defined in the answer.

Instead of combining the separate models, you can create a model which uses image and numerical data side by side. Keras allows you to use different types of data using multi input structure via functional API. And then you can combine them to create a single machine learning model. Basic idea is like this:

This image was taken from here, where you can find further details and the code implementation as well. Actually, the discussion in the introduction of that post is almost the same as your question: What to do when the inputs are:

  Numeric/continuous values, such as age, heart rate, blood pressure
  Categorical values, including gender and ethnicity
  Image data, such as any MRI, X-ray, etc.

Also, section 5.1 of this blogpost details the same process.


I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous control with Deep RL" paper - which I'd recommend as a starting point for your investigation:

An obvious approach to adapting deep reinforcement learning methods such as DQN to continuous
domains is to to simply discretize the action space. ... Such large action spaces are difficult to explore efficiently, and thus successfully training
DQN-like networks in this context is likely intractable. Additionally, naive discretization of action spaces needlessly throws away information about the structure of the action domain, which may be essential for solving many problems.

The last sentence in the quote above is the most important point for dealing with your problem. The fundamental issue is inability to efficiently explore such a large  action space - so the idea is to use its structure. I'm sure that your >2k discrete action set has a certain structure on it. Like some actions might be "closer" to others. If so, then you (1) can infer some information about "neighbor" actions even if you never took them (2) do some exploration by adding noise to your policy-preferred action.
The Actor-Critic class of algorithms matches perfectly the steps (1) and (2) above. The Critic Q-value network learns about your state-action space and the Actor policy network returns actions that you could smear.

Are policy gradient methods good for large discrete action spaces?

The Actor-Critic class of RL algorithms is a subclass of the Policy Gradient algorithms. So, the answer is yes - it looks like going that way is your best shot at making progress.

Does this effectively mean DQN will not work?

I don't think that there is a strict "it will not work" statement. The problem is that it just would be extremely difficult to make the DQN training stable "by hand". In principle you might be able to construct a neural architecture that captures the state-action space structure. And then figure out how to perform efficient exploration. And then ensure that nothing explodes anywhere. But that's exactly what DDPG and then TRPO/PPO approaches would do for you.


Facial classification and tracking is easier compared to tracking any other motion. This is due to the fact that the face have a large number of easily identifiable features.

Facial tracking is an additional layer on top of facial detection. Facial detection works by finding characteristics such as the cheekbones, chin, nose, eyes etc. These features are easy to detect as they have very specific properties. These points are found with the help of shadows and brightness, for example, the nose and cheekbones are highlighted, whereas the the eyes and lips have a shadows.

Using these feature points it is possible to create a mesh to construct the face. This has been a classical way to detect and classify faces. The TrueDepth camera helps to make a better structure by providing depth with the help of an IR sensor.

Facial tracking makes use of this mesh to study and track changes in the facial structure. For example when you smile, it will detect the cheekbone points changing, similarly when you open your mouth, your lips will move apart.


How can I process neural network with 25000 input nodes?

Yes, it should be no problem.
When you decide to use a CNN, you have to make sure that this makes sense. Another answer mentioned using 3x3 convolutions -- which I would recommend against. For that to work, you would need to turn your vector into a rectangular array, and you would be implying a structure that isn't there.
Use one-dimensional convolutions instead.


Network representation for Q-Learning in carrom

Discretizing the output will probably be counter-productive in this situation; it would remove flexibility by taking away the fine-grained continuous ranges between each discretization bucket, but also blow up the size of the network, reducing manageability and performance. Fragmenting the outputs into buckets in this way may also lead to information loss and more difficulty in convergence because of the fact that each bucket is partially isolated from the others.

After causing myself needless hassle in the past by mismatching input and output dimensions, I'd simply do this if I were in your shoes: 1) keep it simple and use 3 continuous (well, semi-continuous, depending on the highest precision your programming framework allows) outputs for all of the above reasons; 2) clamp the angles between -45 to 225 by whatever method works best for you, like ceiling/floor hard clamping, adding weight terms, etc.; 3) go big on the hidden layer(s) to maximize information sharing across the inputs and eventual outputs, force, angle of strike, position of strike, etc. This is more likely to fine-tune the precision of the outputs, thereby making good use of the semi-continuous scales.

I'm also wondering if a convolutional neural net might work in this situation; their most popular use case is in image processing, but I don't see why you can't treat the force, angle and position as surrogate spatial dimensions. I'm not sure how many or what type of inputs you have, but 3 continuous outputs might be conducive to a 3D rather than a 2D space. Convolutionals are often used for those, as well as higher-dimensional and temporal data. I hope that helps.


Should the weights of a neural network be updated after each example or at the end of the batch?

The backpropagation step is generally used to compute the gradients and update the weights.

Let us say, you are implementing gradient descent, select the whole training batch, perform the forward propagation using the current set of neurons/weights to get the classification output. Then compute the loss/cost with respect to the actual labels, (Note: This step contains dividing the loss and divide it by the number of training example to get the cost function). Once you get the cost you backpropagate by computing the gradients at each layer with respect to cost from output to input and perform the weight updates.

If you are using mini-batch, you compute the cost for that mini-batch, divide by the number of examples in the mini-batch, then compute the gradients and perform the update in the backprop step.


Yeah, it seems like it's a wrong implementation.
vals_ref_v is a matrix of 1 row, and 128 columns.
value_v.detach() is a matrix of 128 row


How do we actually sample an action from a policy in policy gradient methods?

In those policy gradient algorithms, we typically have:

Input features $x(s)$ for any state $s$
A fixed action set $\mathcal{A}$ of a fixed size (in reinforcement learning (RL) it is often assumed that all actions are legal in all states. In some situations, like board games, we'll filter out illegal actions manually in a postprocessing step).
A function approximator $\pi(x)$ (often a neural network), which takes as input the features $x(s)$ of some state $s$, and produces as output a vector of probabilities, where the vector has a length equal to the size $\vert \mathcal{A} \vert$ of the action space.

Those bolded parts in the last point there are critical for your question. We do not normally have a function that "spits out some number" as you described, we typically have one that "spits out" multiple numbers; one for every action. Often, these numbers can also be directly interpreted as probabilities, because they are forced to all lie in $[0, 1]$ and sum up to $1$ (this is often done using a softmax layer at the end of a neural network for example).

Once you have such a vector of "probabilities", one for every action, it is relatively straightforward to sample from that. Just give every action a probability of being selected equal to its corresponding output.

Note that in some cases, people describe their function approximator as producing "logits" rather than "probabilities" as outputs. These are real-valued outputs which do not necessarily sum up to one, typically produced by a linear layer at the end of a neural network rather than a softmax layer (see e.g. AlphaGo Zero). It is then still implicitly assumed that we convert them into numbers that can be interpreted as probabilities through the application of a softmax, before sampling from it.


What are some resources with exercises related to neural networks?

The book Grokking Deep Learning, by Andrew Trask (a PhD student at Oxford University and a research scientist at DeepMind), a wonderful, clean, and plain-English discussion of the basic mechanics that go on under the hood of neural networks - from data flow to updating of weights.  It is written without a slant on normally-wonky math, the concepts are presented and then advanced at a digestible pace for anyone.

Here are a few more possibly useful resources.

Neural Networks: Playground Exercises
Getting Started With Deep Learning: Convolutional Neural Networks
Deep learning focuses on practical aspects of deep learning
First lab assignment in Deep learning
Deeplearning stanford


Why aren't compiled languages as popular as Python in AI?

There are a few advantages of interpreted languages (compared to compiled languages)

platform independence (you only need the interpreter for your platform, even though this is not true if e.g. your interpreted language is only a wrapper library around a library written in another programming language)
dynamic typing (no need to specify the types of the variables)
dynamic scoping (e.g. you can access variables in other scopes)
automatic memory management (but there are compiled languages, like Java, that also have a garbage collector)
Rapid prototyping (for various reasons, including dynamic typing), hence software can be written more quickly

Hence, the main advantages of an interpreted language compared to a compiled language are flexibility and dynamism. Given that AI is still an evolving field, these characteristics are widely appreciated.

There is also at least one disadvantage of interpreted languages (compared to compiled languages)

Slower running times compared to compiled languages, which, once compiled, are quite fast, because they are often compiled to a code that is quickly executable by the machine or virtual machine

Python and R are widely used in data science and artificial intelligence because of the advantages above, which possibly contributed to the rapid growth of the communities around them and the development of software libraries.

However, note that the core of the most common machine learning libraries today, including TensorFlow and PyTorch, is written in a compiled language like C and C++. In the specific case of TensorFlow, Python is just a wrapper library. Consequently, under the hood, the code is not executed by the Python interpreter, but first compiled, which implies that, when you're using e.g. TensorFlow, your code will run (more or less) as fast as if you were using a compiled language like C. A similar argument can be made for libraries like NumPy, where Python is just a wrapper library.


What evaluation metric are used for sequence-to-sequence prediction problems?

I would recommend taking a look at Bilingual Evaluation Understudy(BELU) score which is commonly used in evaluating machine translation results by sequence to sequence model. Here is the reference https://en.wikipedia.org/wiki/BLEU


What kind of output should be used for predicting angles in DNNs?

As you said, the first option is not very suitable due to the cyclic nature of the angles. However, if you don't mind discretizing the values, you could represent the output as a binary vector.

A variant of the second option seems perfect to me. You may output a 2D vector and use that vector's angle as output. You'll probably need a regularizer for the vector's norm, but that's it.


How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control?

By definition of $V_{n+1}$, we have:

$V_{n+1} = \frac{\sum_{k=1}^{n} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{1}$

Then, taking the $n^{th}$ term out of the sum in the numerator, we have:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; \sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{2}$

Then, from the definition of $V_n$, $V_{n} = \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}$, we have:

$\sum_{k=1}^{n-1} W_{k} G_{k} = V_{n}*\sum_{k=1}^{n-1} W_{k} \; \tag{3}$

Then, substituting $(3)$ in the numerator of $(2)$, we get:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{4}$

Then, adding and subtracting $V_{n}W_{n}$ in the numerator of $(4)$, we obtain:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k} \; + \; V_n W_n \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{5}$

We factor $V_n$ in the numerator of $(5)$:

$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}(W_n \; + \; \sum_{k=1}^{n-1} W_{k}) \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{6}$

We simplify, taking into account that the denominator $\sum_{k=1}^{n} W_{k} = W_n + \sum_{k=1}^{n-1} W_{k}$, we get:

$V_{n+1} = V_n + \frac{W_n G_n - W_n V_n}{\sum_{k=1}^{n} W_{k}} \; \tag{7} $

Further rearrangements of the terms give us:
$V_{n+1} = V_n + \frac{W_n}{\sum_{k=1}^{n} W_{k}}[G_n - V_n] \; \tag{8}$

Finally, by definition of $C_n$ as the cumulative sum of the weights up to time $n$, we get the desired incremental update equation:
$V_{n+1} = V_n + \frac{W_n}{C_n}[G_n - V_n] \; \tag{9}$


"AI will kill us all! The machines will rise up!" - what is being done to dispel such myths?

Nothing.

Its in almost everyone's favor for it to stay that way financially. Having non-technical individuals associate AI with terminators makes a perception that the field has greater capabilities than it does $\rightarrow$ this leads to grants, funding, etc...

Is there any negative? Yes. Misconceptions always have drawbacks. We see the creation of dumb ethics boards and such cough cough Elon Musk.

But if history has anything to say about this, as the field gains popularity (which it is dnagerously quick), information will spread by definition, and eventually misconceptions will be laid to rest.

Note that this answer is biased and based upon my own opinions


Why is Information Filter called Information Filter?

I am probably not qualified to answer your question, but I will attempt. At the very least I hope to start a discussion about the matter because I am unsure.

The (linear) Kalman filter uses the error Covariance as it's main contributor to the Kalman gain. We usually notice the Error covariance decreases over time which can be thought of as the decrease in uncertainty. In the information filter the inverse is true. Our "information matrix" increases with more measurements, because our information increases.

Additionally the measurement update of the information state vector y resembles the way we simply add information. Since

y(i+1) = y(i) + i(i+1)

here we are simply adding information.

This is how I reason and remember the Information filter. I'm not sure where the name came from.

Additionally:
Arthur Mutambara mentions: "The information filter essentially tracks information about the states and not the states them selves". This may be more correct and concise.