What is the difference between simulated annealing and deterministic annealing?

After diving deeper into the material I am able to answer my own question:
Simulated Annealing tries to optimize a energy (cost) function by stochastically searching for minima at different temparatures via a Markov Chain Monte Carlo method. The stochasticity comes from the fact that we always accept a new state $c'$ with lower energy ($\Delta E < 0$), but a new state with higher energy ($\Delta E > 0$) only with a certain probability
$$p(c \to c') = \text{min}\{1, \exp(-\frac{\Delta E}{T}) \},$$
$$\Delta E = E(c') - E(c).$$
Where we used the Gibbs distribution $p(c) = \frac{1}{Z}\text{exp}(\frac{-E(c)}{T})$ to calculate probabilities for each state, with $Z$ being the partition sum. The temperature $T$ plays the role of a scaling factor for the probability distribution. If $T \to \infty $ we have a uniform distribution and all states are equally possible. If $T \to 0$ we have a Dirac delta function around the global optimum. By starting with a high $T$, sampling states and gradually decreasing it, we can make sure to sample enough states from the state space and accepting energetic higher states in order to escape local minima on the way to the global optimum. After sampling long enough while slowly decreasing the temperature, we theoretically arrive at the global optimum.
Deterministic Annealing on the other hand directly minimizes the free energy $F(T)$ of the system deterministically at each temperature, e.g. by Expectation-Maximization (EM-algorithm). The intuition behind it is that we like to find an optimum at a high temperature (where it is easier to find one because there are fewer local minima), accept this as intermediate solution, lower the temperature, thus scaling the cost function such it is more peaked around it's optima (making optimization a bit more difficult) and start deterministically looking for an optimum again. This is repeated until the temperature is low enough and we (hopefully) found a global solution to our problem. Major drawback is that there is no guarantee to arrive at a global optimum in contrast to simulated annealing. The whole idea of scaling the energy function is based on the concept of homotopy: "Two continuous functions [...] can be "continuously deformed" into each other."


This paper Can Machines Design? An Artificial General Intelligence Approach (2018, presented at AGI-18 and published in the related proceedings here), which proposes the design Gödel machine, may be useful to you.
After a quick search, I have not found other relevant papers, so I suppose that the research on GMs is currently not very active.


What are recent AI software systems and research papers close to J. Pitrat's ideas?

Today one of the challenges is learning representations/concepts that are causally invariant. Once we have good representations then we can work on the reasoning aspect. There are 2 camps of people today. One believes that symbolic manipulation cannot be achieved properly by deep networks. Hence, they separate the task of extracting a lower-dimensional representation of objects from the visual scenes from the task of reasoning with knowledge-graphs. The other camp feels that we can do end-to-end training of a neural network and it can learn how to jointly learn a good lower-dimensional representation for each symbol along with learning how to reason with them. I am no expert at this but here are a few papers that I find are worthy for you to read -

Neuro Symbolic Concent Learner Paper, Code
Learning Reasoning Strategies in End-to-End Differentiable Proving Paper, Code
Neuro-Symbolic Visual Reasoning: Disentangling “Visual” from “Reasoning” Paper
Knowledge Infused Learning (K-IL):
Towards Deep Incorporation of Knowledge in Deep Learning Paper
Visual Concept-Metaconcept Learning Paper Project Page
CVPR 2020 workshop on Neuro-Symbolic Visual Reasoning and Program Synthesis youtube videos

If you are looking for community maintained lists, here is one  list of papers


What is the current state-of-the-art in Reinforcement Learning regarding data efficiency?

There is a very interesting learning agent. They call it Neural-Episodic-Control. Here is the link for the paper: https://arxiv.org/abs/1703.01988 . Their experiments show that NEC requires an order of magnitude fewer interactions with the environment than agents previously proposed.


How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode.
You have two "natural" options to tweak DQN to match to theory at the end of a pseudo-episode:

Store the state, action, reward, next_state tuple as normal and use the standard one step TD target $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$

Completely ignore the last step and don't store it in memory. There is no benefit to this as opposed to the above option, but it might be simpler to implement if you are using a pre-built RL library.

Both these involve ignoring any done flag returned by the environment for the purposes of calculating TD targets. You still can use that flag to trigger the end of a loop and a reset to new starting state.
You should also take this approach if you terminate an episodic problem early after hitting a time step limit, in order to reset for training purposes.

As an aside (and mentioned in comment by Swakshar Deb), you can also look into the average reward setting for non-episodic environments. This solves the problem of needing to pick a value for $\gamma$. If you have no reason to pick a specific $\gamma$ in a continuing problem, then it is common to pick a value close to 1 such as 0.99 or 0.999 in DQN - this is basically an approximation to average reward.


Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning?

In rehearsal, you do not necessarily train with all old training data, but you can just use some of it [1], which you add to your current (or new) training data.
In batch learning, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have $N$ training examples and your batch size is $M$, then you will have $\lfloor N/M \rfloor $ gradient descent steps.
There is also pseudo-rehearsal, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [1] (for example, because it is expensive to store).


Why evolutionary training of neural networks is not popular?

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers building on or using the algorithm.

NEAT is not widely used in commercial applications because if you have a clean objective function, a topology that is optimized for gradient decent via backpropogation, and an implementation that is highly optimized for a GPU, you are almost certainly going to see better, faster, results from a conventional training process. Where NEAT is really useful is if you want to do something weird, like train to maximize novelty, or if you want to try to train neurons that don't have cleanly decomposable gradients. Basically, you need to have any of the usual reasons you might prefer an evolutionary algorithm to hill-climbing approaches:

You don't have a clean mapping from loss function to individual model components.
Your loss function has many local maximia.


Why Autoencoder Weights Are Not Always Tied

We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.
If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.


How would we know if AI development will continue to be multipolar, or will become monolithic?

Oligopoly vs Monopoly

The terms in the question, multipolar and monolithic, appear to be referring to the micro-economic concepts of oligopoly and monopoly respectively.  Although these concepts are not AI specific, they certainly apply to such development in the way the question suggests.  Leading AI R&D is occurring in  a relatively small number of corporate, governmental, and university research labs.

A simple search for Oligopoly vs Monopoly will cover the differences between these two scenarios sufficiently for all sectors of products and services, and those concepts and observations apply to a large degree in the AI case.

Those who have not studied economics or political science might incorrectly conclude that oligopoly is necessarily related to capitalistic competition and monopoly is related to communism. However, R&D monopolies frequently exist within capitalistic economies and oligopolies frequently exist in a planned economies, such as communist states.

Strengths of Each

Research oligopoly strengths:

Lead to diversity of approaches and target products and services
Lower the probability of ethical, environmental, or social technology abuses because multiple independent teams, even with corporate confidentiality are in place, exert the force of at least a modicum of accountability

Research monopoly strengths:

Efficient in terms of return on research investment for two reasons (1) the division of a fixed amount of available research funding between several labs can increase the R&D required to produce usable results, and (2) unfruitful approaches are more quickly abandoned when their workers can be shifted to other teams within the same organization
Attract all those who have the most talent and experience in the fields needed by the specific R&D into one place to potentially collaborate

Predicting Transitions Between Monopolistic and 'Oligopolistic' Forms

The leading edge of any R&D tends to begin as a monopoly or a 'biopoly'.  Three or more units beginning the same research at the same time or being funded by government at the same time (as in planned economies) is unusual.

Over time, employees tend to leave and start their own organizations, more so in capitalistic societies, but also in planned economies, where government may hear cases to branch off and grant permission or the scientist may, in one way or another, seek relocation to a free economy.

We can see these trends throughout technological history, back to early forms of writing, crop irrigation and plowing, chariot development, and ship building.  Modern examples include the application of the Westinghouse-Tesla model of electrical energy distribution and sales, genetic engineering, speech recognition, and HTTP clients (browsers starting with Mosaic, then Netscape, then IE, then Opera, Chrome, and others).

Unknowns

How far ahead one global AI leader is than another cannot be so easily predicted.  Some companies choose to monetize R&D results sooner in the research cycle than others, depending on their strategy and the diversity of their revenue stream.  For instance, as of this writing, Apple and Facebook can wait a decade to release to the public manifestations of R&D without jeopardizing their financial status.

Furthermore, governments may declassify material (another form of what the question calls time lag) several decades after discovery.  In the case of the Von Neumann model for achieving critical temperature and pressure of fissile or fusion materials, the government may never declassify it.  What the NSA or equivalent organizations outside the United States may decide must fall under an equivalent strategic wall of secrecy cannot be known outside of treasonous disclosure.

The insight in an organization is not sustainable in practice.  Most people do not even know where first insights occurred and two whom they occurred first because the current icons in business related to the products and services stemming from the insight are otherwise completely unrelated to origins.

Few would buy a new car and thank Isaac Newton or even Henry Ford.  Absolutely no one buys a cell phone and thanks Alonso Church, Claude Shannon, or Gene Roddenberry when the first call or SMS message goes through.  We don't thank the water wheel researchers of Fourth Century Alexandria when we turn on a light, or even Nicola Tesla.

We do in some ways worship computer technology entrepreneurs when we buy an iPhone or require a document in docx format and vaguely understand that it is a MS Office document type.  Few know that not a single innovation we attribute to those entrepreneurs came from within their respective organizations.  Everything from personal computing, desktop publishing, touch screens, and windowing interfaces were developed in the R&D facilities of other corporations.

In some ways, the company to come up with the next insight is less likely to be the last one that did.  Innovators usually find better pay after they have a choice item on their resume like, "Invented first adaptive cell tower switching protocol for mobile device communications."  That person is not only no longer working for the company that filed the corresponding patents, but is retired, and the company is now struggling and without recent productized innovation.

Insurmountable Competitive Edges

Thus far competitive edges have always become surmountable.  The proliferation of nuclear weaponry, now spanning at least seven countries, is a current and certainly important example of this.  German and Japanese aeronautics R&D edges are another.


Why are the terms classification and prediction used as synonyms in the context of deep learning?

Many people confuse and misuse the two terms, classification and prediction (or classify and predict).  This is because in many cases classification techniques are being used for prediction purposes which creates part of the confusion to others who then use the term ‘prediction’ (or ‘predict’) inappropriately.

Your understanding of the definitions of classification and prediction is mostly correct and you are absolutely correct that there are many people using the terms synonymously, sometimes correctly but I believe mostly erroneously.  There are many good articles elaborating on the two and I have added some links and excerpts at the end of this answer.  What these articles don't cover is that many forecasting (i.e. prediction) researchers and practitioners will use conventional classifiers to predict the future state of a time series or data sequence.  More advanced researchers and practitioners will use time-recurrent models, which learn temporal patterns.  These are still called classifiers but the for purpose of prediction.

There are more papers written on this use of classifiers, conventional and time-recurrent type classifiers, for time series than the use of regressor models!

This adds to the confusion in the data science and machine learning community in the usage of the terms ‘classify’ and ‘predict'.

Galit Shmueli sums it up best in his paper, “To Explain or to Predict?”, where he states: “Conflation between explanation and prediction is common, yet the distinction must be understood for progressing scientific knowledge.”

There is also the opposite problem where people will confuse regression models with classification.  See the first article below.

Classification vs. Prediction, by Professor Frank Harrell

Excerpt:
By not thinking probabilistically, machine learning advocates frequently utilize classifiers instead of using risk prediction models. The situation has gotten acute: many machine learning experts actually label logistic regression as a classification method (it is not). It is important to think about what classification really implies. Classification is in effect a decision. Optimum decisions require making full use of available data, developing predictions, and applying a loss/utility/cost function to make a decision that, for example, minimizes expected loss or maximizes expected utility. Different end users have different utility functions. In risk assessment this leads to their having different risk thresholds for action. Classification assumes that every user has the same utility function and that the utility function implied by the classification system is that utility function.

To Explain or to Predict?, by Galit Shmueli

Abstract. Statistical modeling is a powerful tool for developing and testing
theories by way of causal explanation, prediction, and description. In many
disciplines there is near-exclusive use of statistical modeling for causal explanation and the assumption that models with high explanatory power are
inherently of high predictive power. Conflation between explanation and prediction is common, yet the distinction must be understood for progressing
scientific knowledge. While this distinction has been recognized in the philosophy of science, the statistical literature lacks a thorough discussion of the
many differences that arise in the process of modeling for an explanatory versus a predictive goal. The purpose of this article is to clarify the distinction
between explanatory and predictive modeling, to discuss its sources, and to
reveal the practical implications of the distinction to each step in the modeling process.

What is the difference between classification and prediction?, from KDnuggets

If one does a decision tree analysis, what is the result? A classification? A prediction?

Gregory Piatetsky-Shapiro answers:
The decision tree is a classification model, applied to existing data. If you apply it to new data, for which the class is unknown, you also get a prediction of the class.

The assumption is that the new data comes from the similar distribution as the data you used to build your decision tree. In many cases this is a correct assumption and that is why you can use the decision tree for building a predictive model.

When Classification and Prediction are not the same?

Gregory Piatetsky-Shapiro answers:
It is a matter of definition. If you are trying to classify existing data, e.g. group patients based on their known medical data and treatment outcome, I would call it a classification. If you use a classification model to predict the treatment outcome for a new patient, it would be a prediction.

gabrielac adds
In the book "Data Mining Concepts and Techniques", Han and Kamber's view is that predicting class labels is classification, and predicting values (e.g. using regression techniques) is prediction.

Other people prefer to use "estimation" for predicting continuous values.


Are convolutional neural networks inspired by the human brain?

In the eye, the retinal ganglion cells have a receptive field that is equivalent to some types of convolution filters, most of them edge detectors.
The brain is a big unknown, nobody knows how it does to organize, memorize, create concepts, learns the language, ... . Thus, it is not possible to establish a parallelism.
In particular, brain has a capacity of handle independence in scale and rotation that CNN's are not able to reproduce.
As general remark about NN and brain: even when it is always said that "neural network cells" are "inspired" in biological neurons, there are critical differences that made this similitude only a "inspiration". Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison. The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the learned parameters.


Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it as a normal MDP again, but with some extra domain knowledge that you can exploit primarily for computational efficiency:

Only at time steps that really correspond to "events" in your SMDP can you observe non-zero rewards; at all other time steps you just get rewards equal to $0$.
Only at time steps that really correspond to "events" in your SMDP do you have an action space greater than $1$; in all the "fake" time steps, you have no agency, you just have a single action available (say, a "dummy" or "null" action). So all of these "fake" time steps do not contribute in any way to the "credit assignment" problem in RL, and you can kind of ignore them in your learning steps; only the time spent in them can still be important for discount factors $\gamma < 1$.

If $\tau$ (real number not an integer number) shows the time between two arrivals, should I update Q-functions as follows:

Yes, an update rule like that looks correct to me. Let's take an example situation, where $\tau = 2.0$, and instead of using the update rule you suggest, we take the "proper" approach of discretising into smaller time steps, and treating it as a regular MDP. In this simple example case, it is sufficient to discretise by taking time steps that correspond to durations of $1.0$.
In the SMDP, we'll have only a single transition $s_0 \rightarrow s_2$ (it will become clear why I use slightly strange time-indexing here soon), after which we observe a reward, and this transition takes time $\tau = 2.0$. In the corresponding MDP, we'll have two state transitions; $s_0 \rightarrow s_1$, and $s_1 \rightarrow s_2$, with two reward observations $R_1$ and $R_2$, where we know for sure that:

$R_2 = 0$ (because it does not actually correspond to any event in the SMDP)
We have a meaningful choice between multiple actions at $s_0$, each of which can have different transition probabilities for taking us into different "dummy" states $s_1$, and yield possibly-non-zero rewards $R_1$. In the dummy state $s_1$, we'll always only have the choice for a single dummy/null action (because this state does not correspond to any event in the SMDP), which always yields $R_2 = 0$ as mentioned above.

So, the correct update rule for $s_1$ where we picked a forced dummy action $\varnothing$ and are doomed to receive a reward $R_2 = 0$, would be:
$$Q(s_1, \varnothing) \gets Q(s_1, \varnothing) + \alpha \left( 0 + \gamma \max_{a'} Q(s_2, a') - Q(s_1, \varnothing) \right)$$
and the correct update rule for $s_0$, where we picked a meaningful action $a_0$ and may get a non-zero reward $R_1$, would be:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \max_{a'} Q(s_1, a') - Q(s_0, a_0) \right)$$
In this last update rule, we know that $s_1$ is a dummy state in which the dummy action $\varnothing$ is the only legal action. So, we can get rid of the $\max$ operator there and simplify it to:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma Q(s_1, \varnothing) - Q(s_0, a_0) \right)$$
Since we know that $s_1$ is a dummy state in which we are never able to make meaningful choices anyway, it seems a bit wasteful to actually keep track of $Q(s_1, \varnothing)$ values for it. Luckily, we can easily express $Q(s_1, \varnothing)$ directly in terms of $Q(s2, \cdot)$ -- which is exactly the next set of $Q$-values that we would be interested in keeping track of again:
$$Q(s_1, \varnothing) = \mathbb{E} \left[ 0 + \gamma \max_{a'} Q(s_2, a') \right]$$
So if we want to skip learning $Q$-values for $s_1$ (because it's kind of a waste of effort), we can just use this definition and plug it straight into the update rule for $Q(s_0, a_0)$. $Q$-learning is inherently an algorithm that just concrete samples of experience to estimate expectations (and this is a major reason why it typically uses learning rates $\alpha < 1.0$, so we can simply get rid of the expectation operator when doing this:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \left[ \gamma \max_{a'} Q(s_2, a') \right] - Q(s_0, a_0) \right)$$
and this is basically the update rule that you suggested. Note; here I assumed that you receive your rewards directly when you do take actions in your SMDP, which is why I had $R_1$ as a possibly-non-zero reward, and always $R_2 = 0$. I suppose you could also in some cases envision SMDPs where the reward only arrives on the next SMDP-time-step, and that the amount of time that ends up being elpased in between the two events is important to take into account via the discount factor $\gamma$. So you could also choose to model a problem where $R_1 = 0$ and $R_2$ may be non-zero, and this would yield a different update rule (I think one where the reward gets multiplied by $\gamma^{\tau - 1}$? not sure, would have to go through the steps again).

What measure should be used in the SMDP setting? I would be thankful if someone can explain the Q-Learning algorithm for the SMDP problem with this setting.

I think it would be important to involve the amount of time that you take somehow in your evaluation criterion. You could run episodes for a fixed amount of time, and then just evaluate agents based on the sum of rewards. If you don't run for a fixed amount of time (but instead for a fixed number of steps, each of which may take variable amounts of time, for example), you would probably instead want to evaluate agents based on the average rewards per unit of time. You could also include discount factors in your evaluation if you want, but probably don't have to.

Moreover, I am wondering when Q-functions are updated. For example, if a customer enters our website and purchases a product, we want to update the Q-functions. Suppose that the planning horizon (state $S_0$) starts at 10:00 am, and the first customer enters at 10:02 am, and we sell a product and gain $R_1$ and the state will be $S_1$. The next customer enters at 10:04 am, and buy a product, and gain reward $R_2$ (state $S_2$). In this situation, should we wait until 10:02 to update the Q-function for state $S_0$?

This depends on you state representation, and how you model a "state", and to what extent previous actions have influence the state you end up in. Keep in mind that the update rule for $Q(S_0)$ also requires for $S_1$ (or even $S_2$ if $S_1$ is a "dummy state" that you skip) to have been observed. So, if your state representation includes some features describing the "current customer" for which you want to pick an action (do you offer them a discount or not, for example?), then you can only update the $Q$-value for the previous customer when the next customer has arrived. This model does assume that your previous actions have some level of influence over the future states that you may end up in though. For example, you might assume that if your actions make the first customer very happy, you get a better reputation and are therefore more likely to end up in future states where other customers visit more frequently.


Value iteration algorithm from pseudo-code to C++

At line 10: The value function is represented as V[s], which has a bracket notation like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?

This is just a notation that the value function is a mapping between S and the real numbers. When implementing you would want to store V(s) and π(s) as either arrays or some kind of hashmap like unordered_map (in which case your states must be hashable). Here we also have to assume that this container will fit in the memory, otherwise the value function has to be approximated with function approximation which is not covered explicitly by this pseudo-code.

  At line 12: Vk would be the element in index k inside of array V?

No. This line says that you would need to store multiple value functions during the algorithm, basically a list of functions (or think of it as an array of functions or a two-dimensional array, where k goes along one dimension and s goes along the other dimension) and if you look at the loop following it, it shows you how you need to keep appending new value functions to the end of this list. However notice (in line 19) that you only need the value function from the previous iteration to calculate your new value function, which means that you will never need to store more than two value functions (the new one and the previous one).

  At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.

Yes. This loop calculates your value function.

  Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?

Here you have to iterate through all the actions a in state s and calculate the sum according to the formula in the max function. Then you take the highest value and set the new V(s) to be this value.

  Line 20: I'm interpreting this a the end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So Would I would have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)

You have to check for all s if the condition satisfies and you may only break the loop if all these values are lower than the threshold θ.


What are pros and cons of Bi-LSTM as compared to LSTM?

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to learn better representations of single words. There is no special training step or units added, the idea is just to read a sentence forward and backward to capture more information.

And as trivial as the idea sounds, it works, in fact in the original paper the authors managed to achieve state-of-the-art scores in three tagging tasks, namely part-of-speech tagging, chunking and named entity recognition.

Even though it must be said that these scores were not dramatically higher compared to other models, and also the complete architecture included a Conditional Random Field on top of the Bi-LSTM.

Probably the most important aspect to stress out is that the authors performed two interesting comparison tests: one using random embedding initialisation and another one using only words (unigrams) as input features. Under these two test conditions Bi_LSTM (with CRF on top) outperformed significantly all other architectures, proving that Bi-LSTM representations are more robust than representation learned by other models.

I would like also to make a side note regarding human reading. It make sense to consider unidirectional sequence models as the most reasonable to emulate human reading, because we experience reading as a movement of the eyes that goes from one direction to the opposite. But the reality is that saccades (really rapid unconscious eye movement) and other eye movements play an enormous rule in reading. Which means that also we humans do continuously look to past and future words as well in order to understand the purpose of a word or sentence we're processing. Of course in our case these movements are directed by implicit knowledge and habits that allow us to direct our attention only to important words/parts (for example we barely read conjunctions) and it is interesting to notice that now state-of-the-art models based on transformers try to learn exactly this, where to pay attention rather than single probabilities for each word in a vocabulary.


What is the fundamental difference between an ML model and a function?

A model as a set of functions
In some cases in machine learning, a model can be thought of as a set of functions, so here's the first difference.
For example, a neural network with an arbitrary vector of parameters $\theta \in \mathbb{R}^m$ is often denoted as a model, then a specific combination of these parameters represents a specific function. More specifically, suppose that we have a neural network with 2 inputs, 1 hidden neuron (with a ReLU activation function, denoted as $\phi$, that follows a linear combination of the inputs), and 1 output neuron (with a sigmoid activation function, $\sigma$). The inputs are connected to the only hidden unit and these connections have a real-valued weight. If we ignore biases, then there are 3 parameters, which can be grouped in the parameter vector $\theta = [\theta_1, \theta_2, \theta_3] \in \mathbb{R}^3 $. The set of functions that this neural network represents is defined as follows
$$
f(x_1, x_2) = \sigma (\theta_3 \phi(x_1 \theta_1 + x_2 \theta_2)) \tag{1}\label{1},
$$
In this case, the equation \ref{1} represents the model, given the parameter space $\Theta = \mathbb{R}^3$. For any specific values that $\theta_1, \theta_2,$ and $\theta_3$ can take, we have a specific (deterministic) function $f: \mathbb{R} \rightarrow [0, 1]$.
For instance, $\theta = [0.2, 10, 0.4]$ represents some specific function, namely
$$
f(x_1, x_2) = \sigma (0.4 \phi(x_1 0.2 + x_2 10.0)) \tag{2}\label{2}
$$
You can plot this function (with Matplotlib) for some values of the inputs to see how it looks. Note that $x_1$ and $x_2$ can be arbitrary (because those are just the inputs, which I assumed to be real numbers).
This interpretation of a model is roughly equivalent to the definition of a hypothesis class (or space) in computational learning theory, which is essentially a set of functions. So, this definition of a model is useful to understand the universal approximation theorems for neural networks, which state that you can find a specific set of parameters such that you can approximately compute some given function arbitrarily well, given that some conditions are met.
This interpretation can also be applied to decision trees, HMM, RNNs, and all these ML models.
A model in reinforcement learning
The term model is also sometimes used to refer to a probability distribution, for example, in the context of reinforcement learning, where $p(s', r \mid s, a)$ is a probability distribution over the next state $s'$ and reward $r$ given the current state $s$ and action $a$ taken in that state $s$. Check this question for more details. A probability distribution could also be thought of as a (possibly infinitely large) set of functions, but it is not just a set of functions, because you can also sample from a probability distribution (i.e. there's some stochasticity associated with a probability distribution). So, a probability distribution can be considered a statistical model or can be used to represent it. Check this answer.
A function as a model
A specific function (e.g. the function in \ref{2}) can also be a model, in the sense that it models (or approximates) another function. In other words, a person may use the term model to refer to a function that attempts to approximate another function that you want to model/approximate/compute.


Which loss function is the brain optimizing in order to learn advanced visual skills without expert/human supervision?

I think you are slightly confusing 2 problems. 1 being classification of meta visual elements and the other being the visual system itself.

Our visual system, when it comes to processing information, has had billions of years of iteration(training), so that at birth(and before), we are already tuned for the processing of visual stimuli, as well as have the mechanisms to decipher objects in our spatial field of view.

These two papers(L1, L2), have a great deal of information about the evolution of our visual system and its processing. The second speculates on the connection of said evolution and the construction of "seeing systems" very interesting.

For further inquiry on this in particular, check out David Marr. He was probably the most influential early computer vision mind. He still is mentioned in many top-down AGI and computer vision research projects to this day.


How can I generate natural language sentences given logical structures that contain the subject, verb and target?

It might be simpler to generate the language output yourself since you've already got a concrete concept structure in code.

Maybe you'll also want to be aware of potential future applications such as parsing your own output back to the inputs.

With that said, you could investigate Markov Chains or Google around a bit for Natural Language Processing and Natural Language Generation.

https://stackoverflow.com/questions/33068943/library-for-generating-natural-language-verbs-in-javascript


Why MLP momentum term must be in the range 0-1?

If gradient descent is like walking down a slope, momentum would be the literal momentum of the agent traversing the hyperplane.

Under that analogy then, momentum factor would be analogous to the friction coefficient, with 1 being max friction and 0 being no friction.

You should be able to see why there can't be friction beyond that range: if friction = 1 it would be identical to having no friction; if friction <= 0 then by conservation of energy gradient descent will not find a local minima; if friction > 1 then gradient descent would be moving backwards.


Can the attention mechanism improve the performance in the case of short sequences?

They shouldn't have any issues with short sequences, as short dependencies are easier to learn. The only difficult cases are long dependencies which is where most of the research is geared at. However, this is assuming that by "short sequence" you mean a sequence of text that is fully contained within itself, i.e. there is no cross-sequence dependencies.
For example, if you have a really long paragraph that doesn't fit in a transformer model, you would have to break that paragraph into many "short sequences", but each of these sequence may have a dependency that depends on another sequence, i.e. cross-sequence dependencies. For these cross-sequence dependencies, any model with recurrence should do better than ones without (e.g. RNN, LSTM, Transformer-XL).
If each short sequence is self-contained, then all of the models should perform pretty well.


Can I use one-hot vectors for text classification?

One hot encoding is a good strategy to apply with categorical variables that assume few possible values. The problem with text data is that you easily end up with corpora with a really large vocabulary. If I remember correctly the IMDb dataset contains around 130.000 unique words, which means that you should create a network with an input matrix of size 130.000 x max_length where max_length is the fixed maximum length allowed for each review. Apart from the huge size, this matrix would also be extremely sparse, and that's another big issue in using one-hot encoding with text.
For these reasons, I really doubt you would achieve any good results with a simple one-hot encoding. Embeddings where actually designed precisely to overcome all these issues, they have fixed reasonable size, they assume continue values between 0 and 1, which is desirable for deep neural networks, and they can be treated as "extra" trainable weights of a network.
If you really want to avoid embeddings I would suggest you to use (or implement, I don't think it will be so hard) a term frequency–inverse document frequency vectoriser. It is closer to one-hot encoding in the fact that it is based on the creation of a huge co-occurances matrix between words, but at least the values are continuous and not dichotomous. Nevertheless I would not expect high performances with the tf-idf either, simply because this type of encoding works best with shallow models like the Naive Bayes rather than deep models.


Is there research that employs realistic models of neurons?

It looks like you really have two questions here. I'll try to answer the first one, and you should think about making a separate question for the second.

There is research into using simulated models of biologically realistic neurons. While there are large projects like the Human Brain Project aimed at simulating human brains, there is also a lot of lower-level AI research. SPAWN is an interesting system that got a lot of press a few years ago, and has continued to be developed since then. It uses realistic neurons to simulate several brain-regions at once, creating a surprisingly general AI system that could perform many types of motor and vision tasks using the same basic design.


What is the meaning of the square brackets in ant colony optimization?

The square brackets $[]$ in $[\tau_{ij}]^\alpha$ and $[\eta_{ij}]^\beta$ may be just a way of emphasing that the elements $\tau_{ij} \in \mathbb{R}$ and $\eta_{ij} \in \mathbb{R}$ of respectively the matrices $\mathbf{\tau} \in \mathbb{R}^{n \times n}$ and $\mathbf{\eta} \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the graph) are respectively raised to $\alpha$ and $\beta$, so they could have used also other type of brackets, for example, $()$. It may also be a way of indicating that $[\tau_{ij}]$ and $[\eta_{ij}]$ are $1 \times 1$ matrices or vectors that contain respectively the scalars $\tau_{ij}$ and $\eta_{ij}$, so you are multiplying matrices or vectors (dot product).

This notation is also used in the paper that introduced the ant colony system (ACS) (and it is probably used in many other papers related to ant colony optimization). See equation 1 of Ant Colony System: A Cooperative Learning Approach to the Traveling Salesman Problem (1997) by Dorigo and Gambardella.


How to implement SVM algorithm from scratch in a programming language?

Based on this repository:
https://github.com/arkm97/svm-from-scratch/blob/master/SVM_from_scratch.ipynb
I will try to reverse engineering that concept:
So firstly there is issue for DataCleaning(removing 0,values, serialize, normalize)
normalizing dataset (replaced by its difference from the mean)
credit_df_norm = (credit_df - credit_df.mean())/(credit_df.std())

Divide dataset into Train & Test:
train_df = credit_df_norm.drop(target, axis=1).loc[:training_points]
train_target = credit_df_norm[target].replace(0, -1).loc[:training_points]

And here is starting the clue of that algorithms (with maths etc.)

given data points $\vec x_j \in \mathbb{R}^{1 \times N}$ and targets
$y_j = \pm 1$, where $j = 1, \dots, M$, find the maximum-margin
hyperplane that separates the two classes ($y_j = 1$ and $y_j = -1$).
Let $\vec w$ be the vector normal to the hyperplane. We want to find
$\vec w$ that satisfies
$$ y_j (\vec w \cdot \vec x_j + b) \geq 1 $$ The dual formulation of
the above is equivalent to maximizing the following over the
multipliers $\vec \alpha$:
$$ L(\vec \alpha) = \vec y \cdot \vec \alpha  - \frac 1 2  \vec \alpha
> K \vec \alpha^T$$ subject to the constraints $\sum_{j=1}^M \alpha_j =
> 0$ and $y_j \alpha_j \geq 0$. The matrix $K$ defines the kernel of the
SVM; I've chosen $K_{jk} = k(\vec x_j, \vec x_k) = \vec x_j \cdot \vec
> x_k$. The parameters of the plane are recovered from $\vec w = \vec
> \alpha \cdot \vec x$ and $b = y_j = \vec w \cdot \alpha_j$ for $j$
such that $\alpha_j \neq 0$

Let's divide it to the single factors:
we need to find maximum-margin-hyperplane for separate into two classes:
Separation cause SVN is classification - so we need to classify elements into categories.
 = 1
 = −1
This pattern allow to find hyperplane(?)
(⃗ ⋅⃗ +)≥1
The next pattern:
(⃗ )=⃗ ⋅⃗ −12⃗ >⃗
The author for kernel SVN choose those pattern:
=(⃗ ,⃗ )=⃗ ⋅>⃗
And the parameters for plane are here:
⃗ =>⃗ ⋅⃗  and ==⃗ ⋅ for  such that ≠0
Reference:
https://arxiv.org/pdf/1307.0471.pdf

In this line are added the kernel
Popular kernels are:

Polynomial Kernel,

Gaussian Kernel,

Radial Basis Function (RBF),

Laplace RBF Kernel,

Sigmoid Kernel,

Anove RBF Kernel
k_value = np.array(train_df @ train_df.T + np.identity(len(train_target))*1e-12)

The structure of k_value (kernel matrix) is an array.

Next there is use of cholesky decomposition—cholesky - which is next complicated concept.
np.linalg.cholesky(k_value)

Next some bits of data converting:
alpha = cp.Variable(shape=train_target.shape)
beta = cp.multiply(alpha, train_target) # to simplify notation
K = cp.Parameter(shape=k_value.shape, PSD=True, value=k_value)

# objective function
obj = .5 * cp.quad_form(beta, K) - np.ones(alpha.shape).T @ alpha

# constraints
const = [np.array(train_target.T) @ alpha == 0,
        -alpha <= np.zeros(alpha.shape),
        alpha <= 10*np.ones(shape=alpha.shape)]
prob = cp.Problem(cp.Minimize(obj), const)

The next step is recreate the hyperplane:
w = np.multiply(train_target, alpha.value).T @ train_df
S = (alpha.value > 1e-4).flatten()
b = train_target[S] - train_df[S] @ w
b = b[0]
# b = np.mean(b)

And finally - the classification and evaluation:
def classify(x):
    result = w @ x + b
    return np.sign(result)

correct = 0
incorrect = 0
validation_set = credit_df_norm.drop(target, axis=1)
predictions = []
for i, x in validation_set.iterrows():
    my_svm = classify(x)
    if my_svm==credit_df_norm[target].replace(0, -1)[i]: correct +=1
    else: incorrect +=1
    predictions.append(my_svm)
predictions = np.array(predictions)

print(f"fraction correct: {correct/(correct + incorrect)}")

That was the more detailed implementation - for shorcut there is an interesting article:
https://towardsdatascience.com/svm-and-kernel-svm-fed02bef1200
There are existing function from another library - that can be reused in simple manner:
# Fitting SVM to the Training set
from sklearn.svm import SVC
classifier = SVC(kernel = 'rbf', C = 0.1, gamma = 0.1)
classifier.fit(X_train, y_train)

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# use seaborn plotting defaults
import seaborn as sns; sns.set()

Visualized Dataset:

Dataset after apply hyperplanes:


When will the number of neurons in AI systems equal the human brain?

Some back of the envelope calculations :

  number of neurons in AI systems

The number of neurons in AI systems is a little tricky to calculate, Neural Networks and Deep Learning are 2 current AI systems as you call them, specifics are hard to come by (If someone has them please share), but data on parameters do exist, parameters are more analogous to synapses (connections) than neurons (the nodes in between connections) somewhere in the range of 100-160 billion is the current upper number for specialized networks.

Deriving the number of neurons in AI systems from this number is a stretch since these AIs emulate certain types of connections and sub assemblies of neurons, but let's continue...

  equal those of the human brain?

So now let's look at the brain, and again this are all contested numbers. Number of neurons ~ 86 Billion, Number of Synapses ~ 150 Trillion, another generalization: average number of synapses per neuron ~ 1,744.

So now we have something to compare, and I can't stress this enough, these are all wonky numbers, so let's make our life a little easier and divide :

Number of Synapses (Brain ) : 150 trillion /  Number of parameters AIs : 150 billion = 1,000 or in other words current AIs would have to scale by a factor of one thousand their connections to be on par with the brain...

Number of Neurons (Brain ) : 86 Billion / Number of Neurons AIs ( 150 billion / 1,744 )  = 86 Million equivalent AI Neurons

Which makes sense, mathematically at least : you can multiply the factor ( 1000 ) times the current number of equivalent AI Neurons ( 86 million) to get the number of neurons in the human brain (86 Billion)

  When ?

Well,let's use  moore's law ( number of transistors processing power doubles about every 2 years ) as a rough measure of technological progress:

     #AI NEURONS            YEAR
     86,000,000             2016
     172,000,000            2018
     344,000,000            2020
     688,000,000            2022
     1,376,000,000          2024
     2,752,000,000          2026
     5,504,000,000          2028
     11,008,000,000         2030
     22,016,000,000         2032
     44,032,000,000         2034
     88,064,000,000         2036

     # NEURONS HUMAN BRAIN
     86,000,000,000

So, if all this made sense to you, somewhere around the year 2035.


Why was the VC dimension not defined for all configurations of $d$ points?

I would say, that meaning of VC dimension is at least a possibility to implement any function on the given number of points for some case, not an express any function on $n$ points.
Yes, you are right, that this definition, unfortunately, is not very useful in practice.
Say, family of functions $\text{sign}(\sin(ax))$ has infinite $\text{VC}$ dimension. There exists an infinite sequence $x_n$, such that by tuning the parameter $a$ one can implement any logical function $0 1 0 \ldots 1$ on these points. However, this doesn't make this class of functions an outstanding ML algorithm.


What is meant by "real-valued argument" in this context of the convolution operation?

In its most raw form, convolution is defined as: $(f*g)(t) = \int_{-\infty}^\infty f(\tau) \cdot g(t-\tau) d\tau$.
Here, t doesn't represent the time domain. Infact, it represents the real valued argument the book is talking about. In this notion, at moment t, convolution can be thought of as a weighted average of the function $f(\tau)$ weighted by $g(–\tau)$, which is simply shifted by amount t.


How to use DQN when the action space can be different at different time steps?

There are two relevant neural network designs for DQN:

Model q function directly $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient.

Model all q values for given state $Q(s,\cdot): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, so neural network takes input of current state and outputs all action values related to that state as a vector.

For the first architecture, you can decide which actions to evaluate by how you construct the minibatch. You pre-filter to the allowed actions for each state.
For the second architecture, you must post-filter the action values to those allowed by the state.
There are other possibilities for constructing variable-length inputs and outputs to neural networks - e.g. using RNNs. However, these are normally not worth the extra effort. A pre- or post- filter on the actions for a NN that can process the whole action space (including impossible actions) is all you usually need. Don't worry that the neural network may calculate some non-needed or nonsense values.


Can we use Multiple data as Input in a NN for a single Output?

Yes, it can be those multiple inputs, you would have to design the network like that (to have 3 input layers and then pool them afterwards).
But, an easier solution would be to flatten this array and put it through 1D CNN and finally put it through a softmaxx.


Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?

Yes this is possible and is exactly what a Markov process aims to accomplish.

The Hidden Markov Model can be considered a simple dynamic Bayesian network. This means that it will generate a probabilistic outcome for a number of states (classes) given a sequence of inputs.

This can be used for a classification task based on a threshold probability to make a confirmed decision.

Let's take a simple example

The above example is a model to determine  how a person will be feeling given some sample data of the previous days.

# Sample data
data = ['Healthy', 'Healthy', 'Fever']

Using this data, it's possible to calculate the probability of each possible state (outcome).

# Probability after 2 Healthy and 1 Fever day
P = 0.6 * 0.7 * 0.3

# Final day state 'Fever'
# Calculating probabilities for each state
Dizzy  = P * 0.6
Cold   = P * 0.3
Normal = P * 0.1

We can calculate the probability of each state and this will be unique for every unique sequence of historical data.


Is Really "AI" Light Years Away from achieving Cognitive Ability of Human?

If certain philosophers are correct, Artificial General Intelligence will be, like fusion power, "always twenty years away".  For the true believers, it is an inevitability, and opinions vary.

It may be most useful to look at the unreliability human predictions in this area.

There was an article in the MIT Tech Review in 2017 that contained this graphic, based on a survey of researchers in 2015:

 SOURCE: Experts Predict When Artificial Intelligence Will Exceed Human Performance (MIT Tech Review)

Most notable is that AlphaGo soundly bested the top human player in Go in March of 2016, years before even the most optimistic expert projections.

Everyone is just guessing, and we still don't if AGI is possible, or merely a myth.

Nevertheless, recent breakthroughs are promising!


Algorithms and strategies to help judges rule cases

Genuine success in this area would be beyond the state-of-the-art in research, since it likely requires analogising from relational knowledge extracted from text. In recent years, techniques for working with natural language have tended to be statistical, and are therefore somewhat deficient in this respect. You could look at 'bag of words'/latent semantic analysis approaches, but they are likely to generate many false positives unless a lot of ad hoc conditions are added manually. More recent work on 'treenets' (paper here) is more structurally informed, but is still a relatively new area.


Which simulation platform is used by DeepMind (and others) to handle inverse kinematics musculoskeletal?

DeepMind used MuJoCo (see also the related paper MuJoCo: A physics engine for model-based control) for the simulations, as they stated in section 3.1 of their paper Emergence of Locomotion Behaviours in Rich Environments (2017), which is the paper you should read to know more about their results related to those animations of skeletons that try to walk or jump (but do it weirdly).


Do we have cross-language vector space for word embedding?

You can try to read about MUSE (Multilingual Unsupervised and Supervised Embeddings) by Facebook. You can read it from its Github or this article. They also provide the FastText dictionary format (.vec file) for some languages.

Their original paper shows how it aligns the vector of words from two different languages:


How could self-driving cars make ethical decisions about who to kill?

It shouldn't. Self-driving cars are not moral agents. Cars fail in predictable ways. Horses fail in predictable ways.

  the car is heading toward a crowd of 10 people crossing the road, so
  it cannot stop in time, but it can avoid killing 10 people by hitting
  the wall (killing the passengers),

In this case, the car should slam on the brakes. If the 10 people die, that's just unfortunate. We simply cannot trust all of our beliefs about what is taking place outside the car. What if those 10 people are really robots made to look like people? What if they're trying to kill you?

  avoiding killing the rider of the motorcycle considering that the
  probability of survival is greater for the passenger of the car,

Again, hard-coding these kinds of sentiments into a vehicle opens the rider of the vehicle up to all kinds of attacks, including "fake" motorcyclists. Humans are barely equipped to make these decisions on their own, if at all. When it doubt, just slam on the brakes.

  killing animal on the street in favour of human being,

Again, just hit the brakes. What if it was a baby? What if it was a bomb?

  changing lanes to crash into another car to avoid killing a dog,

Nope. The dog was in the wrong place at the wrong time. The other car wasn't. Just slam on the brakes, as safely as possible.

  Does the algorithm recognize the difference between a human being and an animal?

Does a human? Not always. What if the human has a gun? What if the animal has large teeth? Is there no context?

  Does the size of the human being or animal matter?
  Does it count how many passengers it has vs. people in the front?
  Does it "know" when babies/children are on board?
  Does it take into the account the age (e.g. killing the older first)?

Humans can't agree on these things. If you ask a cop what to do in any of these situations, the answer won't be, "You should have swerved left, weighed all the relevant parties in your head, assessed the relevant ages between all parties, then veered slightly right, and you would have saved 8% more lives." No, the cop will just say, "You should have brought the vehicle to a stop, as quickly and safely as possible." Why? Because cops know people normally aren't equipped to deal with high-speed crash scenarios.

Our target for "self-driving car" should not be 'a moral agent on par with a human.' It should be an agent with the reactive complexity of cockroach, which fails predictably.


You're looking for a heatmap. Check out e.g. https://stackoverflow.com/q/33282368/3924118 (if you like Python more than the others). See also this documentation.


What is meant by "generate the data" in describing the difference between on-policy and off-policy?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
In both the SARSA and Q learning pseudocode from the book, there is a behaviour policy that selects the next action to take. Other than the initial start state, this policy drives the observations that the learning process must handle - every action is directly selected by it, and due to how MDPs work, every next state and reward are indirectly influenced by it. It is this behaviour policy therefore that must "generate the data".
You can see from the pseudocode that both algorithms describe this behaviour policy as "policy derived from Q (e.g. $\epsilon$-greedy)".
The difference in the two algorithms is in which target polcy is being used:

In SARSA, the update is based on the Bellman equation for the same policy that generated the most recent set of data for S, A, R, S' and A' - all of these values will be directly or indirectly caused by the behaviour policy. So the whole equation uses the same data that was generated. Another way to put that is behaviour policy and target policy are the same.

In Q learning, the $\text{max}_a Q(S',a)$ from the Bellman optimality equation removes the need to use A' in the update. Effectively it is a local search for actions to find one that is potentially better than A', and runs the update as if that one was the one that was taken. This revision of A' is what makes the off-learning evaluate a different target policy to the behaviour policy that generated the rest of the data used in the update.


How do I change the values of a neural net

This is too broad a topic to answer directly.

If you are at the beginner stage with neural networks, you will need to learn some basic theory of the maths of neural networks, before the code will make sense. Although it is possible to write neural network code with only a vague understanding of what is going on, it is not a great way to learn for the future, and more advanced neural network features will likely remain beyond your comprehension.

The maths for back propagation are not that difficult conceptually, it is literally just the Chain Rule from basic calculus applied repeatedly. You do this to get a gradient that tells you the direction that an error function would increase in, then take a small step in the opposite direction. Repeat this over time and the error should reduce.

Despite the simplicity once you already know it, there are a lot of moving parts to training a neural network. The formulae for back propagation include multiple symbols with different meanings, and typically indexed in at least 3 dimensions all at once. It can look like a wall of impenetrable maths, especially if you have got a bit rusty at basic calculus and matrix multiplication, and need to review it.

The answer is to take your time and study the basics carefully. There are many resources out there. Introductory material should cover:

Revision of basic linear algebra. Just vectors, matrices and how to add and multiply them.
Revision of basic calculus. Just differentiation and gradients. The chain rule is handy for later.
Linear regression and logistic regression models trained using gradient descent. These are introductory models that naturally lead to layered neural networks.
Neural networks and back propagation.

There are many courses and books that attempt to teach you these things. It is hard to say in general which one would work best for a particular student. The best thing to do would be to go to a learning resource that you already like, such as Coursera or EDX, and search for "neural networks beginner" or similar.

Here are a couple of courses that I can personally recommend. I cannot create a comprehensive list, so if these do not apply on a quick browse (or if the links finally go stale after a few years), then you should search for something more suitable bearing in mind the syllabus suggestions above:

Andrew Ngs' Machine Learning covers more than just neural networks, but the extra parts apply to other aspects of ML, such as sensible approaches to validating and testing that anyone building predictive models should get to know. Code exercises are in Matlab or Octave.
Andrew Ngs' Introduction to Deep Learning starts with basic regression, and moves very quickly to practical builds of neural networks using TensorFlow. It is part of a five course "Specialisation" that works up to advanced NN models developed in the last three years.
Geoffrey Hinton's Neural Networks for Machine Learning covers some historical models (such as perceptrons) and some more obscure models, and the material is hard to understand, but quite rewarding if you are up to it. Not generally recommended to beginners, but if you have solid maths and computing knowledge already, at degree level, but never looked at a neural network before, this might be the right style of course for you.

They all happen to be Coursera courses . . . I am not affiliated, it is just that the relevant courses that I have personally taken have been on that platform.


How can we efficiently and unbiasedly decide which children to generate in the expansion phase of MCTS?

The first thing to consider in this question is: what do we mean when we talk about "generating a child/node". Just creating a node for a tree data structure, and allocating some memory (initialised to nulls / zeros) for data like deeper children, visit counts, backpropagated scores, etc., is rarely a problem in terms of efficiency.

If you also include generating a game state to store in that node when you say "generating a node", that can be a whole lot more expensive, since it requires applying the effects of a move to the previous game state to generate the new game state (and, depending on implementation, probably also requires first copying that previous game state). But you don't have to do this generally. You can just generate nodes, and only actually put a game state in them if you later on reach them again through the MCTS Selection phase.

For example, you could say that AlphaZero does indeed generate all the nodes for all actions immediately, but they're generally "empty" nodes without game states. They do get "primed" with probabilities computed by the policy network, but that policy network doesn't require successor states inside those nodes; it's a function $\pi(s, a)$ of the current state $s$ (inside the previous node), and the action $a$ leading to the newly-generated node.

But if you're really sure that, for your particular problem domain, the generation of nodes itself already is inefficient, then...

  [...] This is clearly biased in favor of children earlier in the stream.

Yes, you would get a significant bias with such a stream-based approach, probably wouldn't work well.

  [...] This may be harder to implement than the previous version because computing the number of children may be as hard as computing the children themselves. [...]

Again I agree with your observation, I don't think there are many problems where this would be a feasible solution.

  I believe that if the number of iterations of MCTS remaining, X_t, is larger than the number of children, N, then it doesn't matter what you do, because you'll find this node again the next iteration and expand one of the children.

This would only be correct for the children of the root node. For any nodes deeper in the tree, it is possible that MCTS never reaches them again even if $X_t > N$, because it could dedicate most of it search effort to different subtrees.

I think your solution would have to involve some sort of learned function (like the policy network in AlphaZero) which can efficiently compute a recommendation for a node to generate, only using the inputs that are already available before you pick a node to generate. In AlphaZero's policy network, those inputs would be the state $s$ in your current node, and the outward actions $a$ (each of which could lead to a node to be generated). This would often actually be very far from unbiased, but I imagine a strong, learned bias would likely be desireable anyway if you're in a situation where the mere generation of nodes is a legitimate concern for performance.


What is the definition of a heuristic function in the BayesChess paper?

By far the most common form of heuristic evaluation functions for Chess-playing (or, really, any game-playing) agents are simple linear functions. At least when we're talking about handcrafted features that's the case, of course all the hype with Deep Neural Networks in more recent years is different. So, when it's not specified in a paper like this exactly what their heuristic evaluation function looks like, you can relatively safely assume it's just a linear function.

With linear function, I mean that you have vectors of features $\boldsymbol{\phi}(s)$ for your states $s$, and a vector of weights $\boldsymbol{\theta}$, and the evaluation $f(s)$ of a state $s$ is simply given by the dot product (summing up all the multiplications of feature values with their corresponding weights):

$$f(s) = \boldsymbol{\phi}(s)^{\top} \boldsymbol{\theta} = \sum_i \phi_i(s) \times \theta_i(s),$$

where the subscript $i$ indicates taking the $i^{th}$ element of a vector.


When to convert data to word embeddings in NLP

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part of the graph.


What are some good papers or resources for aspect extraction and opinion modelling from video or audio?

The paper A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews could be useful for your purposes, although I have read only the abstract

Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews. In light of this issue, we propose a multimodal approach for mining fine-grained opinions from video reviews that is able to determine the aspects of the item under review that are being discussed and the sentiment orientation towards them. Our approach works at the sentence level without the need for time annotations and uses features derived from the audio, video and language transcriptions of its contents. We evaluate our approach on two datasets and show that leveraging the video and audio modalities consistently provides increased performance over text-only baselines, providing evidence these extra modalities are key in better understanding video reviews.

and looked at some of the diagrams and figures in the paper.


What are the current approaches for AI to learn a foreign language just from English books?

Current approaches for learning a language require having a large corpus of that language; it also doesn't seem reasonable to expect that it will ever be possible to learn about language A by extracting information from a corpus from an unrelated language B.

Even if you want to learn about human languages in general (what sorts of things are true about grammar, vocabulary, and so on), that relies having many languages as training data, so that you can see the different ways of doing things instead of assuming that the way they're done in English is the way they're done in every language.

(There is work in automatic translation that goes from a language to 'concept-space', then goes from that 'concept-space' to another language, so that you can build an English-Chinese translator by building two separate English-Concept and Chinese-Concept translators, instead of ever needing material that directly links English and Chinese. The obvious benefit of this is scalability; in order to make translators for a new language to any other language, you just need to learn that language and the models build themselves.)


Is there a ReLU-like activation function that concatenates positive and negative values?

It seems I have found it. It is called concatenated ReLU (CReLU).

  Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (the original paper that introduced CReLU).
A TensorFlow implementation

There is also Negative CReLU. It seems that the difference is only the sign.

$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$


Understanding the application of Sobel kernel followed by ReLU to a zero-padded image

I assume that you would like to use convolution with padding with the same size of output matrix as this picture. You messed up calculations with convolution with "full padding". If we imagine these two matrices as windows that slide on each other, you can see that the filter is symmetrically inverted. I used a little bit different filter to show you in a better way how it works (I changed the last row to [-3, 0 3]).

Assuming these matrices:

You should add to your picture matrix two rows and columns of zeros:

Then you can start matrix multiplication, but notice that the filter is symmetrically inverted. The result shown as matrix $4 \times 4$ is the convolution with "full padding". The $2 \times 2$ matrix in the middle is a result of convolution with the "same padding", that you requested.

Next step:

Some iterations later:

And later:

And finally:

For convolution with the same size, the result will be the small $2 \times 2$ matrix in the middle.

After using the ReLU function, the result will be exactly the same.

So with using of your filter, the result would look like [[0,90], [0,90]].


Can layers of deep neural networks be seen as Hopfield networks?

Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.
There are two kinds of neural networks:

Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)

Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at this paper which compares the three models.
Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.


What technologies are needed for a self-driving car?

You're going to need some way to 'see' the area around the car, and to track the speed of nearby objects. Google uses a combination of LIDAR, radar, conventional cameras, and occasionally sonar (see  here for a high-level overview). This technology is quite expensive, and can easily cost thousands of US dollars.

However, a bigger obstacle than the expense of the hardware (which would be smaller for a table-top prototype) is the software complexity. Like many major projects, the software for self-driving cars is the result of years of work from AI research teams, and thus extremely difficult to duplicate on your own.

That said, you're not trying to make a state-of-the-art self-driving car.  Assuming you're an expert in image processing and robotics, you can probably create a basic prototype, (like something that drive in a limited table-top environment). However, it's still going to take a lot of time and money.


What difference does it make whether Actor and Critic share the same network or not?

One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for both policy and value function – you are essentially parameter sharing the feature-extraction part of your neural network, and fine tuning the different heads of your network on the two different tasks: action choice and value prediction.
Using two vs one networks is mostly a question of sample efficiency: theoretically, in both case your AC algorithm should work. In practice however, it will usually be useful to have parameter sharing as the representations encouraged by one of the tasks might be highly useful for the other and vice-versa, enabling one task to cause the other task to get unstuck from local optima. Another reason why this might work better is simply because you do not have to learn the same (or at least similar) representations from scratch twice – leading to a more sample efficient training.


XOR-solving neural network is suffering from local minima

From the log you can see that the loss value is still big compared to the first image, but at least it decrease on each iteration (even very very small). So I suggest you to:

try to use bigger learning rate, I think you model is underfiting.
try to use another weight initialization method

I think the first solution will solve this problem as I see no problem in the code.


What is a good model for this objective?

I will try to give another perspective: Solve it without machine learning model

Your problem is try to find the most overlapping point. If the image above is image that you used in your case, you can solve it directly by applying some computer vision algorithms.

Try to create some binary image based on the color of the dots. if you are not sure with the available colors on your image, you can list of pixel colors that not black and white uniquely at first. So if there are four colors you need to generate four different binary image. Create a simple condition or a complex one, for example:

if pixel[i,j]=red then
    pixel[i,j]=white
else
    pixel[i,j]=black

Get its location by searching "the white" over your image or use blob detection method (it'll be a little bit tricky if the actual image always have different axis scale). You can save it as a list of coordinate of each color.
What happen if you can't see the dot fully because it's overlapping with another dot? Find the pattern. In your image, dots appear with certain pattern. If you can find two consecutive dots horizontally and vertically, you can predict the position of all your dots.
Find the most overlapping position from your list.

Pros

The result may more accurate than using machine learning model
Faster, you don't need to train it first

Cons

Finding dot's location in image with different axis will be difficult, but it's still solvable
It'll be difficult to predict the pattern if many dots are missing because overlapped by other dots


Do models train better if the labelling information is more specific (or dense)?

It depends on what is your ultimate goal. If your goal is to simply classify the object in the image, having more complex output won't help. Simpler output representation yields better result. If your goal is to detect the bounding box, output the bounding box. There is no need for a more complex output feature. If you use a segmentation method for bounding box detection, it is more prone to error because of it's excess output features.

Assume you are given a gradr 6 math test. If you do the questions using grade 12 maths knowledge and do it with calculus and stuff to make your calculations seems more complex, will you have a higher mark than doing it the normal way? No! The marks is the same or even less due to higher chance of error in doing complex calculations.

In short, higher complexity on your labels won't help your task if it is a simple task. Hope this would help you and have a nice day!


Why does unsupervised pre-training help in deep learning?

Unsupervised pre-training was done only very shortly, as far as I know, at the time when deep learning started to actually work. It extracts certain regularities in the data, which a later supervised learning can latch onto, so it is not surprising that it might work. On the other hand, unsupervised learning doesn't give particularly impressive results in very deep nets, so it is also not surprising that with current very deep nets, it isn't used anymore.

I was wondering whether the initial success with unsupervised pre-training had something to do with the fact that the ideal initialization of neural nets was only worked out later. In that case, unsupervised pre-training would only be a very complicated way of getting the weights to the correct size.

Unsupervised deep learning is something like the holy grail of AI right now and  hasn't been found yet. Unsupervised deep learning would allow you to use massive amounts of unlabeled data and let the net form its own categories. Later you can just use a little bit of labeled data to give these categories their proper labels. Or just train it immediately on some task, in the conviction that it has a huge amount of knowledge about the world already. This is also what the problem of common sense comes down to: a huge and detailed model of the world, that could only be acquired by unsupervised learning.


What do we mean by 'principal angle between subspaces'?

Let's consider the case where you have two photos, one base photo and one other photo which is a scaled version of the base photo. Consider then that you could create a 'mapping' from the base photo to the scaled photo as defined by a set of vector changes for each pixel in the base photo.
That is to say, if you had a pixel at point (0,0) in the base photo it would be in position (0,0) on the scaled photo. If you had a pixel at point (0,1) on the base photo, it would be at point (0,s) on the scaled photo, where s is the scale factor between them. The same would be true for (1,0) mapping to (s,0) and (1,1) mapping to (s,s).
We can understand each of these pixels as a subspace on the vector space of "all possible pixels" (possibly with an included vector for the 'value' of each pixel as a third+ dimension, based on how it's represented). Note though, that they are different subspaces. There are vectors that exist in the scaled picture for scale factors greater than 1 which are not in the original picture. For scale values less than one, there are vectors which are in the original picture which are not in the scaled picture. For scale vector equal to one, it's basically the identity and they're the same picture.
What's more, we can do the same for rotation. If you have base photo and rotated photo, then there is a mapping from one pixel on the base photo to one pixel on the rotated photo, I'll leave the math up to you. The same is further true for translating, skewing, and even changing the colors of the image. Each one denotes a type of mapping that you could make from one subspace to another.
So what does this mean? Well, one thing that you could do is take two images, one "base" photo and two "scaled" photos and determine "how different are these two photos from the base photo"? That is, if the first scaled photo has a scaling factor of 10 and the second has a scaling factor of 100, then you could say that the larger scaled-photo is "further away" from the base photo. That is, we can compare the two mappings together to determine which mapping is closer to the original.
What's more, these mapping functions are often composable and commutative. That is if you do a "scale-then-rotate", that is very similar if not exactly identical to doing "rotate-then-scale". Let's take a look at why this is. Let's consider that we take a unit vector from either one. We then apply "scale" which scales both the x and y components, and then rotate this unit vector about some axis (let's assume the origin, though maybe there's a translation that happens as well). You will then yield a new vector which represents all of these applied changes. What's more, we can do this to both of the "scale-then-rotate" function and the "rotate-then-scale" function.
Now we have two vectors, and we want to determine if they are the same. Well, one way to do that is to calculate the angle between two vectors. If the angle is 0, then we can say that the vectors are the same (with the assumption that they are also of the same magnitude). In this case, both would yield the same vector, the angle between them is 0, so we can say they are the same.
So then how do we get back to the question of "how different are these two objects"? Well, one way that we could do it is to attempt to answer the question "What set of composed functions maps from one object onto the other one?". If we can create a new "vector" which is the representation of the full set of changes from one picture to the other, we can then attempt to determine how different these two objects are, as represented in the dimensional spaces of their image.
But the question that you may then ask is "why is this useful"? Well, imagine that you are a neural network. You've been given a picture and tasked with answering the question "Is this a cat?". How would you go about answering this? Well, one thing as a neural net that you've learned is "cats have ears". You have a canonical representation of a "cat ear" encoded in the weights of your neural network, so you go about looking at different subsets of the picture to say "is this a cat ear"? You can do this, for example, by taking a cropped section of the picture, determining the angle and dimensions apart that cropped section is from your canonical "cat ear" is, and if the angle between your canonical "cat ear" and your sample is below some amount, you can say with confidence that what you have found is a "cat ear". A couple cat ears, eyes, a nose, some fur, and a tail (though maybe the tail is occluded), and you can be confident that you have found a "cat".
(Note: This is an oversimplification and this is usually done in deep networks by breaking pictures out into a continually composing series of features, for instance first "lines" vs "curves" vs "textures", which then yield "circles" vs "rectangles" vs "cross-hashes", which then yield "eyes" vs "fur" vs "skin", etc. Each different layer could be implemented with its own comparison in this same method though!)


How to manage large amounts of image data for training?

I suggest you fine-tune an existing model. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training.

  This way you will use few data to tune the model which leads to less
  memory use and less training time. You will also take advantage from a
  ready-to-use architecture and get accurate results.

Depending on your case-study you can choose from this list of computer vision pre-trained models.


What is the actual quality of machine translations?

Who claimed that machine translation is as good as a human translator? For me, as a professional translator who makes his living on translation for 35 years now, MT means that my daily production of human quality translation has grown by factor 3 to 5, depending on complexity of the source text.

I cannot agree that the quality of MT goes down with the length of the foreign language input. That used to be true for the old systems with semantic and grammatical analyses. I don't think that I know all of the old systems (I know Systran, a trashy tool from Siemens that was sold from one company to the next like a Danaer's gift, XL8, Personal Translator and Translate), but even a professional system in which I invested 28.000 DM (!!!!) failed miserably.

For example, the sentence:

  On this hot summer day I had to work and it was a pain in the ass.

can be translated using several MT tools to German.

Personal Translator 20:

  Auf diesem heißen Sommertag musste ich arbeiten, und es war ein Schmerz im Esel.

Prompt:

  An diesem heißen Sommertag musste ich arbeiten, und es war ein Schmerz im Esel.

DeepL:

  An diesem heißen Sommertag musste ich arbeiten und es war eine Qual.

Google:

  An diesem heißen Sommertag musste ich arbeiten und es war ein Schmerz im Arsch.

Today, Google usually presents me with readable, nearly correct translations and DeepL is even better. Just this morning I translated 3500 words in 3 hours and the result is flawless although the source text was full of mistakes (written by Chinese).


What AI applications exist to solve sustainability issues?

The paper The role of artificial intelligence in achieving the Sustainable Development Goals (2020, published in Nature) should contain the information you're looking for.

In the introduction, the authors write

  Here we present and discuss implications of how AI can either enable or inhibit the delivery of all 17 goals and 169 targets recognized in the 2030 Agenda for Sustainable Development. Relationships were characterized by the methods reported at the end of this study, which can be summarized as a consensus-based expert elicitation process, informed by previous studies aimed at mapping SDGs interlinkages. A summary of the results is given in Fig. 1 and the Supplementary Data 1 provides a complete list of all the SDGs and targets, together with the detailed results from this work.

For example, as stated in the supplementary data, goal 1 is

  End poverty in all its forms everywhere

and the first target (1.1.) of goal 1 is

  By 2030, eradicate extreme poverty for all people everywhere, currently measured as people living on less than $1.25 a day

Then the authors suggest that, according to their studies, AI may act as an inhibitor or enabler (i.e. it may be used to fight poverty) for this target.

  We identified in the literature studies suggesting that AI may be an inhibitor for this target, due to the potential increase in inequalities which would hinder the achievement of this goal (1). Other references however identify AI as an enabler for this goal, in the context of using satellite data analysis to track areas of poverty and to foster international collaboration (2).

Therefore, techniques for satellite data analysis are one of the AI techniques that may be used to tackle suistainability issues.


Does a bias also have a chance to be dropped out in Dropout layer?

Only the non-bias ones,

It is discouraged to include the bias weights under norm penalty regularization for example, so why should it be included in the drop-out regularization scheme?

Drop out can be implemented by multiplying units with zero and the bias term is rather special. The bias term determines the distance from the origin the linear decision boundary of the node implements. It is included in every sum, but it does not receive any inputs.

So to take your example, say you have 80 nodes per layer in an MLP, where one is a bias node, the output of these layers will consist of 79 nodes every time. This output is then put under the drop-out effect.

From Goodfellows et al. Deep Learning book under the regularization chapter, present online: "http://www.deeplearningbook.org/contents/regularization.html", they write that they implement the drop-out by canceling some outputs to zero:

"
In most modern neural networks, ... , we can effectively remove a unit from a network by multiplying its output value by zero. ... . Here, we present the dropout algorithm in terms of multiplication by zero for simplicity, but it can be trivially modiﬁed to work with other operations that remove a unit from the network.
"

Drop-out by cancelling outputs, the output from the activation function, has nothing to do with the bias vector. It is not receiving any inputs.
Therefore I think the implementation typically only deals with output-to-input connections, and the bias vector does not receive inputs so it can safely be let alone from the drop-out process.


What do I need to study for machine learning?

As a master's degree student in Artificial Intelligence, I strongly advise you to study some basics in Machine Learning.

To do that, you can get a good book (Machine Learning, Tom Mitchell, McGraw Hill, 1997) for the theory and practice by yourself trying some Kaggle competitions.

I suggested the book of Mitchell because he is an expert in the field, and lot of Machine Learning courses uses his book. You can also follow his videolectures online

On Kaggle, you can find many useful tutorials (named as Notebooks) to get started working with the available datasets.
Some tutorials about Titanic Challenge here


Are CNN, LSTM, GRU and transformer AGI or computational intelligence tools?

CNNs, LSTMs, GRUs and transformers are or use artificial neural networks. The expression computational intelligence (CI) is often used interchangeably with artificial intelligence (AI). CI can also refer to a subfield or superfield of AI where biology is often an inspiration. See What is Computational Intelligence and what could it become? by Włodzisław Duch.

RNNs are Turing complete and CNNs have been shown to be universal function approximators (they can approximate any continuous function to an arbitrary accuracy given a sufficiently deep architecture), but that doesn't mean we will be able to create AGI with them, unless you believe that AGI is just a bunch of algorithms, but, IMHO, that alone doesn't produce AGI. See also the computational theory of mind.

To conclude, CNNs, LSTMs, GRUs and transformers are deep learning tools (so they could also be considered CI tools, given some definitions of CI), which might be useful for the development of AGI.


What is meant by the expected BLEU cost when training with BLEU and SIMILE?

It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function over that distribution. Section 3 has the loss function being minimized:

$ \mathcal{L}_{Risk} = \sum\limits_{u \in \mathcal{U}(x)} cost(t, u) \frac{p(u|x)}{\sum_{u' \in \mathcal{U}(x)} p(u'|x)} $.

One of the cost functions used is $1 - \texttt{BLEU}(t, h)$, where $t$ is the target and $h$ is the generated hypothesis. I'm not sure where $p(u|x)$ is coming from, but $1 - \mathcal{L}_{Risk}$ for the BLEU cost function is probably what they're refering to when they mention Expected BLEU.


Is it possible to train one part of the network with a particular learning rate and the other part with a different one?

Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call


Is there a family tree for reinforcement learning algorithms?

I highly recommend looking at Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto.

In it they write:

  Reinforcement learning, like many topics whose names end with “ing,” such as machine
  learning and mountaineering, is simultaneously a problem, a class of solution methods
  that work well on the problem, and the field that studies this problem and its solution
  methods. It is convenient to use a single name for all three things, but at the same time
  essential to keep the three conceptually separate. In particular, the distinction between
  problems and solution methods is very important in reinforcement learning; failing to
  make this distinction is the source of many confusions.

On the solution side of things, rather than a family tree of reinforcement learning, there is a spectrum of different approaches to the problem.

Many of the points within the spectrum above are TD($\lambda$) methods. When the state space is large, one might use a neural network to help generalize across similar states, such as DQN for instance.

It should be noted that there are many applications of reinforcement learning algorithms that do not involve training an agent to perform a task. Instead one might want to evaluate an existing agent or use a general value function to predict an event using an agent's policy.

Nevertheless, many popular problems consist of training an agent to perform a task. These algorithms can be categorized into a "family tree" based either on what values they are updating or on what problem they are tackling. (Source : https://github.com/NervanaSystems/coach)

On the left are Value Based methods which update the value function of a policy and find policies that optimize said value function.
To the right of those are policy gradient methods which optimize for the policies directly without explicitly finding a value function.
To the far right is Imitation Learning whereby one wants to copy a policy given demonstrations of an existing but unacceptable policy.
And in the middle is Direct Future Prediction which is its own weird beast that I found out while finding that picture. Here is the arxiv paper that describes DFP.


Which type of feature extractor do you suggest to classify sensor data?

There could be multiple possible ways to extract the features. One would be to use RNNs for a temporal relationship as the input data is time-series.