Applications of Information Theory in Machine Learning

Apart from the entropy and the cross-entropy, which are widely used in deep learning and you seem to be aware of, there is also the Kullback-Leibler divergence (also known as relative entropy), which is widely used in the context of variational Bayesian neural networks and variational auto-encoders, given that it's often part of the loss function that is minimized, i.e. the Evidence Lower BOund, which is a proxy objective function for the KL divergence between the prior and posterior distributions (which actually corresponds to the minimum description length needed to encode the data: huh?). See this answer for more details. There is also the mutual information, which has also been used as a measure of uncertainty in the context of Bayesian neural networks.


What does "In each generation, 25% of offspring resulted from mutation without crossover" mean in the context of NEAT?

In genetic algorithms, mutation without crossover simply means that part of the population is randomly changed. In this case this is applied to 25% of the population.

The remaining 75% either remain unchanged (generally the best performing specimen), or will be combined with other specimen (using crossover). It's a bit more complex here, as the genome is connecting weights in networks, and they use 'species', where certain individuals are treated as separate groups from the rest of the population.


Is there any AI system for finding the best way to schedule university classes?

Welcome to AI.SE @Israr Ali.

The problem of scheduling a timetable is an example of a constraint satisfaction problem, a topic long studied in AI.

There are many possible techniques to apply to this kind of problem. They can be organized into three broad categories:

Global search algorithms, like backtracking search can be used to try and find an assignment of times to classes that results in no conflicts at all. With proper heuristics, these can be quite fast in practice, even though their worst-case runtimes are poor. Russell & Norvig's AI: A Modern Approach has a very good overview of these approaches in Chapter 6.
Local search algorithms, like hill climbing search start by assigning every class a randomly selected time, and then making small adjustments that monotonically decrease the number of conflicts (e.g. swapping the times of two classes). These are often faster than backtracking approaches and are especially so if you are willing to accept a "pretty good" schedule that might still have fewer conflicts than an optimal one. Chapter 4 of AI: A Modern Approach provides a good introduction.
Planning approaches, particularly non-classical algorithms like GraphPlan can also be used for these domains. They make use of the structure of planning or scheduling problems in particular to re-frame the search problems addressed by Backtracking techniques. By using this domain-specific representation, they are able to achieve very high-quality solutions very quickly. Chapter 11 of AI: A Modern Approach covers Graph Plan in some detail, and could be a good introduction to these more specialized techniques.


Are there any projects that can measure the concept similarity between texts?

Doc2Vec

Doc2Vec comes to mind, here's the original publication. The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet.

There are a number of implementations of Doc2Vec. If you're using Python, one to look at is gensim.

Word2Vec

Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the original publication, and another publication that does a nice job explaining it further.

Tensorflow has a tutorial for setting up a Word2Vec model. Gensim also has a Word2Vec implementation.


Why is an embedding of dimension 400 enough to represent 70000 words?

The specific term you are looking for is "word embedding" and not just "embedding".
How to numerically represent textual data?
Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or convert) the text into a numerical representation. There are different ways to do it, such as

one-hot encoding (in that case, if you have 70000 words, you would have sparse vectors with 70000 entries where only one of those entries is equal to $1$ and all other entries are $0$: see this article  for more info)

map each word to a number (in this case, you would have 70000 numbers, one for each word)

word embeddings

Each of these representations has different benefits and drawbacks. For instance, if you map each word to a number, then you just need to keep track of $70000$ numbers. In the case of one-hot encoding or word embeddings, you will need more memory. However, nowadays, word embeddings are widely used in natural language processing/understanding/generation tasks (and given that your question is about word embeddings), so let me briefly describe them.
Word embeddings
There are different word embedding techniques (such as word2vec). However, they are all based on the same ideas

Words that are similar (or related) in meaning should be mapped to vectors (i.e. the "word embeddings") that are also similar in some sense (for instance, their cosine similarity should be high). For instance, the words "man" and "boy" should be mapped to vectors that are similar.

These word embeddings are learned (rather than hard-coded or manually specified) given the data

The size of the word embeddings is a hyper-parameter (this should answer your question!)

Hyper-parameters
To answer your question(s) more directly, the choice of the dimension of the embeddings or the number of "hidden features" (which are both hyper-parameters) was probably more or less arbitrary or based on the instructor's experience. In general, it is difficult to determine the optimal choice of any hyper-parameter. Sometimes you can just use numbers that other people have used in the past and have noticed that work "well enough". If you really want to find more appropriate values of the hyper-parameters, you could use some hyper-parameter optimization technique, such as Bayesian optimization or a simple grid search.
Further reading
You can find many resources online that explain the concept of "word embeddings" more in detail. For instance

What Are Word Embeddings for Text? (by Jason Brownlee)
Word embeddings (A TensorFlow article/tutorial on word embeddings).


How to define the "Pre-Processing" in machine learning?

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy).

The book Data Preprocessing in Data Mining (2014), by Salvador García et al., which provides a good overview of the data preprocessing techniques and their connection with data mining and machine learning algorithms and models, defines data preprocessing as follows.

  Data preprocessing includes data preparation, compounded by integration,
  cleaning, normalization and transformation of data; and data reduction tasks; such as feature selection, instance selection, discretization, etc. The result expected after a reliable chaining of data preprocessing tasks is a final dataset, which can be considered correct and useful for further data mining algorithms.

From page 10 onwards, there is a description and categorization of the main data preprocessing techniques. I will just list them, so refer to the book for a definition and explanation of each of these techniques.

Data Preparation

Data Cleaning
Data Transformation
Data Integration
Data Normalization
Missing Data Imputation
Noise Identification

Data Reduction

Feature Selection
Instance Selection
Discretization
Feature Extraction/Instance Generation

Here are two screenshots (from the cited book) that illustrate some of the data preparation

and data reduction techniques.


How to express a fully connected neural network succintly using linear algebra?

The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i.e. a neural network with 3 layers, 1 input layer, 1 hidden layer, and 1 output layer, where

the input layer is connected to the hidden layer (all scalar inputs are connected to every neuron in the hidden layer)
the hidden layer is connected to the output layer (all neurons in the hidden layer are connected to all neurons in the output layer

As an example, suppose that we have $N$ real-valued features, there are $L$ hidden units (or neurons) and $M$ output units, then the elements (feature vector and parameters) of equation \ref{1} would have the following shape

$x \in \mathbb{R}^{1 \times N}$
$W_\color{green}{1} \in \mathbb{R}^{N \times L}$
$W_\color{blue}{2} \in \mathbb{R}^{L \times M}$

$\sigma$ is an activation function that is applicable to all elements of the matrix separately (i.e. component-wise). So, $\hat{y} \in \mathbb{R}^{1 \times M}$.
The equation
$$
\hat{y} = \sum_{\color{red}{i} = 1}^{L} W_{\color{blue}{2}, \color{red}{i} \cdot} \sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )\label{2}\tag{2}
$$
is another way of writing equation \ref{1}.
Before going to the explanation, let's try to understand equation \ref{2} and its components.

$W_{\color{green}{1}, \cdot \color{red}{i}} \in \mathbb{R}^N$ is the $\color{red}{i}$th column of the matrix that connects the inputs to the hidden neurons, so it is a vector of $N$ elements (note that we sum over the number of hidden neurons, $L$).

Similarly, $ W_{\color{blue}{2}, \color{red}{i} \cdot} \in \mathbb{R}^M$ is also a vector, but, in this case, it is a row of the matrix $ W_{\color{blue}{2}}$ (rather than a column: why? because we use $\color{red}{i} \cdot$ instead of $\cdot \color{red}{i}$, which refers to the column).

So, $\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle$ is the dot (or scalar) product between the feature vector $x$ and the $\color{red}{i}$th column of the matrix that connects the inputs to the hidden neurons, so it's a number (or scalar). Note that both $x$ and $W_{\color{green}{1}, \cdot \color{red}{i}}$ have $N$ elements, so the dot product is well-defined in this case.

$ W_{\color{blue}{2}, \color{red}{i}} \sigma(\langle x, W_{\color{green}{1}, \color{red}{i}} \rangle))$ is the product between a vector of shape $M$ and a number $\sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )$. This is also well-defined. You can multiply a real-number with a vector, it's like multiplying the real-number with each element of the vector.

$\sum_{\color{red}{i} = 1}^{L} W_{\color{blue}{2}, \color{red}{i} \cdot} \sigma(\langle x, W_{\color{green}{1}, \cdot \color{red}{i}} \rangle )$ is thus the sum of $L$ vectors of size $M$, which makes $\hat{y}$ also have size $M$, as in equation \ref{1}.

Now, the question is: is equation \ref{2} really equivalent to equation \ref{1}? This is still not easy to see because $xW_\color{green}{1}$ is a vector of shape $L$, but, in equation \ref{2}, we do not have any vector of shape $L$, but we have vectors of shape $N$ and $M$ (and the vectors of shape $M$ are summed $L$ times). First, note that $\sigma(xW_\color{green}{1}) = h\in \mathbb{R}^L$, so $hW_\color{blue}{2}$ are $M$ dot products collected in a vector (i.e. $\hat{y} \in \mathbb{R}^M$), where the $j$th element of $\hat{y}$ was computed as a summation of $L$ elements (a dot product of two vectors is the element-wise multiplication of the elements of the vectors followed by the summation of these multiplications). Ha, still not clear!
The easiest way (for me) to see that they are equivalent is to think that $\sigma(xW_\color{green}{1})$ is a vector of $L$ elements $\sigma(xW_\color{green}{1}) = \ell = [l_1, l_2, \dots, l_L]$. Then you know that to multiply this vector with $\ell$ (from the left), you actually perform a dot product between $\ell$ and each column $W_\color{blue}{2}$. A dot product is essentially a sum, and that's why we sum in equation \ref{2}. So, essentially, in equation \ref{2}, we first multiple $l_1$ with the first row of $W_\color{blue}{2}$ (i.e. by all elements of the first row). Then we multiply $l_2$ by the second row of $W_\color{blue}{2}$. We do this for all $L$ rows, then we sum the rows (to conclude the dot product). So, you can think of equation 2 as first perform all multiplications, then summing, rather than dot product-by-dot product.
So, in my head, I have the following picture. To simplify the notation, let $A$ denote $W_\color{blue}{2}$, so $A_{ij}$ is the element at the $i$th row and $j$th column of matrix $W_\color{blue}{2}$. So, we have the following initial matrix
$$
A =
\begin{bmatrix}
A_{11} & A_{12} & A_{13} & \dots & A_{1M} \\
A_{21} & A_{22} & A_{23} & \dots & A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
=
\begin{bmatrix}
W_{\color{blue}{2}, \color{red}{1} \cdot}  \\
W_{\color{blue}{2}, \color{red}{2}  \cdot} \\
\vdots  \\
W_\color{blue}{2, \color{red}{L}  \cdot} \\
\end{bmatrix}
$$
Then, in the first iteration of equation \ref{2}, we do the following
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
A_{21} & A_{22} & A_{23} & \dots & A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
$$
In the second, we do the following
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
l_2 A_{21} & l_2 A_{22} & l_2 A_{23} & \dots & l_2 A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
A_{L1} & A_{L2} & A_{L3} & \dots & A_{LM} \\
\end{bmatrix}
$$
Until we have
$$
\begin{bmatrix}
l_1 A_{11} & l_1 A_{12} & l_1 A_{13} & \dots & l_1 A_{1M} \\
l_2 A_{21} & l_2 A_{22} & l_2 A_{23} & \dots & l_2 A_{2M} \\
\vdots & \vdots & \vdots & \dots & \vdots \\
l_L A_{L1} & l_L A_{L2} & l_L A_{L3} & \dots & l_L A_{LM} \\
\end{bmatrix}
$$
Then we do a reduce sum across the rows to end the dot product (i.e. for each column we sum the elements in the rows). This is exactly equivalent to first performing the dot product between $\ell$ and the first column of $W_\color{blue}{2}$, then the second column, and so on.


Do images submitted to a segmentation network after training need to be the same size as the training images?

Yes, you will need data that has the same dimensionality as your training set. Otherwise the model will either discard excess data, or will supplement missing data with a random number. This will likely lead to very poor results.

I would suggest to rescale your images to have the same pixel density as those you used for training (250x250) rather than scaling your model.

Methods for rescaling images and maintaining details are very well developed. Many exist and you can use existing tools such as Photoshop to change the pixel density of an image.


Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I think instead of:

if np.argmax(act_values[0]) in actions_allowed:
        return np.argmax(act_values[0])

you can use something like:

if np.argmax(actions_allowed_values[0]):
        return np.argmax(actions_allowed_values[0])

So you didn't random your act if the best action is not allowed, but you choose the best action that allowed. You can do that by checking for each action value, whether it is a valid action or not, then take the best value. Or you can change invalid action value to a very small number.


How is a deep neural network different from other neural networks?

The difference is mostly in the number of layers.

For a long time, it was believed that "1-2 hidden layers are enough for most tasks" and it was impractical to use more than that, because training neural networks can be very computationally demanding.

Nowadays, computers are capable of much more, so people have started to use networks with more layers and found that they work very well for some tasks.

The word "deep" is there simply to distinguish these networks from the traditional, "more shallow" ones.


Representation of real numbers in Genetic Algorithm

The authors want to use a binary genome which contains specific interpretation/mapping to a fixed precision real number. They could use one or more existing schemes, but these typically won't represent the exact precision and range of real numbers they want to use in the example problem.

They use their own encoding scheme and explain it in the text for completeness. Any similarities between their scheme and other binary representations of real numbers are to be expected, and if you understand them, it will help understand what they are doing here too.

GAs don't have to use this sort of scheme for real numbers. Often they use standard floating-point representation as separate elements. But the authors want a binary code here to help teach some basic concepts, which is what they mean by "on the other hand, binary code is good for pedagogy"


Formal definition of the Object Detection problem

This is just an idea
Given a set of pixels, the task is to decide:

Which pixel is the center of an object?
What is the size of the bounding boxes with the center is the pixel in part 1?

Formula, consider this is a 2D image, call $(x,y)$ is the horizontal and vertical coordinate and $(w_i,h_i)$ is the size of bouding box of object $i$:
$\text{For }m \in[x,x+w_i] \text{ and } n\in[y,y+h_i]$
$c_i(m,n) = \begin{cases}
    1, \text{if pixel at position (m,n) is belongs to object i,}\\
    0, \text{else}
\end{cases}$


What is the difference between a fitness function and a reward function?

In many cases, a fitness function (FF) is indeed similar to a reward function (RF), but, in other cases, it's more similar to a cost function (CF) as used in supervised learning (SL), and I explain below why. The FF, RF, and CF are used to evaluate the individuals, actions, and predictions, respectively, hence they can all be thought of as evaluation functions (although this latter term has a more specific meaning in the context of search algorithms, like A*). Their primary difference is their purpose and how they are used (i.e. for which problems they are used or what they evaluate), and not much what they return, which is typically a number (in all cases).
Reward function
As stated in the question, an RF can be viewed as evaluating different situations, depending on its definition or what we are interested in.

$r(s)$ evaluates the goodness of being state $s$
$r(s, a)$ evaluates the goodness of taking $a$ in the state $s$
$r(s, s')$ evaluates the goodness of being in state $s$, at the current state time step, then, independently of the action $a$ taken in $s$, ending up in state $s'$
$r(s')$ similar to $r(s)$
$r(s, a, s')$ the quality of being in state $s$, taking action $a$ and ending up in state $s'$.

Although I will not dwell on the details in this answer (and you can find more details here), depending on the problem or the RL algorithms that you want to use to solve the problem, one definition may be more interesting or useful than another.
Fitness function
In evolutionary algorithms (EA), such as genetic algorithms (GA) or genetic programming (GP), the solutions are generally known as individuals, which can be represented in different ways depending on the specific approach.
However, in all cases, we often distinguish between genotypes (aka genomes or chromosomes), which are representations of the actual solutions to a problem, which are more amenable to changes (i.e. to the mutation and crossover operations), and phenotypes, which are the actual solutions to the problem. The reason why we use genotypes is that, often, it's easier to find solutions in a different but related space rather than in the original space of solutions, although some EAs can also just work with phenotypes.
To be more concrete, in traditional genetic algorithms, the genotypes are binary vectors $\{0, 1 \}^N$, which restricts the space of possible solutions, which, in certain cases, it's a good thing, because the original space may be too huge (so it may be harder to converge to a good solution). These binary vectors can then be converted to the actual solutions using some encoding-decoding mechanism that we need to specify depending on the problem. More precisely, maybe each element of a binary vector corresponds to some binary choice in some game where you need to take a sequence of choices, and each time you can either say "yes" or "no".
So, in EAs, an FF evaluates the quality (or fitness) of an individual and, more precisely, either a genotype or a phenotype (depending on the representation of the solutions).
Often, the fitness function is really a cost function (CF), like in supervised learning. For example, if you want to solve a symbolic regression problem using genetic programming (i.e. find an analytical or closed-form expression of a function), then the fitness function is really just a cost function, i.e. an individual would be like a prediction and the label would be generated by the actual true function (provided that we know it). More concretely, let's say that we want to find the analytical expression
$$f(x) = x^2.$$
In this case, we can generate a labeled dataset of input-output pairs by just evaluating $f$ at different inputs, so we may get the following labeled dataset
$$
D = \{(1, 1), (2, 4), (3, 9) \}
$$
We can then evaluate an individual, which in the case of GP is a function (or program) $g(x)$, using e.g. the mean squared error (MSE). For example, if we have the individual $g(x) = x^3$, we can compute the mean squared error as follows
$$
\text{MSE}(D, g(x)) = \frac{1}{3} \left( (1 - 1)^2 + (4 - 8)^2 + (9  - 27)^2 \right)
$$
So, in this case, the FF is the MSE. Of course, we could have bigger datasets than $D$. In any case, the point is that you can use GP to solve SL problems.
You could also use EAs to solve other types of problems, i.e. not just SL problems, for instance, RL or unsupervised learning problems. In the case of RL problems, the FF is more similar to an RF in RL, given that you will probably need to define the FF as a function that evaluates the qualities of actions, states, or combinations of these.
So, to conclude, the definition of the FF in EAs depends on the problem that you want to solve, and it can be more similar to an RF or CF, depending on the problem you want to solve.
Further reading
If you are interested in how to design reward functions or fitness functions for your problem, you can take a look at this (and this) and this questions, respectively. This answer provides a brief explanation of the difference between RL and EAs.


What are some resources to improve practical expertise in machine learning?

Since you are a software engineer, you already have some experience with one or more programming languages, so this is already a good thing. Moreover, you say that you have a strong mathematical background, so you will probably learn certain machine learning concepts more easily.
When it comes to machine learning, there are open source projects that you can contribute to via Github, and this really boosts your skills and experience.
I also am contributing to an open-source conversation agent via Github.
If you are able to understand and contribute to such projects, then implementing ML/DL algorithms is possible in your target language. I myself have been using Python, but that's just a personal choice, and it's been utilized by many AI engineers and data scientists, globally.

Do you think that attempting kaggle challenges from scratch would be
stimulating? Or too confusing? Do you think delivering a personal
project (i.e. not a byproduct of my current job) in which I employ one
of such methods to solve a potentially real problem would be enough to
be relevant in my cv?

Contributing to open-source projects will link you to other engineers in the same field, and this will eventually open you many paths.
Also, try the book: Artificial Intelligence: A Modern Approach Stuart Russel, Peter Norvig, Prentice Hall, 3rd edition, 2009


Google has an API you can use. https://cloud.google.com/translate/. Their API can translate audio to text. They also have an API for converting speech to text. The language detection feature should let you detect the language in the resulting text. They have client libraries for the most popular programming languages.


How's the action represented in MuZero for Atari?

The bias plane is a layer everywhere equal to the constant $a/18$ where $a$ is the action. So each of the 32 frames has three frames for RGB and a fourth frame which is the bias plane making for 128 input layers. This is explained in the Network Architecture section where it mentions that the actions are "broadcast" to the planes.

For this I do not have conclusive evidence, but I think tiling a vector means arranging parallel copies of it into e.g. a grid shape. In other words, the input is 6x6x18, and action 1 is represented as all ones in the first plane with all zeros in the remaining planes. One problem with the way you've depicted is that the input is subject to convolutions, but there is no inherent reason why the fifth position and eleventh actions (which are next to each other vertically) should be included in the same 3x3 filter application, but the fifth and seventh (for example) actions should not.


Best Camera and protocol for embedded real time CNN project

The Raspberry Pi ZeroW with V2 camera doesn't quite have the specs you need, but it is close: https://picamera.readthedocs.io/en/latest/fov.html#sensor-modes

Here's a waterproof enclosure for it: https://www.youtube.com/watch?v=5CWksss_5lQ


Biological organisms (such as animals or plants) are the main examples of intelligent systems that we are aware of (excluding artificially intelligent systems, so as not to discuss whether current AI systems are really intelligent or not). Consequently, biological life is often an inspiration for AI researchers to develop AI systems.
There are numerous examples of AI systems that have been introduced  (at least, partially) based on or just inspired by the biology. Here are a few examples.

Reinforcement learning is based on a similar way that animals (such as dogs or pigeons) can learn. For more details, see Sutton & Barto's book (especially chapters 14 and 15).

Artificial neural networks are very approximative models of human neural networks.

Genetic algorithms are roughly based on Darwin's theory of evolution.

Ant colony optimization algorithms (and, in general, swarm intelligence) are based on the way real ants (and, respectively, biological swarms) behave. (There is even a rap song dedicated to ants).

There are probably other examples that don't come to my mind right now. See also this and this questions.
There are cases where AI discoveries have also helped the development of biology or related fields (such as neuroscience and psychology). For instance, Sutton & Barto (on page 4) write

Of all the forms of machine learning, reinforcement learning is the closest to the kind of learning that humans and other animals do, and many of the core algorithms of reinforcement learning were originally inspired by biological learning systems. Reinforcement learning has also given back, both through a psychological model of animal learning that better matches some of the empirical data, and through an influential model of parts of the brain's reward system.


Which classifier should I use for a dataset with one feature?

It is not really a metter of what model, but if it is possible at all to predict what you're trying to predict. Let's take a similar dataset from kaggle: California Housing Prices

This dataset contains house prices and other information among which the number of bedrooms per house. As suggested by Oliver in the comments we can compute the Person coefficient to estimate the correlation between the two variables.

import pandas as pd
from scipy.stats import pearsonr

df = pd.read_csv('housing.csv')
df = df.apply(lambda row: row[df['total_bedrooms'] <= 20]) # select subset of dataframe for sake of clarity
df.dropna(inplace=True)

x = df['median_house_value'] # our single feature
y = df['total_bedrooms'] # target labels

print('Correlation: \n', pearsonr(x,y))

Out:

>>Correlation:
>>(-0.14015312664251944, 0.12362969210761204)

The correlation is pretty low, which means that the price and number of bedrooms are basically not related. We can also plot the points to check that indeed there is no correlation at all.

df.plot(x='total_bedrooms',y='median_house_value',kind='scatter')

Out:

Training a model to predict the number of bedrooms uniquely from the price would mean to find a function that can interpolate all those points, which is an impossible task since we have several different prices for houses with the same amount of bedrooms.

The only way to tackle a problem like this would be to expand the dimensionality of the data, for example by using a Support Vector Machine with a non linear kernel. But even with non linear kernels you can't do miracles, so if you're dataset looks like this one, the only solution would be to expand your dataset to include extra features.


Can an AI be trained to generate the outline of a story?

As far as I am aware, this has not been done yet.

I see several problems with this. A neural network is basically a classifier, which matches an input to an output. Both input and output are usually numerical values, though they could be matched to concepts or words.

To train a NN you provide an appropriately encoded input, and the corresponding output. The NN learns the associations between the two, and can then classify unseen input accordingly. This has recently been used to transform images in a particular style etc.

What would the input and output be for generating screenplays? You could use previous scripts as inputs, but what would the output be? It could be narrative 'moves' of some sort, perhaps. So you could train an NN to recognise narrative elements from screenplays.

However, you are still not creating anything, but just recognising stuff. You would need some other input. I guess you could train an NN on "The Simpsons", get a narrative structure, and then present it with an Episode of "Friends" and see what happens. It won't be a new episode of a screenplay, though.

The other way round might work: you feed it narrative moves (a kind of story skeleton), and get a script out. But it would need a lot of (human) post-editing to be at all useful.

I think an NN is the wrong tool to use here. There has been work done with generating stories and screenplays, even way back in the early days of AI. But that was all based on symbolic AI, not on the kind of ML which seems to currently be en vogue. Have a look at James Ryan's website; he has recently written an overview over historic approaches to story (and screenplay) generation.


Deep Learning models train really slow Jetson Nano

First of all, you mentioned that you installed the correct version of Tensorflow on Jetson. You can list the available Tensorflow devices with:

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

And make sure that you see the GPU available.

More importantly, Jetson Nano has a 128-core NVIDIA Maxwell™ architecture-based GPU. This might not be a powerful GPU for training a neural network with. As you can see here, in NVIDIA's official benchmark, they only tested trained models (i.e. classifications, object detections and segmentations). You can consider training your Keras model using another hardware and use it for labeling, classification, or etc. in your Jetson.


How to represent players in a multi agent environment so each model can distinguish its own player

When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square.
I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.


ReLu, Sum and Convolution Layers to Count Pixels of Certain Color

I can judge from the excerpt only, but I don't see any mention of a convolutional layer there. This doesn't mean that convolution isn't suitable for this task (in fact, it's the best method for image classification so far), it's just seems the instructor proposes this particular network architecture, without convolutions.

Since this architecture is more illustrative and by no means a standard, I can't be sure I fully understand it. But it seems like the first layer outputs a vector of size 4, which contains the counts of four selected colors. The ReLu operation here is a fancy way of filtering extra colors, based on channel value (note that they are close to either 0 or 255 in all four colors). If you select different four colors, this filtering will have to be done differently too, so it's basically hardwired. The second layer is more straightforward.

In general, you shouldn't consider this a "true" deep network, but merely a puzzle to get familiar with ReLu operation.


Why are all weights of a neural net updated and not just the weights of the first layer

However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer.

I don't see how the second statement follows from the first.

Each weight $w_i$ (not just the ones in the first layer) affects the loss $\mathcal{L}$ according to the partial derivative of $\mathcal{L}$ with respect to $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$. Intuitively, the partial derivative with respect to parameter tells you how the function is changing with respect to that parameter.

  My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place.

I am not sure I understand your reasoning, but, typically, you update the parameters only after having computed all the partial derivatives. In other words, first, you compute all partial derivatives, i.e. the gradient with back-propagation (a fancy name to denote the application of the chain rule), then you update the parameters.

Why do you do this? In this case, the loss function is multi-variable function, so it depends on multiple variables. The gradient $\nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_N} \right]$ represents the direction (note that the gradient is a vector and vectors have direction) towards which your function is increasing or decreasing (depending on the sign of the gradient).


Autoencoder produces repeated artifacts after convergence

Perhaps you are getting checkerboard artifacts Explained here, solutions involve changing the kernel and stride size to prevent them from being not divisible. Besides that, a solution could be to apply Gaussian smoothing to minimize the artifacts.
For example, using Gaussian smoothing in OpenCV with your image results in
import cv2
img = cv2.imread('s.png') # I took a screen shot to see how it would look
blur = cv2.blur(img,(8,8))
plt.imshow(blur)

The artifact is gone using a kernel of size (8,8). I hope this can help


When did Artificial Intelligence research first start?

The earliest research into thinking machines was inspired by a confluence of ideas that became prevalent in the late 30s, 40s and early 50s (e.g. formal logic, automata, robots). Although the Turing test was proposed in the 1950s by Alan Turing, the work culminated back in the 1940s in the invention of the programmable digital computers, an abstract essence of mathematical reasoning. These ideas were inspired by a handful of scientists from a variety of fields who began seriously considering the possibility of building an electronic brain. The field of artificial intelligence research was officially founded as an academic discipline in 1956 during the Dartmouth workshop.
However, the concept of artificial beings is not new and it's as old as the Greek myths of Hephaestus and Pygmalion, which incorporated the idea of intelligent robots (such as Talos) and artificial beings (such as Galatea and Pandora).
See the following articles at Wikipedia for further details:

Artificial intelligence (AI)
History of artificial intelligence
Timeline of artificial intelligence


Will LMS always be convex function? If yes, then why do we change it for neural networks?

Square loss is fine for regression, since minimizing it is the same as maximizing the likelihood of the model parameters (under assumption that the error is Gaussian). However, if the model directly produces probabilities, then it is natural to use these probabilities directly within the loss. Hence, in all classification models we prefer to minimize negative log-likelihood of the correct class.

Note that choosing a natural loss leads to several practical advantages. In particular, applying a quadratic loss after a sigmoid activation leads to very poor gradients when the sigmoid is saturated in the wrong direction. The negative log-likelihood loss has no such problems.

This issue is not specific for neural networks. Logistic regression has used the negative log likelihood loss ever since 1958.


Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else uses it, but I can confirm that what they mean by "logit probability" is basically the same thing that is more commonly referred to simply as "logits": they are the raw, unbounded scores of which we generally push a vector through a softmax function to generate a discrete probability distribution that nicely adds up to $1$.

This definition fits the one you linked from wikipedia (although that link only covers the binary case, and AlphaGo Zero would have multinomial logits since it has more than two outputs for the policy head).

In the AlphaGo Zero paper, the described architecture has a "linear output layer" (i.e. no activation function for the outputs, or the identity function as activation function for the outputs, or however you like to describe it) for the policy head. This means that these outputs are essentially unbounded, they could be any real number. We know for sure that these outputs cannot directly be interpreted as probabilities, even if this isn't stated quite explicitly in the paper.

By calling them logits (or logit probabilities for reasons unknown to me), they are essentially implying that these outputs will still be post-processed by a softmax to convert them into a vector that can be interpreted as a discrete probability distribution over the actions, even if they do not explicitly describe a softmax layer as being a part of the network.

It is indeed possible that in Leela Zero they decided to make the softmax operation explicitly a part of the Neural Network architecture. Mathematically they end up doing the same thing... the AlphaGo Zero paper implies (by using the word "logit") that a softmax is used as a "post-processing" step, and in Leela Zero they explicitly make it a part of the Neural Network.

Here are a couple more sources for the reasoning that usage of the word "logit" basically implies usage of a softmax, though indeed they do not cover the term "logit probability":

https://stats.stackexchange.com/a/52836/144483
https://stackoverflow.com/a/34243720/6735980


Understanding how the loss was calculated for the SQuAD task in BERT paper

These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help:

What do they mean by "maximum scoring span is used as the prediction"?

As you know in SQuAD the input sequence is divided to 2 parts: Question and Document (from which we extract the answer if possible).

Sometimes the input length exceeds the max_seq_length parameter, in this case the document is truncated to as many parts as needed and we end up having more than one input for the same question/document. Mention that the question is replicated in all the resulting inputs (see line_350 for details).

So in such cases, in order to determine the predicted span among all the generated-inputs we use the maximum scoring i.e (max_start + max_end) / 2 = ( max(Softmax(S*Ti)) + max(Softmax(E*Ti)) ) / 2 of all the inputs related to the same question.

"The training objective is the loglikelihood of the correct start and end positions"

The loss is the average of the start_position loss start_loss and the end_postition loss end_loss. Each loss is computed in the same way: after applying the Softmax to the final output (i.e S*Ti or E*Ti) we use the real start/end postitions to compute the loss (see code below).

From run_squad.py in Bert_repo:

 def compute_loss(logits, positions):
        one_hot_positions = tf.one_hot(positions, depth=seq_length, dtype=tf.float32)
        log_probs = tf.nn.log_softmax(logits, axis=-1)
        loss = -tf.reduce_mean(tf.reduce_sum(one_hot_positions * log_probs, axis=-1))
        return loss

 start_positions = features["start_positions"]
 end_positions = features["end_positions"]

 start_loss = compute_loss(start_logits, start_positions)
 end_loss = compute_loss(end_logits, end_positions)

 total_loss = (start_loss + end_loss) / 2.0


How to estimate the capacity of a neural network?

VC dimension
A rigorous measure of the capacity of a neural network is the VC dimension, which is intuitively a number or bound that quantifies the difficulty of learning from data.
The sample complexity, which is the number of training instances that the model (or learner) must be exposed to in order to be reasonably certain of the accurateness of the predictions made given some data, is proportional to this number.
The paper VC Dimension of Neural Networks (1998) by Eduardo D. Sontag provides a good introduction to the VC dimension of neural networks (even though these concepts are quite abstract and you may need to read them several times to fully grasp them). The information in this answer is highly based on that paper.
Shattering and VC dimension
In section 2, Concepts and VC Dimension, he describes the basic concepts behind the VC dimension (not only for neural networks), such as the concept of shattering (i.e. what does it mean for a set of sets to shatter another set?), which is a well-known concept in computational learning theory and is used to define the VC dimension (see definition 2), so you definitely need to get familiar with this concept to understand the VC dimension and, therefore, the capacity of a neural network (calculated with the VC dimension).
VC dimension of functions and neural networks
He then provides an equivalent definition of the VC dimension but for functions (equation 6). Given that neural networks represent functions, then we can also define the VC dimension of a neural network. A specific combination of weights of neural networks represents a specific function, for which the VC dimension can be defined. To be more precise, a parametrized function (and a neural network) can be denoted as
$$
\beta : \mathbb{W} \times \mathbb{U} \rightarrow \mathbb{R}
$$
where $\mathbb{W} = \mathbb{R}^p$ and $p$ is the number of weights (or parameters) of the neural network, $\mathbb{U}$ is the input space and $\mathbb{R}$ the output space. So, in this case, $\beta$ can also represent a neural network, with a certain parameter space $\mathbb{W}$, an input space $\mathbb{U}$ and an output space $\mathbb{R}$.
The vector $\mathbf{w} = (w_1, \dots, w_p) \in \mathbb{W}$ represents a specific combination of weights of the neural network, so it represents a specific function. The set of all functions for each choice of this weight vector can be denoted as
$$
\mathcal{F}_{\beta} = \{ \beta(\mathbf{w}, \cdot) \mid \mathbf{w} \in \mathbb{W} \}
$$
The VC dimension (VCD) of $\beta$ can then be defined as
$$
\text{VCD}(\beta) := \text{VCD}(\mathcal{F}_{\beta})
$$
Therefore, the VC dimension is a measure of the capacity of a neural network with a certain architecture. Moreover, the VC dimension is equivalently defined for a certain set of functions associated with a neural network.
How to calculate the VC dimension?
To calculate the actual VC dimension of a neural network, it takes a little bit of more creativity. Therefore, I will just report the VC dimension of some neural networks. For more details, you should fully read the cited paper (more than once) and other papers and books too (especially, the ones described in this answer, which provide an introduction to CLT concepts).
VC dimension of a perceptron
The VC dimension of a perceptron is $m + 1$, where $m$ is the number of inputs. Given that a perceptron represents a linear and affine function, the VC dimension of the perceptron is also equal to the number of parameters. However, note that, even though the VC dimension of the perceptron is linear in the number of parameters and inputs, it doesn't mean the perceptron can learn any function. In fact, perceptrons can only represent linear functions. See section 3.1 of VC Dimension of Neural Networks for more details.
VC dimension of a single hidden layer neural network
Let $n$ be the number of hidden units, then the VC dimension of a single hidden layer neural network is less than or equal to $n+1$. See section 3.2 of VC Dimension of Neural Networks for more details.
VC dimension of multi-layer neural networks with binary activations
The VC dimension of multi-layer neural networks (MLPs) with binary activations and $p$ weights (or parameters) is $\mathcal{O}(p \log p)$. See theorem 4 (and related sections) of the paper VC Dimension of Neural Networks for more details.
VC dimension of MLPs with real-valued activations
The VC dimension of MLPs with real-valued activations is no longer bounded by $\mathcal{O}(p \log p)$ and can be exponential in the number of parameters. See section 5.3 of VC Dimension of Neural Networks.
VC dimension of MLPs with linear activations
The VC dimension of MLPs with linear activations is $\mathcal{O}(p^2)$.  See theorem 5 of the paper VC Dimension of Neural Networks.
Notes
The VC dimension is often expressed as a bound (e.g. with big-O notation), which may not be strict.
In any case, the VC dimension is useful because it provides some guarantees. For example, if you use the VC dimension to describe an upper bound on the number of samples required to learn a certain task, then you have a precise mathematical formula that guarantees that you will not need more samples than those expressed by the bound in order to achieve a small generalization error, but, in practice, you may need fewer samples than those expressed by the bound (because these bounds may not be strict or the VC dimension may also not be strict).
Further reading
There is a more recent paper (published in 2017 in MLR) that proves new and tighter upper and lower bounds on the VC dimension of deep neural networks with the ReLU activation function: Nearly-tight VC-dimension bounds for piecewise linear neural networks. So, you probably should read this paper first.
The paper On Characterizing the Capacity of Neural Networks using Algebraic Topology may also be useful and interesting. See also section 6, Algebraic Techniques, of the paper I have been citing: VC Dimension of Neural Networks.
The capacity of a neural network is clearly related to the number of functions it can represent, so it is strictly related to the universal approximation theorems for neural networks. See Where can I find the proof of the universal approximation theorem?.


How does 'Bright score' assess how strong the connection is between users and their jobs?

Take a look at this 2012 paper by three people at Bright. (Sadly, it's paywalled and I couldn't easily find a ungated copy, so I don't have a summary for you.)

The abstract:

  Bright has built an automated system for ranking job candidates against job descriptions. The candidate's resume and social media profiles are interwoven to build an augmented user profile. Similarly, the job description is augmented by external databases and user-generated content to build an enhanced job profile. These augmented user and job profiles are then analyzed in order to develop numerical overlap features each with strong discriminating power, and in sum with maximal coverage. The resulting feature scores are then combined into a single Bright Score using a custom algorithm, where the feature weights are derived from a nation-wide and controlled study in which we collected a large sample of human judgments on real resume-job pairings. We demonstrate that the addition of social media profile data and external data improves the classification accuracy dramatically in terms of identifying the most qualified candidates.


What role the activation function plays in the forward pass and how it is different from backpropagation

Activation function is a non-linear function. Operation in a neuron without activation function is just a linear function. If we don't put activation function between operation of neurons, then the function "Layer" is useless.

for example if you have two layer network, when you are doing forward-propagation, your output (without activation function) of your first layer will be calculated as:

$O_1 = W_1X+b_1 $

Then your output of your second layer will be:

$O_2 = W_2O_1+b_2 $

If we substitute $O_1$, so the output of your second layer can be calculated as:

$O_2 = W_2(W_1X+b_1)+b_2 $

or simply

$O_2 = W_2W_1X+W_2b_1+b_2 $

As we train neural network to optimize the value of $W$ and $b$ (we train to find the best value of it) so instead of training neural network with two layers, we actually just train a one layer network. From the latter formula we can said $W_2W_1 = W_3$ and $W_2b_1+b_2 = b_3$ so our two layer network is just another linear model:

$O_2 = W_3X+b_3 $

We don't want that, we add layers to get more complex model. That's why we use Activation function that is non-linear function. To prevent our deep model is just become a simple linear function.


Does leaky relu help learning if the final output needs negative values?

In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.


There are multi-objective optimization problems, where the objective functions may be in conflict with each other, which can potentially have multiple Pareto-optimal solutions. The paper Multi-objective optimization using genetic algorithms: A tutorial (2006) gives a good overview of the multi-objective optimization problem with genetic algorithms, which can be called evolutionary multi-objective optimization (EMO) or multi-objective optimization evolutionary algorithms (MOEAs).

A common multi-objective genetic algorithm is NSGA (or NSGA-2 and NSGA-3), which stands for Non-dominated Sorting Genetic Algorithm, which is based on the concepts of non-dominated sorting, Pareto front and optimality, niches (sub-populations), and elitism (the best individuals of the current population are carried over to the next generation).

If you want to play with MOEAs, you may wanna try the Python deap package, which supports, for example, the NSGA-2 algorithm.


Can Neural Networks be considered as "Strong AI"?

In this excerpt there is something very misleading

  The principle behind Weak AI is simply the fact that machines can be made to act as if they are intelligent. For example, when a human player plays chess against a computer, the human player may feel as if the computer is actually making impressive moves. But the chess application is not thinking and planning at all. All the moves it makes are previously fed in to the computer by a human and that is how it is ensured that the software will make the right moves at the right times.

Intelligent systems do not in general make use of a look up table. For example, the first chess computer Deep Blue made use of complex evaluation functions.

Neural networks could be trained to act as these evaluation functions.

Additionally, there are planning agents - this is a very large sub-field in AI.

Below again the article mistakes all artificial agents for simple deterministic Turing-like machines

  Weak AI is focused towards the technology which is capable of carrying out pre-planned moves based on some rules and applying these to achieve a certain goal but, Strong AI is based on coming up with a technology that can think and function very similar to humans.

A highly recommended book is "Artificial Intelligence: A modern Approach." This reference covers many important sub-fields of AI and provides enough information to realize when people have mistaken the operations of their laptop as the final measure of artificial agency.

Here Comes the hand waving

The strong vs weak AI dichotomy is fueled by what appears to be a fundamental question: Is there something "special" about humanity that sets us apart from the rest of the material realm?

If we are just a complex arrangement of matter with no metaphysical properties then in theory we only need to solve two problems for strong AI:

Powerful enough hardware
Knowing how to correctly assemble said hardware

Justification of 1 & 2:

In the absence of metaphysical properties, we know that a correct arrangement of matter will result in the epiphenomenal emergence of human like experience - this idea is strongly supported by Big Bang theory and the Theory of Evolution.

Thus, all that is left is finding how to correctly arrange matter (hardware) so as to ensure said emergence.

Though, this philosophical thread quickly descends into the rabbit hole where we must tread softly when discussing whether a synthesized agent actually has qualia. Subjectivity is arguably the fundamental characteristic of human experience and we can't even assure ourselves that another human has this - we just accept that they do.

I hope that I have clarified where the science and philosophy stand.


How can I fix jerky movement in a continuous action space

I think you should try to reason in terms of total "area" explored by the agent rather than "how far" it moves from the initial point, and also you should add some reward terms to push the agent steering more often. I think that the problem with your setting is more or less this: The agent go as straight as it can because you're rewarding it for it, it start sensing an obstacle so it stops, there is no reward for steering so the best strategy to go away from the obstacle and not end the episode is just to go backwards.
Considering that you have information about the grid points at any time you could rewrite the reward function in terms of grid squared explored by checking at each move if the agent end up in a new square grid:
$$
R_t=
\begin{cases}
r_{\text{collision}}\\
\lambda^d\left(\|\mathbf{p}^{x,y}_t-\mathbf{p}_0^{x,y}\|_2-\|\mathbf{p}_{t-1}^{x,y}-\mathbf{p}_0^{x,y}\|_2 \right) + r_{new-squared-explored}
\end{cases}
$$
Moreover it would be useful to add some reward terms also related to how the agent avoid the obstacle, for example a penalisation when the sensor goes and remain under a certain threshold (to make the agent learn to not go and stay too close to an obstacle) but also a rewarding term when an obstacle is detected and the agent manage to maintain a certain distance from it (even though if not well tuned this term could lead the agent to learn to just run in circles around a single obstacle, but if tuned properly I think it might help to make the agent movements smoother).


How successfully can convnets detect NSFW images?

The 2015 paper entitled "Applying deep learning to classify pornographic images and videos" applied various types of convnets for detecting pornography. The proposed architecture achieved 94.1% accuracy on the NPDI dataset, which contains 800 videos (400 porn, 200 non-porn "easy" and 200 non-porn "difficult"). More traditional computer vision methods achieved 90.9% accuracy. The proposed architecture also performs very well regarding the ROC curve.

There does not seem to exist any works regarding the other aspects of NSFW yet.


Should the prediction of the body temperature given a camera image be modelled as classification or regression?

I think it depends on you application and what data you have available.

If the prediction of body temperature itself doesn't have to be accurate and classes like COLD, NORMAL, and HOT will suffice, you should stay with a classification. There isn't a cut off but as you increase the number of classes that represent numbers on the same scale, it may become more difficult to interpret the result as there will be a distribution across the classes.

If you choose regression on the other hand, you are not restricted by your classes anymore and may be able to tell the difference between 36.5 and 36C which (according to wikipedia) can be the difference between normal and cold. This is something classes may not be able to capture.

Another thing to consider is what your training data looks like and how accurately you want to predict the temperature. If you have pictures of people and a temperature reading, where was the reading taken and how accurate is the reading? If it isn't accurate (+- 1 degree) you may not be able to give as accurate predictions as you would like and may only be able to do 3 different classes like above.

If you don't have a data set, then that is another problem altogether and might require another question as it depends on your application.

I think that your problem is interesting and I hope this can help you to understand how best to apply deep learning to it.


Is there a good ratio between the positive and negative rewards in reinforcement learning?

It usually does not matter, but I'm sure there are situations where it could matter.  In theory, if a reward for good behavior is higher than the rewards for bad behavior, then the neural network will be trained such that the higher rewards are preferred, even if those higher rewards are negative.  For example, if a bad reward is -100, then a relatively good reward could be -50, and the network will then be more likely to choose the action with -50 reward over the action with -100 reward.


Face liveness detection using face landmark points

One approach would be to obtain labeled ("yes/no") samples, e.g. via Mechanical Turk, it costs though) and approach it as supervised learning task. E.g. a neural network will discover itself through the learning what features (landmark positions) are helpful for detection.

There should be examples around for mood detection etc.


Why does off-policy learning outperform on-policy learning?

This post contains many answers that describe the difference between on-policy vs. off-policy.
Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-X, R2D2, Agent57 are technically "off-policy", since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number of benefits, such as reusing experience and not forgetting important experiences.
Another benefit is that you can collect a lot of experience distributedly. Since RL is typically not bottlenecked by the computation for training but rather from collecting experiences, the distributed replay buffer in Ape-X can enable much faster training, in terms of seconds but not sample complexity.
However, it's important to emphasize that these replay-buffer approaches are almost on-policy, in the sense that the replay buffer is constantly updated with new experiences. So, the policy in the replay buffer is "not too different" from your current policy (just a few gradient steps away). Most importantly, this allows the policy to learn from its own mistakes if it makes any...
Off-policy learning, in general, can also refer to batch RL (a.k.a. offline RL), where you're provided a dataset of experiences from another behavior policy, and your goal is to improve over it. Notably, you don't get to rollout your current policy in any way! In this case, algorithms that worked well with a replay-buffer (like DQN, SAC) fail miserably, since they over-estimate the value of actions when they extrapolate outside the "support" of the dataset. See the BCQ paper which illustrates how a lot of "off-policy" algorithms like DQN fail when the "distance between the two policies is large". For this task, SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR).
It's also worth noting that importance sampling can correct off-policy gradients to be on-policy; but the farther away your target policy is, the larger the variance. This is especially deadly for long horizon tasks (often called curse of horizon).
To summarize, using replay-buffer (which makes the algorithm off-policy), especially a distributed one, can offer a lot of benefits over pure on-policy algorithms. However, this is a very special class of off-policy algorithms, where the behavioral policy is close to your policy.
But in general, off-policy is a lot harder than on-policy; you'll suffer from extrapolation bias if you use DQN-based approaches, and exponential variance blow-up if you use importance sampling to correct for it.


How is parallelism implemented in RL algorithms like PPO?

OpenAI have a post on that: https://openai.com/blog/openai-five/

They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.

PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).


Why is the policy implied by Q-learning deterministic, when it always chooses the action with highest probability?

Q-learning uses the maximizing value at each step,

Mostly true. The target policy that Q-learning learns the action values of is the one with the maximum value. While training, Q-learning will take one action randomly (typically with a high probability of taking the action with maximum value), whilst it makes updates to value estimates assuming it will always take the maximising action next.

which implies that there is a probability distribution and it happens to choose the one with the highest probability.

Not true. This is not implied at all. The action values that Q-learning estimates are not probabilities, but expected sums of future reward. The probability distribution for exploring actions is added outside of the Q table or Q function estimate. There is no implied probability distribution of actions in the target policy. When implementing the target policy, you may decide to break ties randomly, but that is not an important detail.
It is worth noting that even if you were working with a table of action probabilities, then an agent that always chose the one with the maximum probability would be deterministic. A list of numbers from a table or non-stochastic function (which is how a Q table or Q function is implemented, and also how policy functions are implemented for methods that do process probabilities), even if it represents probabilities, cannot be random in itself. Instead, it must be interpreted by a process that decides how to use them. A process that includes a random number generator can use the numbers to generate a probability distribution that it samples from.
In Q-learning, the process that updates the Q table estimates does not include a random number generator, so as a consequence it is deterministic. However, the choice of which actions to explore is often random, and it is sometimes the case that the environment includes randomness in state transitions or reward values. So Q-learning taken as a whole is a random process.


What are the societal risks associated with AI?

The biggest risk is algorithmic bias. As more and more decision-making processes are taken on by AI systems, there will be an abdication of responsibility to the computer; people in charge will simply claim the computer did it, and they cannot change it.

The real problem is that training data for machine learning often contains bias, which is usually ignored or not recognised. There was a story on BBC Radio about someone whose passport photo was rejected by an algorithm because he supposedly had his mouth open. However, he belonged to an ethnic group which has larger lips than Caucasian whites, but the machine could not cope with that.

There is a whole raft of examples where similar things happen: if you belong to a minority group, machine learning can lead to you being excluded, just because the algorithms will have been trained on training data that was too restricted.

Update: Here is a link to a BBC News story about the example I mentioned.


Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?

In your case the most probable explanation would be the case of overfitting. The model with too many hidden layers have lots of parameters. By means of all these parameters the model is remembering stuff from the training data itself instead of generalizing by learning the useful patterns.
As a rule of thumb if you increase the number of hidden layers more and more at some point model would perform poorly. (I am assuming there is non-linearity in between. In case there is no non-linearity, it doesn't matter how much you stack, it would give the same result because it just boils down to one single layer).
As an experiment, you can try to add regularization and you will see the model won't be performing that bad. Because now model is being punished for being too confident about the things it is remembering. As a result, it won't overfit to the training data.


What framework for a project with a custom environment?

You can modify the environments and create custom models using Ray RLlib: https://docs.ray.io/en/latest/rllib.html


Why can't MLPs perform non-linear regression and classification?

In Single Perceptron / Multi-layer Perceptron(MLP), we only have linear separability because they are composed of input and output layers(some hidden layers in MLP)

This is wrong.

A multi-layer perceptron (i.e. a feed-forward neural network) with non-linear activation functions can perform non-linear classification and regression. In fact, an MLP with one hidden layer with an arbitrary number of hidden nodes, each of them with a sigmoid (which is a non-linear function), can approximate any continuous function (up to an approximation error).

On the other hand, perceptrons can't do that. They perform only linear classification/regression.

  I thought the MLP was a non-linear classifier.

You're right, unless the MLP only uses linear activation functions. In that case, it won't be able to perform non-linear classification/regression.

(P.S.: I suggest you always question the truth and correctness of what you read on the web, especially, on sites like Medium, as you actually did!)


Would this work to prevent forgetting: train a neural net with N nodes. Then, add more nodes and stop training the original nodes

Per Neil's line of questioning:

If you're doing offline, supervised or reinforcement learning, where you have the dataset and decide what to show to the network next, there's no reason to do this. It's better to just adjust your training to show rare examples to the network more often.
If you're doing online reinforcement learning, then the agent typically controls which examples it sees next through its choice of actions. Increasing the agent's exploration rate will cause it to see a more diverse set of examples, but these examples are also less likely to be useful for solving the task the agent is working on.
If you're doing supervised, online, learning, and you don't control the order in which examples appear, it might be useful to freeze some of the weights, but it might be better, and would certainly be simpler, to increase the learning rate for rare classes instead (in effect, showing the network a given rare example repeatedly when it does show up, so that it takes longer to forget it).


Which machine learning approach can be used to predict a univariate value?

Assuming your data is not purely random (otherwise it will be difficult to make useful predictions), you can try the following:

Hidden Markov Models: Treating your discrete numbers as states, you can use HMMs to predict the next state (next number) by looking at the last or the $n$ last states.
Recurrent Neural Networks: Assuming you have a lot of number sequences, you can treat your numbers as time series. Train a model and predict the next number(s). Here is a nice tutorial which uses LSTM cells.
Autoregressive models: AR models are quite powerful as well and have already been successfully used in finance and signal processing before the rise of deep learning.

Further this post (Problem in discrete valued time series forecasting) on Cross Validated might answer your question as well.


Are Modular Neural Networks more effective than large, monolithic networks at any tasks?

A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. Modularization as systems grow in complexity and demands on performance and capability grow is as old as industrialization.

Our laboratory works with robotic control, thermodynamic instrumentation, and data analysis, artificial networks are components in these larger system contexts. we have no single MLPs or RNNs that by themselves perform any useful function.

Contrary to the conjecture about hierarchies decades ago, the topology approach that seems to work well in most cases follows the more common system module relationships that are seen in power plants, automated factories, aeronautics, enterprise information architectures, and other complex engineered creations. The connections are those of flow, and if those are designed well, oversight functions are minimal. Flow occurs between modules involving protocols for communications, and each module performs its function well, encapsulating the lower level of complexity and functional detail. It is not one network overseeing another that seems to emerge most effective in actual practice but balance and symbiosis. Identification of clear master-slave design in the human brain seems to be equally slippery. It may be naive to assume that some all knowing subsystem can be a single causal factor in the remarkable collaborative synchronization of a number of other subsystems.

The challenge is not finding the information paths that make up the system information topology. The information flow is often obvious upon problem analysis. The difficulty is in discovering the best of strategies to train these independent networks. Training dependencies are common and often critical, whereas in animals, the training occurs in situ or not at all. We're discovering conditions under which that kind of learning in our systems is practical and how to achieve it. Most of our research along these lines is intended to discover ways to achieve higher reliability and lower burden in terms of research hours to get it.

Higher functionality is not always of benefit. It often produces lower reliability and consumes additional development resources with little return. Find a way to marry higher level automation, resource thrift, and reliability into one development process, and you might win an award and honorable mention around the web.

Parallel systems that have the same objective is a good idea, but not a new one. In one aeronautics system, nine parallel systems have the same objective, in groups of three. Each group uses a different computing approach. If two of the systems using the same approach provide the same output and the third differs, the matching output is used and the difference in the third is reported as a system fault. If two of the different approaches provide similar results and the third differs substantially, a merge of the two similar results is used and the third is reported as a use case to further develop the dissenting approach.

The improved fault tolerance has a cost, eight more systems and associated computing resources and connectivity plus the comparators at the tail, but in systems that are a matter of life and death, the extra costs are paid and the reliability is maximized.

Dynamic topological adaptation is related to redundant systems and fault tolerance but in some ways is quite distinct. In that area of development, the technology to follow is neuromorphic computing, which is partly inspired by neuroplasticity.

One last distinction to consider is between process topology, data topology, and hardware topology. These three geometric frames can produce greater efficiency together if aligned in specific ways that produce more direct mappings between the relationships between flow, representation, and mechanics. They are, however, distinct topologies. The meaning of alignment may not be apparent without diving deeply into these concepts and the details that emerge for specific product or service objectives.

Footnotes

[1] Deep networks that are trained as a single unit and function without connectivity to other artificial networks are not necessarily monolithic. Most practical deep networks have heterogeneous sequence of layers in terms of their activation functions and often their cell types.


How can we create a vector space where word spelling and pronunciation can be easily compared?

If you only need the vector space as a way to obtain a similarity measure, you may want to consider a distance measure instead. Similarity and distance are inversely related: identical words have maximum similarity or zero distance, and as the similarity decreases, the distance increases.

For instance, the Wagner-Fischer algorithm computes the edit distance between two strings of characters. This edit distance takes into acccount insertions and deletions, as in your examples, but also substitutions (for example "gray" vs. "grey").

The article linked above includes pseudocode that should translate easily to actual code.


Have any AI systems yet been developed that can knowingly lie to / deceive a human?

The Saturday Papers: Would AI Lie To You? is a blog post summarizing a research paper called Toward Characters Who Observe, Tell, Misremember, and Lie. This research paper details some researchers' plans to implement "mental models" for NPCs in video games. NPCs will gather information about the world, and convey that knowledge to other people (including human players). However, they will also "misremember" that knowledge (either "mutating" that knowledge or just forgetting about it), or even lie:

As a subject of conversation gets brought up, a character may convey false information—more precisely, information that she herself does not believe—to her interlocutor. Currently, this happens probabilistically according to a character’s affinity toward the interlocutor, and the misinformation is randomly chosen.

Later on in the research paper, they detailed their future plans for lying:

Currently, lies are only stored in the knowledge of characters who receive them, but we plan to have characters who tell them also keep track of them so that they can reason about past lies when constructing subse- quent ones. While characters currently only lie about other characters, we plan to also implement self-centered lying (DePaulo 2004), e.g., characters lying about their job titles or relationships with other characters. Finally, we envision characters who discover they have been lied to revising their affinities toward the liars, or even confronting them.

The research paper also detailed how other video game developers attempted to create lying NPCs, with an emphasis on how their system differs:

TALE-SPIN characters may lie to one another (Meehan 1976, 183-84), though rather arbitrarily, as in our current system implementation. GOLEM implements a blocks world variant in which agents deceive others to achieve goals (Castelfranchi, Falcone, and De Rosis 1998), while Mouth of Truth uses a probabilistic representation of character belief to fuel agent deception in a variant of Turing’s imitation game (De Rosis et al. 2003). In Christian (2004), a deception planner injects inaccurate world state into the beliefs of a target agent so that she may unwittingly carry out actions that fulfill ulterior goals of a deceiving agent. Lastly, agents in Reis’s (2012) extension to FAtiMA employ multiple levels of theory of mind to deceive one another in the party game Werewolf. While all of the above systems showcase characters who perceive—and in some cases, deceive—other characters, none appear to support the following key components of our system: knowledge propagation and memory fallibility. ...
Like a few other systems noted above, Dwarf Fortress also features characters who autonomously lie. When a character commits a crime, she may falsely implicate someone else in a witness report to a sheriff, to protect herself or even to frame an enemy. These witness reports, however, are only seen by the player; characters don’t give false witness reports to each other. They may, however, lie about their opinions, for instance, out of fear of repercussions from criticizing a leader. Finally, Dwarf Fortress does not currently model issues of memory fallibility—Adams is wary that such phenomena would appear to arise from bugs if not artfully expressed to the player.


The term it comes from cognitive science - depending on the paradigm, it can have many meanings correlated with neuroscience as well as the brain.
Activities based on this sphere is :

language
learning
thinking
perception
awareness
decision making
intelligence

The mentioned cognitive processes - activate the appropriate regions in the brain narrow cognitive intelligence - it's an effective response to in particular in unforeseen and uncertain situations.
However, it requires the ability to set goals, learn, plan and think about your own thinking.


How to train a logical XOR with reinforcement learning?

Reinforcement learning is used when we know the outcome we want, but not how to get there which is why you won't see a lot of people using it for classification (because we already know the optimal policy, which is just to output the class label). You knew that already, just getting it out of the way for future readers!

As you say, your policy model is fine - a fully connected model that is just deep enough to learn XOR.  I think the reward gradient is a little shallow - when I give a reward of +1 for "3 out 4" correct and +2 for "4 out of 4", then convergence happens (but very slowly).


In PyTorch, why does the sequence length need to be provided as the first dimension of the input tensor for an RNN?

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN.
Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.
To answer the second part of your question, normally, each sequence in a batch is independent of the others (since they commonly get sampled at random). So, there is no direct dependence between any two inputs in a batch (except, of course, for the fact that they are commonly expected to stem from some underlying shared data generating process which you want to approximate by the RNN).
And a hidden state is commonly maintained per batch element, i.e. there is one hidden state per batch element (i.e. per sequence).


With Monte Carlo off-policy learning what do we correct by using importance sampling?

We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.

The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,a)$) of a state (or a state action pair). These value functions are always defined with respect to some policy $\pi(a|s)$ and is given the notation of $V^{\pi}(s)$ (or $Q^{\pi}(s,a)$).
The off-policy learning problems are arising when you have two policies: the generation policy $\mu(a|s)$ and the target policy $\pi(a|s)$. Your MC sampling data came from an agent following $\mu$, while you want to improve your target policy $\pi$. It is pretty straightforward from here that you'd need to weight your calculations with factors like $\frac{\pi(a_i|s_i)}{\mu(a_i|s_i)}$ - that's what importance sampling is.


Use AI to auto-correlate the words of human-translated texts?

You are basically describing the way Google Translate works.

There has been a lot of research in text alignment in the area of multi-lingual corpus linguistics. An early paper (with sourcode) is Gale and Church's A Program for Aligning Sentences in Bilingual Corpora (PDF).

In linguistics these are called parallel texts. On the wikipedia page you will find links to a number of alignment programs which attempt to solve that issue.


"Word Sense Disambiguation" refers to the idea that words can have different meanings in different contexts. Here are some examples

"I went to the river river bank" vs "I deposited my check at the bank"
"He's mad good at that game" vs "I am so mad at you"

How it effects NLP, comes down to the way we process text. This generally includes the steps of tokenization and embedding them into some form of vector space. These embeddings in many cases are trained either through some self supervised task on some corpora (examples include Word2Vec or Glove) or doing it from scratch under whichever task/data-set is being used.

Now regarding that paper it does not solve the problem but it does extend a new methodology that assists in helping achieve a better learned generalizable representation for this task. The way I interpreted it is that they don't just use a sense-label (which would be a one-hot encoding / what they call discrete) but instead use a continuous sense representation and do the comparison there. This difference allows for words with similar but different senses to not be equidistant from words with completely different senses.


How to obtain a formula for loss, when given an iterative update rule in gradient descent?

You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: https://github.com/JamesChuanggg/pytorch-REINFORCE/. First of all, I would like to note that a policy can be represented or implemented as a neural network, where the input is the state (you are currently in) and the output is a "probability distribution over the actions you can take from that state received as input".

In the Python module https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py, the policy is defined as a neural network with 2 linear layers, where the first linear layer is followed by a ReLU activation function, whereas the second is followed by a soft-max. In that same Python module, the author also defines another class called REINFORCE, which creates a Policy object (in the __init__ method) and defines it as property of that class. The class REINFORCE also defines two methods select_action and update_parameters. These two methods are called from the main.py module, where the main loop of the REINFORCE algorithm is implemented. In that same main loop, the author declares lists entropies, log_probs and rewards. Note that these lists are re-initialized at ever episode. A "log_prob" and an "entropy" is returned from the select_action method, whereas a "reward" is returned from the environment after having executed one environment step. The environment is provided by the OpenAI's Gym library. The lists entropies, log_probs and rewards are then used to update the parameters, i.e. they are used by the method update_parameters defined in the class REINFORCE.

Let's see better now what these methods, select_action and update_parameters, actually do.

select_action first calls the forward method of the class Policy, which returns the output of the forward pass of the NN (i.e. the output of the soft-max layer), so it returns the probabilities of selecting each of the available actions (from the state given as input). It then selects the probability associated with the first action (I guess, it picks the probabilities associated with the action with the highest probabilities), denoted by prob (in the source code). Essentially, what I've described so far regarding this select_action method is the computation of $\pi(A_t \mid S_t, \theta)$ (as shown in the pseudocode of your question). Afterwards, in the same method select_action, the author also computes the log of that probability I've just mentioned above (i.e. the one associated with the  action with the highest probability, i.e. the log of prob), denoted by log_prob. In that same method, the entropy (as defined in this answer) is calculated. In reality, the author calculates the entropy using only one distribution (instead of two): more specifically, the entropy is calculated as follows entropy = -(probs*probs.log()).sum(). In fact, the entropy loss function usually requires the ground-truth labels (as explained in the answer I linked you to above), but, in this case, we do not have ground-truth labels (given that we are performing RL and not supervised learning). Nonetheless, I can't really tell you why the entropy is calculated like this, in this case. Finally, the method select_action then return action[0], log_prob, entropy.

First of all, I would like to note that the method update_parameters is called only at the end of each episode (in the main.py module). In that same method, a variable called loss is first initialized to zero. In that method, we then iterate the list of rewards for the current episode. Inside that loop of the update_parameters method, the return, R is calculated. R is also multiplied by $\gamma$. On each time step, the loss is then calculated as follows

loss = loss - (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

The loss is calculated by subtracting the previous loss with

(log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

where log_probs are the log probabilities calculated in the select_action method. log_probs is the part $\log \pi(A_t \mid S_t, \theta)$ of the update rule of your pseudocode. log_probs are then multiplied by the return R. We then sum the result of this multiplication over all elements of the vector. We then subtract this just obtained result by the entropies multiplied by 0.0001. I can't really tell you why the author decided to implement the loss in this way. I would need to think about it a little more.

The following article may also be useful: https://pytorch.org/docs/stable/distributions.html.


I can't answer definitively without a detailed breakdown of game mechanics of dota2 vs. Starcraft, but assuming the games have similar complexity, the AI controlling multiple in-game agents that form a team would be more complex, and therefore more difficult to create, than a single agent "team".

My supposition is based on the fundamentals of combinatorics and computational complexity:

in dota2, the AI has to not only to identify winning strategies, but coordinate multiple agents.

by contrast:

the Starcraft AI only has to identify a winning strategy, with no coordination between discrete agents which form a team.

Coalitions have traditionally been difficult to analyze in Game Theory and Combinatorial Game Theory (in the latter case especially, restricting mathematical study, for the most part, to games involving only 2 players.)  In this case the issue is not coalitions--the multiple game agents are already on the same team--but complexity is nevertheless dramatically increased by the existence of multiple agents on a team as compared to a team consisting of a single agent.


How is the transformers' output matrix size arrived at?

Aha, I understand now! In the paper, the diagram for SHA has its inputs in the order Q, K, V, while the diagram for MHS has its inputs in the order V, K, Q! In the grand diagram for the entire net, I thought the arrows for the inputs were in the order for SHA, but they are actually in the order for MHA. It is a bit confusing that when changing to the MHA diagram, they decided to swap the order of inputs in the paper, but it all makes sense now. output_seq x d_q multiplies d_q x input_seq to get output_seq x input_seq, which finally multiplies with input_seq x d_v to get output_seq x d_v, which then concatenates to output_seq x hd_v, so after the final linear layer it becomes output_seq x d_model, the standard in the decoder layer. Hopefully, anyone else who mixed up orders from the diagram will understand now.


Is logistic regression more free from the conditional independence assumption than naive Bayes?

I refer you to Prof. Tom Mitchell's (the lecturer in the video) draft chapter as the best explanation I could find. I will try to explain it in layman's terms here.

Given a boolean problem, our logistic regression classifier will assign the label $Y=0$ iff $w_0 + \sum_{i=1}^{n}w_{i}X_{i} > 0$ (from the video you posted; eqn. 18 in the draft chapter). Otherwise, it will assign the label $Y=1$. Remember the goal of logistic regression is to learn the weights $w_0, w_1, ..., w_n$ that will give us a linear function that would seperate our data best.

It can be shown that $w_0=\ln\frac{1-\pi}{\pi} + \sum_{i=1}^{n} \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_{i}^2}$ and $w_i = \frac{\mu_{i0} - \mu_{i1}}{\sigma_{i}^2}$ (eqns. 19-23 in the draft chapter), where:

$\pi$ is $P(Y=1)$,
$\mu_{i0}$ and $\mu_{i1}$ are the means of feature $X_i$ in classes $Y=0$ and $Y=1$, respectively, and
$\sigma_i$ is the standard deviation of feature $X_i$

This means that we can express our weights $w_0, w_1, ..., w_n$ in terms of the Gaussian Naive Bayes classifier's parameters. Now, here is the difference:

Our goal is to find $P(Y=y_k|X_1 ... X_n)$. Assuming conditional independence, $P(Y=y_k|X_1 ... X_n) \propto P(Y=y_k) \prod_{i}^n P(X_i | Y=y_k)$.

If we estimate $\pi$, $\mu_{i0}$, $\mu_{i1}$, and $\sigma_i$, then use those values to estimate $w_0, w_1, ..., w_n$, then we use the conditional independence property directly in our calculations. This is what the Naive Bayes classifier does.

If we estimate $w_0, w_1, ..., w_n$ directly, by repeatedly minimizing the error of an assumed linear function, we are not strictly abiding by the conditional independence property, because we don't explicitly calculate an estimate of the conditional probability $P(X_i|Y=y_k)$. We completely bypass that step. This is what the Logistic Regression classifier does.

From the section 4 of the draft chapter (GNB is short for Gaussian Naive Bayes):

  ... Logistic Regression directly estimates the parameters of $P(Y|X)$, whereas Naive Bayes directly estimates parameters for $P(Y)$ and $P(X|Y)$... if the GNB assumptions hold, then asymptotically (as the number of training examples grows
  toward infinity) the GNB and Logistic Regression converge toward identical classifiers.

Another way to understand this is that it has to do with the fact that Logistic Regression is a discriminative classifier, whereas Naive Bayes is a generative classifier. For more on that, read Prof. Andrew Y. Ng and Prof. Michael I. Jordan's paper, On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes.


Can the IQ of an AI program be measured?

Short answer: No.

Longer answer: It depends on what IQ exactly is, and when the question is asked compared to ongoing development. The topic you're referring to is actually more commonly described as AGI, or Artificial General Intelligence, as opposed to AI, which could be any narrow problem solving capability represented in software/hardware.

Intelligence quotient is a rough estimate of how well humans are able to generally answer questions they have not previously encountered, but as a predictor it is somewhat flawed, and has many criticisms and detractors.

Currently (2016), no known programs have the ability to generalize, or apply learning from one domain to solving problems in an arbitrarily different domain through an abstract understanding. (However there are programs which can effectively analyze, or break down some information domains into simpler representations.) This seems likely to change as time goes on and both hardware and software techniques are developed toward this goal. Experts widely disagree as to the likely timing and approach of these developments, as well as to the most probable outcomes.

It's also worth noting that there seems to be a large deficit of understanding as to what exactly consciousness is, and disagreement over whether there is ever likely to be anything in the field of artificial intelligence that compares to it.


When are Q values calculated in experience replay?

Because the Q value is different, I don't see how the reward signal at time $t$ is of any relevance for $Q_{t+x}(s_t,a_t)$ at $t+x$, the time of learning.

The $r_t$ value for any single step is not dependent on $Q$ or the current policy. It is purely dependent on $(s_t,a_t)$. That means you can use the Q update equation to use it to calculate new TD targets, by combining your knowledge of the old reward signal and the value functions of the latest policy.

The value $r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta_i^-)$ is the TD target in the loss function you show. There are many possible variations to calculate the TD target value in RL, with different properties. The one you are using is biased towards initial random values unrelated to the problem, and also deliberately biased towards older Q values to prevent runaway feedback. It is also low variance compared to other possibilities, and simpler to calculate.

  Also, it is likely that following a policy which is derived from $Q_t$ would lead to $a_{t}$, whereas following a policy which is derived from $Q_{t+x}$ would not.

Yes that is correct. However, you don't care about that for a single step, you just care about calculating a better estimate for $Q(s,a)$ regardless of whether you would choose $a$ in state $s$ with the current policy. That is what an action value measures - the utility of taking action $a$ in state $s$ and thereafter following a given policy.

This is a strength of experience replay, that you are constantly refining your estimates of off-policy action values to determine which is the best.

This does become more of an issue when you want to use longer trajectories in the update step (which you may want to do to reduce bias of your TD target estimate). A series of steps in history $s_t,a_t,r_t ... s_{t+1}, a_{t+1},r_{t+1}...s_{t+n}$ may not have the same chance of occurring under the latest policy as it did when it was stored in memory. For the first step $s_t,a_t$ again you don't care if it is one you would currently take because the point is to refine your estimate of that action value. However, if you want to use $r_{t+1}, r_{t+2}$ etc plus $s_{t+n}$ to create a TD target, then you have to care whether your current policy and the one used to populate the history table would be different.

It is a problem if you want to use more sophisticated estimates of TD target that use multiple sample steps along with experience replay. There are some approaches you can take to allow for this, such as importance sampling. For a single step update mechanism you don't need to worry about it.

  I don't see in the experience replay algorithm that the Q value $Q_t(s_t, a_t)$ is saved, so I must assume that is is not.

This is correct. You must re-calculate the TD target from a more up-to-date policy to get better estimates of the action value. With experience replay, you are not interested in collecting a history of values of Q. Instead you are interested in the history of state transitions and rewards.

  Why does calculating the Q value again at a later time make sense FOR THE SAME SAVED REWARD AND ACTION?

Because it will change, due to the learning process summarising effects of state transitions.

As a toy example, consider a maze solver with traps (negative rewards) and treasures (positive rewards). At one point in history, the agent finds itself in a location and its policy told it to move into a trap on the next step. The agent would initially score location and steps leading up to it with negative Q values. Later it discovers through exploration that there is also some treasure if it takes a different turning towards the end of the the same series of steps. With experience replay, and re-calculating Q values each time, it can figure out which section of that path should be converted to high scores because they lead to treasure as well as the trap and now the agent has a better policy for the end of the path it has better estimates of value there.


Are machine learning and self-learning really possible?

Yes of course they are possible, I have built some! You are correct that there is an aspect of randomness to the process of machine learning but it is more accurate to describe this as trial and error. Each successive try in a machine learning system is evaluated against a goal and if it is an improvement or is closer to the goal, then this try is stored and some aspect that made it successful is incorporated into similar trys for this type of input. Therefore, machines learn by trying all possible combinations of a problem, albeit with some clever human described short cuts or "heuristics" to make the task easier.