How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?

First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.
Loss is defined as:

$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$

If we use definition of expectation (for discrete case):

$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$
we get

$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$

probability $p_\theta(w_t)$ is defined as $softmax(s_t)$ = $S(s_t)$. If we calculate derivative of $L(\theta)$ with respect to $s_t$, we are basically calculating derivative of softmax $S(s_t)$ with respect to $s_t$.

It can be shown and i won't derive it here that derivative of softmax function is

$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$

where $1_{ij}$ here is defined as yours. If we plug this derivative into derivative of  $L(\theta)$ and replace $S(s_t)$ with $p_\theta(w_t)$ (remember that probability distribution is defined as softmax here) we get:

$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$

we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:

$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$

now, if we ignore expectation we get:

$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$

I probably butchered notation is some places, but base idea should be clear


Into which subcategories can reinforcement learning be divided?

Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I recommend, if you can, to describe some sort of "RL taxonomy" instead. By this I mean describing different RL characterizations without assuming they're mutually exclusive.
To add to your list :

On-policy or off-policy
Value based or policy gradient


Given an image and two points $A$ and $B$ on that image, how could we find a path from $A$ to $B$?

Let's say I have a white image, a green blob in the middle, and points $A$ and $B$ at either side of the blob, I need to get from $A$ to $B$. I don't have points to traverse. I just have this image.

If your image is as "simple" as you describe here, with very easily distinguishable colours, the easiest solution would likely be to construct a graph as expected by algorithms such as A* as follows:

Every pixel becomes a node in the graph, with connections to all adjacent pixels
Green pixels (the green blob in the middle) are marked as non-traversable (or simply don't have connections to adjacent pixels) (I assume this thing is your obstacle).
Pixels belonging to the points on either side of the obstacle are marked as starting and goal vertices.

Then, you have the graph that a traditional pathfinding algorithm like A* would expect, and can simply run the algorithm there.

If you get less "clear" images (i.e. not brightly coloured, easily distinguishable things, but more like real-life top-down images of an area), then the above won't really work anymore. In such cases, you'd likely want to look into Reinforcement Learning approaches, where you give an agent a reward when it successfully manages to find a connection between start and goal. In particular, you'd need Deep Reinforcement Learning approaches, because otherwise you're not going to be able to handle complex images.
Note that these approaches are not going to be simple to implement or immediately understand for 100% if you're a beginner. There won't be any "beginner-friendly" approaches for the problem you describe, it simply is quite a complex problem (requiring a mix of "decision-making" and "understanding" images).


What are the common techniques one could use to deal with collisions in a transposition table?

The term you're looking for is "replacement schemes". As far as I'm aware, the primary reference on this is still Replacement Schemes for Transposition Tables, although it is a fairly old paper from 1994.

I'll very briefly summarize the seven different schemes listed in this paper, but full text of the paper is also freely available and contains more info:

Deep: preserve position for which the deepest subtree below it was searched.
New: Always replace an old entry with the newest entry.
Old: Opposite of New. Paper mentions only including it for the sake of completeness, implies that it's probably not a good scheme.
Big1: similar to Deep, but uses the size (in number of nodes) of subtree searched, rather than just its depth. If the same position (i.e. table entry) occurs multiple times in a subtree, it is only counted once.
BigAll: same as above, but really counts the number of nodes rather than number of positions (so a position that occurs multiple times is counted multiple times).
TwoDeep: In this scheme, the TT is modified to contain two positions per table entry, rather than just one position. You could view it as the table having two "layers". In this scheme, a new position is moved into the first "slot" if it has the deepest subtree (as in the Deep scheme), moving the previous position in the first slot into the second slot. If a new position doesn't have a new deepest subtree, it is instead moved into the second slot.
TwoBig1: Similar to the above, but using Big1-style replacement rather than Deep-style replacement.

If I recall correctly, the Two-layered schemes tend to perform best (of course they do require approximately twice as much memory for the same number of bits per key).


Are there well-established ways of mixing different inputs (e.g. image and numbers)?

A more efficient way would be creating a multi input model, with something like this:

___________    _____________
|__Image__|    |Other input|
_____|_____     _____|_____
|___CNN___|     |__Dense___|
_____|______    _____|______
|_Features1_|   |_Features2_|
         __|_____|__
         |__Merge___|
         _____|______
         |___Dense__|
         _____|_____
         |__Output__|

However, you could also combine the unstructured data to the image, as stated in the quora answer:

  The out-of-the-box method:

  If you want to just take your CNN library and use it without much
  thought, there's an easy way to do it.

  Your image has “channels”: red blue and green channels, for example.
  Just add another channel for each unstructured feature. Those channels
  will just be 2D-arrays whose entries are all the same value: the value
  of your outside feature.

  It means more memory and more parameters though. If you have a lot of
  unstructured data, this can become prohibitively expensive.

  The more efficient method (and still not hard):

  You use one or more deconvolutional filters to bring the unstructured
  data up to the size of the structured data, concatenate them along the
  channel dimension, and keep going as if nothing happened.

Source: How can l train a CNN with extra features other than the pixels? (Quora)


How to evaluate the goodness of images generated by GANs?

I understand that you want to know about methods that we can use to evaluate GANs (Generative Adversarial Networks).

How can GANs be evaluated?

One Discriminator on Separate GANs

We can train a Discriminator beforehand and then we can use this Discriminator on various Generators to see what does this Discriminator say about the images generated from each Generator. The average output of the Discriminator for images generated from one Generator can be compared with that of the images generated from another Generator. This means that according to this Discriminator, one Generator is better than another.

Comparing Probabilistic Models

This is the class of evaluation metrics that is attracting research attention recently. Basically, all generative models are probabilistic in nature, even GANs. When we ask a generative model to generate something like an image, we are simply sampling from a probability distribution. This means that if we can compare the probability distributions of our Generator and that of our original data then we will have an evaluation metric. We can come up with a kernel function to define these probability distributions more accurately. Then we can take our generated samples from the Generator and see the probability of that sample being taken from the original distribution. Researchers have also utilized KL-divergence for comparing samples generated from two distributions.

Note: KL-divergence can not be directly applied in this case because corresponding ground truth is not available. However, there are some modifications proposed in the research work mentioned.

Recommended Readings

A NOTE ON THE EVALUATION OF GENERATIVE MODELS
UNLEARNING FOR BETTER MIXING
A QUANTITATIVE MEASURE OF GENERATIVE ADVERSARIAL NETWORK DISTRIBUTIONS


How can one intuitively understand generative v/s discriminative models, specifically with respect to when each is useful?

The intution that I have about these is that generative are "from abstract to concrete" whereas discriminative models are "from concrete to abstract".

For example: Detecting if a photo has a cat or not is about going from the photo i.e concrete to the abstract concept of a cat. Whereas generating a photo of a cat given some abstract properties about the cat is going from abstract to concrete.


How is the distance between pointers in Stochastic Universal Sampling determined?

As originally conceived in James Baker's 1989 paper Reducing bias and inefficiency in the selection algorithm, Stochastic Universal Sampling accepts a population containing $N$ individuals, and a number of parents to sample, denoted $n$. Assuming fitness values are normalized so that they sum up to $N$, at each step, a new pointer is placed a step equal in size to the fraction $\frac{N}{n}$ ahead of the location of the previous pointer (and the location of the first pointer is set to a random value in the range [0, $\frac{N}{n}$) ). So, for example, if you want to sample 6 individuals from a population of size 10, you would make steps of size $\frac{10}{6}$, spacing your pointers at even intervals of $\frac{10}{6}$.

Modern implementations, like the one on Wikipedia sometimes do not document this fact clearly, although it is apparent what is intended if you already understand the method. They often write the step size as $\frac{F}{n}$, where $F$ is the total fitness of the population, without discussing its relation to the size of the population. The extra normalization step is actually not essential, so modern implementations generally seem to skip it.

So in summary, the step size $\frac{F}{n}$ used if fitness values of a population sum to $F$, and you want to select $n$ individuals. If you want to select more individuals, use a higher value for $n$. If you want to select fewer, use a lower value for $n$, which updates your step size accordingly.

Values of this parameter of $\frac{1}{4}$ or $\frac{1}{6}$ suggest that the implementation may be normalizing the sum of fitness values is being normalized to N, and then using the parameter as a multiplicative factor automatically. This is a fairly reasonable design. You could interpret these values as "Select $\frac{1}{4}$ of the population" and "Select $\frac{1}{6}$ of the population".

Note that this sort of bumps your question up a level: how do you pick the fraction of the population to keep? That question doesn't have a clean answer, and picking it is generally an art developed by experts through practice. It is very closely related to the exploration/exploitation tradeoff.

Some ways you might pick $n$:

Use a fixed value, for instance, keep half the population at each step. The exact proportion you will want to pick is not something you can know in advance. Expert practitioners can make effective guesses. Others will need to just try out different values using a technique like cross validation, and pick whichever one seems to work best.
You can use a value that changes over time. A common strategy for this would be to use one of the temperature schedules developed in the simulated annealing literature, and keep a portion of the population that was inversely proportionate to the temperature. That is, early on, you'd use a large $n$, and keep most of the population around (probably with mutations). Later, you'd use a small $n$ and keep only the best individuals around.
You could use a value of $n$ that changes in response to the fitness of the population. This is much like the adaptive learning rates used in some algorithms to train neural networks (most notably: the ADAM optimizer). When fitness levels have a lot of variety, use low values of $n$ to encourage more exploitation. When fitness levels are all within a narrow band, use high values of $n$ to encourage more exploration.


How to figure out which words have the same meaning in two different languages?

You are implying that such ideas are novel, and that such tools do not exist. But the idea is very popular, and there are numerous tools.

  We need to write a program that would recognize that a word is connected to other words in the same way in both language. Then it would know those two words must have the same meaning.

You are describing the essence of known natural language processing (NLP) tasks such as word alignment (link words in different languages that have the same meaning) and, of course, machine translation.

While learning a machine translation model, we actually do discover which words (or parts of words, or sequences of words) in different languages have the same meaning.

Here are some concepts I would recommend for further study of this subject:

Word alignment, an example for a well-known and popular tool would be fast_align
Word embeddings, word2vec is a widely used tool
Modern machine translation with sequence-to-sequence models, well-known tools are fairseq, or Sockeye


You can find a decent sized collection on archive.org. Just browse through the snapshots and they'll contain a few images. It probably doesn't contain every single one, but it has quite a decent set to start from. :)

Here's one I found:


Do AIs based on MCTS start each game from scratch?

Your intuition is right. The tree is grown after each game.

Before the first play through, the values of each decision point are initialized (randomly or to some constant). Then at the end of one play through, the weights of the decision points are updated using Monte Carlo methods. In this way, on the next play through, the updated weights help the agent make a decision in the next game.

As for how the decision points are added to the tree, it depends on the application. For simple games like TicTacToe, we know all of the decision points. For big games, there are methods to trim the tree of bad performing branches.


What are examples of promising AI/ML techniques that are computationally intractable?

AIXI is a Bayesian, non-Markov, reinforcement learning and artificial general intelligence agent that is incomputable, given the involved incomputable Kolmogorov complexity. However, there are approximations of AIXI, such as AIXItl, described in Universal Artificial Intelligence:  Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability (2005), by Marcus Hutter (the original author of AIXI), and MC-AIXI-CTW (which stands for Monte Carlo AIXI Context-Tree Weighting). Here is a Python implementation of MC-AIXI-CTW: https://github.com/gkassel/pyaixi.


Training the generator in a GAN pair with back propagation

Let's take a look at the images that fooled the D-network. When this happens the binary cross entropy loss of the D-network is important to look at. The network said a fake image was real. As usual, you back propagate to the very beginning of the D-network. What is the back prop doing? It is telling the weights in the D-network to change such that you lessen the loss (remember you are minimizing the loss function by going down hill --> using gradient descent). So you changed the weights of the D-network, but you really don't have to stop there.

The complex nonlinear function the D-network is calculating is really f(Image, weights of D-network). So the output of the D-network is really dependent on the Image fed in and the weights of the D-network. The cost is a function of the output f. So we have c = g(f(Image, weights of D-network)) where g is really the binary cross entropy. Looking at this, we see that the cost is also a function of the image!

We can take the partials with respect to the image just like we take the partials with respect to the weights to update the weights. And so after doing back prop to the image, we don't stop -> we take the partial of the cost function with respect to the image that is fed into the D-network and we update the image. And essentially we don't have the original image when we continue to back prop to the beginning of the G-network. Let me know if you'd like more clarification. Again this is my understanding and I haven't coded a GAN yet, so someone please confirm. Also, I am open to others' interpretations.


What is the effect of parallel environments in reinforcement learning?

Do parallel environments improve the agent's ability to learn or does it not really make a difference?

Yes they can make a difference. There are two ways improvement is seen:

Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with i.i.d. data.

Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.

Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.

Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:

Using an on-policy method, or where experience replay is not an option.

Using a function approximator for policy and/or value function.

A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.
These effects are discussed in sections 1 and 4 of the paper Asynchronous Methods for Deep Reinforcement Learning which introduced A3C - thanks to DeepQZero for that reference.


Which edges of this tree will be pruned by Alpha-beta pruning?

If you prune k and L then you could miss the optimal solution. Assume L=9, if you prune L then the value of the tree is 8. If you don't prune L then the value of the tree is 9. Now I will try and address what I think your actual question is

  But no matter what, the decision of the max(root) will not change, the max will choose the right side no matter what 'k' and 'l' are, because the right side will either be 8 or 9, which is still higher than 4 (returned value from left side).

From this sentence it seems like you don't care about the value of the tree, you only want to find the optimal first move based on alpha beta. You are correct in saying that the correct first move will always contain the right most child of the root, but oftentimes that is not the only information we want. Sometimes we want to know the value of the tree as well or what the complete correct path is, but if we had pruned k and L we would not know these.

Edit: I have changed all 'L's to uppercase, because lower case 'L' looks to much like uppercase 'i'


What is the correct name for state explosion from sensor discretization?

I would refer to your problem as having a continuous state space.
By using a 32-bit float variable you discritize it. However, creating states for every possible value of a 32-bit float variable is probably too much. You should decide on:

the variable range: what is the real range of the position variables (e.g. from 0 m to 10 m),
and what is the resolution you require for your problem (e.g. 0.01 m or 0.1 m).

Note that you should take into account:

the sensor range resolution,
the required range and resolution for the problem.

Then, based on the number of states you could decide to discretize the states or to use a Monte Carlo approach.

See for example the work of Brechtel et al. (2013).


How do you find the gender of a noun in German using spacy

It seems from this old thread German lemmatization and noun gender that they never actually implemented this feature. But in the same thread you can find links to these two GitHub projects that should provide you an alternative:

IWNLP
DEMorphy


Genetic Algorithm vs Particle Swarm Optimization

These kind of questions cannot be answered without looking at a particular project. Each algorithm has its particular strengths and weaknesses; and trade-offs in terms of use of resources (processing power and/or storage space, for example). If there was an objective answer, then the worse algorithm would surely fall in disuse.

It also depends what you mean by "better". Faster? Better score according to some evaluation measure? More robust (ie works with many diverse data sets)?

I would recommend looking at both algorithms in more detail, and trying to understand how they work. Then you should be able to find out which best fits your problem.

However, one problem with Particle Swarm Optimisation is that it is not well understood, so you might have to resort to trial-and-error.


Using NVIDIA VOLTA V with Mac or Windows externally

The NVidia Volta architecture is capable of delivering an astounding 110 teraflops of computing power. This coupled with the NVidia CUDA Neural Network library (cuDNN). Which can be used to speed up DNN's on learning frameworks such as  Caffe 2, Microsoft Cognitive Toolkit, TensorFlow, Theano, and PyTorch. Makes GPU's based on the Volta architecture such as the Titan V specifically suited to accelerate AI workloads.

Regarding your MacBook Air. Unfortunately, it comes with an integrated GPU that can't be replaced because it is soldered to the logic board. So, unfortunately, we have to rule out upgrading it because there isn't any other port to connect a GPU. You can read more about this on Apple's official discussion forum https://discussions.apple.com/thread/5206086 .

Having said that, you possibly can upgrade your GPU within the same series which is the 10 series. Other powerful GPU's within the 10 series include The Titan X, Titan Xp, GeForce GTX 1070 Ti and the GeForce GTX 1080 Ti. Unfortunately, the Geforce 10 series is based on the Pascal architecture meaning your laptop might not by default accept a Volta based GPU such as the Tesla V100 or the Tesla Titan V. Consequently it could involve BIOS modification, chassis modification or heatsink assembly modification to make it work. This kind of upgrade will be diving into uncharted territory since it will void your warranty.

My recommendation is that a Volta based GPU might be unsuitable for your Asus laptop. I'd recommend a GPU that uses an architecture that is similar to your current GPU. Meaning a Pascal-based GPU such as the Titan X or GeForce GTX 1080 would be better suited to satisfy your demands.


structure of neural network for classification problems with large amounts of null classifications

I see multiple reasons to take a different route:

While in "classical" pattern recognition you might have done things like feature engineering outside of your model, one idea of deep learning was to "insource" it into the model and let the model take care of "everything". Following that, I have seen a general tendency in deep net architectures to let your deep net handle all the work in one single model. So your idea kind of contradicts the architectural mainstream.
There are so many deep net architectures being published which are well engineered and tested for all kinds of tasks. Therefore, I would check if there is anything "ready off the shelf" for a task like yours in the very first place. If there is, then it will probably be better than your self-engineered model architecture. And if there is not, you might want to check related tasks and see how they solved this problem, e.g. medical applications where specific cells need to be detected with most examples being negative (and if there is a positive example the area in a picture might still be mostly negative as it is just a single positive cell among many others).
Your thought of feeding the "present/not present" binary classification to your task of locating an object is very closely related to what has been discussed as auxiliary tasks. I don't know if it has been applied to exactly your problem but there are similar applications. For example there is an application where "name recognition" is complemented by the auxiliary task of deciding whether any name is present in a given sentence or not. Which is somewhat similar to your case. The paper An Overview of Multi-Task Learning in Deep Neural Networks provides an overview. A famous example using auxiliary tasks would be GoogleNet. This also goes back to my first point of letting the deep net handle "everything" internally.

However, as a disclaimer, this is a theoretical perspective as I cannot speak from experience regarding your problem.


Why do we sample vectors from a standard normal distribution for the generator?

It has become our human bias that data will arrive from a normal distribution. It is also the most prevalent distribution in nature occurring in many places. Hence, we sample from a normal distribution. Also, central limit theorem works around means lying around normal distribution.
It is not taboo to use others if they are helpful to your network. But, a data from one distribution can transformed into other and if that is something that is required, the network will to do it since it can approximate anything (although a weak assumption, but, hey, its working right?).


It may come as a surprise, after learning all about dynamic programming, temporal difference learning, SARSA/Q-learning, to then discover that there is yet another whole dimension to reinforcement learning (on top of choices for on-policy/off-policy, model-based/model-free, bootstrap/monte carlo etc). That is value-based vs policy-based methods. Policy-based methods are often taught after value-based methods, because they are more complex.

You can learn the parameters of a policy function by training it using a policy gradient method. The archetypical policy gradient method is REINFORCE, although that is not very efficient. You may have heard of policy gradient methods developed recently: A3C, A2C, DDPG, TRPO, PPO . . . there are a few.

  How do you calculate the loss when you only have the 1 move that was played in the game to work with?

You can pre-train a policy network using supervised learning (perhaps using the moves of winning players in high quality games) - that would use multi-class cross-entropy loss that you may be familiar with from supervised classification problems.

Policy Gradient methods work with a reward summation function defined as expected reward given the distribution of states. If your network parameters are $\theta$, then that may look like this:

$$J(\theta) = \sum_{s \in \mathcal{S}} \rho_{\pi}(s) \sum_{a \in \mathcal{A}} \pi(a|s,\theta)q_{\pi}(s,a)$$

where $\rho_{\pi}(s)$ is the expected proportion of time steps spent in state $s$. There is a way to take a sample gradient of this that can be used for gradient ascent - the derivation is called the Policy Gradient Theorem. It's a bit long to include in this answer, but the upshot is that you can use your sampled single step to generate an approximate gradient towards improving the policy. There are a few variations, but for instance advantage actor critic uses this:

$$\nabla J(\theta) = \hat{A}(s,a)\nabla\text{log}(\pi(a|s,\theta))$$

where $\hat{A}(s,a)$ is your current estimate of the advantage (or $Q(s,a) - V(s)$) for taking a specific action in state s.

The related loss function is

$$\mathcal{L}(\theta) = -A(s,a)\text{log}(\pi(a|s,\theta))$$

The $\text{log}$ function looks like an odd addition, but is just a consequence of adjusting $\nabla J$ to take account of ratios in which actions are taken in the current policy. In fact $\nabla\text{log}(\pi(a|s,\theta)) = \frac{\nabla\pi(a|s,\theta)}{\pi(a|s,\theta)}$ and it may help your intuition to keep it in that form (the $\text{log}$ form is concise and used elsewhere in statistics as the "score function", but it is not necessary for anything specific in RL).

The variations in policy gradients may use other functions than the advantage function, and it is not clear that there is any "best" one. The policy gradient theory basically gives us ways of estimating relative benefits of actions, and allows for any offset to estimated return from $(s,a)$ that does not further depend on the choice of action $a$. So you can use any method for getting an estimated return, and offset it with anything that you think might normalise the updates - common choices for the latter include subtracting average reward, or subtracting the state value function.


Which models accept numerical parameters and produce a numerical output?

You're probably looking for regression, either linear or non-linear, which usually refers to a set of methods that can be used to predict a continuous (or numerical) value (the value of the so-called dependent variable), given one or more possibly numerical values (the values of the independent variables). (The other common task is called classification, where the outcome variable is discrete.)

The dependent and independent variables can also be vectors. In that case, it is called multivariate linear or non-linear regression. If there is more than one independent variable (or, also called, predictor), then it is called multivariable (or multiple) linear or non-linear regression. See, for example, Multivariate or Multivariable Regression? (2013), by Bertha Hidalgo and Melody Goodman.

In linear regression, the dependent variable is assumed to be a linear combination of the independent variables, while, in non-linear regression, it is assumed to be a non-linear combination (which is any combination or relationship that is not linear).

The independent variables (or predictors) do not actually need to be independent of each other and, of course, the dependent variable does not need to be independent of the predictors (otherwise you would not be able to predict the value of the dependent variable, given the predictors), so the expression predictor is possibly more appropriate than independent variable. See, for example, In Regression Analysis, why do we call independent variables "independent"?, by Frank Harrell. However, the dependent variable is assumed to be dependent on the independent variables, so you could think of the independent variables as the variables that are not the dependent variables (if this helps you to memorize the concept).

There are more synonyms for the dependent and independent variables, depending on the context or area. For example, the predictors (or independent variables) can also be called covariates, regressors or explanatory variables. The dependent variable can also be called regressand, outcome variable, or explainable variable.

There are other types of regression models. In particular, there is the generalized regression model (GLM), which is (as the name suggests) a generalization of several regression models. The famous logistic regression (where the outcome is actually binomial or binary, so discrete) is an application of the generalized regression model, where the link function (which is a concept that is used in the context of GLMs) is the logit function. See, for example, Why is logistic regression a linear model?. Note that linear regression is also an application of a GLM. See also this article 1.1. Generalized Linear Models, which gives you an overview of different GLMs.

Have also a look at this article Choosing the Correct Type of Regression Analysis, which provides a few guidelines to help you choose the most appropriate regression model for your task.

In Python, you could use, for example, sklearn.linear_model.LinearRegression. The Python library sklearn provides several other regressors. For example, sklearn.ensemble.AdaBoostRegressor or sklearn.ensemble.RandomForestRegressor. (Of course, you will need a training dataset to train these regressors, when calling their method fit).


What is the purpose of the noise injection in the generator network of a GAN?

Your goal is to model a distribution when constructing a GAN, therefore you need a way to be able to sample that distribution. The noise's purpose is so you can do this. Generally, it's drawn from a distribution that is computationally easy to draw from (like a gaussian).

You are modeling the generator $G(X)$  where $X \sim N(\mu, \sigma^2)$. this means $G(X)$ is a random variable itself. The forward pass of the network transforms the $X$ samples into our $G(X)$ samples, allowing us to formulate a loss function (by solving the expectation as the mean of drawn samples) and train the model.

Takeaway: The noise injected is just a parametrization of our Generator in another space, and the training goal is to learn the ideal transformation (we use neural networks because they are differentiable and are effective function approximators)

Also, note to your point of why it doesn't learn the training data (in its entirety) exactly is because generally $G(X)$ is continuous, and therefore if it has 2 images in its codomain, there also exists some path in pixel space from one to the other containing an uncountable (or quantized countable) number of images that don't exist in the training, and this would be reflected in the loss, therefore in the min-max game of the optimization, its difficult for it learn the training set on the nose.


What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)?

The behaviour when playing against "cheats" depends on how the agent has been trained, and how different the game becomes from the training scenarios. It will also depend on how much of the agent's behaviour is driven by training, and how much by just-in-time planning.
In general, unless game playing bots are written specifically to detect or cope with opponents that are given unfair advantages, they will continue to play in the same style as if the cheating had not occurred, and assuming that the rules are still being followed strictly. If the cheating player only makes one or two rules-breaking moves, and the resulting game state is still something feasible within the game, then the agent should continue to play well. If the agent significantly outclasses the human opponent, it may still win.
A completed, trained agent will not adapt its style to "now my opponent can cheat". An agent still being trained could do so in theory, but it would take many games with cheating allowed for it to learn tactics that cope with an opponent that had an unfair advantage.
Agents that plan by looking ahead during play can cope with more unusual/unseen game state - things that may not have been seen in training. However, they still look ahead on the assumption that game play is as desiged/trained for, they cannot adapt to new rules unless those rules are added to the planning by the bot designers. For instance if the allowed cheating was a limited number of extra moves, but only for the human player, the effects of that could be coded into the planning engine, and the bot would "adapt" with help from its designers.

[AlphaGo] adapted to an absurd move made by Lee Sedol in 2015?

Assuming you are referring to game 4, then as far as I know, AlphaGo did not "adapt" to this play, after Lee Sedol managed to put it in a losing position then it started playing badly as it could not find a winning strategy from the board positions it was in, and could not recover. I don't think any effort was put into refining AlphaGo during this game or afterwards to patch it for game 5.


How AlphaGo Zero is learning from $\pi_t$ when $z_t = -1$?

Intuitively I think there's definitely something to be said for your idea, but it's not a 100% clear case, and there are also some arguments to be made for the case that we should also be training the policy from data where $z_t = -1$.
So, first let's establish that if we do indeed choose to discard any and all data where $z_t = -1$, we are in fact discarding a really significant part of our data; we're discarding 50% of all the data we generate in games like Go where there are no draws (less than that in games like Chess where there are many draws, but still a significant amount of data). So this is not a decision to be made lightly (it has a major impact on our sample efficiency), and we should probably only do it if we really believe that policy learning from any data where $z_t = -1$ is actually harmful.

The primary idea behind the self-play learning process in AlphaGo Zero / AlphaZero may intuitively be explained as:

When we run an MCTS search biased by a trained policy $\pi_t$, we expect the resulting distribution of visits to be slightly better than what was produced by $\pi_t$ alone.
According to the expectation from point 1., we may use the visit counts of MCTS as a training target for the policy $\pi_t$, and hence we expect to get a slight improvement in the quality of that trained policy.
If we were to now run a new MCTS search biased by the updated policy in the same situation again, we would expect that to perform even better than it previously did because it is now biased by a new policy which has improved in comparison to the policy we previously used.

Of course, there can be exceptions to point 1. if we get unlucky, but on average we expect that to be true. Crucially for your question, we don't expect this to only be true in games where we actually won, but still also be true in games that we ultimately end up losing. Even if we still end up losing the game played according to the MCTS search, we expect that we at least put up a slightly better fight with the MCTS + $\pi_t$ combo than we would have done with just $\pi_t$, and so it may still be useful to learn from it (to at least lose less badly).
On top of this, it is important to consider that we intentionally build in exploration mechanisms in the self-play training process, which may "pollute" the signal $z_t$ without having polluted the training target for the policy. In self-play, we do not always pick the action with the maximum visit count (as we would in an evaluation match / an important tournament game), but we pick actions proportionally to the MCTS visit counts. This is done for exploration, to introduce extra variety in the experience that we generate, to make sure that we do not always learn from exactly the same games. This can clearly affect the $z_t$ signal (because sometimes we knowingly make a very very bad move just for the sake of exploration), but it does not affect the policy training targets encountered throughout that game; MCTS still tries to make the best that it can out of the situations it faces. So, these policy training targets are still likely to be useful, even if we "intentionally" made a mistake somewhere along the way which caused us to lose the game.


Understanding the equation of TD(0) in the paper "Learning to predict by the methods of temporal differences"

When lambda = 0 as in TD(0), how does the method learn? As it appears, with lambda = 0, there will never be a change in weight and hence no learning.

I think the detail that you're missing is that one of the terms in the sum (the final "iteration" of the sum, the case where $k = t$) has $\lambda$ raised to the power $0$, and anything raised to the power $0$ (even $0$) is equal to $1$. So, for $\lambda = 0$, your update equation becomes

$$\Delta w_t = \alpha \left( P_{t+1} - P_t \right) \nabla_w P_t,$$

which is basically a one-step update (just like Sarsa).


Better to learn the same small set for multiple epochs then go to the next or learn from each one time repeatedly for multiple times?

Small set for multiple epochs causes overfit, so expose entire data in one epoch, then augment (change a littel bit), and train again. Use small amount of dropout also.

I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.

Do not store augmented data, just do augment with function right after next epoch and overwrite the dataset array


Maybe later I can work through your math though I can say this.
First, a forward pass is done to obtain predictions of $z$, at every $t$. Then the adjoint state is run backward in time for every $t$. Which gives the learning impulse. So an initial run is done to obtain values of the dynamical system at all time points and the last of these is the initial point for the backward pass.
This is probably not enough to satisfy your question, hopefully, I can get more specific with respect to the math later.


Does Google's deep dream project have a specific website for playing with it?

Here are a few websites that help run the deep dream generator:

1) http://deepdream.psychic-vr-lab.com/deepdream/

2) https://dreamscopeapp.com/deep-dream-generator

This blog by Google AI explains the concept and here is the GitHub repo for the same. It has a python notebook for reproducing the results.


When exactly is a model considered over-parameterized?

Ok so after a little more reading, I am currently satisfy with what I found for this question.

Yes, the "under-parameterized" and "over-parameterized" terms do not currently have a widely accepted definitions.
Any definition for those term should consider the input data domain as well as the architecture and training procedure.

In a recent paper Deep Double Descent from OpenAI Nakkiran et. al 2019, the authors tried to formalized and generalized the concept of "interpolation threshold" in the "double descent" phenomenon, both terms popularized by Belkin et. al 2019 Reconciling modern Machine Learning practices with the Bias-Variance Tradeoff.

In the Deep Double Descent paper, they define a concept called "Effective Model Complexity (EMC)", which include model architecture, training procedure and data to describe the "interpolation threshold" (the moment at which model can fit near perfect the training data).

EMC below the interpolation threshold is considered "under-parameterized" and above interpolation threshold is considered "over-parameterized".

So according to this definition of EMC, I suppose:

  1M-parameters model on 10M samples of 2 binary features, then it should not be over-parameterized

is over-parameterized because of the simplicity of input data.

  1M-parameters model on 0.1M samples of 512x512 images, then is over-parameterized, or

is probably not parameterized, if it cannot fit the training data with near 0 loss.

  the model in the paper Exploring the Limits of Weakly Supervised Pretraining "IG-940M-1.5k ResNeXt-101 32×48d" with 829M parameters, trained on 1B Instagram images, is not over-parameterized

is not over parameterized because it cannot fit the entire training data perfectly.

I am curious to see if EMC will catch on and be a popular measure for model complexity in the future.


Why is baseline conditional on state at some timestep unbiased?

Using the law of iterated expectations one has:

$\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \mathbb{E}_{a_t \sim \pi_\theta(a_t | s_t)}
 \left[ b(s_t) \right]\right] =$

written with integrals and moving the gradient inside (linearity) you get

$= \sum_{t=1}^T \int_{s_t} p(s_t) \left(\int_{a_t} \nabla_\theta b(s_t) \pi_\theta(a_t | s_t) da_t \right)ds_t =$

you can now move $\nabla_\theta$ (due to linearity) and $b(s_t)$ (does not depend on $a_t$) form the inner integral to the outer one:

$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta \left(\int_{a_t} \pi_\theta(a_t | s_t)  da_t \right)ds_t= $

$\pi_\theta(a_t | s_t)$ is a (conditional) probability density function, so integrating over all $a_t$ for a given fixed state $s_t$ equals $1$:

$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta 1 ds_t = $

Now $\nabla_\theta1 = 0$, which concludes the proof.


How to fix time dimension in time varying data-sets using deep learning model for classification?

In deep learning, it is very common to use Recurrent Neural Networks (RNNs) to handle time-series data with varying input sequence lengths. Check out the RNN Wikipedia for more detail.


Is gradient descent scale invariant or not?

What are you hoping to get out of the answer for this question?
Feature scaling is a method you CAN(but don't have to) use, so that your algorithm performs faster and reaches better general accuracy.
I would say that on a simple regression task, where the feature value ranges do not vary a lot, the output would probably be almost the same, but as soon as you introduce data that has one feature of range 1-10, and other range of 10000-100000, that is where you would notice that you NEED to standardize/normalize your features in order to reach optimal results.
That's why it's almost a general rule to just scale your data, so your algorithm can generalize better, and you don't have to worry about your algorithm giving higher importance to a feature with higher values, instead of the one with lower values(just an example).


What is the difference between the uniform-cost search and Dijkstra's algorithm?

The answer to my question can be found in the paper Position Paper: Dijkstra's Algorithm versus Uniform Cost Search or a Case Against Dijkstra's Algorithm (2011), in particular section Similarities of DA and UCS, so you should read this paper for all the details.
DA and UCS are logically equivalent (i.e. they process the same vertices in the same order), but they do it differently. In particular, the main practical difference between the single-source DA and UCS is that, in DA, all nodes are initially inserted in a priority queue, while in UCS nodes are inserted lazily.
Here is the pseudocode (taken from the cited paper) of DA

Here is the pseudocode of the best-first search (BFS), of which UCS is just a particular case. Actually, this is the pseudocode of UCS where $g(n)$ is the cost of the path from the source node to $n$ (although the title indicates that this is the pseudocode of BFS).


Berkeley AI Course Question on Nearly Zero Sum Games

(using X for epsilon because keyboard)

This is just a hypothesis, but if you have a maximising agent and a minimising agent, then the optimal outcome for A (maximising) is to sweep the board (X,0), while the optimal outcome for B (minimising) is (-X,0) because B is minimising A's score, not their own.

A's optimal outcome is then complicated by the factor for sub-optimality, which we then imagine approaches zero.

There seem to be a bunch of assumptions that are not articulated, though, if this hypothesis is true.


A deep neural network (DNN) is nothing but a neural network which has multiple layers, where multiple can be subjective.

IMHO, any network which has 6 or 7 or more layers is considered deep. So, the above would form a very basic definition of a deep network.


I think the best thing to use here is a form of "structured prediction". Our "target" is a sequence of operations. The framework of structured prediction allows us to chain together as many filters as we want.
With a neural network of fixed architecture, you would have to make sure you have enough space for all the filters you might need.


Can you learn parameters in nonlinear function?

In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $\rho$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $\alpha$.

In this paper, $\rho$ is a non-linear function (that is, a function that is not linear, e.g. the sigmoid function) that applies element-wise to its input. So, if you pass a vector to this $\rho$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $\rho$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $\rho$ is also followed by a matrix. In general, this is not forbidden, even though it is not common.

In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections.


Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

They are not maximizing the gradient, the gradient is of the form
\begin{equation}
\nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t))
\end{equation}
that means that when implementing it in software you can form your objective as
\begin{equation}
J = \sum_{t=0}^T G_t \log(\pi_{\theta}(a_t|s_t))
\end{equation}
and then taking the gradient of that objective is equal to the policy gradient.


How do I increase the size of an (almost) balanced dataset?

Random over sampling creates duplicates of existing examples, so applying this to your training data would be the same as increasing the weight of the oversampled examples. If it's done to all of the examples uniformly then the effects will probably cancel out. SMOTE, on the other hand, creates synthetic examples that are linear combinations of existing examples. Thus it can be thought of as a type of data augmentation, and in some situations this might improve your model's predictions.


Convert input dataset given in hex addresses to int

Add train_data 1st

import pandas as pd

df = pd.read_excel('data.xlsx', index_col=None, names=['train_data'])
df['train_data'] = df['train_data'].apply(lambda x: int(x, 16))
df.to_csv('data.csv', index=False)


How do I keep track of already visited states in breadth-first search?

Dennis Soemers' answer is correct: you should use a HashSet or a similar structure to keep track of visited states in BFS Graph Search.

However, it doesn't quite answer your question. You're right, that in the worst case, BFS will then require you to store 16! nodes. Even though the insertion and check times in the set will be O(1), you'll still need an absurd amount of memory.

To fix this, don't use BFS. It's intractable for all but the simplest of problems, because it requires both time and memory that are exponential in the distance to the nearest goal state.

A much more memory-efficient algorithm is iterative deepening. It has all the desirable properties of BFS, but uses only O(n) memory, where n is the number of moves to reach the nearest solution. It might still take a while, but you'll hit memory limits long before CPU-related limits.

Better still, develop a domain specific heuristic, and use A* search. This should require you to examine only a very small number of nodes, and allow the search to complete in something much closer to linear time.


How to create neural network that predicates result of exam?

You can add as many layers (with any arbitrary number of nodes) as you want.
Please note that as you add more learning parameters (layers and nodes), your model complexity increases. This means the model can potentially learn a more complex input-output relationship. However, it also increases the risk of overfitting. Overfitting generally happens when the model you build is more complex than the data you have. In such a scenario, the model memorizes the data instead of learning from it. In other words, it can produce a very good result for the same data as it was trained on but cannot generalize well. So, it performs poorly when the inputs are slightly different from what is used to be fed at the training stage.
In practice, you may try different architectures and parameter configurations, and measure the generalization capacity of the models (via cross-validation for example) to choose the best model. In English, a generalizable model is the one that performs (almost) equally good on both training/validation and testing sets.


Do smaller loss values during DQN training produce better policies?

I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is much more clear.


How can we find the value function by solving a system of linear equations without knowing the policy?

Your equations all look correct to me.
It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy.
There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance, Monte Carlo policy evaluation or single-step TD learning can both do this. It is also common to work with $\pi(a|s)$ known but $P^{a}_{ss'}$ and $R^{a}_{s}$ unknown in model-free control algoithms such as Q learning.
However, in your case, you are correct, in order to resolve the simultaneous equations you have presented, you do need to know $\pi(a|s)$
This is not as limiting as you might think. You can construct a control method using simultaneous equations, by starting with the policy set to some arbitrary policy. Either a randomly-chosen deterministic policy or the equiprobable policy are reasonable first guesses. Then, after each solution to linear equations, you improve the policy so that each action choice maximises the expected return. This is essentially the policy iteration algorithm but replacing the policy evaluation step with the linear equations method for calculating the values.


Detailed explaination of Facenet Model for face recogniton?

Facenet is a Siamese network. It's basic architecture is this:

The input(a face) is fed through a deep convolutional neural network and also a fully connected layer at the end. The fully connected layer at the end output an embedding of the input image which is a predefined size. The embedding can contain feature that human understand or maybe not. The embedding represent the input image, just in a "compressed" form.

To further explain that, let me give an example. Let's say that you have to describe a face. What's will you say? You will probably say something like the face is round, the eyes are blue, it is a female face and more. The neural network is doing what you are doing, describing the face, but using numbers instead of words.

To do a face recognition task, the network take an pre taken image of the list of people to recognize and the unknown new data from the people to be recognised. It then feed both images into the network and get the embeddings. The network then calculate the distance of the two embeddings, using some metric such as squared error or absolute error. In the image it uses the squared error. If the error is below a certain threshold, the face is recognised. If not, it then loops through the other pre taken images in the set of faces of the system and do the task again. The system stores embedding of the pre taken images before hand.

For training the FaceNet, triplet loss is used. Triplet loss has been explained in another of your post, by me. What is the formula used to calculate the accuracy in the FaceNet model?

Basically, the model is trained using the triplet loss as it can train the network to output a similar embedding for the same person and a very different embedded for a different person.

Sometimes, a binary classification end is also used. It removed the need of the triplet loss and outputs a number from 0 to 1 for similarity instead. This removes the triplet loss part.

Hope my answer can help you and have a nice day!


Why is AI safety so much harder than Isaac Asimov's "Three Laws of Robotics"?

The reason this is hard is because it is not trivial to understand what a law means. Many humans still have a hard time understanding laws and thus we have millions of judges and lawyers who study years to be able to even debate whether a law was broken at all.
More generally to AI, the problem of understanding laws is a byproduct of the bigger problem that AIs today do not yet have mechanisms to turn input (text, audio, other sensor data) into concepts. This is evident in even the most advanced conversation systems which struggle with dialog which is off of a particular set of subjects.
However, there are promising results with probabilistic logic programming languages that attempt to combine first-order logic with statistical inference ($\text{LP}^{\text{MLN}}$ is an example). These languages could provide, in theory, a way for future AI-lawyers (or whatever one should call them) to encode exactly what laws mean. Using this set or encoded laws, a system with a rich enough representation could determine if an action would brake a law or not. This sounds great but the "rich enough representation" is beyond what even today's best deep learning systems (or any other architecture) can do. For comparison, today's best deep learning systems don't even have good object permanence, something humans have around 4-7 months old. Asking those deep learning systems to also be able to represent the complexities of laws is profoundly outside of current AI capabilities.
Another promising area is only including text data and having artificial judges. This wouldn't immediately mean that AIs would obey laws but it is the start of AIs understanding the word of law.


Text classification of non-equal length texts, should I pad left or right?

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference.
I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short.
Either way, when a model is trained a certain way, it shouldn't make a difference so long the testing is also padded the way it was presented in training.


What is the relationship between degrees of freedom and the size of the training dataset?

When you define a straight line of the form $y=mx+c$, you need 2 points $(x_1,y_1)$ and $(x_2,y_2)$, to solve for the 2 variables $m$ and $c$ (you can easily visualise this graphically). Similarly, a parabola of the form $y=ax^2+bx+c$ will require 3 such points.

Now viewing it as a ML problem, you are given the points and you have to estimate the parameters such that the training error is 0 (Regression). So just like the previous case you have a bunch of $(x_i,y_i)$ and you have to fit a curve whose degree of freedom you have to choose. Here $m,c,a,b$ are all replaced with more generic $w$ called as a parameter

If you have $10$ degree of freedom and $10$ data-points you can solve for the parameters of the model (unambiguous solution i.e only one and one unique solution will exist). Whereas , if the degree of freedom is lower you'll get a solution which may miss one point. For, example if you are given 3 points and ask to fit a straight line through it, you may or may not be able to (depending on collinearity). On the opposite case if you have more degrees of freedom you can get multiple values for a single parameter. Assuming a single extra degree of freedom, you can take a parameter hold it to some fixed value and solve the rest of the equation as mentioned above you get some value for the rest of parameters. Now hold the same parameter at a different fixed value and repeat the same process, you get some different value of the other parameters.

In general, it is easier to view it this way:

If let's say you have 3 degrees of freedom $y=w_2x_1^2+w_1x_1+w_0$ and 3 data-points $(x_1,y_1), (x_2,y_2), (x_3,y_3)$, you can get a system of equation:

$$y_1=w_2x_1^2+w_1x_1+w_0$$
$$y_2=w_2x_2^2+w_1x_2+w_0$$
$$y_3=w_2x_3^2+w_1x_3+w_2$$

Thus you get a system of linear equations (NOTE: $x_i,y_i$ are known here). The only unknowns here are $w_i$'s for which there are multiple methods to solve. You can extend this for $n$ equations.


Why is my Soft Actor-Critic's policy and value function losses not converging?

I would say it is the nature of data. Generally speaking, you are trying to predict a random sequence, especially if you use the history data as an input and try to get the future value as an output.


Let's say you have a big box (0.999, 0.999) and a small box (0.001, 0.001).

Euclidean distance = sqrt((0.999 - 0.001)^2 + (0.999 - 0.001)^2) = 1.41.

Now, we look at a small box (0.001, 0.001) and an even smaller box (0.00000001, 0.000000001).

Euclidean distance = sqrt((0.001 - 0.00000001)^2 + (0.001 - 0.000000001)^2) = 0.00141

If you look at the number of the 2nd example, the Euclidean distance
thinks that the second box is pretty similar to the first box. But
this is not true.

IOU solves the problem because it measures the boxes relative to each
other. The result is a percentage, so it would give both boxes a
percentage near 0 which is correct.

  As for the Jaccard Index, why when the denominator equals the numerator, J(b1,b2)=1   ?

  I suppose we still have terms (w1*h1+w2*h2) in the denominator, so how can denominator equals numerator ?

J(b1,b2) = 1 iff intersection/(w1h1 + w2h2 - intersection) = 1 where
intersection = min_of_width * min_of_height

Assume w1h1 = w2h2, then intersection/(2 * w1h1 - intersection). But
intersection is just w1h1, then w1h1/(2 * w1h1 - w1h1) = w1h1/wh1h1 = 1


How to validate that my DQN hyperparameters are the optimal?

If possible, I would try to calculate what the (theoretical) maximum  throughput through the intersection is for a given time interval. If the control behavior that the DQN produces comes empirically close to the maximally possible throughput score, the model is good. Otherwise, you could measure and compare the throughput of different models and choose the best performing one.


Is it still called linear separation with a layer of more than 1 neuron

What you have depicted is a nonlinear classificator. Although each stage does a linear separation, the sequential composition of linear separations is nonlinear. The nonlinearity of the neuron is key in this regard, as otherwise it would all be equivalent to a matrix multiplication, which is linear. You were right about the degree, although it's rarely called like that. People usually describe just the number of layers, and I guess the main reason is that it's not directly equivalent to such polynomials, as that depends on other factors (e.g. the activation function).


How to use a VAE to reconstruct an image starting from an initial image instead of starting from a random vector?

The thing is, the decoder samples from a latent mu and sigma, so you cant sample from a raw image directly. But if you’re trying to put a random image into the encoder of a trained VAE to match it to some sample image (via reconstruction loss), then your random input image will converge to the target sample.

This will work when the following VAE architecture constraints are satisfied:

The target sample is contained in the previously used training distribution.
The parameters of the VAE are frozen after training.
The input image values are “backpropagate-able”. (Interpret the input image as optimizable parameters.)


Did Alphago zero actually beat Alphago 100 games to 0?

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games?

There were 100 different games. You can view some example games between AlphaGo [Lee] and AlphaGo Zero here. They are clearly all different.
This statement in the question shows a misunderstanding:

My understanding of AlphaGo and AlphaGo [Zero] are that they are deterministic, not stochastic.

The Monte Carlo Tree Search (MCTS) algorithm used for look-ahead planning in AlphaGo and Alpha Zero is inherently stochastic. It samples from the huge tree of possibilities in a game like Go by making weighted random choices at all branch points. That means play can progress stochastically with two such agents opposing each other, as many board states will resolve into selecting semi-randomly between "best" moves that would be very closely ranked by each agent in the limit of very long search times.
Whilst this solves the main point of your question, it is worth noting that there can be a related effect in self-play algorithms, even if they are partially stochastic. That is, it is possible to have one agent that develops a specific weakness by chance, that another agent consistently takes advantage of, such that agent A consistently beats agent B, and wins in a very similar fashion each time (maybe deterministically, maybe across a range of different games all with a similar mistake). However it may be the case that also:

Neither agent is strong in general.

Another agent C can beat B consistently, but will lose to A consistently. There would then be no clear way to rank agents A, B, and C without further measurements.

Agents trained through self play therefore do need to be trained and tested against a wide range of opponents to verify this is not happening and that the  skill level assessment is valid more generally. I believe this was done with all the AlphaGo variants built by DeepMind.
The MCTS algorithm does help a little with this scenario as it can correct for weaknesses in how a trained neural network rates early board positions. The look-ahead planning of MCTS makes initial ratings less relevant to eventual action selection - effectively it refines those learned ratings using the samples from current position.


How can a genetic algorithm adapt and get better in a changing environment?

There are a couple of ways of dealing with this. Very often, the approach is just to design your representation and operators in a way to account for the fact that the world changes. The idea is to give the algorithm something that can be used to learn general behaviors or solutions rather than specific ones.
Take an example of learning to steer a race car around a track. You want to represent the state of the world and have the GA learn to select an appropriate action. You might choose to represent the state of the world as a vector of $[x, y, v, a]$, where $(x, y)$ is your location on the track, $v$ is your current velocity vector, and $a$ is an acceleration to apply. The fitness function could return how "good" it was to apply that acceleration. If you do this, your algorithm can probably learn to navigate this track, but a different track will be hopeless, as the locations aren't corresponding to the same turn locations on the new track.
However, what if you encode the world as $[s, v, a]$, where instead of an $(x, y)$ pair representing your current position, you have $s$ as a vector of sensor readings? Is there a wall coming up in front of you or is the track starting to bank? Now, your algorithm can learn to be more general. It doesn't need to be the exact track it learned on, because what it's learning is not to brake at a specific point, but to brake when it detects a wall coming up.
I didn't do a lot of digging into the example you linked, but if you run it for a few years, you can see evidence of this. You see agents that appear to have learned to move in lines horizontally or vertically until they encounter a green square, and then they'll stick around and eat in the patches around that square. That behavior is general, because every environment it encounters has lots of blank space with clusters of green. It's not learning "go to square $(20, 30)$". It's learning to move in a pattern until it finds green and then move around that location.
You can do this in a lot of cases where the specific environment can change, but the objective is the same. There are problems where the actual fitness function changes over time, however. For those problems, there are specific techniques to deal with dynamic fitness functions. Generally, this involves doing something to maintain diversity so that your whole population isn't getting stuck on whatever the current "best" looks like. That's a bit more advanced topic though, and I think your question was really more about the former type of problems.


What is the intuition behind the TD(0) equation with average reward, and how is it derived?

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
\begin{equation}
TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is
\begin{equation}
TD_{\text{error}} = R_{t+1} - \bar R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
where $\bar R_{t+1}$ is estimate of $r(\pi)$.

You can use $\delta_t$ in all 3 updates because neither of these updates depend on each other. For example if you update $\mathbf w$ you don't use that then to update $\mathbf \theta$, or if you update $\bar R$ you don't use updated version to update $\mathbf w$ or $\mathbf \theta$ so you're not introducing additional bias. In each separate update you also don't have $\delta_t$ present multiple times so that you require multiple sampling per timestep to get the unbiased update.

Additionally this is semi-gradient algorithm, it uses bootstrapped estimate $V_{t+1}$ but it doesn't calculate full derivative with respect to it aswell, only with respect to $V_t$ so the algorithm is biased by default, but works well enough in practice for linear case.


This answer describes the "word vector" toolkit in NLP. The result of analyzing a large corpus to find words that occur in similar context provides dense vectors for each word that can then be used for similarity. For bots, the goal is generally a similarity and not exact synonyms. Synonyms can be hard-coded using WordNet if needed. For your greeting question, following blog post can help: Do-it-yourself NLP for bot developers.


What is a simple game for validation of MCTS?

A good choice might be smaller-scale games of Go, like a 9x9 board. This was the original application domain MCTS was designed for, and the original paper by Brugmann from 1993 details parameters that should lead to an agent that can play above beginner level in what is today a minuscule amount of computational time, in a scaled-down 9x9 grid.

Go is a good choice for a benchmark because most learning algorithms fail at it pretty badly. The fact that MCTS worked here was a major breakthrough at the time, and helped cement it as a technique for game playing. If your algorithm is not working properly, it is therefore unlikely that it can learn to play Go at the level described in Brugmann's paper.


Understanding arrangement of applying filters to input channels

Ok, here's the break down:

The depth of an input to a convolutional layer is termed channels. The depth of a convolutional layer is the number of kernels (aka filters). The depth of a kernel is equal to the number of channels in the input.

See below:

The input (of 7x7, pad of 1) has 3 channels. The convolutional layer has 2 kernels (or filters). Each filter has a depth of 3, equal to the number of channels in the input. Using the notation you used in your question:

Filter 1, channel 1 to input channel 1
Filter 1, channel 2 to input channel 2
Filter 1, channel 3 to input channel 3
Sum all three channels of filter 1, then add bias
Filter 2, channel 1 to input channel 1
Filter 2, channel 2 to input channel 2
Filter 2, channel 3 to input channel 3
Sum all three channels of filter 2, then add bias

These steps are repeated for each frame the filter slides over the input image.

To answer question 2, if the output is 128, that simply means there are 128 filters. There could be an infinite number of filters if you so choose.

EDIT:

Here's the link to the interactive graphic: http://cs231n.github.io/convolutional-networks/


How should I define an MDP for this problem where we need to guess a number and minimise the number of guesses?

When formulating a problem into an MDP you need to define the states, of the system, the possible actions you can take, the transition probabilities between states depending on the action and the rewards earned (or costs paid) for the state transitions. The important part here is to create a state-space that has a Markov property which in plain words means that in any given state you have enough information to make the optimal decision.

Rewards: in this case, I think it is fairly obvious that a positive reward should be given if we find out the number and zero reward if we don't.
Actions: the actions we can take is making a guess, that is the possible actions will come from the action-space A=(1,2,3).
States: this is a bit more difficult to come up with. Here the intuition should come from thinking about the actions you can make and how they change the information you have about the system. In our state-space, each state will represent the set of numbers that the answer might possibly be. For example in state the (1,3) we know that the guessed number is either 1 or 3. Also, we will denote these states with capital letters to simplify the state transitions.

A=(1, 2, 3): the initial state as at the beginning of the search the guessed number can be either of 1, 2 or 3;
B=(1, 2): the guessed number is either 1 or 2;
C=(1, 3): the guessed number is either 1 or 3;
D=(2, 3): the guessed number is either 2 or 3;
E=(1): the guessed number can only be 1;
F=(2): the guessed number can only be 2;
G=(3): the guessed number can only be 3;
H=(): final state, we found the guessed number.

State transitions: this is very simple, just imagine which states are reachable in a state depending on the guess (action) we make. Here the notation P(A|B,1) will mean the probability that we reach state A from state B given that we guessed 1. (Note: we will assume uniform distribution for the guessed number). I think I'll be lazy here and not write down the all the transitions as they get very repetitive once you understand how they are made, I'll just provide examples for all cases.

P(D|A,1)=2/3: we guess 1 in the initial state and we miss with 2/3 chance, the reply will be that the guessed number is greater than 1.
P(H|A,1)=1/3: we guessed right, so we reach the final state.
P(B|A,3)=2/3: similarly we guess 3 and miss.
P(H|A,3)=1/3: we guess 3 and we are right.
P(C|A,2)=0: if our guess 2 is wrong we get a reply of whether the guessed number is greater or smaller than 2 therefore we won't get into this state.
P(E|A,2)=1/3: we guess 2 and the reply is that the number is smaller than 2.
P(G|A,2)=1/3
P(H|A,2)=1/3
P(B|B,3)=1: guessing 3 in state B doesn't provide more information so we reach the same state with probability 1.
P(H|B,1)=1/2: 1 is the right guess with 1/2 probability in this state.
P(F|B,1)=1/2: similarly we miss with 1/2 chance if we guess 1 so we reach state F where we know what is the good solution however we haven't won yet as we still need to make one more guess to reach the final state.
P(F|F,1)=1: again guessing 1 in state F doesn't give us extra information so we get back to state F.
P(H|F,2)=1: however, guessing 2 in state F will give us the final state, as it is the right guess.

Note that I defined 8 states earlier, however, state C=(1,3) is never reached so we don't actually need it.

I hope this helped and you will be able to finish the rest.


Why are activation functions independent layers in CNNs rather than part of convolutional layers?

These are just two equivalent interpretations (or illustrations) of the application of an activation function. In other words, in a multi-layer perceptron (MLP), you could also illustrate the application of the activation function as a separate layer that follows a linear combination layer. However, in the context of MLPs, the math is relatively simple and elegant, so a fully-connected layer of an MLP can simply be represented as follows

$$
\sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right)
$$

where $\sigma$ is some activation function and $\mathbf{W} \mathbf{X}$ is the linear combination of the inputs, $\mathbf{X}$, and the weights, $\mathbf{W}$, and $\mathbf{b}$ is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a composite (or nested) function.

In the context of convolutional neural networks (CNNs), people might illustrate the application of the activation function as a separate layer because the application of an activation function to the result of the convolution operation of the CNN is optional and out-of-favor (as stated in this article http://cs231n.github.io/convolutional-networks), as opposed to the case of MLPs, where activations functions usually follow the linear combination. However, note that the last layers of a CNN are usually fully-connected layers (and not convolutional or pooling layers), that is, they are a linear combination of their input and their weights followed by an application of an activation function.


How can I have a computer learn the equation with known dependent variables?

You might want to try symbolic regression. This is a machine learning approach that tries to generate an equation with arbitrary form that best fits a set of examples. I've used it before when working with physicists who wanted an equation in terms of known numeric constants and specific variables, but didn't expect the equation to be linear or one of the other shapes for which we have standard regression models.

Symbolic regression often (usually?) uses genetic programming for the underlying optimization. A decent tool is Eureka. It used to be free, but I think they want a payment now. You can also do it yourself from any genetic programming toolkit with a little know-how.


How do we make our outputs to have the same size as the true mask?

You can create a mapping from classes to colors for a simple one is:
y # y.shape = (W, H, n_classes)
_, y_color = y.max(dim=-1, keepdim=True) / n_classes # y_color.shape = (W, H, 1)
y_color = torch.cat([y_color] * 3, dim=-1) # y_color.shape = (W, H, 3)

(using pytorch like code)
This mapping is visualizable, of course, you may get nicer visualizations if your class to color mapping is more sophisticated.


What is a neural network?
Many neural networks can be defined as a function $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$, where $n, m \geq 1$.
Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) $f^i$ that receive some input and produce output, i.e. $f^i(\mathbf{x}^i) = y^i$, where $\mathbf{x}^i \in \mathbb{R}^k$. The actual function $f^i$ is variable and depends on the application or problem you want to solve. For example, $f^i$ can just be a linear combination of the inputs, i.e. $f^i(\mathbf{x}^i) = \sum_{j} \mathbf{w}_j^i \mathbf{x}_{j}^i$, where $\mathbf{w}^i \in \mathbb{R}^k$ is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.
If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).
What are the most common feedforward neural networks?
Perceptron
The simplest (non-binary) FFNN is the perceptron, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.
Before the perceptron, McCulloch and Pitts had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.
Multi-layer perceptron
An FFNN with more layers (of units) between the input and the output is often called a multi-layer perceptron (MLP). The layers in the middle are often denoted as hidden layers. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.
Convolution neural network
You can have other forms of FFNNs that perform other operations.
For example, a convolution neural network (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the convolution operation (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)
However, note that CNNs can also have recurrent connections, but this is not usually the case.
Residual neural network
There are also residual neural networks, i.e. neural networks where a node in a certain layer $l$ can be connected to other nodes in layers $l+j$, for $j \geq 1$, as opposed to being connected only to the nodes in layer $l+1$, which is the typical case.
Auto-encoders
Auto-encoders are neural networks that compress the input and then decompress it. The answers to this question may help you to understand why AEs would be useful.
What kind of inputs do they receive?

What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?

In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).
How can you further classify the NNs?
Based on chapter 2 of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas, you can also divide neural networks into other categories

Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)
Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)
Neural networks that store states vs NNs that don't store states

You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.


Can AI come up with scientific theories of past when provided with sufficient data available at that time?

Yes.

Some good examples of this are Lipson's work using evolutionary models, and Wu & Tegmark's work on a theory-based life-long learner, and Iten et al.'s work with deep neural networks. There are many, many, other research papers in this area, and there is a lot more work that is ongoing. The endgame for a lot of this work is the hope that we can synthesize new theories automatically.


Being confused of distribution notations in Deep Learning book

At page 130 of the same book, the author states that $\hat{p}_\text{data}$ is an empirical distribution defined by the training data. Similarly, at page 129, he states that $p_\text{data}$ is the true distribution that generates the set $\mathbb{X} = \{ \boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)} \}$.

What is the difference between $\hat{p}_\text{data}$ and $p_\text{data}$? You can think of $\hat{p}_\text{data}$ as a histogram that is calculated from the set $\mathbb{X}$ and $p_\text{data}$ as the true distribution from which the elements in $\mathbb{X}$ are drawn.

The subscript ${\boldsymbol{x}, y \sim \hat{p}_\text{data}}$ in the expectation $\mathbb{E}_{\boldsymbol{x}, y \sim \hat{p}_\text{data}}$ indicates that the expectation is taken with respect to the samples drawn from the empirical distribution $\hat{p}_\text{data}$. In other words, you will optimise the objective function $J$ using the training data. Have a look at this question for more info.

The subscript ${\boldsymbol{x}, y \sim p_\text{data}}$ in the expectation of formula $5.101$ is a typo. In fact, in this online version of the book, at page 151, the subscript of the expectation is  ${\boldsymbol{x}, y \sim \hat{p}_\text{data}}$.


Can a multi-camera system be used for localization, mapping, and obstacle detection in self-driving cars to within 10 cm of accuracy? Whither lidar?

It seems that LIDAR presents a problem for resolving the car's environment at higher speed. While I'm not too familiar with the dynamics of LIDAR I do know that it's a physical system that relies on sending and receiving laser pulses to various points around the car by way of rotating mirrors. As speeds increase, it seems different arrangements of mirrors and light collectors might have to be used to maintain a high-resolution image. There's some evidence that Doppler LIDAR (developed in the 1990s) became less accurate with higher velocities. However, LIDAR is partly preferred over radar because of its higher accuracy even when tracking objects at high speeds - this is why LIDAR guns are increasingly being used by police instead of radar guns to track speeding vehicles. It seems natural that a set of high-resolution cameras paired with a well-trained neural network would not be subject to the same physical limitations as LIDAR.

I think that an important intuition to consider is that while LIDAR is used to generate clouds of data points whose shapes and patterns can be analyzed by autonomous car software, cameras can pick up non-topograhical features such as road lines, the content of roadsigns, and additional location context such as storefronts and intersection layouts. Considering that these cameras can use pattern recognition and stereoscopy to also generate a 3D topographic map of the environment, it seems plausible that Level 5 self driving cars would not require LIDAR.

Here's an interesting look at the problem.


What would be the best approach to teach an AI to learn how to "sing" along a beat?

You have to define a function between voice and a corresponding sound, but there is no a best way to do it, it depends on what kind od output you care, for example you could create an algorithm that receives music and siging as separate "files" and then that algorithm finds the relation between them now you can give that algorithm a new music only file and it will create the singing part with the function of the previous music-singing "system".
This is sing this music as the other one was sang.

At least that is the way i would do it.


Why Monte Carlo epsilon-soft approach cannot compute $\max Q(s,a)$?

If $\pi$ is a random policy, and after running through this algorithm, and for each state take the $\max Q(s,a)$ for all possible actions, why would that not be equal to $Q_{\pi^*}(s, a)$ (optimal Q function)?

Assuming that the estimates for $Q_{\pi}(s,a)$ have converged to close to correct values from many samples, then a policy based on $\pi'(s) = \text{argmax}_a Q_{\pi}(s,a)$ is not guaranteed to be an optimal policy unless the policy $\pi$ being measured is already the optimal policy.

This is because the action value $Q_{\pi}(s,a)$ gives the expected future reward from taking action $a$ in state $s$, and from that point on following the policy $\pi$. The function does not, by itself, adapt to the idea that you might change other action choices as well. It is a measure of immediate differences between action choices at any given time step. Therefore if there are any long-term dependencies where your action choice at $t$ would be different if only you could guarantee a certain choice at $t+1$ or later, this cannot be resolved by simply taking the maximum $Q_{\pi}(s,a)$ when $\pi$ was a simple random policy.

However, if you do decide to change the policy such that you always follow actions $\pi'(s) = \text{argmax}_a Q_{\pi}(s,a)$ for all states, then you can say this: For each state, $V_{\pi'}(s) \ge V_{\pi}(s)$. I.e.  $\pi'(s)$ is no worse than, and may be a strict improvement over $\pi(s)$. Better than that, $\pi'(s)$ will be a strict improvment over $\pi(s)$ if $Q_{\pi}(s,a)$ is accurate and $\pi(s)$ is not already the optimal policy $\pi^*(s)$. This is the basis for the Policy Improvement Theorem which shows that if you repeat the process of measuring $Q_{\pi^k}$ and then creating a new policy $\pi^{k+1}(s) = \text{argmax}_a Q_{\pi^k}(s,a)$ that you will eventually find the optimal policy. You only have to repeat your idea many times to eventually find $\pi^*$

The Dynamic Programming technique Policy Iteration does this exactly. All other value-based Reinforcement Learning methods are variations of this idea and rely at least in part on this proof.


How to determine the target value when using ReLU as activation function?

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.

The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability?  For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.

You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.

See this question How to choose cross-entropy loss in TensorFlow? on Stack Overflow for more info about different cross-entropy functions.

By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the mean squared error).


Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

It is not so much the problem of using Reinforcement Learning to train the neural networks, it is the assumptions made about the data given to standard Neural Networks. They are not capable of handling strongly correlated data which is one of the motivations for introducing Recurrent Neural Networks, as they can handle this correlated data well.


Why is MSE used over other quadratic loss functions?

I can comment on several properties of MSE and related losses.

As you mentioned MSE (aka $l_2$-loss) is convex which is a great property in optimization in which one can find a single global optimum. MSE is used in linear and non-linear least squares problems which form the basis of many widely used statistical methods. I would imagine the math and implementation would be more difficult if one would use a higher-order loss (e.g. $x^3$) and that would also prove to be futile because MSE already possesses great statistical and optimization properties on its own.
Another important aspect, one wouldn't use higher-order loss functions in regression is because it would be extremely prone to outliers. MSE on its own would weigh the outliers much more than l1-loss would! And in real world data there is always noise and outliers present. In comparison l1 loss is more difficult in optimization, one reason for which is it's not differentiable at zero.

Other interesting losses you might want to read about are $l_0$ and $l_{inf}$ loss, all of which have their own trade-offs in optimization-sense.


Which methods or algorithms to develop a learning application?

even though that's not really AI, the easiest way to do that would probably be to put coefficients on each question

e.g. your question would have something like

grammar=0.55
vocabulary=0.45
if(won){
  success=-1
}else{
  success=1
}

the lower the level, the less questions of this type will appear

at the end of the level, you sum each question times how much did the player succeed

player.level is a real number array

for each question{
   player.level["grammar"]=player.level["grammar"]+question.grammar*question.success
   player.level["vocabulary"]=player.level["vocabulary"]+question.vocabulary*question.success
}

then when you initialize the level, generate the questions according to the player's level
choice is a random real number between 0 and sum of all player.level (offseted to be above 0)

for each player.level{
   if choice>player.level{
      choice=choice-player.level
   }else{
      return question_type=player.level.type
   }
}

though a "true" AI can make the same or better choices of questions, it will probably be slower, and require more data than a simple algorithm like this one.


Computers (the processor in combination with the memory) are designed to be deterministic.  Otherwise no software would ever work, because the computer would be executing it randomly.

The computer's outcomes are not determined by the chaotic movements of electrons, but by the much strong deterministic currents and voltages between the transistors.  A digital machine is, simply speaking, not affected by noise.  The noise is always present,  but 0.13 volts or whatever gets rounded to 0 V (logic 0) and 3.02 volts or whatever gets rounded to 3.3 V (logic 1).

It is however possible to take advantage of the chaotic movement of electrons as well.  This is done in special hardware called "hardware random number generators", (some use different forms of true randomness such as nuclear decays).


Understanding the math behind using maximum likelihood for linear regression

Note first that the first $=$ (equals) in $\frac{dl(\theta)}{d\theta}
 = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$ should be interpreted as a "is set to", that is, we set $\frac{dl(\theta)}{d\theta}
 = 0$. Given that (apparently) $\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$, $\frac{dl(\theta)}{d\theta}
 = 0$ is equivalent to $0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$.

Now, let's apply some basic linear algebra:

\begin{align}
0 &= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\
0 &= −(0−2X^TY + X^TX \theta) \iff \\
0 &= −0 + 2X^TY - X^TX \theta) \iff \\
0 &= 2X^TY - X^TX \theta \iff \\
X^TX \theta &= 2X^TY \iff \\
(X^TX)^{-1}(X^TX) \theta &= (X^TX)^{-1}2X^TY \iff \\
\theta &= (X^TX)^{-1}2X^TY
\end{align}

Now, you can ignore the $2$, because it is just a constant, and, when optimizing, this does not influence the result.

Note that using $\hat{\theta}$ instead of $\theta$ is just to indicate that what we will get is an "estimate" of the real $\theta$, because of round off errors during the computations, etc.


Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are:

strategies encoder-decoder (autoencoders) where the encoder typically  has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes.

bidirectional recurrent networks where in the forward direction number nodes decreases and in the backward increases.

generators, that from a random vector generates, by example, a full image.

As general rule: decrease number of nodes forces the net to filter/resume/abstract/summarize the internal signal information (discarding useless information or noise) while increase number of nodes means apply current information to generate an answer value for a specific question/target.
Allow me a strongly simplified example: assume you want a system that, from a photo of an animal, answers the questions: number of legs? has beak ? flies ? . Net inputs are images of birds and dogs.
The net architecture can have layers of decreasing size until a single node that will decide "is bird or dog ?". From this single item of information (the only one need to answer all the questions) the output layer will have 3 nodes, each one answering one of the specific target questions: number or legs ? 4 if dog, 2 if bird, etc .