Are training sequences for LMs sampled in an IID fashion?

Language models explicitly assume that word sequences are not independent and identically distributed (iid). A word-based model that assumed iid within each sequence could only predict probabilities of words according to some other context than surrounding words, which would not be very useful for a language model.
The training processes for statistical models often require iid input/output pairs when sampling e.g. minibatches. Neural networks definitely train better when datasets are shuffled to remove any chance of correlation between examples as they are used to update parameters.
How to resolve these two different needs? When training a sequence-based model on many sequences, it is the distribution of sequences that needs to be iid and representative of the overall population. The distribution of items within the sequence is what is being learned, so should not be obscured or removed.
As an analogy, you do not usually want to shuffle the rows of pixels in an image when training image classifiers. The spatial pattern in the image needs to be preserved in the same way that the sequential pattern in a sentence needs to be preserved, because the pattern is part of the data being modelled. With image data, you accept non-iid relationships between the pixels that are next to each other within a single image, then apply shuffling and stratifying algorithms at the level of individual images.

If so, are there any papers that try and deal with this?

There may be some original papers from 50 years ago which compare iid with non-iid data when training sequences fed into RNNs, but it has been a standard part of engineering practice for decades now to shuffle datasets, and the separate sequences used to train RNNs are no different.
From your comments:

However the sequences that are sampled for training are not iid if they are multiple sequences generated sequentially from the same document, which from what I understand happens often?

You are correct, the raw set of sequences is not iid if they are collected in that fashion. However, the dataset is always shuffled or resampled for training purposes, it is not fed into training routines in that raw state. The shuffling (of selected sequences which are kept intact internally) happens in-between the raw data collection and the training.

There are some simple statistical models that do not require iid data to train on. This occurs for example in tabular reinforcement learning, which can learn online from a single continuous sequence of states, actions and rewards. An equivalent language model would be word- or letter-based ngrams.


Would it be ethical to use AI to determine a user’s gender from the content they upload, without them knowing?

Ethics aside, I think this could potentially create a number of different issues. What if your algorithm guesses wrong and I'm stuck with a UI that's targeting incorrectly? What if my wife and I share an account? What if I'm a (insert orientation here) (insert gender here) who [coaches, supports] a [men's, women's] [volleyball, football] team? People are very diverse, and you could make many mistakes. Would you then provide tools for the user to correct these mistakes? How would you be able to do so without the user being offended?

Instead, I see many fewer issues that might arise from a section like the following:

  Let's get to know each other!

  If you'd like, we can help tune your profile to match your interests.

  You can start by selecting some of your interests from the list below, or try searching for your own.

  Fishing Cooking Soccer Video Gaming Social Media
  Search for my interest...

  Skip this section →

You could continue with other questions that might actually be relevant to your shaping your UI or application, allowing the user to omit/delete details for any or all questions.

Being purely an opt-in experience prevents any issues with the user feeling like your app may be "talking about them" behind their back.

In this case, specifically at this point in time, I think it's wise to be transparent about what data your application knows (or thinks it knows) about your users.


The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that the current state $s$ is known. However, in POMDP, at each time step, the agent does not know the current state, but it maintains a "belief" (which, mathematically, is represented as a probability distribution) in what the current state might be, so it cannot maintain (an approximation of) the function $Q(s, a)$. Hence, the usual Q-learning and SARSA algorithms shouldn't be directly applicable to a POMDP.

However, $Q$-learning is often used in the contexts where observations emitted by the environment (or transformations of the raw observations) are used to build the current state (which is assumed to be Markov, even if it is not). For example, in the original DQN, the action taken at the current step and the raw observation and the reward emitted by the environment (after this action is taken) are combined to produce the current (Markov) state. It might not be the case that the way they combine the action, the reward and the observation is sufficient to fully describe the current state (which might not even be Markov).

In this report, Deep Reinforcement Learning with POMDPs, the author attempts to use Q-learning in a POMDP setting. He suggests to represent a function, either $Q(b, a)$ or $Q(h, a)$, where $b$ is the "belief" over the states and $h$ the history of previously executed actions, using neural networks. So, the resulting parameterized functions would be denoted by $Q(b, a; \theta)$ or $Q(h, a; \theta)$, where $\theta$ is a vector representing the parameters of the corresponding neural network. Essentially, the author uses a DQN (with an experience replay buffer and target network), but the results are not great: $Q$ values converge, but policies do not and they are not robust (in that they are sensitive to small perturbations).


"Porpoising" in latter stages of validation loss and MSE charts in Keras

What might be causing these, and how to eliminate (or greatly reduce) them?

It is difficult to be sure just from the graph, but I note you are using the Adam optimiser. It shares with a few other optimisers (most notably RMSProp) that it divides current gradients by a rolling mean of recent gradients to set step sizes. This can cause some minor instability when gradients get close to zero for a long while before growing again. This might happen at a saddle point where only some fraction of the network parameters are critical to results for a few iterations before hitting some other direction of slope where changing the "quiet" parameters becomes important again. This is more likely to occur as loss values become small, and close to perfect convergence.
There are a couple of hyperparameters in Adam that may reduce the effect:

Reduce learning rate. The default learning rate in Adam is often set to 0.001, but you will find a lot of researchers will reduce that to magnitudes around 0.0001 or 0.00001

Increase epsilon. This is effectively the minimum rolling average gradient. The default is 1e-7 but it sometimes needs to be increased significantly, it depends on the loss surface. The official documentation suggests that it may even be useful to increase it up to 1.0


How to perform gradient checking in a neural network with batch normalization?

You should be able to do gradient checking as long as you fix the randomness by fixing the random seed, on python you might want to look at numpy.random.seed.

From http://cs231n.github.io/neural-networks-3/#ensemble :

  When performing gradient check, remember to turn off any non-deterministic effects in the network, such as dropout, random data augmentations, etc. Otherwise these can clearly introduce huge errors when estimating the numerical gradient. The downside of turning off these effects is that you wouldn’t be gradient checking them (e.g. it might be that dropout isn’t backpropagated correctly). Therefore, a better solution might be to force a particular random seed before evaluating both (f(x+h)) and (f(x-h)), and when evaluating the analytic gradient.


Method to check goodness of combinatorial optimization algorithm implementation

This is a very large question that could be answered in a variety of ways depending on the context.

For some optimization problems operating under specific conditions you can make theoretical guarantees that your optimization will solve your problem.  A specific example of this is running the gradient descent algorithm on a convex function.  If the function being optimized is convex then gradient descent is guaranteed to converge to the correct solution.  Given the nice properties of convex functions, there are many types of optimization where you can make these theoretical guarantees.

When evolutionary algorithms (and many non-derivative based optimizations) are run, it's much harder to make these types of guarantees.  Often in the literature I've seen people try their functions on several baseline optimization functions and report back the minimum or maximum found.  Many of these are included in Facebook's nevergrad optimization library to be used for evaluation.  An example of one of these functions is included below,

Even if you can prove that your algorithm minimizes several objective functions you could be testing on, there could be other properties that should be considered.  Some of these include the following: the speed at which the algorithm converges; the number of evaluations of the objective function (if not closed form), and even the amount of space in memory which is required for your optimization.  All of these (and many more) factors could come into play when evaluating your algorithm.

Scientists may use functions like this one to evaluate their optimization and compare it to other algorithms that are similar.  To answer your final question though, "How can I see if my optimization works for every problem", I would say that's a bit of a loaded question.  Most of the time these optimizations need some sort of constraint to make a generalized guarantee.  I would look at the constraints of your problem and go from there.

If you're interested, here's a good paper that goes much more in depth than I did here.


This is sort of a tricky question to answer, by results you could mean 2 possible things. History, which is a culmination of actions, observations and rewards. Or the value func/policy which is the thing you are trying to improve.

With that being said, you for most instances of RL will only be dealing with the last few events in the history, or sampling uniform randomly from it according to your route of exploration. So I don't think this is what you are talking about.

As far as storing your value func/policy OpenAI offers a way to post and to a certain extent store your result. This Readme will show you how to do so.

Edit: OpenAI does not offer anything to store experience out of the box, as I noted in my comment you could pretty easily whip up some code that writes JSON to some file and saving that would be a store of experience. Although, it looks like you already found something that does this in tensorflow.

OpenAI likely hasn't included it up to this point because they don't put a ton into this framework. I would encourage you check out SerpentAI on Github, as it has a lot of features missing in OpenAI.

Hope I answered your question!


Does L1/L2 Regularization help reach an optimum result faster?

I am not aware of any empirical results regarding this question.
But in theory, adding a regularization term shall make the learning task actually even harder, since there is suddenly a second loss term that the network has to be optimized for, which is not even directly related to achieving the original task of fitting the model to the data.
It is true that the regularization term will try drive as many of the weights towards low values as it can. But, at the same time, the other loss term (computed on the original optimization criterion) will try to drive many of the weights to larger values unequal 0 in order to achieve the original training task. (If that wasn't the case, you would be good to go without regularization in the first place.)
Having these competing interests shall then make achieving the original task harder, which I would in theory expect to be more time consuming than, alternatively, allowing the model overfit until a certain admissible error is reached.
After all, the goal of regularization is to improve generalization of a learned model (i.e. to prevent overfitting, as you said), which is about the quality of the outcome and not about how quickly you get to the desired outcome.


"Vocal captcha" for robots on the phone?

your question is a very similar to "turing-test". you could narrate a simple story and ask questions based on that , considering the state-of-art algorithms in "question-answering" are still far beyond human skills.


Getting better results in improving the configuration

Try using an RMSProp optimizer instead of Adam optimizer. Also try decreasing the batch size and keep a small learning rate like 0.001.


How is exponential moving average computed in deep Q networks?

However, looking at DQN, how exactly is the exponential moving average implemented in deep networks?

It is not implemented directly as exponential moving average.
Instead, the ability of neural networks to learn online and incrementally forget older input/output associations is used to achieve the same goal.
If you use the simplest mini-batch stochastic gradient descent methods - i.e. just a simple gradient step $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}}\sum_i(g_i - \hat{q}_i)^2$ where $g_i$ is measured (or bootstrap estimated) discounted return for a single state/action pair and $\hat{q}_i$ is the current estimate, then the learning rate $\alpha$ is analagous to the same factor in exponential moving average approach, and in fact would be the same thing mathematically if you one-hot-encoded the states and only had a single layer in the neural network.
Typical implementations of DQN will have deeper networks, will not one-hot-encode the entire state space, and will typically use some gradient modifier such as momentum or Adam to improve performance. So the match to exponential moving average is not exact. But the behaviour is similar in the most important aspect for RL - the ability to learn online and forget older values as the target distribution of expected returns changes due to changes in policy.


How to combine several chatbots into one?

Define features that describe C1 and C2 (like their ids) and add these features to inputs of your model. Thus your only model will be able to generate the next line for any pair of (C1, C2).


The networks in NEAT are still implicitly layered. There are neurons that need to be evaluated before other neurons can be evaluated and so this gives us our layers.

If you don't know the structure of your network then you can use Kahn's algorithm to find an arbitrary (by arbitrary I just mean one of the possible partially ordered sets) ordering of the nodes in the network. Then you evaluate your neurons in the order given to you by Kahn's algorithm. This works because the ordering of your network (which is a directed acyclic graph) is a partially ordered set.


Understanding the derivation of the first-order model-agnostic meta-learning

$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$ in a similar way that $\frac{d f}{dx} = 1$ for $f(x) = x$. Strictly speaking, $\mathbf{I}$ should be a vector of $1s$ with the same dimensionality as $\theta_{i-1}$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $\theta_{i-1}^j$ with respect to $\theta_{i-1}^k$, for all $k$, for all $j$, which will make up an identity matrix.

Regarding your first question, $\nabla_{\theta} \theta_{0}$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $\nabla_{\theta} \theta_{0}$ actually means $\nabla_{\theta_0} \theta_{0}$. I would need to dive into it.


Dropout causes too much noise for network to train

why dropout of 0.5 effectively kills the networks ability to train.

because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25%  remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network.
Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers.  And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.


Neural network architecture for author name as an input?

The most straightforward approach I would recommend would be the one-hot encoding solution without a feature for ''other author''. If you use drop-out during training, the network should learn how to deal with input vectors that don't have any author-features set to 1. Then, whenever you have an unknown author, you simply have no 1s at all for any of the author-features, and it should still have learned how to deal with that.

Another possible approach would be a one-hot encoding with a feature for ''other author''. In order to still get training data for that weight, you could simply use data augmentation. The most straightforward data augmentation approach for this would be to create copies of the instances in your training data, but for those copies set the ''other author'' feature to 1 instead of the actual author.

The most complex solution I can think of would be taking the string representation of the author, trying to have your program find information about that author online (for example try to look up a wikipedia page), push all that text through an LSTM. The ''output layer'' of the LSTM can then be merged with your other features (those two together will be your ''input vector''), stack a few more layers on top of that, and train the entire thing end-to-end. Unless the competition with your friend involves a serious amount of money, this is probably going to be too complicated to be worth it though.


How close have we come to passing the Turing Test?

No one has attempted to make a system that could pass a serious Turing test. All the systems that are claimed to have "passed" Turing tests have done so with low success rates simulating "special" people.  Even relatively sophisticated systems like Siri and learning systems like Cleverbot are trivially stumped.

To pass a real Turing test, you would both have to create a human-level AGI and equip it with the specialized ability to deceive people about itself convincingly (of course, that might come automatically with the human-level AGI).  We don't really know how to create a human-level AGI and available hardware appears to be orders of magnitude short of what is required.  Even if we were to develop the AGI, it wouldn't necessarily be useful to enable/equip/motivate? it to have the deception abilities required for the Turing test.


Why isn't the ElliotSig activation function widely used?

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.

ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions,

  the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.

Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.

Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.

As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.


How can I predict the true label for data with incomplete features based on the trained model with data with more features?

Assuming that you have access to the training data set, you could use an autoencoder network to predict what features f4, f5, f6 'could be' for the test data set. The way to do this is to train the autoencoder on the training data set with features f1, f2, f3 as inputs, and then use f1,f2,f3,f4,f5,f6 as the output of the network. The autoencoder then effectively learns to map any input samples with (f1,f2,f3) to (f1,f2,f3,f4,f5,f6). By passing your test data through the autoencoder, you can then use the output and pass it to your model.


What is the reason for mode collapse in GAN as opposed to WGAN?

I don't have a definite answer, but only a suspicion/idea:
Looking at Figure 1 from the WGAN paper, we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable at $0$. However, the EM plot on the left is continuous also at $0$. You could now argue that we have a kink there, so it should not be differentiable there either, but they might have a different notion of differentiability, I am honestly not sure about it right now.


Confusion between function learned and the underlying distribution

You're right! The generative model $f$ is not the same as the probability density  (p.d.f.) function $p_{data}$. The kind of phrases you've referred to are to be interpreted informally. You learn $f$ with the hope that sampling a latent vector $z$
from some known distribution (from which it is easy to sample), results in $f(z)$ that has the probability density function $p_{data}$. However, merely learning $f$ does not give you the power to estimate what $p_{data}(x)$ is for some image $x$. Learning $f$ only gives you the power to sample according to $p_{data}(\cdot)$ (if you've learned an accurate such $f$).


Which parsing algorithm can I use for NLP question answering system?

I have been reading and reading, and found answers to almost all my questions.
I am sticking to Earley algorithm, given that it offers a dynamic programming approach (CKY does the same). Both algorithms are chart parsing algorithms.
Earley is a context-free, top-down parsing algorithm, which makes it a goal-driven algorithm. From start symbol down. Furthermore, it is more efficient than the CKY algorithm.
Here are the slides that provide more info.


Implementing experience replay in reinforcement learning

Gradient descent should be performed using the sum (or average) of the losses in the minibatch.

This is in fact also how I read the pseudocode in your question, though I understand it can be confusing. Note that, in the pseudocode, j is not specified in detail. They do not, for example, have j ranging from 0 to the size of the minibatch.

When they say:

Sample random minibatch of transitions (phi_j, a_j, r_j, phi_{j+1}) from D

they mean multiple transitions in the minibatch (with minibatch size of 1 being a special case), and they use the index j to collectively refer to the entire set of indices in that randomly sampled minibatch. It's not one particular number / index, j is a set of indices. When further lines of code do something with index j, they actually do something with all indices j.


Python doesn't require an IDE. You just need to install the Python interpreter to run a Python program.

Running machine learning on another computer is in fact industry standard. Google, Amazon and Azure can provide you a virtual machine, which you can run anything you want.

I personally rent a powerful machine with GPUs on Google to do my machine learning analysis. It's not free but very powerful.


What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

The correct number of child processes will depend on the hardware available to you.

Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running.

If your problem fits nicely in your computers' memory, then processes will spend almost all of their time running. If it's too big for memory, they will periodically need to wait for disk.

You should use approximately 1 Child process per CPU core. If you are training on a GPU, then it depends whether the process can make use of the entire GPU at once (in which case, use just 1), or whether a "process" is really more like a CUDA thread here (in which case you'd want one per CUDA core).

If you think your processes will wait for disk, use more than one per core. About 50% more is a good starting point. You can use a program like top to monitor CPU usage and adjust the number of processes accordingly.

To answer your question more explicitly:

Having more child processes (up to a point, discussed above), will increase hardware utilization, and make your training run faster. With a Core i7 CPU for instance, you might be able to run 8 or 16 child processes at a time, so you'd train 8-16x times faster.
Having more child processes than processing units (CPU cores, CUDA cores), will begin to cause frequent context switching, where the processing units have to pause to change between different jobs. Changing jobs is extremely expensive, and ultimately, your program cannot train faster than it would by using all the available hardware. If you have more processes than processing units, reducing the number should make your program train faster.


Are there any artificial intelligence systems with an internal dialectical approach and multiple minds which develop within a community of nodes?

There are a lot of systems that follow the ancient maxim: "Always two there are; no more, no less. A master and an apprentice."
In reinforcement learning a class of such setups is called Actor-Critic-Method. There you have a master, whose duty it is to create feedback for the actions of the apprentice, who acts in a given environment. This would be comparable to how a human learns some physical activity, like playing table tennis. You basically let your body do its thing, but your consciousness evaluates how good the result is.
The setup of AlphaGo might be even closer to Kahnemann's system 1 and system 2. AlphaGo has two neural networks which provide actions and evaluations (system 1, fast, intuitive, etc.) and the Monte Carlo tree search, which uses these actions and evaluations to prune a search tree and make a decision (system 2, deliberate, logical).
In the end, this kind of structure will pop up again and again because it is often necessary to do some kind of classification or preprocessing on the raw data before your algorithm can be run on it. You could frame the whole history of gofai as the story of how scientists thought system 1 should be easy and system 2 should be doable in a few decades, where the reality is that we have no idea how difficult system 2 is because it turned out that system 1 is extremely difficult.


Q table not converging for an arbitrary experiment

If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit.

The current reward structure provides the incentive to just move to a stage where it gets a reward.

For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a higher inverse value.

To make the agent learn to move to state 2, you would have to provide it with more incentives to go to state 2.

def reward(s,a,s_dash):
    if s_dash == 2:
        return 5
    elif inverse_values_for_states[s]<inverse_values_for_states[s_dash]:
        return 1
    elif inverse_values_for_states[s]>inverse_values_for_states[s_dash]:
        return -1
    else:
        return 0

I tried using this and it converges to 2. This is a hard-coded version, but I guess you get the idea.


What are non-held-out data or non-held-out classes?

Held-out simply means "not included" particularly in the sense of:

This part of the data was not included in this specific training run.

Depending on the context of all of these text non-held-out data/classes means the data that actually was included in a particular modeling exercise.

Consider this excerpt from your first example:

  For instance, Owen and Perry (2009) show a method for holding out data, computing SVD on the non-held-out data, and selecting k so as to minimize the reconstruction error between the held-out data and its SVD approximation.

It actually means:

  For instance, Owen and Perry (2009) show a method for excluding data, computing SVD on the remaining data, and selecting k so as to minimize the reconstruction error between the excluded data and its SVD approximation.

So it simply talks about a particular way of train-test-validation splitting the data.


Is there any example of using Q-learning with big data?

Or in RL does the agent built its own dataset?

Essentially this is the case.

RL is a very general learning mechanism, based on trial-and-error. You could create an environment where the agent's goal is to correctly predict classification or regression problems, where the agent is rewarded for getting close to the correct prediction. However, these problems are most often better addressed using supervised learning techniques. Using RL in such cases will most of the time make the learning slower, less efficient and less accurate.

The relationship between RL and supervised learning is more about how RL generates the target data for learning. In some cases, such as Deep Q Learning (DQN), you can see quite clearly from looking at the algorithm or code, that the RL agent contains a supervised learning component. The "supervised learning" in DQN learns a regression problem to predict the action values for the current agent's target policy. Whilst the outer RL logic that contains this supervised learning model, is built around testing and changing that behaviour in order to act optimally in the longer term.

Using RL to predict from a big data set would involve using the same kind of supervised learning model on the inside, with RL using that to guess at the correct result, and sometimes choosing to guess at some other random prediction instead. This guessing process is extra processing, plus it will add noise and variance to the error signal used to train the model. In a few cases, this might even be helpful, but in the majority of cases studied for regression and classification tasks, it will result in a poorer training process, less accurate model, or both.


My money would be on something much simpler like Naive Bayes.

In my experience for small data NB outperforms the more exotic methods.

Also, if you want to get more value out of your training data, try 10 fold cross validation


Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_1}
&= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
&= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
&= 2(\hat{y} - y) (1) \\
&= 2(\hat{y} - y)
\end{align}

The derivative of $\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_2}
&= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
&= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
&= 2(y - \hat{y})(-1)\\
&= -2(y - \hat{y})\\
&= 2(\hat{y} - y)
\end{align}

So, the derivatives of $\mathcal{L_1}$ and $\mathcal{L_2}$ are the same.


Let's first take a look at equation 6:

$$ \mathbf{E} \left[ r_{t,a} \vert \mathbf{x}_{t, a} \right] = \mathbf{z}_{t, a}^{\top} \boldsymbol{\beta}^* + \mathbf{x}_{t, a}^{\top} \boldsymbol{\theta}_a^* $$

Some quick observations:

The feature vector $\mathbf{z}_{t, a}$ has time $t$ and arms $a$ as subscripts. This means that we can have different vectors $\mathbf{z}_{t, a}$ for every time step $t$ and every arm $a$.
The feature vector $\mathbf{x}_{t, a}$ has time steps $t$ and arms $a$ as subscripts. These are exactly the same subscripts we also saw in $\mathbf{z}_{t, a}$. So, again, this means that we can have different feature vectors $\mathbf{x}_{t, a}$ for every time step $t$ and every arm $a$.
The parameters or weights vector $\boldsymbol{\beta}^*$ does not have any subscripts. This is the optimal vector (the star denotes optimality) that we aim to approximate through learning. This means that we learn only a single vector $\boldsymbol{\beta}$ that is used across all timesteps and all arms (of course, since we're learning it, it does actually actually change over time in practice; not because it should inherently change over time, but because we're still learning it).
The optimal parameter vector $\boldsymbol{\theta}_a^*$ (which we aim to approximate as $\boldsymbol{\theta}_a$ through learning) has arms $a$ as a subscript. This means that, for every arm $a$, we'll learn a separate parameter vector $\boldsymbol{\theta}_a$.

First a couple of corrections, some things I don't think are correct / clear in the paper:

In the paper, as you also quoted in the question, they say "it is helpful to use features that are shared by all arms, in addition to the arm-specific ones." This is in reference to the new feature vectors $\mathbf{z}_{t, a}$ that they're introducing there. However, we've just observed above that, due to having subscripts $a$, these feature vectors are in fact arm-specific, and not shared by all arms. However, we use these feature vector to take a dot product with a learned parameter vector $\boldsymbol{\beta}$, which in turn is shared across all arms. Intuitively, this means that we're saying that these feature vectors $\mathbf{z}_{t, a}$ can have different feature values per arm, but the "importance" of every value in such a vector is the same regardless of the arm we're looking at.
Since we assume that feature vectors $\mathbf{z}_{t, a}$ can be relevant for predicting rewards, it should also be included in the expectation on the left-hand side of equation 6, which should therefore be changed to:

$$ \mathbf{E} \left[ r_{t,a} \vert \mathbf{x}_{t, a}, \mathbf{z}_{t, a} \right] = \mathbf{z}_{t, a}^{\top} \boldsymbol{\beta}^* + \mathbf{x}_{t, a}^{\top} \boldsymbol{\theta}_a^* $$

Example

Suppose at every time $t$, a new customer enters our shop, and we have to pick one of a set of books (arms $a$) to try selling to that customer (note; this is actually very similar to news article recommendation, or selecting which Ad to display, or any other common MAB problem).

In feature vectors $\mathbf{x}_{t, a}$, we want features that we expect to have different influences on our chances of a successful sale per book. For example:

Age (kids will tend to prefer certain books, adults will tend to prefer others)
A measure of how many other books in the same genre that particular customer has bought before
etc.

In feature vectors $\mathbf{z}_{t, a}$, we want features that we expect to have the same influence on our chances of a successful sale regardless of what book we're looking at (note; the feature values may still be different per arm $a$ or time step $t$, we just expect their influence or importance to be the same / similar. For example:

A variable that is e.g. $1$ if the customer $t$ has enough money to buy a book $a$, or $-1$ if they don't have enough money. Note that the feature value can differ per customer and per book (different customers have different amounts of money, and different books have different prices), but we expect the influence to always be the same; if they have enough money, they might buy it, otherwise, they're very unlikely to buy it.
Age: yes, I'm aware that I already put this feature in the other list above as well. We might want to have it in both feature vectors. We put it in the previous list because we expect there to be book-specific preference levels depending on age. However, it might also be the case that certain age groups tend to buy more books than other age groups in general, so it might be beneficial to simply have it in both feature vectors (for different expected effects).


Perfect play in information incomplete games

This may be an evolving answer, because the question is, in some sense, a (useful) rabbit hole. I apologize if I don't go deeply into meta-games per se, as it's a little outside of my scope, which is non-chance games of perfect information, but I think it's worthwhile to think about the underlying problem of indeterminacy in relation to games in general.

Bounded Rationality* is a useful concept because it pre-supposes a condition of computational intractability. Computational intractability can be introduced into games in several forms:

Complexity
Hidden Information
Randomness ("quantum" indeterminacy)

[For more details on my use of "quantum" in regards to randomness, see Deterministic Games.]

The underlying purpose of game theory is to determine "optimal" strategies for any given problem.  I put optimal in quotes because optimality is a spectrum, and subjective in a condition of computational intractability.

Thus, we cannot know if AlphaGo plays optimally, only that it played more optimally than Lee Sedol in 4 out of 5 games.

This is distinct from strongly solved games such as tic-tac-toe, where we can know with total certainty that a choice is optimal, because the problem of tic-tac-toe is computationally tractable.

Part of the confusion may be semantic, because the concepts are subtle and profound, and require language, what TS Eliot might have called "the intolerable wrestle with words and meanings." (For instance, I used hidden information above to avoid having to distinguish between incomplete and imperfect information.)

Perfect Play is generally defined as a strategy that leads to the best possible outcome for a participant, regardless of the choices of the opponent.

Thus minimax is of central importance, and provided the foundation for game theory.

Even in games with incomplete information, whether "deterministic" (Battleship) or involving "quantum indeterminacy" (Prisoner's Dilemma), there are optimal strategies. For simultaneous games such as Dilemma and all of the numerous extensions minimax is used.  In Battleship, there are at least three strategies of increasing optimality, and although there doesn't appear to be a strategy that can yield P > .5, if one player employs a more optimal strategy, they will win in aggregate. Even Rock, Paper, Scissors seems to have an optimal strategy, which blows my mind, and carries the caveat that I need to look into it more.

Thus, perfect play, as defined, is certainly achievable, but does not necessarily connote (objectively) optimal choices, which is a little confusing, because "perfect" implies objectivity, a condition which is only possible in regard to tractable problems.

It is also important to note that there may not be a "winning" strategy in the sense of being better off than the opponent, and in this condition, perfect or optimal play is mitigation of loss.

*In terms of incomplete information games specifically, I think there's a case for extending the concept of Bounded Rationality is extended to include information that cannot be observed or "known".

Colloquially, this would include the "unknowns" (both known and unknown) and the "unknowable" (quantum indeterminacy and superpositions).


How to understand 'losses' in Spacy's custom NER training engine?

A critical goal of training a neural network is to minimize the loss.  Loss is not explained for spaCy because it is a general concept for machine learning and deep learning.  Loss is not specific to spaCy and although there are some finer details I don't believe that is your inquiry.
In general, to understand loss functions, I recommend the following resources:

Loss and Loss Functions for Training Deep Learning Neural Networks by Jason Brownlee.  Jason writes some great articles and tutorials and I highly suggest reading his other articles and books.
Introduction to Loss Functions
Loss Functions in Machine Learning for Beginners by John Kaller

If you like videos watch:

Loss Functions Explained by Siraj Raval.  Siraj has many great videos on machine learning.


CAPTCHA based on text comprehension and random tokens

(Assuming English.) In your specific example, there would be $26^2$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $26 * 26 * 7 = 4732$ possible combinations. My intuition is that the key space is too small.
It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $(26 + 10)^6 = 2,176,782,336$ possibilities. That keyspace is $460,013$ times larger than your system with the longer string. It gets worse though.  An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. It would not be a guaranteed win for the attacker. But it would be guessing at a higher probability than $\frac{1}{4732}$.
Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.
I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just "Pick the first pair of adjacent letters." But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.


Which is best: evaluation of states or probability of moves?

Which do you think is the best method?

As with most machine learning, each approach has its strengths and weaknesses, and other than a little bit of intuition:

Policy-based methods are strong in large or continuous action spaces, and/or where there is a simple relationship between state and optimal action. E.g. controlling a robotic arm with continuous action space.
Value-based methods are strong where there is a simple relationship between state and value under an optimal policy. E.g. in a maze game.

It may not always be clear which is best, in which case experimentation is required. If using neural networks, there will then be a large number of hyper-parameters on each approach, so it may be hard to come to a strict conclusion about which is better. Although you can include "easy to find a working model" or "robust for a range of hyper-parameter values" as benefits of any type of model if you wish - these are important practical benefits of any approach, developers rarely want to do 100s of experiments to tune a learning rate parameter for example.

  Which is the best way to do AI? Or some combination of the two?

Actor-Critic, as seen in Asynchronous Advantage Actor Critic. A3C and A2C (deterministic variant of A3C) is producing current state-of-the-art results in video games. This is a combination of both approaches, where the agent maintains two related models - one, the Actor, generates a policy directly by looking at the state, and the second, the Critic, tracks the estimated value of each state. Often, these two models share some parameters - e.g. using neural networks, the initial layers may be the same for both.


What is the relation between semi-supervised and self-supervised visual representation learning?

Semi-supervised learning
Semi-supervised learning is the collection of machine learning techniques where there are two datasets: a labelled one and an unlabelled one.
There are two main problems that can be solved using semi-supervised learning:

transductive learning (i.e. label the given unlabelled data) and
inductive learning (generalization) (i.e. find a function that maps inputs to outputs, like classification).

Self-supervised learning
Self-supervised learning (SSL) is a machine learning approach where the supervisory signal is automatically generated. More precisely, SSL can either refer to

learn data representations (i.e. learn to represent the data) by solving a so-called pretext (or auxiliary) task, in a self-supervised fashion, i.e. you automatically generate the supervised signal from the unlabelled

automatically label an unlabelled dataset by exploiting data coming from different sensors (this is the usual definition of SLL in the context or robotics)

What is the relationship between the two?
SSL (for data representation) can be considered a semi-supervised learning approach, if you fine-tune the learned data representations with a labeled dataset to solve supervised learning problem (i.e. the so-called downstream task), which you will probably do otherwise the data representations are pretty useless. Read this answer for other details.


What is the input to AlphaGo's neural network?

The input to the neural network is a $19 × 19 × 17$ image stack
  comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the
  presence of the current player’s stones ($X^i_t = 1$ if intersection $i$ contains a stone of the player’s
  colour at time-step $t$; $0$ if the intersection is empty, contains an opponent stone, or if $t < 0$). A
  further $8$ feature planes, $Y_t$
  , represent the corresponding features for the opponent’s stones. The
  final feature plane, $C$, represents the colour to play, and has a constant value of either $1$ if black
  is to play or $0$ if white is to play. These planes are concatenated together to give input features
  $s_t = [
X_t, Y_t, X_{t−1}, Y_{t−1}, ..., X_{t−7}, Y_{t−7}, C]$.

This and all the other architecture details can be found in the "Neural Network Architecture" section in the paper.


Do VR, AR and MR use any machine learning or deep learning?

TL;DR

Yes for all the 3 cases

Details

VR

Even if the environment is virtual there is another thing to perceive which or better who is real: the user

Deep Learning can be used for things like: eye tracking, gesture recognition, voice recognition, ...

Furthermore, in games,  you could think about more complex applications like understanding the way the user plays the game and react accordingly

AR and MR

In addition to what has been said for VR, in these cases you also have to integrate the environment in the final user experience so you ultimately have to perceive it and there multiple specific applications: object detection, planes detection, depth estimation, Visual Odometry (e.g. as feature detection module), SLAM, ...

These days Deep Learning is achieving Starte-Of-The-Art (SOTA) in many different tasks far beyond the classification and detection, but one things is research so published papers and so on and another thing is a product
so to sum up every VR, MR, AR vendor is probably relying on an hybrid solution where traditional algorithms and deep learning coexist but so far it seems the trend is deep learning is eating the traditional algos up day after day

References

The rising force of deep learning in VR and AR

Virtual Reality And Machine Learning Go Hand In Hand


What does the Markov assumption say about the history of state sequences?

A stochastic process has the Markov property if the probability distribution of future states conditioned on both the present and past states depends only on the present state or, more formally, the following equality holds.

$$
p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t}), \forall t
$$

The hidden Markov model (HMM) is an example of a model where the Markov property is often assumed to hold. In other words, the Markov assumption is made in the case of the HMM.

There are also the variable-order (or higher-order) Markov models, where the future state can depend on the previously $n$ states or, more formally, the following equality holds.

$$
p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t:t-n}), \forall t
$$

In this context, a hidden Markov model is called a first-order Markov model ($n=0$). Therefore, there can also be second-order ($n=1$), third-order ($n=2$), etc., Markov models. In fact, there are also higher-order hidden Markov models.

To conclude, the expressions Markov property and Markov assumption are not exactly interchangeable. The Markov property is an attribute that a stochastic process can be assumed to possess. In that case, the Markov assumption is made. The expression Markov property usually refers to a first-order Markov property, but it can more generally refer to a higher-order Markov property.


Applications of CNN for detecting crime from video surveillance cameras

After a bit of research I found something kind of close:

Artificially intelligent security cameras are spotting crimes before they happen
New surveillance cameras will use computer eyes to find 'pre crimes' by detecting suspicious behaviour and calling for guards
CCTV 'fightcams' detect violence 'before it happens'  at Dailymail, also check at Telegraph

They, however, makes no mention of what specific methods they use.

So a crime detection system as that is written does not exist, but abnormal behaviour detection systems do.

An accurate generalized system seems intuitively infeasible, however. Commiting a crime, unlike falling, is a complex behavior, and takes so many forms. A camera watching a store's counter like at a 7-11 could perhaps see that the 'customer''s arm is strangely reaching across the counter, and the attendant is moving a lot more than usual suddenly, but aside from very specific cases like this such a system is currently quite unfeasible.

Crimes are unusual, relatively speaking, and their dramatic nature means that even the simplest crimes play out in very different ways. Perhaps you could in this case you could try to look for images of a gun, or someone with their hands up. So, looking for unusual, detectable behavioural mannerisms may be possible, but not crime detection.

Ultimately, while you may be able to make (possibly pretty good) systems to detect specific crimes in specific environments, that's all we got for now.

P.S. - Do these camera's also get audio signals? That is also an interesting facet to consider ("PUT YOUR HANDS UP / GIVE ME ALL YOUR MONEY")


Why does TD Learning require Markovian domains?

The Markov assumption is used when deriving the Bellman equation for state values:
$$v(s) = \sum_a \pi(a|s)\sum_{r,s'} p(r,s'|s,a)(r + \gamma v(s'))$$
One requirement for this equation to hold is that $p(r,s'|s,a)$ is consistent. The current state $s$ is a key argument of that function. There is no adjustment for history of previous states, actions or rewards. This is the same as requiring the Markov trait for state, i.e. that $s$ holds all information necessary to predict outcome probabilities of the next step.
The one step TD target that is sampled in basic TD learning is simply the inner part of this:
$$G_{t:t+1} = R_{t+1} + \gamma \hat{v}(S_{t+1})$$
which when sampled is equal to $v(s)$ in expectation *, when $S_t = s$. That is, when you measure a single instance of the TD target and use it to update a value function, you implicitly assume that the values or $r_{t+1}$ and $s_{t+1}$ that you observed occur with probabilities determined by $\pi(a|s)$ and $p(r,s'|s,a)$ as shown by the Bellman equation.
So the theory behind TD learning uses the Markov assumption, otherwise the sampled TD targets would be incorrect.
In practice you can get away with slightly non-Markov environments - most measurements of state for machinery are approximations that ignore details at some level, for instance, and TD learning can solve optimal control in many robotics environments. However, Monte Carlo methods are more robust against state representations that are not fully Markov.

* Technically this sample is biased because $\hat{v}(S_{t+1})$ is not correct when learning starts. The bias reduces over time and multiple updates. So the expected value during learning is approximately the same as the true value as shown by the Bellman equation.


Loss function for singular object detection

The terms error and loss are always relative to some ideal of fitness, and it is always against some model, the extension of the concept of curve fitting.  The ideal is the optimized state.

When we are analyzing system behavior and optimizing PROCESS parameters as we do in machine learning, we don't always use the same terms.  Just as people in medicine and alternative medicine speak of wellness, systems that process data instead of food also can be considered either well or dysfunctional.  With machine vision, we optimize the processing of an arbitrary signal, an image with an object depicted in your case, not the parameters of a curve to fit a set of data points.

It is common to wish to judge the wellness of the learning state across a range of sizes.  You cannot normalize the image by re-sizing before you recognize the image boundaries, so you have to design the process (circuit) that must be trained accordingly.

It is not in the error function that this can be done.  There are only two places where it can be done.

The model of the object you are trying to recognize
The process topology you are attempting to train

The standard approach for size-independent recognition involves a sequence in the process being trained, reflected in activation and interconnection choices.

Edge detection
Corner detection
Shape detection
Illumination detection (for higher reliability)

Note that edge detection and corner detection can overlap in learning layers but not necessarily.

The key is obviously between corner and shape detection.  The feed of corners into the layer that detects shape has to be angular rather than Cartesian yielding this sequence.

Edge detection
Corner detection
Angles at corners
Shape/form detection
Illumination detection (for higher reliability)

A designer can hope a deep network will learn these through back-propagation, but convergence upon the ideal behavior (wellness) will be faster and more reliable if the layers are designed to perform those functions.  That means your model for the shape or form you hope to detect must be in terms of angles not coordinates.

Florian Pierre and Joseph Piewak seem to be doing it correctly: https://arxiv.org/pdf/1709.03138.pdf. Search the document with ctrl-f for "angl".  These researchers show excellence in their treatment: https://arxiv.org/pdf/1806.07996.pdf [2]. Search for "size indep" in that one.

References

[1] Fully Convolutional Neural Networks for Dynamic Object Detection in Grid Maps; Master’s Thesis of Florian Pierre, Joseph Piewak; Department of Computer Science, Institute for Anthropomatics and FZI Research Center for Information Technology respectively; 2016

[2] Novel Convolution Kernels for Computer Vision and Shape Analysis based on Electromagnetism; Beaini, Achiche, Cio, Raison; 2018; Polytechnique Montreal, 2900 Edouard Montpetit Blvd, Montreal, H3T 1J4, Canada


What are some resources on computational learning theory?

Although I have only partially read (or not read at all) some of the following resources and some of these resources may not cover more advanced topics than the ones presented in the book you are reading, I think they can still be useful for your purposes, so I will share them with you.
I would also like to note that if you understand the contents of the book you are currently reading, you are probably already prepared for reading some (if not most of) the research papers you wish to read. Initially, you may find them a little bit too succinct and sometimes unclear or complex, but you need to get used to this format, so there's nothing stopping you from trying to read them and learn even more by doing this exercise.
Books

An Introduction to Computational Learning Theory (1994) by Kearns and Vazirani

No free PDF available

The Nature of Statistical Learning Theory (1995, 2000) by Vapnik
Machine Learning (1997) by Mitchell
Statistical Learning Theory (1998) by Vapnik
Prediction, Learning, and Games (2006) by Cesa-Bianchi and Lugosi
Foundations of Machine Learning (2012) by Mohri et al.
Boosting: Foundations and Algorithms (2012) by Schapire and Freund
Bandit Algorithms (2020) by Lattimore and Szepesvári (with associated solutions to the exercises)

Papers

A theory of the learnable (1984) by Valiant
An overview of statistical learning theory (1999) by Vapnik
Introduction to Statistical Learning Theory (2014) by Bousquet et al.

Courses (videos)

Learning from Data (Caltech) (2012) by Yaser Abu-Mostafa
Machine Learning Theory (2015) by Shai Ben-David

Lecture notes

Introduction to Machine Learning (2009) by Shai Shalev-Shwartz

Concentration-of-measure inequalities (2009) by Gábor Lugosi

Stat 928: Statistical Learning Theory (2011) by Sham Kakades

COMS E6253, Advanced Topics in Computational Learning Theory (2012) Rocco Servedio

9.520: Statistical Learning Theory and Applications, Spring 2012 by Tomaso Poggio et al.

STAT928: Statistical Learning Theory and Sequential Prediction (2014) by Alexander Rakhlin and Karthik Sridharan

ECE 543, Statistical Learning Theory (2019) by Bruce Hajek and Maxim Raginsky

Other
See also this list of resources https://kiranvodrahalli.github.io/links/#resources-notes-textbooks-monographs-classes-etc compiled by Kiran Vodrahalli.


Is it ok to struggle with mathematics while learning AI as a beginner?

I think the key part of your question is "as a beginner". For all intents and purposes you can create a state of the art (SoTA) model in various fields with no knowledge of the mathematics what so ever.

This means you do not need to understand back-propagation, gradient descent, or even mathematically how each layer works. Respectively you could just know there exists an optimizer and that different layers generally do different things (convolutions are good at picking up local dependencies, fully connected layers are good at picking up connections among your neurons in an expensive manner when you hold no previous assumptions), etc.. Follow some common intuitions and architectures built upon in the field and your ability to model will follow (thanks to the amazing work on opensource ML frameworks -- Looking at you Google and Facebook)! But this is only a stop-gap.

A Newton quote that I'm about to butcher: "If I have seen further it's because I'm standing on the shoulders of giants". In other words he saw further because he didn't just use what people before him did, he utilized it to expand even further. So yes, I think you can finish your object detection project on time with little understanding of the math (look at the Google object detection API, it does wonders and you don't even need to know anything about ML to use it, you just need to have data. But, and this is a big but, if you ever want to extend into a realm that isn't particularly touched upon or push the envelope in any meaningful way, you will probably have to learn the math, learn the basics, learn the foundations.


How can I model a problem as an MDP if the agent does not follow the successive order of states?

Your problem fundamentally is that you are confusing what the state and actions are in this setting. Webpages are not your states; your state is the entire priority queue of (website-outlink) pairs + the (new_website-outlink) pairs. Your action is which pair you select.
Now this is a variable sized state-space and variable sized action-space problem setting at the same time. To deal with this lets start by noting that state==observation need not be (in general). So what is your observation? Your observation is a variable-sized batch of either:

(website-outlink) pairs or
next_website (where each next_website is determined by its corresponding pair)

Both of these observations could work just fine, choosing between one or the other is just a matter of whether you want your agent to learn "which links to open before opening them" or "which links are meaningful (after opening them)".
What your priority queue is essentially doing is just adding a neat trick that:

Saves the computational complexity of keeping the state ordered (remember that your state is not a website, but the list/batch of website-outlink)
Avoids needlessly recomputing the Q-values for each of your actions (remember that an action is not selecting an outlink from new_website, but selecting an outlink from all available choices in the queue)

Note however that to actually have the second saving it is crucial to store the Q-values for each pair!!!
Last important thing to note is that in a scenario where you use a Replay Buffer (which I guess is likely given that you chose a DQN), you can't use the priority queue whilst learning from the RB. To see why (and to see in detail how your learning process actually looks like), start by remembering that your Q-value updates are given by the formula here; your state s_t is a (quasi-ordered1) batch of pairs. Q(s_t, a_t) is just the output of running your DQN regression on just the best website/pair in this batch (you have to add an index to denote the best choice when adding transitions to the RB, in order to be consistent about which action was taken from this state). To compute the estimate of the optimal future value however you will have to recompute the Q-value of every single website/pair in the next state. You CANNOT use the priority queue when training from the RB.
1 You have the priority queue ordered for all websites you had in it whilst looking at the last website, but all the new_website-outlink pairs you are now adding are not ordered yet. You still have to run the agent on them and then you can order them with the rest of the priority queue to generate the next state (which still will not be ordered because you will have new_new_website-outink pairs).


What may be more informative in terms of whether it is learning or not, is to track gradients.
Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.
You are developing with tf.keras, in which case you can add to your tensorboard callback:
tf.keras.callbacks.TensorBoard(write_grads=True)
Additional experiments may include:

trying shorter length sequences and compare grad flow
try replacing tanh with alternative activation functions in the LSTM layers of your model tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu()) (see tanh saddle points problem)

(graph borrowed from d2l)


Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Discretizing state/action space will always be a very expensive strategy, in fact I don't think you can do better than exponential time in state/action dimension that way.
Now, you still haven't really explained your algorithm with the discretized states and actions. Given your known models, it sounds like you're planning on doing dynamic programming. This would definitely be an exponential time algorithm. You also haven't explained how you'd use your knowledge of the model in your policy gradient implementation. If you're just using it to sample some trajectories to have more data for updates, policy gradient methods will be much more efficient (poly-time). Otherwise, if you're planning on using your model for some sort of MCTS planning, depending on how you implement that, it could be very inefficient as well.


What are the possible social consequences of training neural networks with artificially generated data?

We can already observe information bubbles on social media, where the circle is that the ML algorithms learn what content people like and give more similar content based on clicks and so on. From a single wrong click, you could enter a bubble and never come out if you don't take care or be aware.
This happens with humans, so the same may apply to computers. Checking backgrounds, like click saver for click titles as an ultimate example, would save the machines from circles too. Information is not different whether it is made in consecutive rounds by humans or machines.
For example, if you make a thesis, not necessarily all things you write are your own investigation and research. You use quotes, refer to papers, and so on. For information sources, you have to be careful if the source is respected and considered to be meaningful and correct.
The same carefulness needs to be applied to machine-generated content. To the responsible - the last responsibility is with the reader to believe or not what you're reading.


The environment is :

Deterministic : the future state of the environment (and the agent) is defined by the agent's action.
Partially Observable (partially accessible) : since his sensors don't see what's beyond the shaded barrier, the environment is partially observable (accessible).
Sequential (non-episodic): Since the agent's current action will constrain and affect the space of possibilities (options, choices) of future actions, current decisions are not independent of future ones.
Discrete : Environment is not continuous, that is, every action/step/case is finite (either turn left, right, forward or backward), you don't have other actions, which makes it a discrete task environment.
Static : the environment is not changing, it's static, only the agent is moving.


Why are denser layers needed in computer vision neural nets?

Convolutional layers are added in order to extract features from the image (like edges, corners, textures). After extracting those features, you feed them to a fully connected neural network to get the prediction.

Let's take an example, consider you want to classify the cat's image. But you decided to do this by only using the convolutional layer. So, you feed the image to a convolutional layer. After passing through some layer it extracted the key features from the cat's image and at the end of the layer, you add a single neuron(since we have a single class to predict). Now it's ready to classify it as a cat. But unfortunately, it's misclassified the image. Why? Now let us answer the question.

Since convolutional layers are not fully connected layers, the neuron added at the last layer is only connected with a handful of neurons from the previous layer. So it misses the key features (encapsulated by some other neurons it is not connected with them) in order to detect this image as a cat. In order to get the prediction right, you have to add a dense layer(a fully connected layer) at the end of the convolutional layer so that it can get all of those extracted features.

If this does not satisfy you, please, ask the question more specifically and also edit the question because linear layer does not call fully connected layer.


How to solve the problem of variable-sized AST as input for a (convolutional) neural network model?

You discovered already one solution for your problem: Zero-Padding.

There are two other common possibilities:

Using Recurrent NNsThis is often used at text processing, where you feed each word one after another into your model.
Using Recursive NNs (I wont recommend this for your use case)This method is also frequently used in word processing, but is more often applied in the semantic analysis of text. You reduce the text to the essential, until it has reached the desired length. However, information is lost in this process.


Why is this deep Q agent constantly learning just one action?

I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked.

So think about your input guys


How can I design and train a neural network to play a card game (similar to Magic: The Gathering)?

I think you raise a good question, especially WRT to how the NNs inputs & outputs are mapped onto the mechanics of a card game like MtG where the available actions vary greatly with context.

I don't have a really satisfying answer to offer, but I have played Keldon's Race for the Galaxy NN-based AI - agree that it's excellent- and have looked into how it tackled this problem.

The latest code for Keldon's AI is now searchable and browseable on github.

The ai code is in one file. It uses 2 distinct NNs, one for "evaluating hand and active cards" and the other for "predicting role choices".

What you'll notice is that it uses a fair amount on non-NN code to model the game mechanics. Very much a hybrid solution.

The mapping of game state into the evaluation NN is done here. Various relevant features are one-hot-encoded, eg the number of goods that can be sold that turn.

Another excellent case study in mapping a complex game into a NN is the Starcraft II Learning Environment created by Deepmind in collaboration with Blizzard Entertainment. This paper gives an overview of how a game of Starcraft is mapped onto a set of features that a NN can interpret, and how actions can be issued by a NN agent to the game simulation.


I understand your question as: "How did the author select the number of neurons in their hidden layer?"

The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more intricate decision barriers. However, the more complex function is harder to optimize and will lead to lower performance scores. The goal here is to find the right tradeoff to maximize your performance. You can tune the number of hidden neurons as a hyper-parameter using cross-validation.

There isn't any formula to determine the number of neurons you will need, however you can get an intuition based on the number of inputs and outputs you will have. Generally, you want more hidden neurons than input and output neurons. Since most people are programmers who are writing neural networks, we are used to working with units in $2^n$. Thus, 16 is chosen over 10, and 32 would be chosen over 28.


Do I have to crossover my node genes in NEAT, and how?

Okay, I'm first going to review how NEAT works. I hope this helps you model NEAT successfully as a whole, not just limited to your question.

We use neuro-evolution to create a specific behavior that solves a given task. The behavior can be simple and complex.

Now let's focus on behavior... Different neural networks can create the same behavior (A.K.A the competing conventions problem).

We also want to end up with a neural network that is very efficient.

So we want to solve two problems with one algorithm (NEAT): find the behavior that solves a task and find the most efficient neural network that creates the behavior.

There's a simple way to search for the most efficient neural net: start with the simplest neural net and slowly build the neural net up.

The hard part is defining behavior (how do we define behavior in neural networks?). NEAT introduces their very interesting definition of behavior: an atomic unit of neural network behavior is the connection of the neural network: which node connects to which and with what weight.

Now I'm approaching your question(concern): you want to make sure your genomes can grow complex, which means that you want to preserve the capacity for your genome to express complex behavior. As the paper states, behavior does not have anything to do with nodes, let alone node genes. Behavior is a set of neural network connections.

So here's my answer to your question: care about crossing over connections, not for the node genes. Find an efficient way to calculate the node genes. For instance, you'll consume a lot of memory if you just concatenate node genes from two parents without overlaps. There is a chance that some of the node genes might not be in use (i.e. disabled connections, not inheriting some connection genes due to probability).

Hope this helps :)


Why not use the target network in DQN as the predictor after training

The learning model and target model are only different by N steps (typically a few thousand) out the entire taining process. If the process is near complete, they will also be quite similar.
The target model is not inherently more stable in terms of producing "correct" or "better" Q values. Instead it is kept static for a period of time in order to stabilise the temporal difference (TD) target $R_{t+1} + \gamma \text{max}_{a'} \hat{q}(S_{t+1}, a', \theta^{-})$
Due to the copying stage that you listed:

If you return the target model, this is identical to returning the learning model from N steps beforehand.

Which in turn means that there was no point in doing the last N steps of training. You may as well have returned the learning model at the same point as copying.

Or another way of thinking about it: Returning the learning model at the end of the training process is identical to copying it to the target model as normal, then returning the target model immediately.


Is there any way to apply linear transformations on a vector other than matrix multiplication?

You can sometimes exploit the structure of your matrix to perform faster matrix multiplication. For example, if your matrix is sparse (or dense), there are algorithms that exploit this fact.

In your case, you can actually compute $A^n$ in less time than $\mathcal{O}(n^3$). For example, have a look at this question at CS SE and this one at Stack Overflow (SO). Note that the provided solutions may not be numerically stable, so I am not suggesting you use them.

Moreover, if you perform your operations on the GPU, they could be faster in practice. See e.g. this question at SO and this one at SciComp SE.


As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too).
You may require the output of the neural network to be a probability, for example, if you use a cross-entropy loss function, although you could in principle produce only $0$s or $1$s. The probability $p$ can then be used to decide the class (or label) of the input. For example, if $p > \alpha$, then you purposedly decide that the input belongs to class $1$, otherwise, it belongs to class $0$. The parameter $\alpha$ is called the classification (or decision) threshold. The choice of this threshold can actually depend on the problem and it is one of the reasons people use the AUC metric, i.e. to avoid choosing this classification threshold.

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?

The most famous universal approximation theorem for neural networks assumes that the activation functions of the units of the only hidden layer are sigmoids, but it does not assume that the output of the network will be squashed to the range $[0, 1]$. To be more precise, the UAT (theorem 2 of Approximation by Superpositions of a Sigmoidal Function, 1989, by G. Cybenko) states

Let $\sigma$ be any continuous sigmoidal function. Then finite sums of the
$$G(x) = \sum_{j=1}^N \alpha_j \sigma (y_j^T x + \theta_j)$$
are dense in $C(I_n)$.
In other words, given any $f \in C(I_n)$ and $\epsilon > 0$, there is a sum, $G(x)$, of the above form, for which
$$|G(x) - f(x)| < \epsilon $$

Here, $f$ is the continuous function that you want to approximate, $G(x)$ is a linear combination of the outputs of $N$ (which should be arbitrarily big) units of the only hidden layer, $I_n$ denotes the $n$-dimensional unit cube, $[0, 1]^n$, $C(I_n)$ denotes the space of continuous functions on $I_n$, $x \in I_n$ (so the assumption is that the input to the neural network is an element of $[0, 1]^n$, i.e. a vector $x \in \mathbb{R}^n$, whose entries are between $0$ and $1$) and $y_j$ and $\theta_j$ are respectively the weights and bias of the $j$ unit. The assumption that $f$ is a real-valued function means that $f$ can take any value on $\mathbb{R}$ (i.e. $f: [0, 1]^n \rightarrow \mathbb{R}$). You should note that $G(x)$ is the output of the neural network, which is a combination (where the coefficients are $\alpha_j$) of the outputs of the units in the only hidden layer, so there's no restriction on the output of $G(x)$, unless you restrict $\alpha_i$ (but, in this theorem, there's no restriction on the values $\alpha_j$ can take).
Of course, if you restrict the output of the neural networks to the range $[0, 1]$, you cannot approximate all continuous functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (because not all of these functions will have the codomain $[0, 1]$)! However, the sigmoid has an inverse function, i.e. the logit, so you can reverse the output of such a neural network. So, in this sense (i.e. by reversing the output of the sigmoid), a neural network with a sigmoid as the activation function of the output layer can potentially approximate any continuous function too.
The UAT above only states the existence of $G(x)$ (i.e. it's an existence theorem). It doesn't tell you how you can find $G(x)$. So, if you use a sigmoid as the activation function of the output layer or not is a little bit orthogonal to the universality of neural networks.


How to use pre-trained BERT to extract the vectors from sentences?

Did you mean:

  How do you use a pre-trained BERT model in a feature-based setting
  to get pre-trained word contextual embeddings?

Here is the BERT paper. I highly recommend you read it.

Firstly, by sentences, we mean a sequence of word embedding representations of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned, and so a (usually fixed) sequence of such vectors represent the sentence input. (We don't need the input to always be divided to individual sentences)

There are mainly two ways to train the modern neural network models for sequence modeling tasks such as language modelling, machine translation and question answering:

1. The pretrain - finetune based approach (also called Transfer Learning)

In this approach, a model is first pre-trained on some auxiliary dataset whose domain usually overlaps with the domain of the dataset corresponding to the downstream task, that is, the actual task for which we need to build the model.

In the case of Natural Language Processing (NLP) tasks, such as machine translation and question answering, the pre-training that is done is usually unsupervised (no labels in the dataset) where our training objective is to maximize the log likelihood of the sentences. You can read more about Maximum Likelihood estimation here.

Its almost always a good idea to pre-train models (whether it is for NLP tasks or for computer vision tasks etc). The more we pre-train a model, the closer its parameters are to the optimal ones, and the work needed in the fine-tuning phase is minimized. This is why we use large scale datasets for pre-training.

Later, during the fine-tuning phase, we fine tune the parameters of the model to suit a downstream task, such as a task of translation between two specific languages of text. Since the parameters of the model are already at a position in the error/loss space that is already sort of good enough for the model (because we already pre-trained it with text in, say, 10 languages), the fine-tuning can, in a sense, tailor the model to suit the task a hand.

2. The feature based approach

In this approach, we take an already pre-trained model (any model, e.g. a transformer based neural net such as BERT, which has been pre-trained as described in the previous approach) and then we extract the activations from one or more layers of this pre-trained model.

This is done by simply inputting the word embedding sequence corresponding to a sentence to the pre-trained model and then extracting the activations from one or more layers (one or more of the last layers, since the features associated to the activations in these layers are far more complex and include more context) of this pre-trained model.

These activations (also called contextual embeddings) are used as input (similar to the word embeddings described earlier) to another model, such as an LSTM or even another BERT. That is, fixed features are extracted from the pre-trained model.

Usually the size of these layers are the same as the size of the input layer, i.e. they are also a sequence, but a sequence of far more complex features instead of word embeddings. If the word embedding vector of the input to this model id $D$ and the length of the sentence which is input to the model is $L$, then each of the hidden activation layers have the same length $L$, but the hidden size of these vectors is $H >> D$, i.e. far higher to incorporate more complex features.

Even though the dimensionality of the hidden layers is far higher, these layers are equal in length to the input layer, and so we have a one to one correspondence between the input tokens in the sentence and each of these hidden layer activations.

The bonus in using these hidden representations is that we get to use representations of the input sentence that also incorporate CONTEXT, dependency between the words of the sentence, an important feature that is to be modeled by all models used for NLP tasks. Models such as BERT, are based on Transformer models that use an attention mechanism in order to model these contextual features.

This is a superb article describing to the last detail the attention mechanism and the Transformer model based on which models like BERT operate. For more about contextual embeddings I recommend taking a look at this article. The figure used here is from this article.

Models such as BERT make use of one half of these Transformer models, called the encoder, since the Transformer is a seq2seq or encoder-decoder model. Here is the link to the Transformer paper.

Here is a great blog on extracting contextual word embeddings from BERT using Tensorflow and Keras.

I'll also provide a link to a Kaggle Python Notebook on using Pipelines functionality from the HuggingFace community repo on github that also is used for feature extraction (contextual embeddings).

Please let me know if you have any questions, happy to help!


Reinforcement learning is the best fit for problems like these. To get started explore how DeepMind solved Atari games. The key is self play. RL works through trial and error and eventually figures out the best strategy. You also need to tweak so that it does not play at God level making it impossible for users to win.


What do we mean by saying "VC dimension gives a LOOSE, not TIGHT bound"?

But, the literature (i.e. Learning from Data) states that VC gives a loose bound and that in real applications, learning models with lower VC dimension tend to generalize better than those with higher VC dimension.

It's true that people often use techniques such as regularisation to avoid over-parametrized models. However, I think it's dangerous to say that, in real applications, those models are really generalizing better, given that you're typically assessing their generalization ability by using a finite validation dataset (possibly selected in a biased way). Moreover, note that the validation dataset is often ignored in certain bounds and the only thing that is taken into account is the expected risk and the empirical risk (on the training data).

In any case, the $\mathcal{VC}$ bounds may be "loose" because

the $\mathcal{VC}$ dimension is often expressed with the big-$\mathcal{O}$ notation (e.g. see this answer)
the bounds often involve probabilities and uncertainties (e.g. see this other answer)
there aren't stricter bounds (i.e. no one found better bounds yet)

  Is the VC generalization bound loose due to its universality? Meaning, its results apply to all hypothesis sets, learning algorithms, input spaces, probability distributions, and binary target functions.

I think the answer is yes. In fact, often, the specific tasks or learning algorithms are ignored in the analyses, but, in general, you may exploit your knowledge of the problem to improve e.g. the number of useful examples that the learning algorithm could use.


Are PAC learnability and the No Free Lunch theorem contradictory?

There is no contradiction.
First, agnostic PAC learnable doesn't mean that the there is a good hypothesis in the hypothesis class; it just means that there is an algorithm that can probably approximately do as well as the best hypothesis in the hypothesis class.

Also, these NFL theorems have specific mathematical statements, and hypothesis classes for which they apply are often not the same as the hypothesis class for which PAC-learnability holds. For example, in Understanding Machine Learning  by Shalev-Shwartz and Ben-David, a hypothesis class is agnostic PAC learnable if and only if has finite VC dimension (Theorem 6.7). Here, the algorithm is ERM.
On the other hand, the application of the specific version of NFL that this book uses has Corollary 5.2, that the hypothesis class of all classifiers is not PAC learnable, and note that this hypothesis class has infinite VC dimension, so the Fundamental Theorem of PAC learning does not apply.

The main takeaway is that in order to learn, we need some sort of inductive bias (prior information). This can be seen in the form of measuring the complexity of the hypothesis class or using other tools in learning theory.


How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I'm not sure what do you mean by one input. The input to the pruning agent is always the same, it's the convolutional layer $W$ of dimension $m \times h \times w$. The layer is taken from baseline model that is pretrained. The input doesn't change it's always the same. The output of the pruning agent is an array of probabilities to prune a specific filter. For example if you have $3$ filters in a layer, the output of the pruning agent will be array of $3$ elements . Let's say its
\begin{equation}
y = [0.1, 0.6, 0.7]
\end{equation}
Each of these elements represents probability of pruning filter $i$ in layer $W$. So $0.1$ would be probability to prune filter $1$, $0.6$ to prune filter $2$ and $0.7$ to prune filter $3$.
Let's say you sample this distribution $2$ times and you get: $[0, 1, 1], [0, 0, 1]$. That means you would make 2 different models from the original baseline model. First model would have 2nd and 3rd filter pruned in layer $W$, and second model would have 3rd filter pruned. The you run those 2 new models on your train and validation set, calculate objective function $R$. Then you update parameters $\theta$ of your pruning agent based on $R$. The original weights of layer $W$ stay untouched. Then you do another inference of the pruning model $\pi$ with updated parameters $\theta$ (the input is still original $W$). You will get another array of probabilities and you keep repeating previous steps that i described until parameters $\theta$ converge. When they converge you make final pruning.


What is a heavy node in neural networks?

Yes, firstly you are correct to acknowledge the term node with reference to the neural network.

A neural network is made to mimic the working of a human brain hence the term neural. It comprises of several layers and each layer comprises of several nodes. A network can be a deep neural network, signifying the presence of numerous layers, in which the output of the one, serves as an input for the other.
Further a node in a particular layer has some weights and biases attached with it that are responsible for the computation and producing output.

To your context, a heavy node may refer to a node that holds importance for the network and has got significant weight and bias attached. Henceforth as per the statement if the network does make any significant change to the node with importance in terms of weights and biases, it is noticed else not.


How can genetic programming be used for path planning?

Take a robot that we want to be able to move from the bottom right corner to the top left corner of a 4x4 matrix full of random holes it should avoid. With holes represented by 1s, it could look something like:

exit
\/
[0,0,0,1]
[0,1,1,0]
[0,1,1,1]
[0,0,0,0]
      /\
      enter

As we want it to get to an exit from a start, we have a natural fitness function: closeness to exit door in smallest number of moves.

The genetic programming approach to solving this is to create random computer programs (the second chapter of your link does a pretty good intro to the tree like nature of this process) and let them loose. The vast majority of these strategies will be/are utterly terrible, things like 'go right once' or 'go left ten times'.

Say we make 100 random programs on our first run. We firstly score them on how well they did according to our fitness function (the random programs that did the best). We take a set % of these to survive and get rid of the rest, lets say 10% survive.

We take these surviving 10% of the best performing programs and use them to create new programs for the next generation by modifying them randomly again, but not completely. Say we randomly modify half their structure and leave the other half as is across however many we want for the next generation. We now let this generation loose again, and again rank, score, take the top 10% and breed a new generation from them and so on for n number of generations.

In this case, if we say left the grid as is, the program would generally come up with a rule roughly like 'go left x4, go up x4) as it solves this problem in the easiest way, but if we were to say, continuously randomise the position of 1s in the grid during this evolutionary process, we will force the program to come up with much more generalisable rules, such as checking the cells it can move into for 1s and not moving into any space containing a 1 etc.

Thus we can build a program with a flexible strategy able to cope with different environments for our robot in terms of number/configuration of holes - much more useful than having to program it for every configuration.

Just like with regular evolution, over millions of trials of taking the top performers and modifying them slightly, these programs become very specialised and high performing, able to solve highly complex games, paths with highly complex features etc.


How to actually teach the ANN the resulting weights of different training inputs?

Your network must have something which persist like weights add bias

Your new implementation would be like this :

trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)
weights = []

while(trainingData not empty) {
    for(0<iterations) {
       set = trainingData.pop()

       weights = updateInput(network, set.in)

       forwardPropagate(network, links)

       linkUpdate = backPropagate(network, links, set.out, weights)

       updateLinks(linkUpdate, links)}
}

Inshort retain weights, and backprapogate.


How should I encode the input which are 5 cards from a deck of 52 cards?

Depends on how your game is played. Is there any meaning assigned to the order of cards, or are all 5 played simultaneously? If order matters, use 5 one-hot vectors so you can choose how to order them, otherwise use a single 5-hot input vector. I would also add that if temporal order matters, you could also use a recurrent net with a 52-element input and feed the five one-hot vectors one after another.


What are some alternatives to "Papers with Code"?

Recently arxiv.org added a Code Tab towards the end of paper descriptions. Which contains links to both the official and community code.

I don't know if this is the case for all the papers or not till know. But I'm sure it'll be extended to all the papers in a short while.


I never used a k-WL in practice, but I did apply weisfeiler-lehman for my graph tasks.
As you can know, the WL provides the coloring by interactive procedure that's assign each node a 'color' (basically some kind of label reflecting the node neighborhood). Counting colors allows to compare two graphs on isomorphism, but it's not that important here, the key is that you have kind of feature for each node.
The k-WL make similar thing, but it operates not on graph, but in space of k-tuples of nodes. Basically, it's pairs or triplets of nodes. Then you define neighborhood as all pairs/triplets that differ in one element from the target pair/triplet. So, now you have have some kind of objects and defined neihgborhood for each of them, that's basically enough to apply the WL coloring.
Ok, now to k-GNN part. You repeat k-WL procedure on pairs of nodes, gets color for each of pair. You use one-hot encoded colors as input features and make graph convolution for pairs (with neighborhood defined as above), then getting some kind of embeddings for pairs. To get the node classification, you make average pooling by all pairs containing the node and then use usual fully-connected neural net.
As of the examples, the paper author provided several in article repo, you can try them.


Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):
MSE:

L2 loss is continuously-differentiable across any domain, unlike L1 loss.  This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator").  Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator.  If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian.  From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.


Why does the discriminator minimize the cross-entropy while the generator maximize it?

Your mistake is that you think that the referenced $V(D,G)$ is the deifinition of the cross entropy! Indeed, the cross entropy is defined base on the negative value of the $V(D,G)$. Hence, if you consider the minus behind the $V(D,G)$ ($-V(D,G)$) the sentence will be meaningful.


What is the intuition behind the attention mechanism?

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask someone to pick out the key words of the sentence, i.e. which ones encode the most meaning, they would likely say "dog" and "Labrador." Articles like "the" and "a" are not as relevant in translation as the previous words (though they aren't completely insignificant). Therefore, we focus our attention on the important words.

Attention seeks to mimic this by adding attention weights to a model as trainable parameters to augment important parts of our input. Consider an encoder-decoder architecture such as the one Google Translate uses. Our encoder recurrent neural network (RNN) encodes our input sentence as a context vector in some vector space, which is then passed along to the decoder RNN which translates it into the target language. The attention mechanism scores each word in the input (via dot product with attention weights), then passes these scores through the softmax function to create a distribution. This distribution is then multiplied with the context vector to produce an attention vector, which is then passed to the decoder. In the example in the first paragraph, our attention weights for "dog" and "Labrador" would hopefully become larger in comparison to those for the other words during training. Note that all parts of the input are still considered since a distribution must sum to 1, just some elements have more effect on the output than others.

Below is a diagram from Towards Data Science that illustrates this concept very nicely in terms of an encoder-decoder architecture.

The advantages of attention is its ability to identify the information in an input most pertinent to accomplishing a task, increasing performance especially in natural language processing - Google Translate is a bidirectional encoder-decoder RNN with attention mechanisms. The disadvantage is the increased computation. In humans, attention serves to reduce our workload by allowing us to ignore unimportant features, however in a neural network, attention entails overhead as we are now  generating attention distributions and training our attention weights (we are not actually ignoring the unimportant features, just diminishing their importance).