If IQ is used as a measure of intelligence in humans could it also be used as a measure of intelligence in machines?

It depends on how the IQ test is presented:

If as for humans (effectively, as a video of the book containing the
test questions being opened etc), then all AI programs would score
zero.
If presented as the test set of a supervised learning problem (e.g. as for Bongard Problems) then one might imagine that a number of ML rule induction techniques (e.g. Learning Classifier Systems, Genetic Programming) might achieve some limited success.

So all current AI programs require the problem to be 'framed' in a suitable fashion. It doesn't take too much thought to see that removing the need for such 'framing' is actually the core problem in AI, and (despite some of the claims about Deep Learning), eliminating framing remains a distant goal.

More generally (just as with the Turing test), in order for an IQ test to be a really meaningful test of intelligence, it should be possible as a side effect of the program's capabilities, and not the specific purpose for which humans have designed it.

Interestingly, there is only one program that I'm aware of that sits between 1. and 2.:

Phaeaco (developed by Harry Foundalis at Douglas Hofstadter's research group) takes noisy photographic images of Bongard problems as input and (using a variant of Hofstadter's 'Fluid Concepts' architecture) successfully deduces the required rule in many cases.


How can 3 same size CNN layers in different ordering output different receptive field from the input layer?

It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.

Lets take a simple example :

Dimensions are in the form of (channels x height x width)

Input Image - 3x5x5 (I)

Conv 1 kernel - 3x2x2 (K1) : we consider only 1 filter

Conv 2 kernel - 1x2x2 (K2)

Pad = 0 (P)

Stride = 1 (S)

The output dimensions are calculated by the following formula taken from the lecture CS231n.

Output = (I - K + 2P)/S + 1

When you do a convolution of the input image with the first filter K1, you get an output of 1x4x4 (this is the activation of CONV1 layer). The receptive field for this is same as the kernel size (K1) that is 2x2.
When this layer (1x4x4) is convolved with the second kernel (CONV2) 1x2x2 (K2), the output would be 1x3x3. The receptive field for this would be 3x3 window of the input because you have already accumulated the sum of the 2x2 window in the previous layer.

Considering your example of three CONV layers with 3x3 kernels is also similar. The first layer activation accumulates the sum of all the neurons in the 3x3 window of the input. When you further convolve this output with a kernel of 3x3, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.

This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts etc because they cover a larger receptive field of our original input image.

Hope this clarifies.


The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a numerical library, but you do need to be aware of the fundamental Boolean nature of the data. The $0$ and $1$ values allow the maths for calculating TP et al, but are not fundamental to it, they are a representation.

Your derivations of gradients for the functions you give seem correct, barring the odd typo. However, the gradient doesn't really apply to the value of $\mathbf{y}$, because all components of $\mathbf{y}$ are either $0$ or $1$. The idea that you could increase $\mathbf{y}$ slightly where  $\mathbf{\hat{y}}$ is $1$ in order to increase the value of the TP metric slightly has no basis. Instead the only valid changes to make an improvement are to modify FN values from $0$ to $1$ exactly.

You could probably still use your derivations as a gradient for optimisation (it would not be the only time in machine learning that something does not quite apply theoretically but you could still use it in practice). However, you then immediately hit the problem of how the values of $\mathbf{y}$ have been discretised to  $0$ or $1$ - that function will not be differentiable, and it will prevent you back propagating your gradients to the neural network weights that you want to change. If you fix that follow-on problem using a smoother function (e.g. a sigmoid) then you are likely to end up with something close to either cross-entropy loss or the perceptron update step.

In other words, although what you have been told is an over-simplification, you will not find a way to improve the performance of your classifier by adding cost functions based directly on TP, FP etc. That is what binary cross-entropy loss is already doing. There are other, perhaps more fruitful avenues of investigation - hyperparameter searches, regularisation, ensembling, and if you have an unbalanced data set then consider weighting the true/false costs.


What is the difference between TensorFlow's callbacks and early stopping?

Early stopping and callbacks are two different concepts:

Early stopping is a machine learning concept about when to stop training your model to avoid overfitting: You monitor a target value (e.g. validation loss) and stop learning after it hits a minimum. If the monitored value keeps increasing for a couple of epoch, you can restore the weights from the minimum.
Callbacks are a general software engineering concept and a technical implementation detail on how to trigger certain functions.

A callback function (e.g. early stopping) is a function that is passed as an argument to another function (e.g. fitting an ML model), and called in certain situations (e.g. after finishing an epoch). The callback function will perform its business (e.g. monitor the validation loss and decide whether to trigger the end of the training) and return control to the calling function (fitting the model).

In TensorFlow, early stopping is implemented using a callback function, but this is not the only way to do it. Also, there are further features in TensorFlow that are implemented using callbacks.


How to estimate a behavior policy for off-policy learning based on data?

You can simply train a policy from the inputs to predict the actions in your dataset. You can use the cross entropy loss for this, i.e. maximize the the log probability that the policy assigns to the actions in the data set when given the corresponding inputs. This is called behavioral cloning.

The result is an approximation of the behavioral policy that lets you compute probability densities of actions. It is an approximation because the dataset is finite, and even more so when you restrict the learned policy to a class of distributions, e.g. Gaussians.


Is GAIL applicable if the expert's trajectories are for the same task but are in a different environment?

The authors of the paper Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2018, published in ICRL), which introduced the inverse RL technique AIRL, argue that GAIL fails to generalize to different environment's dynamics. Specifically, in section 7.2 (p. 7), they describe an experiment where they disable and shrink the two front legs of the ant, then, based on the results, they conclude

GAIL learns successfully in the training domain, but does not acquire a representation that is suitable for transfer to test domains.

On the other hand, according to their experiments, AIRL is more robust to changes in the environment's dynamics.


How should I encode a categorical input?

As with many general questions about how to represent features, or best learn from them, the answer is "it depends".

If there is a natural sequence to the separate items, and that sequence is informative somehow for the prediction, then the classes may work best as a single feature which takes discrete values, scaled to the network. A good example might be predicting house prices where one of the features is property tax bands (in locations which have these such as UK Council Tax).
If the classes have no natural sequence to them relating to the problem, then one-hot-encoding is usually a better interpretation. An example of this might be predicting the price of a car based on the manufacturer.

In both cases, a neural network with enough layers and connections - and enough data - could resolve a more difficult representation, and make little effective difference between the representations in practice. However, if you know something useful about the feature, it is normal to choose the most "natural" representation based on that knowledge, and you will often see a small improvement by making the correct choice. In the first case, the neural network can benefit from requiring less parameters, and learn more efficiently.

If you are not sure, then choosing one-hot versus a single scaled input should be an experiment which you perform as part of your hyper parameter tuning, along with any other feature engineering that you are not sure of.


In online one step actor critic, why does the weights update become less significant as the episode progresses?

This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:

$$J(\theta) = v_{\pi_\theta}(s_0)$$

That is, the expected return from the first state of the episode. This is equation 13.4 in the book edition that you linked in the question.

In other words, if there is any discounting, we care less about rewards seen later in the episode. We mainly care about how well the agent will do from its starting position.

This is not true for all formulations of policy gradients. There are other, related, choices of objective function. We can formulate the objective function as caring about the returns from any distribution of states, but in order to define it well, we do need to describe the weighting/distribution somehow, it should be relevant to the problem, and we want to be able to get approximate samples of $\nabla J(\theta)$ for policy gradient to work. The algorithm you are asking about is specifically for improving policy for episodic problems. Note you can set $\gamma = 1$ for these problems, so the decay is not necessarily required.

As an aside (because someone is bound to ask): Defining $J(\theta)$ with respect to all states equally weighted could lead to difficulties  e.g. the objective would take less account of a policy's ability to avoid undesirable states, and it would require a lot of samples from probably irrelevant states in order to estimate it. These difficulties would turn up as a hard  to calculate (or maybe impossible) expectation for $\nabla J(\theta)$


These two are not "general" ways in which AI interact with humans , especially not the first one. In fact most of the applications we have complex AI algorithms deeply rooted in them , your search engine , smart-reply , translating systems all employ some kind of "AI" or to be more specific "machine-learning" algorithms in them. and many organizations are already using business-intelligence systems which you can call "AI" in some sense.


Why do we apply the mutation operation after generating the offspring?

The mutation operation is (usually) needed to introduce new genes not found in the population.

For example, suppose that you have 4 possible genes $A$, $B$, $C$, and $D$, and that your chromosomes have a non-binary encoding. In that case, if no member of your population has the gene $D$, then no amount of crossover operations will result in the introduction of that gene.

However, if your chromosomes use a binary encoding, then new genes could be introduced as a side effect of crossover operations. But it is always safer to have some kind of mutation to ensure that all genes can be accessed.

See also the paper Genetic Algorithm for Traveling Salesman Problem with Modified Cycle Crossover Operator, which mentions several non-binary encodings for the TSP.


Simulated Annealing: Why is e-function used as propability function to decide to accept a worse solution

You can find the explanation by asking some question about the function. Suppose, the value of $\frac{E(y)-E(x)}{kT} >> 0$ is much more greater than zero. What does it mean? It means the value of $E(y)$ is much greater than $E(x)$ related to the $kT$ that is as a measure of temperature decreasing. Now, you want in this situation a probability which is near to zero. Hence, $e^{-\frac{E(y)-E(x)}{kT}}$ could be a good value for the probability of selection of worse solutions!

Why $e$ instead of $2$ or other values greater than $1$? Because it could be a good function in optimization problems as its derivative is more simple than others!


Local minima.
You have the exact same issue of this question. If you randomize your initial weights, you'll see sometimes you get the correct results, and others you won't. It's because when the weights are initialized with a certain range of values, they will converge to a local minima which you cannot escape with a low learning rate.
A simple solution is to increase the size of your hidden layer, which will make the network more robust to such issues.

When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.
The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.


What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?

We can first consider what kind of elements are in this list.  They are all quantifications that guide a process of adaptation, learning, or innovation — controls a process that optimizes or attempts to optimize the development of another process.  Quantification allows for comparison and the determination of first and second-degree derivatives to use in approximating steps in improvement.
On the basis of this generalization, the list may be expanded to include other potential quantifications that can control.

Pain
Pleasure
Disparity
Value
Advantage
Expense
Profit
Gain
Return
Distance
Burden
Optimality

In English, there is quite a bit of nuance in terms of normal contextual use.  In academic papers on machine learning topics there are some conventions forming there, producing jargon, meanings beyond the normal English use.  We can first divide them symmetrically around the most basic qualifier.  Is the increase in the number what is intended or something to be avoided?
Decreases are Desirable

Cost
Loss
Error
Distance
Pain
Disparity
Expense
Burden

Increases are Desirable

Fitness
Utility
Objective
Criterion
Pleasure
Value
Advantage
Profit
Gain
Optimality
Complete

Other Features
Some of these are relative, such as Advantage being a relative measure between two Values associated with two Action options.  Some are direct inversions, as in that the quotient of Gain over Loss equals -1 in the usual contexts of those words.  Some appear to be opposite but in fact are not, as in the case of pain and pleasure occurring concurrently and via different neural mechanisms in biological neural systems.
Some are discrete and have only two possible values, such as Criterion and Complete.  Those can be well represented by Boolean values rather than integers, fixed-point, or floating-point numbers.
Some are conditionally synonymous, as in the case of Distance and Error if the output dimensionality is one or the output is multidimensional but the error aggregation method used is the square root of the sum of squares.
Some seem better in financial contexts ... others in robotic contexts.
Words like Disparity and Optimality are generalizations.
One, in particular, is a quotients based on investment &mdash Return — which could be applied to energy return on energy invested (as Hall suggests is the fitness function for genetic evolution).
What is missing from this list is learning based on multiple degrees of freedom, not used in mainstream ML yet, and probably will emerge in papers and then libraries and then VLSI designs in time.  This likely direction would introduce the plurals of many of these quantifications because it will be vectors of corrective signaling back-propagating to perform curved modifications on network parameters.  It is likely that brain signaling (pulses, energy balances, neurochemical secretions and uptake, and genetic expression) will not be well simulated until this happens.
In all cases, it is the definition of these quantifications that drive adaptation, learning, and innovation in biological and artificial systems.  It is thus likely that the application of ethics and risk analysis might best be applied to AI in the formulation of the calculation of these quantities in adaptive systems.


It can make sense to incorporate the standard deviation of Monte-Carlo-based evaluations in some ways to reduce risk, but I don't think the way of using it that you described would work well.

For MCTS evaluations, if you're taking a zero-sum-style approach (which you are if you are trying to estimate win/loss probabilities), it is very important that your estimates are "symmetric" with respect to the players. If you evaluate the probability of winning for player A to be $p$, it is important that your algorithm simultaneously evaluates the probability of the other player B to be equal to $1 - p$ (or, if you prefer winning chance, if the chance of winning for one player is $p\%$, it should be $100\% - p\%$ for the other player). This appears to be violated by your idea, which subtracts standard deviation regardless of player "perspective".

A better place to take standard deviation (or, similarly, variance) into account would be, for example, the Selection phase of MCTS. The most common strategy for the Selection phase is using the "UCB1 equation". You can modify that to include the variance in your observations, for example using the "UCB1-Tuned" strategy as described in the beginning of Section 4 of Finite-time Analysis of the Multiarmed Bandit Problem.

In my answer above, I assume that you were talking about evaluations "inside" the algorithm, while it is still running. If you were rather thinking of the final move selection for the "real" game after having run the algorithm for a while, the most common approach there is to simply play the move with the maximum number of visits (also referred to as robust child), rather than playing the move with the maximum score. It should not be necessary to include standard deviation at this stage anymore.


Are PAC learning and VC dimension relevant to machine learning in practice?

Yes, PAC learning can be relevant in practice. There's an area of research that combines PAC learning and Bayesian learning that is called PAC-Bayesian (or PAC-Bayes) learning, where the goal is to find PAC-like bounds for Bayesian estimators.

For example, Theorem 1 (McAllester’s bound) of the paper A primer on PAC-Bayesian learning (2019) by Benjamin Guedj, who provides a nice overview of the topic, shows a certain bound that can be used to design Bayesian estimators. An advantage of PAC-Bayes is that you get bounds on the generalization ability of the Bayesian estimator, so you do not necessarily need to test your estimator on a test dataset. Sections 5 and 6 of the paper go into the details of the real-world applications of PAC-Bayes.

See e.g. Risk Bounds for the Majority Vote: From a PAC-Bayesian
Analysis to a Learning Algorithm (2015) by P. Germain et al. for a specific application of PAC-Bayes. There's also the related Python implementation.

See also these related slides and this blog post (by the same author and John Shawe-Taylor) that will point you to their video tutorials about the topic.

The VC dimension can also be useful in practice. For example, in the paper Model Selection via the VC Dimension (2019) M. Mpoudeu et al. describe a method for model selection based on the VC dimension.


What are the variables used in a Gaussian radial basis kernel in the context of SVMs?

$\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{x}' \in \mathbb{R}^p$ are two inputs (or feature vectors).

In the context of classification with an SVM, you are given a dataset $D = \{(\mathbf{x}_i, y_i) \}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^p$ is an input (or point) and $y_i$ the corresponding label. The goal is to find a hyperplane that classifies the points $\mathbf{x}_i$. The hyperplane actually corresponds to a binary classifier that splits the plane into two, so the assumption is that there are two labels. However, these points $ \mathbf{x}_i$ may not be linearly separable in $\mathbb{R}^p$, i.e. there may not be a hyperplane (in 2d, i.e. when $p=2$, a hyperplane is a line) that separates them. The kernel trick, i.e. the use of kernels (such as the Gaussian radial basis), allows an SVM to perform non-linear classification by transforming the inputs to a space where they are linearly separable.


What is the purpose and benefit of applying CNN to a graph?

There are some problems that involve graphs and manifolds (sometimes collectively called non-Euclidean data), such as molecule design and generation, drug repositioning, social networks analysis, brain imaging, fake news detection, recommender systems, neutrino detection, computer vision and graphics and shape (e.g. hand or face) completion or generation (generative models).

The main benefit of geometric deep learning (deep learning applied to graphs and manifolds) is that you do not lose the information encoded in the graphs (or manifolds), which, otherwise, you would likely lose because you would need to convert your graphs (or manifolds) to an equivalent vector representation that you can feed into the existing CNN or other standard neural networks.

Note that you cannot directly apply the usual convolution operation to graphs, because, for example, graphs do not have the notion of relative positions of the nodes. Furthermore, note that graph networks have little to do with CNNs, even if they are sometimes called graph convolution networks.


Unable to understand V* at infinite time horizon using Bellman equation for solving MDP

Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).


How to assign rewards in a non-Markovian environment?

Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is to try and make the agent's representation of it Markovian.

After reducing Agent's model of the dynamics to a Markovian process, rewards are assigned from the environment in exactly the same way as before. The environment simply sends the agent a reward signal in response to each action.

The Markovian assumption is essentially a formalism of the idea that the future can be predicted from the present. It says that if you know the dynamics of a system, and you know the state of the system now, you know everything you need to predict the state of the system later, and how we got to this state is not important. Formally, we write this as $P(s_t∣s_{t−1:0})=P(s_t∣s_{t−1})$.

That said, the models we use in AI are usually simplifications of the real world. When we simplify the world, we can introduce non-Markovian dynamics. However, if the model grows too complex, and the state space too large, learning will take too long. The goal is then to define a state space that is small enough to be learnable, and not too bad an approximation of the real dynamics of the world. AI researchers have several tools to do this.

As a working example, imagine that the future position of a robot depends mainly on the current position, and current velocity, along with the action the robot takes right now. Using these variables to define a state, we get almost Markovian dynamics, but as the robot moves over time, its battery drains and the movements become very slightly more imprecise. If we wanted to remove this error, we can:

Expand the state variables. If we add "current battery level" to the state, then our process will become Markovian again. This is the most obvious approach, but it only works up to a point. The state space grows exponentially in size as you add new variables. It will quickly become too large to learn, so in many problems, a complex state is subsequently decomposed into simpler sub-states. These simpler states may not depend on one another at all, or may depend only to varying degrees. The main limiting factor in learning to navigate an exponential state space is that the number of parameters in our model will grow exponentially. If the original space had $O(2^n)$ parameters, splitting it in half will yield two seperate learning problems of size $O(2^{n/2} + 2^{n/2} = 2^{n/2 + 1})$. Splitting the problem in two reduces its size by a factor $O(2^{n/2})$, which is big. This is the technique exploited by Dynamic Bayesian Networks. In the case of our robot, we could make current $x$ position depend only on previous $x$ position, previous $x$ velocity, and the robot's action, rather than on all previous variables, and likewise for the other variables.
Use Higher Order Models of the Environment. This is really a generalization of (1) above. Instead of the state being the current location and velocity of the robot, we could define the state to be the current location/velocity and the location/velocity of the robot 1 step in the past (or 2, or 3, or $k$ steps). This increases the size of the state space enormously, but if we don't know why there is an error in the robot's movements, this can still allow us to model it. In this case, if we know the size of the change in the robot's position last time, and we observe that it is smaller this time (for the same action), we can estimate the rate at which the change is changing, without understanding its cause.

As an example, consider the process of setting the price of a good. The agent's reward is non-Markovian, because sales increase or decline gradually in response to price changes. However, they don't depend on all of the history of prices. Imagine that instead they depend on the last 5 prices (or the last k). We can use technique (2) above to expand the agent's model of what a state is. The now the agent learns that when prices have been $p_{t-1:t-5}$ in the last 5 steps, and it sets the price at time $t$ to some value, it's reward is $x$. Since the reward depends only on the prices now, and in the last 5 steps, the agent is now learning a Markovian process, even though the original process is non-Markovian, and the reward function is non-Markovian. No changes are made to the reward function, or the environment, only the agent's model of the environment.


What made your DDPG implementation on your environment work?

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment:

Reducing the neural network size, compared to the original paper. Instead of:

  2 hidden layers with 400 and 300 units respectively

I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this.

As suggested in the paper, I added batch normalization:

  ... to manually scale the features so they are in similar ranges across
  environments and units. We address this issue by adapting a recent technique from deep learning
  called batch normalization (Ioffe & Szegedy, 2015). This technique normalizes each dimension
  across the samples in a minibatch to have unit mean and variance.

The implementation you used does not seem to include this.

Reducing the value of $\sigma$, which is a parameter of the Ornstein-Uhlenbeck process used to enable exploration. Originally, it was $0.2$, I used $0.05$. (I can't find where this parameter is set in your code.)

I am not entirely sure if this will help in your environment, but it was just to give you some ideas.

PS: Here is a link to the code I followed to build DDPG and here is a plot of rewards per episode.


What part of the game is the value network trained to predict a winner on?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to likelihood of winning.

Yes, although the data for this supervised learning problem was provided by self-play. As AlphaZero learned, the board evaluations of the same positions would need to change, so this is a non-stationary problem, requiring that the ML forgot the training for older examples over time.

  What part of the game is the network trained to predict a winner on?

Potentially all of it, including the starting empty board. I am not sure if the empty board was evaluated in this way, but it is not only feasible, but can be done accurately in practice for simpler games (Tic Tac Toe and Connect 4 for example), given known player policies.

  Obviously after only five moves, the winner is not yet clear, and trying to predict a winner after five moves based on the game's eventual winner would learn a meaningless function.

Not at all. This is purely a matter of complexity and difficultly. In practice at such an early stage, the value network will output something non-committal, such as $p=0.51$ win chance for player 1. And it will have learned to do this, because in its experience during self-play similar positions at the start of the game lead to almost equal numbers of player 1 and player 2 winning.

The function is not meaningless either, it can be used to assess results of look-ahead searches without needing to play to the end of the game. It completely replaces position-evaluation heuristics as used in more traditional game tree searches. In practice, very early position data in something as complex as chess or go is not going to be as useful as later position evaluations, due to ambivalent predictions. However, for consistency it can still be learned and used in the game algorithms.

  How is the network trained to understand that, if all it is told is who eventually won?

If a supervised learning technique is given the same input data $X$ that on different examples predicts the labels $A, B, B, B, A, A, B, B$, then it should learn $p(B|X) = 0.625$. That would minimise a cross-entropy loss function, and is what is going on here.


Beginner - Object classification data in a neural network

The answer is yes, retraining a neural network on a new dataset will alter its internal state such that it would no longer give the same output as before (in your case 'messing it up').

There are techniques to allow you to reuse sections of trained networks for other problems (this is called 'Transfer Learning'). Transfer Learning involves freezing the weights/biases for parts of the already trained network while adding extra new layers to the end of it and training those (with Back-propagation) for your new problem. For your digit recognition problem you would normally just train the network on all the digits at the same time.

Tensorflow is lower level than the idea of neural networks and layers. It is a library that allows you to write mathematical pipelines that implement the tensor calculations required to build a neural network. To do the sort of thing that you mentioned you would use a higher level library such as Keras (https://www.tensorflow.org/guide/keras) which now comes conveniently bundled with Tensorflow.


How to train a neural network for a round based board game?

Great question!  NN is very promising for this type of problem: Giraffe Chess.  Lai's accomplishment was considered to be a pretty big deal, but unfortunately came just a few months before AlphaGo took the spotlight. (It all turned out well, in that Lai was subsequently hired by DeepMind, although not so well for the Giraffe engine;)

I've found Lai's approach to be quite helpful, and it is backed by solid results.

You may want to use "sequential" as opposed to "round based" since sequential is the preferred term in Game Theory and Combinatorial Game Theory, and these are the fields that apply mathematical analysis to games.

The games you list are further termed "abstract" to distinguish them from modern strategy boardgames, or games in general, which utilize a strong theme and are generally less compact than abstract games in terms of mechanics and elements. This carries the caveat that abstract games are not restricted to sequential games or boardgames, or even games specifically, as in the case of puzzles like Sudoku.

The formal name for this group of games is generally "partisan, sequential, deterministic, perfect information" with the further categorization of Tic-Tac-Toe as "trivial" (solved and easily solvable) and non-trivial (intractable and unsolved) for games like Chess and Go.


Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?

You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:

https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c

To run the notebook click Open in playground and run the code. There is a commented line which add 1 extra layer. The syntax should be easy to understand even though it is in python.

For back propagation, you can try to see this python implementation of multi layer perceptron backpropagation.

https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb

A network will not usually decrease it's accuracy by almost a half in normal scenario when you add an extra layer, though it is possible to have the network decrease accuracy when you add an extra layer due to overfitting. Though if this happen the performance drop won't be that dramatic.

Hope I can help you.


Make 9 AIs to replace Supreme Court justices

Politics are driven by a combination of things such as culture, ideology, religion, ethics, morality, desire for money, fame, and power, etc.  Although God has one truth, most people don't know this truth, a few do, and the rest either disagree about it or just don't believe in Him. The Constitution was written by man.  It does not tell us what to do in every situation. It is open to interpretation.  A person's interpretation of the Constitution is driven by his morality, culture, and all the things I mentioned above. AI systems are designed by people so they are driven by their biases.


In deep learning, is it possible to use discontinuous activation functions?

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition shows how to train such networks on pages 170-174.

Error backpropagation algorithm can be modified to accommodate discontinuous activation functions. The details are in paper [2]. That paper points out a possible application: training a neural network on micro-controllers. As the multiplication of the output of the previous layer $x_j$ by the weigth $w_{ij}$ is exspensive, the author suggested to approximate it with a left shift by $n$ bits (multiplication by $2^n$) for the corresponding $n$ in which case the activation function is discontinuous (a staircase).

An example of a neural network with discontinuous activation functions applied to a restoration of degraded images is in Ref. [3]. Applications of recurrent neural networks with discontinuous activation functions to convex optimization problems are in Ref. [4]. Probably more examples can be found in the literature.

References

Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev. 1958 Nov; 65(6):386-408. PMID: 13602029 DOI: 10.1037/h0042519
Findlay, D.A. Training networks with discontinuous activation functions. 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313), London, UK, 1989, pp. 361-363.
Ferreira, L. V.; Kaszkurewicz, E.; Bhaya, A. Image restoration using L1-norm regularization and a gradient-based neural network with discontinuous activation functions. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 2512-2519.  DOI: 10.1109/IJCNN.2008.4634149
Liu, Q.; Wang, J. Recurrent Neural Networks with Discontinuous Activation Functions for Convex Optimization. Integration of Swarm Intelligence and Artificial Neural Network, pp. 95-119 (2011) DOI: 10.1142/9789814280150_0004


In LSTM text generation can low amount of training data be compensated?

A.) The algorithm can only learn patterns from the data, so if you want it to align to a specific functional form of language, it would make sense to curate your data to only contain sentences of that very structure.

If you tune your hyper-parameters such that the model is not as complex as the default, it should work for datasets that are smaller. But this relies on the observation that the thing you are trying to learn isn't as complex (as opposed to learning from a large corpus of Nietzsche).

B.) What is happening is that you are overfitting the data, such that the LSTM isn't generalizing to your intended goal. In essence, overfitting means that your model is learning irrelevant details that by chance happen to predict the intended goal in the training data. To alleviate this, use a validation set.

And finally (but very importantly), these machine learning techniques don't learn the same way as humans do. To understand why you'd need to learn the math behind NNs, and I can't think of an intuitive explanation as to why.


Can neural network help me with detecting center coordinates of particles in an image?

This is a very hard problem, you have many overlapping points with objects which aren't completely round. I'm not very knowledgeable on CV but I suspect you will find it very challenging.

I would probably say a handcrafted detection algorithm would probably be easier, something like an edge detector which fit circles to arcs and labeled the points. But it's still going to be nontrivial and maybe impossible to get it working with high accuracy.


Unsupervised learning to optimize a function of the input

According to your description, you already know your function $f$ to be optimized. So you should use it directly instead of the standard loss functions. In this other post there is an explanation of how to use $f$ as a custom loss function in Keras.


How does Ubers rating System algorithm work?

I highly recommend that you start reading on the Netflix challenge. It has tonnes of useful and interesting examples dealing with this sort of thing.
You will need an Algorithm that builds a score on both 'quality' and 'quantity'. That is, it needs to add a 'weight' to the final rating based on the number of reviews that an individual has. This is so that, for example, an individual with 50 8 score reviews would be rated higher than an individual with only one 9 score review.
I recommend that you implement Bayesian estimates to calculate weighted voting.
IMDb (Internet Movie Database) utilizes this algorithmn to determine its IMDB top 250 movies. (Robert C 2010)

The formula for calculating the Top Rated 250 Titles gives a true Bayesian estimate:
weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 3000)
C = the mean vote across the whole report (currently 6.9)

Please note that in addition to the rating Algorithm, Uber has a dispatch algorithm which takes into consideration factors i.e. Drivers who are online and drivers who are nearest to the passenger.


Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always predicts 0 and the agent doesn't collect any reward, meaning very poor behavior.


What ML algorithm should I use that suits this data?

Regression Model will definitely work on That problem.
You only need to change shape of predicting variables like(Day, Weather, people arrived) into 1D array if you got error..
Otherwise you can simply apply Linear Regression, SVM etc to get your output with good accuracy.


Is the next state drawn from the joint distribution of the previous state and action?

Question 1. Are the states $s'$ drawn from a from a joint probability distribution $P_{sa}$? In other words, if you are in an
  initial state $s$, take an action $\pi(s)$, then $s'$ is the random
  state you would end up in according to the probability distribution
  $P_{sa}$?

This is tricky, because you don't show a definition of $P_{sa}$. My first thought was that you meant the transition matrix $P_{ss'}^a$, but that doesn't fit with the phrase joint probability distribution.

If you really mean joint probability distribution then the answer is generally "no", because $P_{sa}$ should be the probability of observing state $s$ and action $a$ when taking a random sampled time step:

$$P_{sa} = \pi(a|s)\rho_{\pi}(s)$$

where $\rho_{\pi}(s)$ is the distribution of states under policy $\pi$. Note this makes no reference to $s'$ at all.

However, there are ways that this could relate to the distribution of $s'$. Probably the most direct relationship would be when there is a deterministic environment, thus knowing $s$ and $a$ would determine $s'$. If, in addition to that, each $s'$ could only be reached from a single $(s,a)$ combination, then knowing $P_{sa}$ would also give you knowledge of $P_{s'}$ - this is not the same thing as the question is asking though.

If you did mean the transition matrix $P_{ss'}^a$ instead in the question, then the answer is yes, because

$$P_{ss'}^a = \sum_r p(r,s'|s,a)$$

  Question 2. What are the advantages of looking at $q_{\pi}(s,a)$ versus $v_{\pi}(s)$?

The main advantage is that you can derive a policy more easily from $q_{\pi}$:

$$\pi'(s) = \text{argmax}_a q_{\pi}(s,a)$$

Compare with deriving a policy using $v_{\pi}$:

$$\pi'(s) = \text{argmax}_a \sum_{s',r} p(r, s'|s,a)(r + \gamma v_{\pi}(s'))$$

Note that these policies are not necessarily the same as $\pi$ on which the $q$ or $v$ values are evaluated. In fact this is a common situation whilst searching for an optimal policy, and it is possible to show that $\pi'(s)$ will result in same or higher returns as $\pi(s)$ across all states . . . the proof of this is called the policy improvement theorem.

The important thing about the first equation using $q$ is that it does not involve using the MDP model $p(r, s'|s,a)$ directly. This is the basis of model-free RL. Whilst the version using $v$ is more complex (taking more computation) and requires that you know $p(r,s'|s,a)$.

The main disadvantage of looking at $q_{\pi}(s,a)$ is that it has a larger dimension, it maps $S \times A \rightarrow \mathbb{R}$, as opposed to using $v_{\pi}(s)$ which maps $S \rightarrow \mathbb{R}$. So it can take longer to get good approximations of $q$ compared to $v$.


How to use CNN for making predictions on non-image data?

You can use CNN on any data, but it's recommended to use CNN only on data that have spatial features (It might still work on data that doesn't have spatial features, see DuttaA's comment below).
For example, in the image, the connection between pixels in some area gives you another feature (e.g. edge) instead of a feature from one pixel (e.g. color). So, as long as you can shaping your data, and your data have spatial features, you can use CNN.
For Text classification, there are connections between characters (that form words) so you can use CNN for text classification in character level.
For Speech recognition, there is also a connection between frequencies from one frame with some previous and next frames, so you can also use CNN for speech recognition.
If your data have spatial features, just reshape it to a 1D array (for example in text) or 2D array (for example in Audio). Tensorflow's function conv1d and conv2d are general function that can be used on any data. It look the data as an array of floating-point, not as image/audio/text.
But if your data doesn't have spatial features, for example, your features are price, salary, status_marriage, etc. I think you don't need CNN, and using CNN won't help.


How should the values of the filters of a CNN change?

The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $-\infty$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $-\infty$. You will have to increase one or more weights.


Should you reload the optimizer for transfer learning?

When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.


Is there any literature on the design of dialogue systems for interviews and questionnaire administration?

[Disclaimer: I work for a company that provides a platform for developing conversational AI systems]

The platform used by the company I work for has a sentiment analysis component, so you can recognise if the user input expresses certain emotions. The dialogues are encoded in 'flows', which are graphs with an initial trigger consisting of output nodes and transitions that correspond to user input. In order to react to emotions expressed by the user you'd have a trigger that does not react to the user input text, but instead to a setting of the sentiment flags.

For example, in a call centre bot, if the user repeatedly uses angry sentiment in their utterances, the bot could detect this and trigger a flow that says "It seems you are not happy with my responses -- please wait while I transfer you to a human agent".

This all depends on the mechanics of organising the dialogues. If you have a ML model, this will probably not be as easy as if you have a manually designed graph structure (which gives you more control over such matters).

Being a commercial system, there are obviously no academic papers on this. But as it's so easy to do within the platform, it doesn't seem too much of a problem in the first place.


How do I check that the combination of these models is good?

Goodness is subjective.  Reliable knowledge isn't possible with that flimsy a quality objective.

The sturdy objective criteria you gave is 95%, so it is bad by that criteria.  (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)

However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy.  (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.)

I have four questions.  (My apologies that this question leads to the two assumptions above and a bunch more questions.)

Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?
Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?
Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?
What economic, contractual, or operational hard stop is dictating the 90%?

If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.


What kind of problems require more than 2 hidden layers?

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal Approximation Theorem.

Finding the best topology for a given problem is an open research problem. As far as I know, there are few universal 'rules of thumb' for this.

For a given problem, one option is to apply a neuroevolutionary approach such as NEAT, which attempts to find a topology that works well for the problem at hand.


Is there more than one Q-matrix update formula?

No, your second statement does not correctly implement the Q-learning update rule, which the first statement correctly implements.


When is it necessary to manually extract features to feed into the neural network rather than providing raw data?

Feature engineering may be necessary when one cannot achieve acceptable error rate — within a budget or in principle.
NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That's why starting fat and pruning later deems superior, although not always affordable.)
Not a necessity, but a no-brainer nonetheless: having strong insight into underlying processes is a clear call for feature engineering. Let's say, it's apparent that known meaningful transformation can hardly be approximated by (reasonably small) subnetwork: mapping between periodic and non-periodic functions, as an example.
Personally in my check-list, feature stage is coupled with a reminder to search for a domain expert. In some practical sense features are being extracted from people rather than data. Our neural networks are pretty good too!


Fundamentally, what is a perfect language model?

One of your hypothesis is very close to the truth, it's 1.2
So, a language model measures the probability of a given sentence in a language $L$. The sentences can have any length and the sum of probabilities of all the sentences in the language $L$ is 1. It's very difficult to compute, thus people use some simplifications, like say if the words are located far enough from each other, then the occurrence of a current word doesn't depend on a word which was occurred far away in the past.
Each sentence is a sequence $w_1, \dots, w_n$ and a language model computes the probability of the sequence $p([w_1, \dots w_n])$ (it's not joint distrribution yet). It can be decomposed into a joint distribution with some special tokens added $p(BOS, w_1, \dots w_n, EOS])$. BOS is begin of the sentence and EOS is end of sentence. Then this joint distribution can be decomposed using the chain rule $p(BOS, w_1, \dots w_n, EOS]) = p(BOS) p(w_1 | BOS) \Big[ \prod\limits_{i=1}^n p(w_i | BOS, w_1, \dots, w_{i-1}) \Big] p(EOS | BOS, w_1, \dots, w_n)$. There are 2 types of probabilities that are usually modelled differently: a prior probability $p(BOS)$ which is always equal to 1, because you always have BOS as the first token in the augmented sequence. Then conditional probabilities can be computed as follows $p(w_i | BOS, w_1, \dots, w_{i-1}) = \frac{c(BOS, w_1, \dots, w_{i-1}, w_i)}{\sum_{w_i \in W} c(BOS, w_1, \dots, w_{i-1}, w_i)}$. Where $c$ is a counter function that measures how many times a given sequence occured in the dataset you specified to train your model. You can notice it's a maximum likelihood estimate of the unknown conditional probabilities. Obviously if you're using a certain dataset you compute a model of that dataset, not of a language, but that's the to approximate true probabilities of sentences in a language.
The EOS token is needed to make difference between a probability of a non-finished yet sequence and that which has finished, because if you take those counters from above and forget about adding the EOS into your dataset in the end of all sentences, you'll get probabilities that don't sum into 1 (which is bad).


Sutton and Barto 2nd Edition Exercise 13.1

There are problems with both the approach and the expressions that you have. I don't want to just give the correct solution, though, that's an exercise for you to go through and learn from your own experience trying to accomplish it. Instead, let me illustrate that your expressions for $v(s_i)$ are wrong. To do that we'll just do a Monte-Carlo estimate for a range of values of $p$ and compare to your expression.
Here's the Python code that runs a single payout starting from state state and following the policy with probability p of choosing "right". It returns the collected reward:
import numpy as np

def playout(state, p):
   reward = 0
   while state != 0:
       action_right = np.random.rand() < p
       move_right = action_right if state != 2 else (not action_right)
       state = state - 1 if move_right else state + 1
       state = 3 if state > 3 else state
       reward -= 1
   return reward

Then we make a simulation code that runs the playout multiple times, collects the reward counts, and returns the average reward (so, essentially estimates $v^\pi(s_i)$):
from collections import defaultdict

def simulate(state, p, n):
    rewards = defaultdict(lambda : 0, {})
    for _ in range(n):
        rewards[playout(state,p)] += 1
    results = np.array([[a,b] for a,b in rewards.items()]).T
    reward , nplayouts = results[0] , results[1]
    value = (reward * nplayouts).sum() / nplayouts.sum()
    return value

Finally, I make a grid in p and run each playout 10000 times:
p = np.linspace(0,1,51)[1:-1]
v3 = [simulate(3,p,10000) for p in p]
v2 = [simulate(2,p,10000) for p in p]
v1 = [simulate(1,p,10000) for p in p]

I've plotted the resulting value estimates for each state. Together with your expression for them (blue curves). And the correct expression (red curve) that I've obtained by actually writing down and solving the equations:

As you can see, the expressions you've presented are all too far off from the results that the simulation returns. More than that - the asymptotic behavior of your solutions at $p\to0$ and $p\to1$ doesn't make much sense.
I've obtained the expressions for the red curves above by solving the system for $v(s_i)$ on my own account - without relying on weird and wrong solutions that I googled on the internet. Which I'd recommend you do as well.
Finally, the question of the exercise is to find an optimal $p$ for a policy that starts at $s_3$ - not "irrespective of starting state" as you've thought is should be. Unlike your expression the correct expression for  $v(s_3)$ has a maximum, which can be found analytically and it is
$$\max_p v(s_3) = -6-4\sqrt2 \simeq -11.6$$
$$ \text{at}\quad p = ??? \simeq 0.59$$


What is the difference between a normal processor and a processor designed for AI?

Parallel processing is very important when computing ANNs. Though they are not designed for AIs, GPUs are widely used in machine learning because of their higher capability of paralel processing and it is due to their higher number of cores comparing CPUs.

All in all, the dedicated hardware should have GPU-like architecture.


Why we don't use importance sampling in tabular Q-Learning?

In Tabular Q-learning the update is as follows

$$Q(s,a) = Q(s,a) + \alpha \left[R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a) \right]\;.$$

Now, as we are interested in learning about the optimal policy, this would correspond to the $\max_aQ(s',a)$ term in the TD target because that is how the optimal policy chooses its actions - i.e. $\pi_*(a|s) = \arg\max_aQ_*(s,a)$, so eventually the greedy TD update would be greedy with respect to the optimal state-action value function due to the guaranteed convergence of Q-learning.

The action $a$ in the update rule, i.e. the action we chose in state $s$ to receive the reward $R_{t+1}$, was chosen according to some non-optimal policy, e.g. $\epsilon$-greedy. However, as the $Q$ function is defined as the expected returns assuming we are in state $s$ and have taken action $a$ -- we thus don't need an importance sampling ratio for the $R_{t+1}$ term, even though it was generated from an action that the optimal policy might not have taken, because we are only updating the $Q$ function for state $s$ and action $a$, and by the definition of a $Q$ function it is assumed that we have taken action $a$ as we condition on this.


Why is my validation/test accuracy higher than my training accuracy

It is a bit rare that the validation and test  accuracy exceed the training accuracy. One thing that could cause this is the selection of the validation and test data. Was the data for these two sets selected randomly or did you do the selection yourself? It is generally better to have these sets selected randomly from  the overall data  set. That way the probability distribution in these sets will closely match the distribution of the training set. Normally the training accuracy is higher (especially if you run enough epochs which I see you did) because there is always some degree of over fitting which reduces validation and test accuracy. The only other thing I can think of is the effect of Dropout layers. If you had dropout layers in your model and the drop out ratio was high that could cause this accuracy disparity. When the training accuracy is calculated it is done with drop being active. This can lower the training accuracy to some degree. However when evaluating validation accuracy and test accuracy drop out is NOT active so the model is actually more accurate. This increase in accuracy might be enough to overcome the decrease due to over fitting. Especially possible in this case since the accuracy differences appear to be quite small.


In classification, how does the number of classes affect the model size and amount of data needed to train?

The most obvious way more classes increase the network size it the output layer, but I don't believe there is a rule of thumb for the size of the entire network.
As I understand it, there is no clear answer how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.
Regarding the data needed, in The Deep Learning Book (https://www.deeplearningbook.org/) (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need abut 10 million labeled examples.


How to represent the weights of a neural network as binary strings for a genetic algorithm?

I would first say consider the advice of Thomas W in the comment above and think about whether you really need to discretize your variables. I'd also question the wisdom of training a reasonably sized network with a GA instead of a dedicated neural net training algorithm that's very likely to exhibit much better performance. However, assuming you really do need to do this, the basic approach is generally to do the following.

Decide on your range and precision. For the sake of simplicity, I'm going to assume you use say $L=16$ bits to represent every weight. You can have some variables encoded at a higher precision than others -- you're in charge of writing the encode/decode functions, so it's entirely up to you, but here, I'm assuming you don't. Your encoding will now be a vector of length $16n$, where $n$ is the number of weights in the network. You can use a string if you like, but it's probably a net loss of efficiency, as mostly what you'll be doing is converting those bits to integers so that you can do arithmetic necessary to decode the encoded weights, so I'd just make it a vector of $16n$ integers.

For the range, you just need the minimum and maximum values you're going to allow. Again, for simplicity, I'm going to say the values range between $-10$ and $10$, denoted as $R_{\min}=-10$ and $R_{\max}=10$.

Your GA then just manipulates the $0$s and $1$s. To evaluate fitness, you have to decode each of those $16$-bit strings into a double between $-10$ and $10$. With $16$ bits, you have $2^{16}=16536$ possible values. The total range of output values is $10-(-10)=20$. $20/16536=0.0012$. Let $0000000000000000$ be the smallest possible string, and map that onto the value $-10$. Now, every increment to that binary string just adds $0.0012$ to the output value.

Putting all this together, you have $w_i = (R_{\max} - R_{\min})/2^L + R_{\min}$. Do that for each block of $16$ bits giving you each weight in the network. Run your network on your training data and use whatever accuracy metric as the fitness function for the GA.

For the GA itself now, you don't really have to do anything special. The GA can just see an arbitrary string of $16n$ bits. You can do crossover and mutation on just the apparently meaningless string of bits.

Your question was asking about going from a vector of weights to a vector of strings, which is the opposite conversion from that described above. Generally, for a GA, you don't need a phenotype->genotype conversion. You just generate genotypes using genetic operators and use the genotype->phenotype conversion to evaluate candidate solutions. However, there are some reasons why you might want to go the other way. For instance, you might want to seed the population with a known set of weights, and so you'd need the ability to convert those weights to binary.
Basically, in that case, just invert that conversion function. The idea is to convert the floating-point weight between $R_{\min}$ and $R_{\max}$ into the integer that's closest to the same distance into the range when mapped to $0$ to $2^{L}-1$. For example, if the weight is $-10$, then it's the smallest possible value. It maps onto the smallest possible value between $0$ and $2^{L}-1$, namely $0$. Similarly, a weight of $0$ maps onto $(2^{L}-1)/2$, because it's halfway between the min and max of both ranges. If you play around with it a bit, you should see that the expression here is given by $\big \lfloor \frac{w_i - R_\min}{R_\max - R_\min} 2^L \big \rfloor$. That gives me an integer in the range $0$ to $2^L$. From there, just do a standard binary encoding (or gray code, whatever) to get into a bit string you can manipulate.


In the case of invalid actions, which output probability matrix should we use in back-propagation?

I am also quite new to this field, but I think you should use the normalized outputs for the backpropagation. In general, you would want to backpropagate all the calculations you did in the forward pass, so why would you want to exclude the step of the normalization in your backward pass? This would essentially make the renormalization to have no effect (different loss values but no different model weight update).
For example, in policy gradients, you backpropagate through the log propability of the selected action. In the forward pass, the sampling of the probability (that determines which action is selected) is not affected by the renormalization (you might just get different loss values in the end in your loss function). But, compared to this, in the backward pass, you need the actual value of the log probability to calculate the gradient that updates the model weights.
So (I think) the normalization is mostly just done for the backpropagation to get "renormalized" gradients. And no unbalanced gradients between states with more/less allowed actions.


Predict time series from initial non-time dependant parameters

I don't think you need a recurrent neural network for this. Why not just train a feedforward model with angle of attack etc as input and translation velocities as output? The size of the output will depend on how frequently you want updates to the velocities. e.g. if the update rate is 0.5 seconds and the network is predicting 2 seconds in advance it could have 12 outputs, being x,y,z values every 0.5 seconds.
You may need to massage the way you represent the outputs to find the best approach. For example, will the network be predicting absolute velocities or just the initial velocities plus changes to velocity over time? If the latter you can always get back to absolute velocities.


Neural network as (BDI) agent - running in continuous mode (that do inference in parallel with learning)?

You might be interested in the Clarion cognitive architecture, developed by Prof. Ron Sun and collaborators.

Full disclosure: I am a student in Ron Sun's Cognitive Architecture Lab.

Brief Description of Clarion

Clarion agents are composed of several subsystems, each of which may contain several neural networks. These subsystems include the Action-Centered Subsystem (ACS), the Non-Action-Centered Subsytem. The ACS controls action decision making, the NACS stores general knowledge. During an activation cycle, the ACS might request information from the NACS, setting the NACS activation cycle in motion.

Strictly speaking, Clarion does not adopt the BDI framework. But, various components of a Clarion agent can be put in correspondence with BDI concepts. BDI beliefs correspond roughly to knowledge in the ACS and NACS. The Motivational Subsystem (MS) contains agent drives and sets agent goals, which roughly correspond to BDI desires and intentions respectively.

Clarion agents do not directly control when learning happens, but may control which subsystems are active based on task demands, and may output mental actions as part of their processing. Learning generally happens at the end of a subsystem activation cycle based on feedback from the environment or from the agent's own motivational and metacognitive subsystems.

Links

A page on Prof. Ron Sun's site links to several resources on Clarion.

Currently the most up-to-date reference on Clarion is the book Anatomy of the Mind. A precis is available in the journal Cognitive Computation.


Can neuroevolution be combined with gradient descent?

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and Christian W. Omlin, uses genetic algorithms to train a recurrent neural network and then uses gradient descent to fine tune the trained model.

The paper Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks (2018), by Xiaodong Cui, Wei Zhang, Zoltán Tüske and Michael Picheny, also combines evolutionary algorithms and gradient descent, but, in this case, they alternate between a gradient descent step and an evolution step. This is an example of a evolutionary stochastic gradient descent (ESGD) method, as opposed to a population-based training (PBT) method, which uses only evolutionary algorithms to train neural networks.


What is the reason AMD Radeon is not widely used for machine learning and deep learning?

The main reason that AMD Radeon graphics card is not used for deep learning is not the hardware and raw speed. Instead it is because the software and drivers for deep learning on Radeon GPU is not actively developed. NVIDIA have good drivers and software stack for deep learning such as CUDA, CUDNN and more. Many deep learning library also have CUDA support. However for AMD there is little support on software of GPU. There is ROCM but it is not well optimized and also a lot of deep learning libraries don't have ROCM support.

Also on the hardware side, AMD lacks deep learning specific features like tensor cores. AMD also don't have data centre specific cards like the Tesla lineup. The performing of AMD cards also is not very good until recently with the RDNA architecture.

Lastly, data centres don't change hardware very often. NVIDIA flourished in the deep learning field very early on so many companies bought a lot of Tesla GPU. Even if AMD caught up in the deep learning field it is very hard as many companies have used NVIDIA from a long time ago, and switching to a very different architecture of GPU is troublesome, especially for a data centre with couple hundred or more servers.

AMD has also "given up" on deep learning market. They don't actively tries to make effort in deep learning data centres using AMD hardware. Many data centre also use very old Tesla hardwares like Tesla K80 of M series.

Hope this can help you.


Clarification regarding "Image Crowd Counting Using Convolutional Neural Network and Markov Random Field"

I will try to do it part by part :

  We employ the residual network, which is trained on ImageNet dataset for image classication task, to extract the deep features to represent the density of the crowd.

If you look at the figure 2, you can see that they use the Neural network architecture ResNet. This is a deep network, here is the paper. It has good performance and do image classification.

  This pre-trained CNN network created a residual item for every three convolution layer to bring the layer of the network to 152

If you are in the layer k, it means this layer has in input, the ouput of the k-3th layer. See the paper, figure 5 explains it well without much explications needed. Furthermore, Resnet has 3 different architectures wih different number of layers, and they take the deeper one, the 152 layers deep Resnet.

  We resize the image patches to the size of 224 × 224 as the input of the model and extract the output of the fc1000 layer to get the 1000 dimensional features

The input of Resnet is images of size 224x224, so they need to resize them to fit the input requirement of Resnet. The output of Resnet is 1000 because Imagenet is a dataset of 1000 classes.

  The features are then used to train 5 layers fully connected neural network. The network's input is 1000dimensional, and the number of neurons in the network is given by 100-100-50-50-1.

Then they give the output of Resnet to their own Network, which is 5 layer deep. See figure 2 of their paper. Obviously, the input layer has 1000 inputs because of the output of Resnet. The network have layers of 100, 100, 50 and finally 1 neuron. See figure 2.

  The network's output is the local crowd count

I don't think I need to explain it, they want only the number of people in the crowd, so they need only one output. This is obviously not a classifcation problem, but a regression problem.

Has you don't really point out what you don't understand, I don't explain it in details. Feel free to ask more precise question if some part are still blurred to you !


Why do you not see dropout layers on reinforcement learning examples?

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise).

In Reinforcement Learning, additional variance is not really what we're looking for. There already tends to be a large amount of variance in the learning signals that we get, and this variance already tends to be a major issue for learning stability and/or learning speed. For example:

Randomness in action selection leads to variance in returns that we observe
There may be randomness inherent to the environment itself, leading to extra variance in our observations (some environments are nondeterministic)
Unlike Supervised Learning settings, in Reinforcement Learning we often actually use our own predictions as a part of our loss function / training signal. For example, in temporal-difference learning (like Q-learning / DQN), the target that we update towards looks like $r + \max_{a'} Q(s', a')$. In that term, only the $r$ is a ground-truth observation (like we would use in supervised learning), and the other term is our own prediction. During a learning process, those latter parts (our own predictions) are changing over time. This is a "moving target'' problem, which can be viewed as additional variance in our learning signals.

Many important parts of Deep RL algorithms (without which our training processes empirically turn out to destabilize and break down) are very much tailored towards reducing that variance. For example, Target Networks in DQN were introduced specifically to reduce the moving target problem. From this point of view, it's not surprising that if we were to add more artificial variance through other means again (such as dropout), that this would hurt performance / destabilize learning.

  Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

In the majority of current (Deep) Reinforcement Learning research, overfitting is indeed not viewed as a problem. The vast majority of RL research consists of training in one environment (for example Cartpole, or Breakout, or one particular level in Pacman, or navigating in one specific maze, etc.), and either constantly evaluating performance during that learning process, or evaluating performance after such a learning process in the same environment.

If we were to compare that evaluation methodology to what happens in supervised learning... we are basically evaluating performance on the training set*. In supervised learning, this would be absolutely unacceptable, but in RL it is very much treated as acceptable and more rule than exception. Some say this is simply a problem in current RL research, something that needs to change. It could also be argued that it's not necessarily a problem; if we really are able to train the agent in precisely the same environment that we wish to deploy it in later... well, then what's the problem with it overfitting to that environment?

So, when we're using the evaluation methodology described above, indeed we are overfitting to one specific environment, but overfitting is good rather than bad according to our evaluation criteria. It is clear that this methodology does not lead to agents that can generalize well though; if you consistently train an agent to navigate in one particular maze, it will likely be unable to navigate a different maze after training.

*Note: the truth, in my opinion, is slightly more nuanced than that we are really "evaluating on the training set" in RL. See, for example, this nice thread of tweets: https://twitter.com/nanjiang_cs/status/1049682399980908544

  I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

Note that your evaluation methodology described here indeed no longer fits the more "common" evaluation methodology. You have a problem with concept drift, with nonstationarity in the environment. This means overfitting may be a problem for you.

Still, I'm not sure if dropout would help (it's still additional variance which may hurt). First and foremost, you'd want to make sure that there's some way to keep track of the time / month in your inputs, such that you'll at least have a chance of learning a policy that adapts itself over time. If you have a clear, solid boundary between "training phase" and "evaluation phase", and you know that concept drift occurs across that boundary (you know that your environment behaves differently in the training phase from the evaluation phase)... you really don't have much hope of learning a policy only from experience in the training phase that still performs well in the evaluation phase. I suspect you'll have to get rid of that clear, solid boundary. You'll want to keep learning throughout the evaluation phase as well. This enables your learning algorithm to actually collect experience in the changed environment, and adapt to it.


How can one distinguish between an AI and a "sufficiently advanced algorithm"?

As you correctly pointed out, people tend to misinterpret the expression AI since they do not know what's behind an AI; it is pretty clear that in AI there is no more than just a bunch of algorithms and flowing bits. Talking about the nature of an AI without talking about the algorithmic paradigm of that AI is pointless and antiscientific.

This point of view is quite cynical; what we call intelligence is just the capability of solving particular problems.

The quote you cited in the title it's a derived version of the quote below, by a science fiction writer.

  Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke

Hence I doubt that a scientific answer exists since science lacks a formal definition of sufficiently advanced algorithm and advanced algorithm.


How do you generate the transition probabilities of a non-trivial MDP?

In non-trivial cases, the transition matrix is (generally) not maintained in the traditional tabular form. If the representation used factored notation (Factored MDP) then Dynamic Bayesian networks can be used.
Another approach would be to abstract state spaces so that you have reduced number of states in the representation of P. These can be roughly classified as model minimization techniques.
Hierarchical representations can be adopted if some tasks can be split. E.g., the classical "taxi problem" has an explicit Get-passenger and Put-passenger task, that form the hierarchy. Now each of the smaller tasks can possibly have a well formed transition function.
It may also be prudent to use model free learning methods in such cases.


3D environment for RL research in Academia

For example DeepMind [Lab] is not a standard or Open Source friendly

I'm not sure where you got that info from... as far as I'm aware, DeepMind Lab is definitely used in various publications (maybe primarily publications from DeepMind, but still). Considering the github repo has the GNU GPL 2 license, it also seems Open Source-friendly to me.

Another framework of which I'm sure that it would widely be considered suitable within academia would be the Unity ML-Agents Toolkit, which uses the Unity game engine.

I suppose you could also consider using ViZDoom, which is also used in various publications, but (as far as I'm aware) it only supports one specific game (Doom).

I do not have enough experience with using any of the above personally to be able to recommend one of them over the others... but they would all seem suitable to me.


What is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.
The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greedy policy with respect to its own previous values. The latter choice - using Q learning to find an optimal policy, using generalised policy iteration - is by far the most common use of it.
A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.
The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction). Or you could write out a table of states showing the chosen action in each one.
The maths notation for how you derive the policy from a Q table can be written:
$$\pi(s) = \text{argmax}_a Q(s,a)$$
Or a bit more formally:
$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$


Defining Path

You are right to be confused if your professors did not clarify (see warning at end). The term "path" can mean a few things:

"Concrete" Path:
Recall, a graph is a collection of vertices and edges. A path on the graph is then:

$$v_1\xrightarrow{e_1} v_2\xrightarrow{e_2} \cdots \xrightarrow{e_{n-1}} v_n$$

Where $v_i$ are vertices on the graph and the arrows denote the direction traversed over the undirected edges $e_j$. Observe, this definition explicitly defines some "chain" of vertices connected by specific edges.

"Abstract" Path: Say there is some task that involves a serious of repeated actions but does not depend on specific objects. We can define this as a chain of composed functions:

$$f_n\circ\cdots \circ f_1$$

The idea, is that this definition is little more abstract. Though, there are many other ways one could define "path."

"Solution Path"

A Subtlety

  So, in this case, is the solution path A -> B -> C, i.e. the order in which the nodes are visited? Or is the solution path A -> B -> A -> C? Basically saying that we go from A to B, but to go from B to C, we must go through A again.

BFS does not "go through A again." Considering you are using a queue based implementation, once A has been visited there will be no need to "visit" it again. That is, we have added all nodes we need to the queue and marked A as being seen. This leads into:

  How to report the solution path of a search algorithm on a graph?

Since, these algorithms do not revisit nodes, what is most likely being asked of you is for you to report the order of the visits - this is what my professors wanted in my AI & Algorithms courses.

A Warning: I can only answer what my professors wanted. Please clarify this with your instructors.


What are the implications of the "No Free Lunch" theorem for machine learning?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's discussed in the ML community. That said, the theorem is true, but what it means is open to some debate.

To restate the theorem for people who don't know it, the NFL theorem for machine learning is really a special case of the NFL theorem for local search and optimization. The local search version is easier to understand. The theorem makes the following, somewhat radical claim:

  Averaged across all possible optimization problems, the average solution quality found by any local search algorithm you choose to use is exactly the same as the average solution quality of a local "search" algorithm that just generates possible solutions by sampling uniformly at random from the space of all solutions.

Another formulation, when people want an even stronger reaction, is to say that if you want to find the best solution to a problem, it's just as good to try things that seem to be making your solution iteratively worse as it is to try things that seem to be making your solution iteratively better. On average, both these approaches are equally good.

Okay, so why is this true? Well, the key is in the details. Wolpert has sometimes described the theorem as a specialization of Hume's work on the problem of induction. The basic statement of the problem of induction is: we have no logical basis for assuming that the future will be like the past. Logically, there's no reason that the laws of physics couldn't all just radically change tomorrow. From a purely logical perspective, it's totally reasonable that the future can be different from the past in any number of ways. Hume's problem is that, in general the future is like the past in a lot of ways. He tried to formulate a philosophical (logical) argument that this needed to be so, but basically failed.

The No Free Lunch theorems say the same thing. If you don't know what your search space looks like, then if you iteratively refine your guess at what a good solution looks like, in response to the observations you've made in the past about what good solutions look like (i.e. learning from data), then it's just as likely that operation you make helps as it is that it hurts. That's why the "averaged over all possible optimization problems" part is key. For any optimization problem where hill climbing is a good strategy after $k$ moves, we can make one that is identical, except that the kth hill climbing move leads to an awful solution. The actual proof is more subtle than that, but that's the basic idea.

A very brief lay summary might be:

  A machine learning algorithim can only be made to work better on some kinds of problems by being made to work worse on another kind of problem.

So what does this mean in a practical sense? It means that you need to have some apriori reason for thinking that your algorithm will be effective on a particular problem. Exactly what a good reason looks like is the subject of vigorous debate within the ML community. This is very closely related to the bias/variance tradeoff.

Some common responses are:

When you're looking at a new optimization problem, although it could have any random kind of structure, the problems we actually encounter in the real world are a lot more regular, and certain common themes are present, like the the fact that moving "uphill" (minimizing error) iteratively tends to lead to good solutions. Basically, this school of thought says NFL is an ornamental theorem: most ML algorithms work better on "the kind of problems we see in real life", by working worse on "the kind of problems we don't see in real life".
When you're looking at a new optimization problem in [insert your favourite application domain], although it could have any random kind of structure,  problems tend to look like [whatever you think], which makes [your favorite algorithm] a lot more effective than random guessing.
Wolpert & McCready themselves published an interesting result showing that there actually are specialized optimization processes, based on co-evolution, that are consistently better than random guessing.

Regardless, it's indisputable that some algorithms are better than others, in certain sub-domains (we can see this empirically). NFL tells us that to be better there, they need to be worse somewhere else. The question up for debate is whether the "somewhere else" is real problems, or purely artificial ones.


As explained in an answer to this AI SE question, GAs are 'satisficers' rather than 'optimizers' and tend not to explore 'outlying' regions of the search space. Rather, the population tends to cluster in regions that are 'fairly good' according to the fitness function.

In contrast, I believe the thinking is that novelty affords a kind of dynamic fitness, tending to push the population away from previously discovered areas.


Why did the developement of neural networks stop between 50s and 80s?

I will first address your main question "Why did the development of neural networks stop between 50s and 80s?"
In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)!
However, Minsky and Papert have proved in 1969 that a single-layer architecture is not enough to build a universal approximating machine (see e.g. Minsky, M. & Papert, S. Perceptrons: An Introduction to Computational Geometry, vol. 165 (1969)). That led to the first disappointment in "AI". Which lasted until several major breakthroughs in 1980s: the proof of universal approximating capabilities of multi-layer perceptron by Cybenko, the popularisation of backpropagation algorithm (Hinton and colleagues), etc.
I agree with LeCun that using continuous activation functions have enabled the backpropagation algorithm at the time. It is only recently that we have learned to backpropagate in networks with binary activation functions (2016!).


Techniques and semantics in better training of deep learning models

I try to answer the things I know for sure:

One effect of bigger images is the increasing computation time due to more pixels (input to your training)

4.Grayscaling reduces the information, which might decrease training time, but also model performance (accuracy, precision, recall). What I have seen is that grayscaling is used in for example face detection where structural information (shapes) is important and colors can be regarded as secondary. For your example you have to determine whether such structural differences between positives and negatives exist and if color is a valuable information.

If possible, I would somehow alter these copies by clipping the objects (sun lights, lamps) and rotate, scale, make them transparent etc.
In the end your training data should be representative of the data it should generalize to. If you would for example take my user icon as an additional non fire example, this wouldn't help your cause.


How classification neural nets are different from simple dimension reduction + clustering?

PCA is linear, NN are nonlinear, more generally it's a universal function approximator.
That said basic NN are not terribly usefull, the real value of NN is for structured data which is obvious for us but hard to describe analytically. Basically NN architecture has been designed to learn this structure using operations like convolution and maxpooling.
There are still problems of course, and in some ways I think clustering algorithms still have value for things like anomaly detection.


How do we compute the target value when the agent ends up in the terminal state?

Now, let us assume the agent is in the penultimate state, $S_1$, and
chooses the action $A$ that leads him to the completion state, $S_2$,
and gets a reward $R$.
How do we form the target value $Q_\text{target}$ for $S_1$ now? Do we still include the $Q_\text{future}$ term? Or is it only the reward
in this case?

Your term "completion state" is commonly called "terminal state". In a terminal state, there are no more actions to take, no more time steps, and no possibility to take any action. So, by definition, in your state $S_2$, the expected future reward is $0$.
Mathematically, this is often noted like $v(S_T) = 0$ or $q(S_T,\cdot) = 0$ with the $T$ standing for last time step of the episode, and dot standing in for the fact that no action need to be supplied, or the specific action value is not relevant. So, therefore using your terms, $Q_\text{future} = \text{max}_a Q(S_2, a) = 0$
That makes the equations work in theory, but does not explain what to do in code. In practice in your code, you would do as you suggest and use just the reward when calculating the TD target for $Q(S_1, A)$. This is typically done using an if block around the done condition e.g.
if done:
  td_target = r
else:
  td_target = r + gamma * np.max(q_future_values)
end

Of course, the details depend on how you have structured and named your variables. You will find code similar to this in most DQN implementations though.
You should not really try to learn $V(S_2)$ or $Q(S_2, A)$, or calculate TD target starting from $S_2$, because the result should be $0$ by definition.


How to design a good evaluation function for a go-like game?

If you have the best combination of distance between the stones, you should choose the best move to win. In this case, you have to be close to where your opponent plays. It is best to do this by surrounding your opponent's stones. You should always put the first stone in middle or corner of the table.


Why are decision trees and random forests scale invariant?

Scaling only makes sense when there is something that reacts to that scale. Decision Trees though, just make a cut at a certain number.
Imagine: For a feature that goes from 0 to 100 a cut at 50 may be improving performance. Scaling this down to 0 to 1 making the cut a 0.5 doesn't change a thing.
Now on the other hand NN have some kind of activation function (leaving RELu aside) that react differently to input that is above 1. Here Normalization, putting every feature between 0 and 1 makes sense.


Could clustering be used to parse pdf documents to get headings and titles?

Yes, you could use clustering: Encode your features as a feature vector and feed it into a clustering algorithm (see Finding Groups in Data for a comprehensive description of these). You could use agglomerative clustering, which would give you groups of similar items; perhaps different level headings will be clustered together.

Alternatively you could try a decision tree, something like ID3, which would also be suitable; for this you'd need some annotated training data, though. But with a small amount of data you might solve it, if your items are clearly separated.


Are there examples of neural networks (used for control) implemented on a FPGA or on a neurochip?

Assumption

In this answer it is assumed that with "neurochips" you mean chips made (using neuromorphic engineering) for neuromphic computing.

Related example

From what I currently understand from this article neuromorphic chips, in particular the TrueNorth chip, are being used (or emulated) for embedded systems related signals processing.

Doubt

The signal processing performed by these (emulated) neuromorphic chips might be part of real-life applications involving control systems.


Why does L1 regularization yield sparse features?

In L1 regularization, the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor).
Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0. So, the only way how a training procedure can considerably reduce the L1 regularization penalty is by driving all (unnecessary) weights towards 0, which results in a sparse representation.
Of course, the L2 regularization will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the squared value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. $abs(weight) < 1$, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: $x^2 < x\ for\ all\ x\ with\ abs(x) < 1$.
So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with $abs(x) < 1$ much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.