How robust are deep networks to class imbalance?

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC AUC drops by 5-10%, and accuracy decreases by 20-30%. These effects are worsened on more complex tasks.
There are 3 noteworthy class imbalance approaches to partially alleviate this:

Undersampling: sampling the over-represented class less often
Oversampling: sampling the under-represented class more often
Thresholding: after the neural network learns the weights, during inference, multiply the output class probability by a weight (the class prior, which is different for each class). The weight is the inverse of the class representation of the dataset (i.e. the inverse of $\frac{numInstances(c)}{\displaystyle \sum_i numInstances(i)}$, where $c$ is the current class and $numInstances(i)$ is the number of unique instances of class $i$ in the training set.

The paper concludes with the following best practices:

Regarding the choice of a method to handle CNN training on imbalanced dataset we conclude the following.

The method that in most of the cases outperforms all others with respect to multi-class ROC AUC was oversampling.

For extreme ratio of imbalance and large portion of classes being minority, undersampling performs on a par with oversampling. If training time is an issue, undersampling is a better choice in such a scenario since it dramatically reduces the size of the training set

To achieve the best accuracy, one should apply thresholding to compensate for prior class probabilities. A combination of thresholding with baseline and oversampling is the most preferable, whereas it should not be combined with undersampling.

Oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance. The higher a fraction of minority classes in the imbalanced training set, the more imbalance ratio should be reduced.

Oversampling does not cause overfitting of convolutional neural networks, as opposed to some classical machine learning models.

The last point is very interesting, because oversampling is known to cause overfitting in classical machine learning models and many have advised against doing it.


Choice of inputs features for Snake game

Before we start to tweak with you Agent-Environment setup, there are couple of important things to note

Q-Learning

Q-learning as a fundamental is a greedy approach based on Action Value functions. When I say action value what I mean to say is this
$q_\pi(s,a) = p(\hat{s},r | s, a)[r + \gamma(V(\hat{s}))]$.

This means according to $\pi$ policy the Action value for taking action "$a$" at state "$s$" is the probability of taking reaching state "$\hat{s}$" and getting a reward "$r$" given the current state is "$s$" and action taken is "$a$" multiplied by sum of immediate reward "$r$" and expected future reward given by future state value(s) "$V(\hat{s})$".

Please note the $p(\hat{s},r | s, a)$ is only available in deterministic modelling technique (model based methods) and this part is not available for stochastic modelling techniques (model-free methods). For model-free methods we execute multiple experiments and take the expected value in accordance with Law of Large Numbers. Which is what we do in common Deep Reinforced Learning implementations.

The above formulation for future rewards can be expanded in series using the state-value formulation as $V(s) = \pi(a,s)q_\pi(s,a)$, where $\pi$ is the policy we are following aka probability of taking action $a$ when in state $s$.

Why is it termed Greedy?

This is because while using the fundamental of multiple experiments, with every action I take the immediate reward, and the max q_value ($q$) associated with the next_state to update my current q_value.

In Q-learning the future state reward is what you calculate by submitting the next_state to the network and taking max of the q_values attained. (Hence, Greedy!!)

Crux of the above

Having said all the above (Duh!), what is in it for us?

In Q-Learning, your reward is key for your model learn, i.e. "$r$" immediate reward and the future reward ($V(\hat{s})$)

When the state-space is limited

For example, a numerical tic-tac-toe game, in such cases it is ideal to have discrete reward system, such as the one suggested by you.
When the state-space is very large (the problem statement you are tackling)

In such cases, it is ideal to have a continuous reward mechanism, i.e. rewards should be more variable in nature, like continuous values.
Suggestions

You can define the reward function in the environment has the 10$\sqrt{2}$ - Euclidean distance between the face of your snake and the fruit it is trying to reach. The reason I am doing it (10$\sqrt{2}$ minus) is because the maximum distance could be 10$\sqrt{2}$ (considering it is 10X10 board). This would help you to have a more variable/continuous rewards for your model to learn with (since you are using neural networks for training them). As your Snake goes closer to the fruit this value will keep on increasing. One cavet, if by taking an action, your snake reaches fruit, you can provide it a specific constant (extra) reward, and if it hits its own self, or the borders you penalize it with a specific constant. So that the model understand better.
You can define your state (input vector to your NN) as 100 dimension array (10X10 grid) with each element signifying if there a object in the that area or not. 1 means it is fruit, 0 means it is snakes own body and 0.5 means it is free for movement. The idea is to limit the input values of the vectors between 0~1.  As your agent takes action, the values in the state will keep on changing.
The output of your network should have 3 neurons, each signifying the $q$ of each of action (front/right/left movement). For get_action function you will get the action pertaining to max(q_value), and keep on adding things in your memory d_que. Subsequently you can trigger model_train methods at every episode/or even at after every step.
Lastly, the hyper parameters would be the discount factor $\gamma$ you choose and the learning rate $\alpha$ you use for the NN network. That's something you will have to experiment. Along the depth of your ANN. I think a single hidden layer should be enough.
For the last layer of your NN, please use a linear activation function, and not a relu activation function.
Please run many-many-many experiments, something in millions for optimum training. There could be multiple ways of monitoring your model's learning process. How many games won, how much fruit eating it is able to achieve in 1 game etc.

Hope the above helps.


How do map providers like Google calculate the distance between two coordinates and find turn by turn directions?

Roads maps are well defined and you can access them online. For example you can visit OpenStreetMap to download road network of a given region.

Such a road network definition contains nodes (junctions) with lat/lon coordinates and edges between nodes (roads). An edge is defined by a connection of two nodes. Edges can represent one-way roads: you need to define double ways as two edges going in both ways. Haversine formula gives you the straight distance between two nodes: the length of the edge.

This definition of a road network gives you a graph (in the mathematical point of view) and is largely covered by the literature. You can search the graph and compute many of its parameters.

From a road network definition you can run shortest path algorithm to find the closest way between two nodes. From a given coordinate you can either pick the closest node or get the closest edge (and compute distance to the edge and the coordinates of the perpendicular projection point).

Algorithms like A* and Dijkstra are not so time consuming and you can find shortest path efficiently.


How do Google cars recognize the traffic signs?

I'm not sure what Google is using to perform that task, but most companies use region based convolutional neural nets to locate traffic signs and other objects.

But other companies use a deep neural network + bag of words approach to find objects.

See: Bag-of-Words Based Deep Neural Network for Image Retrieval which shows a general approach, to get the exact location you can use Feature Matching or Random Boxes.


Testing, Validation Percentage & Test, Validation Batch Size Difference?

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the validation and testing percentages to be 10% each, your model will train on 80,000 samples, validate them on 10,000 and save additional 10,000 samples for the final test.

The batch sizes refer to the number of samples in each batch during the test and validation evaluations.
Your model probably can't process 10,000 samples in a single run (due to memory limitations) so during evaluation it breaks the dataset into batches, which are processed sequentially and the result is the mean of all batches.

When you are training, the batch size is an important hyper-parameter which has an affect on the properties and final results of the training process. During test/validation it has no affect and only needs to be small enough for your model to be able to run it (evaluation with different batch sizes will produce the same results).


What is the difference between out of distribution detection and anomaly detection?

You observation is correct although the terminology needs a little explaining.
The term 'out-of-distribution' (OOD) data refers to data that was collected at a different time, and possibly under different conditions or in a different environment, then the data collected to create the model.  They may say that this data is from a 'different distribution'.
Data that is in-distribution can be called novelty data.  Novelty detection is when you have new data (i.e. OOD) and you want to know whether or not it is in-distribution.  You want to know if it looks like the data you trained on.
Anomaly detection is when you test your data to see if it is different than what you trained the model.  Out-of-distribution detection is essentially running your model on OOD data.  So one takes OOD data and does novelty detection or anomaly detection (aka outlier detection).
Below is a figure from What is anomaly detection?

In time series modeling, the term 'out-of-distribution' data is analogous to 'out-of-sample' data and 'in-distribution' data is analogous with 'in-sample' data.


What is Statistical relational learning?

The University of Maryland published some slides (PDF) from an introductory presentation on this topic.

The fourth page explains why SRL is interesting. "Traditional statistical machine learning approaches" process one sort of thing in which there is some uncertaintly. Image identification is a good example of that. "Traditional ILP/relational learning approaches" use several kinds of information to produce hypotheses about the data set, but apparently allow for no noise in the data.

Statistical relational learning models are intended to work with data sets that have several types of objects connected to each other via various links (hence "relational"). They also are meant to deal with uncertainty (hence "statistical").

Skipping past some slides that aren't really useful without a transcript of what was said over them, we come to slide 17, which has comprehensible definitions and examples:

  Object classification

  Predicting the category of an object based on its attributes and its links and attributes of linked objects
  e.g., predicting the topic of a paper based on the words used in the paper the topics of papers it cites the used in the paper, the topics of papers it cites, the research interests of the author

  Object type prediction

  Predicting the type of an object based on its attributes and its links and attributes of linked objects
  e.g., predict the venue type of a publication (conference, journal, workshop) based on properties of the paper

As you can see, these models can keep track of several things and the interactions between them. The next slide talks about link prediction, the ability to predict several attributes of connections between objects, like the importance/quality of a citation. As previously mentioned, these models don't require 100% accurate data to give interesting results; academic citation lists might occasionally be less than comprehensive, and the importance of a citation is challenging to quantify.

Like ILP, SLP will hopefully be able to "see" new kinds of links between "entities", as with the presentation's example of identifying research communities.

Past slide 20, the presentation goes into some serious mathematics. It does have a much less technical conclusion starting at slide 198.


What is meant by "reproduction" in the description of this exercise?

The terminology of this exercise is not standard. What is referred to as "reproduction" in the exercise is usually referred to as "selection".

The term "reproduction" does indeed seem conceptually closer to the notion of crossover or recombination (these two are the same thing), which is probably where your confusion has arisen.

See the excellent (and freely-downloadable) 'Essentials of Metaheuristics' for an introduction to the usual terminology for evolutionary algorithms.


Is there a machine learning algorithm to find similar sales patterns?

I would recommend a hierarchical cluster algorithm, after normalising your numbers into proportions. Then the clustering should be able to identify similar patterns. Depending at which level you make the cut, you can decide how many clusters you want.

A great resource on this topic is Kaufman, L., & Roussew, P. J. (1990). "Finding Groups in Data - An Introduction to Cluster Analysis". John Wiley & Sons


What is the difference between distant supervision and self-supervision?

The main difference between distant supervision (as described in the link you provided) and self-supervision lies on the task the network is trained on.
Distant supervision focuses on generating weak labels for the very same task that would be tackled with supervised labels, and the final result could be directly used for that matter.
Self-supervision is a means for learning a data representation. It does so by learning a surrogate task, which is defined by inputs and labels derived exclusively from the original input data.
I can imagine cases in which an implementation of self-supervision could be considered distant supervision (if the task casually matches the target task). On the other hand, if external data sources would be employed for training on a surrogate task, that would be a case of representation learning (that could incorporate self-supervision too).


How does the Jetson AI write its screenplays?

It appears to use Recurrent NNs (RNNs) that have a 'Long Short-Term Memory' (LTSM) architecture.

Here's a summary of the development process that the author, Ross Goodwin, went through to create it.

It seems to me (and is also observed in the above link) that the output is rather poor - simply comparable to what one might expect from Markov chains, a technique that is over 100 years old.

I haven't dug deeply into the technique, so I could be misktaken, but perhaps one of the reasons that it's so bad is that (as far as I can see), the model-building process is essentially lexical - i.e. it is linking together tokens (words) without any more informed language model to guide it. In particular, the generated output doesn't seem to know anything about the functional roles played by objects (chairs are supporting objects, used by humans for sitting on etc), which is something that might be fairly readily incorporated.


How is AlphaZero different from Stockfish or Rybka?

Stockfish is free software (source while AlphaGo can only be used by employees at Deepmind. This is important, because it means all answers to this question can only rely on the AlphaGo paper. Some decisions during the matches (e.g. giving limited time per move and not per game; giving different computational power) lead to unclarity how good alpha go really is.

Stockfish  uses Alpha-beta search with a hardcoded metric to evaluate how good a position is.

In contrast, AlphaGo uses Monte Carlo tree search and convolutional neural networks. They are trained in an reinforcement learning setting.

AlphaGo is promoted as being almost directly applicable to different settings (Go, Chess), while stockfish is only applied to Chess. The techniques of stockfish can also be applied to Go, but the problem is the board evaluation. For Chess it's easy compared to Go.

References

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, December 2017.
Mastering the Game of Go without Human Knowledge is about Alpha Go Zero


How is BERT different from the original transformer architecture?

What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the non-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper "Attention is all you need", i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.
Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at this paper).  The novelty of the transformer and this cited paper is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.
The original transformed architecture is depicted in figure 1 of the cited paper. Both the encoder and decoder are composed of

attention modules
feed-forward (or fully connected) layers
residual (or skip) connections
normalization layers
dropout
label smoothing
embedding layers
positional encoding

The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).
What is BERT?
BERT stands for Bidirectional Encoder Representations from
Transformers, so, as the name suggests, it is a way of learning representations of a language that uses a transformer, specifically, the encoder part of the transformer.
What is the difference between the transformer and BERT?

BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part Representations in its name, Bidirectional Encoder Representations from
Transformers.
BERT can be trained in an unsupervised way for representation learning, and then we can fine-tune BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. this one) and used to solve your specific supervised learning task. You can play with this TensorFlow tutorial to use a pre-trained BERT model.
On the other hand, the original transformed was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.

BERT is only an encoder, while the original transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformed-based neural networks that use only the decoder part of the transformer, for example, the GPT model.

BERT uses different hyper-parameters than the ones used in Attention is all you need to achieve the best performance. For example, it uses 12 and 16 "attention heads" (please, read the transformer paper to know more about these "attention heads") rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).

BERT also uses segment embeddings, while the original transformer only uses word embeddings and positional encodings.

There are probably other small differences that I missed, but, after having read the paper Attention is all you need and quickly read some parts of the BERT paper, these seem to be the main differences.
When to use BERT and the transformer?
Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.


Choosing Data Augmentation smartly for different application

In fact, choosing smartly the values of the image augmentation can help the performance of your system. Where I work we developed an object detector for cars. We had the following image augmentation parameters:

Apect ratio distorsion: it changed the cars dimensions
Additive noise: it blurred the image
Change colorspace: change the cars colors
Saturation change: brightness
Rotation: rotate image axis
Flipping (horizontally): mirror effect
Zooming: zoom in/out of image

At first we just combined all the effects (within a certain range of change per effect) randomly in each augmented image. But then we did an study for our use case and we found that our model performed better with: no aspect ratio deformation and no change in color space. After giving it a bit of a thought it really made sense for our use case because it is a detector for a very specific type of car.

However, although the improvements accomplished by optimizing the parameters of image augmentation boosted the performance of the model, the improvements were not huge, so in the end it did not really worth the effort. To give some numbers it improved $5\%$ in our custom test metrics, and we were expecting something around $10\%$ so from that moment on we left the specific image augmentation parameters but stopped thinking too much about it.

So, in conclussion, does it helps? yes, how much? well, that would depend on the use case. For us, it turned out it did not improve as much as we wanted.

Hope this helps :)


Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:

IT SHOULD OVERFIT.

Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.


Is it possible to design a neural network from the plot of data points?

From the diagram you have given, it's quite clear that you have to design a supervised network. Also, it's clear that you are dealing with a problem that's not linearly separable i.e. you can't separate two classes using a single line. Particularly, it will require three lines to separate these two classes.
From the above observations, what you can do is:

Make training data sets of form [(x,y), o] where (x,y) are the co-ordinates on the plot and o is the class that point belongs to.
Note: You can actually divide your data set into training data and test data. Just, extract the data, randomly shuffle it and take first 80% of data to train and rest 20% of data to test.

Design a 2-2-2-1 neural network with bias units and some random initial weight values. This is because we will require minimum three lines to separate two classes, so 2 hidden layers (each consisting of 2 nodes and bias nodes) and 1 output layer (with bias node). The first layer is input layer with 2 nodes (no bias) as we have x-coordinate and y-coordinate.

Train your training data using this network using the Error Back Propagation algorithm i.e. update the weights between the layers and the bias weights too. Run the EBP algorithm for 100-2000 epochs.

Then, use the test data to see, if you get the desired output.

Note: I am NOT quite sure how to obtain the coordinates from the diagram. You can either manually note all points or research for some methods to get those points automatically from the image (some image processing algorithms).
Edit: I found a good video which explains neural networks. In the video, there is non-linear data like the one in your image which is separable using multiple lines and the network is designed to try to find curves such that data can be separated and the output resembles to that when used straight lines.


Is the paper "Reducing the Dimensionality of Data with Neural Networks" by Hinton relevant?

No; there are too many publications around for anybody to keep track of everything, so unless it is a seminal paper, you cannot draw any conclusions from this. They could simply have missed it.

Especially if it is a textbook for beginners, more advanced papers are often not mentioned, as they might be too complex to understand.

So you have to decide for yourself if that paper is relevant to you. To me it sounds like a specific application of neural networks; if the dimensionality of the input data is an issue for you, it might be, otherwise probably not.


What amount of ressources is involved in building an image recognition system?

One answer is infinite amount of time because it can always be better.
Another answer is:

10k for training set
A PC with a GPU (3~4k USD), google colab (10 USD per month), or other cloud service (probably more expensive than colab)
One developer, 1 day lol
Two kinds is easier than multiple kinds
There is no paper that seeks to answer your question the way you put it. I wouldn't even recommend a paper. In fact, for you I'd recommend an AutoML tutorial. Check these. * no offence if I've misjudged your knowledge/skill level.
Here's a paper anyway :) https://paperswithcode.com/lib/torchvision/alexnet

To conclude, please be aware that your question is super open ended, and my answer is bad (but good enough for now maybe), but a good answer doesn't really exist. It's always going to be context dependent. For instance, you never said whether you need 90% or 99% accuracy.


how to normalize the state space for articulated robot environments?

This paper might provide some answers https://arxiv.org/pdf/1810.05762.pdf

For the observations / states they used not only angles, but also velocities, heights and positions (Table 2).

In 4.2 Learning algorithm you can see that they mention this, which is related to your question about normalization:

  Additionally, for stability we whiten the current observations by
  maintaining online statistics of mean and standard deviation from the
  history of past observations.


Can the AI in a box experiment be formalized?

I think that something like "strength" would be difficult to quantify in this context.  I do think that formal experimentation around the "AI in a box" scenario could be interesting.  I know that experiments have been done where a human plays the role of the AI, attempting to get naive test subjects to "release" him by interacting with them over a chat interface.  In all cases, the "AI" tends to be extraordinarily effective.  It is often simply not possible to anticipate every way someone can trick you into revealing information or creating security holes.  This is how human hackers do their jobs too.  But I think a fully automated "game" could produce some interesting data.

One way I think it may be interesting to go about creating an "AI in a box" game is to have an ambiguous win condition.  Essentially, make it possible to let the AI out and still think you are "winning" the game.  One example that pops into my head right away is to use a (pointless) point system.  You tell the player that the primary goal of the game is to keep the AI in the box, but you have a score that is displayed in the game, and tell the player that they win, no matter what, if their score is higher than the AI's by the end of the game.  Of course, they never get to see the AI's score so they never know how well they are doing by comparison.  This could be justified as representing human vs AI "strength" or "leverage" or whatever else.

In reality the AI doesn't have a score, but it does have the ability to effect the player's score.  It can raise the player's score if they do things that help it escape and lower it when the player stands in its way.  The score has no impact on the game and the player really wins by keeping the AI in the box.  It only serves as a red herring that the AI can use to manipulate the player.  I think this sort of experiment could provide an interesting model of how an AI might leverage bribes, threats, promises, and deception, as well as technical tricks, to convince a human to "let it out of the box".


Why can't we use Google Translate for every translation task?

Google has achieved significant progress in AI translation, but it's still no-where near a qualified human translator. Natural language translation is already very challenging, adding domain knowledge to the equation is too much even for Google.

I don't think we have the technology to translate an arbitrary book from one language to another reliably.


Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero).
Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:

I've write the code for this task in colab and you can find it here, and you can play with it if you want.
If you run the network several times you may get different results (because of different initializations) and you can see some of them at the Results section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with tanh rather than sigmoid and if you want to know why, I highly recommend you to look at this lecture of CS231n. In summary it's because tanh has the negative part and the network can learn better with it.
But actually their power of approximation are almost similar because 2*sigmoid(1.5*x) - 1 almost looks the same as tanh(x) and you can find it by looking the picture below:

So why you can't get the same results as tanh? that's because tanh suits the problem better and if the network wants to get the same result as tanh with sigmoid it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with sigmoid but it's harder. And to show you that its possible, I have set the parameters of the network using sigmoid manually and got the result below (you can get better results if you have more time):

At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.
If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):

Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies sigmoid or tanh to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).


How can we conclude that an optimization algorithm is better than another one

See here for a potential way to do it:

http://infinity77.net/global_optimization/#motivation-motivation

http://infinity77.net/global_optimization/#rules-the-rules

You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.


Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

Your examples are equivalent. But it is possible to find a constant yielding  a different optimal policy.
Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so is by reaching $s^*$.
Consider $r_3$ :
$$
r_3(s, a)=
\begin{cases}
1, & \text{ if }  s \neq s^*\\
2, & \text{ otherwise}
\end{cases}
$$
With a sufficiently large $\gamma$, moving infinitely without reaching $s^*$ is now the optimal solution.
For the generic case
$$
r_4(s, a)=
\begin{cases}
\alpha, & \text{ if }  s \neq s^*\\
\beta, & \text{ otherwise}
\end{cases}
$$
the threshold is found by comparing the results of the series $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t_m}$, where $t_m$ is the maximum episode length, and $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t^*}$, where $t^*$ is the length of the episode following the fastest policy.
In the example of $r_3$, it is trivial to find examples where the fastest policy isn't optimal. Imagine a race, the agent starts on the left and gets either $\alpha$ or $\beta$ points, depending on where it is. With $\gamma = 0.9$ and no time-limit (infinite episodes) the optimal policy is  to move randomly, but in the second-to-last house, avoid the goal state. With $\gamma = 0.1$, the optimal policy is to move randomly (not really, probably there would be a slight advantage in moving right), but in the second-to-last house, enter the goal.


What does a joint probability density function have to do with Stochastic Optimal Control and Reinforcement Learning?

Extracting a joint PDF just means that you create a model that models the behavior of several variables combined instead of in isolation.

If these variables aren't independent and your loss functions is influenced by all of them, you obviously have to learn this joint PDF to minimize your loss.

So I don't see this statement as particularly mysterious.


Why is the denominator ignored in the Bayes' rule?

The answer from @kiner_shah in the comments has solved it:

  They have eliminated it because in comparing probability for best outcome, it would just introduce additional division and the divisor p(x) is same for all candidates. For example, if I tell you to compare the largest of the following numbers: 3/2, 5/2, 1/2, you will without thinking pick 5/2 since the divisors are same and 5 is greatest of all.


Why do we have to train a model from scratch every time?

Disclosure: I am a product manager on Google Cloud Platform.

  [...] why does everyone have to repeat the effort of learning the same things?

  If Google has already learned cats, or if someone already has a program to recognize handwritten digits, can this knowledge be shared and re-used? Or is it just a matter of paying for them?

You don't have to rebuild these machine learning models from scratch; you can reuse prebuilt machine learning algorithms, e.g., Google Cloud provides the following hosted APIs as a service:

Google Cloud Vision API to recognize images and provide text labels
Google Cloud Speech API to convert speech to text
Google Cloud Natural Language API to understand language, find entities, etc.
and more...

You can put these APIs together to build interesting applications, e.g.,

build a smart Raspberry Pi bot with Cloud Vision and Speech APIs (video presentation; blog post + video overview)


Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder?

The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training.
This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder:
class Decoder(nn.Module):
    "Generic N layer decoder with masking."
    def __init__(self, layer, N):
        super(Decoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)

    def forward(self, x, memory, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, memory, src_mask, tgt_mask)
        return self.norm(x)


Why do we need convolutional neural networks instead of feed-forward neural networks?

Why are CNNs useful?
The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations).
Convolution
The convolution is an operation (more precisely, a linear operator) that takes two functions $f$ and $h$ and produces another function $g$. It's often denoted as $f \circledast h = g$, where $\circledast$ represents the convolution operation and $g$ is the function that results from the convolution of the functions $f$ and $h$.
In the case of CNNs,

$f$ is a multi-dimensional array (aka tensor) and it represents an image (or a processed version of an image, i.e. a feature map)
$h$ is a multi-dimensional array and it is called kernel (aka filter), which represents the learnable parameters of the CNN, and
$g$ is a processed version (with $h$) of $f$ and it is often called the feature map, so it's also a multi-dimensional array

Images as functions
To be consistent with the initial definition of the convolution, $f, h$, and $g$ can indeed be represented as functions.
Suppose that the input image is a greyscale (so it is initially represented as a matrix), then we can represent it as a function as follows
$$f: [a, b] \times [c, d] \rightarrow [0, 1],$$
i.e. given two numbers $x \in [a, b]$ and $y \in [c, d]$, $f$ outputs a number in the range $[0, 1]$, i.e. $f(x, y) = z$, where $z$ is the grayscale intensity of the pixel at coordinates $x$ and $y$. Similarly, the kernel $h$ and $g$ can also be defined as a function $h: [a, b] \times [c, d] \rightarrow [0, 1]$ and $g: [a, b] \times [c, d] \rightarrow [0, 1]$, respectively.
To be more concrete, if the shape of the image $f$ is $28 \times 28$, then it is represented as the function $f: [0, 28] \times [0, 28] \rightarrow [0, 1]$.
Note that the domain of the images doesn't have to range from $0$ to $28$ and the codomain doesn't have to range from $[0, 1]$. For example, in the case of RGB images, the codomain can also equivalently range from $0$ to $255$.
RGB images can also be represented as functions, more precisely, vector-valued functions, i.e.
$$
f(x, y) =
\begin{bmatrix}
r(x, y) \\
g(x, y) \\
b(x, y)
\end{bmatrix}
$$
where

$r: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the red channel,
$g: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the green channel, and
$b: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the blue channel

Or, equivalently, $f: [a, b] \times [c, d] \times [0, 1]^3$.
Why is the convolution useful?
The convolution of an image with kernels (e.g. the median kernel) can be used to perform many operations.
For example, the convolution of a noisy image with the median filter can be used to remove noise from that image.

This is a screenshot of an image from this article, which you should read if you want to understand more about noise removal. So, on the left, there's the noisy image, and, on the right, there's the convolution of the median filter with the noisy image, which removes (at least, partially) the initial noise (i.e. those dots, which are due to the so-called "pepper and salt" noise).
The convolution of any image with the Sobel filter can be used to compute the derivatives of that image (both in the $x$ and $y$ directions, from which you can compute the magnitude and orientation of the gradient at each pixel of the image). See this article for more info.
So, in general, the convolution of an image with a kernel processes the image and the results (i.e. another image, which, in the case of CNNs, is called a feature map) can be different depending on the kernel.
This is the same thing as in CNNs. The only difference is that, in CNNs, the kernels are the learnable (or trainable) parameters, i.e. they change during training so that the overall loss (that the CNN is making) reduces (in the case CNNs are trained with gradient descent and back-propagation). For this reason, people like to say that CNNs are feature extractors or are performing feature extraction (aka feature learning or representation learning).
(Moreover, note that the convolution and cross-correlation are the same operations when the kernels are symmetric (e.g. in the case of a Gaussian kernel). In the case of CNNs, the distinction between convolution and cross-correlation doesn't make much sense because the kernels are learnable. You can ignore this if you are a beginner, but you can find more details here.)
Other useful properties
There are other useful properties of CNNs, most of them are just a consequence of the use of the convolution

Translation invariance (or equivariance), i.e. they can potentially find the same features (if you think of them as feature extractors) in multiple places of the image independently of their position, orientation, etc. See this answer for more details.

The equivalent FFNN has a lot more parameters (so CNNs may be less prone to overfitting)

They often use a sub-sampling operation  (known as pooling) to further reduce the number of parameters (which can possibly help to avoid overfitting) and introduce non-linearity.

Notes
Note that the FFNN can also be used to process images. It's just that the CNN is more suited to deal with images for the reasons described above.


What are preferences and preference functions in multi-objective reinforcement learning?

In MORL the reward component is a vector rather than a scalar, with an element for each objective. So if we are using a multiobjective version of an algorithm like Q-learning, the Q-values stored for each state-action pair will also be vectors.
Q-learning requires the agent to be able to identify the greedy action in any state (the action expected to lead to the highest long-term return). For scalar rewards this is easy, but for vector values it is more complicated as one vector may be higher for objective 1, while another is higher for objective 2, and so on.
We need a means to order the vector values in terms of how well they meet the user's desired trade-offs between the different objectives. That is the role of the preference function and preferences. The function defines a general operation for either converting the vector values to a scalar value so they can be compared, or for performing some sort of ordering of the vectors (some types of orderings such as lexicographic ordering can't readily be defined in terms of scalarisation). So, for example, our preference function might be a weighted sum of the components of the vector. The preferences specify the parameters of the preference function which define a specific ordering (i.e. based on the needs of the current user). So, in the case of a weighted sum for the preference function, the preferences would be specified in terms of the values of the weights.
The choice of preference function can have implications for the types of solutions which can be found, or for whether additional information needs to be included in the state in order to ensure convergence.
I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area)

Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.


The allegation was based on the fact that Deep Blue made a choice that did not yield the immediate (or short term) benefit that was synonymous with systems back then (1997). Computational capability was significantly less powerful then, and Kasparov claimed that only a grand master would have made the decision that the system did - so the deep blue team cheated by having a human perform the move instead of the system.

He asked for a rematch, but IBM did not allow this, which only added to the suspicion.

This is a great article with deep analysis on the specific moves and circumstances - however suffice to say that Kasparov was trying to bait the system into making a decision for a weak pawn, but the system chose otherwise and instead put Kasparov into a compromised position:
https://en.chessbase.com/post/deep-blue-s-cheating-move


Why are Neural Networks considered Artificially Intelligent?

Why in every aspect we are now considering artificial intelligence as a neural network?

"We" aren't. It is generally due to reporting by media sources that simplify science and technology news.

The definition of AI is somewhat fluid, and also contentious at times, but in research and scientific circles it has not changed to the degree that AI=NN.

What has happened is that research into neural networks has produced some real advances in the last decade. These advances have taken research-only issues such as very basic computer vision, and made them good enough to turn into technology products that can be used in the real world on commodity hardware.

These are game-changing technology advances, and they use neural networks internally. Research and development using neural networks is still turning out new and improved ideas, so has become a very popular area to learn.

A lot of research using neural networks is also research into AI. Aspects such as computer vision, natural language processing, control of autonomous agents are generally considered parts of AI. This has been simplified in reporting, and used by hyped-up marketing campaigns to label pretty much any product with a NN in it as "Artificial Intelligence". When often it is more correctly statistics or Data Science. Data Science is another term which has been somewhat abused by media and technology companies - the main difference between use of AI and Data Science is that Data Science was a new term, so did not clash with pre-existing uses of it.

The rest of AI as a subject and area of study has not gone away. Some of it may well use neural networks as part of a toolkit to build or study things. But not all of it, and even with the use of NNs, the AI part is not necessarily the neural network.


NEAT - Managing species across generations

In some implementations, the specie representatives change each generation. This allows for a dynamic definition of what a specie is. If you speciate from scratch(meaning each specie is assigned a new representative) every generation, you won't have the problem where a representative leaves its specie.


Each network has a limit that it can learn. To check that limit, you need to try. Especially those kind of simple networks, it can be achieved so quickly. Idea is that, you need to give X data and check the accuracy and increase it to 2X data and check the accuracy. If there is no change in accuracy, it means that your network has reached its limit and you may consider making it deeper and wider.

By nature of the training, the last sample that you bring to training is the result that optimizer will try to converge. Hence, optimizers are already doing what you are searching for. However, if you want to create some weights for them, try considering RNNs. Their nature is suitable for that purpose.


What's the best practice for Boltzmann Exploration temperature in RL?

Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.
Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf
Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): https://arxiv.org/pdf/1612.05628.pdf


What would be the result of an AI that does not find any pattern according its available methods of pattern recognition?

During my master thesis I struggled with this issue.

In strict sense, a real AI would be able to say, my sensors (or some of them) are gathering data i cannot understand (yet), and would ideally have a routine what to do whith such cases and ideally one day, even understand that data. But it would most probably now, if that is happening outside in the real world, or if it's due to damaged or wrong sensors.
But that broadly depends how it was programmed, so your use case would have to be further defined: what specific AI are we talking about, what can it do, what not, how is it built.

AI is actually able to reproduce
a) a subset of an adult's intelligence (speach, face recognition, etc.) but just on a case by case basis
b) or, if you seak to build a full grown intelligence, it has to go thorugh the development stages as humans (eyesight, voice, imitation, learning, etc.)
So AI is still in babysteps and is not even able to reproduce let's say an adolescent behaviour, or to reproduce the human evolution of a baby to an adolescent.

So, after a long speach, to come back to your original question: At the very moment an AI is not able to decide by itself, if it has no method to analize the data, if the data analized are too random, or if one of it's sensors is giving random input.
Perhaps in some years there are better methods, but at the moment, machines, as babys or childs, need adults hands to teach and show them what to do in what case and how to difference each use-case. Which often means that they would need to add new modules to the AI, which, nowadays is still far away from building anything we could really define as "intelligent" or even "artificially intelligent" (PD: Intelligence here means by ways much more than plain rationality)

I deeply recommend some books of Andy Clark concerning filosofical issues and AI:
e.g. "Being there: Putting Brain, Body and World together". Andy Clark is a filosofer which takes a deeper look at AI, the human brain, cognition sciences, filosofy, psychology, sociology, etc.


How should we interpret this figure that relates the perceptron criterion and the hinge loss?

I can't fully explain the part because I forgot what it talks about.
However, regarding the hinge loss, it is basically allowing your SVM to tolerate misclassifications without increasing the cost function.
For example, you give someone 1 dollar or 1 euro. You can forgive them, you tolerate it. Your hinge loss is 0 for lending someone 1d dollar. However, if you give them 10 dollars or 100, you will ask them to refund you ASAP because you can't tolerate that much loss!


Are Bayesian networks important to learn in 2018?

*AI, A Modern Approach," was given that title to break from previously narrow approaches to duplicating desirable qualities of human thinking.

Although Bayesian networks require somewhat resource intensive computational elements, the importance of Bayesian inference and probability are still of paramount importance in that some of the highest scientific thinking require mastery of them.  Furthermore, developing in silicon dies (or possibly graphene nanites) the machinery to perform elementary probability computations in massively parallel architectures may arise over the next few decades.  The use of video DSP circuits to implement ANNs is a notable segue into this kind of development.

I would not dismiss the techniques you just read about.  If your intention is to capitalize on the recent crazes, you may enter the river of wannabees chasing every current trend, implement many systems that other people will rewrite later, and have a meaningless yet profitable career.  I would recommend following your inquisitiveness instead.


When could a linear discriminant give excellent or possibly even the optimal classification accuracy?

A linear model has a linear decision boundary. So in the case of the question, you need to draw two multimodal distributions whose domains do not overlap at all and then you can just say the linear model would should all to the right of some number would be class 1 and all to the left would be class 2


How do I classify an image that contains only polygons?

To make it easy take small known CNN network like Alex Net, train like : input  is image, output is [1,0,0,] separate, [0,1,0], encloses, [0,0,1] overlap. Cause you task is easy i guess that will be fine. But it can be done without ML just analysing image with constant algorithm by where are the points of poligons.


How does the AlphaGo Zero policy decide what move to execute?

The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo Zero as a whole.

There are 4 "phases" to the Monte-Carlo tree search in AlphaGo Zero as depicted in Figure 2. The first 3 expand and update the tree and together are the "search" in Monte-Carlo tree "search" in AlphaGo Zero.

Select an edge (action) in the tree with maximum action-value Q (plus upper confidence bound U)
Expand and evaluate the leaf node using the network
Backup Action-values Q are updated to track the evaluations of the Value in the subtree.
Play - After the "search" is complete, the search probabilities are returned proportional to the visitation count of the nodes of the tree.*

* This is where the formula in question comes into play (pun intended). During the "search", nodes that looked good were expanded on and thus their visitation counts were updated. So the formula is essentially describing this logic:

Good nodes have higher counts, so choose the nodes with higher counts often

But what if there is a really good node but it hasn't been visited a lot so its not chosen?

This is where the temperature parameter comes in:

If the temperature is 1, this selects moves proportionally to their visit counts.
If the temperature is 0 (not actually but rather in infinitesimal that approaches 0), and with some added noise, it "ensures that all moves may be tried, but the search may still overrule bad moves."

So altogether, the formula is saying : Pick good things most of the time

The mathematical evaluation of the formula is described below:

AlphaGo Zero defines the probability of each action (aka the policy) by that formula. If there are 3 nodes, A, B and C, and they have each been visited 10, 70, and 20 times (100 times in total), respectively, then the probability of taking those actions is:

P(A) = 10/100 = .10
P(B) = 70/100 = .7
P(C) = 20/100 = .2


What AI technique should I use to assign a person to a task?

What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of this area, although I think they don't talk too much about Task Allocation in particular.
There are two basic approaches to this problem: centralized approaches, and decentralized approaches.
In centralized approaches, the properties of each task (or sub-task) and the skills of each processing entity are recorded in a central database. The task is phrased as an optimization problem. For example, given the skills of the processors and the tasks' types, find the schedule that minimizes average processing time (or cost, or usage of rare-resource types, or whatever you're interested in). Common approaches include phrasing the optimization task as a linear-programming problem; phrasing the problem as a graph and using something like the graphplan algorithm; or phrasing the problem as a constraint satisfaction problem and using some kind of heuristic-guided local search.
There are all kinds of other more modern techniques too. I'm not aware of a survey paper for translation tasks in particular, but there are lots of examples in robotics and distributed computing.
Although good AI techniques exist for scheduling the task, they are predicated on being able to quantify the tasks' properties and the abilities of the agents, and on the translators accepting the decisions of the system. If you want an interactive system, you may need to look at techniques from Natural Language Processing. The work on Mixed Initiative Scheduling Systems might also be relevant if you have to go that route.


Handling imbalanced data with multiple targets

You can try weighting your training data instances. So, if for example class A has proportion $p_A$, you weight every instance of class A with $1/p_A$. There also exists more sophisticated approaches to train on unbalanced data, such as generating synthetic samples to create a balanced dataset and so on. You can start learning more here.


How can artificial intelligence predict the next possible moves of the player?

Artificial Intelligence can predict such a thing because before they release the game, the train the BOT or AI to play the game million times so they have a model or BOT that can predict every next move or combo that they can do or predict all moves that can finish the game.
for example snake game. what they do to predict moves is train the model or bot to play the game when the snake perform some action . the snake got reward which can be positive or negative reward. the goal of the snake is to learn what action that maximize the reward, given every possible state. States are the observations that the agent receives at each iteration from the environment

this is the link that can give you the detail : https://towardsdatascience.com/how-to-teach-an-ai-to-play-games-deep-reinforcement-learning-28f9b920440a


What is the gradient of the objective function in the Soft Actor-Critic paper?

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me.
I'll disregard expectation here to make things simpler. First, note that policy $\pi$ depends on parameter vector $\phi$ and function $f_\phi(\epsilon_t;s_t)$, and value function $Q$ depends on parameter vector $\theta$ and same function $f_\phi(\epsilon_t;s_t)$. Also, one important thing that authors mention in the paper and you didn't mention is that this solution is approximate gradient not the true gradient.
Our goal is to calculate gradient of objective function $J_\pi$ with respect to $\phi$, so disregarding the expectation we have:

$\nabla_\phi J_\pi (\phi) = \nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) - \nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t))$

Let's see the gradient of first term on right hand side. To get the full gradient we need to calculate derivative w.r.t to both variables, $\phi$ and $f_\phi (\epsilon_t;s_t)$, so we have:

$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) = \frac {\partial \log\pi(\phi,f_\phi (\epsilon_t;s_t))}{\partial \phi} + \frac{\partial \log\pi(\phi,f_\phi (\epsilon_t;s_t))}{\partial f_\phi(\epsilon_t;s_t)} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$

This is where approximation comes, they replace $f_\phi (\epsilon_t;s_t)$ with $a_t$ in some places and we have:

$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) \approx \frac {\partial \log\pi(\phi,a_t)}{\partial \phi} + \frac{\partial \log\pi(\phi,a_t)}{\partial a_t} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$
$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) \approx \nabla_\phi \log\pi(\phi,a_t) + \nabla_{a_t} \log\pi(\phi,a_t) \nabla_\phi f_\phi (\epsilon_t;s_t)$

For the second term in first expression on right hand side we have:

$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) = \frac {\partial Q(s_t,\theta,f_\phi (\epsilon_t;s_t))}{\partial \phi} + \frac{\partial Q(s_t,\theta,f_\phi (\epsilon_t;s_t))}{\partial f_\phi(\epsilon_t;s_t)} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$
$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) \approx \frac {\partial Q(s_t,\theta,a_t)}{\partial \phi} + \frac{\partial Q(s_t,\theta,a_t)}{\partial a_t} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$

Fist term on right hand side is 0 because $Q$ does not depend on $\phi$ so we have:

$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) \approx \nabla_{a_t}Q(s_t, \theta,a_t)\nabla_\phi f_\phi(\epsilon_t;s_t)$

Now you add up things and you get the final result.


Continuous ground truth in supervised (metric) learning?

See Deep Metric Learning Beyond Binary Supervision in CVPR 2019


Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results.

Technically this does not need to be 100% accurate or complete. It could even be a learned model. However, in the case of applying minimax to classic two player games, such as chess or Connect 4, then usually the game rules are used to create perfect predictions.

This difference between planning and learning is not quite the same as model-free vs model-based RL, but the ideas do overlap considerably. You can for instance consider the experience replay approach used in DQN as a form of "background planning" where the model used is the memory of previous events, whilst the core Q-learning algorithm used inside DQN is normally considered model-free.


Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. That's not why Q-learning is model-free. Q-learning assumes that the underlying environment (FrozenLake or MountainCar, for example) can be modelled as a Markov decision process (MDP), which is a mathematical model that describes problems where decisions/actions can be taken and the outcomes of those decisions are at least partially stochastic (or random). More precisely, an MDP is composed of

A set of actions $A$ (that the RL agent can take); for example, up and down, in some grid world
A set of states $S$ (where the RL agent can be);
A transition function $p(s_{t+1} = s' \mid s_{t} = s , a_t = a)$ (aka the model), which represents the probability of going to state $s'$ at time step $t+1$, given that at time step $t$ the RL agent is in the state $s$ and takes action $a$.
A reward function $r(s, a, s')$ (sometimes also denoted as $r(s)$ or $r(s, s')$, although these can have different semantics); the reward function gives the reward (or reinforcement) to the RL agent when it takes an action in a certain state and moves to another state; the reward function can also be included in the transition function, i.e., often you will also see $p(s_{t+1} = s', r_{t+1} = r \mid s_{t} = s , a_t = a)$, and this is the model: this is what we mean by model in reinforcement learning, it's this $p$ (which is a probability distribution)!

A model-free algorithm is any algorithm that does not use or estimate this $p$. Q-learning, if you look at its pseudocode, does not make use of this model. Q-learning estimates the value function $q(s, a)$ by interacting with the environment (taking actions and receiving rewards), but, meanwhile, it does not know or keep track of the dynamics (i.e. $p$) of the environment, and that's why it's model-free.
And, no, the value function is not what we mean by "model" in reinforcement learning. The value function is, as the name suggests, a function.

How does one know that a problem is "model-free" in reinforcement learning?

A problem is not model-free or model-based. An algorithm is model-free or model-based. Again, a model-free algorithm does not use or estimate $p$, a model-based one uses (and/or estimates) it.

Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?

No. As stated in the other answer, you could apply the model-based algorithm Dyna-Q to these environments.


How to let the agent choose how to populate a state space matrix in RL (using python)

I think this question is hinting at the problem of choosing an exploration strategy.
The simplest strategy is to use the so called epsilon-greedy strategy (or $\epsilon$-greedy). This means that you select an action at random $x$ percent of the times that an agent has to select an action. The other times, the agent takes the action that its current policy dictates. Usually, $x$ declines throughout the learning process.
There are many different exploration strategies to choose from, with the aforementioned one being really the most simple one. For getting an impression of how diverse these different approaches to exploration strategies are, please be referred to Google Scholar (searching for "reinforcement learning exploration strategies").


What to do with a GAN that trained well but got worse over time?

This is from my own experience with (Vanilla) GANs, so it might not translate exactly to your application, but maybe it gives some orientation.

your learning rate seems quite high. I've quite frequently found that 1e-5 is a good value for me. The training might take longer but will probably be more stable.
have you tried using dropout? It's a good regularisation mechanism that can prevent from overfitting and also make training more stable. I've made good experience with a dropout rate of ~0.4 and keeping it in both training and testing stages.
what is your discriminator vs generator update ratio? In my experiments, ~5 discriminator updates per generator update has often shown to improve things.
how many training samples do you have? If they are limited, it might be helpful to add a bit of noise on them. This manipulates the data a bit, but might make training more stable.
I have often not seen a major change when varying the batch size.
what are your network architectures? I have the feeling that small networks are often sufficient...

This is entirely based on my own experience, but maybe there are some interesting directions for you to explore. Happy to hear back from you about the success of these tests.


Why not use a single agent to control the intersection with the following rules:

define 8 traffic lights
Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green
Lights are as follows:

0 = North to South
1 = North to East (Left Turn)
2 = East to West
3 = East to South (Left Turn)
4 = South to North
5 = South to West (Left Turn)
6 = West to East
7 = West to North (Left Turn)

map lights into a bitmap of 8 bits, with the least significant bit (left bit) is the North to south

Given the above, we have the following valid moves, which are both State Space  and Action Space :

State/Move 0 - Vertical Moves only   = 10001000
State/Move 1 - Horizontal Moves only = 00100010
State/Move 2 - Originated from North only = 11000000
State/Move 3 - Originated from East only = 00110000
State/Move 4 - Originated from South only = 00001100
State/Move 5 - Originated from West only = 00000011
State/Move 6 - Vertical Diagonal only = 01000100
State/Move 7 - Horizontal Diagonal only = 00010001

i.e. S = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

and  A = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

Now we have a ∈ A and s ∈ S and at any given s, a will cause the next state s' to have the same value of a

Final component to complete the moel, is the reward signal, which can be simply the negative sum of time spent per car in the intersection

i.e.     rt+1 = -∑ci ∈ Cars time(ci)

a single agent, will be able to optimize the solution to avoid punishment (negative value caused by wait time)

Update

You can increase the state space to double or triple its size to accommodate for time, by appending a time counter component to the bitmap.
in addition to an extra (9th) action of do nothing,

This should give a more subtle traffic controller. e.g. Vertical Traffic Only, at timer = 0 can be represented by 100010000000 or in hex 880, then at time 1 = 881, and so on until time 15 to be 88F.

Update 2

You can find the source code here and here


Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties.
The 'human' representation of a word, a sequence of letters and other symbols, is not related at all to its meaning or use in actual text. It only serves as a look-up key into our cognitive language processing facility (however that actually works) which enables us to understand the meaning in the context of its usage. However, a computer system does not have such a facility.
In order to convert the character string into something more usable for language processing, embeddings are created. These are typically vectors describing other words that surround a word in question (as the meaning of a word depends on its context). There are obvious problems with ambiguities (eg is bank the side of a river or a place where you deposit money?), but there are probably ways around that.
So the embedding (a vector) represents the usage of a word, which strongly correlates with its meaning. Because words as character sequences are useless for most sub-symbolic processing, the terms embedding and word are possibly used interchangably.


Does the number of parameters in a convolutional neuronal network increase if the input dimension increases?

If I have a convolutional neuronal network, does the input dimension change the number of parameters? And if yes, why?

If the convolutional neural network (CNN) only uses convolutional layers, then the number of parameters does not increase as a function of the spatial dimensions ($x$ and $y$) of the input. This is one of the advantages of CNNs!
The reason is quite simple: the parameters of the convolutional layers are the kernels (aka filters), which typically have fixed dimensions and can, nevertheless, be applied to inputs of different spatial dimensions, provided that the necessary padding is used. However, note that padding can create bigger feature maps, but feature maps are not the parameters of the neural network: they are the output of the convolutional layers. That's probably what confuses you, when you see a diagram of a CNN, because you see bigger feature maps and you might think that the number of parameters increases.
Here's a TensorFlow 2 (and Keras) program that shows that the number of parameters does not change as a function of the $x$ and $y$ dimension of the input.
import tensorflow as tf

input_shapes = [(2 * k, 2 * k, 3) for k in range(2, 6)]

for input_shape in input_shapes:
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
    model.summary() # The total number of parameters is always 280

The parameters of a convolutional layer can increase if you increase the size of each kernel and the number of kernels, but this does not necessarily depend on the input.
The parameters of the CNN can also increase if you increase the depth of the input, but that's typically fixed (either $3$ for RGB images or $1$ for grayscale images). The reason is quite simple too: the kernels in the first convolutional layer (connected to the input layer) will have the same depth as the depth of the input.
If your CNN also has fully connected layers, then the number of parameters also depends on the dimensions of the inputs. This is because the parameters of the fully connected layers depend on the number and dimensions of the feature maps (remember the flatten layer before the fully connected layers?), which, as I said, can increase as a function of the input.
If you don't want to use fully connected layers, you may want to try fully convolutional networks (FCNs), which do not make use of fully connected layers, but can, nonetheless, be used to solve classification (and other) tasks.


Which evaluation methods can I use for image segmentation?

See:

  Martin Thoma: A Survey of Semantic Segmentation, Section III

Subsection A is about metrics and B is about datasets.

Metrics include: accuracy, IoU, frequency weighted IoU, F-beta score, speed, ...


Use Reinforcement Learning instead of genetic algorithm for optimization

In order to have anything resembling reinforcement learning you must at the very least have a set of states $S$ and a set of actions $A$.
In your formulation I can vaguely identify the set of states $S$ as all possible $(x,y,z)$ triplets. But don't see anything in your description that could be interpreted as a set of actions $A$. You either oversimplified the description of your problem or reinforcement learning is not applicable here by lack of very basic ingredients for it.


Ontology learning is a relatively new field that aims to automatically (or semi-automatically) learn or create ontologies (using machine learning, text mining, knowledge representation and reasoning, information retrieval and natural language processing techniques) from some text or corpus.

Ontology learning can be divided into different phases or tasks

the acquisition of terms that refer to specific concepts (named-entity
recognition)
the recognition of synonyms among these terms
the identification of taxonomic relations (such as the "is-a" relation)
the establishment of non-hierarchical relations
the derivation of new knowledge, i.e. knowledge that is not explicitly
encoded by the ontology.

See also Ontology Learning from Text: An Overview (2003) and A survey of ontology learning techniques and applications (2018) for more details.

In the paper Ontology Learning with Deep Learning: a case study on Patient Safety using PubMed (2016), the authors investigate how continuous bag-of-words (CBOW) and skip-gram (two language models based on artificial neural networks) can be used to aid ontology development for patient safety, using PubMed citations as a corpus.

Latent Dirichlet allocation (LDA) has also been used for ontology learning, for example, in the paper Terminological ontology learning and population using latent Dirichlet allocation. (2014).


How Graph Convolutional Neural Networks forward propagate?

I think the picture you're presenting is mostly for educational purposes and that's why they are excluding the node itself from it's neighbors and using two distinct networks (most of the papers I've read they are using the same network for the neighbors and for the center node). But you are right the two networks needs to have the same input and output shapes otherwise the point-wise summation between the two terms is not possible.


What is the difference between a Bayesian Network and a Markov Chain?

The main difference between a Bayesian network and a Markov chain is not that a Markov Chain is not directional, it is that the graph of the Bayesian network is not trivial whereas the graph of a Markov chain would be somewhat trivial, as all the previous $k$ nodes would just point to the current node. To illustrate further why this would be trivial, we let each node represent a random variable $X_i$. Then the nodes representing $X_i$ for $ t-k \leq i < t$ would be connected by a directed edge to $X_t$. That is, the edges $(X_i, X_t) \in E$ for $ t-k \leq i < t$ and where $E$ is the set of edges of the graph.

To illustrate this please see the examples below.

Assume that we have a $k$th order Markov chain, then by definition we have $\forall t > k$ $\mathbb{P}(X_t = x | X_0,...,X_{t-1}) = \mathbb{P}(X_t = x | X_{t-k},...,X_{t-1})$.

The main difference between the above definition and the definition of a Bayesian Network is that due to the direction of the graph we can have different dependencies for each $X_t$. Consider the Bayesian Network in the Figure below

We would get that $\mathbb{P}(X_4 = x| X_1, X_2, X_3) = \mathbb{P}(X_4 = x | X_2, X_3)$ and $\mathbb{P}(X_5 = x| X_1, X_2, X_3, X_4) = \mathbb{P}(X_5 = x | X_3)$.

So, the past events that the current random variable depends on don't have to have the same 'structure' in a Bayesian Network as in a Markov Chain.


Is a super-intelligence limited by the intelligence of its creator?

Of course the intelligence of a product is limited by the intelligence of its creator. Just not to the intelligence of its creator.

That would be about as reasonable as the idea that the speed of a car is limited to the speed of its creator.

Or the playing strength of a chess program to the Elo of its creator.

Or the ability of a neural network to differentiate between dozens of dog breeds to the dog expertise of its creator.

So, not very reasonable.


What are the aspects that most impact on the inference time for neural networks in embedded systems?

You can expect that the inference time will strongly depend on particular hardware and software present on your platform. First, GPU equipped devices (eg NVidia TX) will outperform non-GPU equipped devices (eg. Intel Movidius). Second, software support (eg. cudnn, TensorRT) will make dramatic further impact.

For instance, we have measured the inference time of two convolutional models. The model A requires 250% more floating point operations than the model B. Yet, the two models take roughly the same time to evaluate on our device, since the layers of model A are better optimized in software. Conclusion: algorithmic complexity and practical execution time on a particular computing platform are not bound to be proportional any more.


How can I design the input layer of a feed-forward neural network to be trained with a medical dataset with three features?

I would give some advice here:

You should probably use Convolutional Neural Network for image data or you'll end up with too many input features, and too much computation. What are your constraints on this point?

I would not mix image data with numerical data. Maybe you can implement two neural networks, one handling image data and another managing the numerical data, each one outputting their results. A second step would be to use their results to handcraft a global result or let another NN manage this.


Does depth-first search always stop when it has found the leftmost solution?

One of the more standard assumptions when first introducing new students to search algorithms (like Depth-First Search, Breadth-First Search which you've also likely heard about or will hear about soon, etc.) is indeed that our goal is to find some sort of solution, and only find one.

If our intention is to find just a single solution, then yes, you will need to check at every node whether that is a solution node, and you can stop the search process once you've found one.

In practice, there can be all kinds of variants of this idea. Maybe in a different case you are interested in finding all solutions, rather than a single one; in such a case you would naturally not stop the search process after finding the first one, but continue searching.

So, to conclude, it really depends on exactly what you want, why are you using a search algorithm. If you only care about finding a solution, you can stop when you have one.


What are the prerequisites to start using the TensorFlow Probability library?

Although this question is slightly primarily opinion-based and too broad (and I will probably close it as such) and a good answer will necessarily depend on your background, I will list some of the main theoretical prerequisites that everyone should ideally be familiar with before diving into TensorFlow Probability (TFP).

I am familiar with TFP, given that I've been using it for a project, but I've not used all of its functionalities, such as the bijectors. I've only used the Bayesian layers, distributions, etc., so I will try to give an answer based on my experience.

You definitely need to be familiar with the basic concepts of probability theory, such as distributions, random variables, expectations, etc. A full university-level course in probability and statistics would definitely be helpful!

If you want to use Bayesian layers, you also need to be familiar with Bayesian neural networks (BNNs). To understand BNNs, you need to understand the basic concepts of Bayesian inference (BI), given that most BNNs are based on BI. If you are already familiar with variational auto-encoders (VAEs), then it will be easier to understand BNNs. To understand BNNs, you should start reading the paper Weight Uncertainty in Neural Networks. If this paper is difficult to follow, you should first read the paper Variational Inference: A Review for Statisticians and then it may also be useful to read and be familiar with the VAE paper. Therefore, you will need to be familiar with concepts such as Kullback–Leibler divergence and Monte Carlo sampling.

The blog post Regression with Probabilistic Layers in TensorFlow Probability is the first blog post you should read. If you don't understand it, then it probably means you need to learn its prerequisites (i.e. you need to be familiar with at least linear regression).

TFP also provides several implementations of other more advanced concepts such as BNNs. The easiest example to follow is probably the logistic regression example. LR is relatively easy to follow compared to the topics I mentioned above, but, of course, you need to be familiar with logistic regression.

Of course, I cannot list all the theoretical prerequisites (and that's why your post is too broad!) and there are definitely others, but this is a start. Bear in mind that these are not trivial topics, so it is normal to get stuck.


How to calculate probability from fuzzy membership grade?

No, you can't extract any probability from a fuzzy membership grade. The uncertainty expressed by fuzzy logic is about partial truth, not about probability. $ \mu_S(x) = 0.9 $ doesn't mean that "$ x $ is tall" is true with a probability of 0.9, but that "$ x $ is tall" is 90% true (notice the difference in semantics). You have to think about fuzzy logic as an extension of logic (as its name implies), rather than an extension of probability.
It's true, however, that fuzzy logic is flexible and lets you define how the membership grades are combined in logic formulae, to the extend you can replicate probability theory within the fuzzy logic framework. Wikipedia has a good overview on this: https://en.wikipedia.org/wiki/Fuzzy_logic#Comparison_to_probability.
However, please understand that, in general, fuzzy membership $ \neq $ probability. How we come up with the fuzzy membership grade is subjective and application-dependent. Conversely, probabilities have a well-defined and unambiguous interpretation. The point of being fuzzy is to replicate our reasoning process which, even if is not necessarily formal and rigorous, is often very accurate. To do so it needs a set of (admittedly arbitrary) rules on how to calculate the "truthfulness" of logical formulae. This may turn to be very useful in applications where manipulating probabilities or coming up with them in the first place is not tractable.


Why is the Backpropagation algorithm used to train the Multilayer Perceptron?

According to wikipedia of backpropagation:

In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the network for a single input–output example, and does so efficiently, unlike a naive direct computation of the gradient with respect to each weight individually.

Backpropagation is a special case of reverse accumulation of automatic differentiation, and it was announced in Rumelhart, Hinton & Williams (1986). Automatic differentiation has 2 modes: forward accumulation and backward accumulation.

Automatic differentiation is distinct from symbolic differentiation and numerical differentiation (the method of finite differences). Symbolic differentiation can lead to inefficient code and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where complexity and errors increase. Finally, both classical methods are slow at computing partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems.

Usually, two distinct modes of AD are presented, forward accumulation (or forward mode) and reverse accumulation (or reverse mode). Forward accumulation specifies that one traverses the chain rule from inside to outside, while reverse accumulation has the traversal from outside to inside...

Forward accumulation is more efficient than reverse accumulation for functions $ f:ℝ^n → ℝ^m$ with $m ≫ n$ as only $n$ sweeps are necessary, compared to $m$ sweeps for reverse accumulation.

Reverse accumulation is more efficient than forward accumulation for functions $ f:ℝ^n → ℝ^m$ with $m ≪ n$ as only $m$ sweeps are necessary, compared to $n$ sweeps for forward accumulation.

Under this historical computational differentiation techniques context, since the loss function in a MLP maps $ℝ^n \rightarrow ℝ^m$ with $m ≪ n$ for most cases, that's one of the main reasons why backpropagation is most often used to train a MLP as a special case of reverse accumulation introduced above.


How does a neural network that has been trained keep learning while in a real world scenario

You are right. If you don't continuously train the neural network after you have deployed it, there is no way it can continuously learn or be updated with more information. You need to program the neural network to learn even after it has been deployed. There is no such thing as a neural network that decides what it does without a human deciding first what it needs to do: this is a very common misconception (probably caused by the media and science fiction movies). It's also true that you need to label your data if you intend to train the neural network in a supervised fashion, but there are other ways to train neural networks (e.g. by reinforcement learning), depending also on the problem you want to solve.
If you want to develop neural networks that can learn continually, you probably want to look into continual learning techniques for neural networks. Another term that you may be looking for is online machine learning.


Is it possible to guide a reinforcement learning algorithm?

The programmer already guides the RL algorithm (or agent) by specifying the reward function. However, the reward function alone may not be sufficient to learn efficiently and fast, as you correctly noticed.

To attempt to solve this inefficiency problem, one solution is to combine reinforcement learning with supervised learning. For example, the paper Deep Q-learning from Demonstrations (2017) by Todd Hester et al. describes an approach to achieve this.

The paper Active Reinforcement Learning (2008) by Arkady Epshteyn et al. also tries to solve this problem but by incorporating approximations (given by domain experts) of the MDP.

There are probably many other possible solutions. In fact, all model-based RL algorithms could probably fall into this category of algorithms that estimate or incorporate the dynamics of the environment to find a policy more efficiently.


What is the most accurate pretrained sentiment analysis model by 2019?

For best results, I'd recommend Google Cloud Machine Learning. It has [Natural Language Processing API] (https://cloud.google.com/natural-language/docs/basics) with Sentiment, Entity, and Entity-Sentiment analysis.

You can implement in C++, PHP, Python, or other languages. This does require running a virtual machine instance on Google Cloud. TensorFlow can also be used for NLP sentiment analysis. The only drawback is it's not economical.

As an alternative, this is something I came across recently by Fast Text : http://fasttext.cc.
They have some pre-trained models trained on yelp and some other datasets. They support a lot of languages


Did Minsky and Papert know that multi-layer perceptrons could solve XOR?

There does not appear to be a historical consensus on this.
The Wikipedia page on the Perceptrons book (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts).
However, this page gives an account by someone present at the MIT AI lab in 1974, claiming that this was not common knowledge there, alluding to documentation in "Artificial Intelligence Progress Report: Research at the Laboratory in Vision, Language, and other problems of Intelligence" (p31-32) which is claimed to support this.


Can a neural network with linear activation functions produce a connection of linear functions?

A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.
If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input, where the next layer's neurons would also calculate a weighted sum on that input, and, in turn, it fires based on another linear activation function.
No matter how many layers and neurons there are, if all are linear in nature, the final activation function of the last layer is also a linear function of the input of the first layer.
That means that any or all of these layers can be replaced by one layer. This completely loses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation, given that a combination of linear functions in a linear manner is still another linear function.
A good way to visualize this messing around with Tensorflow playground, which has a spiral data set similar to your data.
In contrast to the failure of linear activation functions for this data in the previous link, check out this much smaller network using a Tanh activation function which can separate the data within 100 or so iterations.
For further reference on activation functions, checkout this blog post.


How does general image background removal AI work?

In image segmentation the target is actually an image, with the same dimensions as the input, where each pixel has a label depending on which class it represents. It is not uncommon for such a dataset to have a "background" class that essentially consists of the pixels not belonging to any other class. If not you can always group together classes typically associated with background (e.g. "sky", "cloud", "grass", "mountain", etc.) to form the class "background". Likewise you could group all other possible classes of interest (e.g. "person", "car", "horse", etc.) into the class "foreground". With this dataset you could train an image segmentation model that predicts if a pixel belongs to the background or the foreground, without actually classifying it into a "person" or a "car".
So suppose you want to make your own removal.ai, you could:

find one or more diverse image segmentation datasets (it needs to be diverse so that it will work on any generic photo uploaded to the site)
check all the unique classes in the labels
group all classes associated with backgrounds into class 0 (i.e. "background" class)
group all classes associated with foregrounds into class 1 (i.e. "foreground" class)
train an image segmentation model with these two classes
profit


How can I determine the bias and variance of a random forrest?

To gain a good understanding of this, I recommend first reading about the trade-off between bias and variance in ML and AI methods.

A great article on this topic that I recommend as a light mathematical introduction is this:
https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229

In short:
Bias represents the models effort to generalize samples, as opposed to Variance that represents the models effort to conform to new data.
A high bias, low variance model will thus look more like a straight(underfitted) line, while a low bias, high variance model will look jagged and all-over the place(overfitted).

In essence, you need to find a balance between the two to avoid both overfitting(high variance, low bias) and underfitting(high bias, low variance) for your specific application.

  But how can I determine this for a model such as a Random Forrest classifier?

To determine your models bias and variance configuration(if either is too high/low), you can look at the models performance on the validation and test set.
The very reason we divide our data into training-validation-test sets, is so that we can validate the models performance when it is presented with samples it has not seen during training.


What limits, if any, does the halting problem put on Artificial Intelligence?

The halting problem is an example of a general phenomenon known as Undecidability, which shows that there are problems no Turing machine can solve in finite time. Let's consider the generalization that it is undecidable whether a Turing Machine satisfies some non-trivial property P, called Rice's theorem.

First note that the halting problem applies only if the Turing machine takes arbitrarily long input. If the input is bounded, it is possible to enumerate all possible cases and the problem is no longer undecidable. It might still be inefficient to calculate it, but then we are turning to the complexity theory, which should be a separate question.

Rice's theorem implies that an intelligence (a human) cannot be able to determine whether another intelligence (such as an AGI) possesses a certain property, such as being friendly. This does not mean that we cannot design a Friendly AGI, but it does mean that we cannot check whether an arbitrary AGI is friendly. So, while we can possibly create an AI which is guaranteed to be friendly, we also need to ensure that IT cannot create another AI which is unfriendly.


What are the best known gradient-free training methods for deep learning?

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed:

Particle Swarm optimization - I would say that this is one of the better optimization algorithms to train neural networks other than back propagation. I am currently using it and have achieved quite good results.
Genetic Algorithms - I have tried to use genetic algorithms to train neural networks in the past and I was not able to get it to work. However, I was using deep neural networks with almost a million parameters and the performance was not that good.
Simulated annealing - simulated annealing is based off of metals cooling. I have seen simulated annealing work fairly well but maybe not as well as particle swarm optimization.
Derivatives of genetic algorithms - derivatives of genetic algorithms such as NEAT have been shown to work pretty well. I have not personally used them extensively but some of the things that people have used them for are pretty cool.


How would researchers determine the best deep learning model if every run of the code yields different results?

I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?

That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is that it is the best one given the hyperparameters you have selected and that specific seed. Sometimes the value of the seed is highly relevant, other times less so.

  Since results might be stochastic, how would researchers know what their best performing model is?

In general, as with any experiment where measurements are variable, by running the experiment multiple times and taking statistics over the set of results. This will give you a much better sense of how the algorithm does in general, independently of specific seeds. You can still fix your RNG seeds for repeatability, but you will need multiple sets of them.

For certain goals, such as making the best possible model that you can, independently of whether the approach you take is "best in general", this is not necessary. A single run which creates a state-of-the-art performance is still of interest, for instance. Or if you are creating a model that you want to use in production, you may care less about the stability of the technique (and being "lucky") than being in possession of a high performing agent for the task.


Must people tell an AI which algorithm it should use? Can an AI learn algorithms by itself?

There are problems we for which we don't have a known, optimal, deterministic algorithm. By and large we use heuristics to "solve" those problems.  A closely related idea is that of satisficing where we seek out answers that are "good enough" for immediate purposes.

Likewise, machines can also use heuristics, whether they are programmed in explicitly or, presumably, learned.  Within the range of ways that a machine can use heuristics, there are meta heuristics and hyper heuristics.

Going beyond that, there are other ways that machines an learn "algorithms" or "rules" for solving problems.  One are that I'm particularly interested in is known as rule induction.

This is all an area of open and active research BTW... so if you're interested in exploring any of these approaches, you'll probably find a lot of ground to cover.


Is the traditional meaning of "strong AI" outmoded?

The meaning of strong AI has changed as you correctly indicated, and the term "strong narrow AI" is more appropriate as now people are shifting towards the practical uses.
This also bring to light the number of breakthroughs that AI is having and literature and other various related resources will often(unlike in other fields) need to be changed constantly.