How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

Here is what I discovered empirically, trial and error. Since tuning the parameters are going to be environment specific, I'll lay out mine to give a better understanding of what I found to work for my case. Hopefully someone with better understanding of the algorithm will weigh in:
Environment: A 2D map where an agent controls a simulated PC mouse pad and must navigate from a random spawn point to a random reward point.
Action Space: Discrete(24) -- This was flattened from the original implementation to allow for use in the Q* algorithms.

0-7, mouse buttons, each with one of two states pressed or depressed

public ButtonPressState Left;
public ButtonPressState Middle;
public ButtonPressState Right;
public ButtonPressState Touch;
public ButtonPressState ScrollUp;
public ButtonPressState ScrollDown;
public ButtonPressState IncreaseScroll;
public ButtonPressState DecreaseScroll;

8-15: Angle to move if Touch is in state pressed, incremented by 45°, starting at 0 °up to 315°

public enum D8Dir
{
    UP = 0,         //   0°
    UP_RIGHT = 1,   //  45°
    RIGHT = 2,      //  90°
    DOWN_RIGHT = 3, // 135°
    DOWN = 4,       // 180°
    DOWN_LEFT = 5,  // 225°
    LEFT = 6,       // 270°
    UP_LEFT = 7     // 315°
}

16-23: Speed, an exponent of 2^N representing how may cells to move per step.

Distance normalization: Trial and error led me to normalizing the space. This also brings the benefit of allowing the same trained algorithm to be used for different world sizes, eg a 2x3 vs a 3x2 vs a 3x3 on up to large common monitor dimensions. Specifically, distances are normalized between -1 and 1 for the height and the width of the space. So no matter the dimensions all points map to x and y between -1 and 1. In the case the height and the width differ the smaller is scaled to the size of the larger.
Reward space: After having agents reward exploit iteration after iteration of reward scheme, Rewards are given when an agent moves closer to the target - with closer taking into consideration both "birds eye" and Euclidian distance. After first, took "theoretical" maximum sum of the total rewards summed up to 1 and were assigned by taking the maximum distance an agent might move in a given 2D space to gain a reward. This works out to be the diagonal of the 2D space. I say "theoretical" because unless the agent randomly spawns exactly in one corner and the reward is placed exactly in the other corner, the actual total reward that can be achieved is lower. So in practice the maximum reward for a given episode is the sum of normalized distances between the agent and the reward.
To avoid reward exploitation, only .70% of the reward if the agent moves closer vertically or horizontally when a diagonal move was available. This is to get the agent to take the shortest path moving diagonally with only the sqrt(2) bird's eye reward. Otherwise and agent might move up, then left and collect a reward of 2 normalized units instead of the ~=1.4142 normalized reward units collected for moving diagonally. The .70% keeps the reward for an vertical then horizontal (or vice versa) move to 1.4 which is less than the ~=1.4142 diagonal reward.
Originally, the agent could collect up to a total reward of up to 1 and then 3 additional reward points for reach the target. Using the stock TensorFlow c51 DQN agent against this reward scheme, using the default min_q_value=-10 and max_q_value=10 values (I believe they are referred to as VMIN and VMAX in the literature) I found learning achieved the following results on very small world sizes:
[5/2/2021 5:39:19 PM] Training Started
[5/2/2021 5:39:33 PM] Training Environment (1, 2): 5/2/2021 5:39:33 PM
[5/2/2021 5:40:11 PM] Mastered (1, 2) in 00:00:38.8244035 - Steps 2565 Episode 948  Average: 3.000000000000002
[5/2/2021 5:40:11 PM] Training Environment (2, 1): 5/2/2021 5:40:11 PM
[5/2/2021 5:41:01 PM] Mastered (2, 1) in 00:00:49.1426355 - Steps 5802 Episode 1242  Average: 3.000000000000002
[5/2/2021 5:41:01 PM] Training Environment (2, 2): 5/2/2021 5:41:01 PM
[5/2/2021 5:43:42 PM] Mastered (2, 2) in 00:02:41.2581146 - Steps 16573 Episode 3248  Average: 3.0551471839999977
[5/2/2021 5:43:42 PM] Training Environment (2, 3): 5/2/2021 5:43:42 PM
[5/2/2021 5:53:44 PM] Mastered (2, 3) in 00:10:01.8163250 - Steps 56732 Episode 11226  Average: 3.0981306360000063
[5/2/2021 5:53:44 PM] Training Environment (3, 2): 5/2/2021 5:53:44 PM
[5/2/2021 5:59:13 PM] Mastered (3, 2) in 00:05:29.5753945 - Steps 78612 Episode 6276  Average: 3.108986006000001
[5/2/2021 5:59:13 PM] Training Environment (3, 3): 5/2/2021 5:59:13 PM
[5/2/2021 6:22:50 PM] Mastered (3, 3) in 00:23:37.2302437 - Steps 173577 Episode 25603  Average: 3.1401708113999938
[5/2/2021 6:22:50 PM] Training Environment (4, 4): 5/2/2021 6:22:50 PM
[5/2/2021 7:10:04 PM] Mastered (4, 4) in 00:47:13.5879491 - Steps 363685 Episode 46441  Average: 3.1596743723999965
[5/2/2021 7:10:04 PM] Training Environment (5, 5): 5/2/2021 7:10:04 PM
[5/2/2021 7:20:08 PM] Mastered (5, 5) in 00:10:04.1529582 - Steps 404256 Episode 8916  Average: 3.182016046800004
[5/2/2021 7:20:08 PM] Training Environment (6, 6): 5/2/2021 7:20:08 PM
[5/3/2021 12:33:49 AM] Killed 6x6 - No Convergence: Step 1671400: Reward 1.8765 loss = 2.90894175 (00:00:02.9459997) (eps: 2.9459996999999998)

As can be seen, even for a simple 6x6 world C-51 DQN did not converge even after over 4 hours of wall time on GTX-3080 sampling 1,671,400 steps so I killed it.
So I modified the total possible reward to for the agent moving to sum up to .5. This was simply a matter of dividing the old reward scheme in half (or rather *.5).
I then changed the target reached reward from 3 to .5.
The theoretical maximum reward then totaled to 1, so I changed min_q_value=0 and max_q_value=1 to match the sum of the rewards an agent might achieve. This resulted in wall times ever larger than before.
My latest attempt I used min_q_value=0 and max_q_value=.5 to match the maximum cumulative reward the agent could recieve before the collecting the .5 target reached reward, (which BTW is also a terminal state in my environment).
The new rewards, normalized to total to 1 run orders of a magnitude faster for the larger spaces. The 6x6 world "converged" after 05 minutes, 43 secs using 88721 total steps in  88721 in 4624 total episodes.
I still find this slow, but clearly the q values matching the range of cumulative rewards is an improvement.
[5/3/2021 12:49:35 AM] Training Environment (1, 2): 5/3/2021 12:49:35 AM
[5/3/2021 12:50:12 AM] Mastered (1, 2) in 00:00:36.8244207 - Steps 2424 Episode 981  Average: 0.5000000000000002
[5/3/2021 12:50:12 AM] Training Environment (2, 1): 5/3/2021 12:50:12 AM
[5/3/2021 12:50:37 AM] Mastered (2, 1) in 00:00:24.6645133 - Steps 4075 Episode 700  Average: 0.5000000000000002
[5/3/2021 12:51:52 AM] Mastered (2, 2) in 00:01:15.0585713 - Steps 8135 Episode 1419  Average: 0.5314644659999993
[5/3/2021 12:51:52 AM] Training Environment (2, 3): 5/3/2021 12:51:52 AM
[5/3/2021 12:52:39 AM] Mastered (2, 3) in 00:00:47.0126118 - Steps 11266 Episode 961  Average: 0.5455624971999997
[5/3/2021 12:52:39 AM] Training Environment (3, 2): 5/3/2021 12:52:39 AM
[5/3/2021 12:52:56 AM] Mastered (3, 2) in 00:00:17.6139995 - Steps 12447 Episode 395  Average: 0.5650996157999993
[5/3/2021 12:52:56 AM] Training Environment (3, 3): 5/3/2021 12:52:56 AM
[5/3/2021 12:55:56 AM] Mastered (3, 3) in 00:02:59.7056623 - Steps 24472 Episode 3403  Average: 0.5863068707999991
[5/3/2021 12:55:56 AM] Training Environment (4, 4): 5/3/2021 12:55:56 AM
[5/3/2021 1:00:07 AM] Mastered (4, 4) in 00:04:11.2932882 - Steps 41259 Episode 4240  Average: 0.6052226153999957
[5/3/2021 1:00:07 AM] Training Environment (5, 5): 5/3/2021 1:00:07 AM
[5/3/2021 1:06:11 AM] Mastered (5, 5) in 00:06:03.8448782 - Steps 65641 Episode 5502  Average: 0.6209708242799977
[5/3/2021 1:06:11 AM] Training Environment (6, 6): 5/3/2021 1:06:11 AM
[5/3/2021 1:11:54 AM] Mastered (6, 6) in 00:05:43.2033837 - Steps 88721 Episode 4624  Average: 0.6089312343400013

FYI: The paper states that Transitions to a terminal state are handled with γt = 0.

Update:
The 9x9 environment took 294,897 steps, in 37,025 episodes over 51 minutes and 38 seconds. It may be the case that the max_q_value=1 works better in the larger world sizes where an agent might collect more movement reward. In any case, these values are nearly quadratically better than the default values from the tutorial.
I will experiment with them more.
Additionally, tuning n-steps might help. My implementation may also have an issue using the replay buffer memories from smaller world sizes as I am changing the world size dynamically an currently not clearing the buffer.
[5/3/2021 2:03:33 AM] Mastered (9, 9) in 00:51:38.4201434 - Steps 294897 Episode 37025  Average: 0.6425936848799915


Can a LFSR be approximated by a Neural Network?

According to the abstract of the following book, yes it's possible:

"Prediction of Sequences Generated by LFSR Using Back Propagation MLP"

Abstract:
Prediction of the next bit in pseudorandom sequences is one of the main issues in cryptology in order to prove the robustness of the systems. Linear complexity has served as a reference measurement to evaluate the randomness of the sequences, comparing them with the shortest LFSR that can generate those sequences. Several tools based on artificial intelligence have also been used for the next bit prediction, such as the C4.5 classifier. In this paper, we apply a different approach, the back propagation neural networks, to predict the sequences generated by LFSR. The results confirm that these networks can predict the entire sequence knowing less input patterns than techniques based on classifiers.


Is there a name for this approach to evolutionary algorithms?

It's just a genetic algorithm, only your population is a set of instructions that generate a subject to be tested. There are minor differences in the way you apply multiple mutations, so you evaluate groups of them instead of each individually, but for the scoring you effectively single out the commonalities among the worse individuals.
Hard to tell whether it's a good idea; discarding poor mutations based on them being part of a sub-optimal individual might not be as straight forward as it sounds. And where do you get new mutations from?


Adding a dense layer after a conv2d layer in a convolutional autoencoder

Hi for me this worked perfectly. I encoded with conv2d and dense and then I flatten I and reshape in the decoder after the dense layer so the encoder and decoder are symmetrical. The only difference is that in my case I use images (224,224,1)
        # create encoder
        # 28,28 -> 1st conv2d (filter 3x3,relu activation, padding, strides == 'jumps')
        self.encoder = tf.keras.Sequential([layers.Input(shape=(224,224,1)),
                                            layers.Conv2D(16,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2D(8,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2D(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Flatten(),
                                            layers.Dense(units=3136,activation='sigmoid')]) # (28,28,4)

        # deconvolution -> decoding
        self.decoder = tf.keras.Sequential([layers.Input(shape=(3136)),
                                            layers.Dense(units=3136,activation='sigmoid'),
                                            layers.Reshape((28,28,4)),
                                            layers.Conv2DTranspose(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                            layers.Conv2DTranspose(8,kernel_size=3,strides=2,activation='relu',padding='same'),
                                            layers.Conv2DTranspose(16,kernel_size=3,strides=2,activation='relu',padding='same'),
                                            layers.Conv2D(1,kernel_size=(3,3),activation='sigmoid',padding='same')])


what will i be able to do in the end of AI: modern approach?

"Artificial Intelligence: A Modern Approach" (AIMA) by Russel and Norvig is a general introductory book on AI. That means it not only covers sub-symbolic AI (like machine learning) but also symbolic AI. Therefore, it can "only" give you an overview of each topic (I put only in quotation marks since it is actually quite ambitious to cover all topics of AI in a single book!).

Regarding the topics you have mentioned that means:

you will have read about the basics of neural networks but will not have seen much about deep nets (the 3rd edition only has 10 pages on neural networks, i.e. it's really basic)
you will have read the basics of reinforcement learning (the 3rd edition has 23 pages on this, i.e. it's basic but more comprehensive than neural networks and enough to read papers about the topic)
you will not have read anything about NEAT but have read about the basics of genetic algorithms and neural networks which NEAT is based on (and can read the corresponding papers afterwards)

So, regarding your preferred topics, AIMA is one way to get started if you're not only looking for a selectively deep but also a broad understanding of AI. The road could go like this:

Read a general AI book, e.g. AIMA
Read a general ML book, e.g. "The Elements of Statistical Learning" by Hastie, Tibshirani, Friedman
Read deep dive books on selected topics, e.g. "Deep Learning" by Goodfellow, Bengio, Courville
Read papers for your topics of interest, e.g. "Mastering the Game of Go without Human Knowledge" by Silver et al.

The quicker start would be of course to start at step 2 or 3 and only go to previous steps to look something up if it is required.


How can an artificial general intelligence determine which information is true?

Not possible without some big restrictions. What it can do is look at known "good" sites and compare news with site that is potentially "bad". Obvious problem here is defining some sites as absolute truth. For example it can recognize, while reading text, that some politician said something. These sentences can be compared with other sites, and if there is significant difference, that news is candidate for false news.

In practical terms, program would extract sentences "i like cats", "says he likes cats", "cats that John likes" etc. We need part that recognizes something as a quote, part that extracts it and finally parser so we end up with structure stored in some form that contains meaning of sentence (john-like-cats). Also it can keep information of time and context in which it was said, like timestamp of an article, some proper nouns that can indicate place (XY conference, London...). Now, suspicious article can be compared and checked if it matches time, place, some context and contains quote that is similar. Finally it needs to compare how different it is from other quotes. "...hates cats" should be labeled as potential fake news, but "likes dogs", "thinks cats are OK", "sings well" etc. should not. This can be expanded into comparison of whole articles.

There are many features that can be used to define particular article as fake. Interesting feature  for finding fake sites could be bias when it comes to particular (political, economical, ecological...) opinion. But in the end machine can't decide if the article is fake without comparing it to other articles. It is bound to closed system that reflects real world in subjective way.


What is the difference between AI architecture and AI model?

Well simply put AI model can be seen just as a flowchart showing how the control flow moves where it moves how it moves why it moves etc.
However AI architecture refers to the next step after building an AI model AI architecture involves representation of the functions that you use in your program. It also involved declaration of the variables you're going to use, the functions that you are going to use, etc.


How does the "Lorem Ipsum" generator work?

Lorem ipsum generators don't typically use anything considered as AI. Usually they just store large pieces of text and select sections from it randomly - they are very simple. The main goal is to produce "nonsense" text that fills space but does not distract from issues of layout and design. The variations of it are usually just for fun, and like the original, are mostly simple generators which select strings of text from a core data source randomly and without using any AI techniques.

It is possible to build more sophisticated random text generators that work using data structures from Natural Language Processing (NLP).

One popular and easy-to-code data structure is N-grams, which store the frequencies/probabilities of the Nth word given words 1 to N-1. E.g. a bigram structure can tell you all the possible words to come after "fish" e.g. "fish" => ["food" => 0.2, "swims" => 0.3, "and" => 0.4, "scale" => 0.1] To use that structure to generate text, use a random number generator to select a word based on looking up the Nth word's frequency, then shift the list of words being considered and repeat.

A more recent text generating NLP model is recurrent neural networks (RNNs), which have a variety of designs. Popular right now are LSTM networks, and these are capable of some quite sophisticated generation, provided they are trained with enough data for long enough. The blog The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy has quite a few really interesting examples of using RNNs for text generation. In practice this works similarly to n-grams: Use the RNN to suggest probabilities for next word given words so far, choose one randomly, then feed back the generated word into the RNN and repeat.


What's an appropriate algorithm for classification with categorical features?

What you have is called a classification problem with categorical features. That is, the features can be represented numerically, but the numbers have no relative meaning.

Algorithms that rely on smooth function approximation will probably not work well here. These would include classic approaches to regression, and also function approximation via a neural network. That's because the data are anything but smooth!

In contrast, classic classification algorithms like Quinlan's C4.5 decision tree learner, (implemented in the Weka Toolkit as J48, and possibly in SciKitLearn as DecisionTreeClassifier, though the documentation is less clear), are ideal for this: they actually work by splitting up numeric values into discrete categories anyway, so there's no issue at all for them. Most versions also support a way to pre-tag features as categorical, and the algorithms rely on the cross-entropy of each feature's categories, without making assumptions of smoothness.


Is there any disadvantage of the maximum number of fitness function call as a stop criterion?

I imagine that using the MaxNFFC as a stop criterion only happens in very particular implementations. And this is its main disadvantage.
Normally, you'd evaluate each individual, each generation. So, NFFC will always be the same as the size of the population times the number of epochs (+1 to consider the initial population).
$$N*(E+1)$$
As the population size is generally constant, MaxNFFC looks a lot like an MaxEpoch stop.
$$ \frac{\text{NFFC}}{N} = E + 1 $$
So, it seems that it might be used in scenarios where:

the population size is not constant (in this case you want to make sure that at least K individuals have been evaluated before stopping).
not all individuals are evaluated each epoch. There might be a high abortive rate (individuals with invalid genomes that are kept in the population but not considered). In this case, as above, you want to perform a minimum amount of evaluations.

I wouldn't see the usage of MaxNFFC in other cases.


Do AlphaZero/MuZero learn faster in terms of number of games played than humans?

According to Table S3 of the AlphaZero paper (p. 15)

AlphaZero was trained for 9 hours and, during these 9 hours, it played 44 million games of chess.
According to this Wikipedia article, the longest human lifespan is that of Jeanne Calment, who lived to age 122 years and 164 days.
Let's assume that humans cannot live more than 123 years (which is a reasonable assumption, although this record could eventually be broken). Let's also assume that a chess game lasts at least 10 minutes, which means that you can play at most 6 games in 1 hour, which means that you can play at most $6*24 = 144$ games in one day (assuming that you never sleep, which is, of course, impractical, but I'm just trying to show you an upper bound). Let's say that a year has 365 days. So, here is roughly the maximum number of games that a human could play
$$
6*24*365*123 = 6464880 \tag{1}\label{1},
$$
which is smaller than 44 million games by a factor of more than 6, i.e. any human could at most play 1/6 of the games that AlphaZero played, and \ref{1} is a very loose upper bound that doesn't take into account that humans need to sleep, eat, and do many other things.
So, humans learn to play chess a lot slower than AlphaZero. This is not surprising at all, given that computers can perform calculations a lot faster than us (that's why they are called computers), and this has been the case for many years. We (humans) just made computers make the right calculations for them to approximately play chess better than us. That's it.


How does neural network classifier classify from just drawing a decision plane?

If you are working with supervised learning, each training example has a label. That label is your classification of the provided input. Just like linear or logistic regression, if your problem only has 2 classes (e.g. determine whether a tumor is malignant or not), your network will have a single output. An output value of 1.0 could represent one class and an output of 0.0 represents the other. This is basically determining which side of multi-dimensional plane your input is on.

If you have more than 2 (m) classes, then you will have m outputs from your network. Each output will represent the likelihood of the input matching that class. For example if you have an image recognition network that has 4 classes: Dog, truck, boat, person, you would have 4 outputs with each one representing how likely that image is one of the classes. Think about this as 4 independent planes, where each one is making a determination: does this image match this class(1 of {dog, truck, boat, person}) or not. If you had outputs of dog: 0.3, truck: 0.25, boat: 0.7, person: 0.4, One simple algorithm would be simply pick the class with the maximum value -- in this case, it would classify this image as a boat.


How could we estimate the square footage of a room from an image?

Welcome to AI.SE Hadrien!

A possible approach is:

Gather many example images of rooms for which you know the square footage. Record the square footage of each room together with each image.
Pick a machine learning model that is well suited to learning relationships between images and numerical outputs, like a Convolutional Neural Network.
Train a machine learning model using an optimisation algorithm, like gradient decent. In the case of training a CNN, this algorithm starts by setting up the network with randomly chosen connection strengths between different 'simulated' neurons. It then exposes the network to an input image, and observes the number the network outputs in response. The algorithm then makes small adjustments to the strengths of the connections in the network, so that if the network were exposed to the image a second time, it would output a number that was less wrong (i.e. closer to the correct square footage). By repeating this process many thousands of times, and with many images, the network eventually becomes quite good at guessing the square footage of new images that it hasn't seen before.

In practice, the network is quite likely to pick up on any patterns in the images that correlate with square footage, not necessarily the ones you want. For example, it might pick up on, say, the fact that humans in the picture are usually a good object to guess at the scale of the rest of the room.


What approach to use for selecting one of the category according to short category text?

For this application, you can frame it as text classification. Look at SpaCy. You just need to create embeddings for your text and put a Softmax in the end. You can get those embeddings from BERT or anything else out there. You can in fact just use GLOVE vectors and others like it, concatenate them and then train a classifier.


What does "class-level discriminative feature representation" mean in the paper "Semi-Supervised Deep Learning with Memory"?

Discriminative models give the probability of an element in the feature space $x \in X$ belonging to a class $y \in Y$, i.e. $p(Y|X)$, where $Y$ is the set of classes in a classification problem.
The discriminative feature representation in this context means the feature map/s  which are the output/s of the convolutional layers of the backbone (the convolutional layers of the neural network) which (presumably) differ on the basis of their class, and can be used to discriminate which class the original image belongs to, in the case of image classification.


Is there any use of using 3D convolutions for traditional images (like cifar10, imagenet)?

As far as I've seen there is no use of 3D CNNs for traditional image classification tasks.

The reason I think is that, while these images do have multiple channels, there is no spatial information in those channels for the 3D convolution to extract. On the other hand it makes more sense to take the weighted sum of those pixels along that dimension (as the 2D convolution does).

3D CNNs have been used, as far as I know, only for applications where you have volumetric data, i.e. the images are sequential and, when combined, form a large 3D image.


Why is it illegal for Google's autonomous car to drive on the road by itself?

The question is based on a false fact: in Michigan, it is currently legal (under certain conditions described here ) for an autonomous car to operate without a driver.

The reason that the federal government has not enacted any direct legislation (although they have enacted guidelines) on autonomous cars is because it is still a developing technology (as great as those Cruise videos look, I wouldn't trust most companies working on the technology to have driverless cars on the road without operators) and that it arguably falls under the state governments jurisdictions. Some companies have appealed to the House to pass national legislation to allow autonomous testing (GM, Toyota, Lyft), but nothing has come of it yet.


I'm not an expert in this area, but it would appear to depend on the choice of activation function:

$e^x$ is not Lipschitz continuous. See Analytic functions which are not Lipschitz continuous.
$\tanh(x)$ is.

That said, this paper appears to give some conditions (specifically for dynamic ANNs) for which networks with activation function involving $e^x$ can be Lipschitz continuous, so possibly the above is not the whole story.


How to make machine learning model that reports ambiguity of the input?

Another specific way to do this if one uses a neural network for this. Use a dropout a layer in your network and instead of scaling the activations at test time, one can sample the activations (just like in training-time) and predict multiple times for a given input, then look at distribution of your outputs. Intuitively this would add "probabilistic, bayesian effect" to you neural network.
I think this method was first proposed in Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning which is called Monte Carlo Dropout.


An infinite VC dimensional space vs using hierarchical subspaces of finite but growing VC dimensions

In the first case, the VC dimension of $H$ being infinite implies that $H$ is not (agnostic) PAC learnable (see p. 48 of Understanding Machine Learning: From Theory to Algorithms). So, in general, your classifier is not guaranteed to succeed.

In the second case, your division of $H$ implies that $H$ is nonuniformly learnable (chapter 7 of the cited book). This implies that you can get a generalization bound by using structural risk minimization.


How to use a conv2d layer after a flatten?

To answer the question in the title, your enclosed method is a valid way to use 2d convs after a flattened feature vector. However, the bad results you experience could come from the structure of your model or from the way you train it. Regarding you last question, it is very hard to give you an advice without knowing your intentions in detail. Regardless, here are my two cents.
First of all, in the case you actually want to use this approach, to have a pretrained model and add your layers after one of its layers, you might want to keep the parameters of the original network intact at least until your newly initialized layers get trained properly. To achieve that, you need to use the gradients for updating only the parts you added as described in this comment. (You definitely should read the other comments there as well to get a better picture.)
Secondly, it might be worth reconsidering if you really want to add your layers after the very last layers of the pretrained network. Depending on your goals, using the output of some prior layers as the input to your layers might be more beneficial to you. (Just do not forget to keep the parameters of the pretrained model intact as I advised in the prior point.)
Lastly, the structure of your layers should also be reconsidered. 2d convs with a kernel size of 1x1 in this situation seems strange to my limited experience, but without knowing what you want to do, its hard to give any solid advice in this regard.
Therefore you might be better off splitting your question into smaller parts and work your way through them one by one.
(My reputation is not high enough otherwise I would have left a comment.)


How to use convolution neural network in Deep-Q?

Is a 3d convolution a good idea? Is 256 filters a good idea? Are the filters (4,4,2) and (2,2,1) suitable?

It's not so much that answers are subjective, but you are performing an experiment, and this should be driven by results. If you can find something published about a similar environment that might help you narrow down your choices.

That said, intuitively you don't gain much going from 2d to 3d convolutions when one of your dimensions is only sized 2. Purely from gut feeling I would suggest simply concatenating your two frames by having 8 channels instead of 4, and use 2d filters. This is simple to enough try and compare though, so perhaps you can do both.

You will likely want to explore deeper networks that have fewer initial filters and build up over a few more than two convolutional layers.

Definitely try your network without the dropout layer. I have not had much luck with using dropout in DQN, and I have seen others having similar problems. I am not sure what the exact issue is.

The accuracy metric won't do you much good on a regression task, so you can drop that and just use MSE loss. Bear in mind that training loss is a less useful metric overall in RL, since the prediction target (action value) is continuously changing as the policy changes. Low loss values mean that the value predictions are self-consistent, they don't necessarily mean that the learning has converged to an optimal policy.


Do we have to consider the feasability of an action when defining the reward function of a MDP?

Your issue is related to how you are representing your rewards, and not anything to worry about for the MDP.

You have chosen to use a matrix to represent your reward function, which maps $(s,s')$ to $r$. If some transition $s \rightarrow s'$ doesn't happen, then it doesn't matter at all what you put there for the reward value for $(s,s')$. It is only because you have decided to use a matrix for this that you even need to think about it.

A complete reward function would map $s, a, s'$ to a scalar value, and can also include a random factor. You don't need to use all the factors - a MDP can use any or all of them, and all the MDP theory is still correct. In your case, it looks like arriving in a "goal state" $s' = C$ is what triggers a +1 reward. So instead of a matrix you could define a function:

$$r(s,a,s') = \begin{cases}
    -1,& \text{if } s' = A\\
    1,& \text{if } s' = C\\
    0,              & \text{otherwise}
\end{cases} $$

This is equally valid in the MDP as your matrix, and avoids the issue of storing data about impossible transitions.

  I thought the transition function would handle this issue since I will define there what is possible and what is not.

Yes that should be the case. If your transition function does not allow for certain state changes, then there is no need to handle them in any particular way in the reward function.

  But anyway it is said that for a horizon of one the imm[e]diate reward is given and the transition function isn't even considered

I am not sure where you have read this, but it does not relate to your representation issue or resolving "impossible" state transitions.

If you are using a model-based method, such as Policy Iteration or Value Iteration, then you do use the transition function and reward function - more or less directly in the form of the Bellman equations. In this case, the transition function will assign a weight of 0 to the impossible transitions, so the reward value doe not matter (and efficient code would likely skip even looking up or calculating the reward in that case).

If you are using a model-free method, such as Q learning, then the agent does not use the transition function or the reward function directly. They are part of the environment that it is learning. However, any code for the simulated environment has to implement the physics of your world, and that includes using the transition function to resolve what happens when the agent takes an action. In that case, the simulated model of the environment would prevent the agent ever experiencing impossible transitions, so there is never any need to calculate the reward for them or allow for them as edge cases in a reward function (you might still choose to check validity and/or return something as defensive programming of the reward function, but it is not required for the MDP and reinforcement learning to work).


I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.

From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.

So using its pre-trained understanding, it analyzes the structure of sample texts and replicates this structure.
For scientific articles this structure could be:

Abstract
Context of research topic
Introduction of researchers
Explanation of methods/experiments/discoveries
Results and interpretation
Future research and application

For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.

TL;DR: The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).

P.S.: Take this with a grain of salt. It's 90% conjecture from incomplete information.


Will training an AI still work if the input data is somewhat sparse?

First, the title mentions "sparse data". Recently the expression has taken a clear meaning: The agent input is data with mostly zeros. In the question a different meaning: A "sparse data stream", where data flows and vanishes sometimes. I understand the question as: "Will training an AI still work if the training data stream breaks?"

Note the explicit "training data stream": The question suggests the agent has at least 2 inputs: Training data you want to feed "anytime", and "inference data" sent to the agent for actual recognition.

This question enters (to my eye) the realm of distributed AI and multi-agent systems, and ultimately a common issue in distributed systems.

If we cast your problem to two humans S and L communicating, when S talks to L over a reliable channel, L gets all the information. When the channel breaks, L gets nothing. Does it prevent L from living normally? It merely cuts out whatever is expected out of the conversation from S to L.

Back to your scenario, whenever the data stream is broken (S), the learning agent (L) will just be unable to learn from that data source. The impact on the pattern recognition agent is bounded to what it could have learned from the new data. The agent recognition performance remains constant while the data stream is interrupted.

Now if the learning agent is just learning, and cannot perform recognition without learning, there is an architectural or implementation issue. Continuous learning entails the agent is active (performs actual recognitions) and learns out of what it does.

Update, for clarification:

The performance remains constant is "true", but subtle. At time t some metric like precision can be 99% with respect to what the agent has seen so far. Assuming continuous learning is interrupted and new recognition requests come in, the performance has "two faces":

As long as new recognition requests are "close" to what the agent has seen so far, performance is "constant"---the agent still scores 99%.
If the request is quite different, the performance will drop. The size of the drop depends on how different the input is.

A concrete example: The agent is trained to find mushrooms with a dataset where all images are taken in the forest. Assuming learning stops, when an image of mushroom on a concrete crack comes in, the agent will probably do worse. And it would then keep doing worse on such kind of image, as long as it cannot "refresh" by learning from this experience.


Does quiescence search even improve the minimax algorithm?

Your logic is flawed because you negated "stand-pat" (i.e. do nothing) and alpha-beta. Let's take a look at the pseudocode (https://www.chessprogramming.org/Quiescence_Search#Pseudo_Code):

int Quiesce( int alpha, int beta ) {
    int stand_pat = Evaluate();
    if( stand_pat >= beta )
        return beta;
    if( alpha < stand_pat )
        alpha = stand_pat;

    until( every_capture_has_been_examined )  {
        MakeCapture();
        score = -Quiesce( -beta, -alpha );
        TakeBackMove();

        if( score >= beta )
            return beta;
        if( score > alpha )
           alpha = score;
    }
    return alpha;
}

Your Qxd6 capture will make return a score far below the alpha. The line:

  if( score > alpha )

will prevent your blunder being reported. Instead the engine would report either stand_pat (do nothing), or something like Nf3, Nc3 etc.


Can we use genetic algorithms to evolve datasets?

This question raises a lot more questions. It seems like a solution looking for a problem, instead of the other way round.

How do you measure the fitness of a feature?
What would one of the "evolved datasets" mean? What does it represent?
What would your overal purpose be? If you just wish to generate simulated datasets, there are easier ways to do this, with more control over the various aspects of the resulting datasets.

If you want to compute a new set of features to "better" describe a given dataset, there are many approaches to this, such as PCA, ISOMAP, self-organizing maps, ... If this is the kind of thing you're thinking about, I would recommend starting there.


Optimizing image recognition results for unknown labels

Welcome to AI.SE @Stromgren!

A likely explanation is that the animals in the "nothing" group do not have much in common with each other.

This means it will be difficult for the network to learn which features from the images are associated with that label (in fact, there aren't any!). As a result, the network is probably assigning very low confidence to any estimate for the nothing label. You should be able to check if this is the case (i.e. examine whether it is ever confident about the label "nothing").

I am not completely sure how the Inception network encodes its labels. A common scheme though is to use one output neuron for each class. The correct label for an image is thus always a "1-hot" vector, where one of the elements is set to 1, and the others all to zero.

If that representation is being used, you can incorporate the "nothing" data by labelling it with a vector that is all zero: none of the output neurons should activate for it. That would produce precisely the training signal you want.


Are there papers about automatic punctuation of written text?

It seems like either of those 2 options you listed would work in this situation. It second, in which you have a binary classifier first, would be a bit more efficient at least, as your deeper net(or whatever classifier you use) would not need to deal with as much.

It looks like you have a decent idea where to go from here, I'm interested to see what happens with this! Update us when you make some progress.


What are examples of optimization problems that can be solved using genetic algorithms?

There are numerous problems that can be solved with genetic algorithms or, more generally, with evolutionary algorithms (which includes also genetic programming and evolutionary strategies), even though they may not necessarily be the most efficient approach.
Here are a few examples.

Evolution of the topology of neural networks. This is called neuroevolution.
Automatic test case generation (in particular, for self-driving cars). AsFault is one specific example.
Design of novel quantum computing algorithms. Specifically, genetic programming has been used to solve this problem (see this reference for more details).
As an alternative to reinforcement learning algorithms to solve RL problems. Specifically, evolution strategies have been successfully used in this case (see this).

There is a Wikipedia article that lists many other applications of genetic algorithms: List of genetic algorithm applications.


Are there any artificial neuromorphic systems which can mimic the brain?

Vernor Vinge said that if we can scan a human brain and then simulate it: We can run it at 1000 times the speed. The brain will be able to do 1000 years of thinking in 1 year ect.

At this stage in history we have the computer power.

The trouble lies in cutting a brain up and scanning the 100 billion neurons and 12 million kilometres of axons and 100000 billion synapses.
And piecing together the connectome from all the data.

Sebastian Seung at MIT is working on automating this scanning process with machine learning. By gathering training data from thousands of people playing his Eyewire game

Henry Markram in Europe tried to do something similar with his Blue Brain Project.
He attempted to simulate the neocortical column of a rat. The EU gave him a billion euros to do this. Unfortunately he has been heavily criticised by the Neuroscience community. They claim that we don't know the physiology  well enough to make a valid simulation.

Check out his Ted Talk.

In the 1970s Sydney Brenner achieved a full brain scan of a C Elegans worm. This worm has one of the simplest biological neural networks having only 302 neurons.
Here is a picture of its connectome:

An accurate computer simulation of this worm would be a major stepping stone to uploading a human brain.


There is a big difference between the 2 datasets. When you train on the GTSRB dataset, your model performs image recognition, while for the GTSDB dataset you need to perform object detection. These are 2 related but different tasks (with detection being considerably harder).

You can consider using a sliding window approach, or use some prior attention model that will learn to extract the area of the signs for later classification.

I don't think that it is recommended to down sample images of 1360x800 pixels to 224x224.
(224x224)/(1360x800)=0.046. You are loosing 96% of the information (including your street signs which might reduced to nothing), granted most of it is not interesting, but simple down-sampling doesn't get to choose what to forfeit.


Why do we need to provide false labels to the discriminator on purpose to train GANs?

in this tutorial, it taught us to intentionally provide false labels to "fool" the discriminator, does it make discriminator actually inaccurate?

When training GANs, the training steps for the generator and discriminator are separate:

There is a training stage for the discriminator, where it is presented with a mix of generated and real data, all correctly labelled. It is important at this stage to not update the generator (otherwise it will get worse by helping to make the fake data look more fake).

There is a training stage for the generator, where the discriminator is presented with only generated data, all labelled incorrectly as if it were real. It is important at this stage to not update the discriminator.

Typically you will alternate between the two stages frequently, making small updates to discriminator and generator separately. Some GANs use metrics to decide how much of each to do, because you don't want either the generator or discriminator to win outright and stop progress - at least at the start.
So yes there is a stage where you deliberately "fool" the discriminator, because being able to do so is the goal of the generator. However, one key detail of this stage is that the discriminator weights are not updated from that faked data. Instead the gradients from that stage are used only to update and improve the generator.
It may help if you don't think of the false labels as being "fool the discriminator", but instead they are "measure how well the generator is fooling the discriminator".


How much can an inclusion of the number of iterations have on the training of an MLP?

If anything, you want the learning rate to decrease as the number of iterations increases.
When you're looking for a good spot and you're clueless, take large steps. When you've found a pretty good spot, take small steps, so you don't end up far away.
In other fields of machine learning, there are studies of how the learning rate should scale. For example, in traditional reinforcement learning methods, if $\alpha_i$ is the learning rate at step $i$, then we want to have the following two criteria, to make sure we get convergence to the optimal policy:

$\sum_{i=0}^{\infty} \alpha_i = \infty$. This makes sure that, no matter how bad our initial experience was, we can eventually forget it and replace it with better information.
$\sum_{i=0}^{\infty} \alpha_i^2 < \infty$. This guarantees eventual convergence.

A typical choice here is $\alpha_i = \frac{1}{1+i}$, which fits both criteria.
I am unaware of similar criteria for MLPs, but if you're going to modify the step sizes, I would follow a similar approach. Make the step sizes decrease, but not too fast.


Calculating gradient for log policy when variance is not constant

It's true that computing the log prob of a sample from a gaussian requires inverting a matrix and dealing with the determinant in the general case of a full covariance matrix. If you wanted to backprop through the log prob, you'd need to backprop through these operations, which by the way are available as differentiable operations in pytorch. See the gaussian distribution class which deals with these operations: https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L177
You can see there's a half_log_det and although you won't find an explicit matrix inversion, you'll see that it uses a differentiable triangular linear solve (torch.trtrs) at some point, which is essentially doing a matrix inversion in a more efficient way.


Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

Good question.

First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, the engines are 500 ELO points stronger than the top human players. This is a massive difference. The amount of work that has gone into contemporary chess engines is staggering. We are talking about millions of hours in programming, hundreds of thousands of iterations. It is a massive body of knowledge and work. To overcome and surpass all of that in 4 hours is staggering.

Secondly it is not so much the result itself which is surprising to chess masters but instead its how AlphaZero plays chess. It's quite ironic that a system which had no human knowledge or expertise plays the most like we do. Engines are notorious for playing ugly looking moves, those lacking harmony etc. Its hard to explain to a non-chess player but there is such a thing as an "Artificial move" like the contemporary engines come up with often. AlphaZero does not play like this at all. It has a very human-like style where it dominates the opponent's pieces with deep strategic play and stunning position sacrifices. AlphaZero plays the way we aspire to, combining deep positional understanding with the precision of an engines calculation.

Edit
Oh and I forgot to mention something about the result itself. If you are not familiar with computer chess it may not seem staggering but it is.

These days the margins of victory which separate the top contemporary engines are razor thin. In a 100 game match you could expect to see a result like 85 games drawn, 9 victories, and 6 losses to determine the better engine.

AlphaZero 28 wins and 72 draws with zero losses was otherworldly crushing and was completely unthinkable right up to the moment it happened.


Can policy iteration use only the immediate reward for updates?

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Technically yes.

The value update step in Policy Iteration is:

$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$

The discount factor $\gamma$ can be set to $0$, making the update:

$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$

However, there are two key details that are important, and make this a technically yes, not some alternative way of solving problems:

Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.
Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.
Due to the lack of bootstrapping between states, all the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.

However, with these caveats in mind, yes this is still policy iteration. It is the same update process, just with a particular choice of one of the parameters.


How can it be shown clearly and transparently that the outcomes of data-driven health and care technology are validated?

The current version of the document Code of conduct for data-driven health and care technology provides more details about all principles, including principle number 7, which I will quote here.

Consider how the introduction of AI will change relationships in
health and care provision, and the implications of these changes for
responsibility and liability. Use current best practice on how to
explain algorithms to those taking actions based on their outputs.
When building an algorithm, be it a stand-alone product or integrated
within a system, show it clearly and be transparent of the learning
methodology (if any) that the algorithm is using. Undertake ethical
examination of data use specific to this use-case. Achieving
transparency of algorithms that have a higher potential for harm or
unintended decision-making, can ensure the rights of the data subject
as set out in the Data Protection Act 2018 are met, to build trust in
users and enable better adoption and uptake.
Work collaboratively with partners, specify the context for the
algorithm, specify potential alternative contexts and be transparent
on whether the model is based on active, supervised or unsupervised
learning. Show in a clear and transparent specification:

the functionality of the algorithm
the strengths and limitations of the algorithm (as far as they are known)
its learning methodology
whether it is ready for deployment or still in training
how the decision has been made on the acceptable use of the algorithm in the context it is being used (for example, is there a
committee, evidence or equivalent that has contributed to this
decision?)
the potential resource implications

This specification and transparency in development will build trust in
incorporating machine-led decision-making into clinical care.


What is the mapping between actions and numbers in OpenAI's gym?

There is no way to tell via the Gym API, and to any RL-based learning agent this is entirely unimportant, discrete actions are just arbitrary labels, and their effects are learned by trial and error in terms of reward and changes to state.

If you need to know, you can read the code for each environment. Often it is described in a comment, or as constants.

E.g. in FrozenLake-v0, you can find this code:

LEFT = 0
DOWN = 1
RIGHT = 2
UP = 3

  If I env.step(0), which direction is my agent moving?

Left, according to those constant names at least.


How can a system recognize if two strings have the same or similar meaning?

Getting the intent of the sentence is not an easy task. To get you started on what to do, have a look on word vectors. You can also download pre-trained word2vec models. They help in getting similarity of words and reasoning with words. To get the intent of a sentence, you can use LSTM.

Fun fact most NLP algorithms strip away punctuation with is sufficient for most cases, but to give a counter example.

The defendant, who looked apologetic, was found guilty.
The defendant who looked apologetic was found guilty.

They mean different things and are difficult to catch the intent even with the best algorithms.

PS: For those wondering the difference, in the second sentence it seems like there were two defendants, and it was the one who looked apologetic who was found guilty while the other walked away free.


Does the encoding of a restricted Boltzmann machine improve with more layers?

First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs.

There are some elements to answer this question in the article Representational Power of Restricted Boltzmann Machines and Deep Belief Networks by Nicolas Le Roux, which can be found summarized in these course slides. The main results are:

  Restricted Boltzmann Machines:

  Increasing the number of hidden units improves representational ability.
  With an unbounded number of units, any distribution over $\{0,1\}^n$ can be approximated arbitrarily well.

  Deep Belief Networks:

  Adding additional layers using greedy contrastive divergence training does not provide additional benefit.
  There remain open questions about the benefits additional layers add.

I emphasized the $4^{th}$ point that covers the most your question. It does not mean that additional layers are useless, but since RBM are universal approximators ($2^{nd}$ point), the benefits of adding layers seem less straightforward. They seem dependent on the first layer, the training procedure... You can see the article for more details and the open questions raised about DBNs. Note that this answer is an entry point on the topic, there might be more recent results following this article which I'm not familiar with...


How does the neural-network know how to tweak weights for a specific neuron?

tl;dr
The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.
A deeper look...
Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a loss $J$. The loss (or cost) function tells us how far off we are from the target. This is what we want to reduce, so that the predictions fall closer to the target.
By computing the partial derivative of the loss function $J$ w.r.t a parameter $w_i$ (so this is just one partial derivative and not the full gradient vector)
$$
\frac{\partial J}{\partial w_i}
$$
the NN uncovers two pieces of information:

The slope of $J$ w.r.t $w_i$, which tells the NN how much $w_i$ affects $J$.
Its sign, which tells the NN which way to tweak $w_i$ to decrease (or increase) the value of $J$.

By making the parameter updates depend on the derivative
$$
w_i^{new} \leftarrow w_i^{old} - \lambda \frac{\partial J}{\partial w_i}
$$
the NN causes parameters that affect the loss the most, to be updated the most.
You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.


How can thought vectors be used outside of an Artificial Neural Network (ANN) context?

I'll take a shot at answering this, though I'm no expert in Neural Nets or Deep Learning.

Given that practical thought vectors (TVs) don't yet exist, and may be impractical or impossible, I think answering your question will require a lot of conjecture and speculation.  So here goes...

For thought vectors to be useful in or outside NNs, the vector values will have to be normalized, probably using the local context of the application problem's 'frame'.  Without a NN to create new baseline vector values (weights) and to normalize them to match each new context, any non-NN mechanistic alternative means of employing TVs will somehow have to fill that void.

Could vector values created by NNs be used by an alternative technique?  Could that technique also normalize them?  Sure.  We're just talking about turing-computable functions performed by NNs.  If NNs aren't magic, then there should exist other means to compute the same results -- creating or editing, or employing TVs.

What might such an alternative to NNs be?  Well, if its to shape vector weights, I suspect it too will have to learn those values through statistical iteration and feedback (as opposed to logical induction, say).  I doubt such a mechanism exists yet, since it'd probably resemble NNs in sufficiently many ways that, thus far, it would have seemed too derivative of NNs to gain acceptance as sufficiently novel.  Of course to be as powerful as deep nets, it too would have to propagate learning weights both forward and backward without incurring much error.  Not an easy thing to accomplish.

Less ambitiously, could TVs be simply interpreted by another technique usefully?  I think so.  I can see several existing techniques, like decision trees or even expert systems, importing thought vectors and being shaped by them, and then function in accordance.  But could these same techniques create or revise TVs usefully?  Beyond a trivial extent, I'm doubtful.  I think TVs are too complex a knowledge representation format for most general learning methods to both use and create/modify them, unless they employ an iterative statistical and feedback-based learning process, like those of NNs, which would allow novel and complex features to be learned and integrated into the vectors.


How to deal with evolutionary/genetic fitness function that can have both negative and positive values?

If I understand correctly your problem, you always want to maximize some function $h(x)$, which is defined as follows $h: \Gamma \rightarrow \mathbb{R}$, where $\Gamma$ is the space of genotypes. However, at every generation $g$, you don't know exactly $h(i)$ of each individual $i$, i.e., maybe in one generation $g$, all $h(i)$, for $i=1, \dots, N$ (where $N$ is the size of the population), are negative, but in the next generation $g+1$, due to the mutations and crossovers, that may not be the case anymore, so you don't know how to shift $h(i), \forall i$, so that they are all positive and you can compute the probability of being selected, assuming you are using the fitness proportionate selection.
If my intepretation is correct, then, at every generation $g$, you just need to find $w = \operatorname{min}_i h(i)$, then shift all $h(i)$ by adding $|w| + \epsilon$ (where $\epsilon$ can be zero or a very small number) to all $h(i)$, so you would compute the fitness as follows $f(i) = h(i) + |w| + \epsilon$, for all $i$. Here are all the steps to compute the probability of individual $i$ being selected $p(i)$

$w = \operatorname{min}_i h(i)$
$f(i) \leftarrow h(i) + |w| + \epsilon, \forall i$
$p(i) \leftarrow \frac{f(i)}{\sum_{j=1}^{N} f(j)}, \forall i$

This technique is known as windowing, which I have used in the past to solve the exact same problem that you seem to be trying to solve (check it here).


How can auto-encoders compute the reconstruction error for the new data?

It is computed just like in training. You take an MSE or something along these lines between the input and the output. You set a threshold for it. If new data's reconstruction error is higher than your threshold, then it is anomalous otherwise it isn't.


Technology for predicting body measurements of a person, with a full body photo of them

If the user will be imputting the body height/largest width measures, and you want to have a couple of groups where you final results will be stored at, you should look at the classification algorithms(Logistic Regression, KNN, Random Forests).
If on the other hand, you want your program to detect the user from an image and take the measurements itself, you would need to look into Convolutional Neural Networks, along with person detection/segmentation


Why does Simulated Annealing not take worse solution if the energy difference becomes higher?

Note that you can't really predict whether your escape from a local minimum will work or not - you might just wind up in another, worse local minimum. The probability function you describe increases the likelihood of this happening. By upweighting the likelihood of allowing small energy differences, you allow for the possibility of escaping local minima, while ensuring that whatever new minimum you find can't be that much worse that where you started. If you make the acceptance of large energy differences more likely, you can escape local minima more often, but you increase the likelihood that you'll just wind up in a region with an even higher local minimum.


Why does PyTorch use a different formula for the cross-entropy?

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.

Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $ for the correct prediction.

Therefore, your loss reduces to:
$$
H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})
$$


Can Q-learning be used to find the shortest distance from each source to destination?

Welcome to AI.SE Adarsh!

This is fairly simple to do with Q-Learning.

If you assign a reward equal to -1 for each location other than the goals, and 0 for the goals, then Q-learning with no discount factor will learn the length of the shortest path from each location to the goal. Alternatively, you can use a discounting factor, and then invert the equation on the learned values of the Q function to obtain the number of steps.

Hope this helps!


Is there a connection between the bias term in a linear regression model and the bias that can lead to under-fitting?

In machine learning, the term bias can refer to at least 2 related concepts

A (learnable) parameter of a model, such as a linear regression model, which allows you to learn a shifted function. For example, in the case of a linear regression model $y = f(x) = mx + b$, the bias $b$ allows you to shift the straight-line up an down: without the bias, you would only be able to control the slope $m$ of the straight-line. Similarly, in a neural network, you can have a neuron that performs a linear combination of the inputs, then it uses a bias term to shift the straight-line, and you could also use the bias after having applied the activation function, but this will have a different effect.

Anything that guides the learning algorithm (e.g. gradient descent with back-propagation) towards a specific set of solutions. For example, if you use regularization, you are biasing the learning algorithm to choose, typically, smoother or simpler functions. The bias term in the linear regression model is also a way of biasing the learning algorithm: you assume that the straight-line function does not necessarily go through zero, and this assumption affects the type of functions that you can learn (and this is why these two concepts of bias are related!). So, there are many ways of biasing a learning algorithm.

The bias does not always lead you to the best solutions and can actually lead to the wrong solutions, but it is often useful in many ways, for example, it can speed up learning, as you restrict the number of functions that can be learned and searched. The bias (as described in point 2) is often discussed in the context of the bias-variance trade-off, and both the bias and the variance are related to the concepts of generalization, under-fitting, and over-fitting. The linked Wikipedia article explains these concepts quite well and provides an example of how the bias and variance are decomposed, so you should probably read that article for more details.


Can analog quantum computer implement real-valued neural networks and hence do hypercomputation?

Digital and Analog

The question about analog computing is important.

Digital circuitry gained popularity as a replacement for analog circuitry during the four decades between 1975 to 2015 due to three compelling qualities.

Greater noise immunity
Greater drift immunity (accuracy)
No leakage of stored values

This quickly led to digital signaling standards, architecture of a general purpose computing, and central processing units on a chip. The later, combined with an array of registers to perform elementary operations is the meaning of the word microprocessor.

Quanta and Computing

Regarding quantum computing, there have been some interesting proposals to pack digital gates into much smaller volumes, but the notion that a computer can be made of transistors the size of electrons might be a bit fantastic. That's what the term quantum computing implies. That degree of miniaturization would have to defy principles of particle physics that are very strongly supported by amassed empirical evidence. Among them is Heisenberg's uncertainty principle.

All computing involves quanta, but statistically. For a transistor in a digital circuit to be statistically stable, there must be a sufficient number of Si atoms with at least 0.1% molar concentration of the atoms used to dope the Si to create a PN junction. Otherwise the transistor will not switch reliably.

The lithographic limit of most mass produced VLSI chips is around 7 nm as of this writing. Crystalline Si, nucleus to nucleus, is about .2 nm, so the miniaturization of a stable transistor is near its quantum limit already. Exceeding that limit by a considerable amount destabilizes the digital circuitry. That's a quantum physics limitation, not a lithographic limitation.

Projections, Models, and Techniques to Push Limits

Moore's law was simply an approximate model for the chip industry during the period between the invention of the integrated circuit to the physical limitation of the atomic composition of transistors, which we are now approaching.

Field effect transistors (FETs) can take the miniaturization only slightly further than the mechanics of PN junctions. 3-D circuitry has theoretical promise, but no repeatable nanotechnology mechanisms have yet been developed to form complex circuitry in the third dimension.

Returning to the Primary Question

Placing aside the magical idea that quantum computing will somehow completely revolutionize circuitry, we have a question that is both feasible and predictable.

  Can an analog computer implement real-valued neural networks and hence do artificial network computation better?

If we define better in this context as cheaper and faster, while maintaining reliability and accuracy, the answer is straightforward.

It definitely takes fewer transistors to create the feed forward part of an artificial network using an analog approximation of the closed forms resulting from the calculus of artificial networks than a digital one. Both are approximations. Analog circuitry has noise, and drift and digital circuitry has rounding error. Beyond rounding, digital multiplication is much more complex in terms of circuitry than analog, and multiplication is used quite a bit in artificial network implementations.

Limitation Interplay of Gödel and Turing

The idea from the tail end of the title of the book this question referenced, "Beyond the Turing Limit," is also a little fantastic. The thought experiment of Alan Turing leading to the Turing machine and the associated computability theory (including Turing completeness) was not developed to be a limit. Quite the opposite. It was an answer to Gödel's incompleteness theory. People in Turing's time saw the work Gödel's genius as the annoying but indismissable limit threatening the centuries-old vision of using machines to automatically expand human knowledge. To summarize this work, we can state with assurance this.

  The theory limiting what computers can do is not related to how the numbers are represented in electronic circuit implementations. It is a limitation of information mechanics.

These principles are important but have little to do with the limitation.

Analog computing
Miniaturization
Parallel computing
Ways that stochastic injection can help
How algorithms can be defined in programming languages

The above has to do with the feasibility of a project for which some person or corporation must pay and the intellectual capacities required to completing it, not the hard limit on what is possible.

Defining what a super-Turing capability might be would be a dismissal or a dismantling of what mathematicians consider to be well constructed theory. Dismantling or shifting the contextual frame of some computability theory is plausible. Dismissing the work that has been done would be naive and counterproductive.

Real Numbers are Now Less Real Than Integers

The compelling idea contained in the question is the reference to continuity, physicality, and the real valued nature of parameters that acquire a learned state during the training of artificial networks.

To multiply a vector of digital signals by a digital parameter matrix requires many transistors and can require a significant number of clock cycles, even when dedicated hardware is used. Only a few transistors per signal path are required to perform the analog equivalent, and the throughput potential is very high.

To say that real values cannot be represented in digital circuits is inaccurate. The IEEE standards for floating point numbers, when processed in a time series, represent real valued signals well. Analog circuits suffer from noise and drift as stated above. Both analog and digital signals only appear to be comprised of real number values. Real numbers are not real except in the world of mathematical models. What we call quantities in the laboratory are essentially measurements of means of distributions. Solidifying and integrating the probabilistic nature of reality into science and technology may be the primary triumph of the twentieth century,

For instance, when dealing in milli-Amps (mA), electric current seems to be a continuous phenomenon, but when dealing with nano-Amps (nA), the quantum nature of electric current begins to appear. This is much like what happens with the miniaturization of the transistor. Real numbers can only be represented in analog circuits through the flow of discrete electrons. The key to the advantage of an analog forward feed in artificial networks is solely that the density of network cells can be considerably higher, reducing the cost of the network in its VLSI space.

In summary, real numbers received the name for their type prior to the emergence of quantum physics. The idea that quantities formerly considered real and continuous were actually statistical averages of discrete activities at a quantum level revolutionized the field of thermodynamics and microelectronics. This is something that disturbed Einstein in his later years. In essence, mathematics using real numbers is effective in engineering because it simplifies what physicist now believe are distributions of a large numbers of quantum phenomena occurring in concert.

Summarizing the Probable Future of Analog Computing

This phrase from the question is not precisely scientific, even though it points to a strong likelihood.

  It is quite possible that quantum computers will be analogue.

This modified version is more consistent with scientific fact in the way it is phrased, and is also factual.

  It is possible that computers dealing with signals at a near quantum level of miniaturization will have a higher proportion of analog circuitry.

This question and its answers have many links to papers and research work regarding analog computing: If digital values are mere estimates, why not return to analog for AI?.


State representation of position in 2D plane for Reinforcement Learning (Q Learning)

I think your net should have the various actions as outputs, but I am not an expert in Deep Nets. I just think that that light form of multi-task learning might be better. The idea of multi-task learning is that a predictor predicting multiple variables (in this case the various Q(s,a1), Q(s,a2), ...) using mostly the same structure (varying only the output weights) will learn more sensible things. Though I admit applying this here might be a bit of a stretch.

As for the real question, a popular technique in Reinforcement Learning is Tile Coding.

The basic idea is to discretize the (2-dimensional, in your case) state-space - imagine a grid laid over the 2D space - and assigning an input feature to each cell; all of these variables are set to zero except for the one your continuous variables fall into. For example, if your grid is 20x20, you will have 400 variables, 399 of which are set to zero, and 1 set to one.

Tile Coding takes this one step further and repeats this using slight offsets for the grid. Imagine you create an identical grid but you move it slightly to the right by 1/10 of the width of a cell: you will have another set of 400 variables like before, but it is possible that the cell set to one is not the same. Then you repeat this moving the grid by 2/10 and you have another set of 400 variables, again, only 1 of which is set to one. In total you have 10 sets of 400 variables (if you repeat more than that, you get the same grids as before); of your 4000 variables, only 10 are set to one.
Now you repeat this by adding a 1/10 of a cell offset in the Y axis and obtain another 4000 variables. Repeat with 2/10 and you get another 4000. By the end of it, you have 40000 variables, 100 of which are set to one.

Now your net can more easily learn different weights for different positions. I recommend you to follow the link above for a better explanation than mine (and figures!)

My suggestion is to feed all of these variables to your net and have it predict the Q-value for all of the actions. But, again, I am no expert in deep nets so I may be wrong.

Also, according to Andrej Karpathy, "most people prefer to use Policy Gradients, including the authors of the original DQN paper who have shown Policy Gradients to work better than Q Learning when tuned well.". This means that you may be better off not using Q-learning (as they did in the original DQN formulation) to train your net. Have a look at Andrej's blog and the paper he points to.


This sounds to me like a use case for a chatbot. You would have different intents reflecting the types of user queries that your system can respond to. The intent matching can be done by pattern matching, machine learning (classification), or a combination of the two (hybrid). You can then use the chatbot to ask clarification questions or elicit more information to identify which live agent would be the best person to take over the call. Essentially each live agent would have a list of intents plus added information (such as geographical area etc) which you then compare against the data from the caller to find the best match.

If this is a voice call you'd need to put an ASR system at the front of the pipeline. Chatbots can usually do live-agent handover to then pass control to a human agent at any time in the conversation.

[Disclaimer: I work for a company that operates in exactly that area and whose system works as described above]


Transformers: how does the decoder final layer output the desired token?

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models

Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of
the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of this model is the union of both the source
and target vocabularies. In this model, both in the
encoder and decoder, all subwords are embedded
in the same duo-lingual space.

It seems like they learned a single embedding matrix ($n^e + n^f$) x $d^{model}$ in dimension.


How Does AlphaGo Zero Implement Reinforcement Learning?

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience.

RL could in fact be considered supervised learning (SL) or, more specifically, self-supervised learning, where the experience corresponds to the labels in SL, especially nowadays with techniques like experience replay.


If you only need 5 frames to predict the next frame, then I'd recommend a U-Net architecture, wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.

If using a longer history of frames can help, then I recommend taking a look at "Recurrent Environment Simulators" and combining it with the ideas above.

Hope it helps!


Why is the actor-critic algorithm limited to using on-policy data?

It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to importance sampling relative to the other policy.


What is relation between gradient descent and regularization in deep learning?

Usually, when talking about regularization for neural networks there are 3 main types:
L1, L2 and dropout. All affect the gradient descent procedure.

L1 and L2 regularization is implemented in the loss function, and therefore are part of gradient descent directly by altering the derivatives of the loss function thereby altering the weight update rules of the network during gradient descent.

For L1 you add a penalty based on the $\mathcal L^1$ norm of the weight vector, while for L2 you add a penalty based on the $\mathcal L^2$  norm.

For dropout, there is no direct impact on the loss function, but you are still interfering in the gradient descent procedure indirectly by masking nodes to alter the forward and backward propagation.


This question seems to be specifically about your idea so I'll answer as such - any general 'can digital activity be used to predict suicide attempts' is probably too broad for this site.
As I see it you want to use what a person types, across all types of digital media, to get some measure of their risk of suicide attempt.
Two key questions in any machine learning problem

Can we create variables as predictors?
Is the data there? i.e. do we have labelled data already

As far as creating variables goes a lot of people don't realise that machine learning (or 'AI') isn't about dumping a load of unedited data into an algorithm and letting it figure it out - 90% of the work is in making those variables into something sensible. In this case it may involve a fair bit of psychology. Perhaps you look at changes in message length - has someone gone from writing essays to short answers? (I'm not a psychologist, that may well be way off the mark). The use of particular words or phrases may come into it - you would need to do a fair bit of research into the differences between those who attempted suicide and those who didn't (perhaps ranking words by frequency and see if there are distinct differences).
This brings us to our second question (or perhaps it should be the first) 'is the data there'? You want to predict suicide risk from digital activity so you'll need the digital activity of all different types of people with a label of whether or not they attempted suicide (and when - no point collecting the data two years after their attempt). Do you think this dataset exists? I'm sceptical - the level of detail you would need (both into someone's digital activity and their personal mental state) is unlikely to be recorded.
It isn't to say I think its an idea to be thrown out, only that you need to be realistic about the amount of effort required to do something like this. You would need to carry out research to collect this data, need to plan out your ideas before hand to know what you would like to collect (the worst thing is to find, after a year of data collection, that you didn't monitor a variable you would have now found vital).


Adding voices to voice synthesis corpuses

The original wavenet and the implementation you linked to is globally conditioned on speaker embeddings, which means that the network was given a unique identifier for the person speaking each time it trained on an audio clip. This allows the network to learn to mimic the voice of each person in the training data, but it only learns their voices, not arbitrary people's voices.
You might be able to do what you're describing by globally conditioning on different features of the speakers, such as age etc, though this would require a lot a speakers to span those dimensions.
I think it would definitely be possible to use style transfer for this, similar to the work done on tweaking people's age in images [1]. Some similar work has been done for audio in order to change the accent of a voices between accents [2], so I wouldn't be surprised to see more examples of this in the near future.
[1] https://www.youtube.com/watch?v=2qMw8sOsNg0
[2] https://ai.googleblog.com/2018/03/expressive-speech-synthesis-with.html


What are filters in image processing?
In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the image. For example, you can use a Gaussian filter to smooth the image (including its edges).
What are filters in CNNs?
In the context of convolutional neural networks (CNNs), the filters (or kernels) are the learnable parameters of the model.
Before training, the kernels are usually randomly initialised (so they are not usually hardcoded). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are typically fixed. Incredibly, the filters learned by CNNs can be similar to the Gabor filter (which is thought to be related to our visual system [1]). See figure 9.19 of chapter 9 (p. 365) of the Deep Learning book by Goodfellow et al.
The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a hyper-parameter.
What are the differences and similarities?
In both contexts, the words "kernel" and "filter" are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the convolution (or cross-correlation) operation. More specifically, the application of a filter, which is a function $h$, to an input, which is another function $f$, is equivalent to the convolution of $f$ and $h$. In mathematics, this is often denoted by $f \circledast h = g$, where $\circledast$ is the convolution operator and $g$ is the result of the convolution operation and is often called the convolution (of $f$ and $h$). In the case of image processing, $g$ is the filtered image. In the case of CNNs, $g$ is often called an activation map.
Further reading
Take a look at this and this answers for more details about CNNs and the convolution operation, respectively.
You may also want to have a look at Visualizing what ConvNets learn
 for some info about the visualization of the kernels learned during the training of a CNN.


Which NLP applications are based on recurrent neural networks?

Speech recognition and Character recognition are not part of NLP. Everything else on your list can in principle be done with RNNs. But the field is quickly moving towards using transformers.


When using PCA for dimensionality reduction of the feature vectors to speed up learning, how do I know that I'm not letting the model overfit?

I'm not sure if I understood your question correctly, but here's my take anyway.
So, PCA is a technique that you can apply to data to reduce the number of features.
In return, (i) this can speed-up training, as there are less features to do computation with, (ii) and can prevent overfitting, as you lose some information on your data.
To detect overfitting, you usually monitor the validation and training losses during the training. If your training loss decreases, but your validation loss stays constant or increases, it's likely that your model is overfitting on the training data. In practice, this means your model generalizes worse and you can observe this by measuring the test accuracy.
All in all, you can apply PCA, train a new model, and measure your model's test accuracy to see if PCA has successfully prevented overfitting. In case it didn't, you can re-train with other regularization techniques such as weight decay and so on.
After your edit that put the slides
Basically, what slides claim is, PCA could be a bad way to prevent overfitting when compared to using standard regularization methods. To actually see whether this is the case, the standard way would be measuring your model's performance on a validation dataset. So, if PCA throws away lots of information, and hence causes your model to underfit, your validation accuracy should rather be poor wrt to using standard regularization techniques.


Is it possible to flip the features and labels after training a model?

The body of your post seems to be asking a completely separate question than the title of your post, so I will answer both:
"Body: How do I complete the goal of this program?"
Your dataset does not have the dependent variable, which is the outcome of the game (win/loss/draw). What I am assuming is that you have a way of looking up the outcome of the game from either the "id" field or the "gameReferenceId" field.
So you would have to augment the dataset with a new column, "gameOutcome", which has values (win/loss/draw), by looking up the outcome of each game (each row) and adding that to your dataset.
Once you have this, you have the 12 independent variables (the 12 columns already there), and the 1 dependent variable (the "gameOutcome"), and the prediction task should be straightforward from there.
"Title: Given a label, how do I predict features?"
(note: this section will not help your program)
What you are looking for are generative models. Generative models can generate instances given the label and/or some random seed. This is a completely different model than discriminative models, which given the instance predicts the label (this is the one you have).
The simplest generative model is a Naive Bayes model. While Naive Bayes is normally used as a discriminative model (to classify a label), it has enough information to generate instances given a label as well. Here are some tips on how to turn Naive Bayes into a generative model.
If you are looking for generative deep neural network models, this blog by OpenAI has a nice overview explaining the three modern approaches (generative adversarial networks, variational autoencoders, and autoregressive models). The blog explains at a high level what the strengths and weaknesses are of each approach, and links to some open source projects for you to try it out as well.


In Fast R-CNN, how are input RoIs mapped to the respective RoIs in the feature map before RoI pooling?

The ROIs in the input space are mapped to the feature map space, by dividing it by the net stride at that layer. Say, in a network, after a sequence of four 2x2 pooling layers, your image is reduced to 1/16 of the original size. (A 32*32 image is reduced to 2x2) So, the bounding boxes in the original space are mapped to the feature space by dividing by the net stride, which is 16 here. But here's the catch, the ROI co-ordinates could be a floating point number when divided by 16, so, it is adjusted by either flooring it or ceiling it. This is why ROIPooling is quantized. There is a loss of information when you are rounding off the co-ordinates of the ROI. Nevertheless, now, each region is pooled to a single-size feature map, and each ROI is fed one-by-one to the following layers.
The Mask-RCNN paper brings a change to the mapping to the feature space, by not rounding off the co-ordinates, and by bilinear interpolation, and the loss of information is reduced, thus, the ROIAlign algorithm(which has been described here) performs better at object detection, than the quantized ROIPool algorithm.

As said, each fixed-size feature map vector is processed one-by-one.

Reference


Training neural network for good taste in art

This question reminds me of a project I saw that used Deep Learning to rate selfies on twitter. But a quick google search shows that there are plenty of projects that are much closer to what you are interested in:

Rating Image Aesthetics using Deep Learning

Predicting Image Aesthetics with Deep Learning

Deep Understanding of Image Aesthetics (with data and model linked)

Understanding Aesthetics with Deep Learning

and probably dozens more.

Of course if you are interested in predicting subjective pleasantness the above is only a beginning. In that case you may also take a look at recommender systems.


How do we know the classification boundaries of the data?

This is the classic question of what structure is or can be. It relates directly to the concepts of generalization, pattern recognition, over-fitting in surface fitting strategies, and learning tabula rasa, Latin for blank slate. The underlying questions are these:

How can it be determined whether the organization of data in a set, which appears to correlate well with model $\mathbb{A}$, isn't just a random data set that merely appears to have $\mathbb{A}$ organization.

How can it be determined whether the organization of data in a set, which appears to correlate well with model $\mathbb{A}$, isn't generated by a phenomenon that generally exhibits organization that correlates well with $\mathbb{B}$?

How can we determine the interrelationship between $\mathbb{A}$ and $\mathbb{B}$, especially since inclusion and overlap relates to the above question?

In the second image in the question, if $\mathbb{A}$ is defined as clustering of a type the smallest boundaries of which are not breached by instances of another type with at least five members per cluster, then several false positives would occur if the locations of the points were generated with quantum or thermal noise and the model was intended to illuminate clustering. In such a case there may be the appearance of clusters, but proximity would not be indicative of organization.
In the first image in the question, if $\mathbb{A}$ is the compound angular projection of the orbital paths of the planets in this solar system, then the image would appear as noise.
If a third image existed of the moon's sky, the same model would likely produce a false negative if the tolerance of inaccuracy in input was high.
It is difficult to reliability detect organization in the measured properties of a complex system. The goal of feature extraction, using RBMs or some other approach is to detect patterns. Whether those patterns are organized or merely apparent is a hypothesis that may be strengthened or weakened as the amount of data analyzed increases.
If an AI system is trained to detect rats but knows nothing of cats and dogs, one cannot assume it will differentiate cats from dogs on the basis of the closer relationship between rodent and cat features than between rodent and dog features unless the AI system was meta-trained or programmed statically with a specific conception of what a feature is.


How powerful is OpenAI's Gym and Universe in board games area?

OpenAI's Gym is a standardised API, useful for reinforcement learning, applied to a range of interesting environments many of which you can then access for free with little effort. It is very simple to use, and IMO worth learning if you want to practice RL using Python to any depth at all. You could use it to ensure you have good understanding of basic algorithms such as Q learning, independently of and before you look at using RL in a board game context.

There are limitations for Gym and Universe when dealing with multiple agents. The API is not really designed with that in mind. For instance, there is no simple way to add two agents to an environment, you would have to write a new environment and attach an opposing agent inside of it. This is still possible, and not necessarily a terrible idea (it depends on the training setup you want to investigate).

If you want to look into classic two-player games, and write bots like AlphaGo and Stockfish, then I would point out that:

Game-playing bots often make extensive use of planning that can interrogate potential future game states. OpenAI's Gym doesn't prevent you doing that, but it doesn't help in any way.
Algorithms for AlphaGo are public, with many nice tutorials. It would be quicker to follow one of these and develop your own bot training code in most cases, than to try and adapt an OpenAI solution for single agent play.
Probably the biggest time-saver you could find for any game is a rules engine that implements the board, pieces and game rules for you. If Gym already has a game environment for the game you want your bot to play, it might be worth checking the Gym code to see what it is integrating, then try to use the same library yourself, but not the Gym environment directly.
Many decent game-playing algorithms don't use RL at all. You can frame most of them as search (finding best moves) plus heuristics (rating moves or positions), and can usually make independent choices for algorithms that perform each sub-task. You can apply RL so that a bot learns game heuristics, then use a more traditional search e.g. negamax in order to make decisions during play. Or you can use any analysis of the game you like in order to generate heuristics. Very simple games usch as tic-tac-toe (noughts and crosses in UK) can just have heuristic of +1 if X has won, -1 if O has won and 0 otherwise, and still be quickly solved with a minimax search for perfect play.
DeepMind's AlphaGo uses a variant of MCTS for the search algorithm, which can be considered a RL technique, but the lines are a bit blurry around definitions there - it is safer to say that AlphaGo incorporates MCTS as the chosen search technique for both self-play training and active play against any other opponent.


What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
If you're interested in actor-critic methods then a popular off-policy method is the Deep Deterministic Policy Gradient.