Can games be solved without an evaluation function?

Human chess and go experts clearly use evaluation functions. They do come up with moves that look sensible without evaluating the board position, but to validate these candidate moves they evaluate board positions that occur at the end of the variations they calculate. Pretty similar to AlphaGo.

Inputting two board states and outputting a preference is a (much) more complex task than mapping one board state into the real numbers. And it gives you less information. So its a lose-lose choice. (I did try something very similar and it didn't work at all. The reason is that you didn't just double the input size, rather you made the input space quadratically bigger.)

If you compare two board states that differ just by one move, then your input space doesn't quite explode as much, but you have to do a ton of comparisons to make a decision. The logical choice would be to output a preference distribution over all possible moves - but that's exactly what AlphaGo does in its policy network. There is also an earlier paper which trained a network to predict expert moves, which comes down to the same thing. And yes, both these networks play quite strongly without any search or board evaluation. But nowhere near AlphaGo-level.


Will a neural network always predict the correct label if it sees the exact same input during training and testing?

No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...

For an easy counterexample, consider what happens if you have two instances with precisely identical inputs, but different output labels. If your classifier is deterministic (in the sense that there is no stochasticity in the procedure going from input to output after training), which a Neural Network is (unless, for example, you mess up a Dropout implementation and accidentally also apply dropout after training), it cannot possibly generate the correct output for both of those instances, even if they were presented as examples thousands of times during training.

Of course the above is an extreme example, but similar intuition applies to more realistic cases. There can be cases where getting the correct prediction on one instance would reduce the quality of predictions on many other instances if they have somewhat similar inputs. Normally, the training procedure would then prefer getting better predictions on the larger number of instances, and settle for failure on another instance.


Are there any good tutorials about training RL agent from raw pixels using PyTorch?

I am using Jetson reinforcement for my quadcopter reinforcement in simulation. Maybe it will help you. because you can create AI agents that can learn from the interactive environment, gather experience, and system of reward with deep RL. You can also use end-to-end neural networks that translate raw pixels into action as per your need and use that RL-trained agent to complete complex tasks.

The best thing is you can easily transfer RL-agent which simulated in the simulator to real-world robots. and We are using multiple Nano's easily to perform the complex tasks of navigation and co-ordination of Our Quad-copter.

Webinar of Deep RL on Jetson

Google group to get help for Deep RL on jetson nano

Hope it helps you.


What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

There are different possible ways to handle huge datasets:

If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in the article Why and How to Use Pandas with Large Data, section 1. Read CSV file data in chunk size. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.

If the data is really big, probably it's better to store and process it by multiple computers using special software, A specific tool depends on what you want to do with the data:

2.1. You can even keep using Pandas: there is an extension named Dask which wraps interfaces of Python iterables, NumPy, and Pandas to run on multiple machines. Also, there are other similar tools.
2.2. A more independent yes simple approach is to use some analytical, distributed DBMS like Google BigQuery or Cassandra.
2.3. And if you need something more powerful and complex, MapReduce systems like Apache Hadoop, Spark can be utilised. Moreover, there are many articles and scientific papers that describes usage of MapReduce for the machine learning.

So, as always, it is a matter of your opportunities and your aims of dealing with the data.


Perturbed here means adding a small random value to the weight. That random value comes from a uniform distribution or from a gaussian (or any distribution really).
Imagine just nudging the weight by a little.

It’s done to overcome the problem of local minima where models can get stuck with a good set of weights but not the best set of weights. By perturbing the weights a little, the model has a chance of finding a better set of weights. Searching for methods that deal with local minima like stochastic gradient descent will give you a better intuition.


What else can boost iterative deepening with alpha-beta pruning?

First thing you're going to want to add is probably a Transposition Table, as also suggested by SmallChess.

Afterwards, I'd look into Aspiration Search and/or Principal Variation Search (also see this page).

Then I'd look into things like the Killer Move Heuristic, and maybe also see if you can simply implement existing parts of your engine more efficiently (e.g. use bitboards for your state representation).

Other than all of that, the chess programming wiki probably has lots of other interesting pages as well.


When should we use algorithms like Adam as opposed to SGD?

Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also observed that Adam and RMSProp are highly sensitive to certain values of the learning rate (and, sometimes, other hyper-parameters like the batch size) and they can catastrophically fail to converge if e.g. the learning rate is too high. On the other hand, in general, SGD have not led me to the highest performance, but they did not catastrophically fail (at least, as much as Adam and RMSProp) in my experiments (even when using quite different hyper-parameters). I noticed that the learning rate (and the batch size) are the hyper-parameters that mainly affect the performance of all these algorithms.

In my experiments, I use SGD without momentum and I used the (PyTorch) default values of Adam and RMSProp. I only compared SGD with Adam and RMSProp, on the relatively simple task of recognising MNIST digits. You can have a look at this repository https://github.com/nbro/comparative-study-between-optimizers, which contains the code I used to perform these experiments. You also have the instructions there to perform the experiments (if you want).


How can a machine learning problem be reduced as a communication problem?

Information-theoretic view of Bayesian learning

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?

Yes, this is indeed possible. More precisely, there is an information-theoretic view of Bayesian learning in neural networks, which can also be thought of as a communication problem, which explains both maximum a posteriori estimation (MAPE) and full Bayesian learning [1], i.e. finding the posteriors over the weights of the neural network: the neural networks that maintain a probability distribution over the weights are now known as Bayesian neural networks (and, in terms of theory, they are strongly related/similar to the famous variational auto-encoders).
The oldest relevant paper (I am aware of) that interprets Bayesian learning in neural networks as a communication problem is the 1993 paper by Hinton and Van Camp entitled Keeping the neural networks simple by minimizing the description length of the weights (COLT), which is the paper that introduces variational Bayesian neural networks (sometimes called ensemble learning in some papers from the 1990s), i.e. variational inference (VI) applied to neural networks (yes, the same VI used in VAEs). Hinton (yes, the famous Hinton that won the Turing award) and Van Camp (who is this? probably a Dutch guy from the name!) write in this paper

We can think in terms of a sender who can see both the input vector and the correct output and a receiver who can only see the input vector. The sender first fits a neural network, of pre-arranged architecture, to the complete set of training cases, then sends the weights to the receiver. For each training case, the sender also sends the discrepancy between the net's output and the correct output. By adding this discrepancy to the output of the net, the receiver can generate exactly the correct output.

You should read this seminal paper if you want to understand all the details.
Another relevant paper is Practical Variational Inference for Neural Networks (2013, NeurIPS) by Graves, who cites the 1993 paper immediately at the beginning of the paper. Essentially, as the title of the paper suggests, Graves tries to make VI in neural networks practical.
There are other relevant papers that still attempt to provide this information-theoretic view of Bayesian learning, such as Variational learning and bits-back coding: An information-theoretic view to bayesian learning (2004, IEEE Transactions on Neural networks), but most current papers on Bayesian neural networks, such as Weight Uncertainty in Neural Networks (2015, PMLR) don't do it (at most they may mention that this interpretation exists, but they don't go into the details).
Minimum description length
To give you a few more details, the information-theoretic view of Bayesian learning in these papers is that of the minimum description length (MDL), i.e. Bayesian learning (i.e. the application of Bayes rule to find the posteriors over the parameters of the model) is equivalent to finding a model that gives the "shortest description of the data" (hence the name MDL), where a description is some code/encoding of the data: in the case of the NNs, this encoding is contained in their weights.
Given that you want to find the simplest code, then this is a direct application of Occam's razor: if you have multiple hypotheses/functions that describe your data (or are consistent with your observations), then choose the simplest one. Occam's razor underlies many other mathematical/ML theories and frameworks, for example, AIXI, a framework for artificial general intelligence developed by Marcus Hutter. Jürgen Schmidhuber is also a good fan of Occam's razor and compression as a means to act intelligently (see e.g. the speed prior). If you are familiar with deep learning, a light bulb should turn on in your brain now. Yes, regularization techniques to avoid over-fitting and improve generalization can also be viewed as an application of Occam's razor principle.
Bits-back coding
How do we find the simplest weights? The bits-back coding, used by the 1993 paper and described in the 2004 and 2013 papers, essentially states that you can find the simplest encoding (i.e. posterior over the weights) by minimizing the Kullback-Leibler divergence (aka relative entropy: say what?!) between the posterior (which is unknown: so how can we compute the KL divergence?) and some prior (coding distribution), which is zero when the prior is equal to the posterior (but we don't know the posterior) [1]. Given that we don't know the posterior, we need to use a proxy objective function that doesn't involve the posterior, such as the Evidence Lower BOund (ELBO), also known as variational free-energy, which leads to a non-optimal coding (i.e. possibly, you will find some posteriors that are not optimal given the data).
Conclusions
Using MAPE or performing (approximate) Bayesian learning in a neural network (which finds one function or a probability distribution over functions, respectively) can be interpreted as finding the MDL, i.e. an optimal or near-optimal encoding of the data that needs to be communicated from a sender to a receiver.
Side notes
Information theory was pioneered by Claude Shannon in his 1948 seminal paper A Mathematical Theory of Communication.
Claude Shannon was also one of the participants at the Dartmouth workshop, which officially started the field of artificial intelligence, so he is one of the fathers of the AI field, and his impact on the field is definitely huge (although most people are not aware of it, but, hopefully, this answer will change that).
Further reading
Apart from the papers that I cited above, you may also be interested in Information Theory and its Relation to Machine Learning (2015) by Hu.


Minimum video quality for facial recognition

On page 2 of Axis' web page Identification and Recognition there is an estimate of the minimum number of pixels needed for identification, recognition and detection.


How to recognize non-circular radial symmetry in images?

The Hough Transform extended to orthogonal ellipses uses this model, accumulating on $\theta$ for all $\{x, y\}$ with parameter matrix

\begin{Bmatrix}
c_x & c_y \\
r_x & r_y
\end{Bmatrix}

where

$$1 = \dfrac {(x - c_x) \, \cos \theta} {r_x} + \dfrac {(y - c_y) \, \sin \theta} {r_y}$$

The question is looking to detect the normal lines, so any of the several algorithms for the above model can be modified to accumulate on $r$ for all $\{x, y\}$ with parameter matrix

\begin{Bmatrix}
c_x & c_y  \\
r_x & r_y
\end{Bmatrix}

where

$$0 = \dfrac {x - c_x} {r_x} + \dfrac {y - c_y} {r_y}$$

Lines that intersect $(c_x, c_y)$ don't rely on $r_x$ or $r_y$.  However, it may be useful to recognize that, if radially equally spaced, viewing the lines from a position other than one that projects into the plane of the lines at $(c_x, c_y)$ will present a line density that is a function of $\arctan (r_x, r_y)$.


Monte Carlo Tree Search: What kind of moves can easily be found and what kinds make trouble?

Whether the move is found and how quick it is found depends on a few things. If I understand correctly, there is a sequence of many "bad" moves which lead to the "big win" move, and you are afraid that the MCTS algorithm will not get to the "big win" move because it will be selecting more promising moves further up the tree. Some things to think about (read also the Wikipedia MCTS article):

when doing playouts, you can do play your game only for a few further moves or down to the game end. Playing only a few moves further is obviously quicker, but in the extreme case you described it would not be the best choice. If you know about the existence of such scenarios, ensure to play the game to the end in the playouts.
when doing playouts, you can choose your moves/actions either randomly or based on some simple, greedy (quick) heuristics tailored to your problem. Are there maybe greedy heuristics designed to find or take into account such scenarios for your game/problem? If yes, implement them. It is then called a "heavy playout". Compare the results to playouts using random moves.
If you choose the actions using UCT (Upper Confidence Bound applied to Trees), then the first part of the expression is responsible for exploitation. Moves with high average win ratio are preferred. The second part though corresponds to exploration. If the exploration parameter is set high enough (test empirically for your problem), then moves with few simulations will be preferred. High exploration would be another way to find your golden move, in detriment of exploitation (read about the exploration/exploitation dilemma).

If you describe a realistic game or problem scenario, we may be able to help you come up with a suitable strategy.


Is the target assumed to be a noisy version of the output of the model in machine learning?

Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.


Can artificial intelligence be thought of as optimization?

A good answer to this question depends on what you want to use the labels for.

When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible answers that could be returned and we can know what the cost is of any particular answer.

In this view, the answer is "yes"--pattern recognition is a case where each pattern is a possible answer, and the optimization method is trying to find the one where the cost is lowest (that is, where the answer matches what you want it to match).

But most interesting optimization problems are characterized by exponential solution spaces and clean cost functions, and so can be thought of more as 'search' problems, whereas most pattern recognition problems are characterized by simple solution spaces and complicated cost functions, and it might feel unnatural to put the two of them together.

(In general, I do think that optimization and intelligence are deeply linked enough that optimization power is a good measure of intelligence, and certainly a better measure of the practical use of intelligence than pattern recognition.)


ML model that is most suited to analyse Google Analytics data

I understand that in your example you are interested in modelling the outcome of the 'result' column.

One easy model I would suggest is to model it using the Bernoulli distribution (https://en.wikipedia.org/wiki/Bernoulli_distribution) with the probability of success p.

Then you can model p with something like this

x = a + b * log(page_views) + c * log(time_spent) + e

p = exp(x) / (1+exp(x))

where e is normally distributed,
e ~ N(0, sigma^2)
(or simply centered around zero).

a, b and c are parameters that you can estimate.

I.e. the probability of success (conversion) is modeled as a sigma function of a certain variable that depends on page_views and time_spend. You can also add squares (and higher powers) of page_views and time_spend in the equation of x (i.e. upto a certain threshold page_views can have a positive effect on conversion and then a negative one, then again a positive effect).

Also reading about logistic regression should put you on the right track: https://en.wikipedia.org/wiki/Logistic_regression


What should the input and output of the Q-network be in the case of an ordinal action space?

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor:

If all actions are in the output, your neural network function is $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|} = [\hat{q}(s,a_1), \hat{q}(s,a_2), \hat{q}(s,a_3) ...]$, and you take the maximum value from the output vector as the greedy action.

If the action is provided as an input argument, your neural network function is $f(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \hat{q}(s,a)$, and to find the maximum value you construct and run a mini-batch over all possible values of $a$ for the given state.

Also see the answer to this question: Why does Deep Q Network outputs multiple Q values?
In your case, you would like to take advantage of similar values of $a$ because you expect that to work well with the approximation. As you correctly suggested, this will only work with the second approach using action as an input. So, use the steering angle, normalised into a suitable range for input to a neural network, as an input. Every time that you need to find $\text{max}_a Q(s,a)$ for the Q-learning algorithm, you must construct a mini-batch of the current state concatenated with each of the discrete steering angles that you want to consider as actions in the DQN, and run the current (or target) neural network forward.
If you want to go further and use a continuous action space, you will need to change which reinforcement learning method you are using. The various policy gradient and actor-critic approaches, such as REINFORCE, A3C, DDPG etc can cope with continuous actions, because they drop the need to find $\text{max}_a Q(s,a)$, which becomes impractical for very large action spaces.


Do these FOL formula both represent "You can fool some of the people all of the time"?

(1) can be paraphrased as "There exists an x, and for any t if x is a person and t is a time, then x can be fooled at time t" (I would use fool-able instead of can-fool, as it is closer to the intended meaning).
(2) would be "There exists an x, and for any t, you can fool x is a person and t is a time."
They are not equivalent: person(x)  and time(t) are boolean predicates, which return a truth value: they are true if x is a person, and t is a time, respectively. So in (1) they act as a constraint on the values that x and t can take. If x was a saucepan, then person(x) would be false, and thus you wouldn't be able to claim that you can fool a saucepan.
So can-fool takes two arguments: one for which person is true, and one for which time is true. But in (2), the arguments are actually the boolean truth values: if x was "Falstaff" and t was "yesterday", then in (1) the premise would be true, as person("Falstaff") and time("yesterday") are true, and so you conclude can-fool("Falstaff", "yesterday").
In (2) that becomes can-fool(person("Falstaff"), time("yesterday")), which evaluates to can-fool(true, true), and that won't work.


Deep Q Learning for Simple Game Not Effective

Your implementation of single-step Q-learning with neural network and experience replay is basically correct.

There are a few blocking issues preventing you seeing it working correctly.

Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of next_state (when it gets copied to state you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of scale to not do this. A very simple re-write of your routine would be:

def scale(s):
    return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]

In addition, you need to change random action selection to:

np.random.randint(0, 3)

because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.

  What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)).

I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.

  Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern

Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.

Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the scale function:

Starting epsilon of 1.0
Repay memory size 10,000
Only start learning when replay memory has greater than 1,000 entries
Discount factor $\gamma$ 0.99
Neural network with 20 neurons per layer with tanh activation instead of relu

There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!


Which neural network should I use to approximate a specific function?

If the concept class specified is

$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$

and the optimum fit to example data is expected occur when $k \approx 1 \land f_x \approx 1 \land f_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.

If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be

$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$

The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.

Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?

Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.

Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).

If the objective of this question is to take images and sequential data and produce something useful without a priori defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.


How do deepfakes work and how they might be dangerous?

In general, deepfakes rely on advanced context-aware digital signal manipulations - usually image, video or audio - that allow for very natural looking modifications of content that previously have been costly or near impossible to produce in high quality.

The AI models, often based on generative adversarial networks (GANs), style transfer, pose estimation and similar technologies, are capable of tasks such as transferring facial features from subject A to replace those of subject B in a still image or video, whilst copying subject B's pose, expression, and matching the scene's lighting. Similar technologies exist for voices.

A good example of this might be these Star Wars edits, where actors faces have been changed. It is not perfect, you can in a few shots see a little instability if you study the frames - but the quality is still pretty good, and it was done with a relatively inexpensive setup. The work was achieved using freely-available software, such as DeepFaceLab on Github.

The technology is not limited to simple replacements - other forms of puppet-like control over output are possible, where an actor can directly control the face of a target in real time using no more than a PC and webcam.

Essentially, with the aid of deepfakes, it becomes possible to back up slanderous or libelous commentary with convincing media, at a low price point. Or the reverse, to re-word or re-enact an event that would otherwise be negative publicity for someone, in order to make it seem very different yet still naturally captured.

The danger of this technology is it puts tools for misinformation into a lot of people's hands. This leads to potential problems including:

Attacks on integrity of public figures, backed by realistic-looking "evidence". Even with the knowledge that this fakery is possible (and perhaps likely given a particular context), then damage can still be done especially towards feeding people with already-polarised opinions with manufactured events, relying on confirmation bias.
Erosion of belief in any presented media as proof of anything. With deepfakes out in the wild, someone confronted with media evidence that went against any narrative can claim "fake" that much more easily.

Neither of these issues are new in the domains of reporting, political bias, propaganda etc. However, it adds another powerful tool for people willing to spread misinformation to support any agenda, alongside things such as selective statistics, quoting out of context, lies in media that is text-only or crudely photoshopped etc.

A search for papers studying impact of deep fakes should find academic research such as Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News.

Opinion: Video content, presented feasibly as a live capture or report, is especially compelling, as unlike text and still image media, it directly interfaces to two key senses that humans use to understand and navigate the world in real time. In short, it is more believable by default at an unconscious and emotional level compared to a newspaper article or even a photo. And that applies despite any academic knowledge of how it is produced that you might possess as a viewer.


Which neural network has capabilities of sorting input?

Even a simple multilayer perceptron can sort input data to some extent, as you can see here and here.

However, neural networks for sequential data seem more appropriate, as they can handle sequences of variable lengths. It has been done with an LSTM (Long Short-Term Memory), LSTM+HAM (Hierarchical Attentive Memory) and an NTM (Neural Turing Machine).


Is stable learning preferable to jumps in accuracy/loss

There is an approach to machine learning, called Simulated Annealing, which varies the rate: starting from a large rate, it is slowly reduced over time. The general idea is that the initial larger rate will cover a broader range, while the increasingly lower rate then produces a less 'erratic' climb towards a maximum.
If you only use a low rate, you risk getting stuck in a local maximum, while too large a rate will not find the best solution but might end up close to one. Adjusting the rate gives you the best of both.


Find the expected reward in an expectimax-based dice rolling game?

If I understand,

With probability $p$ the robot select no re-roll (action $D$).
With probability $1-p$ the human uniformly select an action between $A,B,C,D$.

Thus the expected reward for if the robot helps is
$$R_{AH}= p\cdot D+(1-p)\frac{A+B+C+D}{4} = \frac{(A+B+C)(1-p)+D+3pD}{4}$$


Programming an inference AI that computes the best outcomes like a quantum computer

Welcome to the AI stack Jake.

This probably isn't going to be possible. Modern Psudo-random number generators, like Mersenne Twister, are designed not to have any patterns in them, so there's nothing to learn from.

You could however, try something like predicting the values of a broken random number generator, like RANDU. These aren't used anymore, precisely because it's possibly to predict future numbers on the basis of past numbers.

Hope this helps.


How to calculate the entropy in the ID3 decision tree algorithm?

Suppose you have data:

color  height  quality
=====  ======  =======
green  tall    good
green  short   bad
blue   tall    bad
blue   short   medium
red    tall    medium
red    short   medium

To calculate the entropy for quality in this example:

X  = {good, medium, bad}
x1 = {good}, x2 = {bad}, x3 = {medium}

Probability of each x in X:

p1 = 1/6 = 0.16667
p2 = 2/6 = 0.33333
p3 = 3/6 = 0.5

for which logarithms are:

log2(p1) = -2.58496
log2(p2) = -1.58496
log2(p3) = -1.0

and therefore entropy for the set is:

H(X) = - (0.16667 * -2.58496) - (0.33333 * -1.58496) - (0.5 * -1.0)
     = 1.45915

by the formula in the question.

Remaining tasks are to iterate this process for each attribute to form the nodes of the tree.


Based on the presentation by Praveen Venkateswaran on the same paper, KWIK framework is any learning algorithm that makes the following:

if it knows (by its learning) the answer already, it does tell that
if it is uncertain, it tells the "I don't know" mark
"I don't know" marks should have an upper limit, which is the accuracy/error term of the algorithm.

The idea is that algorithm has a trainer who knows the correct answers, like in any machine learning case, but the difference is that the algorithm agent should know when it has enough knowledge to give the result right away, and when is time to ask and learn. In learning phase the trainer does not calculate errors, only he gives right answers if asked.

There I see two implications: algorithm should know when it has not all the necessary data to answer (when to use "I don't know" answer) and a mechanism to be sure you're correct when the answer is e.g. 1 or 0 in binary case.

How to put that in mathematics, it varies by your chosen algorithm. Also there is a formal equation set for this, but I leave here only this literal definition to give the basic idea.


TLDR Use whatever environment you are most comfortable with.

Today the choice of operating system is of much less importance than few years ago. Generally speaking, you will have more application choices under Linux, however, some of these choices will be less than spectacular :-) Windows works better with some hardware. Hence, under Linux, it is a very good idea to check the forums before buying or installing.

I have played with OpenAI gym under Linux for some time and it worked as advertised.

Early versions of many important pieces of deep learning software get released for Linux only. For instance, PyTorch and Tensorflow were available for Linux at least one year before the release of Windows ports.


Over the years, many people attempted to define artificial intelligence. A lot of those definitions are summed up by Stuart Russell and Peter Norvig in their book Artificial Intelligence - A Modern Approach

The definitions of AI can be summarised as falling into the following categories:

  Those that address thought process and reasoning (how an AI thinks/reasons)
  Those that address behaviour (how an AI acts given what it knows)

Furthermore, the above 2 categories are further divided into definitions that:

  I. assess the success of an AI (to do the above) based on its ability
  to replicate human performance

  II. or an ability to replicate an ideal performance measure called
  'rationality' (does it do the 'right' thing based on what it knows?)

I will cite you definitions that fit into each of the above categories:

  1.I. "The [automation of] activities that we associate with human thinking, activities such as decision making, problem solving, learning.." - Bellman 1978
  1.II. "The study of the computations that make it possible to perceive, reason, and act." - Winston, 1992
  2.I. "The study of how to make computers do things at which, at the moment, people do better" - Rich and Knight, 1991
  2.II. "The study of the design of intelligent agents" - Poole et al., 1998

In summary, AI is devoted to the creation of intelligent and rational machines that can make rational decisions and take rational actions.

I would suggest you read up on the Turing test, which Alan Turing proposed to test if a computer was intelligent. However, the Turing test has a few issues, because it is anthropomorphic.

When Aeronautical engineers created the airplane, they didn't set their goal that planes should fly exactly like birds, but rather, they started learning how lift forces were generated, based on the study of aerodynamics. Using this knowledge, they created planes.

Similarly, people in the AI world shouldn't put, IMHO, human intelligence as the standard to strive for, but, rather, we could use, say, rationality as a standard (amongst others).


How can I classify houses given a dataset of houses with descriptions?

It all depends on what kind of annotations or other variables are present in your dataset. I see 2 possible scenarios here:

your dataset is made only of houses descriptions, without any indication of their luxury level.

you have annotations regarding the luxury level or another similar variables from which you can infer the luxury level (like the house price for example).

In the first case there's not much you can do except for trying to apply some unsupervised algorithms or transfer learning. Usually in NLP unsupervised techniques are used for tasks like Topic or language modeling, both of which are not really helpful for your specific application since they work at a really abstract level trying to learn relationships between words or documents in corpora containing huge variety of texts. The best you could try could be preprocessing the data to extract specific terms like entities (cities names, furnitures names, etc.) and adjectives from each description, and then try cluster them into n clusters were n is the number of classes you're interest in by applying for example Latent Dirichlet Allocation. Even though everything is possible with the right time and patience, I would never follow this road, especially because it relies on a perfect preprocessing, that involves already transfer learning, for the name entity recognition part for example. And even tough libraries like SpaCy offer really good models to perform these tasks, once you have these kind of annotations a rule based approach would probably be faster and easier to build than another unsupervised model, e.g. creating a simple dictionary contacting names of luxurious furnitures name and adjectives that indicate if the house is expensive would probably be sufficiently good.
In the second case, the story change completely because if you have annotations you could rely on supervised learning. If you already have explicit annotations like "luxuriously renovated" "nicely renovated" nothing stops you from trying to train whatever architecture you feel more comfortable with on this classification task. Even simple architecture like CNN that are easy and fast to train usually achieve good results in classification, and you definitely want to leverage some pre-trained embedding vectors like GloVe as an input feature (every deep learning framework like tensorflow or pytorch already implement the possibility to use them).
To conclude, if you don't have annotations you might try LDA just to check if you're lucky but if I were you I would start annotating data as quick as I can. A good practice in this case is to also ask someone else to perform the annotations, not only to be faster in the creation of the dataset but also to then calculate the Inter Annotator Agreement score, that gives an indication about the quality of the annotations (if the score is low, the dataset quality is poor and no model will be able to learn something from it).


How could an AI detect whether an enemy in a game can be blocked off/trapped?

If I've understood the logic correctly, the player tries to build a wall such that the enemy cannot reach the player in any way. This can normally be determined by some path-finding algorithms, Dijkstra Shortest Path algorithm being a reasonable choice for the setting and grid-size.
This algorithm explores the possible paths from a starting point to one or multiple end points, and usually returns the shortest path to the point(s). If there is no path to the mentioned point, it will not return anything, and you will know the two points are blocked from reaching each other.

Of course, if I've truly understood the rules correctly, the bigger question is how to avoid having the player just build a wall around themselves to block of the enemy instead, which is likely going to be trivially easy. And additionally rule of requiring the player to be in the "room" with the bigger area after a room has been blocked off (Something which coincidentally also could be done by extending the Dijkstra's algorithm a bit with custom logic)


Given two neural networks that compute two functions $f(x)$ and $g(x)$, how can I create a neural network that computes $f(x)g(x)$?

Use another network h which takes f(x) and g(x) as input i.e. h(f(x), g(x)).
Training psuedo code(pytorch):
for epoch in epochs:
    for batch, x in dataset:
        train(f)
        train(g)

for epoch in epochs:
    for batch, x in dataset:
        # freeze f and g (using torch.no_grad in pytorch)
        with torch.no_grad:
            fx = f(x)
            gx = g(x)
        pred = h(fx, gx)
        loss = loss_fn(pred, (fx . gx))
        backpropagate()
        optimise()


Reinforcement learning to play snake - network seems to not get trained at all

I cannot comment much on your setup for inputs and outputs. It seems adequate to get some control, but does not cover the fully Markov state for the game, so I would expect that will limit the agent from ever being truly optimal. I would expect it to learn to play the game though, if you were implementing Q learning with a neural network correctly.

In your code, you are implementing a basic Q learning loop. It seems correct. However, this combination of Q learning and neural networks is known not to work - or more accurately, it rarely works this simply. The problem is mainly to do with the network receiving its own initially biased outputs back as new targets, plus receiving updates in correlated form (data on each time step is strongly correlated with data from previous time step). These biases are too large for the Q learning process to overcome, and typically the result is an agent that fixates on a single default action, because it has learned an inflated action value for it.

The problem is well known in RL research and called "The Deadly Triad" by Sutton & Barto.

The usual solution to this with Q learing is called DQN or "Deep" Q Learning ("Deep" is in quotes because this should be applied even if you just have a single hidden layer).

In basic DQN, you need to add the following features:

An experience replay table. Instead of training directly on experience as it is received, instead store $s, a, r, s'$ in memory. When it is time later in the loop to train the NN for a step, take a random sample of some M items (e.g. 32 items) as a mini-batch, calculate a latest target for them, and train once on the mini batch. You will need logic to only start this training process once you have some minimal amount of experience from behaving randomly (e.g. 500 random steps).
A "target network". When generating target Q values, use a cloned copy of the learning network, and only update this clone every N steps (with N typically set at 1000 or 10000).

These two additions are not really optional, even for really basic environments. You will need to add them to your script.


Is text preprocessing really all that necessary for NLP?

It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link
, if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be spectacular.

In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". Also, you should remember that BERT or GPT-2 it's deep learning algorithms so they not need too much processing. Using preprocessing in machine learning is more that needed(prediction of sentiment for example).

Shortly, preprocessing is optional, but highly advisable.


How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

Don't know if you have this doubt anymore, but this would be helpful for those who are facing similar problems-
You will need to find the correct weights with which you add these two loses by hyperparameter search. That is, find the best $\lambda$ for the loss-
$$
L = Loss_1 + \lambda(Loss_2)
$$
Here $Loss_1$ and $Loss_2$ can be any losses. Here, we take them as SSIM and L1-regularization losses, respectively.
You can keep the gradient flow of the regularization loss below some percentage of that of $Loss_1$ by choosing the correct value of gradient clipping for a hyperparameter. Note that by setting this hyperparameter to low, you may even impede its performance(the exact opposite of the case mentioned here). For this specific case, L1 regularization has a constant gradient, equal to the hyperparameter itself. So, by keeping it around 10% of the max gradient(or 10% of the max loss generally works as well), we should not face these kinds of problem.


How can I apply reinforcement learning to solve this asteroid game?

A closely related question and a minimal implementation written in Python.

That program implements the reinforcement learning technique 'Q-Learning'.

The idea is for the program to take in an observation of the environment (which could be a screenshot if learning a computer game, or sensor data for a robot) and output a decision in the form of a vector of values. Each cell in that output vector corresponds to a possible action (left, right, shoot, etc) and the highest valued cell shows the action that the agent/player should take. The values in that output vector are called the Q-Values, and it is the mapping from the input data to the vector of Q-Values that we are trying to learn.
In your case, the function that takes in an observation of the environment and spits out a vector of decision choices is your SVM.

The question linked above contains a description of the training algorithm. It boils down to playing the game multiple times while storing the input vector, output vector and output decision for each step until you reach a termination condition (ie hitting an asteroid) and receiving a score (in this case a negative score since you want to avoid the asteroids). Then, going backwards through the the output vectors, assigning a slowly reducing amount of the output score to the particular cell in each output vector corresponding to the decision that was made at that step of the game.
The array of input vectors from the game, and the array of output decision vectors that you have just created become the training data for your machine learning system. Once you have completed a training run, play the game again with your trained system and keep repeating until it is a good as you need it to be, or it stops getting any better.

To address your question directly, I don't think that you are going to be able to train your system without giving it information about the position of the asteroids (assuming that they are positioned at random). However, you could try just having a very simple input vector such as 3 integer values where each value corresponds to the presence (1) or absence (0) of an asteroid in the 3 squares to the above left, above and above right of the player. That might be enough to encourage it to dodge away from falling rocks...

There are some Support Vector Machines implemented in JavaScript, although I have never used them, but since SVMs are basically two-class classifiers you would have to check that the library can easily support multi-class SVMs. In your case the SVM would be trying to classify each input into a particular player choice (left, right, shoot, etc).


What is convergence analysis, and why is it needed in reinforcement learning?

Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves that theoretically the algorithm achieves the desired function. Without convergence, we have no guarantees that the value function will be accurate or the policy will be any good, so in other words the proposed RL algorithm can completely fail at serving its purpose even in simple cases.


In Deep Learning there are several methods to improve "stuck" gradient - decrease learning rate, use cyclic learning rate - cycle it from bigger to smaller value. More radical method is completely reinitialize last or two last (before loss) layers of the network.

In non-Deep Learning ML out of those only decrease learning rate will work, but there is plethora Numerical Optimization methods to help, like second-order methods - variation of Gauss-Newton, or methods specific to the problem which may include projective methods, alternate directions, conjugate gradients etc. There are a lot of methods which are better then gradient descent for non-Deep Learning optimization.


Extending a neural network to classify new objects

Yes this is standard transfer learning. Using a trained model, we can freeze the first N hidden layers of a classifier, except for the last few. This will allow our previous relevant training to be retained whilst also being able learn new features and target new classes.
We will then initialize a new output layer that works our new context(i.e sigmoid, 1 node for binary classifier). Everything is now set to resume training on our new y_targets.
Take a look at this link for some more info on transfer learning.
If you want no perturbance to your past learning, I would recommend freezing all previous hidden layers and then tacking some additional ones on.


Cross entropy loss function causes division by zero error

Cross entropy loss is given by:

Now as we know sigmoid function outputs values between 0-1, but what you have missed is it cannot output values exactly 0 or exactly 1 as for that to happen sigmoid(z) will have to be + or -infinity.

Although your compiler gives a divide by 0 error, as very small floating point numbers are rounded off to 0, it is practically of no importance as it can happen in 2 cases only:

sigmoid(z) = 0,in which case even though the compiler cannot calculate log(0) (the first term in the equation) it is ultimately getting multiplied by y_i which will be 0 so final answer is 0.
sigmoid(z) = 1,in which case even though the compiler cannot calculate log(1-1) (the second term in the equation) it is ultimately getting multiplied by 1 - y_i which will be 0 so final answer is 0.

There are a few ways to get past this if you don't want the error at all:

Increase the precision of your compiler to float64 or infinity if available.
Write the program in such a way that anything multiplied by 0 is 0 without looking at the other terms.
Write the program in a way to handle such cases in a special way.

Implementation side note: You cannot bypass divide by 0 error with your manual exception handler in most processors (AFAIK) . So you have to make sure the error does not occur at all.

NOTE: It is assumed that the random weight initialisation takes care of the fact that at the beginning of training it does not so happen that $\tilde y$ or $1-\tilde y$ is 0 while the target is exactly the opposite, it is assumed that due to good training that the output is reaching near to the target and thus the 2 cases mentioned above will hold true.

Hope this helps!


Is there ever a need to combine deep learning frameworks? (Eg. TensorFlow & Torch)?

In the time since I asked this question, I have been able to combine Tensorflow and Chainer considerably well. That being said, one should try to avoid combining deep learning frameworks if one can for a few reasons:

It doubles the amount of documentation one needs to reference
It makes it difficult for new developers to become familiar with the
code base.
It creates multiple data transfer bottlenecks when one
has to transfer data between the deep learning different
configurations such as: CPU -> GPU (Tensorflow) -> CPU -> GPU
(Chainer) -> CPU ... etc.

That being said, I combined them because of time constraints on a research project which would not be finding its way into an industrial product. In this research setting, I still wouldn't recommend combining frameworks if one can avoid it but if it means doubling the time it takes to do the project for a proof of concept or experiment, then it seems reasonable and can drastically speed up development until the missing functionality is added to the desired framework.

There are 2 issues with integration:

Abstracting the networks so that the main
training/testing script does not depend on the framework.

This is important in the event that the missing functionality of your desired framework is added. It also helps new developers understand what the overall algorithm or experiment is without being bogged down by framework details.
Tensorflow and similar frameworks which require a Session object.

Chainer, PyTorch (which came from Chainer), and any other framework that doesn't require a something like Tensorflow's Session all play very nicely together and can be used (almost) interchangeably. For Session based frameworks, the main script/loop/experiment must all take place within a Session. With proper abstraction, my experiment only required a with tf.Session() as sess: to be used in the main experiment and was the only reference that Tensorflow was being used.

Although I still don't recommend combining frameworks for the reasons specified above, if one abstracts their framework calls enough, it is almost seamless to integrate deep learning frameworks.


Should I use Monte Carlo or a classifier for this Decision Making problem?

A classifier seems to me to not really fit the problem. I am not really sure. Actually, I don't see how a classifier like a decision tree, for example, would work here. Because I have to predict decision of the counselor to propose or not based on the decision of banks (and I don't know their criteria) to refuse or accept applicants who were proposed loan insurance and accepted it.

  The data I have is former applicants profile who were sent to banks and if they were accepted or not by the bank, if they wanted a loan insurance or not and the type of acquisition they wanted to make with their loan.

Why does this seem to you like something where a classifier wouldn't fit? Unless I'm missing something, it sounds like a prototypical example of a classification problem to me.

You have:

Input features (applicants' profile)
A clear (binary?) prediction target: propose or don't propose (equivalent to predicting whether or not the bank would accept, because you'll always want to propose if the bank would accept, and never propose if the bank wouldn't accept).
Old training data containing both the input features and the matching prediction targets.

Approaches like Monte-Carlo Tree Search can only be used if you have a forward model or simulator. In your setting, you could view the features (applicants' profile) as a "game state", and model the problem as a game with two actions (propose or not propose). However, you don't have a forward model (a function that, given a current state and action, generates a possible reward and subsequent state).

In applications where MCTS is often used (such as games), you do have such a forward model: for a game like Go or chess, you can easily program the game's rules, program how you transition from one state into another when you select an action, etc. This does not appear to be the case for you.


Is there a computer vision service for classifying images on a fixed array of images provided by me?

Actually I'm trying the same thing with the Azure Computer Vision API. Although the API is very good in identifying objects, it has problems identifying specific consumer products (in my experience though). For example it can't really distinguish between two pair of shoes, or two pair of watches.

People recommended me using the: Custom Vision Service combined with Custom Decision Service. I haven't really looked in to them both. But maybe it's useful for you.


There is a good chance that your DQN is already optimized but you would have to take a look at its performance to really check and see whether its actions seem up to par.
Reasons it may not be optimized: If you are tracking the reward after every episode, unstable rewards are very common just due to random chance, but if you are averaging the reward over the past 50 episodes or so, then it may also be your learning rate or epsilon.
If your learning rate is too high or low you may either never be able to reach a fully optimized DQN or be stuck in a local minimum. An easy way to solve a problem like this would be to just add a simple learning rate decay, so that the learning rate would start off high as to not get stuck in local minima yet decay to a small enough number where you know that the agent has found the global minimum.
The other problem could be that your epsilon may be too high or low. A high epsilon will never allow the agent to fully optimize while a low epsilon doesn't allow the agent to explore and discover better strategies, so it may be a good idea to mess around with this as well.
The only way to really gauge the agent's performance would be to look at it making some decisions through a video or by analyzing some of its predictions. And, if it seems to be doing well, then it may very well be optimized, but, if the agent is not performing as well as it should, then it may be a good idea to try out some of the strategies above.


Which algorithm can I use to solve a problem with multiple objectives and constraints?

From your description, the problem you want to solve is a linear optimization problem: suppose we use the indices $i$ and $j$ to denote the $i-$th class and the $j-$th grade. Also, let us call $y_j$ the current value of the $j-th$ grade, $g_j$ the goal value in grade $j$, $c_{ij}$ the impact factor of taking class $i$ in grade $j$, and $x_i$ the binary variable that indicates if you go to class $i$ or not. Now, suppose you decided to take certain classes (which is equivalent to choosing the values $x_i$ for each $i$) and measure the new grades $\tilde g_j$. If your impact factors are accurate, then the new grades should be $\tilde g_j = \sum_i(x_ic_{ij}+y_j)$. Clearly, what you would like is for this new values to be as close as possible to the goal values. So, a possible way to express your optimization problem is that you want to minimize the addition of all these differences:

$$\min_{x_0,\cdots,x_n} \sum_j g_j - \tilde g_j = \sum_j g_j - \sum_i(x_ic_{ij}+y_j), \\ \text{s.t. } x_i\in\{0,1\}, \forall\hspace 2pt i.$$

Most probably, you will have certain time restrictions. For example, maybe each class takes $t_i$ hours from your time and your total available time is just $T$ hours. Or maybe you have a limit of classes $N$ you can take, irrespective of their time duration. With these two constraints, your problem would look like this:

$$\min_{x_0,\cdots,x_n} \sum_j g_j - \sum_i(x_ic_{ij}+y_j), \\ \text{s.t. } x_i\in\{0,1\}, \forall\hspace 2pt i,\\ \sum_ix_it_i\leq T,\\ \sum_i x_i\leq N.$$

Apart from the constraint that $x_i$ should be 0 or 1, this is a linear optimization problem with linear constraints. Integer programming is an area of optimization that studies this type of problems so probably it's a good direction for your research.


How and where can I start learning AI if I have considerable knowledge of Java & C++?

If you are coming from Java, it would make a lot of sense to play around with deeplearning4J at first. From there I would start learning python as this is the primary language used today in ML.

Lectures are a great way to get your feet wet in understanding and applying ML concepts.

My favorites are:

Stanford ML(Andrew NG): https://www.youtube.com/watch?v=UzxYlbK2c7E (covers classic algorithms in depth)

MIT AI(Patrick Winston): https://www.youtube.com/watch?v=TjZBTDzGeGg&t=2s (great starter course that covers basic concepts beautifully)

Deepmind RL(David Silver): https://www.youtube.com/watch?v=2pWv7GOvuf0&t=7s (Best RL course available)

Beyond that, start implementing these concepts. There are hundreds of tutorials on computer vision, natural language processing, and reinforcement learning.


What's the difference between Starcraft and Dota from an AI perspective?

Those AI-learning programs may have very similar scheme. We are changing only inputs and possible actions (like "use skill" or "move here"). Starcraft AI must do a lot of actions and control many units. Dota is MOBA so bot should be good in positioning on map for example. Different opponents to destroy and target for win.

AI needs to play many games for learn to play game with some rules to find best moves in some situations/states.

Of course it's only my newbie-programmer opinion :)


What AI conferences in Europe should I consider submitting papers to explaining the ongoing work on RefPerSys?

Probably to as many as possible. Average accept rate of papers is around 20%. You can find the best conferences on AI & ML Event.


How create an AI that continuously adapts to different users?

Assuming you have enough training data and representational capacity, you can give each user a unique identifier and concatenate that with the other inputs to the neural net. As users give more feedback, the network will learn in the usual way since each situation (e.g. food-user pair) is represented by a unique input.

I'd consider this a general, brute-force approach, and it may not be suited to your application. Others' links to recommendation systems might be more useful. It depends on what your task is and what your constraints are.


What is the benefit of using identity mapping layers in deep neural networks like ResNet?

TL;DR: Deep networks have some issues that skip connections fix.

To address this statement:

  As I understand Resnet has some identity mapping layers that their task is to create the output as the same as the input of the layer

The residual blocks don't strictly learn the identity mapping. They are simply capable of learning such a mapping. That is, the residual block makes learning the identity function easy. So, at the very least, skip connections will not hurt performance (this is explained formally in the paper).

From the paper:

Observe: it's taking some of the layer outputs from earlier layers and passing their outputs further down and element wise summing these with the the outputs from the skipped layers. These blocks may learn mappings that are not the identity map.

From paper (some benefits):

  $$\boldsymbol{y} = \mathcal{F}(\boldsymbol{x},\{W_i\})+\boldsymbol{x}\quad\text{(1)}$$The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition).

An example of a residual mapping from the paper is $$\mathcal{F} = W_2\sigma_2(W_1\boldsymbol{x})$$

That is $\{W_i\}$ represents a set of i weight matrices ($W_1,W_2$ in the example) occurring in the layers of the residual (skipped) layers. The "identity shortcuts" are referring to performing the element wise addition of $\boldsymbol{x}$ with the output of the residual layers.

So using the residual mapping from the example (1) becomes:

$$\boldsymbol{y} = W_2\sigma_2(W_1\boldsymbol{x})+\boldsymbol{x}$$

In short, you take the output $\boldsymbol{x}$ of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.

Limitations of deep networks expressed in paper:

  When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.

The skip connections and hence the residual blocks allow for stacking deeper networks while avoiding this degradation issue.

Link to paper

I hope this helps.


What kind of neural network can be trained to recognise patterns?

The idea is simple, but it requires some time to develop.
Assumption: I am assuming in your problem the final model will have seen all possible shapes.
What your algorithm needs is a convolutional NN to understand each shape by extracting features, but you just need to be very careful with pooling.
Then what you need is a recurrent NN. In the example you showed (the image of shapes) we have bigrams (sequence of 2) which means we have the first shape as an input and the second shape is a target. In this case normal RNN should work.
But if you have sequence of many shapes, for example 10 shapes and let's say the final shape, 10th, is the target; And also if the sequence is in the way that 10th shape could be more depend on initial shapes (e.g. 1st or 2nd shape) then what you also need to consider beside RNN is long-short-term-memory (LSTM).
I cannot think of a simpler solution than this.


How can I minimize the number of answers that are relevant to a machine learning model?

You don't need to re-train on the fly. What you're looking for is an embedded feature selection algorithm, and even more precisely, one that minimizes the number of responses required.

I think this might be one of the rare cases where genetic and evolutionary approaches are the obviously correct choice.

Genetic Programming is a technique for finding models that are simply computer programs. You generate a bunch of computer programs at random, and then breed the "better" ones together. Repeating this process over time leads to highly optimized programs.

A nice feature of GP is that it is extremely flexible when picking what to optimize. So instead of "better" meaning just "more accurate", "better" can mean the sum of accuracy and $\frac{1}{\#answers\_used}$. The algorithm works the same way, and, with carefully chosen rewards, you may be able to get the best of both worlds.

There are lots of variations on this. I would probably start with a simple toolkit like EJC, and standard, boring, genetic programming.

There are specialized techniques for things like your problem too, but you'll probably get 80% of the benefit without needing to pursue them.


Why are shallow networks so prevalent in RL?

Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks

Towards Generalization and Simplicity in Continous Control tries to answer the question "What are the simplest set of ingredients needed to succeed in some of the popular benchmarks?"
Among them is the representation capacity required of the function approximators.
It shows that in popular OpenAI gym locomotion benchmarks (Swimmer 3D,  3D Humanoid, Ant, Walker), using Linear and RBF policy architectures is just as competitive in the reward attained as using neural nets. Here's a summary:

The more Fourier Featurers RBF has, the better the performance
A 2 layer neural net policy's performance is slightly short of a 500-feature RBF
A linear representation is unable to learn some of the locomotion behaviour, e.g walking

I assume RBF, and its use of Conjugate Gradients, has some shortcomings which make shallow Neural networks more preferrable. (I'm not familiar with RBF, and stand to be corrected).
Overall, the the authors use these result to suggest that complex policy architectures should not be a default choice in RL unless side-by-side comparisons with simpler alternatives suggest otherwise.
This might explain the prevalence of shallow networks. It's also possible that since most RL research (presently) doesn't try to show how we can learn better representations using complex policy architectures, but rather how to better make use of representation capacities by addressing issues like sample efficiency, shallow networks are expressive enough in the adapted benchmarks. As expressed in Brail's comment, this is also useful for setting up fair comparisons.
However, the effect of network architecture design in RL has not been widely pursued.
It's also worth mentioning the findings of this work D2RL, (2020) which shows one can gain significant performance improvement by using input concatenations in the policy and value function, and that more layers in a vanilla neural net decreases the agent performance.

The effect of increasing the number of FCs in SAC. Performance drops after 2 layers (D2RL)
Further Reads:
Why is everyone using shallow networks (Reddit thread)


Speech recognition software implementation by open-source API

The difference between a Speech API and a Speech Engine is: Speech API's enable developers to integrate speech recognition technologies into developer apps. On the other hand a speech engine is software that gives your computer the ability to play back text in a spoken voice. (Source msdn library)

Below is a list of speech recognition tool-kits and their features.

Tensorflow - Although tensorflow doesn't arrive packaged with speech recognition libraries by default. You can use seq2seq models which have achieved high levels of accuracy in speech recognition. A few of the advantages of using tensorflow for speech recognition include: It comes with tensorboard which is useful in visualising and fine-tuning your network, it's architecture is highly modular which allows you to experiment with different voice libraries and finally it is portable meaning you can run it on GPU's, CPU's, servers or even mobile computing platforms.

Microsoft's CNTK - Microsofts Cognitive Toolkit delivers excellent results when it come to speech recognition. In recent times CNTK has outperformed humans in transcribing speech to text. Some of its perks are its efficient resource usage additionally it was originally built for speech recognition systems and consequently it is very effective at working with time series data.

CMU Sphinx - CMU Sphinx is a speech recognition system developed at Carnegie Mellon University. The advantages of using CMU Sphinx are: it is multilingual and supports most international languages, it has excellent commercial support, it has a light mobile version called pocketsphinx, it has a wide range of tools for different purposes i.e. keyword spotting, alignment and pronunciation evaluation. It also enjoys active support from the Carnegie Mellon University.

Kaldi - Kaldi aims to provide speech recognition software that is flexible and extensible. Kaldi has powerful features such as pipelines that are highly optimized for parallel computing i.e. training models on the GPU. Additionally it supports speaker identification and detection of errors in transcripts.

Mozilla Deep Speech - This project aims at providing speech interfaces for the web. This project achieved a word error rate on LibriSpeech's test-clean of 6.5% which is commendable.

I found this paper comparing open source speech recognition toolkits to be relevant to your question http://suendermann.com/su/pdf/oasis2014.pdf

Siraj Raval has an excellent tensorflow speech recognition tutorial. It is available here https://github.com/llSourcell/tensorflow_speech_recognition_demo


How to identify the areas to reduce over fitting?

It is a difficult task to identify the areas to be corrected to improve accuracy unless that area is looking into your face. By that I mean unless the regularizing parameter has unusually large or small values it is difficult to pin down and identify which regularizer to tweak. You need to do a grid search or random search over the hyper-parameter space to come up with an optimal combination of hyper parameters.

There could be a lot of things that you could consider to reduce over fitting. Some of them are $L_2$ regularizers, Dropout, depth of the network, number of neurons in the layers, the optimizer,  etc.


Current Simulation of Emotional Behavior

Emotion is used in AI in very limited ways in leading edge natural language systems. For instance, advanced natural language systems can calculate the probability that a particular segment of speech originates from an angry human. This recognition can be trained using labels from bio-monitors. However, the mental features of a human with soft skills tuned from years of experience with people is not nearly simulated in computers as of this writing.

We will not see computers becoming counselors (as once believed) or directors of movies or courtroom judges or customs officials any time soon. Nonetheless, the processes behind emotion are not entirely undiscovered, and there is definite interest in simulating them in computers. Much of that work is company confidential.

The emergence of emotional sophistication in computers likely to begin in the context of sexuality, primarily because flirtation is powerful and primordial emotional expression will probably be easier to simulate in natural language than higher emotional expressions such as love or chaotic ones like rage. Sexy AI will likely be exploited by what businesses might consider legitimate marketing activity.

It is also going to be exploited by the sex industry. The ethical and moral analysis of sexy AI beyond the scope of the question but will probably gain the attention of public media as it unfolds, and that has already begun on FaceBook, originating from third party attacks using fictitious identities.

The Science of Emotion

Emotion isn't a scientific quantity. From an AI perspective, emotion is a quality an individual might recognize through visual and audio queues, specifically through the natural language and affect of another individual. (Affect is a visual clue about a person's emotional and general mental state.)

An individual can also learn to recognize those clues in her or his self. They can be detected by replaying one's own speech as heard through the ear, by linguistic analysis of thoughts not spoken, or through the detection of muscle tension or vital signs. Those skilled in meditation can detect emotional predecessors closer to their causal centers in the brain and control them more directly before emotions even arise.

In the brain, emotion is not in a single geometric location. We cannot say, "That emotion of compassion comes from this group of neurons in Jerome's brain." We cannot say, "Sheri is angry at this 3D coordinate in her cerebral cortex." Emotions are also not strictly system wide either. An individual can be annoyed without going into rage, leaving most of the brain chemistry and electrical signaling unaffected.

Emotions are not entirely electrical and not entirely chemical. On the electric side, emotional states can occur simultaneously through separate circuit pathway connecting distinct and only distantly related regions of the brain. On the chemical side, there is the synaptic chemistry that is part of the electrical signal pathways. There are also many regional signaling systems using specialized pathways that are neither circulatory (blood) nor primary electrical (neuron) pathways. Serotonin is one of dozens of chemical signaling compounds that operate regionally in this way.

Emotions, being a largely social set of phenomena, should not be characterized as purely Darwinian. Although related to survival, emotional processing and communications impact mate selection and, more generally, social patterns within a community, including altruistic and collaborative activity.

Emotions don't always lead to survival. In some cases, emotional states may lead to death prior to reproduction. One could say that emotional balance and the ability to interact on emotional levels may improve odds of having offspring. Imbalance to the degree of any of hundreds of emotional extremes can lead to childlessness.

Emotional intelligence is different than using emotions within an algorithm, but not extremely so.

Discussion of emotional intelligence is one of many advancements in the concept of intelligence since the formation of one-dimensional conceptions of intelligence. Those nineteenth century conceptions, such as IQ and G-factor are poorly supported by genetic evidence and anthropological theory. Mathematically unproven and naive concepts like general intelligence rest on those one-dimensional concepts.

Emotional intelligence is a form of mental capability related to emotional balance. If a person's cognitive skills are honed with respect to their emotions and the assessment of the emotional states of others, then they have greater emotional intelligence than someone who cannot read the affect and linguistic clues of another and cannot integrate cognitive and emotional skill to balance of their own emotions.

Cybernetic Analysis

The interface between natural emotion and artificial emotion fits within the realm of cybernetics, the conceptual study of the interface between humans and machines. Such interaction is clearly related to both algorithms and topology, two important concepts in AI research and development.

Emotion has an algorithmic context because there is clearly some combination of neurons and chemistry that produce this algorithmic difference between a reactive person and one who has developed emotional intelligence.

    emotion[person] = recognize_emotion[person]
    if emotion[person] = anger
        be_in_responses(angry)

    emotion[person] = recognize_emotion[person]
    if emotion[person] = anger
        be_in_responses(extra_calm)

The former is reactive and the later exhibits emotional intelligence. The acquisition of the later skill may be cognitive and conscious or it may be intuitive and unconscious. In either case, the actual algorithms at a lower level may be entirely different than those shown above, yet the external behavior of the person as marshaled by the brain is essentially one of those shown.

The plural, algorithms, is used rather than the singular, algorithm, because it is unlikely that a single synchronous algorithm is involved. The brain is a massively parallel processor. Emotional processing is likely best expressed in artificial form as hundreds of thousands of algorithms operating in parallel and forming millions of balances within the system — multidimensional and highly parallel stasis.

This is why emotional recognition and emotional responses are not very sophisticated in computer systems as of this writing. The balances have much social nuance. It may be easier to simulate rational thought than emotional thought.

Desire as a Systemic Behavior

Hunger and thirst may sometimes be called feelings, but they are not strictly emotional. The detection of the need for air, energy, nutrients, and water may stimulate emotional states if the needs are unmet and other emotional states if met. A person may become frustrated and irritable when lacking something essential and confronted with another person's less important agenda. A robot may someday do the same. A person may become elated and generous when all such essentials have recently been made available in surplus. A robot may someday do the same. These relationships are expressed in the question this way.

  Emotions also sometimes don't just focus on objectives/priorities — but categorize how much certain classifications are "projected" into a particular emotions.

That statement in the question and its explanation is true in some respects. If a robot that needs fuel but is afraid of passing in front of a moving vehicle because it has been hit in the past can be seen in more than one way.

Probabilistic risk management based on past experiences
Fuzzy logic that produces control behavior
Feeling fear because of past experience

In AI design, these three would be handled in different ways.

Development of a function that ties visual and auditory information to a model of collision and produces a projected likelihood of injury based each of a number of paths to obtaining fuel
Rules that relates to travel risk with learned probabilities for each along with rules that relate to energy depletion risk with learned probabilities for each and a fuzzy logic rules engine
Artificial networks that have no audit trail but simulate reptilian emotional circuitry and simulate base instinct

Maintaining Scientific Perspective

Emotion is not hard coded into the brain circuitry or DNA. The reality is significantly more complex.

The DNA provides parameters to a genetic expression system that leads to protein synthesis that leads to brain structure and function that leads to the ability to learn emotional responses that lead to improved social behavior that may lead to higher probabilities of gene pool survival.

Applying digital system traditions to biological process can be counter productive, like anthropomorphic views of programs. Artificial networks don't actually learn; they converge. Nothing is hard coded into biology because the term code applied to DNA isn't anything like a page of Java or Python code.

It is true that some behavioral predispositions are strong forms of stasis within the course of a species. An organism will normally exhibit a strong desire to acquire resources from the biosphere, such as oxygen, proteins, nutrients, carbs, fats, and water. A robot might replace those with a voltage to use for a charge and lubricants for moving parts. An organism will normally exhibit a string desire to reproduce. A robot might be given a simulation of that recursive process and wish to build another like itself.

These are not hard coded in biology. They form a kind of stasis within a population. Some humans don't want children. Some are hospitalized for anorexia nervosa. Some commit suicide by asphyxiation. The statistical mean produces the behavior of the species, not a fixed behavior identical across individuals within the species.

Nature and Nurture

Nature and nurture are useful umbrella terms for general categories of causality in biology and may have equivalents in future robotic products, but they are broad generalities. There are no nature algorithms or nurture algorithms or algorithms that balance nature and nurture. That is where topology is of paramount conceptual importance.

Topology of Algorithmic Components

There is massive interaction between many systems operating independently in multiple dimensions. The visualization of such interactive structure would look more like the topology of all the web sites in a country than a machine learning block diagram. If somehow coded into one algorithm it is possible that all the silicon from all the sand on earth converted to random access memory (RAM) might be insufficient to hold the code expressing the algorithm. Perhaps not. Perhaps a simplicity underlies the interactive system design of life. Perhaps we'll someday know. Perhaps not.

The elegance in the design of life on earth is that multiple independent processes are tuned by billions of years of trial and error to inter-operate and support complex organic processes with billions of moving parts at a molecular level.

Veins of Interdisciplinary Research

Study of these are important for biology, for bioinformatics, for cognitive science, and for artificial intelligence. Emotional recognition and integration of emotional reaction and control into natural communications is part of this research and development.


What is the concept of Tensorflow Bottlenecks?

The bottleneck in a neural network is just a layer with less neurons then the layer below or above it. Having such a layer encourages the network to compress feature representations to best fit in the available space, in order to get the best loss during training.

In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the number of feature maps (aka "channels") in the network, which otherwise tend to increase in each layer. This is achieved by using 1x1 convolutions with less output channels than input channels.

You don't usually calculate weights for bottleneck layers directly, the training process handles that, as for all other weights. Selecting a good size for a bottleneck layer is something you have to guess, and then experiment, in order to find network architectures that work well. The goal here is usually finding a network that generalises well to new images, and bottleneck layers help by reducing the number of parameters in the network whilst still allowing it to be deep and represent many feature maps.


Does use of GPU reduces time of execution of .predict() method in Keras/TF model?

Yes, Keras will use the GPU for inference (predictions) as well as training.

TensorFlow code, and tf.keras models will transparently run on a single GPU with no code changes required.

For more information, you can read the TensorFlow GPU guide, but in effect you shouldn't need to do anything to get inference to run on the GPU.


Is training a CNN object detector on an image containing multiple targets that are not all annotated will teach it to miss targets?

The neural network will learn what we teach it, for example with that image only, after finish training, your model will hard to recognize humans with dark skin, glasses, big eyes, etc, the features that two annotated targets don't have.
If your data is big enough, and contain all the feature of humans face, the result should be good.
If not, I recommend a semi unsupervised learning method which called Noisy Student. Quick explanation, you take a part of data, add noise (augmentation, drop-out, stochastic depth), and train. Then use that model to label the rest of the dataset and train a new bigger teacher (such as YOLOv3 > YOLOv1) and repeat it. They told that this method was even better than you labeled all the dataset.
You will need to choose the data to train the teacher really carefully, then pray for the result.


How can I use the success and failure data to estimate parameters of a Dirichlet distribution?

Dirichlet is the Multi Variate version of the Beta distribution. In general, these distributions can be thought to model the probability of modelling a probability distribution.

The support Dirichlet distribution is defined as follows:

$$ S_K = \{ x:0 \leq x_k \leq 1, \sum_{k=1}^K x_k=1 \} $$

and the PDF is defined as:

$$Dir(x|\alpha) = \frac{1}{B(\alpha)} \prod_{k=1}^Kx_k^{\alpha_k-1}$$

where $B(\alpha)$ is the beta function of $K$ variables:

$$B(\alpha) = \frac{\prod_{k=1}^K \tau(\alpha_k)}{\tau(\sum_{k=1}^K \alpha_k)}$$

and the resultant point estimates are (Define $\sum_{k=1}^K \alpha_k = \alpha_0)$:

$$\mu(x_k) = \frac{\alpha_k}{\alpha_0}$$
$$\sigma^2(x_k) = \frac{\alpha_k(\alpha_0-\alpha_k)}{\alpha_0^2(\alpha_0+1)}$$
$$mode[x_k] = \frac{\alpha_k-1}{\alpha_0-1}$$

Beta distribution is the special case where $k=2$

Clearly, when you run an experiment a large number of times, the success of each $k$ will approach towards its expected value i.e if you define your random variables as $x_k = \frac{N_{k}}{N}$ where $N$ is the total number of trials and $N_k$ is the success of the $k$ th term, it clearly satisfies the support of the Dirichlet Distribution and hence you can use

$$\frac{\alpha_{k}}{\alpha_0} = \frac{N_{k}}{N}$$

This is assuming that the experiment follows Dirichlet Distribution.

(Taken in parts from A Probabilistic Approach to ML)


Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$?

The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible future returns whilst $v_*(S_{t+1})$ is a probability distribution derived over all possible values of $S_{t+1}$. These will be different distributions much of the time, but their expectations are equal, provided the conditions of the expectation match.

In other words,

$$G_{t+1} \neq v_*(S_{t+1})$$

But

$$\mathbb{E}[G_{t+1}] = \mathbb{E}[v_*(S_{t+1})]$$

. . . when the conditions that apply to the expectations on each side are compatible. The relevant conditions are

Same initial state or state/action at given timestep $t$ (or you could pick any earlier timestep)
Same state progression rules and reward structure (i.e. same MDP)
Same policy

More details

The definition of $v(s)$ can be given as

$$v(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$$

If you substitute step s' and index $t+1$ you get

$$v(s') = \mathbb{E}_\pi[G_{t+1} \mid S_{t+1} = s']$$

(This is the same equation, true by definition, the substitution just shows you how it fits).

In order to put this into equation 3.17, you need to note that:

It is OK to substitute terms inside an expectation if they are equal in separate expections, amd  the conditions $c$ and $Y$ apply to both (or are irrelevant to either one or both). So if for example $\mathbb{E}_c[Z] = \mathbb{E}_c[X \mid Y]$ where $X$ and $Z$ are random variables, and you know $Z$ is independent of $Y$ then you can say $\mathbb{E}_c[W + 2X \mid Y] = \mathbb{E}_c[W + 2Z \mid Y]$ even if $X$ and $Z$ are different distributions.
$A_{t+1} = a'$ does not need to be specified because it is decided by the same $\pi$ in both $q(s,a)$ and $v(s')$, making the conditions on the expectation compatible already. So the condition of following $\pi$ is compatible with $\mathbb{E}_\pi[G_{t+1} \mid S_{t} = s, A_{t}=a] = \mathbb{E}_\pi[v_*(S_{t+1}) \mid S_{t} = s, A_{t}=a]$
The expectation over possible $s'$ in $\mathbb{E}_\pi[v_*(S_{t+1})|S_t=s, A_t=a] = \sum p(s'|s,a)v_*(s')$ is already implied by conditions on the original expectation that the functions are evaluating the same environment - something that is not usually shown in the notation.

Also worth noting, in 3.17 $\pi$ is the optimal policy $\pi^*$, but actually the equation holds for any fixed policy.


What programmable devices can be used to teach artificial intelligence?

LEGO Mindstorms is widely used to demonstrate AI in schools and universities [1, 2]. With LEGO as basis, you are very flexible. You can build what you want very easily. The AI programs can be written in different languages from very easy graphical once to Lisp and C++. The newest version has an SD Card drive, USB interface and a powerful ARM processor. You can use four motors and four sensors directly. There exists touch, sound, sonar, gyro, infrared and color sensors. There is also a big community which provides you with a lot of ideas, hardware and programs [3].


How is uniqueness quantification translated in First Oder Logic

According to the Wikipedia entry on Uniqueness Quantification your lecturer is correct. There is no size requirement expressed in the FOL expression.

The point about the implication is that it can be true if the antecedent is false. So, there is a house in area1 (which we call x). And all houses in area1 which are smaller than 200 are the same as x. But if there aren't any, then the antecedent is false, and the consequence (x = y) is false, but the whole statement is still true.

As another example: "If Trump is the 31st president of the USA, then the moon is made of green cheese". Both antecedent and consequence are false, but the whole statement is still logically true. Same as "If there is a house in that area, and there are houses that are less than 200 (which there aren't), then that house is one of them."

Moving on to the correct expression: The unique quantifier (usually written as ∃!) can be rewritten using the existential and universal quantifiers as follows (see the above mentioned Wikipedia page):

  ∃x (P(x) ∧ ∀y (P(y) -> y = x))

This is not what you have got; you have got two different predicates, P1 and P2. Your P1(x) is (house(x) ∧ in(x, area1)), and your P2(x) is (house(x) ∧ in(x, area1) ∧ size(x) < 200)

The correct expression would require the same predicate for the quantifiers and would therefore be

  ∃x ((house(x) ∧ in(x, area1) ∧ size(x) < 200) ∧
    ∀y ((house(y) ∧ in(y, area1) ∧ size(y) < 200)
      -> y = x))

The difference is that you state that there is at least one house in the area with a size of less than 200. So the second predicate, that y is a house in the area with a size of less than 200, cannot be false.


Is my understanding of the value function, Q function, policy, reward and return correct?

Value function: How good it is to be in a state $s$ following policy $\pi$.

There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a function of only one variable, i.e. $s$ (a state). There's the state-action value function $q(s, a)$ (or $Q(s, a$)). A value function is a function, so it's not a number or a vector, or whatever. It's a function, so it maps inputs to outputs. In the first case, it maps states to real numbers. In the second case, it maps states and actions to real numbers. So, we could denote the state value function as $v : \mathcal{S} \rightarrow \mathbb{R}$ (where $\mathcal{S}$ is the set of states in your environment) and state-action value function as $q : \mathcal{S} \times \mathcal{A}\rightarrow \mathbb{R}$ (where $\mathcal{A}$ is the set of actions and $\times$ means "combination of").

So, your definition of a value function is not quite correct. The value function $v(s)$ doesn't represent "how good it is to be in a state $s$ following a policy $\pi$", but "how good it is to be in a state $s$ AND THEN following policy $\pi$". To emphasize this, you often use the notation $v_{\pi}(s)$ rather than simply $v(s)$.

See What are the value functions used in reinforcement learning? for more details about existing value functions in reinforcement learning. And to see the full definition of the value functions, I suggest you read Sutton and Barto's book.

  Q function (also called state-action value, or just action value): How good it is to be in a state $s$ and perform action $a$ while following policy $\pi$. It uses reward to measure the state-action value

As I said above, the $q$ function is a "value function" too. It's just a different value function than $v$.

Again, the same thing I said for $v$ also applies here, so "how good it is to be in a state $s$ and perform action $a$ while following policy $\pi$" is incorrect for the same reason your definition for $v$ was incorrect. The $q$ function can be defined as "how good it is to be in a state $s$ and take action $a$, AND, AFTER THAT, follow a given policy $\pi$. Again, to emphasize that $q$ is defined in terms of $\pi$, we often use the notation $q_\pi$.

  Reward: The metric used to tell the agent how good/bad it's action was. It is a constant value.

This is roughly correct, but the reward doesn't have to be constant and it depends on your problem. Also, there's also the related notion of "reward function", which is the function that assigns rewards to each action. So, when defining your problem as a Markov decision process, you need to define this reward function. Actually, this is probably the most important function in reinforcement learning (because this is the way you teach the agent to behave).

  Return: The sum of rewards in a single episode

This is roughly correct. However, note that the sum can also be a "weighted sum".

  Policy: A set of specific instructions an agent will follow in an episode.

This is roughly correct, but a policy can also have some randomness in it. For example, if you are in state $s$, your policy could say "always take action $a_i$", but another policy could say "take action $a_i$ with probability $p$ and action $a_j$ with probably $1 - p$. Also, note that the policy is not restricted to an episode. It's a general function that tells the agent how to behave independently of the episode.

(Sorry, I didn't look at your examples. Maybe I will review this answer later to look at your examples too, but the information in this answer should already tell you if your examples are correct or not).


Where to publish replications of machine learning papers?

OpenReview (https://openreview.net/) may be what you're looking for. It's both a platform for ML conferences to organize their reviewing process as well as an arXiv-like preprint repository.

One major machine learning conference (ICLR) has been holding "reproducibility challenges," in which participants can submit replication reports of existing papers, but that probably only applies to papers submitted to that conference.

There is a general preprints section, which might be your best bet.

Disclaimer: I used to be on the OpenReview development team