Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces

Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the space to a small number of states. Provided you can sample and update enough times, then a few million states is not a major problem.

However, with large state spaces it is more common to use some form of function approximation for the action value. This is often noted $\hat{q}(s,a,\theta)$ to show that it is both an estimate (the circumflex over $\hat{q}$) and that you are learning some function parameters ($\theta$). There are broadly two popular approaches to Q-learning using function approximation:

Linear function approximation over a processed version of the state into features. A lot of variations to generate features have been proposed and tested, including Fourier series, tile coding, radial basic functions. The advantage of these methods are that they are simple, and more robust than non-linear function approximations. Which one to choose depends on what you state space represents and how the value function is likely to vary depending on location within the state space.
Neural network function approximation. This is essentially what Deep Q Networks (DQN) are. Provided you have a Markov state description, you scale it to work sensibly with neural networks, and you follow other DQN best practices (experience replay table, slow changing target network) this can work well.

Q-learning for continuous action spaces

Unless you discretise the action space, then this becomes very unwieldy.

The problem is that, given $s,a,r,s'$, Q-learning needs to evaluate the TD target:

$$Q_{target}(s,a) = r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$$

The process for evaluating the maximum becomes less efficient and less accurate the larger the space that it needs to check.

For somewhat large action spaces, using double Q-learning can help (with two estimates of Q, one to pick the target action, the other to estimate its value, which you alternate between on different steps) - this helps avoid maximisation bias where picking an action because it has the highest value and then using that highest value in calculations leads to over-estimating value.

For very large or continuous action spaces, it is not usually practical to check all values. The alternative to Q-learning in this case is to use a policy gradient method such as Actor-Critic which can cope with very large or continuous action spaces, and does not rely on maximising over all possible actions in order to enact or evaluate a policy.

Controlling a pendulum

For a discrete action space e.g. applying one of a choice of forces on each time step, then this can be done using a DQN approach or any other function approximation. The classic example here might be an environment like Open AI's CartPole-v1 where the state space is continuous, but there are only two possible actions. This can be solved easily using DQN, it is something of a beginner's problem.

Adding continuous action space ends up with something like the Pendulum-v0 environment. This can be solved to some degree using DQN and discretising the action space (to e.g. 9 different actions). However, it is possible to make more optimal solutions using an Actor-Critic algorithm like A3C.


What activation functions are currently popular?

Currently both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.
Also these AFs, more specifically ReLU, are very fast learning AF which make them even more useful in research.
However, depending on type of NN you working on, it's always a good practice to pay some attentions to new studies.


Using feature learning for a medical text classification problem

Having just looked through a few entries from the corpus, I'd personally be skeptical of the applicability of any naive approaches.

In particular light of your small training set, I'd recommend that
whatever method you use should be able to produce human-readable
explanations for the operation of the classifier it builds (e.g.
decision trees/learning classifier systems/genetic programming):
this allows 'common sense' tuning of the classifier, rather than the
danger of overfitting to the training set via black box parameter
optimization.
Rather than throw the entire 'bag of words' at a classifier and hope
that the appropriate set of features will be extracted, you should
first consider what kind of criteria you as a human being might use
to make that decision, and how you might be able to pre-process to
produce features (e.g. syllable-length, metrics from ConceptNet etc) that
are as close to these as reasonably possible.
Having used some human intuition to obtain a reasonable set of
feature primitives, then you can build your classifier and obtain
higher-level expressions that discriminate between them.


Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?

From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.

For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "dog or muffin". A purely visual approach would find this hard to do.

However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.

But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.

Overall I do not think it is currently possible to solve this problem with high accuracy.


Are A2C or A3C suitable for episodic tasks where the reward is delivered only at the end of the episode?

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the episode.

This seems fairly accurate. The important thing to note is that the trainable value function is trained to predict values (specifically, advantage values of state-action pairs in the case of A2C / A3C, where the first A stands for "advantage"). These value estimates can intuitively be understood as estimates of long-term (discounted) rewards, they're not just short-term rewards.

So yes, initially when the agent only observes a reward at the end of a long trajectory, only state-action pairs close to the end will receive credit for that reward. For example, when using $n$-step returns, approximately only the last $n$ state-action pairs receive credit. However, in the next episode, that longer-term reward will already become "visible" in the form of an advantage value prediction when you're still $n$ steps away from the end, and then that update can again get propagated back $n$ steps further into the history of state-action pairs.

My explanation above is very informal... there are all kinds of nuances that I skipped over. Use of function approximation is likely to speed up the propagation of reward observations through the space of state-action pairs even more, and of course in reality things won't be as "clean" as getting the propagation to get $n$ steps further in the next episode in comparison to the previous episode, since selected actions and random state transitions can be different... but hopefully it gets the idea across.


Why isn't Nilsson's Sequence Score an admissible heuristic function?

I will use the 8-puzzle game to show you why Nilson's sequence score heuristic function is not admissible.  In the 8-puzzle game, you have a $3 \times 3$ board of (numbered) squares as follows.

+---+---+---+
| 0 | 1 | 2 |
+---+---+---+
| 7 | 8 | 3 |
+---+---+---+
| 6 | 5 | 4 |
+---+---+---+

The numbers in these squares are just used to refer to the specific squares (to avoid saying the "middle square" or the "upper-left square"). So, when I say square 1, I refer to the upper-left square. In these games, you have 8 "tiles". Let's denote these tiles by $A, B, C, D, E, F, G$ and $H$. So, in this game, there is always one square which is free (or empty), given that there are $9$ squares. The goal of this game is to reach the following configuration of tiles

+---+---+---+
| A | B | C |
+---+---+---+
| H |   | D |
+---+---+---+
| G | F | E |
+---+---+---+

Note that, in the case of the 8-puzzle game, a "state" is a configuration of the board. So, the following two board configurations are two distinct states

+---+---+---+
|   | A | C |
+---+---+---+
| H | B | D |
+---+---+---+
| G | F | E |
+---+---+---+

and

+---+---+---+
|   | C | A |
+---+---+---+
| H | B | D |
+---+---+---+
| G | F | E |
+---+---+---+

The rules of the 8-puzzle are simple. You can move one tile (at a time) from its current position (or square) to another position, provided that the destination square is free. You can only move a tile horizontally and vertically (and one square at a time).

I will not explain in this answer how the Nilson's sequence score heuristic works. Here is a explanation of how is used in the case of the 8-puzzle game. You should read this explanation and make sure you understand how Nilson's heuristic works before proceeding! You can also find an explanation of how this heuristic works in the book "Principles of Artificial Intelligence" (at page 85), by Nils J. Nilsson (1982).

  Why isn't then Nilson's sequence score admissible?

A heuristic function $h$ is admissible if $h(n) \leq h^*(n)$, for all states $n$ in the state space, where $h^*(n)$ is the actual distance to reach the goal (which is often unknown in practice, hence the need to use heuristics to estimate such distance).

Note that admissibility is a property that must hold for all states. So, if we find a state where the condition above is not satisfied for the Nilson's sequence score heuristic function, then we show that the Nilson's sequence score is not admissible.

Let us create the following state

+---+---+---+
| A | B | C |
+---+---+---+
|   | H | D |
+---+---+---+
| G | F | E |
+---+---+---+

Note that this state only differs from the goal state by one move: if we move the tile $H$ to square $7$, then we reach the goal state. So, the actual distance to reach the goal state is $1$ move. But what does the Nilson's score function tell us regarding the distance of this state to the goal state? You can see from the algorithm presented in this answer to compute the Nilson's sequence score (of a board configuration) that the score of the configuration (or state) above would be more than $1$ (you can immediately see this because you need to multiply by $3$). Therefore, Nilson's sequence score overestimated the distance to the goal (at least for one state), thus it cannot be admissible (by definition).


Getting loss from each sample in CNN (python)

You may use Keras model which is provided in TensorFlow. Keras model lets you evaluate a batch of different number of samples compared to when training.
For example, when training:
myModel.fit(x=X, y=Y, batch_size=BATCH_SIZE, epochs=EPOCHS, verbose=1)

Evaluate every single sample to get output loss of each sample (https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate):
losses = []

for i in range(len(X)):
    sampleX = X[i]
    sampleY = Y[i]
    loss = myModel.evaluate(x=[sampleX], y=[sampleY], batch_size=1, steps=1, verbose=0)
    losses.append(loss)

Another way is to use Keras callback during multiple sample evaluation (https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback#on_batch_end):
losses = []

class myCallback(tf.keras.callbacks.Callback):
    def on_batch_end(batch,log):
        global losses
        losses.append(log["loss"])

myModel.evaluate(x=X, y=Y, batch_size=1, steps=NUM_SAMPLES, callbacks=[myCallback()])
```


Looking for comprehensive material on learning in mammals

I recommend Animal Learning and Cognition: An Introduction by John Pearce.

As an introduction to the field, it's comprehensive. It covers all the topics you mentioned and more. Here is a free preview of the book.
It's well-regarded, having over 500 citations on Google Scholar.
It's relatively modern. Published in 2008, it discusses experimental findings that you won't be able to find in some of the other classics published in the 70s or 80s.

Although it does not specifically focus on mammals, all of the ideas extend to mammals.


Why are large models necessary when we have a limited number of training examples?

If you read the relevant section. it also says:

Model compression is applicable when the size of the original model is driven
primarily by a need to prevent overfitting. In most cases, the model with the
lowest generalization error is an ensemble of several independently trained models.
Evaluating all $n$ ensemble members is expensive. Sometimes, even a single model
generalizes better if it is large (for example, if it is regularized with dropout).

The keyword (I think) here is dropout. Dropout Learning in the referred book has been intepreted as training an ensemble of models, with a model probability same as the probability of a particular dropout architecture of the large Neural Network. Thus, this effectively makes the training as training multiple smaller Neural Nets. According to this paper on dropout, by the original authors, dropout prevents co-adaptation which effectively means you are just training an ensemble of Neural Nets. But this intuition lacks any theoretical justification.
Another paper (understanding the paper might require familiarity with certain statistical ideas of ML) claims this is not true, and dropout doesn't reduce co-adaptation but more likely reduces the variance over dropout patterns. They have provided better empirical and theoretical justifications to this end. So it is still up for debate what actually happens.
But in general the generalization error upper bound very roughly is directly proportional to the size of the Neural Nets. So yes the authors statement in face value is oversimplified and most likely wrong in the general case.


What would be the steps to create an sentiment analysis chatbot?

There are some different approaches to learning this sequential acting. First, you can use RL (reinforcement learning) and define some rewards over the action of the user over that question, to explore and exploit the environment.

Second, you can use RNN to predict the preferred next question from a history of data. In contrast with RL, you need many instances to train the network to predict better next answer more accurately.


How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then?

Policy evaluation is the process of determining state-value $v_{\pi}(s)$ or action-value $q_{\pi}(s, a)$ functions for the current policy. In the context of continuous state and action spaces without a model of the environment, policy evaluation must incorporate the agent's past experience instead of the model dynamics and will generally use a function approximator such as a neural network to estimate the action-values. Many popular approaches apply online updates to the function approximator; e.g., DQN combines Temporal-Difference targets and gradient descent to change the weights of the neural network and the resultant action-value estimates. Since

we gradually change the weights of the neural network at each gradient descent step,
the estimated action-values are solely dependent on the weights of the neural network,
the current policy is solely dependent on the estimated action-values (e.g. DQN takes the action with largest action-value),

then policy evaluation (updating the estimated action-value function to better match the true action-value function under the current policy) and policy improvement (greedily changing the current policy based on the new estimated action-value function) occur simultaneously at each gradient descent step. In DQN, a gradient descent step occurs at each time step.

  I am thinking of Monte Carlo simulation, but that would require many many episodes.

After changing the action-value function, we may get a new policy. We assume that the action-value function of the old policy is similar to that of the new policy (although it is not guaranteed) because the change in weights of the neural network was small. Therefore, we use the estimated action-value function of the old policy as an initial estimate of the action-value function of the new policy. Specifically, we use the same neural network with the same weights as the initial approximation. This is computationally convenient, as it prevents the need to start the next policy evaluation update from scratch (e.g. with a Monte Carlo simulation over a painfully large number of episodes).

  Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming and solving the Bellman equation.

This same technique of using the estimated action-values of the old policy as the initial estimates for the action-values of the new policy is employed by some Dynamic Programming methods such as value iteration, albeit with known dynamics. Generalized Policy Iteration (GPI) is the notion of letting policy evaluation and policy iteration interact on whatever granularity deemed necessary for the problem at hand. A consequence of adopting the GPI paradigm is the choice to halt policy evaluation before convergence of the action-value function. Many deep reinforcement learning algorithms take this to an extreme and perform policy evaluation and policy improvement simultaneously during a single gradient descent step. For reference, Chapter 4 of Sutton and Barto provides a brief summary of these ideas.


Strategy for playing a board game with Minimax algorithm

I'm not familiar with your game so I can't tell you what a good heuristic woul be in your specific case, but I can give you some advice on how to look for a good heuristic function.
As a rule of thumb, the heuristic function for a MiniMax algorithm is best kept simple and efficient, so you can get deeper into the tree. But it depends on how costly it is to compute the heuristic function compared to simulating moves in the game.
If the heuristic takes longer than simulating a game move, it might be worth simplifying it so it runs faster and you can look ahead further. This often leads to more emergent and advanced strategies that are hard to express mathematically. An extreme example of a simple heuristic would be the current player score minus the opponent's score. Since scores only change when someone lands on a bonus tile, many paths you take down the tree would have equal value, so you need to be able to look ahead many moves to find a non-zero heuristic and be able to prune parts of the tree. But because the heuristic is so fast to compute, you can do this and discover more non-obvious strategies, simply by brute force simulation. This leads to more emergent behavior, and would tell you more about different ways to play the game (if that is your goal).
If simulating a game move takes longer than your current heuristic, it's probably not worth making the heuristic faster since the game simulation is the dominating factor in how deep you can go down the tree. This case is much more difficult because it means you have to find optimal strategies (and how to express them as a heuristic function) yourself. This is more of an art than a science - ask any chess grandmaster. I'd look up existing literature on the game and see if any existing strategies can be translated into a heuristic function. If there is no literature (e.g. because the game is new or unpopular), you could spend some time playing it yourself to discover what works. Alternatively (and perhaps more interesting), you could use a simple heuristic function with more emergent behavior, increase the time the MiniMax algorithm has to make moves, and play against your NPC opponent a few times to see what strategies it discovers. Or even have two NPCs with different heuristic functions play against each other. Then try to incorporate those into your final heuristic function. This is also a way to determine which heuristic function is better, if you have multiple candidates.
There are ways to optimize the heuristic function automatically using machine learning (specifically reinforcement learning), but it's probably not worth opening that can of worms in your case.


How does a software license apply to pretrained models?

I am not sure if a pretrained machine learning model is actually protected by copyrights or not. Copyright protection exists to protect the creators of creative works from having their work "stolen", and I am not sure if training a ML model is an act of creativity.

That said, assuming that a pretrained ML model is actually protected by copyrights, then it is more likely that the model is a derived work of the data set used for training than that it is a derived work of the software that uses the model.

The software reads the model in as data, assuming that the software can be used with many differently trained models. In that case, the software and the model are considered completely independent works in the same way that MS Word and the documents you write with it are independent works for copyright.

Thus, if you want to publish the trained model with a license, I would recommend to use the BSD license that was also used for the training set.


Making generated texts from "data-to-text" more variable

The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. This article and this one should give you an example of how powerful it is at text generation.
GPT is a transformer based architecture, somewhat like BERT. The main difference is it only takes into account left context, which is why it is so well suited for text generation.
GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.
Here is a link to a tutorial explaining the basics to get you started with GPT-2.
If you are interested in diving into the relevant research papers:
Here are the open AI papers on GPT 1-3

Improving Language Understanding by Generative Pre-Training
Language Models are Unsupervised Multitask Learners
Language Models are Few-Shot Learners

Additionally, if you have never seen transformers before, take a look at:

Attention Is All You Need


Nice Question!

This is a perennial topic of discussion among AI researchers. The short answer is "we don't really know which topics are hard in general, but we do know which we haven't got good techniques for yet."

Let's start by explaining why AI is not concerned with notions of computational complexity like NP-Completeness. AI researchers figured out in the 90's that most problems that are computationally hard in theory aren't actually hard in practice at all! For example, Boolean Satisfiability, the canonical NP-Hard problem, is known to have hard instances, but in practice, these almost never show up. Even when they do, we can usually get good approximate solutions with minimal computation time. Since many AI problems are reducible to SAT, whole areas of the field just use these approximation techniques and solvers. There's a good, if a bit old, survey here. Since 2008, things have only gotten better. Basically, NP-Hard stuff just isn't that hard. Worst-case complexity is therefore probably the wrong tool to guess at which problems are hard for AI.

At the other end of the scale, we have subjective complexity based on things like how "large" the problem is. This has proven to be pretty unreliable. One example is Go, which I was told would "never be solved by AI" as late as 2010. Clearly, we were completely wrong. We just didn't know which techniques to use yet. Another example is language. Rule-based AI'ers tried at it for decades with minimal success. Probabilistic methods have achieved essentially human-level performance in less time. If you asked a researcher in the 1970s if language was hard for AI, they'd have said yes, but they'd have been wrong. This has been closely related to the advances in computing hardware: techniques that seemed wasteful and slow 40 years ago are now entirely practical. Sometimes they turn out to solve problems really well.

Part of this ties into the issue that AI'ers don't really know or agree on what intelligence is, or what it means to solve a problem. Some AI'ers maintain that language really is hard, and that the systems we use now aren't really solving language, they're just making blind guesses that happen to be right a lot. Under that view, language is still hard. Fodor was a strong proponent of this view in the past, and his writings are a good place to read about it, but I've still heard people espouse views like this at AI conferences as recently as 2015.

Something this usually considered hard right now is making a good general intelligence, that integrates knowledge across many domains and could plausibly pass something like the Turing test. Yet, there has been some progress here (look at virtual assistants), and some AI'ers reckon this might not be as hard as we thought either.

So, basically, we don't know what's hard because we don't know when someone will come up with a new exciting technique that solves problems previously thought to be hard, we know that worst-case complexity isn't a good measurement system for "intelligence" requirements, and we don't know what intelligence really is.


Reduce receptive field size of CNN while keeping its capacity?

One way to keep the capacity while reducing the receptive field size is to add 1x1 conv layers instead of 3x3 (I did so within the DenseBlocks, there the first layer is a 3x3 conv and now followed by 4 times a 1x1 conv layer instead of the original 3x3 convs (which increase the receptive field)). In doing that, the number of parameters can be kept at a similar level. While 1x1 convolutions are good to add non-linearity, they are not useful to learn spatial information where neighboring pixels/values correlate. This might turn out to be a problem.

Therefore, experiments will have to show if this idea of adding 1x1 conv layers to keep the capacity proves useful or not in learning spatial features on a natural image data set.


Can the same input for a plain neural network be used for a convolutional neural network?

There is no restriction on how you input a data to a NN. You can input it in 1D arrays and do element-wise multiplication using 4-5 loops and imposing certain conditions(which will be slow and hence $nD$ matrix notations are used for a CNN).
Ultimately, the library you are using (TensorFlow, NumPy might convert it into its own convenient dimensions).
The main thing different of a CNN from a normal NN is:

The number of parameters of a CNN in a convolutional layer is less than the number of input features.
$parameters \le features$ (in general it is less than).

Different people have different ways of viewing how the convolutional layer work but the general consensus is that the weights of the convolutional layers of a CNN are like digital filters. It will be a $nD$ filter if input dimension is $nD$. The  output is obtained by superimposing the filter on a certain part of the input and doing element-wise multiplication of the values of filter and the values of the input upon which the filter is superimposed upon. How you implement this particular operation depends on you.

So the answer to your question will be same network cannot be used, but it might be used with modifications (a normal NN is the limiting case of a CNN where $features=parameters$.


Are mathematical models sufficient to create general artificial intelligence?

Mathematical models are essentially highly formalised knowledge. When it comes to computer engineering, there is literally no other choice - anything you can write code for, or design a machine for, will have an associated mathematical model. That model may not be fully explored or comprehended analytically by theorists, it may be just too complex (and driven by calculations) or even mathematically intractable. However, that doesn't make it non-mathematical, just more driven by empirical results than theory.

We don't have complete models for how AGI definitely work, nor tight enough definitions of general intelligence to base maths from where we can say "if we implemented a framework based on this maths, we could build an AGI". Right now, exploration and experiments based on intuition of what might work are far ahead of such theory.

The theoretical work behind e.g. neural networks is chipping away at the problem, and there are more general over-arching theories about intelligent rational behaviour available e.g. the equations of AIXI. AIXI doesn't cover emotions and intuition directly, but does attempt to cover knowledge and how a rational agent would approach understanding the world in general from scratch. It is possible that an embodied system driven by a software implementing something like AIXI could exhibit intuition and emotions in an emergent fashion, but whether or not that would happen in practice is not at all clear from the theory. AIXI is just one theory/model out of many, and I am not qualified to analyse it in depth, but its creators have strong pedigree in AI research, so it is as good as any IMO if you are interested in starting to research AI from a theoretical perspective.

Despite all the unknowns, the success of deep learning and the loose analogies between artificial neural networks and biological ones, makes it look likely that neural networks or something like them will form a component of an AGI. The current state of the art for more narrow problem solving, learning from examples or experience through backpropagation of error gradients, might not be the most important or key component. However, whatever the structure, whether it is a small extension of existing systems, or involves some new  science, it will be describable mathematically.


What are the segment embeddings and position embeddings in BERT?

These embeddings are nothing more than token embeddings.

You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.


Are Computer Vision and Digital Image Processing part of Artificial Intelligence?

Artifical Intelligence is a rather fuzzily-defined field. It includes lots of subject areas. Sometimes that's because they replicate some behaviour or mental ability of creatures used to solve problems, and sometimes it is because they were the focus of "classic" AI, and have remained classified as part of AI. These problems include navigation, searching, solving combinatorial or logic puzzles, complex systems control, communication. Many of the problems can include converting visual input to something more useful to the rest of the system.

Computer Vision by comparison appears more tightly-defined. However, it quickly becomes less so when you try to consider what the eventual outputs should be. To what extent does a computer vision system need to "understand" the visual inputs in order for other system components to make sense of them, and use them correctly? Here are a couple of examples to illustrate my point:

Image captioning

For instance, work in automated captioning effectively includes natural language models. The input to the system is an image. The computer vision half of that system does not output a class or direct measurement of something inside the image. Instead it outputs a vector encoding of the description of the whole image - something that the language model can decode into an English sentence.

The vision model and language model are trained together, so this encoding can be very loosely considered an internal "ideagram" that represents the contents of the image and allows the vision and language models to share representations.

Agents using vision systems for decision-making

Another interesting example is recent work on Deep Q Networks, used to give superhuman performance when playing some Atari games. This used a neural network architecture (CNN) already known to be good for image classification problems, and used it to directly connect the pixel output from a game to a system which learned optimal behaviour. This is more clearly work in AI - creating a learning agent that works across a general set of problems - but at its core is a CV system.

In this case, the input is four consecutive images (although there are variations which learn from the stream of individual frames), and the output is a vector which scores the value of taking one of the possible controller actions (left, right, up, down, fire etc). Variations on the system directly output which action would be preferred from the CNN.

You could take this further and require that a vision system produce some kind of explainable, transferable, shared world model for other systems to interact with. An embryonic version of that approach is effectively how the captioning system works. It is not at all clear whether such an imagined design is strictly in the field of CV or some other part of AI.

Whether such outputs are considered part of computer vision or more generally part of AI is less of a pure scientific classification issue than an opinionated or political one (e.g. due to funding either an AI research team or a CV one).

The similarities between current computer vision approaches and ongoing work in AI, such as heavy use of deep learning models, would suggest that CV is strongly related to AI. Intuitively I would say that CV is part of AI, perhaps a key specialism of it, roughly equivalent in scope to Natural Language Processing. However, the practical outcome of that is only important to people who have labelled themselves "AI researcher" or "CV engineer" etc.

I also think the same arguments apply to other studies of artificial perception, most notably speech recognition.


Is it feasible to train a Machine Learning Model (with image inputs) in an average personal computer?

You may play around on an average laptop but training will be very slow and you will be limited on the size of your model.
Once you try to build something more serious you will run out of memory very fast. A system with a GPU is recommended if you want to really do things like image recognition. If you buy something I would not go for any GPU with less than 8 GB memory and for an X99 motherboard to keep the flexibility to add a second GPU on full speed later if needed.
Don't mix up image recognition running on phones etc. Those are trained models and actually just being applied that is much easier process. Training is way much more expensive.


Is average pooling equivalent to a strided convolution with a specific constant kernel?

Yes.

Why then are explicit pooling layers needed if they can be realized by convolutions?

It is probably because the convolution is more expensive than the usual/natural implementation (i.e. just summing and then dividing).
To see why, let's consider your example. If you implement pooling with a convolution (or cross-correlation), we would need to perform $9$ multiplications, then $8$ summations, for a total of $17$ operations. If we implement pooling as usual, we would need to perform $8$ summations and $1$ division (or multiplication), for a total of $9$ operations.
Moreover, convolution may also be more prone to numerical instability (multiplications of numbers in the range $[0, 1]$ are not nice), but I am not completely sure about this, given that we multiply always by the same numbers.


Seq2Seq dialogs predicts only most common words like `you` after couple of epoches

To answer my own question - it was because of 2 things:

Too small number of batches - Model just started to gain statistical knowledge about language dialogs. I needed to train it longer.
I masked sequence too early (wrongly removed the < END > tag) - Because on each sentence the last world is just < END > tag - I removed it on all training examples, which prevented Model from learn what does it mean "the end of the sentence".

The last condition probably caused that strange pattern even further, because if model doesn't know what word to put in (and because of the lack of the < END > tag) it must fill each sentence till the end of max_sequence_len.

So the Model inputed in loop-like-manner - one of the most common word (where there was no signal from target sentence, because it simply ended).


Why do we need both the validation set and test set?

The difference between the validation and test set in my opinion should be explained in this way:

the validation set is meant to be used multiple times.
the test set is meant to be used only once.

I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model training. In every tutorial standard datasets are used so that you don't have to worry about data collection, data labelling (it's really sad to see that lot of people have not a clue about what the inter annotator agreement is), data pre-processing and especially, all the part about the real application of the model is almost never mentioned.

The importance of having a set of instances that you can use for fine tuning (validation) and a set of instances that your model never encountered neither in training nor during fine tuning (test) becomes particularly clear if you focus on the subsequent deployment of the model you trained. No one expect a model to have the same performance scores in training and when applied to some unknown data. And the crucial point is that the performance of a model on the validation set are not representative either of the behaviour of a model with unknown data, because the same validation data have been used to fine tuned the model! So here's why having a set of data completely new to the model is important, because it gives you a much more unbiased view about the model performance on a real use case scenario.


I think the answer depends very much on why you are reading the paper, what are you trying to get out of it? There are plenty of papers that I "read" (or often really just quickly skim through) where I'll definitely not understand all the math. More often than not, this will be because I don't actually care to deeply understand it.
There is plenty of more "practical" research to be done in AI, which definitely doesn't always require a deep understanding of all the math. Intuition can often be enough, at least to get started, for meaningful practical contributions. If this is the sort of research that you're interested in doing, you probably don't need to understand as many of the mathematical parts of AI papers as you do if you're really trying to do research directly in that theoretical area.
Personally, when I write "math-heavy" parts in my own papers (and that will often already be restricted to a rather simple level of math in comparison to the "real theory" ML papers), I always try to make sure to include intuitive, English descriptions of what we're doing around it. Even if you don't immediately understand a full equation, just having the intuitive explanation around it to tell you what it means can be enough for a broad understanding of the paper. Then you only have to dive deep into the details of the equations if -- based on the English text -- you decide that you're actually really interested. So, if there are sufficient, intuitive explanations surrounding the equations, I'd recommend to focus heavily on that first. Not every paper does this though, sometimes there's very little text and very much math, and then this can be difficult.
Even if it turns out that you do have to understand math, you may not have to understand ALL of it right away though. The important parts that I would try to focus on understanding first are:

A mathematical description of the "problem". This could be an objective function, a metric to be optimised/minimised/maximised, or an existing equation from previous literature that the authors take as a starting point and inspect some detail of in greater detail.
Mathematical descriptions of the outcomes/results. These could be equations that they actually use in concrete algorithms (see if you can relate them to any pseudocode that may be present), or the final equations stated in theorems / at the end of proofs.

All the complex parts in between are probably less important. Just a vague idea of what the starting point is, and a vague understanding of the final outcome, can be enough to at least know what the paper is about. Then you can decide for yourself whether you really need to know more about the details in between, or if they're maybe not relevant to you / your work / your research.


How is it possible to get the output size of `n` Consecutive Convolutional layers?

The question was also asked in Pytorch Forums, where it has been sort of resolved and would love to have further discussions over there.
Link to Pytorch thread: https://discuss.pytorch.org/t/how-is-it-possible-to-get-the-output-size-of-n-consecutive-convolutional-layers/87300


What sort of game problems can neural networks trained/evolved with evolutionary algorithms solve, and how are they typically implemented?

Without going in too much detail on how exactly Neural Networks and Generic Algorithms work, I can tell you that both the algorithms are not good candidates for computer games.  They work well in scientific environments where the system is "trained" on a huge data set to adjust the "weights" (variables) for a given problem.  This "training" process requires a lot of processing power, time and a large data set.

Computer games, however either needs to run in real-time (no time for training) or turn-based (not enough data for training).

Another problem is that computer games need to free up as much as possible system resources for physics, graphics, sounds and the user interface to improve the player's experience so game developers usually use other lighter techniques (like a rule-based system) to create the illusion of an AI player.


Ideas on how to make a neural net learn how to split sequence into sub sequences

The problem in the original question is akin to that of inducing a context-sensitive grammar (CSL), except that it is harder because a CSL is assumed to be composed of fixed-length subsequences. It is probably closer to the problem of inducing a Reber grammar, but that in turn seems like an overkill.

LSTMs are known to be able to learn both CSL and Reber grammars. However, I doubt that this is what you really need because of the following comment:

  [...] given an entire book where there is NO spaces anywhere, only characters (including special characters, like commas), in what way can we make the network learn the 'word boudaries' of this book.

This is called morphology induction, and it is a much harder problem than that of simple Reber grammar induction. Note that finding word boundaries is a special case of the problem of finding morpheme boundaries. There have been many attempts to solve this (also see this survey paper for more details and references).

Most approaches developed seem to rely on statistical principles (like MDL) and do not use neural networks (a counterexample using LSTMs). My intuition is that the extreme morphological variability across languages (ranging from Finno-Ugric languages with highly inflectional morphology to Sino-Tibetan languages with hardly any morphology at all) makes it hard to train neural networks in a language-agnostic way. However, you might have better luck if you focus on a single language.

Hope that helps.


Would you term Google's Captchas as Turing Test?

In the standard Turing test (or imitation game), the interrogator can ask multiple arbitrary questions, while, in the case of captchas, usually, there's only one question or problem. Additionally, in the Turing test, the interrogator interactively communicates with both the human and the machine. Furthermore, captchas do not test the conversational skills but only the visual ones of the AI. Therefore, the existing captchas are not an example of the standard Turing test.


Training accuracy vs validation accuracy on deep models

Neither of the above mentioned methods could be a potent indicator of the performance of a model.

A simple way to train the model just enough so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, the model learns the statistical noise within the data and starts overfitting.

This technique of Early stopping could be implemented in Keras with the help of a callback function:

class EarlyStop(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if(logs.get('val_loss') < LOSS_THRESHOLD and logs.get('val_categorical_accuracy') > ACCURACY_THRESHOLD):
            self.model.stop_training = True

callbacks= EarlyStop()
model.fit(...,callbacks=[callbacks])

The Loss and Accuracy thresholds can be estimated after a trial run of the model by monitoring the validation/training error graph.


What are some examples of everyday systems that use AI?

A good, recent, and accessible book which includes many case studies is Prediction Machines. Check it out for more details than I can provide in this answer.

Example applications are all around us, but one of the problems with recognizing them is that the bar for what we call AI is constantly being raised.

Consider that a few decades ago, directions from google maps would certainly be recognized as AI, whereas now most laypeople wouldn't make that association.

Some other commonplace examples:

Amazon.com can guess at what you'd like to buy next.
Facebook and many other companies have programs that decide which ads to show you, without human intervention.
Tesla's autopilot feature will drive your car down a highway.
Google search can instantly return the results you were looking for, even when you enter strange or ambiguous search terms.
Virtual Assistants from many companies, like Alexa recognize your voice, figure out what words you're saying, and then figure out what you want them to do.
StackOverflow determines which questions are hot, and which ones need moderator attention.

There are are also a lot of less obvious examples. I'll pick a few from my research area:

When you spot a sky marshal on a plane, or a dog patrol in the airport it's because an AI system put them there. Some subway systems now use the same technology for fare checkers.
When people trade agricultural products in Uganda, it's an AI system and AI techniques at work behind the scenes.
When you're assigned a donor for a kidney transplant (and now other organs too), an AI system did the heavy lifting in that decision.

Of course, there's lots more going on too, but those should give you some suggestions for everyday conversation.


Relationship between input range and channel means, standard deviations for CNNs

Standardization of the pixel values would bring it mean close to 0 and standard deviation as 1.
Normalization will squeeze the values between 0 and 1.
For RGB images, divide each channel i.e each pixel value by 255.
For reference regarding Standardization see here and for normalization see here.


Is a switch from Java to Python worth it?

While dyedgreen is right in some respects, I don't agree entirely with that sentiment. Sure, you can theoretically use any language as long as you know the maths and understand the concepts inside and out whilst having some applicable knowledge.

However, I don't believe if you are starting from scratch, you should learn to develop models in Java. While the underlying material is the same, all you are doing is wasting time IMO learning a language that is rarely used in the field and lacks sufficient support by most modern ML frameworks. Also, while this is rather subjective, python feels better when used as a data science language. Working with data especially is where that difference is clear to me.

That being said, this could all change, and Java could become the ML language of choice(extremely unlikely).


Is there any classifier that works best in general for NLP based projects?

The accuracy depends on various factors. Might not always be the algorithm. For example a cleaner data with a poor algorithm might still give better results and vice versa.

What are the preprocessing techniques you are using? This preprocessing techniques article is a good starting point for html data. And by vectorising I assume you mean word2vec, use a pre-trained word2vec model. Like google's word2vec model it's trained on a lot of data(about 100 billion words).

LSTM performs good whenever the intent of the sentence is important. Check out this.
Ram hit Vijay and Vijay hit Ram, might mean the same to most algorithms. Example, Naive Bayes.


Using machine learning to identify CAD model features

This is a classification problem. Look at some common classification methods. This article by Microsoft gives some pretty good guidance.

https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice


Graph Neural Networks

The term Graph Neural Network, in its broadest sense, refers to any Neural Network designed to take graph structured data as its input:

  To cover a broader range of methods, this survey considers GNNs as all deep learning approaches for graph data.

A Comprehensive Survey on Graph Neural Networks, Wu et al (2019)

However the original paper to propose the term specifically referred to recursive neural networks (RNN) adapted to take graph-structured data as their input:

  This paper presents a new neural model, called graph neural network (GNN), capable of directly processing graphs. GNNs extends recursive neural networks and can be applied on most of the practically useful kinds of graphs, including directed, undirected, labelled and cyclic graphs.

A new model for learning in graph domains (2005)

Subtypes

Note, Wu et al propose a taxonomy dividing GNN's into four subgroups:

Recurrent graph neural networks (RecGNN)
Convolutional graph neural networks (ConvGNN)
Graph autoencoders (GAE)
Spatial-temporal graph neural networks (STGNN)

ConvGNN's can themselves be classified by whether they use Spectral methods or Spatial methods, and GAE's by whether they are designed for Network embedding or Graph generation.


Are there tools to help labelling images?

There are a few tools that you can use to annotate (or label) data. For example, labelme or Labelbox. Have a look at this question for more alternatives.


How to deal with invalid output in a policy network?

You should have a method to generate a possible moves output based on the board state. Use this as a mask before normalization in the policy head.


How to handle classification with label updates?

Doing some unsupervised learning will give you some divisions within each label but you can't be sure it will be split on "E-sports" and "Sports". It might as wel split sports into "land sports" and "water sports" or "long sports" and "short sports".

The only way to reliably split "sports" in 2 subsections of your desire is to relabel this manualy I am afraid.


speech comment detection by deep speech mozilla for data set

The problem you state is a well known problem, and it is called "keyword spotting" os KWS. If you add a wake up word before it (like "hey google/siri"), you can also use "voice command" system to alleviate the problem.
There are two kind of KWS systems: those which develop to detect a hard coded set of keywords, and those who are flexible to amend the keyword set. As you state that you trained an acoustic model using deep speech, it looks that you choose the latter approach.
If you trained a deep speech model, you done the easy half of the solution. The other half which is more tricky is to develop a search algorithm based on your acoustic model. I think wfst based algorithms and prefix beam search algorithms are what you should be looking for.
Regards


When will an artificial intelligence replace a software developer or tester?

The ultimate goal of machine learning is to bypass the developer...
When we will have a "master algorithm" that can learn how to generalize any function or algorithm from examples, it can essentially replace any developer, skip the 'development" stage, going from problem directly to algorithm.

We can't know when this will happen, but as we surrounded with multiple creatures (humans and animals) which can demonstrate learning algorithms and predictive model learning without any "developer" - we can assume that such an algorithm is possible.

If I would have to guess, I would say that we are probably very near the point where "developers" and "testers" will be replaced by learning algorithms. We could be a decade or two away from the point where people will not write any code or any testing at all. Programs and automation will be derived directly from describing the problems themselves in a natural language, visualizations or data collections. However, we still need some breakthroughs in combining feature learning with active memory, unsupervised learning, and artificial common sense.


Can agent based machine learning achieved with any ML algorithms other than neural network?

Neural networks are not inherently part of reinforcement learning (a popular agent-based framework for describing control problems). In general, if you have an agent-based scenario, you are trying to optimise a function:

Policy( State ) -> Action

Where State can be any combination of current observations and history that seem relevant to the problem. The optimisation is usually over some measure of success, achieving goals etc. Reinforcement learning formalises these terms using Markov Decision Processes, which is a very general and successful approach, but not the only one.

Finding the best policy does not require neural networks. There are a lot of basic reinforcement learning algorithms that are defined without reference to NNs. For instance, Q-learning does not need neural networks, they are an optional extra. In addition, you do not need to even use RL - you can try to search for a suitable Policy function directly using a method such as Genetic Algorithms.

Focusing more on RL methods, the usual approach without any kind of function approximator is to just store an estimate of the value of each state or state/action pair. This value is a measure of long term reward, and has a formal definition in RL. Often called the tabular approach , because it is just a table of states and their estimated values, it works for small problems just fine. For example you can train an agent to play tic tac toe, find an optimal path that moves over a grid, or discover when to hold and when to twist in a simplified blackjack game.

However, problems start to occur with the most basic methods when the state space becomes large, which happens very easily. Tabular methods are OK up to a million or maybe ten million states (depending on how easy it is to take trial actions). These are enumerated states, so typically adding a single dimension to a problem multiplies the number of possible states by the number of options in that new dimension. So after a certain level of complexity, a simple table is not good enough, and you need a function approximator. Function approximators for RL ideally have the following properties:

Can generalise from examples.
Can be updated online as new data arrives, progressively forgetting old data.
Can be differentiated with respect to their learning parameters.

Neural networks fit the bill. Some other popular supervised learning techniques do not - for instance Random Forests cannot usually be trained online.

In fact linear regression does work, and it is a very simple approach. For some problems, linear regression works very well in combination with reinforcement learning. You may need to do some careful feature engineering, and it is not always possible for more complex problems, but if you have a "medium-sized" agent-based problem - simpler than learning Go or playing video games - and you want to avoid using neural networks, then you should be able to attempt to train an agent using a combination of RL, such as Q-learning, plus linear regression.

Both Q-learning and linear regression have plenty of examples available online, and can be implemented from scratch using basic Python frameworks such as NumPy.


How does artificial intelligence work in games?

There are many different kinds of AI used in games; AI for historical board games (like chess or Go) tends to be much better than AI for computer games (such as Starcraft or Civilization), in large part because there's more academic interest in developing strategies for those games.

The basic structure of a game-playing AI is that it takes in game state inputs and outputs an action; typically, the internals also contain some sort of goal and some sort of future prediction.

But beyond that, there's tremendous amounts of variability. Some AI are little more than scripted reflexes, some are built like control systems, some do actual optimization and forward thinking.

Getting into the details of how the many different approaches work is probably beyond the scope of this site, though.


Why would adding all the possible embeddings be "worse" than using 1D-convolutions?

N-grams are defined as sets of n contiguous words. We use n-grams because they are more useful than random combinations of words across the sentence. Intuitively, combinations of nearby words have more semantic meaning than combinations of distant words.
Also, using all possible combinations of n embeddings would take much longer, especially since (1D) convolutions are such efficient operations.


How to build my own dataset and model for an LSTM neural network

why do you want to use LSTM network?
lstm is a variant of a recurrent neural network , recurrent neural networks are used for "sequential" tasks , i.e the dataset should have some sequential structure , like poems , songs etc.

Your model should be a simple classifier , once you fit a simple classifier , like a decision forest on the categories dataset  products ,authors etc , you will have a  model that will predict the class based on these attributes then you can say from the decision boundaries of the model what values you must have . if the relationship between the attributes is even more simpler you could try plotting distribution plots.


Does a quantum computer resolve the halting problem and would that advance strong AI?

It depends a bit on what you mean by 'quantum computer'. The 'conventional' notion is that quantum computation buys a (in some cases, exponential) speedup - it doesn't change what can be computed, just how quickly.

In contrast, advocates of hypercomputation claim that quantum effects may make it possible to do infinite computations in finite time. Note, however, that this is not a mainstream belief - the reknowned logician Martin Davis has written an article claiming that hypercomputation is a myth.

Roger Penrose has also claimed that quantum vibrations in neural microtubules may be responsible for consciousness.


Intutitive explanation of why Experience Replay is used in a Deep Q Network?

It is the neural network approximation that suffers, when it attempts to learn from correlated data. Intuitively, this is because the learning algorithm takes gradient steps assuming that the examples it are shown are representative of the dataset as a whole. A neural network update step uses a mini-batch of examples to calculate the gradient of its weights and biases with respect to a cost function. If that mini-batch is not fairly sampled, then the expected value of the loss and of the gradient will not be representative of the population as a whole. They will be biased and can cause an update step to parameters in the wrong direction.

In addition, if you are using a bootstrapping method - any form of TD learning - then the value estimates used to set learning update targets are sensitive to bias in the estimator. This is already something that can cause instability due to positive feedback loops. Adding another source of systemic bias from correlated input data can only make it worse.

You can gain some experience for the effect of this with a simple non-RL experiment.

Goal: To approximate the function $y = x^2$ in the range $-2 < x < 2$. These numbers chosen to make the task simple for a neural network.

Setup: Generate training data in the form $x_i, y_i$ for a few thousand sample points with true values of the function. Optionally add some noise to make the task harder. Keep the data set ordered by $x_i$ values. Create a simple neural network for regression (e.g. 2 hidden layers with 50 neurons and tanh activation), and set it up with a simple optimiser (e.g. SGD)

Run: Train the network twice, using small minibatches (e.g. size 10). Once without shuffling the data on each epoch (or using any shuffling algorithm on the minibatches), and once with standard shuffling and assignment to minibatches. Plot a learning curve of loss vs epoch for each run.

You should find that the ordered, non-shuffled version learns much slower than when using some randomisation to decorrelate the data. Depending on precise hyperparameters, the ordered version may even oscillate with loss quite far away from converging.

It is this effect, or a more subtle version of it, that impacts RL combined with neural networks when learning direct from trajectories. In addition, there is existing starting bias in bootstrap methods, which this effect makes worse.


Correcting 'bad' translations in a sequence-to-sequence neural machine translation model

The basic process you're describing sounds a lot like Boosting, which is also well covered in Chapter 18 of Russell & Norvig, or like active learning.

You're correct that training the model with emphasis on a specific subset of the data is likely to lead to mistakes somewhere else. Boosting gets around this in a clever way.

It begins by training a model as normal.
The new model's performance on the training set is measured.
A new dataset is generated, where examples that were misclassified by the previous model are more highly weighted (this is intuitively similar to adding an extra copy of them to the dataset).
A new model is trained on the weighted dataset. The model suffers a greater penalty for making the same kinds of errors as the previous model(s), and so tends to make different errors.
Steps 2-4 are repeated for some time.
An ensemble of models is output, rather than a single model.

To classify new points, the most common score from all ensemble members is used.

I'm less familiar with NLP work using this approach, but it seems like the same basic idea could be used fruitfully here: the notion is to train many models that make independent errors, and then trust their collective decisions.


Where can I find the original paper that introduced RNNs?

The two tech reports below both call RNNs explicitly "recurrent net(work)s".

Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.

Jordan, Michael I. (May 1986). Serial order: a parallel distributed processing approach. Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, University of California.

Jordan was a student of Rumelhart, so I would lean on identifying 1 as the paper introducing RNNs, with the caveat that the first sentence in the section "Recurrent Nets" of 1 reads:

We have thus far restricted ourselves to feedforward nets. This may
seem like a substantial restriction, but as Minsky and Papert point
out, there is, for every recurrent network, a feedforward network with
identical behavior (over a finite period of time).

This is interesting for two reasons:

After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.
The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.

Unfortunately, I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.


Neural network design when amount of input neurons vary

The best option in your case would probably be zero padding or padding up. This is simply zeroing out inputs for cases in which there is no data. It's done a lot on the borders of images for CNNs.

That or you could just use a RNN which can handle your variable length inputs with ease.


The main difference between a variational auto-encoder (VAE) and an auto-encoder is that the VAE is a generative and statistical model  while an auto-encoder (AE) is just a data compressor and decompressor (it is just a function approximator), so an AE is not a statistical model. They both have an encoder and a decoder and they both convert the inputs to a latent representation, but their inner workings are different and their purpose is also slightly different. An AE could be considered a generative model, in the sense that it generates objects that are similar to the input (but they might not be the same as the input). However, it is not a generative model according to the usual statistical definition of a generative model.

In the case of an AE, given an input $x$ (e.g. an image), the encoder produces one latent vector $z_x$, which can be decoded into $x'$ (another image which should be similar or related to $x$). Compactly, this can be presented as $x'=f(z_x=g(x))$, where $g$ is the encoder and $f$ is the decoder. This operation is deterministic, in the sense that, given the same $x$, the same $z_x$ and $x'$ are produced.

In the case of a VAE, given an input $x \in \mathcal{X}$ (e.g. an image), more than one latent vector, $z_{x}^i \in \mathcal{Z}$, can be produced, because the encoder part of the VAE models the inputs as a probability distribution. In other words, the encoder of a VAE assumes that the inputs $\mathcal{X}$ are drawn from a probability distribution, which can be denoted by $\mathcal{X} \sim q$, where $q$ is some computable probability distribution (e.g. a Gaussian distribution with mean zero, $\mu =0$, and a finite standard deviation, $\sigma$). $q$ is defined over $\mathcal{Z}$ (the space of possible latent vectors). More specifically, $q$ is a conditional probability distribution, which can be denoted by $q(z \mid x)$.

So, the encoder of a VAE, while being trained, creates a probability distribution, which, in practice, can be represented by neural networks, that is, you can train a neural network to represent the parameters of e.g. a Gaussian distribution (which is defined by two parameters: the mean and standard deviation). You can thus more precisely denote $q$ by $q_\phi$, where $\phi$ are the parameters of a neural network that e.g. output two numbers: the mean and a standard deviation of $q$.

Given a probability distribution $q_\phi$, we can sample several $z_{x}^i$ from $q_\phi$. Why is this useful? Given the same input $x$ (e.g. a graphical representation of a molecule $m$), we can sample e.g. $N$ latent vectors $z_{x}^1, z_{x}^2, \dots, z_{x}^N$, which can all be decoded (e.g. into $K$ new molecules). Why would this be useful? For example, in the case you need to generate molecules which have a similar structure to the input molecule $m$. Why would you need this? In some cases, molecules with a similar structure can have similar effects, but could e.g. be more easy or inexpensive to synthesise.

Why can we generate $K$ new molecules out of $N$ latent vectors, where $K$ can be greater than $N$? We have not yet mentioned the details of a decoder of a VAE. The decoder of a VAE also models its inputs, $\mathcal{Z}$, as a probability distribution defined over $\mathcal{X}$, which can be denoted by $p_\theta(x \mid z)$, where $z$ is a latent vector. In other words, given the $i$th latent vector sampled from the latent distribution (the distribution of the encoder), $z_{x}^i$, we can sample more than one reconstructed input from the probability distribution of the decoder. Hence, a VAE often constructs two probability distributions: one that is associated with the encoder, $q_\phi(z \mid x)$, and one that is associated with the decoder, $p_\theta(x \mid z)$.

Given that VAEs create these distributions, they are considered statistical models. An auto-encoder has no notion of a probability distribution. An AE only maps inputs to outputs deterministically, so it is a function approximator (given that it approximates a function that compresses the input and then another function that decompresses it). VAEs are thus a lot more flexible than AEs.


Why is second-order backpropagation useful?

Second-order optimization algorithms like Hessian optimization have more information on the curvature of the loss function, so converge much, much faster than first-order optimization algorithms like gradient descent. I remember reading somewhere that if you have $n$ weights in the neural network, one iteration of a second-order optimization algorithm will reduce the loss function at approximately the same rate as $n$ iterations of a standard first-order optimization algorithm. However, with recent advancements to gradient descent (momentum, adaptive rates, etc), the difference isn't as large anymore -- @EmmanuelMess pointed out a paper that states:

The performance of the proposed first order and second order methods with adaptive gain (BP-AG, CGFR-AG, BFGS-AG) with standard second order methods without gain (BP, CGFR, BFGS) in terms of speed of convergence evaluated in the number of epochs and CPU time. Based on some simulation results, it’s showed that the proposed algorithm had shown improvements in the convergence rate with 40% faster than other standard algorithms without losing their accuracy.

Here is a great post explaining the background behind the math of why this is the case.
Also, second-order gradients can help the optimizer identify states like saddle points, and help the solver get out of those states. Saddle points give standard gradient descent a lot of issues, as the gradient descent has difficulty and is slow to move out of the saddle point. Fixing the saddle point issue is one of the motivations for improving gradient descent over the last two decades (SDG with momentum, adaptive learning rates, ADAM, etc). More info.

The issue though is that to compute the second-order derivative requires a matrix $n^2$ in size, as opposed to gradient descent which requires a matrix $n$ in size. The memory and computation become intractable for large networks, especially if you have millions of weights.
Some approaches exist which efficiently approximates the second-order optimizations, solving the tractability problem. A popular one is L-BFGS. I haven't played around with it much, but I believe L-BFGS is not as popular as gradient descent algorithms (such as SGD-M, ADAM) because it is still very memory demanding (requires storing about 20-100 previous gradient evaluations), and cannot work in a stochastic context (you cannot sample mini-batches to train on; you must train the entire dataset in one pass per iteration). If those two are not an issue for you, then L-BFGS works pretty well I believe.


Can cryptocurrency charts be estimated using neural networks?

There are some academic works on that, but basically what you're asking for is a time-series estimation.
Here's a paper that tries to estimate Bitcoin price, for example: PAKDD: Forecasting Bitcoin Price with Graph Chainlets.


What are some books or state of the art papers about the development of a strong-AI?

There is actually a book called Artificial General Intelligence by  Ben Goertzel and Cassio Pennachin. It's a bit out of date (from 2008), and published as a Springer-Verlag monograph (which tends to have fairly low editorial standards). This one is also an anthology, with each chapter written by a different author. It's probably not suitable as an undergraduate level book, but it does seem to contain something like the information that's wanted.


What loss function to use when labels are probabilities?

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.
You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,
$$H(p,q)=-\sum_{x\in X} p(x) \log q(x).$$
$ $
Note that one-hot labels would mean that
$$
p(x) =
\begin{cases}
1 & \text{if }x \text{ is the true label}\\
0 & \text{otherwise}
\end{cases}$$
which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:
$$H(p,q) = -\log q(x_{label})$$


In which cases is the categorical cross-entropy better than the mean squared error?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number).  However, in principle, you can use the MSE for classification problems too (even though that may not be a good idea). MSE can be preceded by the sigmoid function, which outputs a number $p \in [0, 1]$, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is $1 - p$.

Similarly, cross-entropy (CE) is mainly used for classification problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for binary and multi-class classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a softmax function (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid.

See also Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model? for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also How to choose cross-entropy loss in TensorFlow?, which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions? and Does the cross-entropy cost make sense in the context of regression?.


What algorithm to use to classify data by spatial relations?

The spatial relationships that you describe would correspond to features, and it's not clear that you need to use a neural network for detecting or discovering these features since you have just described them. Could you instead define a feature extractor that detects the correct patterns and returns you a vector of counts of feature occurrences across the image?
In fact gray level co-occurrance matrices might be adaptable to this problem. Since there are only four colours, each matrix would be very small, and you would define one matrix for each local spatial configuration. For example, one matrix might encode the number of times that colour x is below colour y. From these you could then extract counts of interest to use for further downstream processing.


How would one go about generating *sensible* responses to chat?

This seems like a problem for the use of an encoder-decoder pair such as those seen in text summarization (see this paper by Rush et al.: https://arxiv.org/pdf/1509.00685.pdfï%C2%BC).

You would need the following layers:

LSTM layer to encode the given input text into an embedding
LSTM layer the looks over the currently generated output to encode the text into an embedding
A dense soft-max layer for generating words probabilistically based on the output of the two contextual LSTM encoders

Please see the following blog post by Jason Brownlee that outlines this approach and others, while giving implementation details and snippets (https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/)!

Also note that this would require a large set of training examples of input text and reasonable responses. You might be able to scrape Reddit post responses and comments off of those for a start? Let me know if I misunderstood the question.


Is there a full and precise formulation of Theorem 1 in the Integrated Gradients paper?

After working on it for a while this is what I got.
Concerning proposition 1 in the paper, a rigorous statement could be the following version of the Gradient Theorem for line integrals:
Proposition 1. (Gradient Theorem for Lipschitz Continuous Functions).
Let $U$ be an open subset of $\mathbb{R}^n$.  If
$F : U \to \mathbb{R}$ is Lipschitz continuous, and
$\gamma : [0,1] \to U$ is a smooth path such that $F$ is
differentiable at $\gamma(t)$ for almost every $t\in [0,1]$, then
$$
    \int_{\gamma} \nabla F(\mathbf{x}) \cdot d\mathbf{x} = F(\gamma(1)) - F(\gamma(0))
    \,.
  $$
I found theorem 1 about symmetry-preserving harder to state rigurously while still fully capturing its intended meaning - this is the best I got:
Theorem 1.
Given $i,j\in \{1,\dots,n\}$, $i\neq j$, real numbers $a<b$, and a
strictly monotonic smooth path $\gamma : [0,1] \to (a,b)^n$
such that $\gamma_i(0)=\gamma_j(0)$ and $\gamma_i(1)=\gamma_j(1)$,
then the following statements are equivalent:
(1) For every $t\in [0,1]$, $\gamma_i(t) = \gamma_j(t)$.
(2) For every function $F : [a,b]^n \to \mathbb{R}$ symmetric
in $x_i$ and $x_j$ and verifying the premises of
proposition 1 with $U=(a,b)^n$ we have
$\int_{\gamma} \frac{\partial F(\mathbf{x})}{\partial x_i} \, dx_i
= \int_{\gamma} \frac{\partial F(\mathbf{x})}{\partial x_j} \,
dx_j$.
Further details and proofs: Symmetry-Preserving Paths in Integrated Gradients


Diagnosing mental health problems using Artificial Intelligence / Machine Learning techniques

There's a strong sentiment towards the idea that medical diagnosis is largely Abductive Reasoning. See this presentation for additional details.  One approach to automated abductive reasoning is parsimonious covering theory.

If you want a relatively in-depth look at all of this, check out the book Abductive Inference Models for Diagnostic Problem-Solving .

Expert systems have also been shown to work very well for medical diagnosis. As far back as the 1970's, systems like MYCIN could beat human experts in terms of diagnosis and treatment plans.

From what I can tell, there doesn't seem to be any reason in principle to think that AI/ML can't be used across the board for medical diagnosis, mental health or otherwise.


Can one use an Artificial Neural Network to determine the size of an object in a photograph?

Yes: Learning Depth from Single Monocular Images

In the end, depth is just one special form of size.

Of course, you need something partially known, e.g. another car. You don't need to know the exact size of the car, but you know which size cars in general have. If you have an image without any reference, it is impossible.


Why is it useful to track loss while model is being trained?

The loss function (aka cost function) measures the correctness of the predictions of the model.

For example, a simple cost function could be $|y - f(x)|$, where

$\hat{y} = f(x)$ is the prediction of the model $f$ when the input is $x$,
$y$ is the ground-truth label for input $x$ (i.e. what the model is supposed to output when the input is $x$), and
$|\cdot|$ is the absolute value

The loss function is also used to understand whether a model is overfitting (and underfitting) or not. More specifically, if the loss of the model evaluated on the training data (the training loss) keeps decreasing while the loss of the model evaluated on the validation data (validation loss) starts to increase, that's a good sign that the model is overfitting (i.e. just memorizing the training data, but it will likely perform poorly on non-training data). If the training loss does not decrease, that's a good sign of underfitting (i.e. the model is not capable of learning the patterns in the training data), so you may need to use a model with a bigger capacity, change the loss, or do something else.

The loss is typically not directly used as a measure of the performance of the model because the loss doesn't directly represent the performance of the model. In fact, the loss can be relatively big (with respect to another model's loss) and the model still performs well, but typically a lower loss corresponds to higher performance. The performance of the model is calculated differently depending on the task, model, and your goals. The most common performance measure is the accuracy (the number of correct predictions over the total number of predictions), but there are many other performance measures, such as the precision, recall, f1 score, AUC, etc., that emphasize different behavior that you expect from your model.


Is elitism preferred over non-elitism in the cross-over operator?

First of all the answer to your question is largely dependent on the problem you are trying solve, the size of your population, the size of your problem's search space and the rest of your GA's hyper-parameters such as your mutation rate.

If the problem has a large search space, then applying the elites strategy you described above will most likely cause your algorithm to never reach the optimal solution or reach it after too many iterations. The reason for this being that if your search space is big, then since the initialization of your initial population is random, the chances is that the "fittest" individual(s) in the first few iterations are not necessarily the gonna lead you to the optimal solution. Hence applying the elites strategy too early might make you get stuck on a local optima forever or for too long. The way I have personally found to be effective for this kind of situations is to allow your algorithm to explore as large a search space as possible earlier on(i.e use "all the population present in the pool") and then gradually introduce the elite strategy you described above(this is an idea similar to momentum in neural networks). In short, you want to explore earlier on, and then when you gradually start exploiting what you have found.
If the search space is relatively small, I think the elites strategy would be a very good idea. What you might also try to do, if you really want to stick to using elitism, you can try to mutate the elite individual(this kinda allows you to explore a little bit), or choose a higher elitism rate(i.e the number of individuals to be elite).

Now to answer you other questions:

  Is there any certain belief that cross-over on only elites of the population, converges the solutions faster?

No, but I don't have reference for this so I guess I speak under correction, but I will explain why I said no:

The problem with pure elitism is that it causes the genetic algorithm to converge to local maxima instead of the global maxima. Basically pure elitism is just a race to the nearest local maxima and and as soon as you get to the local maxima, if you continue with elitism, you get almost no improved from there.

  I guess in order to escape from local minima, cross-over on all the population is needed; on the other hand I also say why performing cross over on weak population?

"why perform cross-over on weaker individuals in the population?" - to explore. The hope is that applying a cross-over or a mutation on a weaker individual will produce a fitter individual.


Is it possible for visual systems in AI to have Inattentional Blindness?

Presumably what happens to people in the famous Invisible Gorilla experiment, is that an incongruous object is simply filtered out of human perception.

If we wish to interpret this mechanistically, we could hypothesize that a 'gorilla object' is simply not presented by low levels of perception to our higher level pattern recognizers because the lower levels are not biased towards the construction of 'gorilla-like' features in such a context.

The recent Tesla fatality (arising from a failure to distinguish between the sky and a high-sided white truck) could conceivably be considered to be an example of this.

See this AI SE question.


Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.


How good is AI at generating new, unseen [visual] examples?

We are getting pretty good at image generation, some examples:

Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015). https://arxiv.org/pdf/1511.06434.pdf
Gregor, Karol, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. "DRAW: A recurrent neural network for image generation." arXiv preprint arXiv:1502.04623 (2015). https://arxiv.org/pdf/1502.04623.pdf

From (1):

Then there is another research direction around  evolutionary algorithms, for example:

Sims, Karl. "Evolving virtual creatures." In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pp. 15-22. ACM, 1994. https://scholar.google.com/scholar?cluster=6031059536657676358&hl=en&as_sdt=0,22 ; https://www.youtube.com/watch?v=bBt0imn77Zg


It appears to not be published yet; the best available online are these slides for this talk. (Several people reference an earlier talk with this link, but sadly it's broken at time of writing this answer.)

My impression is that it's an attempt to formalize and abstract the creation of subnetworks inside a neural network. That is, if you look at a standard neural network, layers are fully connected (that is, every neuron in layer 1 has access to every neuron in layer 0, and is itself accessed by every neuron in layer 2). But this isn't obviously useful; one might instead have, say, n parallel stacks of layers (the 'capsules') that each specializes on some separate task (which may itself require more than one layer to complete successfully).

If I'm imagining its results correctly, this more sophisticated graph topology seems like something that could easily increase both the effectiveness and the interpretability of the resulting network.


Are these two definitions of the state-action value function equivalent?

These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation.

The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by $\theta$ (i.e. the policy could be represented e.g. by a neural network), while, in the first case, the policy is not parametrized, which is a more general way of writing the definition of the state-action value function, because you do not assume how you will represent the policy.

The second apparent difference (as you have noticed) is that, in the first case, the expectation is taken with respect to transition probability function, while, in the second case, it seems to be taken with respect to the parametrized policy. In the second case, you do not probably know the transition probability function, so you will compute your state-action value function based on your current best estimate of the policy.

Now, as Brale pointed out in the comments (please, read the comments under this answer!), the definition of the state-action value function is always the same, even if you don't know the MDP (because that's a definition and it doesn't matter if you know the transition probability or not). However, in the case of David Silver's slides, by that notation, I think he means he will compute the state-action value function based on the current estimate of the policy.


what are the similarity measure use for both continuous and categorical data?

Sometimes continuous data can be represented as parametric distribution with distribution parameters as variable, essentially continuous stochastic process. In that case cross-entropy would work on that type of continuous data.


Are there any active areas of research in machine learning that do not involve neural networks at all?

If you look into the top conferences on machine learning and neural networks, such as NeurIPS, ICLR, and ICML, you will find many papers related to neural networks and deep learning, given that these are still very hot/promising topics. However, occasionally, you will find accepted papers that do not involve neural networks. Here's a small list of them that I've found after a quick search.

Near-Tight Margin-Based Generalization Bounds for Support Vector Machines (ICML, 2020, pdf) by Allan Grønlund et al.
Implicit Regularization of Random Feature Models (ICML, 2020, pdf) by Arthur Jacot et al.
Quantum Expectation-Maximization for Gaussian mixture models (ICML, 2020, pdf) by Alessandro Luongo et al.
From PAC to Instance-Optimal Sample Complexity in the Plackett-Luce Model (ICML, 2020, pdf) by Aadirupa Saha et al.

So, yes, research in machine learning is not exclusively devoted to neural networks and deep learning. You can probably find more papers here or in the proceedings of similar conferences or journals.


Is reinforcement learning possible for chatbots?

The problem is indeed that the 'rules' of conversations are not as fixed as the rules for games. However, you can make use of  descriptive formalism from Discourse Analysis, called adjacency pairs. These describe regularities between utterances on a local level, for example greeting/reply, which would match your "Hi, I'm Alice" and "Nice to meet you".

You will need to be able to classify utterances by your chat bot according to a set of possible responses, and then you can see if a valid response is produced for any given utterance. If the user asks a question, then a greeting will not be a good answer, but a statement could be, if it was a response to the question. This is leaving aside the content and focuses merely on the formal characteristics of the utterance.

If you want to know more about the topic, have a look at Conversation Analysis, which is the linguistic field dealing with the subject.


As with another answer, I am also skeptical of the distinctions made in the DIKW pyramid.
Nonetheless, a very popular machine learning approach for answering 'Why?' questions is the application of Bayesian reasoning: given a causal data model, reverse inference can be used to find the probability distribution of events which lead to a given outcome.
It could be argued that defining 'cause' in terms of distributions rather than specific concrete mechanisms is a rather limited notion of 'Why?'.
However, it may be that there are some forms of causality that we don't know how to represent, specifically 'first-hand experience'. Indeed, common usage of the term 'wisdom' generally refers to first-hand experience, rather than information gained from some other source.
The idea is that knowledge can be expressed declaratively, whereas wisdom must be derived from experience.
For an AI represented as a computer program, the distinction between declarative and first-hand experience might appear irrelevant, since in principle any experience can be encoded and made available without the program having to 'experience' it first-hand.
However, the following humorous definition of `wisdom' might perhaps shed some light on a distinction that's pertinent to AI research:

Knowledge is knowing that a tomato is a fruit.
Wisdom is knowing that you shouldn't eat it with custard.

This notion of 'Wisdom' could be said to require qualia. It is the subject of much debate whether qualia exist and/or are necessary for consciousness - see for example the thought experiment of 'The Black and White Room'.
So the notion is that there is a distinction between having a Bayesian network representation of wisdom that says: "It is 99.7% likely that putting a tomato in custard is undesirable" and the first-hand experience to the effect that it tastes odd with custard.


How do I encrypt and decrypt my model when I run inference on it?

While @Oliver Mason's comment is correct, and your proposed method won't provide perfect security, you can still protect your models at rest, so that they are stored encrypted in the memory, and your software feed the key at runtime to decrypt it.
On whatever DL inference engine that you have, once it supports loading the model from a buffer (e.g. void*) rather than a file path, you can do the following: read the encrypted model, decrypt it into a buffer, and initialize your neural network model from the decrypted buffer. OpenVino supports loading the model from a buffer.
For encrypt / decrypt the model, any framework such as OpenSSL, or even tiny-AES, can work.
Mentioning again, how to store and use the key is something that should be handled carefully, and a user with sufficient knowledge can read the model and the keys at runtime from the application's memory.


Hand computing feed forward and back propagation of neural network

One important point I missed in first review: error is a summatory, its derivative is also a summatory.

About offsets "b": usually they are different in each cell (if not fixed to some value, as 0). Thus, replace them by b1 and b2. Moreover, they should be optimized in the same way that the weights.


How can the derivative of a neural network be calculated, given no mathematical expression?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?

Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation.

First, take the output of all the neurons (except the one you're about to differentiate by) as a function:

The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.

You then differentiate NN by a single neuron (n) as shown:

$$\frac{\partial NN}{\partial n} = \lim_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$

In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):

$$\frac{\partial A}{\partial n}$$

So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.

I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.


Is there a way to provide multiple masks to BERT in MLM task?

I've found the answer in the original BERT git repo

***** New May 31st, 2019: Whole Word Masking Models *****
This is a release of several new models which were the result of an
improvement the pre-processing code.
In the original pre-processing code, we randomly select WordPiece
tokens to mask. For example:

Input Text:
the man jumped up , put his basket on phil ##am ##mon ' s
head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
phil [MASK] ##mon ' s head

The new technique is called Whole Word Masking. In this case, we
always mask all of the the tokens corresponding to a word at once. The
overall masking rate remains the same.

Whole Word Masked Input:
the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head

The training is identical -- we still predict each masked WordPiece
token independently. The improvement comes from the fact that the
original prediction task was too 'easy' for words that had been split
into multiple WordPieces.
This can be enabled during data generation by passing the flag

--do_whole_word_mask=True

to create_pretraining_data.py.


How feasible is it to perform pose estimation on a Raspberry Pi 4 using a Pi-Cam?

If using TFLite python API, you may reach to 1/2 FPS. On downscaled custom Pose architectures you may get more FPS. You can try the TFLite c++ API as well, it might be a bit faster than the python one.
To reach more than this you would need a TPU or a library that could use GPU computation of the Pi. Nevertheless, I doubt that it would reach 15 FPS with OpenPose open-source library.


Why does this paper say that the Nash-equilibrium of GAN is given by a discriminator which is 0 everywhere on the data distribution?

What this paper is not saying is that the discriminator, $D_{\phi}$, always returns a scalar value of zero. What they are saying is that the generator, $G_{\theta}(z)$, has accurately learned the distribution of the input data, and that the discriminator produces the correct answer for each input from the generator. It's a mathematical description of the global optimum for a GAN. This Medium article by Jonathan Hui talks more about Nash equilibria, the Kullback-Leibler, and more.