What is the difference between graph semi-supervised learning and normal semi-supervised learning?

The authors of your cited paper use the term graph-based semi-supervised learning (G-SSL) to refer to semi-supervised learning techniques which take graph structured data as their input.

Given their main example, the MNIST dataset, is not graph structured, they detail a method for converting the raw Euclidean data $X$ into said form (represented by its adjacency matrix $S$), and then compute the Laplacian $L$ of this graph:

  We consider the graph-based semi-supervised learning (G-SSL) problem. The input include labeled data $X_{l} ∈ \mathbb{R}^{n_{l}×d}$ and unlabeled data $X_{u} \in \mathbb{R}^{n_{u}×d}$, we define the whole features $X = [X_{l}; X_{u}]$. Denoting the labels of $X_{l}$ as $y_{l}$, our goal is to predict the labels of test data $y_{u}$. The learner applies algorithm $A$ to predict $y_{u}$ from available data $\{X_{l}, y_{l}, X_{u}\}$. Here we restrict $A$ to label propagation method, where we first generate a graph with adjacency matrix $S$ from Gaussian kernel:
  $S_{ij} = \exp(−γ\lVert x_i − x_j\rVert ^{2})$, where the subscripts $x_{i(j)}$ represents the $i(j)$-th row of $X$. Then the graph Laplacian is calculated by $L = D − S$, where $D = \text{diag}\{\sum_{k=1}^{n} S_{ik}\}$ is the degree matrix.

A Unified Framework for Data Poisoning Attack to
Graph-based Semi-supervised Learning (2019)

This is consistent with the terminology as used in other literature:

  Semi-supervised learning for node-level classification.
  Given a single network with partial nodes being labeled and others remaining unlabeled, ConvGNNs can learn a robust model that effectively identifies the class labels for the unlabeled nodes [22]. To this end, an end-to-end framework can be built by stacking a couple of graph convolutional layers followed by a softmax layer for multi-class classification.

A Comprehensive Survey on Graph Neural Networks (2019)


How can the EQ of an AI program be measured?

The answer to your question is "In principle, yes" - in it's most general form, EQ testing is just a specific case of the Turing test ("How would you feel about ... ?").

To see why meaningful EQ tests might be difficult to achieve, consider the following two possible tests:

At one extreme of complexity, the film 'Blade Runner' famously shows a test to distinguish between human and android on the basis of responses to emotionally-charged questions.

If you tried asking these questions (or even much simpler ones) to a modern chatbot, you'd likely quickly conclude that you were not talking to a person.

The problem with assessing EQ is that the more emotionally sophisticated the test, the more general the AI system is likely have to be, in order to turn the input into a meaningful representation.

At the other extreme from the above, suppose that an EQ test was phrased in an extremely structured way, with the structured input provided by a human. In such a case, success at an 'EQ test' is not really grounded in the real-world.

In an essay entitled "The ineradicable Eliza effect and its dangers", Douglas Hofstadter gives the following example, in which the ACME program is claimed (not by Hofstadter) to 'understand' analogy.

  Here the computer learns about a fellow named Sluggo taking his wife Jane and
  his good buddy Buck to a bar, where things take their natural course and Jane
  winds up pregnant by Buck. She has the baby but doesn't want it, and so, aided
  by her husband, she drowns the baby in a river, thus "neatly solving "the problem"
  of Bambi.

This story is presented to ACME in the following form:

ql: (neglectful-husband (Sluggo))
q2: (lonely-and-sex-starved-wife (Jane-Doe))
q3: (macho-ladykiller (Buck-Stag))
q4: (poor-innocent-little-fetus (Bambi))
q5: (takes-out-to-local-bar (Sluggo Jane-Doe Buck-Stag))
...
q11: (neatly-solves-the-problem-of (Jane-Doe Bambi))
q12: (cause (ql0 q11))

Suppose the program were to be asked if Jane Doe's behavior was moral. Complex compound emotional concepts such as 'neglectful', 'lonely' and 'innocent' are here simply predicates, not available to the AI for deeper introspective examination. They could just as easily be replaced by labels such as as 'bling-blang-blong15657'.

So in one sense, the absence of success at an EQ test with any depth is indicative of the general problem currently facing AI: the inability to define (or otherwise learn) meaningful representations of subtle complexities of the human world, which is a lot more complex than being able to recognize videos of cats.


Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?

No.

An optimal policy is generally deterministic unless:

Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember previous states, and the state it is given is not enough to disambiguate between locations. If the goal is to get to a specific end location, the optimal policy may include some random moves in order to avoid becoming stuck. Note that the environment in this case could be deterministic (from the perspective of someone who can see the whole state), but still lead to requiring a stochastic policy to solve it.
There is some kind of minimax game theory scenario, where a deterministic policy can be punished by the environment or another agent. Think scissors/paper/stone or prisoner's dilemma.

  Intuitively, if the environment is deterministic (that is, if the agent is in a state  and takes action , then the next state ′ is always the same, not matter which time step), then the optimal policy should also be deterministic (that is, it should be a map from states to actions, and not to a probability distribution over actions).

That seems reasonable, but you can take that intuition further with any method based on a value function:

If you have found an optimal value function, then acting greedily with respect to it is the optimal policy.

The above statement is just a natural language re-statement of the Bellman optimality equation:

$$v^*(s) = \text{max}_a \sum_{r,s'}p(r,s'|s,a)(r+\gamma v^*(s'))$$

i.e. the optimal values are obtained when always choosing the action that maximises reward plus discounted value of next step. The $\text{max}_a$ operation is deterministic (if necessary you can break ties for max value deterministically with e.g. an ordered list of actions).

Therefore, any environment that can be modelled by a MDP and solved by a value-based method (e.g. value iteration, Q-learning) has an optimal  policy which is deterministic.

It is possible in such an environment that the optimal solution may not be stochastic at all (i.e. if you add any randomness to the deterministic optimal policy, the policy will become strictly worse). However, when there are ties for maximum value for one or more actions in one or more states then there are multiple equivalent optimal and deterministic policies. You may construct a stochastic policy that mixes these in any combination, and it will also be optimal.


How does DeepMind perform reinforcement learning on a TPU?

In their blog post, they link to (among many other papers) their IMPALA paper. Now, the blog post only links to that paper with text implying that they're using the "off-policy actor-critic reinforcement learning" described in that paper, but one of the major points of the IMPALA paper is actually an efficient, large-scale, distributed RL setup.

So, until we get more details (for example in their paper that's currently under review), our best guess would be that they're also using a similar kind of distributed RL setup as described in the IMPALA paper. As depicted in Figures 1 and 2, they decouple actors (machines running code to generate experience, e.g. by playing StarCraft) and learners (machines running code to learn/train/update weights of neural network(s)).

I would assume that their TPUs are definitely being used by the Learner (or, likely, multiple Learners). StarCraft 2 itself won't benefit from running on TPUs (and probably would be impossible to even get to run on them in the first place), because the game logic likely doesn't depend on large-scale, dense matrix operations (the kinds of operations that TPUs are optimized for). So, the StarCraft 2 game itself (which only needs to run for the "actors", not for the "learners") is almost certainly running on CPUs.

The actors will still have to run forwards passes through Neural Networks in order to select actions. I would assume that their Actors are still equipped with either GPUs or TPUs to do this more quickly than a CPU would be capable of, but the more expensive backwards passes are not necessary here; only the Learners need to perform those.


Interested in what Google AI can do. Know nothing about AI at all

Google is just one of the vendors for AI and their initial AI self-service is limited to image analysis. Their are other AI vendors (Microsoft, Amazon) that have other services also. AI is definitely capable of continuous video feed analysis. The self-driving car technology uses live video feed analysis. Are you exploring AI in general or specific use cases? I did a presentation on AI use cases for oil & gas engineering and construction that could give you some ideas.

AI Vision 2020 for Oil & Gas


Why one unit in the layers of neural network is not enough?

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.
Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.
Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.
You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.


How can I start learning mathematics for machine learning?

You should begin from Dr Andrew Ng machine learning course on Coursera. It's probably the most popular course for newcomers in machine learning. It's a free course.

You should also grab "Elements of Statistical Learning" ebook PDF. It's a free book.

You may want to focus on:

Regression
Cross validation
Bias-variance tradeoff
Decision surface
Gradient descent

And more...


Dimension after multiple convolutions in ConvNets

The 64 here is the number of filters that are used.
The picture is kind of misleading in that it leaves out the transition of the maxpool.

Below is a text description of the size of the features as they go through the network with the number of filters in bold.

The first 2 layers in the diagram you posted contain 64 3x3 convs
resulting in a 224x224x64 matrix of features.
This is then fed into a maxpool which reduces the size to a 112x112x64 matrix.
This is then fed to 3 layers of 128 3x3 convs resulting in a 112x112x128
matrix.
Then another maxpool giving a 56x56x128 matrix.
Feeding that to 3 layers of 256 3x3 convs results in a 56x56x256 matrix.
This is then fed into another maxpool giving a 28x28x256 matrix,
Which is then fed into 3 layers of 512 3x3 convs resulting in a
28x28x512 matrix.
Another maxpool gives a 14x14x512 matrix which is fed to 3 layers of 512 3x3 convs giving a matrix of 14x14x512 features.
Another maxpool reduces this to a 7x7x512 which is then
given to 3 fully connected layers of 4096 units each before being
sent to a softmax.


Calculating Weights for CNN Max Pooling Output

A Max Pool layer don't have any trainable weights. Only hyperparameters is present and they are non-trainable. The max pooling process calculates the maximum value of the filter, which consists of no weights and biases. It is purely a way to down scale the data to a smaller dimension. Hope this helps you and have a nice day!


How should I deal with variable-length inputs for neural networks?

The most common way people deal with inputs of varying length is padding.

You first define the desired sequence length, i.e. the input length you want your model yo have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired length, usually you'd split it into multiple inputs.


Can neuro-evolution of augmenting topologies (NEAT) neural networks be built in TensorFlow?

Check this implementation in TensorFlow Eager: https://github.com/crisbodnar/TensorFlow-NEAT


"Distilling the Knowledge in a Neural Network" by G. Hinton et al.
https://arxiv.org/abs/1503.02531


A* and uniform-cost search are apparently incomplete

You forgot to calculate and take into account the costs of the actual paths. You forgot to accumulate the cost of the edges for going forward and backward multiple times!

The evaluation function of uniform-cost search (UCS) is $f(n) = g(n)$, where $g(n)$ represents the cost of the path from the start node to $n$. The evaluation function of A* is $f(n) = g(n) + h(n)$, where $h(n)$ is an admissible heuristic. UCS is a special case of A*, with the admissible heuristic $h(n) = 0, \forall n$. The evaluation function is used to choose the next node to visit from the fringe, which is the set of nodes that can potentially be visited. Whenever we visit a node, we remove it from the fringe. To expand a node $X$ means to add the children of $X$ to the fringe. Whenever you visit a node, you will also expand it.

Let's apply UCS to your specific example. Initially, we check whether $B$ is a goal node or not. It is not, so we expand $B$. We can add $B$ to a list of visited (or expanded) nodes (graph search) or not (tree search). Let's use the tree search, so we will not be keeping track of the expanded nodes, which means that we could expand a node more than once. $B$ has two children, $A$ and $C$, so we add them to the fringe, $\mathbb{F} = \{A, C\}$. Should we now visit $A$ or $C$? We will visit the one with the smallest value of the evaluation function, which is $A$, given that $f(A)=g(A)=36 < f(C)=g(C)=70$, so we remove $A$ from the fringe, which is now $\mathbb{F} = \{ C \}$. Is $A$ a goal node? No, so we expand it, but it only has one child, $B$, so we add $B$ to the fringe, so $\mathbb{F} = \{ C, B \}$. The cost of the path to reach $B$ by going first to $A$ and then to $B$ is $f(B) = 36 + 36 = 72$ (given that you go back and forward on the same edge, so you need to accumulate the cost of these trips!) and $f(C) = 70$, so we visit $C$, so we remove it from the fringe, which is now $\mathbb{F} = \{ B \}$.

You should be able to work out the remaining search (I haven't actually done it). I suggest you watch the video Uniform Cost Search, by John Levine, who shows a concrete example of how UCS and A* work.


Spam Detection using Recurrent Neural Networks

Initially, spam detection relied on simple rule based techniques to sort out spam. However following Paul Graham's famed article 'A Plan for Spam'  the Naive Bayes approach became very popular to the point that it became regarded as the baseline for dealing with spam.

However following breakthroughs in deep learning, researchers have now turned their focus to neural networks to help them deal with the perenial problem of spam emails. Google recently reported that by introducing NN's to g-mail's spam filters. It took them from 99.5% to over 99.9% accuracy, suggesting that neural networks especially when used in conjunction with Bayesian classification may be effective for enhancing spam filters. You can refer to the link below to read about Google's success story https://www.wired.com/2015/07/google-says-ai-catches-99-9-percent-gmail-spam/

Developing a spam filter using neural networks is basically a classification problem. You need to follow the steps below to develop such a system. (Nikhil B 2016)

Collect a dataset of spam and legitimate email messages. Label these datasets. You can find email and spam datasets here http://csmining.org/index.php/spam-email-datasets-.html
Process these messages with feature extraction and vectorising techniques i.e. tf-idf vectorizer, word2vec, bag-of-words e.t.c.
Once you have vectorised the dataset succesfully, apply a supervised learning NN algorithm i.e. radial basis network, multi-layer perceptron (MLP) or backpropagation.
Train your labelled data-set on the neural network. Once training is complete you can use cross validation to calculate the precision of your trained model using the test dataset.

Some of the advantages of using NN's for spam detection over other methods include.

Neural networks have a higher accuracy of identifying spam as demonstrated by google.
They have a lower false positive rate compared to other methods such as rule based techniques.
Their main disadvantage is that they require specialised computing hardware to deploy.

Some old influential papers published in the field include.

Machine Learning Techniques in Spam Filtering (2004) http://ats.cs.ut.ee/u/kt/hw/spam/spam.pdf

Detecting Spam Blogs: A Machine Learning Approach (2006) https://www.aaai.org/Papers/AAAI/2006/AAAI06-212.pdf

A review of machine learning approaches to Spam filtering (2009) https://www.sciencedirect.com/science/article/pii/S095741740900181X


Are current AI models sufficient to achieve Artificial General Intelligence?

I am focusing on what you have posted here without going through (or having read) the whole essay you linked:

In the classic ‘brain in a vat’ thought experiment, the brain, when temporarily disconnected from its input and output channels, is thinking, feeling, creating explanations — it has all the cognitive attributes of an AGI. So the relevant attributes of an AGI program do not consist only of the relationships between its inputs and outputs.

The classic brain in a vat experiment does not disconnect the brain from all inputs and outputs but replaces "real" connections with its environment with "fake" connections, e.g. by connection the brain to a computer. This is what Wikipedia says:

In philosophy, the brain in a vat (BIV; alternately known as brain in a jar) is a scenario used in a variety of thought experiments intended to draw out certain features of human conceptions of knowledge, reality, truth, mind, consciousness, and meaning. It is an updated version of René Descartes's evil demon thought experiment originated by Gilbert Harman.1 Common to many science fiction stories, it outlines a scenario in which a mad scientist, machine, or other entity might remove a person's brain from the body, suspend it in a vat of life-sustaining liquid, and connect its neurons by wires to a supercomputer which would provide it with electrical impulses identical to those the brain normally receives.2 According to such stories, the computer would then be simulating reality (including appropriate responses to the brain's own output) and the "disembodied" brain would continue to have perfectly normal conscious experiences, such as those of a person with an embodied brain, without these being related to objects or events in the real world.

Following that, the argument of the author referring to the brain in a vat scenario does not hold.
If you ignore that problem for a moment the next inconsistency arises: The author assumed that because a brain would act "so and so" an AGI would need to act "so and so" as well. That however is not in line with how artificial intelligence is usually defined. Even if you consider the broad range of definitions:

(source: "Artificial Intelligence: A Modern Approach"; Russel, Norvig, 3rd Ed, 2010)
None of these definitions directly references AI to the brain. Therefore, the assumption that an AGI (as a subset of AI) would need to act like a brain in a vat is, without further assumptions,  generally flawed.
Another problem with the author's argumentation lies here:

to address some outstanding problem in theoretical physics — say the nature of Dark Matter — with a new explanation that is plausible and rigorous enough to meet the criteria for publication in an academic journal.
Such a program would presumably be an AGI (and then some).

The assumption that "creativity" (in quotation marks since we actually need to precisely define that in the first place) requires an AGI does not hold either. Let's stick to the authors example related to dark matter:

This is per definition a specialized field of application of an AI. Accordingly, there is not necessarily a need for an AGI since "specialization" is exactly what an AGI does not have.
Moreover, you could make an argument that coming up with new explanations related to dark matter is somewhat related to automated theorem proving. Accordingly, we might already have AI which is, in principle, capable of "solving" this task as of today.


Is the smartest robot more clever than the stupidest human?

First Question

To treat this question in a scientific way, because I think it is a reasonable enough question that draws on the realities of postmodern culture in post industrialized societies to be treated scientifically, we should define some things.

The most difficult is intelligence, which is the realm in which smartness, cleverness, and stupidity reside.

Let's go through the list.

Most humans are not good at chess, but humans invented the game.
Most humans can't write symphonies, but humans invented them.
Most humans don't read novels, but digital computers can't write them yet and can't learn ethical balance through the reading of them like humans can.
Most humans are not Olympic level athletes, yet humans developed Olympics and robots are not yet Olympiads.
Most humans (whether or hot they are good at logical reasoning) don't employ it much other than to, "Get by."

The question is fine until it devolves into the dismissal of the intelligence that people apply to their method of earning income, which for many if not most people requires more than just getting up, commuting in, following some simple instructions, and going to sleep mildly inebriated.  Let's replace this last part with this.

Most humans do not work with the intention of optimizing quality of the product or service by measuring their own quality and seeking educational resources to improve the velocity, reliability, or accuracy of their work output (unless programs are instituted to incentivize these things in the workplace.)

If we define intelligence as the union of these things, for simplicity's sake, we have this (which is subject to change as AI develops).

Playing chess: AI wins
Designing games: Humans win
Writing symphonies: Humans win
Writing novels: Humans win
Absorbing ethics from stories: Humans win
Olympic gold: Humans win
Logical consistency: AI wins

We must, to interpret the above list correctly concede two things:

Machines may have the ability to do something according to specific quality standards but not be configured, trained, or connected appropriately to prevail.
Humans may have the ability to do something according to specific quality standards but not be educated, trained, or be properly motivated to prevail.

Second Question

What are some things that a clever robot can't do that a stupid human can?  These are a few, but they are of particular importance from certain perspectives.

Love their family and friends
Have compassion without reason
Decide what to learn
Hunt
See a future danger approaching
Entertain others
Pray

I would not dismiss these human propensities as irrelevant, even from a scientific perspective.  I would also not dismiss the possibility that these things are beyond the capabilities of silicon based entities.


Ideal score of a model on training and cross validation data

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data.

Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better with unseen data.


Is the playout started from a leaf or child of leaf in Monte Carlo Tree Search?

NOTE: their description of the selection phase actually does not match the "standard implementation". They state to traverse the tree according to the selection strategy until a leaf node is reached. I disagree with this. The standard implementation is to traverse the tree until a node is reached in which there still are legal actions for which no child node has been created (a node that is not yet "fully expanded").

These two are only equivalent in the case where the expansion strategy is to immediately fully expand, to immediately create new nodes for all legal actions in $L$. With such an expansion strategy, $L$ would immediately turn from a leaf node into a fully expanded node. But that is not the "standard implementation". The most common expansion strategy is to only add a single new node to the tree per iteration.

Let $L$ denote a node that is reached at the end of the selection phase. The standard implementation is indeed to first expand in some way, which results in a new child $C$ of $L$, and then start the playout from $L$ onwards.

  And aren't leaves then permanently stuck as leaves, since playouts always start from their children, not them?

No, as soon as you add the new child $C$ to $L$, $L$ is by definition no longer a leaf node; a leaf node is defined as a node with no children. The next time that MCTS decides to traverse the same part of the tree, $L$ will be just like any other internal node, $C$ will be a leaf node, and we may add yet another node below that in a future expansion phase. Or, of course, it is also possible that in a future iteration, our selection phase still ends in $L$ despite it not (anymore) being a leaf, if it still has unexpanded nodes other than $C$ (to-be-siblings of $C$).

  Why is the playout started from a child of the first leaf, not the leaf itself?

This is because we typically require a slightly different mechanism for choosing a node to expand than the playout mechanism. The most common (or, at least, most simple and straightforward) playout mechanism is to select actions uniformly at random from all legal actions. The most common (or, at least, most simple and straightforward) mechanism for selecting a new node $C$ to add to $L$ is to select uniformly at random only from those legal actions for which children have not already been added to $L$. Due to these mechanisms being different, we cannot initiliase the playout at $L$ and declare the first action of the playout to be the one that generates the new node $C$; we have to use a (slightly) different implementation.


What are the pros and cons of studying machine learning before deep learning?

That question doesn't really make sense: deep learning is a sub-topic of machine learning, so you can't really 'skip' it. It's a bit like "I want to learn about trigonometry, but do I need to do geometry first?"

Having said that, in order to make sense of deep learning you should really know about the general principles of machine learning, otherwise you won't understand it. Or, more importantly, you won't understand what problems deep learning can be applied to, and what issues are better solved with other methods.

You don't need to go into much detail, but should at least get an overview.


time-series prediction : loss going down, then stagnates with very high variance

I found the issue, I should have done more unit testing. Upon computing the batch loss before backpropagation, one of the dimension of the "prediction" tensor was not corresonding to the "truth" tensor. The shape match but the content is not the one that was supposed to be.
This is due to how the NLL loss is implemented in pytorch which I was not aware of ...


How to detect frauds in advertising business using machine learning?

There are a couple of different ways you can go about this depending on what kind of data you have.

If you have labels or can separate the normal data from the fraudulent, you can perform either binary classification, or likely more usefully, anomaly detection.

In Anomaly Detection(which is typically now done via autoencoder) you train your model on the normal data, so it learns a compressed representation of that 'signal', from there it will be able to detect any sample that does not fit the learned representation(in theory).

Here is a link to a tutorial in keras: link


How to calculate the size of a 3d object from an image?

There are hundred of papers on this task some older than I am! Normally this is done by trying to form a box shape around the image than estimate the volume. This task is typically done with multiple images so the two can generate a more clear picture of the size of the object than one image alone. An object could be 'infinitely' large but its mass could be behind the surface you can see with the picture. With the height and length dimensions extracted from the image and with the distance from the camera calculating the size of said surface is fairly easy.

 Stack overflow about problem

Frustum PointNets for 3D Object Detection From RGB-D Data (2018) By Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas

From contours to 3D object detection and pose estimation (2017) By Nadia Payet Sinisa Todorovic


What is the role of the hidden vectors in restricted Boltzmann machines?

Generative models

  The hidden units are just structural support and we don't care about what those hidden vectors really are.

Generative modeling is concerned about $P(X)$, to be able to compute it, we use representation learning (aka deep learning) to identify and disentangle the underlying explanatory factors hidden in the observed data,
because if we could separate them out like this

$$P(X) = P(X \mid Z) P(Z),$$

where $Z$ could encode not just the digit identity but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties.

Any classifier for some task will decide which of those $z$ elements could help him to make a better decision, i.e. just the part that holds the digit identity and ignore the rest, this kind of modeling is called distributed representation, where each element is a simple indicator about some feature.

No conditional dependency within a layer

  (my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)

RBM is an undirected graphical model and no conditional dependency within a layer. Thus, the conditional distribution over the hidden units $h$ given the input image $v$ factorizes, that is

$$p(h \mid v) = \prod_i p(h_i \mid v) $$

And the conditional distribution over the visible units $v$ given the hidden units $h$ also factorize.

$$p(v \mid h) = \prod_j p(v_j \mid h) $$

Inference is tractable

Inference in RBM is tractable and we could compute $p(z|x)$, but the tractability of the RBM does not extend to its partition function.

To minimize the negative log-likelihood of the data, consider the following equation.

$$-\frac{\partial \log p(x)}{\partial \theta} = \frac{\partial \mathcal{F}(x)}{\partial \theta}-\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$$

The first term $\frac{\partial \mathcal{F}(x)}{\partial \theta}$ increases the probability of training data. The second term $\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$ decreases the probability of samples generated by the model.

The second term is nothing less than an expectation over all possible configurations of the input $x$ (under the distribution $P$ formed by the model).

Monte Carlo-based algorithms

So, we use Monte Carlo based algorithms to replace the expectation with an average sum. Samples of $p(x)$ from the model can be obtained by running a Markov chain to convergence, using Gibbs sampling as the transition operator.

For RBMs, the set of visible and hidden units, since they are conditionally independent, one can perform block Gibbs sampling. In this setting, visible units are sampled simultaneously given fixed values of the hidden units. Similarly, hidden units are sampled simultaneously given the visible units.

A step in the Markov chain is thus taken as follows:

$v$: input values
$h$: hidden features, latent space, a.k.a $Z$
$W$: the weight parameters; note that I am using the transposed $W$,
since it's an undirected graphical model.

So, we have the formulas.

$$h^{(n+1)} \sim \operatorname{sigm}\left(W^{\prime} v^{(n)}+c\right)$$

and

$$v^{(n+1)} \sim \operatorname{sigm}\left(W h^{(n+1)}+b\right)$$

where $h^{n+1}$ refers to the set of all hidden units at the $n$th step of the Markov chain. What it means is that, for example, $h^(n+1)_i$ is randomly chosen to be $1$ (versus $0$) with the probability of the sigmoid function and similarly for $v^(n+1)_j$.

In theory, each parameter update in the learning process would require running one such chain to convergence. It is needless to say that doing so would be prohibitively expensive. As such, several algorithms have been devised for RBMs, see Contrastive Divergence.

We initialize the Markov chain with a training example (i.e., from a distribution that is expected to be close to $p$, so that the chain will be already close to having converged to its final distribution $p$).
CD does not wait for the chain to converge. Samples are obtained after only $k$-steps of Gibbs sampling. In practice, $k=1$ has been shown to work surprisingly well.

RBM for dimensionality reduction

  RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

According to the paper Reducing the Dimensionality of Data with Neural Networks

  In 2006, a breakthrough in feature learning and deep learning took place (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level
  to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier.

Summary

Train single layer RBM to reconstruct to input and then keep adding layer after layer until we reach some good low-level representation as depicted in the figure below, W1, W2, W3 are the matrices that have some lower dimensionality of the input space.
Transpose those matrices $W_1, W_2, W_3$ to be the decoder part.
Fine-tune the whole encoder-decoder stack to reconstruct the input. image

RBM vs VAE and GAN

Side note: RBM is now obsolete, see VAE, GAN they are way better at modeling the manifold in which the data is concentrated, yet you could sample from the manifold directly with a single inference step.


What is the difference between simple reflex and model-based agents?

A simplex reflex agent takes actions based on current situational experiences.

For example, if you set your smart bulb to turn on at some given time, let's say at 9 pm, the bulb won't recognize how the time is longer simply because that's the rule defined it follows.

A simple reflex agent doesn't compute complex computational problems nor exhibit intelligence.

A model-based agent takes actions based on historical situational experiences. It performs actions basing on its internal state.

The internal state of the model-based agent acts as a "knowledge base", which allows actions to be performed by representing of unobserved aspects of the current state depending on percept history nor historical situational experiences.

For instance, a research-based planetary system can predict the planet's climatical conditions or weather basing on some geo-information knowledge base.


As you are handling with time series data and you want to find trends; A good approach should be consider applying Holt-Winter's seasonal method. This algorithm handle seasonal, trend and smooth parameters. A good implementation of this kind of algorithm is Prophet by Facebook. You can code an exploratory analysis with this library and obtain trend, yearly seasonality, and weekly seasonality of the time series, among other components. Example:


How to test if my implementation of back propagation neural Network is correct

Actually the implementation was correct,

The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.

When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.

Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function


Learning similarities between customers and offers representation

The idea is that I should be able to locate the best offer for any customer given these representations.

I think you need a Recommender System. As you want to map the offers to customers based one their representation you can check Content-Based Recommender System.

The method is by taking pattern from a customer history and try to find similarities with the new offers. You can use many techniques for a recommender system, from a simple TF-IDF or Deep Learning for more complex problems.


Are on-line backpropagation iterations perpendicular to the constraint?

Answer by Theo Bandit at maths stackexchange

  If you choose two points $(w_1, w_2), (v_1, v_2)$ along this line, then
  $$(x_1, x_2) \cdot ((w_1, w_2) - (v_1, v_2)) = x_1 w_1 + x_2 w_2 - (x_1 v_1 + x_2 v_2) = y - y = 0.$$
  That is, the direction $(x_1, x_2)$ is perpendicular to any vector lying along the line, i.e. $(x_1, x_2)$ is normal to the line.


Is it a great misconception that the softmax is an activation function?

I see no problem in regarding the softmax as a particular activation function which takes a vector input and produces a vector output. In fact, the sigmoid function can be viewed as a two-dimensional softmax in which one of the two inputs is hardwired to zero while the corresponding output is neglected.


What are causative and exploratory attacks in Adversarial Machine Learning?

When someone is able to do a causative attack it means there is a mechanism by which they are able to input data into the network. Maybe a website where people can input their images and it outputs a guess on what is in the picture and then you click if it got it right or not. If you continue to input images and lie to it it will obviously get worse and worse if they use the user input to add to the test set. Most people are careful and don't mess around with mixing new data into the testing sample. If they did something like mixed the user input training and test then resampled something like that could occur but most people don't do that. It's bad practice and even worse than leaving your NN open to tampering from malicious user input. Information isn't really added to the knowledge in the model till it is fed into the model and backpropagation occurs.

An exploratory attack is sending tons of inquiries to the model to gain information about the data set they have built into the model even to the point of extracting data about individuals pieces of data that are built into the model. Then, with this information, they could try to reconstruct the data set. They could attempt to trick the network by sending strange generated inputs.

In the paper Adversarial Machine Learning (2011), by Ling Huang et al., in section 2, the authors define these terms, under the category influence.

  Influence

  Causative - Causative attacks alter the training process through influence over the training data.

  Exploratory - Exploratory attacks do not alter the training process but use other techniques, such as probing the detector, to discover information
  about it or its training data.

They also provide other related definitions.

  Security violation

  Integrity - Integrity attacks result in intrusion points
  being classified as normal (false negatives).

  Availability - Availability attacks cause so many classification errors, both false negatives and false positives, that the system becomes effectively unusable.

  Privacy - In a privacy violation, the adversary obtains information from the learner, compromising the secrecy or privacy of the system’s users.

  Specificity (a continuous spectrum)

  Targeted - In a targeted attack, the focus is on a single or small set of target points.

  Indiscriminate - An indiscriminate adversary has a more flexible goal that involves a very general class of points, such as “any false negative.”


How does vanish gradient restrict RNN to not work for long range dependencies?

Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers less than 1 and gradually it will become weaker and weaker and when it arrives to the first layers, it's so weak that makes almost no change in initial layers parameters.
Now in case of RNNs you can unroll the network, and now you can see that it is like a deep network. For clarity look at the image below (taken from a course by Andrew NG and edited):

Red arrows show the way of gradient backpropagation and you can see that in each step they are multiplied by a number (actually a matrix is multiplied in another matrix). If this number is less than one, then it results in vanishing gradient. If this number is greater than one it will result in exposure (which can be controlled by simply clipping it to a max value). The higher the steps, the more of vanishing or exploding effect.
But they have found a solution for this problem, LSTM or GRU. What these units do is that they will make a highway for gradient to backpropagate through (in each state it will be multiplied by 1). So gradient can travel for longer distance (from French into France). Even though, the problem still exists for very long relations.
If you know what is a ResNet and how it works, you can find the same concept behind LSTM or GRU and ResNets as both make a highway for gradient to flow back.
You can have an intuition of how LSTM or GRU works by following the forward pass and assuming what they are doing as locking the concepts in memory cells. Like when the network sees the word France it will understand that it's an important word and maybe it will become handy later. So it will put the France word in one of it's memory cells and keeps it there for several steps and when it wants to guess the language it will use this memory cell to predict French not English or Persian. Then it can release that memory cell and use this cell for something else.
If you want to learn more and get better intuition, I highly recommend you to look at this link: CS231n of MIT - lecture 10


What is sample efficiency, and how can importance sampling be used to achieve it?

An algorithm is sample efficient if it can get the most out of every sample. Imagine learning trying to learn how to play PONG for the first time. As a human, it would take you within seconds to learn how to play the game based on very few samples. This makes you very "sample efficient". Modern RL algorithms would have to see $100$ thousand times more data than you so they are, relatively, sample inefficient.

In the case of off-policy learning, not all samples are useful in that they are not part of the distribution that we are interested in. Importance sampling is a technique to filter these samples. Its original use was to understand one distribution while only being able to take samples from a different but related distribution. In RL, this often comes up when trying to learn off-policy. Namely, that your samples are produced by some behaviour policy but you want to learn a target policy. Thus one needs to measure how important/similar the samples generated are to samples that the target policy may have made. Thus, one is sampling from a weighted distribution which favours these "important" samples. There are many methods, however, for characterizing what is important, and their effectiveness may differ depending on the application.

The most common approach to this off-policy style of importance sampling is finding a ratio of how likely a sample is to be generated by the target policy. The paper On a Connection between Importance Sampling and
the Likelihood Ratio Policy Gradient (2010) by Tang and Abbeel covers this topic.


How do I update the Q values of a Deep Q Network when exploring?

For tabular Q-learning, the q-values for state s and action a are updated according to

$$
Q(s, a) \gets Q(s, a) + \alpha [(r + max_{a'} Q(s', a')) - Q(s,a)]
$$

where $\alpha$ is the learning rate and $(r + max_{a'} Q(s', a')) - Q(s,a)$ is the difference between the current estimate of the q-value, $Q(s,a)$, and the target, $r + max_{a'} Q(s', a')$.

The target q-value is based on the greedy policy, not the exploratory policy. Q-learning is theoretically guaranteed to converge to the optimal policy for any behavior policy (like $\epsilon$-greedy) that is guaranteed to visit every state and action pair an infinite number of times.  See Section 6.5 of the Sutton and Barto book for more details.

In contrast to Q-learning, the target q-value for SARSA is $r + Q(s', a')$, where $a'$ is chosen from an exploratory behavior policy like $\epsilon$-greedy. For SARSA the learned q-values are dependent on the behavior policy and therefore not guaranteed to converge to the optimal policy. A behavior policy that intentionally acted randomly for multiple consecutive actions, as in your example Asteroids exploratory policy, would likely lead to learning different q-values than would be learned for an $\epsilon$-greedy behavior policy.

Unfortunately Q-learning's theoretical guarantees of convergence to an optimal policy go out the window when nonlinear function approximation is introduced, as is the case for deep neural networks. Nevertheless, in the Deep Q-Networks paper, the q-value function is updated using a target value based on the maximum q-value for the next state. Specifically, if $Q(s, a, w)$ is a q-value function parameterized by weights $w$, then the weights are updated by

$$
w \gets w + \alpha [(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)] \nabla_w Q(s,a,w)
$$

where $w^-$ are the parameters of the target network used to stabilize training. (See the paper for more details). This update rule is chosen to minimizes the loss function

$$
L(w) = E[(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)]^2
$$

For your own implementation, it may be helpful to see a code example of the Deep Q-Networks parameter updates.  A tensorflow implementation is available in the function build_train in the OpenAI Baselines DeepQ code.


Can programming cause passive or aggressive behavior in AIs?

The Wikipedia entry on this personality theory says of Type A people:

  The theory describes Type A individuals as ambitious, rigidly organized, highly status-conscious, sensitive, impatient, anxious, proactive, and concerned with time management. People with Type A personalities are often high-achieving "workaholics."

All of those attributes could conceivably be explicitly programmed in. Alternatively, most of them could arise from a basic goal of performing a certain task as efficiently as possible. After all, if you really want to carry out a task, you're going to get organized, you'll only do other things if they're asked of you by someone important, you won't want to get bogged down in irrelevant things, you'll actively pursue the necessary resources, and you'll want to use time as effectively as possible.

Note that this applies only to strong AIs, since weak AIs like image recognizers don't generally have personalities that we can interact with.

Now, just for fun, let's consider an overly aggressive personality, to the point of a disorder.

This Counselling Resource page seems helpful in describing what an aggressive person does. The page includes a bulleted list of common characteristics, which I distill into the following:

They attempt to gain dominance and control
They oppose to anything that places limits on them
They take advantage of others to further their own goals
They hide information from whose who would oppose them
They rarely decide to stop pursuing their desires (even impulses)

This all seems like a characterization of an AI designed to be the best at its task: the best out of any other agent, and the best it by itself could possibly be. Ruthless pursuit of the highest performance would involve taking control of all relevant resources (including other agents), demolishing barriers to the goal, thwarting those who would interfere with progress, and carrying out each possibly-useful idea/desire to completion.

In summary, yes, an AI's behavior and personality are programmable, either explicitly or through some kind of emergence.


How to recognize sequence of digits in an image

Your task is text recognition, however your code is for classification task. So you need to use different approach for that. You mentioned that you're going to give model 123 and get 123. But you can not do that with just convolutional networks. Images with text are sequential, so you need to use CRNN(Convolutional-Recurrent-Neural-Networks), LSTM(Long-Short-Term-Memory), BiLSTM(Bidirectional-LSTM). In most of the research papers there are convolutional networks are being used just for feature extraction stage. For prediction stage they used recurrent units, such as LSTM cells or RNN.


No, AiAngel is not a bot.
It's Rouge's software that changes his voice along with facial recognition software which tracks his movement and copies it to the avatar.

That being said, he has created a very entertaining channel and line of work for himself. By looking at the videos, you can see that he is a true genius at work. Single handedly puts on all hats for that project, from hardware infrastructure development, to software design to graphic art to acting and video editing. Truely one of the great minds of this era.

You can find Ai Angel (Angelica) on Rogue Shadow's

Youtube Channel: https://www.youtube.com/user/TheRogueShadow1/videos
Twitch https://www.twitch.tv/aiangellive
Discord https://discordapp.com/invite/DFhaYgj
Twitter https://twitter.com/AiAngel13
Patreon: https://www.patreon.com/AiAngel

..along with several other platforms.

Hope than answers your question with certainty (instead of philosophically)


The correct syntax is tensorboard --logdir=$PATH, with an equals sign between the switch and the path. Then I would suggest you make sure you are actually telling to the Saver to save whatever data you want to save.


Which pathfinding algorithms can be applied on coloured graphs?

The idea is to apply a well known pathfinding algorithm (such as Dijkstra) on the graph given the rule: "only black and red edges"

Remove all the non-black-and-red edges from the graph first, then run it through any off-the-shelf pathfinding implementation.  Or implement your own, and have it completely ignore the non-black-and-red edges.


Why does the error ensemble use the ceiling of the number of classifiers?

Y ensemble size voting wrong

  k = 50% or majority threshold

If you have 11 models. Then the majority of models is anything bigger than 50% of the number of ensemble models. In the example where you have 11 base models. The majority would be anything bigger than N/2 or 11/2. But since 11 is an odd number and cannot be divided by 2. We have to use the python ceiling function to round 5.5 to 6. The in other words. For your ensemble to be wrong. We must look for the probability that Y of them are wrong. Y is ≥ than K and K is ceil(N/2). Thus we must calculate this probability(6 wrong) like shown above summating the possibility of each of these discrete combinations into one


How to generate labels for self-supervised training?

How can I generate the target label from the other data in the dataset?

If you are asking how you can create the learning signal in SSL, when given an unlabelled dataset, for learning representations of these unlabelled data, then there is no general answer. The answer depends on the type of data that you have (which can be e.g. textual or visual), and which features do you think you want to learn or can be learned from your unlabelled data. This paper and other answers to this question provide some examples of how that can be done (depending on the type of data). Below, I also provide an example.
Let me try to explain this more in detail.
Let's assume that you have both

an unlabelled dataset $U = \{ u_i \}_{i=1}^m$ and

a labelled dataset $D = \{(x_i, y_i) \}_{i=1}^n$

where we may have $m \gg n$ (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the downstream task. So, you may think that $x_i$ and $u_i$ are images and $y_i$ are labels, like "cat" or "dog" (let's say that you want to differentiate between cats and dogs).
You want to solve this downstream task by supervised learning with $D$. However, given that your labeled dataset is not big enough, you may think that training a neural network from scratch (i.e. by randomly initializing its weights) with $D$ may not lead to good performance. So, you think that it could be useful to start training from a pre-trained model that already contains useful representations of data similar to your labeled data, i.e. to perform transfer learning. To pre-train such a model, you could use SSL.
So, to solve your downstream task with SSL, there are 2 different steps

Self-supervised learning (SSL): learn representations of your images $u_i \in U$ by training a neural network $M$ with $U$ to solve a so-called pretext (or auxiliary task); there are many pre-text tasks: you can find many examples here, here and here (see example below too);

Supervised learning (SL) by transfer learning: fine-tune $M$ with $D$ (the labeled dataset), in a supervised way; this task is known as downstream task (as stated above)

In this process, there are 2 different labels.

In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).
So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

In step 2, you use the labeled dataset $D$, which has been typically annotated (or labeled) by a human. As stated above, this dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat" or "dog".
In this step, the pre-trained model $M$, with the SSL technique,  can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

Note that SSL can also refer to something (slightly) different than what has been explained in this answer. See my other answer for more details. Moreover, note that you can perform representation learning with SSL without necessarily solving a downstream task later, which may also not be an SL task (in the example above, I've described a downstream task that is an SL task only for simplicity).
If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques (such as this) for more inspiration.


What does a hybrid Bayesian network contain?

A bayesian network where the random variables represented can be continuous or discrete. Usually the continuous variables are handled with Normal distribution with an input of some linear combination of the variables it is dependent on.

Much more detail can be found in Koller's Probabilistic Graphical Model book: http://pgm.stanford.edu/


I think this game will go pretty crazy, because, at some point, the generator AI will be able to generate absolutely perfect images. Actually, no, just perfect enough that no AI can be sure whether they are real or fake.

So, I think the AI war will go onto more than the image, the detector AI will probably evolve to analyze whether this video is logically plausible, for example, by tracking the celebrities' position to prove that it is impossible that he/she was, for example, let's put it this way, being unloyal to his/her partner.

I mean, currently, AI can tell whether an image is fake or not better than human because it has seen about a million times more samples than us, but if we know who the person in the image is and we are as stalky as the AI I just described, we can probably work out that this image is implausible.

Of course, there will be counter measurements to that. But, at that point, we might as well just let the AI rule the world, given that it will have become this smart (lol).

But, seriously, if it's smart enough to think this far ahead in this 'real world' problem, then strong AI is nigh.


How can a neural network work with continuous time?

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the outside environment is acting, rather than how your code is taking its steps. These are some lines from the book, Artificial Intelligence: A Modern Approach:

  The discrete/ continuous distinction applies to the state of the environment, to the way time is handled, and to the percepts and actions of the agent. For example, the chess environment has a finite number of distinct states (excluding the clock). Chess also has a discrete set of percepts and actions. Taxi driving is a continuous state and continuous-time problem: the speed and location of the taxi and of the other vehicles sweep through a range of continuous values and do so smoothly over time. Taxi-driving actions are also continuous (steering angles, etc.). Input from digital cameras is discrete, strictly speaking, but is typically treated as representing continuously varying intensities and locations.

So, continuous or discrete is not something that should be talked about as a problem of the code. It is basically, what an environment is. Your concern with the device is regarding the code. I will suggest that you upload the code on git and ask people to improve it.
I hope this helps!


Are there any refutation of the original Lovelace Test?

I am a future neurologist with a very complete understanding of linguistic processing in the brain.  I am also an overprotective parent, so I monitor every phrase uttered to my child, and also completely determine all the books she reads in the course of her education.

When my child writes a poem, then, I know the dataset on which her brain was trained, as well as the processes by which her language inputs became language outputs--in broad outline I know these processes are non-linear and are based on how different inputs along with the current collection of trillions of distinct synaptic weights updates the synaptic weights.  I don't know what her poem will be, of course, because there are random factors and the whole history of her synaptic weights are unobservable, but I adhere to the Lovelace test and can therefore conclude that composing the poem was not a creative act.

The Lovelace Test, like the Chinese Room Argument, implicitly assumes that what computers/AI can do in processing symbols and information and what brains can do are distinct.  If you accept that assumption, then the argument ceases to be interesting-- you've merely redefined creativity as one of the distinct things that brains can do.  If you reject the assumption, the argument that computers are incapable of creativity ceases to be valid.  The thought experiment itself does nothing to assist us in evaluating the truth of the assumption.


How can I train a neural network to grade the user's answers to a questionnaire?

use the embedding layers which are mainly used for text.

input the question number and the text that the student wrote to the algorithm. Make the problem into a regression of a number from 0 to 10 ( or a classification of class 10, see which one gives better performance).  So the NN will look at text given a question number and try to figure out which 'points' should it get. It will be a supervised training problem since you already have examples of text rated from 0 to 10.


Concrete example of latent variables and observables plugged into the Bayes' rule

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule

\begin{align}
p(z|x)=\frac{p(x|z)p(z)}{p(x)}
\tag{1}\label{1}
\end{align}

The posterior $p(z|x)$, the likelihood $p(x|z)$, the prior $p(z)$ and the evidence (or marginal) $p(x)$ are Gaussian distributions. You can assume this because Gaussian distributions are closed under conditioning and marginalization.

For simplicity, let's further assume that they are univariate Gaussian distributions. Given that the Gaussian distribution is a continuous probability distribution, it has an associated probability density function (rather than a probability mass function, which is associated with discrete probability distributions, such as the Bernoulli distribution). The probability density function of the Gaussian distribution is

\begin{align}
f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} } \tag{2}\label{2}
\end{align}

where $\mu$ and $\sigma^2$ are respectively the mean and variance of the Gaussian distribution and $x$ is a variable (similarly to a variable $x$ in any mathematical function $f(x)$). So, given a concrete value for $x$, for example, $x=1$, then $f(x=1, \mu, \sigma^2)$ is a so-called density value (rather than a probability, which a probability mass function returns, given an input). For example, let's assume that the mean $\mu=0$ and the variance $\sigma^2 = 1$, then, for $x=1$, the density will be

$$
f(1 \mid 0, 1) = \frac{1}{\sqrt{2\pi} } e^{ -\frac{1}{2} }
$$

So, to obtain the concrete density value, I've just replaced the concrete values of $x$, $\mu$ and $\sigma^2$ in equation \ref{2}.

To calculate the posterior $p(z|x)$ in equation \ref{1}, you just need to replace the likelihood $p(x|z)$, the prior $p(z)$ and the evidence $p(x)$ with the Gaussian probability density shown in equation \ref{2}, so you will have

\begin{align}
p(z|x)=\frac{f_{X\mid Z}(x \mid \mu_{X\mid Z}, \sigma^2_{X\mid Z}, z) f_{Z}(z \mid \mu_{Z}, \sigma^2_{Z})}{f_{X}(x \mid \mu_{X}, \sigma^2_{X})}
\tag{3}\label{3}
\end{align}

I've explicitly added a subscript to the means and variances of each probability density, given that, for example, the mean of the probability density $f_{X\mid Z}$ might be different than the mean of the probability density $f_{Z}$ or $f_{X}$, etc. So, to get the actual density value (a real number) that represents $p(z|x)$, you just need to replace $f_{X\mid Z}$, $f_{Z}$ and $f_{X}$ with the definition of the Gaussian density function in \ref{2} with their actual mean and variance values. I'll let you do this, given that this is really just a matter of picking up a concrete value for the means and variances and doing some algebra.

If you assume the posterior, the likelihood, the prior or the evidence to have a different distribution, you will do the same thing, but using the probability density or mass function of your chosen distribution.

In the context of the variational auto-encoder, you will be learning the mean and variance of the distribution, so the mean and variance will not be fixed, but they will be the parameters that you want to find. However, this does not change the way you apply the Bayes' rule.


How do we define intention if there is no free will?

My thoughts.
The short answer is: you can't.
The long answer is that since we're searching for a new definition of a term when removing a necessary (in my opinion) precondition for it to exist, the question becomes "can you make up a new definition for what you intuitively and empirically understand as intention, while removing free will from the picture?". I'm going to give it a shot.
First of all, there's a lot to be said about whether or not the idea or intuition of intention even exists in our collective discourse based on the latent assumption that free will is a thing. As in, before any rigorous definition, even the intuitions encoded in discourse, philosophy, art, and other languages as "intention" could very well be as invalid as the assumption of free will itself.
That being said, I'm a fan of Deleuze's model for people and other entities as machines of input, output and internal state (not his wording, but I paraphrase and in consequence, interpret and alter for the purposes of my point). It's not perfect, but I run to it a lot to answer these question as I find it very refreshing, often lacking in bias and having good explanatory power compared to the usual romance-foo that dominates these conversations. If that's the case you could pretty much define intention not as a self-started force but as a product of a much blurrier mechanism, namely the non-deterministic characteristic of this rhizomatic soup. Whether or not an input or output will exist, what kind will it be, what internal state will it find or cause in the machine and the long term dependencies between these interactions seem to me like a convincing enough candidate for the cause of any intuition (or illusion if you like a more cynic vocabulary) of "intention". It's pretty much an emergent symbol we use, assuming the form of a force for the setup and function of new connections, that will in their complication or pure non-determinism spawn even more intention in the network.
tl;dr: Intent could be the most basic expression of the RNG of the universe.


What do the variables in the cross-correlation formula mean?

It takes a little bit of time to fully understand the 2D convolution/cross-correlation and to relate it to the usual diagrams of the convolution operation, so, before addressing your questions, let me first try to break the definition of the 2D cross-correlation down, from the left to right.
$$S(i,j) =(K*I)(i,j) = \sum_m \sum_n I(i+m, j+n)K(m,n) \label{1}\tag{1}$$

$S$ is the function that is the cross-correlation of the functions $K$ and $I$, so $S(i, j)$ is the cross-correlation of $K$ and $I$ at the pixels $i$ and $j$

The symbol $*$ in $K*I$ is the cross-correlation/convolution symbol, but sometimes the cross-correlation/convolution is also denoted as $\circledast$

$K*I$ is the function that results from the cross-correlation of $K$ and $I$, i.e. $S$, so $(K*I)(i,j)$ is the value of the function $S$ (the cross-correlation of $K$ and $I$) at the inputs (or pixels) $i$ and $j$. In other words, $(K*I)(i,j)$ is just another way of writing $S(i,j)$ that emphasizes that we took the cross-correlation of $K$ and $I$, but they are exactly the same thing.

The double summation $\sum_m \sum_n$ is because we are computing the 2D cross-correlation, i.e. over the $x$ and $y$ dimensions of the image and kernel. This is just the definition of the 2D cross-correlation.

The $m$ and $n$ are the indices of the summations, one across the $x$-axis and the other across the $y$-axis. Let $m = x$ and $n = y$, so we can rewrite equation \ref{1} as follows.
$$S(i, j) =(K*I)(i, j) = \sum_x \sum_y I(i + x, j + y)K(x, y) \label{2}\tag{2}$$
Now, it should be clearer that we are indexing across the $x$ and $y$ dimensions.

Now, let me further restrict the range of the summations. Let's say from $x=-1$ to $x=1$ and from $y=-1$ to $y=1$, then we can rewrite equation \ref{2} as follows
$$S(i, j) =(K*I)(i, j) = \sum_{x=-1}^1 \sum_{y=-1}^1 I(i + x, j + y)K(x, y) \label{3}\tag{3}$$.

Why do I want to do this? Let me explain why. Consider now the following kernel $K$ (which happens to be a Gaussian kernel)
$$
K = \begin{bmatrix}\ \ \color{blue}{\frac {1}{16}} &\ \ \frac {1}{8} &\ \ \frac {1}{16} \\\ \ \frac {1}{8} &\ \ \frac {1}{4} &\ \ \color{red}{ \frac {1}{8}} \\\ \ \frac {1}{16} &\ \ \frac {1}{8} &\ \ \frac {1}{16}\end{bmatrix}
$$
Note that this is the output of the function $K$ or, more precisely, its support. Let's assume that $\frac {1}{4}$ is at the index/pixel $(0, 0)$. Then, for example, the top-left $ \color{blue}{\frac {1}{16}} $ is at index/pixel $(-1, -1)$ and the middle-right $\color{red}{ \frac {1}{8}}$ at pixel $(1, 0)$

Now, consider any image $I$ represented as a 2D matrix (i.e. its support), with, for example, dimensions $U \times V$. For concreteness, let $U = 5$ and $V=5$. Moreover, the middle pixel of the image is at index $(0, 0)$, as for the kernel. Let's say the image is the following
$$
I =
\begin{bmatrix}
0 & 1 & 0 & 1 & 0 \\
1 & 1 & 0 & 1 & 1 \\
0 & 0 & \color{green}{0} & 1 & 1 \\
1 & 1 & 0 & 0 & 0 \\
0 & 1 & 0 & 1 & 1 \\
\end{bmatrix}
$$
So, $\color{green}{0}$ is at index/pixel $(0,0)$.

Now, let's say that $i = 0$ and $j=0$ in equation \ref{3}. This means that we compute the cross-correlation between $I$ and $K$ at the index $(0, 0)$, where, in the case of the image $I$, the value is $\color{green}{0}$.

With these settings, it should be clearer now that equation \ref{3} means that the cross-correlation between $K$ and $I$ at index/pixel $i=0$ and $j=0$ is a 2D dot (or scalar) product. If it is not clear, then let be take the $3\times 3$ submatrix of $I$ centered at $(i, j) = (0, 0)$ (below, $(0, 0)^{3 \times 3}$ is just the notation that I came up with to indicate that).
$$
I_{(0, 0)^{3 \times 3}} =
\begin{bmatrix}
1 & 0 & 1 \\
0 & \color{green}{0} & 1  \\
1 & 0 & 0
\end{bmatrix}
$$
Then the cross-correlation in equation \ref{3} is just
$$
S(i, j) = S(0, 0) = (K * I)(0, 0) =
\sum
\begin{bmatrix}
1 \frac {1}{16} & 0 \frac {1}{8} & 1 \frac {1}{16} \\
0 \frac {1}{8} & \color{green}{0} \frac {1}{4}  & 1 \frac {1}{8}  \\
1 \frac {1}{16} & 0 \frac {1}{8} & 0 \frac {1}{16}
\end{bmatrix},
$$
where $\sum$ is a sum across all elements, i.e.
\begin{align}
S(0, 0)
&= 1 \frac {1}{16} + 0 \frac {1}{8} + 1 \frac {1}{16} +
 0 \frac {1}{8} + \color{green}{0} \frac {1}{4} + 1 \frac {1}{8} +
1 \frac {1}{16} + 0 \frac {1}{8} + 0 \frac {1}{16}  \\
& = \frac {1}{16} + \frac {1}{16} + \frac {1}{8} + \frac {1}{16} \\
& = \frac {3}{16} + \frac {1}{8} \\
&= \frac {5}{16}
\end{align}

Now, to answer your questions/concerns more directly.

Are they the dimensions of the kernel along the height and width dimensions?

No. They are the indices that determine the neighbourhood around $i$ and $j$ where you want to compute the cross-correlation.

So, for the input image $I$ and kernel $K$, we take the sum product of $I*K$

In this case, the symbol $*$ does not denote the product, but the cross-correlation. Maybe by "sum product" you meant the cross-correlation (or 2D dot product), but I'm not sure.

How is the input image $I$ indexed?

The input image is indexed with $m$ and $n$, but you start at $i$ and $j$, that's why you use the actual indices $i+m$ and $j+n$.


What is the input for the prior model of VQ-VAE?

Some notes about VQ-VAE:

In the paper, they used PixelCNN to learn the prior.
PixelCNN is trained on images.
The discrete latent variables are just the indices of the
embedding vectors. For example, you can put your embedding vectors
in an array.

For a single input image, the number of the output channels of the encoder before quantization equals the dimensionality of the embedding vectors. For an analogy, the output of the encoder is like an image but with a number of channels that equals to the dimensions in the embedding space. With that, each pixel is a vector in the space of the embedding vectors1. The quantization is done on each pixel by mapping each pixel to the nearest embedding vector $e$.
Now, each pixel can be represented by a single integer number which equals to the index of its nearest embedding (what you call the discrete latent variables). Thus, you have two representations for the quantized output, one is a large tensor with many channels (embedding vectors), and another is a simple discrete image with one channel (discrete latent variables).
The one channel discrete images are used to train PixelCNN. This is great because they are small size images. while the large tensor with the embedding vectors is used for the decoder as it holds the information needed for reconstruction.
So, the discrete latent variables for the prior model and the embedding vectors for the encoder.
1This is just an analogy, pixels values belong to a closed interval.


Is a deep technical understanding of neural networks required outside of research?

It depends on what exactly you want to be. You don't need to be mathematician if you just want to run neural networks. Most data scientists don't understand the mathematics, but they know how to run machine learning frameworks. Generally, only PhDs understand the mathematics.

In the industry, most job positions in machine learning (e.g software engineer, big data engineer etc) don't require advanced mathematics. But you should be very comfortable in programming, and understand simple things like matrix manipulation. Data engineers in the industry generally have good programming skills, but they can't read mathematical equations (or just simple equations).

However, quantitative positions in machine learning do require significant mathematics. That happens both in research and industry.

Good news --- if you don't have a PhD (e.g statistics) nobody would expect you understand the mathematics.

Summary:

If your job is more like a development position, mathematics is an advantage but not absolutely necessary
If your job is research, you do need advanced mathematics understanding (e.g. PhD)
If your job is not research but still quantitative based. Mathematics is very important.

PS: I work in research. Everybody I know who works in machine learning has a PhD.


Binary classifier that minimizes false positive error

There is no predefined classifier for any problem. Two main features of a classifier are

its cost function and
its corresponding weight update formula.

Since your problem statement requires a huge cost for falsely classifying a particular class one approach will be.

You have to define a cost function that will penalize hugely for misclassifying for that class only. So your cost function will be $J$ and $J'$ put together. You can look up the cost function of a logistic classifier to see how two separate cost functions are merged together here.

The second approach can be (assuming you are using supervised learning), the learning rate $\alpha$ for both the classes should be different. The larger learning rate will be for the one which is the more important class, since you don't want to classify it improperly (increasing $\alpha$ compared to the other class will reduce or risk of misclassifying it). The exact learning rate depends from case to case.

Thus, I have tailored the two main features of the classifier to solve this problem:

The cost function.

The weight update scheme (by changing learning rate for different cases).


How do I apply reinforcement learning to a game with infinitely many actions?

This looks like the kind of game that a Reinforcement Learning algorithm could learn to play. If you are learning from the game objects directly (as opposed to wanting to have the agent learn from viewing the screen), then it should be relatively straightforward to do so - by which I mean that it will involve some new concepts, a little maths, and maybe take you a few weeks to learn enough and run enough experiments on the AI to the point where you have something good. But at least you should not need a cluster of GPU servers to run your learner.

  However, I have no clue how to apply Q-learning to this task. For example, how could the coder possibly know what future reward another move will bring?

That is a major part of RL - the algorithms that are part of RL all solve this issue. Therefore the coder does not need to know how to calculate future reward, just an immediate reward. For the example game, it looks like you can grant a reward of $+1$ for every frame where the player does not crash. An RL algorithm that can solve this environment should figure out through "trial and error" which actions will keep the game running.

Q Learning achieves this by keeping track of estimates of future value and updating them across time steps - if action $a_1$ in state $s_1$ consistently leads to a later state $s_2$ with a high value, then Q learning will update its estimate of the value of $Q(s_1,a_1)$ to be high as well. The estimates start very poorly, but updates work their way back through time steps as the agent learns, and provided things don't go wrong, eventually it will learn that one action will lead to higher value than another, and it be able to consistently choose it.

This is such a basic question that if you want to use RL, even if you find a library that implements the whole thing for you, that you clearly need to study some foundations of RL. There are a few different places you can do that. I can suggest two that I have used:

The book by Sutton & Barto. Reinforcement Learning: An Introduction
UCL lectures by David Silver. These include videos on YouTube.

Both are online, free and comprehensive introductions to the subject. You can find shorter, more direct introductions online if you search for e.g. "Q Learning Tutorial".

Be prepared to spend hours just learning enough about the topic that you understand how to structure your data and interface with the game in order to create a learning environment for your AI.

  In this case, the player has infinitely many actions. How would I apply RL to this case?

This is a good question. Some algorithms are better than others. Q Learning cannot work with infinitely many actions. However, you could choose a discrete set of mouse points - e.g. maybe 64 - at different angles and distances from the player (red dot) and try and get the values of those. That would allow you to try with Q learning, and is worth considering because RL that properly solves large action spaces is harder to understand and work with. You could look into the DQN algorithm for this. In short, the DQN algorithm uses history of actions and their immediate results to train a neural network to estimate the values of each action.

If you really want to solve large action spaces properly, then you need to be looking into Policy Gradient methods. Two popular and effective PG algorithms are A3C and DDPG. These are tougher to learn and implement. They use neural networks to learn multiple functions, one of which is the policy - this directly selects which action to take, given a current state/observation.

  Is RL a good approach to solving this task?

It is definitely possible to use RL to do what you want.

  Is there a handy Java library which would allow me to use a RL algorithm (as a block-box) to solve this problem?

I could not find one on a brief search. To get started, you might want to look at java-reinforcement-learning which implements some of the algorithms from Sutton & Barto, so would be a good starting point. Most of the other Java packages I found covered basics and learning material, but would not be plug-and-play for your game.

If you do find a library, you will need to have learned enough basic RL to be able to follow how to use it.

  Are there alternatives?

One alternative that might work well for your game environment, and be a little easier to understand and implement would be to combine a policy function (like a neural network) with an evolutionary algorithm that tries out variations of the policy. The go-to algorithm here is probably NEAT, and it is capable of solving simple game agents effectively. Your game should be a reasonable fit for NEAT because the scoring system of survival time makes for a good fitness function.

One thing that could really be worth trying before diving into the (very hard to understand) A3C and DDPG algorithms is the Cross-Entropy Method. It is a really simple RL agent that has some similarities to genetic algorithms (it selects "best" items, but instead of from the population, it is from random behaviour of the same individual), but doesn't always scale too well. However, it might work well for your game.


Overlap between AI and "Game AI"
Nowadays, if you search for AI online, you will find a lot of material about machine learning, natural language processing, intelligent agents and neural networks. These are not the whole of AI by any means, expecially in a historical context, but they have recently been very successful, there is lots of published material about them.
Games, especially action games, tend not to use these new popular technologies because they have other priorities. However, under the broadest definitions, there is good overlap between AI in general and game AI, or more specifically enemy AI within a game:

Computer-controlled opponents within a game are effectively autonomous agents operating within an environment.

The game's purpose is to entertain the player, and this usually translates into requiring challenging and believable behaviour for actions taken by opponents (for some value of "believable").

A 3D map with a physics engine, other moving agents and possible hazards is a complex environment that needs some effort from a developer in order to define working behaviours.

Many games need to solve one or two classic AI problems such as pathfinding.

All the above means that the term "Game AI" is a good choice for describing how enemy agents are designed and implemented. However, there are large differences between goals of AI applied today in a 3D shooter and the goals of AI used in industry to control decision making.

Action games have a low CPU budget available per enemy for decision-making. Control systems in industry can have dedicated machinery, sometimes multiple machines, just to run the AI.

Action game priority is the game experience of the player, and this translates into strict targeting of a definition of "believable" for enemy behaviour that will entertain. In contrast, AI used in industry has accuracy as a high priority.

Typical game AI in 3D shooters
Developing enemy AI in games is a specialist skill, and only partly related to the purpose of this site. For detailed discussion you may want to look into Game Development Stack Exchange which has many questions and answers on topics like Enemy AI
Very roughly, the key traits of enemy AI in a game like Call of Duty could look like this (I have no inside knowledge of how COD does this, so may be inaccurate in places):

Enemies will have one or more modes of behaviour defined. This might be as simple as "Idle" and "Attack Player", or there could be many. For COD I would suspect there are many, and some may be in a hierarchy - e.g. there may be several sub-types of "Attack" behaviour depending on the enemy design.

Within a mode of behaviour, there may be a few different components defined, only some of which use AI routines. For instance, there will be specific animations related to walking or standing which are not AI, but there will also be some pathfinding AI if the agent is attempting to move around.

Decisions to switch between different modes of behaviour are often scripted with simple triggers, such as detecting player visibility using ray-casting between the location of player and enemy. When game AI fails to produce realistic results, it is often these high level triggers being brittle and not covering edge cases that causes it. Depending on complexity of the game and number of behaviours, there may be an algorithm managing the transitions between them, such as a finite state engine or behaviour trees.

Enemies will be presented with highly simplified observations of the game world from their perspective, in order to speed AI decisions.

AI systems will have CPU budget restricted. A 3D game spends significant resources rendering scenes, and will often try to render quickly, e.g. 100 times per second. There are multiple ways that AI budget can be allocated, but it is relatively common for AI calculations to be spread over muliple frames, and for search structures for tasks like path-finding to persist over time.

There will be a middle ground between scripted behaviour and AI-driven behaviour where analysis is done as part of game design. For instance, pathfinding routes may be pre-calculated to some degree. A system of way points is one example of this - it might be set by the game designer, it may be calculated by an AI component of the game asset-building pipeline, or it may be dynamically calculated and cached during a game session so that multiple enemy units can share it.

When complex AI is not needed to achieve a goal, when a simple caclulation or "puppet-like" behaviour would do just fine, then this could be chosen instead. For instance, an enemy "aiming" at a player can be a simple vector calculation, perhaps with a fudge factor of a miss chance depending on the range to make the enemy seem fallible and not as much like a machine.

I do not know how the behaviour of enemies using cover is implemented in COD. A simple variant would be to have each enemy hard-coded to use a specific piece of cover that usually works well against the player due to map design. However, it is definitley possible to have a search algorithm running (perhaps over multiple frames) that assesses nearby locations that the enemy could reach against the player's current position, and then pick those as variables to plug into the "take cover and fire on player" scripted behaviour. That assessment would use the same kind of visibilty detection between enemy and player that is used to trigger changes between "Idle" and "Attack" behaviours for the enemy.

As an aside, one related thing I find interesting is in using modern AI techniques to blend animations and make interactions between actors and the environment look more realistic. Although it is a lower-level feature than the question about enemy behaviour you are asking, it is an interesting cross-over between robotics and game playing that we will likely see in next-generation games, and probably applied first to the more detailed player model animations. Here is another example applied to a humanoid agent switching smoothly between different tasks, which any game player will recognise as something which game engines cannot do well at the moment - there are usually many jarring transitions caused by events in the game.


What technology do people use to create bots for games like LOL or Runescape?

Bot development is more about 'hacking' than AI in a way that in the very first place you need to read and (over) write game data which you are not supposed to (and thereby potentially violating the Terms and Conditions - so be aware of that). The AI part is fairly simple for most hack/bot applications that I can think of.

Read data

To read game data you can for example:

use an assembly code debugger like Ollydbg to locate relevant data in the memory, e.g. amount of gold
Observe graphical objects being rendered, e.g. a unit being drawn
Intercept network packages containing all kinds of game information, e.g. a unit appearing on your screen

Write data

In a similar way there are multiple ways to write data:

overwrite game data in memory
Use the Windows API SendInput() function to emulate keyboard inputs
Use the Windows API SendMessage() function to send messages to the game
Manipulate network traffic

These lists are not comprehensive but to give you an idea of how it is being done.

AI-wise an A* algorithm for example can be deployed to do path finding.

If you are interested in the topic I suggest to read 'Game Hacking' by Nick Cano. The book provides a good introduction.


Why is Jensen-Shannon divergence preferred over Kullback-Leibler divergence in measuring the performance of a generative network?

Lets start with question 1)  how does JS-divergence handles zeros?

by definition:
\begin{align}
D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\
&= \frac{1}{2}\sum_{x\in\Omega} [p(x)log(\frac{2 p(x)}{p(x)+q(x)}) + q(x)log(\frac{2 q(x)}{p(x)+q(x)})]
\end{align}
Where $\Omega$ is the union of the domains of $p$ and $q$. Now lets assume one distribution is zero where the other is not (without loss of generality due to symmetry we can just say $p(x_i) = 0$ and $q(x_i) \neq 0$. We then get for that term in the sum
$$\frac{1}{2}q(x_i)log(\frac{2q(x_i)}{q(x_i)}) = q(x_i)\frac{log(2)}{2}$$
Which isn't undefined as it would be the KL case.

Now onto 2) In GANS why does JS divergence produce better results than KL

The asymmetry of KL divergence places an unfair advantage to one distribution over the other where in this case, its not ideal to consider it this way from an optimization perspective. Additionally KL divergences inability to handle non-overlapped distributions is crushing given that these are approximated through sampling schemes, and therefore there are no guarantees. JS solves both those issues and leads to a smoother manifold which is why its generally preferred. A good resource is this paper where they go more in detail investigating this.


How to detect a Neural Network will work with the whole dataset?

You can try to train it on 1% data, then on 2%, 3%, etc... Then plot it and see if increasing data increases the performance and how it is changing. Not sure if that's correct answer, but at least you can iterate this method pretty fast.


How do we get the true value in the prediction objective in reinforcement learning?

The true value $v_{\pi}(s)$ is a conceptual target for the $\overline{VE}$ in the book. You often do not know it in real problems. However, it is still used in two main ways in the book:

Theoretically for analysis of different aprpoximation schemes, which can be shown to converge to minimise the $\overline{VE}$ objective, or a related one.
In toy problems when exploring the nature of approximation in Reinforcement Learning (RL), it is possible to use tabular methods guranateed to get close to zero error, and then compare them to approximate methods. There are several plots of this type in the book.

The book shows derivation of gradient descent methods that start with minimising $\overline{VE}$ as an objective, and that use samples of $v_{\pi}(s)$ such as Monte Carlo return $G_t$ or the TD target) in place of the unknown $v_{\pi}(s)$ in the loss function. These also rely on the fact that the sample distribution will be weighted by $\mu(s)$ if they are taken naturally from the environment whilst the agent is following the policy $\pi$ so that $\mu(s)$ also does not need to be explicitly known.

Outside of toy problems deliberately constructed to demonstrate that this theory is correct, you will not know $v_{\pi}(s)$ or be able to calculate $\overline{VE}$. However, you will know from the theory that if you follow the update rules derived in the book for approximate gradient descent methods, that the process should find a local minimum for $\overline{VE}$, for whatever state approximation scheme you have chosen to use.

Usually you cannot even approximate $\overline{VE}$ from raw data, as the variance in returns will add noise to the signal, and there is no way to separate variance in returns from error in approximation in the general case. However there are a couple of scenarios that do lend themselves to measuring this objective, provided you already have your estimate $\hat{v}(s,w)$ and the policy remains fixed throughout:

Simple, fast (perhaps simulated), environments which can be solved to arbitrary accuracy using tabular methods. In this case, you first calculate $v_{\pi}(s)$ using a non-approximate method, then sample many approximations by running the environment using policy $\pi$ and treating that as your data set.
Fully deterministic environments where $\pi$ is also deterministic. These have a variance of $0$ for Monte Carlo returns, so each observed return from any given state is already the true value of $v_{\pi}(s)$. Again you can just run the environment many times to get your data set to calculate  $v_{\pi}(s)$ and $\hat{v}(s,w)$ for the observed states in the correct frequencies, and thus have data to calculate $\overline{VE}$.


Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?

There is stuff like the Universal Approximation Theorem.

There are also investigations into the loss surface of neural networks.

And classics like this explanation of the vanishing gradient problem.

But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions can currently only be answered empirically.


How could a NN be trained to output a cyclic (e.g. hue) number?

Your solution is pretty much on spot. It corresponds to the YUV scheme used in television and designed to match human perception characteristics. As you already noticed, such an encoding wouldn't suffer from discontinuities.


Is the word "pose" used correctly in the paper "Matrix Capsules with EM Routing"?

Great question, and one that I think we could have done a better job of answering in the paper.

Essentially, the pose matrix of each capsule is set up so that it could learn to represent the affine transformation between the object and the viewer, but we are not restricting it to necessarily do that. So we talk about the output of a capsule as though it is an affine transformation matrix, but we can't ensure that it will be. We do things explicitly that make it more like such a matrix — like adding in the coordinates to the right-hand column — but we can't be sure. This somewhat embodies a large part of the capsule network theory — we set up scaffolding so that the network can learn to be equivalent to transformations that we think it ought to be invariant to, but we don't ensure that it is.


Your intuition is good. Because "nearby" is only defined with "connected", there could only be 1 station between them. However, it says that the stations are "not_too_far" if at most one station is between them. What about if no stations are between them? If 2 stations are "connected" they should be "not_too_far" as well.

So it should be:

not_too_far(X,Y) :- connected(X,Y) ; nearby(X,Y).

Where ; denotes OR.


Advantages of CNN vs. LSTM for sequence data like text or log-files

The networks with one-dimensional convolution over the temporal dimensions are called Temporal Convolutional Networks. The authors of this study claim that TCNs are comparable to RNNs in performance on common tasks associated with RNNs.


A deep learning algorithm to optimize the outcome

Deep learning models for regression tasks are quite difficult to train so I would suggest not to start with them. Instead I would start with one of the approaches below and maybe try to use deep learning afterwards.

A classical approach to the problem could be to analyze your optimization software and this would probably lead you to some deterministic algorithm.

Different approaches can be to treat your optimization software as a black box - give it a wide variety of inputs, note down the variables you are interested in (execution time, cutting results, etc.) and try to fit some sort or regressor on it.

One option is to follow Kourosh's idea and formulate it as a classical optimization problem.

If you prefer to use machine learning tools than I'd suggest that you start with a simple model like linear regression just to verify that there is any signal in the data that you can use. Afterwards you can look at more powerful algorithms like xgboost, regression trees, etc.


How to know what kind of memory is stored in the connection weights?

You should use google for this question, it is extremely vague. Some technical documents will give very clear insight into what's happening here.

https://blog.openai.com/generative-models/

http://proceedings.mlr.press/v37/gregor15.pdf

If you really want to understand "what the AI is thinking", well, you may never know. The idea of AI is that it can handle complex data (high degree of dimensionality) and is too complex for Humans to comprehend.