How to model inhibitory synapses in the artificial neuron?

Principles of Computational Modelling in Neuroscience by David Sterratt, Bruce Graham, Andrew Gillies and David Willshaw discuss it in Chapter 7 (The synapse) and also in Chapter 8 (Simplified models of neurons). Especially in chapter 8, they discuss how to add excitatory or inhibitory synapses to integrate and fire neuron.

There are various ways to add inhibitory synapse: either substrate voltage, inject negative current.


When does IDA* consider the goal has been found?

According to Artificial Intelligence: A Modern Approach 4th edition in IDA* the cutoff is the $f$-cost($g+h$); at each iteration, the cutoff value is the smallest $f$-cost of any node that exceeded the cutoff on the previous iteration.
In other words, each iteration exhaustively searches an $f$-contour, finds a node just beyond that contour, and uses that node's $f$-cost as the next contour.
And we must test if the node is a goal node when it was selected for expansion, otherwise, the algorithm is not optimal anymore (the proof is similar to that one in A*).
I think the animation that the site provided is misleading because in the code which is written in the last of the same site we have that:
function Search(node, g, threshold)              //recursive function
{

  f = g + heuristic(node);
  if(f > threshold)             //greater f encountered
         return f;
  if(node == Goal)               //Goal node found
         return FOUND;
  integer min = MAX_INT;     //min = Minimum integer
  foreach(tempnode in nextnodes(node))
  {

     //recursive call with next node as current node for depth search
     integer temp=search(tempnode, g + cost(node, tempnode), threshold);
     if(temp == FOUND)            //if goal found
       return FOUND;
     if(temp < min)     //find the minimum of all 'f' greater than threshold encountered
       min = temp;

     }
     return min;  //return the minimum 'f' encountered greater than threshold
}

And in the previous code, we test if the node is the goal node only when it was selected for expansion.


Is this page of Goodfellow's Deep Learning Textbook missing text?

There is no industry standard for psuedocode as it is read by humans (for the moment ;) and the author can decide how best to represent the concepts of the algorithm with the aim of balancing precision and understandability. In this case there is a

FOR ... END FOR

block which is iterated over k for 1 to m. And an

IF ... END IF

block that is run if fine-tuning evaluates to true.


How should I detect an object in a camera image?

Is this a good approach? Will I have a lot of trouble with different backgrounds?

A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without the object in frame. My gut feeling is that your system will have poor performance compared to a properly trained classifier, as the autoencoder will naturally have to get good at constructing background elements in order to score well, so unless your object is very consistently always in a similar place with radically different appearance to the background then the autoencoder will get good reconstructions of background-only frames, and your anomaly detection would need to be set too sensitive. That would cause the detector to fail to spot the object when in frame.

There is a catch-all answer to "how well will my idea work" with ML projects. You should try it and see. Data science is essentially an empirical approach, and building practical models is an engineering discipline where testing is a core part of the process.

In order to test your model, you are going to need a lot of images of just background, including all the kinds of background that you expect the system to be used. Which begs the question, why not collect a range of suitable images without your target object, and use them as the second class?

  This doesn't solve the problem, because I don't know what else could not be an object. It would be impossible to train for 2 classes: object / not object.

Actually, it is easy. Just collect suitable images from locations where you expect your system to be used, without the object in frame. These are your "Not Object" class. A "Not Object" does not have to include some substitute foreground object. Although I recommend that you do have some images like this to prevent accidentally creating an "object is in the foreground" detector*. The primary goal should be to be collect data that matches how the model will be used once deployed. That will depend a lot on how much control and consistency you get over the production cameras where the trained detector will be put to use.

I would do this, then train a standard binary classifier.

If you are still interested in how well your auto-encoder idea could work, you then have plenty of test data to evaluate it.

*  This is one area where your auto-encoder idea might do better than a classifier - reconstructing unfamiliar foreground objects should be hard for it leading to high error.

It is too difficult to tell in advance whether this effect is strong enough to make your auto-encoder approach better than a classifier.


Adding input features - is complete re-training required?

I'll try to explain how I would do it and the intuition behind it, feel free to correct me if something doesn't make sense.
Lets consider you have an input of shape F where F is the number of features. If you were to construct a simple feed forward neural network you'll need to multiply the input with a weight matrix of shape (F, hidden_dim). Now if we want to add one more feature, the input will be of shape F+1 and the multiplication with the first layer will not work. What we could do to overcome this problem is to pad the weight matrix with an extra 0. In theory the new network should be able to reproduce the results of the first because it ignores the new feature. Now if we want to learn the "importance" of the new feature we could train the model to do so. I assume that the training time should be significantly lower since some of the weights might not need to be updated.


Can supervised learning be recast as reinforcement learning problem?

Any supervised learning (SL) problem can be cast as an equivalent reinforcement learning (RL) one.
Suppose you have the training dataset $\mathcal{D} = \{ (x_i, y_i \}_{i=1}^N$, where $x_i$ is an observation and $y_i$ the corresponding label. Then let $x_i$ be a state and let $f(x_i) = \hat{y}_i$, where $f$ is your (current) model, be an action. So, the predicted label of observation $x_i$ corresponds to the action taken in state $x_i$. The reward received after having taken action $f(x_i)$ in state $x_i$ can then be defined as the loss $|f(x_i) - y_i|$ (or any other suitable loss).
The minimization of this loss is then equivalent to the maximization of the (expected) reward. Therefore, in theory, you could use trajectories of the form $$T=\{(x_1, f(x_1), |f(x_1) - y_1|),  \dots, (x_N, f(x_N), |f(x_N) - y_N|)\}$$ to learn a value function $q$ (for example, with Q-learning) or a policy $\pi$, which then, given a new state $x_{\text{new}}$ (an observation) produces an action $f(x_{\text{new}})$ (the predicted label).
However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient).
For more details, read the paper Reinforcement Learning and its Relationship to Supervised Learning (2004) by Barto and Dietterich, who give a good overview of supervised and reinforcement learning and their relationship. The paper Learning to predict by the methods of temporal differences (1988), by Richard Sutton, should also give you an overview of reinforcement learning from a supervised learning perspective. However, note that this does not mean that a reinforcement learning problem can be cast as an equivalent supervised learning one. See section 1.3.3 Converting Reinforcement Learning to Supervised Learning of the mentioned paper Reinforcement Learning and its Relationship to Supervised Learning for more details.
Reinforcement learning can thus be used for classification and regression tasks. See, for example, Reinforcement Learning for Visual Object Detection (2016) by Mathe et al.


What would be the importance sampling ratio for off-policy TD learning control using Q values?

Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not?

You could assign values of 1 to each to get the right answer, but the situation is different. You can see that more clearly in the definition of action value, $q(s,a)$:
$$q_{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1}|S_t=s, A_t=a]$$
The condition on the expectation $|S_t=s, A_t=a$ means that the value of $A_t$ is already assumed. There is no need to know or use the associated probability, and no need to adjust between probabilities of different policies when estimating returns for the first action choice.
In addition, the last action choice used to bootstrap in Q learning is always the maximising action over current Q values. That is an on-policy choice with respect to the target policy $\pi$, so does not need to be adjusted for.
For single-step Q learning, there is no need to use importance sampling to adjust for policy differences between behaviour and target policies. The correct update equation rule is:
$$ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left( R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t,A_t) \right). $$
This does change for n-step updates, where you do need to take account of differences in action choice between target and behaviour policies for $A_{t+1}, A_{t+2}$ up to $A_{t+n-1}$. However, you should bear in mind that $\pi(a|s)$ is zero for any non-maximising action - if you use weighted importance sampling then any trajectory with an exploring action before calculating the TD update contributes zero update.
So, if you are calculating over longer trajectories, in e.g. Q($\lambda$), it is common to see this simplified to a logical test and some kind of shortcut update. You rarely see explicit use of importance sampling - with probability calculations and ratios - in Q learning.


Can AI be used to reverse engineer a black box?

Yes this is entirely possible. As was previously mentioned, complex connectionist systems are often thought of as black boxes(despite us being able to "look in" the box given enough computation and analysis) because of the difficulty in understanding learning and the networks ultimate decision making.

Here, we can model the problem as such: given an input of filter settings(and presumably some information about the audio), predict the target descriptors as an output. All you really need to do is generate a dataset from the program and then train it in a multi-label classification context to predict the output descriptors.


How do two perceptrons produce different linear decision boundaries when learning?

I think your confusion comes from the fact that you are calling those two hidden nodes "perceptrons". You shouldn't call the hidden nodes in your network perceptrons. You should call them "nodes" or "neurons", although the term "multilayer perceptron" comes exactly from that.

You should think of a perceptron as something like

A perceptron simply computes a linear combination of the inputs using the corresponding weights. A single perceptron can (only) linearly separate a set of points.

However, if you have two "stacked perceptrons" (like the hidden nodes in your network), then, provided that both "perceptrons" (or hidden nodes) only compute a linear combination of the inputs, the output of each hidden node can be interpreted as line (like the violet and green lines in your picture).

  I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron?

This is because the two perceptrons will eventually have different weights. You can see from your diagram that $p_0$ has weights $w_{0, 0}$ and $w_{0, 1}$, whereas perceptron $p_1$ has weights $w_{1, 1}$ and $w_{1, 0}$. Note that weights are learned, during training, depending on your loss function (at the output node $p_2$) and the labeled dataset (that you use to train that simple neural network).

  I know the weights can be initialized differently but does this second perceptron classify something else?

If you consider the two hidden nodes (in your network) separately, then (as I said above) the output of these two hidden nodes can be interpreted as a line, and these lines might be different (as illustrated in your picture), because these two hidden nodes will eventually learn something different about the input. However, you should also think of the output of two hidden nodes as contributing to the input of the output node. So, in a certain way, you're combining the decisions of the two hidden nodes (or, as you call them, "perceptrons").

Note that you will need to introduce "non-linearity" in those hidden perceptrons in order to be able to non-linearly separate a set of points.

  Shouldn't the decision boundaries both converge to the same ultimately after training?

No, not necessarily. The decision boundary found by the perceptron learning algorithm depends on the initialization of the weights and the order that the inputs are presented. See chapter 4 (specifically, pages 192-196) of Pattern Recognition and Machine Learning by C. Bishop. Two perceptrons would be guaranteed to converge to the same solution if the loss function was convex (i.e. there's only one optimum).


What AI service can define personality portfolio based on text?

My first recommendation would be before you create an AI or ML based solution. Kindly consider using a business Q&A Software such as Questions for Confluence by Atlassian among others. An enterprise multiple choice solution could be a simple and elegant fit for this problem.

However, if one was to design a solution in accordance to your specification the first step to resolve the problem would definitely be one of applying NLP. In my answer I utilize the open source procedure laid out by EASE (Enhanced AI Scoring Engine) by Edx and a combined initiative of MIT and Harvard towards improving automated essay scoring (Parachuri V 2013).

Computers cannot directly understand text the way people can. People naturally break sentences into units of meaning, however when using a computer we will first have to instruct the computer how to do it. This process is called tokenization. Tokenization is the task of chopping a phrase into pieces called tokens at the same time discarding some characters i.e. punctuation.

After we tokenize the essay answers, we then convert the tokens into a matrix (bag of words model also known as vector space model). This model is a very convenient way to represent text in matrix form. Here we disregard grammar and even the word order but the frequency of words is retained.

Now we can proceed to the machine learning part!

Here I will recommend that we either use regression or classification. The difference between these two algorithms is that regression assumes you are using a continuous scale while classification is discrete.

In statistical modeling, regression is a set of statistical processes for estimating the relationships in variables.

  A simple linear equation is $y=m*x+b$ , where y is the target value(score), m is a coefficient, and b is a constant.
  In linear regression, we would do something like $y=m{1}*x{1}+m{2}*x{2}+\dots+m{n}*x{n}+b$.

Classification (also sometimes called a decision tree) on the other hand is a data mining method for assigning categories to a collection of data which is useful since we want to classify essays on the basis of their scores.

The next step will be to measure the error in the scores that are generated by the algorithm on the previous step.

In order to measure accuracy, lets use cross-validation. Here we split the training dataset randomly into n parts each segment is a 'fold'. This is called n-fold cross validation. We then iterate from 1 to n predicting the scores of the parts [n] from all the data in parts [!n].

Hope this helps.


Does humanism grant moral consideration to sentient artificial general intelligences?

Does humanism also extend to granting rights to AGIs, should they become sentient?

I think that is going to be something that will divide self-identifying humanists into a few different groups, based on understanding about what AGI is.

The front page you link includes "aiming toward the welfare and fulfillment of living things", which does not explicitly include non-living sentient things. However, the humanist philosophy naturally extends to granting rights and protections based on reasoning and scientific evidence to anything that can be argued as needing them.

The big problems are:

Recognising and measuring sentience.
Attaching moral value to events that happen to artificial entities, where subjective measures such as pain, stress, happiness, fulfilment might not be relevant.

These measurement problems already cause a lot of variability in moral stances towards farming animals for instance. These biological creatures share enough with humans that even though we don't fully understand the nature of their subjective experience, it is possible to extrapolate with some backing from scientific studies. For example, we can measure whether a creature suffers stress or pain and reacts similarly to a human, and even if we don't understand what that means subjectively (what it feels like to be such an animal), a combination of Occam's Razor and Precautionary Principle can be argued:

Occam's Razor the simplest interpretation is that if an animal is based on same cellular structure, has same brain regions for experiencing pain and reward, uses same stress hormones etc, that its subjective experience is likely somehow comparable to a humans. This allows us to apply empathy to an animal's situation with a guess that doing so is not 100% inaccurate.
Precautionary Principle the consequences for inhumanely treating a sentient creature are morally worse than the waste of humanely treating a non-sentient creature (provided doing the latter is not causing undue suffering to some other thing). So faced with a lack of precise knowledge, there is still a strong rational ethics argument for extending rights and humane treatment if there is any uncertainty.

When it comes to artificially created entities, we lose the first argument of Occam's Razor, and can only get it back with a better understanding of what sentience actually is, and which parts of it we should value. We need a much firmer theory here than with animals, because there is so little shared between our construction and that of a human-built machine. In addition, just as there are many different kinds of organism which exhibit different amounts of apparent intelligence and possible sentience, there will be different versions of AGI, probably starting with purpose-built non-narrow AIs that combine features that are practically useful and make them seem intelligent (parsing human speech and responding appropriately, which Google Home, Alexa, Siri already do to some depth) with some common-sense understanding of broad task portfolios. These things are near future, and most developers who work on them would not consider them sentient - however, they will be designed to act as if they were, because that makes them practical and usable, which in turn will make it much harder to argue Occam's Razor due to similarity in behaviour.

We still keep the second argument, but it is confounded by not understanding what the analogs are for humane and inhumane when dealing with artificial entities. It is unlikely that anyone would have a problem with switching off a running instance of the next 2 or 3 generations of Siri assistants - but at what point does that become deliberate incapacitation of a sentient being, and does it even matter if the system is designed to be switched on and off, and expresses no particular desire to be in either state?

In summary, the humanist philosophy could extend to cover the welfare of artificial general intelligences, but it is far from clear whether it can go there in practice. Until we have more complete theoretical models of AGIs, or practical working devices, then it is premature to consider it. Most debate is going to be based on imagined qualities of AGI, and any two people discussing this are likely to be starting from different assumptions about what those are.


How to estimate the error during training in deep reinforcement learning

Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.

Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.

In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.


Keras DQN Model with Multiple Inputs and Multiple Outputs

Firstly, concatenate only works on identical output shape of the axis. Otherwise, the function will not work.
Now, your function output size is (None, 32, 50) and (None, 600, 1). Here, '32' and '600' must be same when you want to concatenate.
I would like to suggest some advice based on your problem. You can flatten both of them first and then concatenate. Because you need to flatten feature to use dense layer later.
def create_mlp(self, arr, regress=False):
        # define MLP network
        print("Array", arr)
        model = Sequential()
        model.add(Dense(env.rows * env.cols, input_shape=(len(arr)//2, len(arr)), activation="relu"))
        model.add(Dense((env.rows * env.cols)//2, activation="relu"))
        **model.add.flatten() ### shape = (None, 1600)**
        # check to see if the regression node should be added
        if regress:
            model.add(Dense(1, activation="linear"))
        # return our model
        return model

And just remove the reshape layer in create_cnn function.
(output shape should be = (None, 600)).
then concatenate two model
combinedInput = concatenate([mlp.output, cnn.output])  ## output shape =(None, 2200)
Later you can just use Dense layer as your code. I don't know how can you used dense (next to concatenate layer) without flatten the feature in create_mlp function.
Your code should work this way.
You can read this simple one for better understanding.


What are examples of thought experiments against or in favour of strong AI, apart from the Chinese room argument?

An excellent book summarizing the development of thought in this area over several hundred years is Mind Design II, edited by John Haugeland. This book contains a collection of essays written by the major thinkers in this area through until about the 1990s. A brief summary of some of the major ideas is:

Descartes: Minds are spirits, brains are bodies. Minds attach to brains in a special part of the brain. Descartes has a number of thought experiments, but they mostly have a Theological root. Most modern scientists do not find these to be satisfying because the resolve metaphysical questions about the mind by assuming you already believe in a particular interpretation of a God.
Gilbert Ryle argues in the late 1940s that concepts like the Mind are in some sense vacuous. Asking whether a machine has a mind or not is like asking whether a field contains two cows or a pair of cows: this is a conceptual distinction that has no relation to the real world. Because of this, its study is futile. The parallel movement of Behaviorism in psychology also viewed the study of minds and mental states as a fool's errand. Rather, this movement focused on studying behaviors that could be directly measured.
Turing publishes his seminal paper in AI, in 1950. In this paper, Turing anticipates nearly all later objections to AI, and summarizes reasonable refutations of them. This paper contains a number of short thought experiments. One of the more notable ones is the Turing Test. A key observation Turing makes is that if a machine behaves in a way indistinguishable from a human, then any arguments used to reject its claim to intelligence (and even to consciousness) appear to work equally well on other humans. Searle doesn't really address this in his later argument. Basically, I have no way to know for sure whether you have subjective experiences of the same kind I have. The fact that you're made of roughly the same kind of meat as I am seems like a shaky explanation. For example, would humanoid aliens that act like us also be intelligent? What if it were found that their brains took some radically different form from ours? What if it were found that their brains were really networks of transistor-like machines?
The Cognativist revolution spanned a number of fields starting in the 1960s. This movement held that behaviorism was wrong for two basic reasons. First, Behaviorism does not account for the subjective feelings of existence or understanding. Second, Behaviorism could not explain phenomena like language, that appeared to involve reasoning logically about symbols. Notable authors in this period are Chomsky and Fodor, who argued that minds were essentially computer programs that happened to be running on brains instead of computers. These ideas dominated AI, Psychology and Linguistics for about 30-40 years. Fodor's works contain a number of thought experiments involving language and programs.
John Searle's Chinese Room argument caused problems for the cognativists, because it presents a convincing chain of logic showing that a program that does complicated tasks (like linguistics) does not subjectively understand what it is doing. Various objections are raised and Searle responds to them in his publication of the argument. Objections to Searle's arguments form the basis of several later movements in philosophy of AI.
The Connectionists argue that we can represent the behavior of the brain (and thus, the mind) with a brain-like program. This paralleled increasing interest in artificial neural networks. An example of a cognativist thought experiment is to imagine a simulation of an entire human brain, conducted with perfect fidelity (for instance, we accurately model all the quantum mechanical interactions). If such a machine were not conscious, opponents need to answer the question of where consciousness is hiding (Is it in the meat? Then see Turing. Is it in some yet-undiscovered property of neurons? Then the burden of proof is definitely on the person proposing such a hidden property, and they must explain why we cannot simulate it).
The Churchlands and others argued, much as Ryle, that minds as the public understand them are a sort of "folk theory", that appear to explain something about reality to lay people, but are actually a dead end. Concepts like "beliefs" and "experiences" are, according to these arguments, akin to aether or phlogiston: things that our current, malformed, understandings of neuroscience seem to require, but that in fact have no basis in reality, even though nearly everyone believes in them at present. This line of work contains many extensions of the kinds of thought experiments Turing hinted at.
Brooks and others advance the theory of embodied cognition, in which the mind is not confined to the body, but is actually a property of a body plus the environment it is placed in. An example thought experiment from this school (by Andy Clarke if I recall correctly) is to imagine a person with profound loss of the ability to form new long term memories. This person carries around a notebook. The cover says "Your memory". The first page contains instructions explaining that the person has lost the ability to remember things, and that they write new things they want to remember in the book. The person can remember things for, say, 15 minutes at a time, and the book is very well organized. Where is this person's mind exactly? Is it in their brain (which does not remember anything 15 minutes in the past), or is it in the brain and the book, which together can remember and reason about things essentially like a normal person. If it is in the brain and book, what if we replace the book with a computer inside the person's head?


Training by one batch of examples, what does it mean

This would mean that there is only one gradient update on your dataset of 16 samples. If you are taking the mean of all the squared errors ( mse ), then the loss of each sample is contributing to the 0.1 loss.


I haven't seen it as you describe and I don't think it would be much useful. Pooling layers are being gradually phased out of networks, because they don't seem to be that useful anymore. With the emergence of more and more conv-only architectures, I don't see that likely.


What does it mean "derivative of an image"?

The term Derivative of an Image in the context you mention has two meanings.

A matrix, image, or floating point number that is derived from an image via convolution, passing the image through a two dimensional NN, the application of an FFT analysis, or some other process.  In this context, the word Derivative implies the direction of calculation: Image B is derived from image A.
A matrix or cube that represents the rate of change at in the image being processed.  The change being measured between only two adjacent pixels in a single dimension and only one direction at a time, but the applications of this technique is very limited, and such a sequence is of differences, not at all reasonable approximations of the derivative of light.  What is more useful in real recognition systems are two dimensional or hexagonal windowing (Gausian, Hamming, Hanning, trapazoidal, cosine, ...) across space and, for video, through time.  The calculus term derivative should always reference the theoretical surface being approximated using these techniques, not the discrete matrix or cube that approximates the surface.

Such multidimensional convolution and neural network based approaches are less sensitive to capture noise and orientation nuances.  Two dimensional whole image or windowed FFT techniques have met with much success because filtering the expected frequency range of features to be detected is merely an attenuation process.  Two and three dimensional splines can also be tuned to be useful in the detection of features in an orientation independent way.

In addition to gray scale analysis, color and transparency channels can be selected for independent or parallel analysis or added to the dimension of the fitting model from which the derivative is taken.

Advances in deep networks have blossomed into a new area of image processing and recognition research, bringing new hope to robotics, automated transportation, and cybernetics in general.


What is the difference between the breadth-first search and recursive best-first search?

According to this article

Breadth First Search (BFS) searches breadth-wise in the problem space. Breadth-First search is like traversing a tree where each node is a state which may a be a potential candidate for solution. It expands nodes from the root of the tree and then generates one level of the tree at a time until a solution is found. It is very easily implemented by maintaining a queue of nodes. Initially the queue contains just the root. In each iteration, node at the head of the queue is removed and then expanded. The generated child nodes are then added to the tail of the queue

According to the book Algorithms and Theory of Computation Handbook by Mikhail J. Atallah.

Recursive best-first search is a best-first search that runs in space that is linear with respect to the maximum search depth, regardless of the cost function used.
It works by maintaining on the recursion stack the complete path to the current node being expanded as well as all immediate siblings of nodes on that path, along with the cost of the best node in the sub-tree explored below each sibling.
Whenever the cost of the current node exceeds that of some other node in the previously expanded portion of the tree, the algorithm backs up to their deepest common ancestor, and continues the search down the new path.
In effect, the algorithm maintains a separate threshold for each sub-tree diverging from the current search path.


Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?

The algorithm in the 2012 survey article (your second link) is the most common / standard implementation. Whenever someone mentions using MCTS or UCT, without explicitly stating any other info, it's safe to assume that that pseudocode is what they're using.

The paper by Kocsis and Szepesvári from 2006 (your first link) is (one of) the original publication(s) on UCT. It is very similar to the "standard" implementation as described in the survey paper. If I recall correctly, the only important difference is that the algorithm as described in 2006 keeps track of at which point in time during an episode a reward is observed, and accounts for that timing in the backpropagation phase (i.e. if a reward is observed at time $t$ in an episode, credit for that reward is not assigned to states/action after time $t$). It also used a discounting factor $\gamma$ to discount the importance of reward depending on temporal distance, which is uncommon otherwise in MCTS literature.

Both of those differences are due to the original, 2006 paper having more of a "Markov decision process" or "Reinforcement Learning" flavour, whereas otherwise MCTS has especially become popular in AI for (board) games where we often by default assume that there is only a single nonzero reward (win or lose) at the end of every episode anyway, which makes those differences less meaningful.

Both of the AlphaGo papers (AlphaGo and AlphaGo Zero) use a "foundation" of MCTS that is mostly similar to the one from the 2012 survey article. The core components are all the same. Both of those systems add on a lot of (very important, and some quite complex) bells and whistles though.

Going off the top of my head (should be mostly accurate, but best details are in the original source of course!) AlphaGo (your third link) added Neural Networks trained in various ways to output policies (mapping from states to probability distributions over actions) and value functions (mapping from states to "value estimates", or estimates of win percentage). Trained policy networks were used in a variant of the "selection" phase (no longer UCB1 strategy as in UCT) to bias selection. A different (simpler function, not a deep net) policy was used to run play-outs (no long uniform at random action selection as in UCT). A combination of observed reward at end of play-out + value function estimate at start of play-out was used for backpropagation (no longer only backpropagating reward observed in play-out as in UCT).

AlphaGo Zero is, if we're purely looking at the MCTS part, quite similar to AlphaGo. The Neural Networks were a bit different (a single one, with multiple outputs for policy + value), and play-outs were no longer used at all (just immediate backpropagation of value function estimates after MCTS' selection + expansion phases). Apart from that, the primary differences going from AlphaGo to AlphaGo Zero were in the learning process used to train the Neural Networks really, not so much in the MCTS side of things.


I need to predict ball position from set of Images

There are so many challenges here for a novice or even a very seasoned AI developer!  There hasn't been a lot of development in this area.  I assume you are working with broadcast video.  Your biggest challenges will involve occlusions of the ball and changes in the view as broadcasters constantly switch cameras which have very different positions on the field or around the stadium.  You will need to track the ball and the players that take action on the ball.  The ball can be in motion from a player heading, kicking, or throwing the ball.  The ball can also be in motion from a bounce off a player or the ground.  To get started I recommend building off some work that at least can track the players and the ball and can identify certain actions such as a kick.  If you start there you can build some type of ball trajectory predictor.

The following paper Soccer Event Detection, by Abdullah Khan1, Beatrice Lazzerini, Gaetano Calabrese and Luciano Serafini, presents an event detector for "Ball possession" and "Kicking the ball".

A paper entitled The Newton Scheme for Deep Learning by Junqing Qiu, Guoren Zhong, Yihua Lu, Kun Xin, Huihuan Qian, and Xi Zhu, addresses predicting the trajectory of a ball in sports.  They address the complexities of a ball in motion.  They address situations where "the ball is flying with a self-spinning, resulting in a Magnus force on it, which makes it difficult to predict the position by traditional methods".

Some related articles and papers you might find of interest:

Identifying Player Possession in Spatio-Temporal Data
Deep Learning using CNNs for Ball-by-Ball Outcome Classification in Sports
Detecting events and key actors in multi-person videos
Learning to Track and Identify Players from Broadcast Sports Videos
Video Artificial Intelligence Powers Creation
AI predicts how athletes will react in certain situations
How AI-Based Sports Analytics Is Changing the Game


How to design fitness function for multiple objectives?

Your fitness function has two objectives that are added together, but they are not necessarily on the same scale. The component cos(drone_angle) must have a value from 0..1. The component 1/distToTarget will have a range that depends on how you measure distToTarget; e.g. if distToTarget has a range 0..1000, then this part of the fitness function will always be small far from the target (e.g. distance of 500) and massive when it gets very close (e.g 0.1 distance from the target). So the contribution of both components may not always be equal. Another potential complication is that the cosine function is nonlinear and make a very rapid transition between 0 and 1, as opposed to a smooth transition.
I recommend reworking the fitness function to make the two components more equal, e.g. something like
fitness(angle, distance)=
    w1 * |desired_angle - angle| / maximum_angle_error
  + w2 * distance / maximum_distance

In this function that should be minimised to zero, both components contribute linearly to the final fitness, both components have a range of 0..1, and you can tune w1 and w2 to give different importance to different components depending on your preference (e.g. you may prefer w1>w2 if staying upright is the most important).


Are AI researchers more likely to follow "conservative anthropocentrism" or "post-human fundamentalism" in deciding whether AI has moral standing?

Many large deployments of AI have carefully engineered solutions to problems (ie self driving cars). In these systems, it is important to have discussions about how these systems should react in morally ambiguous situations. Having an agent react "appropriately" sounds similar to the Turing test in that there is a "pass/fail" condition. This leads me to think that the current mindset of most AI researchers falls into "Conservative anthropomorphism".

However, there is growing interest in Continual Learning, where agents build up knowledge about their world from their experience. This idea is largely pushed by reinforcement learning researchers such as Richard Sutton and Mark Ring. Here, the AI agent has to build up knowledge about its world such as:

  When I rotate my motors forward for 3s, my front bump-sensor activates.

and

  If I turned right 90 degrees and then rotated my motors forward for 3s, my front bump-sensor activates.

From knowledge like this, an agent could eventually navigate a room without running into walls because it built up predictive knowledge from interaction with the world.

In this context, lets ignore how the AI actually learns and only look at the environment AI agents "grow up in". These agents will be fundamentally different from humans growing up in homes with families because they will not have the same learning experience as humans. This is much like the nature vs nurture argument.

Humans pass their morals and values on to children through lessons and conversation. As RL agents would lack much of this interaction (unless families adopted robot babies I guess), we would require different ways of judging their moral worth and thus "Post-human fundamentalism".

Sources:
5 years in the RL academia environment and conversations with Richard Sutton and Mark Ring.


Is greedy layer-wise pretraining obsolete?

It depends. It could give you a boost or it could not.

Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).

Ideal World: We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder

Reality years ago: Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).

solution: Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step

Cons: Each step is looking for a greedy solution, rather than a deep one.

Reality today: Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)

solution: We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.

I hope this  helped give some form of intuition of the matter.


My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were consecutives.
After redoing my experience selection method, my DQN is actually working and has reached about +200 points after 400000 experiences (about 500 episodes; only 2-3 hours or training!). Before I couldn't reach that score after days of training.
I'll let it train to see if there are something I can improve. Thanks to everyone who tried to help me! I let this answer here just in case someone has the same problem as me.


Positive bias causes the calculation of incorrect gradients

Ok, so I was right in assuming it was the discontinuity at x=0 for relu(x). The positive bias actually had nothing to do with the incorrect gradients. As for why in my edit I was getting such massive differences between gradient values, it was because I initialised the input to be integers, and upon adding a decimal, no matter how small (> 0 though), numpy would jump entire integers, so -2 + 0.0001 = -1.

This is reproduced by:

>>> a = np.random.randint(-3,3,(1,3,3))
>>> a
array([[[-2,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])
>>> a[0][0][0] += 0.0001
>>> a
array([[[-1,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])

The problem with my input is large amounts of it were 0, so it produced a bunch of discontinuous gradients. As such, I simply check both sides of the graph (have epsilon positive then negative), and if they disagree, I set the value to 0, as in that case I'd just ignore any gradient passing through that node.


How can the convolution operation be implemented as a matrix multiplication?

To show how the convolution (in the context of CNNs) can be viewed as matrix-vector multiplication, let's suppose that we want to apply a $3 \times 3$ kernel to a $4 \times 4$ input, with no padding and with unit stride.
Here's an illustration of this convolutional layer (where, in blue, we have the input, in dark blue, the kernel, and, in green, the feature map or output of the convolution).

Now, let the kernel be defined as follows
$$
\mathbf{W} =
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} \\
w_{1, 0} & w_{1, 1} & w_{1, 2} \\
w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
\in
\mathbb{R}^{3 \times 3}
$$
Similarly, let the input be defined as
$$
\mathbf{I} =
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} & i_{0, 3} \\
i_{1, 0} & i_{1, 1} & i_{1, 2} & i_{1, 3} \\
i_{2, 0} & i_{2, 1} & i_{2, 2} & i_{2, 3} \\
i_{3, 0} & i_{3, 1} & i_{3, 2} & i_{3, 3} \\
\end{bmatrix}
\in
\mathbb{R}^{4 \times 4}
$$
Then the convolution above (without padding and with stride 1) can be computed as a matrix-vector multiplication as follows. First, we redefine the kernel $\mathbf{W}$ as a sparse matrix $\mathbf{W}' \in \mathbb{R}^{4 \times 16}$ (which is a circulant matrix because of its circular nature) as follows.
$$
{\scriptscriptstyle
\mathbf{W}' =
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 & 0 & 0 & 0 & 0 \\
0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 & w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 & 0 & 0 & 0  \\
0 & 0 & 0 & 0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 \\
0 & 0 & 0 & 0 & 0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
}
$$
Similarly, we reshape the input $\mathbf{I}$ as a 16-dimensional vector $\mathbf{I}' \in  \mathbb{R}^{16}$.
$$
{\scriptstyle
\mathbf{I}' =
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} & i_{0, 3} & i_{1, 0} & i_{1, 1} & i_{1, 2} & i_{1, 3} & i_{2, 0} & i_{2, 1} & i_{2, 2} & i_{2, 3} & i_{3, 0} & i_{3, 1} & i_{3, 2} & i_{3, 3}
\end{bmatrix}^T
}
$$
Then the convolution of $\mathbf{W}$ and $\mathbf{I}$, that is
$$\mathbf{W} \circledast \mathbf{I} = \mathbf{O} \in \mathbb{R}^{2 \times 2},$$
where $\circledast$ is the convolution operator, is equivalently defined as $$\mathbf{W}' \cdot \mathbf{I}' = \mathbf{O}' \in \mathbb{R}^{4},$$ where $\cdot$ is the matrix-vector multiplication operator. The produced vector $\mathbf{O}'$ can then be reshaped as a $2 \times 2$ feature map.
You can easily verify that this representation is correct by multiplying e.g. the 16-dimensional input vector $\mathbf{I}'$ with the first row of $\mathbf{W}'$ to obtain the top-left entry of the feature map.
$$w_{0, 0} i_{0, 0} + w_{0, 1} i_{0, 1} + w_{0, 2} i_{0, 2} + 0 i_{0, 3} + w_{1, 0} i_{1, 0} + w_{1, 1} i_{1, 1} + w_{1, 2}i_{1, 2} + 0 i_{1, 3} +  w_{2, 0} i_{2, 0} + w_{2, 1}i_{2, 1} + w_{2, 2} i_{2, 2} + 0 i_{2, 3} + 0 i_{3, 0} + 0 i_{3, 1} + 0 i_{3, 2} + 0 i_{3, 3} = \\
w_{0, 0} i_{0, 0} + w_{0, 1} i_{0, 1} + w_{0, 2} i_{0, 2} + w_{1, 0} i_{1, 0} + w_{1, 1} i_{1, 1} + w_{1, 2}i_{1, 2} + w_{2, 0} i_{2, 0} + w_{2, 1}i_{2, 1} + w_{2, 2} i_{2, 2} = \\
\mathbf{O}'_{0}  \in \mathbb{R} ,$$ which is equivalent to an element-wise multiplication of $\mathbf{W}$ with the top-left $3 \times 3$ sub-matrix of the input followed by a summation over all elements (i.e. convolution), that is
$$
\sum \left(
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} \\
w_{1, 0} & w_{1, 1} & w_{1, 2} \\
w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
\odot
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} \\
i_{1, 0} & i_{1, 1} & i_{1, 2}  \\
i_{2, 0} & i_{2, 1} & i_{2, 2}
\end{bmatrix}
\right) = \mathbf{O}_{0, 0} = \mathbf{O}'_{0} \in \mathbb{R},
$$
where $\odot$ is the element-wise multiplication and $\sum$ is the summation over all elements of the resulting matrix.
The advantage of this representation (and computation) is that back-propagation can be computed more easily by just transposing $\mathbf{W}'$, i.e. with $\mathbf{W}'^T$.
Take also a look at this Github repository that explains how the convolution can be implemented as matrix multiplication.


How would one implement a multi-agent environment with asynchronous action and rewards per agent?

The cleanest solution from a theoretical point of view is to switch over to a hierarchical framework, some framework that supports temporal abstraction. My favourite one is the options framework as formalised by Sutton, Precup and Singh.

The basic idea is that the things that you consider "actions" for your agents become "options", which are "large actions" that may take more than a single primitive time step. When an agent selects an option, it will go on "auto-pilot" and keep selecting primitive actions at the more primitive, fine-grained timescale as dictated by the last selected option, until that option has actually finished executing. In your case, you could:

implement the first "primitive action" of an option to immediately apply all effects to the state, and append a sequence of "no-op" actions afterwards to make sure the option actually has a longer duration than a single primitive timestep, OR
implement the very last primitive action of an option to actually apply changes to the state, and prepend a sequence of "no-op" actions in front of it to make the option take more time, OR
something in between (i.e. actually make partial changes to the state visible during the execution of the option).

Since all legal choices for agents in your scenario appear to be options, i.e. you do not allow agents to select primitive actions at the more fine-grained timescale, you would only have to implement "inter-option" learning in your RL algorithms; there would be no need for "intra-option" learning.

In practice, if you only have a small number of agents and have options that take relatively large amounts of time, you don't have to actually loop through all primitive time-steps. You could, for example, compute the primitive timestamps at which "events" should be executed in advance, and insert these events to be processed into an event-handling queue based on these timestamps. Then you can always just skip through to the next timestamp at which an event needs handling. With "events" I basically mean all timesteps at which something should happen, e.g. timesteps where an option ends and a new option should be selected by one or more agents. Inter-option Reinforcement Learning techniques are basically oblivious to the existence of a more fine-grained timescale, and they only need to operate at precisely these decision points where one option ends and another begins.


confusion matrix is useful to be applied to the result of fuzzy c means clustering or not

Yes, you can use confusion matrix as a performance metric for a classifier where the output of the classifier is class labels.

If your classifier output is a likelihood floating point value for each of the classes, then you have to use some scheme of converting the likelihoods to a class label. For example, use softmax function for one hot encoding.

So, for the Fuzzy K Means clustering based classifier, you'll have to pick output class label as the 'index' if the Fuzzy cluster with maximum likelihood/ membership score. This will constitute predicted labels.

Using array of test labels and predicted labels to compute confusion matrix is straightforward in any computational platform, including Matlab of course.

-
Fuzzy K Means (also called fuzzy c means or FCM) is an unsupervised learning method. This means the assignment of training data points to clusters is not based on their class labels. Therefore, FCM is not used as a discriminative classifier.

Typically, FCM is used as an intermediate module to encode data. The encoded representation is used as a feature vector, in supervised learning, to train a discriminative classifier, such as a SVM.

This is not to say you can not use FCM for classification, but it would only work when data points of the same label occur in reasonably distinct clusters. This almost never happens with real world data and so it's recommended not to use FCM for classification.


What is the big fuzz about SHA-RNN versus Transformers?

Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.


MCTS for non-deterministic games with very high branching factor for chance nodes

You can try using an "Open-Loop" MCTS approach, instead of the standard "closed-loop" one, and eliminate chance nodes altogether. See, for example, Open Loop Search for General Video Game Playing.

In a "standard" (closed-loop) implementation, you would store a game state in every normal (non-chance) node. Whenever there is a chance event, you would stochastically traverse to one of its children, and then have a normal node with a "deterministic" game state again.

In an open-loop approach, you do not store game states in any node (except possibly the root nodes), because nodes no longer deterministically correspond to specific game states. Every node in an open-loop MCTS approach only corresponds to the sequence of actions that leads to it from the root node. This completely eliminates the need for chance nodes, and results in a significantly smaller tree because you only need a single path in your tree for every possible unique sequence of actions. A single sequence of actions may, depending on stochastic events, lead to a distribution over possible game states.

In every separate MCTS iteration, you would re-generate game states again by applying moves "along the edges" as you traverse through the tree. You also "roll the dice" again for any stochastic events. If your MCTS iteration traverses a certain path of the tree often enough, it will still be able to observe all the possible stochastic events through sampling.

Note that, given an infinite amount of time, the closed-loop approach with explicit chance nodes will likely perform much better. But when you have a small amount of time (as is the case in the real-time video game setting considered in the paper I linked above), an open-loop approach without explicit chance nodes may perform better.

Alternatively, if you prefer the closed-loop approach with explicit chance nodes, you could try some mix of:

Allowing MCTS to prioritise promising parts of the search tree over parts that have not been visited at all (i.e. do not automatically prioritise nodes with $0$ visits). For example, instead of giving unvisited node a value estimate of $\infty$ (this is how you could interpret the automatic selection of them), you could give them a value estimate equal to the value estimate of the parent node, and just apply the UCB1 equation directly.
Use AMAF value estimates / RAVE / GRAVE in your selection phase. This allows you to very quickly learn some crude value estimates for moves that you have never selected in the Selection phase yet, by generalising from observations of playing them in the Play-out phase. I have noticed that the "standard" implementation of RAVE / GRAVE, without an explicit UCB-like exploration term, does not mix well with my previous suggestion of using a non-infinite value estimate for unvisited children. It may be good to consider a UCB-like variant with an explicit exploration term instead.


How does one even begin to mathematically model an AI algorithm?

Begin by learning the mathematical treatments at their foundations.

Game theory pioneered John von Neumann and Oskar Morgenstern
Information theory pioneered by Claude Shannon (Bell Labs)
Incompleteness pioneered by Kurt Gödel (which led to Alonso Church's lambda calculus and Turing's completeness which led to a general criteria that defines what a programming language must do to be called general purpose.

You could take a course in Finite Math and Discrete Math, but there is no better way to understand than to go to the authors of the original ideas, which is why MIT and Cal Tech require it of freshmen.  We don't need to be genius to understand genius.  We just need to take the time to read.  That's the genius of them.

It is good to evaluate the pruning of decision trees mathematically, and modeling the general and specific cases is the right approach.  Congratulations for seeing that.  The code for pruning is far more developed than the mathematics for modeling it.  The literature usually shows speed results just before the conclusions, which is more like sitting in the audience with pop corn and a wager ticket than understanding the anatomy and proper care of a horse.

Once you understand what occurred a century ago that led to all computer science and see what mathematical conventions originated and in what context then you can read work like

Appl. Math. L&t. Vol. 4, No. 6, pp. 77-80, 1991 Printed in Great Britain. 08939659191 Pergamon Press plc Turing Computability with Neural Nets T. SIEGELMANN AND EDUARDO D. SONTAG Department of Computer Science, Rutgers University Department of Mathematics

... and ...

A Survey of Decision Tree Classifier Methodology S. Rasoul Safavian David Landgrebe TR-EE 90-54 September, 1990 School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 (NASA-CR-188208).

WARNING: Don't be surprised if you are disappointed by most recent work after reading von Neuman, Shannon, and Gödel.


Are there some notions of distance between two policies?

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance are not necessarily metrics in a mathematical sense).
A common measure is the Kullback–Leibler divergence (which is not a metric, in a mathematical sense, given that it does not satisfy certain required conditions for being a metric). For example, in section 4 of the PPO paper, the KL divergence is used as a regulariser (which is actually quite common, for instance, in the context of variational Bayesian neural networks). The TRPO also uses the KL divergence.
The Wasserstein metric has also been used in RL, for instance, in distributional RL (but, in this case, not to compare policies but distributions over value functions).
You can find more info about statistical distances here. The specific distance that you use may depend on the problem that you want to solve and the properties that you want your distance to have. For example, the KL divergence is unbounded above, so, if that's not desirable, you could choose another one. The paper On choosing and bounding probability metrics (2002, by Gibbs and Su) may also be useful. Here I also talk about the KL divergence and total variation.


How to evaluate an RL algorithm when used in a game?

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of comparing it to humans, you might want to compare the game results of all the games played.
Reward Comparison
Often Reinforcement Learning algorithms are compared by using the rewards (either direct, maximum or average in time/iteration). For example, in this page about RL a comparison of two algorithms is shown:

Or, when you know the optimal actions, you can plot the number of plays/iterations against the percentage of actions. See for example this RL comparison on the 10-armed testbed problem:

Henderson et al. 2017 have a whole section about the evaluation metrics of Reinforcement Learning algorithms. They also comment on the plotting of the average or maximum cumulative rewards, moreover they mention the sample bootstrap method to create confidence intervals for a better comparison. Lastly, they mention that the significance of the improvements of the algorithms should be calculated, using a statistical test, such the two sample t-Test. Note that you should take into account the distributions of the datasets to choose the right statistical test. An interesting article related to this is A Hitchhiker's Guide to Statistical Comparison of Reinforcement Learning Algorithms of Colas et al..
Comparison of plays against humans
To find out how well different algorithms play against humans you should do a large number of games and compare - what you consider - important parameters, for example: did the algorithm won, the time it took to win, number of points gained, etc. These values can then be compared statistically. Note that you have to think well about the setup of these experiments since you only should change the algorithms, the other parameters should stay equal. Therefore, you should - preferably - use a large number of subjects (of different ages, sexes, etc. to cover many types of people), and try to prevent any bias; think about the order, how many games are played, location, time etc.


Knowledge required for understanding AlphaZero paper

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon paper (which is pretty accessible, and which is the basis for the other technique used in AlphaZero) should be enough to get a pretty good idea of what they did.


Which artificial intelligence algorithms could use tensor specific hardware?

There are a lot of other potential applications. It's a good idea to start with GPU related problems, since GPUs are essentially doing a slightly wider set of operations, slightly slower. Some possible problems where TPUs might be advantageous are:

Shaders are algorithms for rendering graphics in one style or another. Since computer graphics can be understood as mostly linear algebra, it is natural to view this as operations over tensors.
Physical simulations, which again involve multiplication of vectors by a series of matrices.
Options Pricing, which again involves the multiplication of vectors by a series of matrices, especially when more complex derivatives are prices and the lattice becomes multi-dimensional.

Within AI, there are many other algorithms optimized to work with GPUs, and that could be modified to work with tensor specific hardware. For example, we have:

Genetic Programming on GPUs
Multiagent Systems on GPUs
SAT solving on GPUs

There does not yet seem to be much work optimizing these other problems for Tensor Processing units, but TPUs are also not yet very old. It took several years following the availability of inexpensive consumer GPUs before we started to see widespread use in AI. I suspect we will see more TPU-tailored code for other problems soon.


What will change when workstations will have ARM Machine Learning Processors onboard?

It is not clear to me that much will change.

ARM makes many devices, mostly designed to consume as little power as possible. My guess is that most workstations will not contain ARM's ML processor, even if they contain an ARM CPU.
ARM's ML processor can do machine learning. It is specifically optimized for training convolutional deep neural networks, and it is a bit faster that most existing non-purpose built chips, especially for its price. In practice, chips like this one should make ML a bit cheaper and faster, but it's not a breakthrough product. This just looks like the next iteration of Moore's law.


What are the applications of ant colony optimization algorithms?

The first ant colony optimisation algorithm was introduced by Marco Dorigo in the report Positive Feedback as a Search Strategy (1991) and his PhD thesis Optimization, Learning and Natural Algorithms (1992). He's still one of the leading figures in the field of swarm intelligence (having also written or co-written several papers and books). Another important person that contributed to ACO algorithms is Luca Gambardella (co-director of IDSIA).

There are several ACO algorithms. They are all based on the way real ants behave, that is, by leaving a substance called "pheromone" on the ground in order to communicate. More specifically, the amount of pheromone is associated with value (e.g. food): more pheromone means more value. (It should now be clear the reason behind the queues real ants form).

A list of ACO algorithms can be found at http://iridia.ulb.ac.be/~mdorigo/ACO/publications.html. For reproducibility, here's a (non-exhaustive) list:

Ant System
Elitist Ant System
Ant-Q
Ant Colony System
Max-Min Ant System
Rank-based Ant System
ANTS
Hyper Cube - ACO

ACO algorithms have been applied to combinatorial and NP-complete (e.g. the travelling salesman problem) problems. ACO algorithms are thus a collection of meta-heuristic and probabilistic algorithms (in the same family of simulated annealing) to tackle often considered intractable problems. The related Wikipedia article contains a more exhaustive section dedicated to the applications of these algorithms. ACO algorithms are often combined with local search algorithms (like the 2-opt or 3-opt).

I would suggest you to start with the travelling salesman problem, which was the first application of these algorithms. You can have a look at the reference implementations at http://iridia.ulb.ac.be/~mdorigo/ACO/aco-code/public-software.html, where you can also find software to solve specific tasks (not just the TSP, such as maximum clique problems).


Expected SARSA vs SARSA in "RL: An Introduction"

Why is the action selection random with Sarsa?

A policy could be stochastic. In the case of SARSA, it is stochastic because of the use of $\epsilon$-greedy.

  Isn't it on-policy and therefore ϵ-greedy?

I don't quite understand the question. SARSA is on-policy evaluation with $\epsilon$-greedy policy. Q-learning is off-policy evaluation with $\epsilon$-greedy policy. $\epsilon$-greedy is just a way to turn an action-value function into a policy.

  Because Expected-Sarsa is off-policy the experience it learns from can be from any policy ... How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an ϵ-greedy policy, especially with the same amount of experience?

It is unfair to compare different natures of experiences because off-policy experience contains less useful information. Thus, both SARSA and Expected SARSA should use their own on-policy experience for comparison.

While Expected SARSA update step guarantees to reduce the expected TD error, SARSA could only achieve that in expectation (taking many updates with sufficiently small learning rate). Through this perspective, there is little doubt that Expected SARSA should be better.

  Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?

Same as the previous answer, it is unfair to compare them without the same quality of experiences.


How does DDPG algorithm know about my action mapping function?

I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[0, 1]$. If the network outputted e.g. 10 in the first dimension then this would just be clipped to $2\pi$.
This way the agent only learns about actions within the action space and the weights of the network would eventually be adjusted to only output actions within this action space, provided that the boundaries aren't the optimal actions.


Is the QuickDraw with Google neural net a convolutional neural network?

I believe they don't use CNNs. The most important reason why it's because they have more information than a regular image: time. The input they receive is a sequence of (x,y,t) as you draw on the screen, which they refer as "ink". This gives them the construction of the image for free, which a CNN would have to deduce by itself.

They tried two approaches. Their currently most successful approach does the following:

Detect parts of the ink that are candidates of being a character
Use a FeedForward Neural Network to do character recognition on those candidates
Use beam search and a language model to find most the most likely combination of results that results into a word

Their second approach is using an LSTM (a type of Recurrent Neural Network) end-to-end. In their paper they say this was better in a couple languages.

Source: I was an intern in Google's handwriting team in summer 2015 (on which I believe quickdraw is based), but the techniques I explained can be found in this paper.


Precise description of one-shot learning

The model has learnt the "features" for the type of inputs, eg. faces.
For the problem to be called one-shot, it needs to also correctly classify/compare any new samples. For example, in face recognition application, any new person's images should also result a positive for their own image and negative for any other seen or unseen image.

Since we are using euclidean distance of the final feature layer and not performing any final classification, we can say we are using the weights of a pretrained network and computing final value using that(distance), thus transfer learning. There is no backprop in this, but what you want to do with the embeddings, such as learning a threshold function can be considered learning.


Changes in flow detection neural network?

A network is able to fit to a certain function over several iteration while training. Now you want the model to be able to detect a change in a list of inputs from the function. This is not possible without first training the model on some data.

Say you want to use a simple function

# Sample function
f(x) = x

Say you create inputs for function using sets of 3 integers for x.

# Data
f1 = [[1, 2, 3],
      [4, 5, 6],
      ...      ]

# Some random values
f2 = [[1, 4, 19],
      [16, 35, 36],
      ...      ]

Now if you use data f1 labelled as 0 and data f2 labelled as 1, in the best case the ANN will only learn to differentiate between data from function f1 and some other data.

To detect change, first the model has to fit to a certain function, this requires it to be trained over a number of epochs. Then it will be able to detect if the values don't match the function, but such a model will only be able to differentiate a single function.


What is the difference between assisted and unassisted learning in relation to AI?

Although there doesn't seem to be standardization of the terminology involving assisted learning, I've been noticing this concept, and the distinction, popping up in articles recently, surely driven by important milestones in unassisted machine learning.

My understanding is that Algorithmic Intelligence has traditionally been based on complex decision-making algorithms that are trained to some degree by the programmers.  (In essence, given a bit of a head start by utilizing human knowledge in regard to the problem.)
By contrast, unassisted machine learning is an Algorithmic Intelligence that learns only by its own analysis of models and problems.  No human knowledge goes into the automata, which develops intelligence entirely on its own.

A passage from Matthew Lai's Giraffe Chess paper may provide some insight:

  "This report presents Giraffe, a chess engine that uses self-play to discover all its
  domain-specific knowledge, with minimal hand-crafted knowledge given by the programmer.
  Unlike previous attempts using machine learning only to perform parametertuning
  on hand-crafted evaluation functions, Giraffe’s learning system also performs
  automatic feature extraction and pattern recognition. The trained evaluation function
  performs comparably to the evaluation functions of state-of-the-art chess engines - all of
  which containing thousands of lines of carefully hand-crafted pattern recognizers, tuned
  over many years by both computer chess experts and human chess masters."

DeepMind, for instance, is setting automata on video games and letting the automata learn with no human guidance whatsoever, and getting very good results.

As to why unassisted machine learning is worrisome relates to the issue of human/AI value alignment and the control problem: if we don't know what an an automata is thinking, or even how smart it has become, how can homo sapiens sapiens be assured of positive outcomes for our species?


Both are used, for example,  in the GATE framework, which is still widely used. I suspect that this also applies to many other applications.
I would think that many recent academic publications are now on other approaches, as FSAs and FSTs are fairly established and mature technologies, but I've been out of academia for a while now.


What kinds of systems have so far failed to be modeled via supervised artificial network training?

I haven't seen any dataset where some standard models worked and neural networks utterly failed.

For columnar data (e.g. Excel files / database dumps / CSV files) which contain structured data usually tree-based models like random forests and gradient boosting work better, but neural networks are also usually way better than random.

If you demand other things, e.g. explanations for the decision then Bayesian models might give you an easier time. Or for baselines/simple implementations linear models. Or for real time applications...


How does a google choose it's autocomplete solution

In general, Google autocompletes (and produces search results) based on wide variety of factors, including (but not limited to) your location, your search history, your other Google accounts, your site visit history, your language settings, etc.

For the specific question, I see a few ways in which Google might have access to the relevant information:

If you are syncing your documents with Google Drive it will index the document that most recently changed (which would be the document you are writing) and it could analyze it for patterns to produce relevant search results and suggestions
If you emailed the document recently anywhere via Gmail it could also index the contents as above
Finally, if you are typing the document in Google Docs, it could also have access, but I assume by "Word" you mean Microsoft Word, so this is probably irrelevant in this case

I don't know specifically whether or not Google uses Drive or Gmail contents in the way described, but it would certainly make sense given that it is well known that they do use this information to target advertising.


Does gradient descent in deep learning assume a smooth fitness landscape?

I'm going to take the fitness landscape to be the graph of the loss function, $\mathcal{G} = \{\left(\theta, L(\theta)\right) : \theta \in \mathbb{R}^n\}$, where $\theta$ parameterises the network (i.e. it is the weights and biases) and $L$ is a given loss function; in other words, the surface you would get by plotting the loss function against its parameters.
We always assume the loss function is differentiable in order to do backpropagation, which means at the very least the loss function is smooth enough to be continuous, but in principle it may not be infinitely differentiable1.
You talk about using gradient descent to find the global minimiser. In general this is not possible: many functions have local minimisers which are not global minimisers. For an example, you could plot $y = x^2 \sin(1/x^2)$: of course the situation is similar, if harder to visualise, in higher dimensions. A certain class of functions known as convex functions satisfy the property that any local minimiser is a global minimiser. Unfortunately, the loss function of a neural network is rarely convex.
For some interesting pictures, see Visualizing the Loss Landscape of Neural Nets by Li et al.

1 For a more detailed discussion on continuity and differentiability, any good text on mathematical analysis will do, for example Rudin's Principles of Mathematical Analysis. In general, any function $f$ that is differentiable on some interval is also continuous, but it need not be twice differentiable, i.e. $f''$ need not exist.


In reinforcement learning, is it possible to make some actions more likely?

For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules.
You don't have to use $\epsilon$-greedy based on current Q function approximation, although that is a common choice because it works well in general cases. However, you should always allow some chance of taking all actions if you want the algorithm to converge - if you fixed things so that bad actions were never taken, the agent would never learn that they had low value.
Probably the simplest way to use your initial idea of best actions is to write a function that returns your assessment of which action to take, and use that with some probability in preference to a completely random choice. At some point you will also want to stop referencing the helper function (unles it is guaranteed perfect) and use some form of standard $\epsilon$-greedy based on current Q values.
I have done similar with a DQN learning to play Connect 4, where the agent would use a look-ahead search function that could see e.g. 7 steps ahead. If that was inconclusive it would use argmax of current Q values. Both these fixed action choices could be replaced, with probability $\epsilon$, with a random action choice to ensure exploration. It worked very well. You could replace the look-ahead search in my example with any function that returned "best" actions for any reason.
There are some other ways you can skew action selection towards better looking action choices. You could look into Boltzmann exploration or upper confidence bounds (UCB) as other ways to create behaviour policies for DQN.


How to deal with padded inputs in a fully connected feed forward network?

If you use fully connected neural network, it will be hard to do what you want. There's no inductive bias to easily generalize from different positions of the strings which you need to extract.

I suggest you use Seq2Seq models here: http://cs224d.stanford.edu/papers/seq2seq.pdf


Why is the change in cost wrt bias in neural network equal to error in the neuron?

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both definitions:

$z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
$\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l  = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$.
The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.


What are the different approaches used in Machine Learning?

Things in italics should give you enough googleable terms to start a deeper dive :P.
There are 3 main branches of statistical ML.

Supervised Learning
This approach is taken when a problem can be phrased as associating some $X$ with some $Y$. For example, classifying a picture of a cat ($X$) with the label “Cat” ($Y$). Training in supervised learning usually means presenting some $X$, having an agent predict the label, comparing the prediction with the answer to get an error metric, and finally using the error to update the agent to make better predictions on future $X$s. MNIST is a great example of a classification domain.
1.1 Supervised learning also works with regression when we want to learn an associated number instead of a label, such as the risk of heart disease given years of smoking and weight.

Unsupervised Learning
This type of learning is used when we want to draw inferences from a dataset without labeled responses. It can be used to find hidden patterns or grouping in data.

Reinforcement Learning
The $X$s and $Y$s in supervised learning are i.i.d., meaning that they don’t relate to the previous $X$ that a model has previously observed. However, there are many problems where the most recent $X$, $X$ at time $t$ ($X_t$), is very important to figuring out what $Y$ to predict ($Y_{t+1}$) for the next $X$ ($X_{t+1}$). The field of reinforcement learning (RL) builds on this notion by framing observations as data that is generated by an agent interacting with an environment. Specifically, the field of RL poses the problem of finding a behaviour (a policy) to select actions such that an agent using the policy can maximize a special signal called the reward from the environment. The agent starts in a state of the world ($S_t$) at time $t$ and takes some action ($A_t$) which puts it in a new state ($S_{t+1}$) and gives it a reward ($R_{t+1}$).

Neural networks are a technique for finding features (some kinda notable differences) in data. They can be used in all of the above learning strategies but are not necessary for them although their application has had some very powerful results recently. Deep ___ learning is simply the application of a multilayered neural network on one of the above learning methods.


In vanilla policy gradient is the baseline lagging behind the policy?

So is the baseline expected to be used in the next iteration when our policy has changed?

Yes.

  To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.

You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.

Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that  policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.

For more on this, see Sutton & Barto, chapter 13, section 13.4.


How is the actor-critic algorithm guaranteed to converge?

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms where the critic is a neural network (function approximation). Most convergence proofs assume that the actor and the critic operate at different time scales, but, for example, in the paper A Convergent Online Single Time Scale Actor-Critic Algorithm (2010) this assumption is not made.

In the paper Incremental Natural Actor-Critic Algorithms (2007), the authors propose four different AC algorithms that use function approximation (neural networks) to represent the critic. Three of these proposed AC algorithms are based on natural policy gradients. In section 6 of the extended and more technical version of the mentioned paper, Natural Actor-Critic Algorithm (2007), the authors prove the convergence of the parameters of the policy and value function to a local maximum of an objective (or performance) function, which corresponds to the average reward (equation 2) plus a measure of the temporal-difference (TD) error inherent in the function approximation.


Should I use neural networks or genetic algorithms to solve Gomoku?

For Gomoku, it seems a bit of an overkill to use neural networks or the genetic algorithm as both take a while, and more often than not, don't go how you want it to. The Gomoku game tree is rather large, but you can get a decent AI from minimax, game tree pruning, and a good heuristic function (that includes counting half and full 2s,3s,4s,...etc.) as opposed to mapping out the full space.
If you are unfamiliar with alpha-beta pruning and minimax see this article.
If you really want to use neural networks or genetic algorithms you can for the learning experience. Regarding neural networks, one way to do it is the following:

Define a heuristic function that receives a board state input (sequence of 0,1,2 for empty, black, white), and outputs a 'goodness' value of the board state. The neural network is our heuristic function.
Assuming that the moves in these game are optimal, train off of the difference between the current best move (by your current parameters) and what move your data says is the best. This is how we define our error function! Thus you are minimizing this difference so that what move your neural network says is the strongest is ideally what your game data says is the strongest (Optimization of this error function can be done via backpropagation or genetic algorithm).
Ideally, by this point, you can now use your ('strong') neural network-based evaluation function for your game tree move evaluations instead of hardcoded heuristics.

Of course, this is just one way, and you would need to find game data first.
As a side note, applying a genetic algorithm can occur in a few ways, such as parameter optimization in a neural network as mentioned above or game tree searching, so make sure you are clear how you define the problem setting with it! The same goes for alternative ways to apply a neural network.
Finally, it's helpful to know Gomoku is solved. See this post for others' thoughts and ideas.


There are already programs that have broken free of our control (Morris worm) so that in itself doesn't imply any great computational demands.

Sentience is ill-defined but is certainly not a pre-requisite for a program to do mischief beyond what its creators intend.

It's difficulty to estimate what sort of processing power is required to support human-like intelligence, since we don't know what the most efficient way to achieve that would be.  If the most processing efficient would be to implement a neural network approaching the number of neurons and interconnects of the human brain processing signals at the same rate, the fastest artificial neural network implementations extant are at least 4-5 orders of magnitude short, is thousands of times less power efficient, and doesn't seem to have a realistic way to scale to the number of interconnects required (see this question)


How to use a Generative Adversarial Network to generate images for developmental analysis?

A generative adversarial network is probably not the best approach for generating the images desired.  We can assume from the comments that the data is not collected.  That's a good thing, because a set of rasterized images, labeled with student age or grade is an inferior input form.

It appears that access to a student population is planned or already negotiated, which is also good.

Although the drawing, as it is being drawn, is seen through each student's eyes, the primary features correlated with drawing skill development is motor control, shape formation, and color choice.  If the sheet of paper is placed over a drawing tablet, the tablet's incoming USB stream events are captured to a file, and the color selection is somehow recorded or automatically determined by having students hold the pencil or crayon up to the computer's camera before using it, a much better in natura input stream can be developed.

Pre-processing can lead to an expression of each drawing experience as a sequence of events arranged in temporal order with the following dimensions for each event.

Relative time from the instruction to draw in seconds
Color
Nearest x grid
Nearest y grid
Pressure

Determining color from camera input may be developed using LSTM approaches.

The dimensions of the label for each of these sequences would be those demographics and rankings that would most closely correlated with developmental stages.

Student age
Student gender
Curriculum grade (-1, 0, 1, 2, ... 12, where -1 is preschool and 0 is kindergarten)
Identifier of the drawing instructions given to the class
Grade ranking of the student in the class

The micro-analysis attached to each ELEMENT in the sequence includes these additional dimensions.

Drawing rate of the utensil given by $r = \frac {\sqrt{(x - x_p)^2 + (y - y_p)^2}} {t - t_p}$ where the subscript p indicates the values are drawn from the previous event in the sequence.
Drawing direction given by $\theta = \arctan (x - x_p, \; y - y_p)$
Curvature $\kappa$ calculated using cubic splines or some other data fitting approach
FFT spectrum $\vec{a}$ and Lyapunov exponent $\lambda$ applied to auto-correlation results

This is a modification of the system Google uses to synthesize speech, based on the WaveNet design. In the diagram, the residual function is defined as follows.

$z = \tanh \, (W_{f,k} x + V_{f,k} y) \, \odot \, \sigma \, (W_{g,k} x + V_{g,k} y)$

The development required is that the $\vec{a}$ must now be accompanied with scalars $r, \theta, \kappa, and \lambda$, but the resulting drawings are likely to have many of the hand-eye developmental features of the examples.


Is Kaggle a good place to find projects in ML to work on?

Yes, Kaggle is a great place to start working on ML projects.
You will start using ML algorithms (at first, using scikit-learn or any other ML library of your choice), learn the concepts of cross-validation (very important concept). You will cope with imbalanced classes, various size of datasets. Moreover, you will be able to ask questions and receive feedback from the community.

It's a great place to start and keep learning, but do not limit yourself at using ML algorithms blindly. Try to understand them, even re-write them if you want.

But it's definitely a lot of fun and there are always several competitions running so you can choose whatever dataset you prefer.