Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.

Therefore, methods used in RL to solve MDPs will work to solve contextual bandits. You can either treat a contextual bandit as a series of 1-step episodes (with start state chosen randomly), or as a continuous problem with discount factor zero.

  Can I use the DQN algorithm with TD Target only the observed reward instead of the reward plus the predicted value of the next state?

Yes. That's identical mathematically to having a discount of zero, or having 1-step episodes.

  Can I use a policy gradient algorithm, like REINFORCE or A2C? If yes, should I use a baseline and what this baseline should be?

Yes. Once converted to an MDP, you can use the same baselines in these algorithms as normal (A2C's use of advantage instead of action value is already a baseline). Generally adding a baseline can help reduce variance, so it may still help when applying RL to contextual bandit problems.

  I have seen in the literature that there are some algorithms for contextual bandits such as LinUCB, LinRel, NeuralBandit, etc. And I am wondering why the DQN, A2C and REINFORCE algorithms, which seem to work well in MDP setting, are not used in contextual bandits

There are a couple of reasons that contextual bandit problems are not solved using RL techniques more often:

The goal in contextual bandits is commonly focused on creating a highly efficient online learner that minimises regret. Regret is the long term difference in total reward between always exploiting the best action choice compared to the exploration necessary to find it. Some RL solvers - e.g. DQN - are poor by this metric.
The lack of timesteps and state transitions can be used in algorithm design to be more efficient.
Improvements to RL methods designed to help with sparse rewards and the assignment problem in MDPs are pointless for contextual bandits and may be wasteful or even counter-productive.

Some RL algorithms do resolve to be nearly identical to their contextual bandit counterparts, and have the same performance characteristics e.g. REINFORCE with baseline for 1-step episodes is essentially the Contextual Gradient Bandit algorithm.

It is also worth noting than many problem domains where contextual bandit algorithms do well - e.g. website recommendations and advertising - have research showing that that a more sophisticated MDP model and RL-like approach can do even better. Although that is not quite the same as your question, it typically means extending the model so that timesteps and state transitions are meaningful.


Is there anything theoretically revolutionary about Deep Neural Networks?

The first neural network machine was the stochastic neural analog reinforcement calculator (SNARC), built in the 1950s. As you can see, it's pretty old. After that, there were several advances regarding backpropagation and the vanishing gradient problem. However, the ideas itself are not novel. Simply put, we have the data and processing power today that we did not have back then.

You could look at the Wikipedia timeline.


Deep Q-learning is not performing well when there are several enemies

1) Vary the number of enemies during the training process.
2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.


Given specific rewards, how can I calculate the returns for each time step?

Perfect.

To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11).
$$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$

You'll see that $G_5$ would be written as a sum with no terms in it, since $T=5$.


Is it possible to create a 'logic system' equivalent to humans?

There is of course a vast amount of work in the area of Automatic Theorem Proving, but most of it is simply concerned with proof, rather than human notions of beauty, elegance, parsimony etc.

There has however been some work in this general area over the years:

Douglas Lenat's famous AM ('Amateur Mathematician').
Douglas Hofstadter's NUMBO program for number sequence extrapolation.
A range of publications by Simon Colton
Shalosh B. Ekhad, the automated proof assistant for Artificial Combinatorics created by Doron Zeilberg and credited by him as a co-author on numerous papers.


What is the relationship between fuzzy logic and objective bayesian probability?

This was a somewhat hotly debated question in the 1980s. The debate was more-or-less ended with papers like Cheeseman's In Defense of Probability.

The short answer is that Fuzzy Logic does not just assign a continuous value to sentences, what it does is assign degrees of membership in different fuzzy sets. These degrees of membership range between 0 and 1.

In contrast, probability says: Among the set of values this variable could take on, what fraction of them are in a certain set? This fraction also ranges between 0 and 1.

This might seem like a semantic distinction, but it has deep implications if you try to use these systems to reason about uncertainty.

For example, consider the question "Will it rain tomorrow?". A probabilistic approach would try to determine the fraction of days like tomorrow that have had rain. A fuzzy logic approach will try to determine whether tomorrow is like a rainy day. The distinction becomes obvious if we then ask whether it will not rain tomorrow. The probabilistic approach will try to find the fraction of days like tomorrow where it did not rain. The chance of it raining or not raining will sum to 1. The fuzzy logic approach will try to determine whether tomorrow is like a non-rainy day. Note that tomorrow may be both like a stereotypical rainy day and not like one. There is no firm requirement that these sets be disjoint. This reflects Cheeseman's critique of Fuzzy Sets: they implicitly reject the additive axiom of probability theory, which (as denoted by the name axiom) is something that is fairly unreasonable to reject.

While there are various approaches to make memberships in fuzzy sets more probabilistic in nature, that's the root distinction.

I think the dominant modern view is that fuzzy sets are a great tool when you need to reason about membership in a fuzzy concept. Whether rice is "done" or not isn't a question of something being true or false. It's a degree of membership in the set "done". On the other hand, whether the object in front of a self-driving car is a person or not is either true or false. It is not necessarily a good idea to reason about this in terms of whether the object is partially in the person set or plastic bag set.


Is there a strong argument that survival instinct is a prerequisite for creating an AGI?

Steve Omohudro wrote a paper called Basic AI Drives that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals.

For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.

In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there are reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."


Selecting the right algorithm to predict disease from questions

There is no defined rules for choosing a machine learning algorithm to learn some type of pattern. However, there are some guidelines to help you better select an algorithm which will yield a higher probability of success.

Some important considerations are:

Number of features: This is the number of questions that each patient had to answer.
Number of instances: This is the number of patients that took your survey.
Number of output classes: This is how specific you want your diagnosis of the disease. Is this a yes/no, or a 5-stage progression.

The larger your feature space and the more output classes you have, the higher the complexity of your model. This is problematic because a more complex model will require more instances of data to learn the underlining patterns. You need to have a good balance between the number of examples you have in your dataset and the complexity of your model.

If you have limited data then you will want to stay very far away from deep learning. In such cases, I prefer to use shallow methods such as SVM, Naive Bayes or Random Forests. These techniques have been shown to be able to capture non-linear relationships. SVM is particularly powerful, you can use the kernel trick! This will transform your feature space into a space that makes the differentiation between the classes easier to distinguish. Do not underestimate the power of this algorithm.

If you have a very large dataset (i.e. 100,000 instances) then you will want to use deep learning. Recently, these techniques have been shown to outperform shallow machine learning algorithms in almost all categories where data is plentiful. You will want to start with a shallow neural network and then increase the complexity of your graph as you see fit. By adding additional nodes per layer, or by adding additional layers. Each node you add will increase the number of variables you need to tune, thus increasing the complexity of your model. If your data is expected to have some type of time dependency which often the case in medical data. Then you can use a long-short term memory neural network (LSTM). These are capable of capturing latent relationships. You can also try a stacked auto-encodder.


How many layers exists in my neural network?

tl;dr I'd say your model has 8 layers (5 conv, 3 dense), however a lot of people count layers in other ways. From what I've seen this is by far the most conventional way for counting layers.

Justification

This is an interesting question because its quite subjective. In most cases only the convolutional and dense layers would count from your network. Bach norm, dropout and flatten are usually considered as operations to other layers rather than layers of their own (much like activation functions).

Note: It is debatable whether or not pooling layers are considered to be layers (as they have no trainable parameters) but in most cases they are not considered to be so.

Note 2: Batch norm, on the other hand, isn't usually considered a layer even though it has trainable parameters. Clearly the authors didn't introduce it as a layer, but as a way to normalize, shift and scale the inputs of a layer. This is apparent in some of the examples below which don't count batch norm as an actual layer.

Note 3: Conventionally all networks are considered to have [at least] one input layer but it doesn't count as a layer.

Examples

Some examples that follow this reasoning when counting layers are the following. I'll also write the pooling layers in each, but they clearly aren't considered as layers be the authors. When available I'll also write the number of layers that keras registers from their official implementations:

The ResNet-50 architecture has 50 layers (49 conv, 2 pool, 1 dense), however keras registers it as 177 layers. ResNets also use batch normalization after each convolution (so 49 batch norms in total), but clearly don't count them as layers.

The Resnet-34 has 34 layers (33 conv, 2 pool, 1 dense). Like the previous, this also uses batch norm but doesn't count it.

VGG-19 has 19 layers (16 conv, 5 pool 3 dense). Keras registers this as 26 layers.

AlexNet is considered to have 8 layers (5 conv, 3 pool, 3 dense).


What are the reasons behind slow YOLO training?

I think you underestimate the size of YOLO. This is the size of one segment of yolo tiny according to the darknet .cfg file:

Convolutional Neural Network structure:
416x416x3                Input image
416x416x16               Convolutional layer: 3x3x16, stride = 1, padding = 1
208x208x16               Max pooling layer: 2x2, stride = 2
208x208x32               Convolutional layer: 3x3x32, stride = 1, padding = 1
104x104x32               Max pooling layer: 2x2, stride = 2
104x104x64               Convolutional layer: 3x3x64, stride = 1, padding = 1
52x52x64                 Max pooling layer: 2x2, stride = 2
52x52x128                Convolutional layer: 3x3x128, stride = 1, padding = 1
26x26x128                Max pooling layer: 2x2, stride = 2
26x26x256                Convolutional layer: 3x3x256, stride = 1, padding = 1
13x13x256                Max pooling layer: 2x2, stride = 2
13x13x512                Convolutional layer: 3x3x512, stride = 1, padding = 1
12x12x512                Max pooling layer: 2x2, stride = 1
12x12x1024               Convolutional layer: 3x3x1024, stride = 1, padding = 1

.cfg file found here: https://github.com/pjreddie/darknet/blob/master/cfg/yolov3-tiny.cfg

EDIT: These networks generally aren't specifically designed to train fast, they're designed to run fast at test time, where it matters


It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.

The approaches/optimizations you could consider:

To make scene change detection and take short piece out of each
To introduce some kind of “novelty” metric and try to maximize
it to get most different parts of video
To convert video to kind of vector with existing solution like
r-cnn and yolo and then process it with recurrent networks.
The task seems to be very close to video capturing/summarization, you
can take inspiration there
Also, the attention approach might be handy, look, for example
self-attention for video,
semantic attention for
video


Why is tic-tac-toe considered a non-deterministic environment?

The game of TIC-TAC-TOE can be modelled as a non-deterministic Markov decision process (MDP) if, and only if:

The opponent is considered part of the environment. This is a reasonable approach when the goal is to solve playing against a specific opponent.

The opponent is using a stochastic policy. Stochastic policies are a  generalisation that include deterministic policies as a special case, so this is a reasonable default assumption.

An action will lead to a known state of the game and an agent has full knowledge of the board and of Its enemies past moves.

Whilst this is true, the next state and reward as observed by an agent may not be due to the postion it plays in (with the exception being if it wins or draws on that move), but the position after the opponent plays.
It is also possible to frame TIC-TAC-TOE as a partially observed MDP (POMDP) if you consider the opponent to not have a fixed policy, but to be reacting to play so far, perhaps even learning from past games. In which case, the internal state of the opponent is the unknown part of the state. In standard game playing engines and in games of perfect information, this is resolved by assuming the opponent will make the best possible (or rational) move, which can be determined using a search process such as minimax. When there is imperfect information, such as in poker, it becomes much harder to allow for an opponent's action.


Is the field of RL is really stochastic approximation theory in disguise? Is RL just a less rigorous version of stochastic approximation theory?

No, but reinforcement learning (RL) is based on stochastic approximation theory (SAT), and these two fields overlap.

In RL, you typically assume that the underlying problem can be modeled as a Markov decision process (MDP), and the goal is to find a policy (or value function) that solves this MDP. To find this policy, you can use stochastic approximation algorithms, such as Q-learning, but RL isn't just SAT, where, in general, there isn't necessarily a notion of MDP.

SAT is the study of iterative algorithms to find the extrema of functions by sampling from them and under which conditions these iterative algorithms converge. SAT isn't just applied in RL, but it is applied in many other fields, such as deep learning. The paper Scalable estimation strategies based on stochastic approximations: Classical results and new insights (2015) by P.  Toulis et al. provides an overview of SAT and the connections with other fields (including RL).

To conclude, RL is based on SAT, but RL isn't just stochastic approximation algorithms, so they are distinct fields. If you want to study e.g. the convergence properties of certain RL algorithms, you may need to study SAT. In fact, for example, the typical proof of convergence for tabular Q-learning assumes the Robbins–Monro conditions. However, you can do a lot of RL without even knowing that RL is based on SAT. Similarly, you can do a lot of SAT without ever caring about RL.


Neural Network Optimizers in Reinforcement Learning non-well behaved environments

The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an amsgrad flag to True in the construction of an ADAM optimizer, and I believe is often also already set to True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.

RMSProper is also still quite popular, and there may be some arguments that it might be better suited for non-stationary learning problems (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).

All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.


Is there any research work that attempts to combine neuroevolution with deep reinforcement learning?

There are 3,810 articles easily found in an academic article search. These are three examples.

Neuroevolution for reinforcement learning using evolution strategies — C Igel — The 2003 Congress on Evolutionary Computation, 2003, ieeexplore.ieee.org — "We apply the CMA-ES, an evolution strategy which efficiently adapts the covariance matrix of the mutation distribution, to the optimization of the weights of neural networks for solving reinforcement learning problems. It turns out that the topology of the networks considerably ..."
Neuroevolution strategies for episodic reinforcement learning — V Heidrich-Meisner, C Igel — Journal of Algorithms, 2009, Elsevier — "Because of their convincing performance, there is a growing interest in using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by the covariance matrix adaptation evolution strategy (CMA-ES), a randomized variable-metric ..."
Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning — FP Such, V Madhavan, E Conti, J Lehman — 2017, arxiv.org — "Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement ..."

To clarify the strategy proposed, we can rewrite the approach as a set of design features. Let's not assume that topology formation is based on weights, which diminishes the concept of morphology and topology. If we were to reduce the model to a traditional artificial network of orthogonal topology, it would then not be neuroevolution; it would then be basic machine learning.

Search for the best network topology through neuroevolution
Train the best candidate selected above through Q-learning

The second item doesn't seem address the relationship between the inputs, outputs, and objectives of neuroevolution designs and those of Q-learning and other reinforcement learning strategies. Q-learning algorithms are not designed to run on feed forward networks in general and certainly not easily mapped to the topologies that may form during neuroevolution. There are probably billions, if not an infinite number, of ways to combine the two strategies, but simplistic concatenation of the two processes is not possible without further research and consideration of how they will interrelate collaboratively toward program goals.

It may be useful to search for articles, study, and then formulate your research trajectory. It is recommendable to start with learning about neuroevolution and reinforcement independently, and then start reading articles like the above three. Pour the foundation, let it dry, and then frame the house.


What could cause a big fluctuation of the loss in the last epochs of training an AlexNet?

I solved my problem with data augmentation and decreasing learning rate value and more epochs.


Can the addition of dropout in a non-overfitting neural network increase accuracy?

Can the addition of dropout, in a non-overfitting neural network, increase accuracy?

Yes, maybe.

  Even if I increase the complexity of the neural network?

Yes, maybe.

As always when making changes to ML algorithms, you need to test carefully to see if your changes have made an improvement. There are very few theories in non-linear machine learning models that make solid guarantees of results. One general difference you should note is that training a network with dropout will take longer (more epochs) than training a similar network without dropout, to reach the same levels of accuracy.

However, as well as the regularisation effect of dropout, it shares some behaviour with ensemble techniques such as bagging. Dropout effectively trains many sub-networks (that share weights) on different samples of the training data. This pseudo-ensemble effect can boost accuracy, and other success metrics. This is not a guaranteed effect, but it does happen in practice.


Calculation of GPU memory consumption on softmax layer doesn't match with the empirical result

GPU DRAM capacity - 7.5G
the below link explains how nVIDIA GPU cUDNN does memory optimization.
https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/
the below link has detailed steps to calculate memory required by parameters and data.
http://cs231n.github.io/convolutional-networks/#case
one data point missing in question is number of output classes in softmax layer.
The above two links will help you to calculate memory required and how software handles large matrix multiplication.


What is the current state-of-the-art in unsupervised cross-lingual representation learning?

The blog post Unsupervised Cross-lingual Representation Learning (2019), the related paper and slides by Sebastian Ruder (a researcher currently at DeepMind) summarize what you are looking for. In fact, the authors write

  We will introduce researchers to state-of-the-art methods for constructing resource-light cross-lingual word representations and discuss their applicability in a broad range of downstream NLP applications, covering bilingual lexicon induction, machine translation (both neural and phrase-based), dialogue, and information retrieval tasks. We will deliver a detailed survey of the current cutting-edge methods, discuss best training and evaluation practices and usecases, and provide links to publicly available implementations, datasets, and pretrained models and word embedding collections.


How do I know that the DQN has learnt an appropriate Q function?

DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)


What is the use of the seed function in the gym Environment 'Pendulum-v0'?

Reinforcement Learning experiments are empirical and you want to be able to reproduce your experiments. Random numbers are generated off a seed; with the seed function you are fixing the seed so the RNG function produces the same sequence of random numbers.


Using a neural network for learning a Motion Graph?

... the neural network has to learn the transition probability between two keyframes. And a second neural network can then produce the motion plan which is also be trained by a large corpus. Is that idea possible or is it the wrong direction?

That would be trivial.

More difficult, but not hard, would be to take a 14 hour video of someone and create a new video with them saying something they have never said previously. Here is a demonstration, from the BBC YouTube channel, titled: "Fake Obama created using AI video tool - BBC News".

The University of Washington demonstrates, using a variety of clips and only audio from an impersonator, that they can create realistic videos of the former president saying anything they want.

It's possible to sketch out an idea and let AI create a finished product of excellent quality.

The YouTube channel ACMSIGGRAPH has a video titled: "Technical Papers Preview: SIGGRAPH 2018", which is discussed on the webpage: "Could These Be The Next High-Tech Tools That Animators Use Daily?".

What you have asked about was possible long ago and not particularly difficult to do.


The use of the Golden Ratio is an interesting suggestion which has intrigued many lovers of the mathematical beauty represented in nature and in AI. The problem lies in the foundations of the AI applications. For example, in designing algorithms for recognizing naturally occurring phenomena, such as face recognition or human body movements (See  https://www.intechopen.com/books/machine-learning-and-biometrics/a-human-body-mathematical-model-biometric-using-golden-ratio-a-new-algorithm) it is suitable. However, for non natural occurrences, the ratio is limited since the data is usually random or chaotic. However, in order to create a master algorithm for the future which encompasses all the best of the current AI algorithms, the use of mathematical concepts such as the golden ration and fractals will be vital. Watch this space...


Which NN Arch? - Given input vec X -> Output vector Y made of "not binary", floating point values

This is a regression problem instead of a classification problem. For both, you can use a feedforward neural network. These are the typical ones you've probably seen where each neuron in a layer is connected to each neuron in the next.

In a classification problem, you want an output like you mentioned above, a one-hot vector corresponding to the correct class. The way you get this vector is by applying a soft-max activation to the outputs of your last layers. If you don't apply this activation, your output vector might be something like [-2.5, 0.6, 10.2, 5.1]. You can see that this looks a lot like the vector you want. So, you can probably use a feedforward architecture (with non-linear activations on the hidden layers if there are non-linearities in your data) with a final layer of 4 neurons. Then, when you train, that should (hopefully) converge to the correct data outputs. If it doesn't, you will need to mess with the architecture/get more training data/lots of other levers to play with.

This link goes to a tutorial for regression using Keras. Keras is a nice package written to take powerful libraries like TensorFlow and Theano and give them a simple interface. TensorFlow can be quite complex (I've never used Theano so I don't know) when you're starting and trying to understand the syntax and the ML concepts. Keras helps take away the confusing syntax.


Corner detection algorithm gives very high value for slanted edges?

I can't go into details of the algorithm but here's some intuition about what's apparently going wrong:

The Sobel transformation identifies mostly-vertical and mostly-horizontal edges. For slanted edges, it also shows a response, just a bit weaker.

By using a window and taking the minimum of vertical and horizontal response, you identify points where you have both a vertical and a horizontal response. However, this is also the case at the slanted edges.

Given the structure of this simplistic corner detection algorithm, you can only expect to get meaningful results for strictly horizontal and vertical lines/edges and their intersections/corners. It is just not suitable for slanted edges.


Neural network for reinforcement learning

From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called Policy Gradient methods, which directly optimizes the policy network $\pi(a_t|s_t)$ from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents high variance.

The methods that you recognize as the ones using two neural networks correspond to a family called Actor-Critic methods. This type of algorithm uses the trajectories of rewards to estimate a value network $q(s_t,a_t)$ (called the critic), and, contrary to the previous family of methods, it uses the value network to train the policy network $\pi(a_t|s_t)$ (called the actor), instead of directly using the trajectory of rewards. This indirect dependence usually makes variance smaller and also learning faster. I recommend you have a look at chapter 13 of the book An Introduction to Reinforcement Learning.

So, to answer your first question: it seems you are missing the family of Actor-Critic methods. I recommend you learn about them since they are very powerful (e.g., read about DDPG or SAC).

About your second question, the standard method to "reward" a policy network is not by training it. Usually, you have a reward function $r(s_t,a_t)$ that depends on your current state $s_t$ and action $a_t$ and you modify the parameters $\theta$ of your network in such a way that the probability of an action $\pi(a_t|s_t)$ increases if the reward is positive or decreases if it is negative. More specifically, you perform stochastic gradient ascend steps like this one:

$$\theta_t\leftarrow\theta_t+\alpha\mathbb E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]\nabla \log\pi(a_t|s_t,\theta_t)$$

What this formula says is that if in the time-step $t$ you take an action $a_t$ in the state $s_t$, wait $T$ steps and collect the rewards from $r(s_t,a_t)$ to $r(s_{t+T},a_{t+T})$, then, you should modify your parameters in the direction that the policy increases the most (i.e., $\nabla \log\pi(a_t|s_t,\theta_t)$) if the expected return $E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]$ is positive, or in the direction that the policy decreases the most if that expected return is negative.


Is time/space estimation of possible actions required for creating AGI?

Time - probably yes; Space - probably no

Every single thing could be time dependent but there is no such thing on which time depends except considering space (black holes, bending of space time), Now we can argue that it can compare memories of present to its previous memories relatively and then call this relativeness time but even there the AGI is putting a time stamp when calling something previous.
Now lets consider how time could be dependent upon space. In this case we can see things getting compared to how much light travels like how the SI unit of time is defined, but that is a really complex observation for a discrete system and it has to go through a lot of time dependent phenomena to reach the level to make the observation.

Space - We can reduce this question by saying that if it could learn the concept of 1-D it could learn about space itself. 1-D space can be compared to a number line, assuming the agi knows the concept of numbers it can compare the differences in location with a number line .And if agi knows the concept of time  it can learn space estimation by always assigning velocity to objects.
The difference between time and space is, time is always moving and there is nothing to compare to.

P.S. Sorry for bad english, english is not my first language.


What is the correct notation for an operation that applies to each element of an array independently?

The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:
$$ \mathcal{L} = - \frac{1}{4} F_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}_i y_{ij} \psi_j \phi + h.c.) + |D_\mu \phi|^2 - V(\phi) $$
But if you try to expand all the indices in all the objects above - then it barely fits on a page.
On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example $f(x_i)$ can be read as:
$$f(x_0, x_1, \dots, x_N)\quad \text{or as}\quad f(x_0),f(x_1), \dots, f(x_N)$$
One way to implicitly resolve this ambiguity is to show that the index $i$ "escapes" the argument brackets:
$$a_i = f(x_i)\quad \text{or e.g.} \quad \sum_if(x_i)$$
This can only be interpreted as $f(x_i)$ being element-wise. Also, at least in my opinion, using $x_i$ with index and $x$ without index in the same expression is extremely confusing.
And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets $f[x_i]$ or capital letters $F(x_i)$ the vector-argument functions.


What is the most appropriate ML algorithm for creating recommendations

You can use Collaborative Filtering, and specifically its memory based approach.
The problem that you have discussed in the question should probably be solved using User-Item collaborative filtering which will calculate similarity between users and then recommend the item. The similarity can be be calculated using cosine similarity or Pearson's similarity formulae.
The best thing about this approach is that no training is required here but if you wish to have very large data, this approach's performance decreases.


how fast does need to be an AI agent to be efficient?

My understanding of efficiency in this context is in regards to optimization of algorithms as opposed to hardware speed, which is more of a "brute force" component.  GPUs may be more energy efficient, but this is distinct from linear optimization of algorithms.

In terms of how much processor speed you need to tackle a given problem, that's in the realm of computational complexity theory and analysis of algorithms.

Amount of GPUs, RAM and storage needed to tackle a given problem are purely a function of the complexity of the problem and the efficiency of the algorithms.


Selecting a Scenario for Comprehension's Sake

Movement through a modeled reality is an area that was under development when I entered the research community.  It is not a problem though.  It is a nearly infinite set of problems, an area of research of interest to robotics engineering and gaming.

The solution cannot be very specific when so many details are left out of the scenario definition.  Although I am fine with specifying a general solution, pure math and system architecture is not met with much enthusiasm by most of those interested in answers in this exchange, so I will make some assumptions.

Those with good mathematical and systems design backgrounds will be able to extrapolate the general approach from the more specific scenario in this answer.  I'll place pertinent general theory inline to facilitate that generalization.

Narrowing the Specification Incompletely

The vehicle for movement is not specified, so I will assume the participants are knights on horses.  Linear movements in Cartesian coordinates as in a CAM move command for a CNC machine head is an unlikely use of reinforcement, and aircraft lift can stall, complicating the problem.  Horses were picked from among the remaining list of typical vehicles (cars, bicycles, on foot, and horses).
Time constraints were not specified, so it will be assumed there are none except the time to reach the goal cannot be infinite unless there is no available path at all to the destination.
The nature of the obstacles and their movements were not specified, so I will assume they are solid and that only other knights move on their own horses.
Risks associated with making contact with an obstacle was not specified, so I will assume the horses will not run into one another or stationary objects and will step over small ground objects or jump over slightly larger ones if approaching at sufficient speed.
What will happen when a collision is about to occur was not specified, so I will assume that the horse will decelerate.
What will happen if a horse passes under an obstacle will be assumed the hospitalization or death of the knight, since they cannot easily duck in their armor, leaving the objective unmet.
The capabilities of the participants is not specified, so I will assume that the knights can see stereoscopically, are equipped with compasses, can turn their head and eyes, can control the reigns in the usual horseback riding way, can urge the horse forward with the vocal sound, "Ya!" and/or a light poke with both feet, hear and detect the volume of an auditory beacon, and sense acceleration in 3D to detect horse behavior.
The specs of the other participants is not given, so I will assume the knights along with their horses are of equal volume.
The obstacle statistics are not given, so I will assume the objects to be chaotically sized, shaped, colored, and placed such that their total volumetric space consumption is 1% and the mean object volume is the same as the volume of a knight and her or his associated horse.
The scene is lit from one distant source so that shadows and shadowing are present and the entire scene is bounded by constants in altitude, longitude, and latitude dimensions.
The coordinate system is not specified, the system is modeling equatorial terrestrial space, so latitude and longitude coordinates are not substantially ansiotropic.
The start and end positions are of specific latitude and longitude and are sufficiently distant from one another so that line of sight between the start and end positions is extremely unlikely.
How the end position will be known is not specified, so I will assume the end position is equipped with an auditory beacon.
We will assume a zero sum game, where the achievement of the goal is not mutually exclusive, and we will assume collaboration linguistically is not possible, although collaborative strategy may emerge out of learned behavior organically.  (That's an entire other topic.)

Mathematical precision is missing in the above definition, with several parameters only roughly defined, and the chaotic sizing, shaping, and positioning becomes a realistic challenge for reinforcement software engineering for the general scenario.

Conversely, engineering a solution for this specific case within the field of VR motion with the application of reinforcement concepts can proceed without mathematical abstractions that require much advance (and advanced) study and laboratory experience.

Generalization

The above adequately defines system E (environment) in combination with system V (vehicles) of which there are N, one for each participant.  Discrete changes to command signals C are multidimensional and A (acquired samples) are also multidimensional.

Acquisition Channels

Sampled audio vectors of spectral spectral distributions in musical half tones (frequency ratio of 21/24), sampled in frames of constant period.
Sampled visual matrices in yuv420p form, sample
Sampled tactile vectors of 3D acceleration force

The audio and  vectors are maintained at constant levels as inputs to the base neural net until the next vector is acquired.  The video matrices are fed into a convolutional neural net as is customary.  (See the work of Google researchers Sergey Levin, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen in August 2016 for their approach and references for background research that guided them.)

Control Channels

Head position relative to the vehicle direction
Left reign position
Right reign position
Activation of, "Ya!" vocalization
Activation of the light poke

The output layer of the base neural net must be a real number with the appropriate range for all five, since the volume of the vocalization and the lightness of the poke have informational meaning to the motor control of the vehicle.

Real Time Learning

Real time learning requires at least one model of wellness to provide the reinforcement signal to the base behavioral network.  In many cases, as suggested in early cybernetics work prior to the advent of digital systems, convergence requires more than one wellness variable.

Vector control of reinforcement is not well developed in open software yet, however the concepts of multidimensional gradients (Jacobian and Hessian matrices) are standard elements in artificial network theory and can be extended from matrices to cubes.  Any intermediate calculus text will provide the theory applicable to gradient descent with a curved surface.

Performing the back-propagation effectively when more than one degrees of freedom are present is an interesting problem that is certainly researched and has been deployed to production in commercial and military applications.

Such cannot be described here because the mechanics of back-propagation with vector reinforcement signaling of which I am aware is currently either company confidential or classified.  As with much technology, it may be released for publication in the future as open source code emerges independently over time.

These are probably the best choices for the first and second channels of learning reinforcement signaling to implement and tune.  I doubt (but have only intuition to offer as reasoning) that only one channel will produce a very effective reinforced learning.  The base network necessary for this scenario will be too deep to train with out both types of proximity estimations (1) beacon and (2) nearest obstacle.

Wellness of Behavior Modelling

The simplest and first cut at modelling participant behavioral wellness (proficiency in searching for the beacon) is the rate of change in beacon volume.  It is a distorted estimate of proximity to the beacon, but systematically so.  The distortions in the correlation between the differential of volume with respect to position (not time) is related to the effects of obstacles on sound.

Wellness of Position Modelling

Further development could add filtering out of transient attenuation created by close proximity to an object, which would require two additional neural nets, (a) to detect closest object proximity from patterns in audio and visual inputs and (b) to use acceleration to approximate change in position in the latitude-longitude plane and correlate filtering of audio volume changes with movement to better filter out transient changes not related to proximity of the beacon.

The tone of the beacon may be added as a third to improve rate of learning.

Crash level accelerations can be added over multiple game plays as a fourth.

Additional Determinations

The determination of initial state and meta-parameters for the base neural network and the connectivity between the models and the reinforcement signal to the base network is beyond the scope of this answer, requiring experimentation and possibly months of intensive analysis for this (or any) semi-specific case.

The only known systems that handle general cases without defining environment E, participant P, and their quantity N are DNA based systems that have developed such general adaptive capabilities over billions of years.


The speaker argued that a dream is a random addition of memories, just as experience replay.

The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. We don't even have strong consensus on why sleep is a necessary feature of animals, let alone what part dreaming plays in it. However, there are some widely-accepted theories, with supporting evidence, that dreams are part of a learning and memorisation process. Studies that manipulate sleep or dreaming have shown changes in the speed that skills are learned for example.

Experience replay in reinforcement learning is a far more precise and well-understood affair, whereby individual time steps that occurred in the past are visited and re-assessed in light of current knowledge about long-term value, at random. If dreams were really like experience replay as it is practiced in RL today, then they would consist of a random jumble of tiny seemingly inconsequential events strung together, and all taken very exactly from the events of the past day. Sometimes dreams do contain content like this, but typically the content is far more varied.

Taken with a large dose of artistic license, then yes, the speaker is referring to real theories and conjectures about dreaming, that do have scientific support. Although it is equally good to draw parallels between dreams and a higher-level management of the memory or experience replay data - which items to replay, and which to keep, depending on what is salient about the information. For instance, there is good evidence that dreams help filter what is forgotten, and also evidence that events associated with strong emotional state are more likely to feature in dreams.

It is important to separate the speaker's analogy, and any suggestion that a current reinforcement learning agent has a subjective experience. We are still a long way away from anything like that, and other similar use of a dreaming metaphor in machine learning - e.g. "Deep Dream" - is equally not an assertion that the devices are having an experience of any kind.


How do I classify strings with possibly no meaning?

I would just use a dictionary. A simple list lookup would tell you whether it's a recognised word or not. As an added bonus you can add some basic language processing, eg to identify inflected forms without listing them in your dictionary. Or use regular expressions to recognise ID numbers. ML is not really the right tool here.


How to decide Linear Separability in my Neural Net work?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, there's no problem to try and run this on a feed forward network. However, the way you give letters as input will be very confusing for a network and will make learning very hard. I'd recommend using one hot encoding or even a binary encoding for the letters. If this is for more than playing around I'd try and add some info (encode whether the letter is in "aeiou" in a separate bit).

As for the hidden layers, try playing around a bit. Two systematic approaches are to start very simple and make the model more complicated, or start complicated and make your model simpler (or just normalize a lot). Look at the performance on the training set and on a separate validation set during training. If the model keeps on improving on the training data but starts to deteriorate on the validation data, you're probably over fitting. In this case you should either make the model simpler (fewer nodes, fewer layers) or regularize (start with l2 normalization on the weights). If the data doesn't perform well on the training data, you may wish to make the model more complex.

Once you've tried the feedforward network, really do try CNN or RNNs for this task.


Possible inconsistency in the Policy Improvement equation

Why are they comparing state value function to action value function?

It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the values at the same $s$ and modifying how $a$ is selected, the proof makes assertions about how that choice impacts the value.

It is important to recall that $v_{\pi}(s)$ measures the expected future reward when starting in state $s$ and following policy $\pi$, and that $q_{\pi}(s,a)$ measures the expected future reward when starting in state $s$ and taking action $a$, thereafer following policy $\pi$. When $a$ is chosen using a deterministic $a = \pi(s)$ then $v_{\pi}(s) = q_{\pi}(s,\pi(s))$

  Isn't it obvious the above equation might hold true

The inequality (it is not an equation) does strictly hold true, because the equation $v_{\pi}(s) = q_{\pi}(s,\pi(s))$ is true. However, that is not terribly useful, it doesn't prove anything new.

What is interesting is if you change the decision for $a$

  (provided we select the best action among the possible actions)

You cannot do that. The policy decides the actions, by definition. Selecting a better action according to the action value function, and showing what that does is precisely what the proof is showing.

  since then the equation will change to $q_\pi(s, \pi(s)) \geq v_\pi(s)$

$q_\pi(s, \pi(s)) = v_\pi(s)$ - yes your inequality holds, but is equivalent to not changing anything.

  and we know $v_\pi(s) = \pi(a|s)q_\pi(s, \pi(s))$?

Here you have switched to using a non-deterministic policy and a deterministic policy in the same statement (you also have not defined $a$ properly), making your equation badly formed. The correct form would be:

$$v_\pi(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s)q_\pi(s, a)$$

This is not a relevant form for the initial policy improvement theorem. However it does become relevant when the theory is adapted to show improving $\epsilon$-greedy policies later.

  NOTE on Convention: As per the convention of the book goes, I think they are using rewards for state to action to state transition sequence rather than state to action transition.

That is not a convention in the book. In the book, the reward distribution is generally given as joint distribution with state transition in $p(s', r | s, a)$ - this is a very generic approach that models any discrete MDP, regardless of how the rewards are described as associated with any of $s, a, s'$ or a random factor.

Possibly the convention you are referring to is labelling the immediate reward as $R_{t+1}$, so that a small section of trajectory starting from state $S_t$ might be $S_t, A_t, R_{t+1}, S_{t+1}$ . . . some other texts will use $R_t$ here. However, that is not relevant to the policy improvement theorem, it would hold the same under either convention, just with slight relabelling of the reward indices.


Back-of-the-envelope machine learning (specifically neural networks) calculations

I have one to share. This is no formula, but a general thing I have noticed.

The number of neurons + neurons should be proportionate, in some way, to the complexity of the classification.

Although this is fairly basic and widely known, it has helped me in many times to consider one thing: how many at a minimum does it need?


Who first coined the term "artificial general intelligence"?

According to Ben Goertzel, the first person that probably used the term "artificial general intelligence" (in an article related to artificial intelligence) was Mark Avrum Gubrud in the 1997 article Nanotechnology and International Security. Here's an excerpt from the article.

  By advanced artificial general intelligence, I mean AI systems that rival or surpass the human brain in complexity and speed, that can acquire, manipulate and reason with general knowledge, and that are usable in essentially any phase of industrial or military operations where a human intelligence would otherwise be needed. Such systems may be modeled on the human brain, but they do not necessarily have to be, and they do not have to be "conscious" or possess any other competence that is not strictly relevant to their application. What matters is that such systems can be used to replace human brains in tasks ranging from organizing and running a mine or a factory to piloting an airplane, analyzing intelligence data or planning a battle.

Note that the term "AGI" could have been used even before that Mark A. Gubrud's article (as Goertzel also suggested).

In any case, Ben Goertzel help to popularise this term, especially with the book Artificial General Intelligence. AGI was previously known as "strong AI", which goes back to John Searle's Chinese Room argument, although strong AI often refers to an AGI with consciousness, and the definition of AGI doesn't necessarily imply consciousness.

You can read more about the history of the term "AGI" in this Goertzel's blog post.


How to define states in reinforcement learning?

The problem of state representation in Reinforcement Learning (RL) is similar to problems of feature representation, feature selection and feature engineering in supervised or unsupervised learning.

Literature that teaches the basics of RL tends to use very simple environments so that all states can be enumerated. This simplifies value estimates into basic rolling averages in a table, which are easier to understand and implement. Tabular learning algorithms also have reasonable theoretical guarantees of convergence, which means if you can simplify your problem so that it has, say, less than a few million states, then this is worth trying.

Most interesting control problems will not fit into that number of states, even if you discretise them. This is due to the "curse of dimensionality". For those problems, you will typically represent your state as a vector of different features - e.g. for a robot, various positions, angles, velocities of mechanical parts. As with supervised learning, you may want to treat these for use with a specific learning process. For instance, typically you will want them all to be numeric, and if you want to use a neural network you should also normalise them to a standard range (e.g. -1 to 1).

In addition to the above concerns which apply for other machine learning, for RL, you also need to be concerned with the Markov Property - that the state provides enough information, so that you can accurately predict expected next rewards and next states given an action, without the need for any additional information. This does not need to be perfect, small differences due to e.g. variations in air density or temperature for a wheeled robot will not usually have a large impact on its navigation, and can be ignored. Any factor which is essentially random can also be ignored whilst sticking to RL theory - it may make the agent less optimal overall, but the theory will still work.

If there are consistent unknown factors that influence result, and could logically be deduced - maybe from history of state or actions - but you have excluded them from the state representation, then you may have a more serious problem, and the agent may fail to learn.

It is worth noting the difference here between observation and state. An observation is some data that you can collect. E.g. you may have sensors on your robot that feed back the positions of its joints. Because the state should possess the Markov Property, a single raw observation might not be enough data to make a suitable state. If that is the case, you can either apply your domain knowledge in order to construct a better state from available data, or you can try to use techniques designed for partially observable MDPs (POMDPs) - these effectively try to build missing parts of state data statistically. You could use a RNN or hidden markov model (also called a "belief state") for this, and in some way this is using a "learning or classification algorithms to "learn" those states" as you asked.

Finally, you need to consider the type of approximation model you want to use. A similar approach applies here as for supervised learning:

A simple linear regression with features engineered based on domain knowledge can do very well. You may need to work hard on trying different state representations so that the linear approximation works. The advantage is that this simpler approach is more robust against stability issues than non-linear approximation
A more complex non-linear function approximator, such as a multi-layer neural network. You can feed in a more "raw" state vector and hope that the hidden layers will find some structure or representation that leads to good estimates. In some ways, this too is "learning or classification algorithms to "learn" those states" , but in a different way to a RNN or HMM. This might be a sensible approach if your state was expressed naturally as a screen image - figuring out the feature engineering for image data by hand is very hard.

The Atari DQN work by DeepMind team used a combination of feature engineering and relying on deep neural network to achieve its results. The feature engineering included downsampling the image, reducing it to grey-scale and - importantly for the Markov Property - using four consecutive frames to represent a single state, so that information about velocity of objects was present in the state representation. The DNN then processed the images into higher-level features that could be used to make predictions about state values.


Is there a clear distinction between Artificial Intelligence and running a sequential program?

No. As of now, there is no clear distinction between AI and a sequential program. The biggest reason is that there is no officially agreed definition of AI. Since AI researchers themselves hold different opinions of AI, the field itself is constantly redefined regularly, there is no clear-cut distinction between AI and non-AI.


You should check out my answer here to your second question.

For your first question, which is a special case, the answer is it might be one of the best fields to study!

Academic and industrial studies agree that working in a job that requires a lot of social interaction, particularly a job that involves caring for others, reduces automation risk. Among the safest sectors are education and personal care assistants, for example.

There are lots of possible reasons for this:

Even if a chat bot can replace your therapist, it probably cannot replace the sensation that another person cares about your problems, precisely because it is not an actual person. For people in dire emotional straits, having someone care might be worth a great deal.
Having a human tutor or therapist might be seen as a status symbol, even if an AI system is equally competent. If Harvard offered a degree where you studied on campus, taught by humans, and a degree where you studied online, taught by chat bots, they would probably not be perceived as equally prestigious by employers and peers.
AI systems like chat bots tend to fail in unpredictable ways when they enter unfamiliar situations. When dealing with an emotionally unstable person in therapy, or with a small child in education, unfamiliar situations might happy often, and the costs for failure might be high. Even if AI can handle most of the job, we might not be willing to trust it to an AI system because of these risks.

To address your third point, Q & A seems closer to AGI than to existing AI. If a program can give compelling answers to free-form questions, then it can almost certainly pass the Turing Test reliably. While you might see simple bots that analyze text and post a related wikipedia link, I suspect you won't see a high quality Q&A bot for many years (or decades) because of this.


Why is there more than one way of calculating the accuracy?

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).

\begin{align}
\text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0}
\end{align}

The following formula corresponds to the threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$).

\begin{align}
{\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1}
\end{align}

In section 7 of the paper A Literature Survey on Algorithms for Multi-label Learning, it is written

  In traditional classification such as multi-class problems, accuracy is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in multi-label classification, predictions for an instance is a set of labels and, therefore, the prediction can be fully correct, partially correct (with different levels of correctness) or fully incorrect. None of these existing evaluation metrics capture such notion in their original form.

Then they say

  Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: evaluating partitions, evaluating ranking and using label hierarchy. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes
  are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.

In section 7.1, the authors state

  To capture the notion of partially correct, one strategy is to evaluate the average difference between the predicted labels and the actual labels for each test example, and then average over all examples in the test set. This approach is called example-based evaluations.

To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper Discriminative Methods for Multi-labeled Classification)

\begin{align}
\frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2}
\end{align}

where $Y_i$ is the true set of labels and $Z_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$ is the accuracy for the instance $i$. If $|Y_i|=1$ and $|Z_i|=1$, then we have a single-label classification problem.

The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.


The code you reference is not part of the learning agent. It is part of:

class BlackjackEnv(gym.Env):

If, as in this case, the environment is provided entirely by software simulation, it is absolutely necessary for it to include a full working model of all state transitions and rewards. That is independent of whether any hidden state makes the problem harder.

  Why is it ok to calculate the reward based on a hidden state?

In the case of Blackjack, this can be treated not as a hidden state that would affect the outcome if only known, but as randomness in the environment over which the agent has no control. Critically, the dealer has no options to behave differently depending on the unknown card, and the dealer's eventual score is entirely unaffected by the player's earlier choices.

It is a subtle difference. If you applied the same environment rules to Poker, where an opponent could behave differently depending on this hidden knowledge, then a simple MDP model is not enough theory to result in an optimal solution. In that case, you would need to look into Partially Observable MDPs (POMDPs). Note this would not affect reward calculation in the environment, just the choices of which agent type to use. If you are just learning RL, you probably don't know of any algorithms that could solve this yet.

In practice, a lot of problems are somewhere between a classic MDP and a POMDP - they contain elements which, if the agent could know them, may allow it to achieve a higher expected reward. In many cases though, these elements can either be treated as random (as here in Blackjack) and thus the system is still theoretically an MDP, or they have a very small effect on the optimal policy, so can be ignored for practical purposes (e.g. think of all the physical details in a real cart pole balancing system - friction, temperature, flexing motions, etc).


What do the terms "front-end" and "back-end" refer to in this article?

I do not think these are formally defined.

The distinction is just to facilitate discussion of the NN architecture: e.g., you may have a few convolutional layers with pooling as a front-end, and a different architecture as a back-end (in a text-book architecture, just a fully-connected layer. But to get wild, maybe LSTM? To really get wild, BERT?).

In the end (no pun intended), computers do not care if a layer is seen by humans as a front-end or a back-end.


How can I reduce combinatorial explosion in an MCTS-like algorithm for program induction?

I think you may get some inspiration from the work on deterministic environments by Javier Segovia et al. See their paper Computing programs for generalized planning using a classical planner (2019).
If you don't have access to Elsevier's papers, I recommend that you check out the first author's profile on dblp.org. From there you will find links to open access versions of the conference paper that led up to the publication above.


Inconsistency in TD-Leaf algorithm in KnightCap chess engine

The second equation is correct. In TD($\lambda$), the $\lambda$ parameter can be tuned to smoothly vary between single-step updates (essentially what Sarsa does) in the case of $\lambda = 0$, and Monte-Carlo returns (using the full episode's returns) in the case of $\lambda = 1$.

In the first equation, $\sum_{j = i}^{N - 1} \lambda^{j - i} d_i$ could be interpreted as a... summing up exactly the same temporal-difference term $d_i$ a number of times (specifically, $N - 1 - j$ times), but multiplied by a different scalar every time. I'm not sure how that could be useful in any way.

In the second equation, $\sum_{m = i}^{N - 1} \lambda^{m - i} d_m$ can be interpreted as a weighted combination;

$1 \times d_i$: this is the difference between what we predict our returns will be at time $i + 1$, and what we previously predicted our returns would be at time $i$.
$+ \lambda^1 \times d_{i + 1}$: a very similar difference between two of our own predictions, now the predictions at time $i + 2$ and $i + 1$. This time the temporal-difference term is weighted by the parameter $\lambda$ (completely ignored if $\lambda = 0$, full weight if $\lambda = 1$, somewhere in between if $0 < \lambda < 1$).
$+ \lambda^2 \times d_{i + 2}$: again a similar temporal-difference term, now again one step further into the future. Downweighted a bit more than the previous term in cases where $0 < \lambda < 1$.
etc.

For people who are familiar with temporal-difference algorithms like TD($\lambda$), sarsa($\lambda$), eligibility traces, etc. from Reinforcement Learning literature, this makes a lot more sense. The notation is still a bit different from the standard literature on algorithms like TD($\lambda$), but in fact becomes equivalent once you note that in this paper they discuss domains where there are only rewards associated with terminal states, and no intermediate rewards.

Intuitively, what they're doing with the $\lambda$ parameter is assigning more weight (or "credit" or "importance") to short-term predictions / short-term "expectations" (in the English sense of the word, rather than the mathematical sense of the word) or observations of rewards, over long-term predictions/observations. In the extreme case of $\lambda = 0$, you completely ignore long-term predictions/observations and only propagate observed rewards very slowly, one-by-one in single steps. In the other extreme case of $\lambda = 1$, you propagate rewards observed at the end of episodes with equal weight all the way to the beginning of the episodes, through all states that you went to, giving them all equal weight for that observed reward. With $0 < \lambda < 1$, you choose a balance between those two extremes.

Also note that Equation (5) in the KnightCap paper (where they similarly discuss the extreme case of $\lambda = 1$, like I did above) is incorrect if we take the first equation from your question, but is correct if we take the second equation.


Comparing the derivation of the Deterministic Policy Gradient Theorem with the standard Policy Gradient Theorem

In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the first couple lines of the PG theorem we have
\begin{align}
\nabla v_\pi(s) &= \nabla \left[ \sum_a \pi(a|s) q_\pi (s,a) \right] \;, \\ &= \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \nabla\sum_{s',r} p(s',r|s,a)(r+ v_\pi(s')) \right] \; ;
\end{align}
you can see that we are taking expectation of $r$ with respect to the policy, so we don't need to write something like $r(s,\pi(a|s))$ (especially because this notation doesn't really make sense for a stochastic policy). This is why we don't need to take the derivative of $r$ with respect to the policy parameters. Now, the next line of the PG theorem is
$$\nabla v_\pi(s) = \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s)\sum_{s'} p(s'|s,a) \nabla v_\pi(s') \right] \; ;$$
so now we have an equation similar to the bellman equation in terms of the $\nabla v_\pi(s)$'s, so we can unroll this repeatedly meaning we never have to take an explicit derivative of the value function.
For the deterministic gradient, this is a bit different. In general we have
$$v_\pi(s) = \mathbb{E}_\pi[Q(s,a)] = \sum_a \pi(a|s) Q(s,a)\;,$$
so for a deterministic policy (denoted by $\pi(s)$ which represents the action taken in state $s$) this becomes
$$v_\pi(s) = Q(s,\pi(s))$$
because the deterministic policy has 0 probability for all actions except one, where it has probability one.
Now, in the deterministic policy gradient theorem we can write
$$\nabla v_\pi(s) = \nabla Q(s,\pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a)v(s') \right)\;.$$
We have to write $r$ explicitly as a function of $s,a$ now because we are not taking an expectation with respect to the actions because we have a deterministic policy. Now, if you replace where I have written $\nabla$ with the notation you have used for the derivatives you will arrive at the same result and you'll see why you need to use the chain rule, which I believe you understand because your question was more why don't we use the chain rule for the normal policy gradient, which I have hopefully explained -- it is essentially because of how an expectation over the action space works with a deterministic policy vs. a stochastic policy.
Another way to think of this is as follows -- the term you're concerned with is obtained by expanding $\nabla q_\pi(s,a) = \nabla \sum_{s', r}p(s',r|s,a)(r(s,a) + v_\pi(s'))$. Because, by definition of the $Q$ function, we have conditioned on knowing $a,s$ then $a$ is completely independent of the policy in this scenario - we could even condition on an action that the policy would have 0 probability for - thus the derivative of $r$ with respect to the policy parameters is 0.
However, in the deterministic policy gradient we are taking $\nabla q_\pi(s, \pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a) v_\pi(s')\right)$ -- here $r$ clearly depends on the policy parameters because the action taken was the deterministic action given by the policy in the state $s$, thus the derivative wrt the policy parameters is not necessarily 0!


What exactly are the differences between semantic and lexical-semantic networks?

According to Wikipedia  and
Semantic Networks
 ,By John F. Sowa
This is an updated version of an article in the Encyclopedia of Artificial Intelligence, Wiley, 1987, second edition, 1992.

  A semantic network is used when one has knowledge that is best
  understood as a set of concepts that are related to one another.

From Wikipedia:

  Most semantic networks are cognitively based. They also consist of arcs and nodes which can be organized into a taxonomic hierarchy. Semantic networks contributed ideas of spreading activation, inheritance, and nodes as proto-objects.

An example of semantic network is WordNet. It groups English words into synsets. Some semantic relations between these synsets are meronymy, hyponymy, etc.

On the other hand, Lexical semantic:

  ...explores whether the meaning of a lexical unit is established by looking at its neighborhood in the semantic net (words it occurs with in natural sentences), or whether the meaning is already locally contained in the lexical unit.


How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward $0$; otherwise, a reward of $0.5$ seems reasonable. These rewards provide model interpretability: an expected return of $p$ (estimated with a state-value or action-value) at a certain state under the current policy would signify a $p$ chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.

That being said, I would focus on the training process instead of a finely-tuned reward signal. Since there is a known goal state in the racing game (the finish line), I suggest training the RL agent by first initializing all racer agents only a few steps away from the goal state at the beginning of each episode. These episodes are shorter and therefore should provide a more dense reward signal. When your RL agent has learned a winning policy (e.g. wins more often than not), then initialize the agents slightly further from the goal state at the beginning of each episode. Also, continue to use and train the same neural network. Since the neural network presumably knows a winning policy at states near the goal state, then by initializing the agents only a few states further back, the RL agent is given an warm start and only needs to learn a policy for a few more states. The policy encoded by the neural network essentially contains a refined reward signal for the states close to the goal state since it is based on a winning policy; this helps prevent the sparsity problem caused by only supplying a reward at episode completion. You can repeat this process by initializing the agents slightly further from the goal state once the RL agent has learned a winning policy while continuing to use and train the same neural network.

Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it $n$ units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase $n$ until $n$ matches the distance from the starting line to the finish line. Since it seems like you have distance traveled and distance to the opponent as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.

A notable benefit of the overall approach is that you can more quickly debug your algorithm on the easier environments (i.e. ones with shorter episode lengths) to be confident that the learning process is correct and focus your efforts elsewhere (e.g. the training process, including the reward signal).


Should I build an environment from scratch myself or it is not always needed?

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking.
If you want a custom environment, you can add an environment to gym like this.


Is there any research on the application of AI for drug design?

Yes, many people have worked on this sort of thing, due to its obvious industrial applications (most of the ones I'm familiar with are in the pharmaceutical industry). Here's a paper from 2013 that claims good results; following the trail of papers that cited it will likely give you more recent work.


How to design a NLP algorithm to find a food item in menu card list?

Since you want a shortcut use the spoonacular API.  Below is a test with your words.  You can see it had trouble with 'Coca' and 'veg'.

What you need is 'named-entity recognition' for food.  This is not a new thing but clearly not a solved problem.  The Foodie Favorites repository attempts to solve the problem from scratch.
If want to do some research and dig deeper see FoodBase corpus: a new resource of annotated food entities.  From the abstract:

It consists of 12,844 food entity annotations describing 2105 unique food entities. Additionally, we provided a weakly annotated corpus on an additional 21,790 recipes. It consists of 274,053 food entity annotations, 13,079 of which are unique.


Freely Available Graphical User Interface for training a network on a dataset

If you want to do machine learning without coding (eg visual programming) you can try :

Orange : https://orange.biolab.si/ . It also come when you download the Anaconda distribution.
Also there is Dataiku Data Science Studio (the free version). https://www.dataiku.com/dss/trynow/

But in general I would not recommend doing machine learning / data science with visual programming : you will probably lose more time learning the software that actually coding with some real library.


What YOLO algorithm can I use for images with noise as I will implement it in real time?

It is much better to know basic mechanics of convnets first ,rather than diving straight into complicated models .

For training data sets (the images) I need to collect, I placed a camera. By recording it and collecting images from there, I am getting images with more noise. Can I use these images for training? Also, which yolo version should I use? What are the important points that I should consider for training datasets?

After you are good with the theory part most of your questions will be answered , otherwise you would endup with nothing but buzzwords.

I want to use any version of yolo compatible with tensorflow.

Tensorflow is a framework for building neural networks , so in theory you can build any network with it so compatibility is not at all a problem.


NLP Identifying important key words in a corpus

I am sure there are complex methods to extract keywords, but the standard one which should serve as a strong baseline is the RAKE graph algorithm https://pypi.org/project/rake-nltk/. It should work reasonably well in most text domains.


Having trouble understanding some of the details of R-CNN (first one)

Yes, your description is mostly aligned with the original paper. However, Figure 1 shows they use features from the second fully connected layer (not the last convolutional layer).
Yes. The training is performed by exploiting hand-annotated bounding boxes. If a proposal has IoU less than 0.3, then it is considered as a negative example (cf. section 2.3).
No. Appendix C explains that box regression parameters are learned separately from the SVM model.


How is the probability transition matrix populated in the Markov process (chain) for a board game?

Transition probability matrix cannot be initialized. Your game world has some rules. Since we do not know these rules we can approximate them. To do this, we should run the game over and over and collect the outcome of each action.

Here is a toy example: assume we control a plane. We want the plane to fly forward, but sometimes there may be wind in a random direction. So the plane won't end up in the desired position. Instead, it has some probability to fly left, right and forward. After the data is collected, we can estimate this probabilities.

Another example is estimating n-gram model: assume we want to train a model that will suggest a word in search field. We would analyze a corpus and calculate all 2 word sequences (for bigram model). Then, if a user starts typing I the model would suggest the word am, because I am occurs more frequently then I did, for example, and hence, the transition probability from I to am is greater.

Here is good book on Language Modeling

But your question suggest me that you want something like reinforcement learning. Check out this video wich is a part of a great lectures series on RL by David Silver


Does the recent advent of a Go playing computer represent Artificial Intelligence?

There are at least two questions in your question:

  What are some of the methods used to program the successful go playing program?

and

  Are those methods considered to be artificial intelligence?

The first question is deep and technical, the second broad and philosophical.

The methods have been described in: Mastering the Game of Go with Deep Neural Networks and Tree Search.

The problem of Go or perfect information games in general is that:

  exhaustive search is infeasible.

So the methods will concentrate on shrinking the search space in an efficient way.

Methods and structures described in the paper include:

learning from expert human players in a supervised fashion
learning by playing against itself (reinforcement learning)
Monte-Carlo tree search (MCTS) combined with policy and value networks

The second question has no definite answer, as you will have at least two angles on AI: strong and weak.

  All real-world systems labeled "artificial intelligence" of any sort are weak AI at most.

So, yes, it is artificial intelligence, but it is non-sentient.


Predicting continous value with CNN (prediction of fruit maturity)

but I have been told that neural networks aren't made to predict values in that way, they really are best suited for classification into discrete classes

I don't agree with this statement. I already trained many CNNs for regressions tasks where a continous output is trained and they generally perform very well.
I think the general "advantage" for a classification approach over a regression approach is that there is usually some margin where the output of the NN can still be clipped to a specific class, which might not be the case in regression tasks. Furthermore a lot of the "better classification performance" is due to the fact that the cross-entropy loss of classification tasks always tries to increase probability of one class while reducing all others. But this advantage might not be really important in your case where a specific range of days (e.g 3-5 days) can be probable instead of just one single day (e.g. exactly day 4).
Thats why I think you should defenitly try the regression approach. For this just normalize/clip your output layer to the min/max range of days. You should also defenitley decrease the learning rate at the end of training, I found this much further improves regression predictions to be more accurate.
And lastly if you want to achieve higher performance and more robustness, in your case I highly recommend a bayesian approach where you average over an ensemble of NNs (this ensemble can be your single NN with different parts truned off via Dropout layers during inferencing). With this average you can get an estimate of the uncertainty (the variance over different predictions) which gives you a much better idea of the probability distribution over the different days for a specifiy input.


How TensorFlow will know if the prediction is true or false?

During the training process, the machine learning program is shown example inputs (in this case pictures of digits) and also what the answer should be for that input. That is how it learns the correct answer.
However, it does not learn an exact mapping between the input data and required output, instead it learns a series of patterns that generally map the input to the required value. The advantage of this approach is that it can perform the same mapping on new inputs that were not included in the training data but are similar (but not necessarily identical) to examples that it has already seen.


Why is exp used in encoder of VAE instead of using the value of standard deviation alone?

In the source code, the author defines sd by

sd       = 0.5 * tf.layers.dense(x, units=n_latent)

which means that $\operatorname{sd}\in \mathbb{R}^n$.
In particular, the support over sd includes negative numbers, which is something we want to avoid.
Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
This is a case where the variable is inappropriately named.
Here, sd is not the standard deviation itself but rather the logarithm of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.


Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?

If a policy is fixed, it is said that an MDP becomes an MRP.

I would change the phrasing slightly here, to:

If a policy is fixed, an MDP can be accurately modeled as an MRP.

  Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?

The transition and reward matrices remain the same in the MDP, but it is possible to flatten them into an equivalent MRP, because in terms of observations of next state and reward, the action that is taken is just part of the transition rules - if the policy is fixed, then so are all the probabilities for next state and reward.

More concretely, if you have an MDP with $|\mathcal{A}|$ transition matrices $P_{ss'}^a$ and a fixed policy $\pi(a|s)$, then you can create a combined transition matrix with a sum:

$$P_{ss'} = \sum_{a \in \mathcal{A}} \pi(a|s) P_{ss'}^a$$

and you can similarly reduce the reward function. Once you have done so you have data that describe an MRP.

  How is it that it switches to an MRP which is not affected by actions?

If the MDP represents a real system where actions are still being taken by an agent, then of course those are still present within the system, and still affect it. The difference is that if you know the agent's policy, then the action choice is predictable, and the MRP representation covers the full definition of probabilities of observed state transitions and rewards.


Can we derive the distribution of a random variable based on a dependent random variable's distribution?

Yes you can, provided you know about $f$ and $g$. Expression $X3 = f(X1, g(X1))$can be written as $X3 = h(X1)$ where $h$ takes into account both $f$ and $g$. After this finding the PDF is simple by differentiating the CDF:
$$ F_{X3} (x3) = P(X3 \leq x3) =  P(h(X1) \leq x3) = P(X1 \leq h^{-1}(x3))$$

$$ \frac {d F_{X3} (x3)}{dx3}  = \frac {d P(X1 \leq h^{-1}(x3))}{dx3} = f_{X3}(x3)$$

NOTE: The conventions followed are the same as used in the field of Probablity

(Take care of the function inversion step in non-monotonic cases)

Check these lectures.


Why is the entire area of a join probability distribution considered when it comes to calculating misclassification?

The misclassifications that could arise if $\hat{x}$ is used as decision boundary are:

a) Classifying a point as $C_2$ when actually it was $C_1$ -- which will only happen when $x > \hat{x}$ as only the points greater than $\hat{x}$ are being classified as $C_2$.

b) Classifying a point as $C_1$ when actually it was $C_2$ -- which will only happen when $x < \hat{x}$ as only the points less than $\hat{x}$ are being classified to be from class $C_1$.

  2) Why is the green area with x < x0 a misclassification?

If we label a point -- the $x$ which is from the interval on the horizontal axis which corresponds to the green area -- which is a probability as being from either class then there is a good chance that it could be misclassified. This is because both the joint distributions' curves are above the axis for that interval on the axis and have some area (probability) for that interval on the horizontal axis.

There is a positive probability that a point that was drawn from the interval (a subset of the sample space, an event), which corresponds to the green area, was drawn from either of the two joint distributions or belongs to either of the two classes $C_1$ and $C_2$. The probability is the area under the curve.

  It's classified as C1 and it is supposed to be C1 right?

The same data point could well be generated from multiple different probability distributions. Here, you have an illustration of two probability distributions for which the author is trying to find the optimal decision boundary that will minimize the misclassification "region".

The point is not supposed to be from $C_1$, even if the density of the joint prob. distribution of $(x, C_1)$ has higher values. It could well be from $C_2$. And, when one classifies the same data point as being from class $C_1$ and make an error. This is why the whole area under the curve $p(x, C_2)$ has been painted green -- which means that there is some probability that a point could be from the distribution $p(x, C_2)$ and if we blindly label all the points to be from $C_1$ will certainly lead to some misclassifications.


Are ANNs just highly distributed lossy compression schemes?

ANNs don't compress, they generalise.

Often this leads to compression, i.e. the internal generalised representation is smaller than the original input, but not necessarily. Imagine a ANN that is trained to use the screen input of a computer game to play it. If the game is very rich and conceptually deep the internal representation of a single screen input might be a lot bigger than the input itself, because ANNs put the single data points into the context of the overall data. Which leads us to the second point:

ANNs (and the neocortex) model data hierarchically.

This is what makes them so powerful. So it is not just about having a large number of parameters, they also have to be arranged in such a way that they capture the structure of the data (or the world), which very often seems to be hierarchical. Just look a two different pictures of a duck. On the pixel level they might be as different as random images, all the similarities emerge in higher levels of the hierarchy, when enough pixels combined give you the patterns of feathers, beak and webs. Bloom filters obviously lack this property. They would only give you a pixel by pixel account of whether you have seen (almost) exactly this picture before.