Is the LSTM component a neuron or a layer?

The diagram you show works at least partially for describing both individual neurons and layers of those neurons.

However, the "incoming" data lines on the left represent all inputs under consideration, typically a vector of all inputs to the cell. That includes all data from current time steps (from input layer or earlier LSTM or time-distributed layers) - the line coming up into the LSTM cell - and the full output and cell state vectors from the whole LSTM layer on previous time step - the horizontal lines inside the cell starting on the left. Technically the top left input line could be read as either be a single neuron cell state value, or the full vector of cell state from the previous time step, depending on whether you were viewing the diagram as describing a single neuron in the layer, or the whole layer respectively.

If you visualise this cell connecting to itself over time steps, then the data lines both in and out must be whole layer vectors. The diagram is then best thought of as representing a whole LSTM layer, which is composed of various sub-layers which get combined, such as the forget gate layer (the leftmost yellow box).

Each yellow box in the diagram can be implemented very similar to a single layer of a simple feed forward NN, with its own weights and biases. So the forget gate can be implemented as

$$\mathbf{f}_t = \sigma(\mathbf{W}_f
[\mathbf{x}_t , \mathbf{y}_{t-1}] + \mathbf{b}_f)$$

where $\mathbf{W}_f$ is the weight matrix of the forget gate, $\mathbf{b}_f$ is the bias vector, $\mathbf{x}_t$ is layer input on current time step, $\mathbf{y}_{t-1}$ is layer output from previous time step, and the comma $,$ is showing concatenation of those two (column) vectors into one large column vector.

If the input $\mathbf{x}$ is an $n$-dimensional vector, and output $\mathbf{y}$ is an $m$-dimensional vector, then:

$\mathbf{c}$, the cell state (top line), and all interim layer outputs are $m$-dimensional vectors
$\mathbf{W}_f$, the forget gate weights, and the three other internal weights matrices are $m \times (n+m)$ matrices
$\mathbf{b}_f$, the forget gate bias, and the three other internal bias vectors are $m$-dimensional vectors

The other gates are near identical (using their own weights and biases, plus the third cell candidate value uses tanh instead of logistic sigmoid), and have their own weight matrices etc. The whole thing is constructed a lot like four separate feed-forward layers, each receiving identical input, that are then combined using element-wise operations to perform the functions of the LSTM cell. There is nothing stopping you implementing each yellow box as a more sophisticated and deeper NN in its own right, but that seems relatively rare in practice, and it is more common to stack LSTM layers to achieve that kind of depth.


What is the most time-consuming part of training deep networks?

Forward pass

The output of a layer can be calculated given the output of the previous layer. So the GPU can parallelize this computation for every layer and over the minibatch which is done by calculating a big matrix. But it needs to be sequential from layer to layer (earlier layers to higher layers). Regarding the layer type convolutions or especially fully connected layers can result in a big matrix calculation.

Backward pass

The gradient of a layer with respect to the layer input (and layer parameters) can only be calculated given the gradient of the layer output (input gradient of a subsequent layer) and input to the layer (output of the previous layer). This again can be parallelized over a layer and minibatch but is sequential from higher layers to earlier layers. Moreover, since the backward pass relies on the outputs of the forward pass all intermediate layer outputs of the forward pass have to be cached for the backward pass which results in a high (GPU) memory usage.

Forward and backward pass take most of the time

So, these two steps take a long time for 1 training iteration, and (depending on your network) high GPU memory usage. But you should read and understand the backpropagation algorithm that basically explains everything.

Moreover, to train a network from scratch, in general, takes lots of iterations because especially in the earlier layers training the parameters is based on gradients that are affected by lots of previous layers, which can result in noisy updates, etc., that do not always push the network parameters in the right direction directly. In contrast, e.g. fine-tuning a pre-trained network on some new task can for example already be done with much less training iterations.


Yes, this is possible. The only restriction on the range of output values is due to the activation function, if there is one. If, for example, the activation function is a sigmoid, the output would be restricted to values between 0 and 1. You can of course choose an activation function best suited to your problem, even if that means having none.

In your case, if you really want the output of your network to be between 50 and 250, you could use a scaled and shifted sigmoid as your activation: 200*sig(w*x+b)+50.

But, as was pointed out in comments below, having a single output neuron may not be the best architecture for this problem.


How can I understand this statement about RNNs and hidden layers?

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.

Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons.

In total there are 10 connections. As the image shows, neuron A receives the combined previous output of both neuron A and B as input. The same goes for neuron B.

This is not the case when we split the neurons up into multiple layers; the context neuron(s) are only used by neurons that are in the same layer. Let say we now use multiple hidden layers and keep the amount of neurons the same. In total there are 7 connections now (image below). That is 3 less than in the first example, which has only one hidden layer. So which connections do we miss? That is shown in the bottom image. (I had to paste these two images together in one image because my reputation only allows me to post 2 links)

Please note the cross; the connection between neuron A and B is not there in the first image, because it would be some kind of random recurrent connection.

The first and the last image are exactly the same. I think that if you compare the first and the last image that you agree that the statement is true.


Is the size of a neural network directly linked with an increase in its inteligence?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without overfitting).
The more nodes (or neurons) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.
So, yes. The more nodes a model possesses, the higher the so-called model capacity, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.
In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.
Generally, the larger the model to be trained, the higher the computational cost to train the network.
A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.


How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with $r(s,a,s')$ rewards would look like this:

$$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,a)(r(s,a,s') + \gamma V(s')) \right\} $$
$$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,a) \cdot r(s,a,s') + \gamma \sum_{s'} p(s'|a,s) \cdot V(s') \right\} $$

Now, $ \sum_{s'} p(s'|s,a) \cdot r(s,a,s') $ is precisely $ \mathbb{E}\left[ r(s,a,s') | s,a \right] = r(s,a) $.

So all in all, the resulting Bellman equation looks like this:

$$V(s) = \max_a \left\{r(s,a) + \gamma \sum_{s'} p(s'|s,a) \cdot V(s') \right\} $$

It is clear, then, that a process with $ r(s,a,s') $ rewards can be transformed to a $ r(s,a) $ process without introducing artificial states and maintaining the optimal policies.

As a side note unrelated to the question itself, that leads me to believe that $ r(s,a,s') $ functions may be convenient in some scenarios, but they do not add "expressive power", in the sense that they do not allow to model problems more compactly (as it happens when we extend $ r(s) $ to $ r(s,a) $).


What is the difference between Inductive Learning and Connectionist Learning?

All of the statistical learning is about inductive learning.

What is the difference between inductive learning and connectionist learning?

Inductive learning is about identifying patterns from examples. It is more related to statistics. Connectionist learning is more about finding a common pattern and predicting as well as self-learning(learning from the experience of prediction).

Connectionist learning is where learning occurs by modifying connection strengths based on experience. This is not the case with inductive learning. In inductive learning, we are not modifying things based on experience. Inductive learning just finds common patterns, not self-learning based on experience.

  Learning requires both practice and rewards

In inductive learning, we learn the model from raw data (so-called training set), and in the deductive learning, the model is applied to predict the behaviour of new data.

  Connectionist Learning is a group of inductive learning and deductive learning.


Why isn't medical imaging improving faster with AI?

AI in healthcare is already playing a big role in diagnosing the diseases, assisting patients or helping the medical staff in supplying various things or performing actions.
The medical imaging helps AI to diagnosis the diseases without help radiologist. And to develop the AI model using medical images to diagnosis various disease need training datasets for machine learning algorithm helping to detect with accuracy.
But just like AI, medical imaging is not improving faster making sometimes difficult or more time taking to diagnosis the diseases. Actually, while developing such highly sensitive models, high-quality training data is required and there is lack of such annotated data to train the models.
The more training data is feed into the model, it will learn the detection process with more accuracy resulting predicting the faster diagnosis. And to improve the quality and quantity of data, more dedicated and fast labeling process is required.
Hence, data annotation companies now use the AI-assisted labeling process to annotate the medical images at faster speed with better accuracy.  Compare to manual annotation, AI companies can get multiple times faster annotated data for machine learning algorithms.
Once the training data will start available in large quantity for medical imaging analysis process through AI will also improve. The AI-assisted data annotation process can only help to produce the medical imaging datasets in large quantity for better predictions in healthcare sector.


Are neural networks prone to catastrophic forgetting?

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general intelligence.
There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper Continual lifelong learning with neural networks: A review (2019), by German I. Parisi et al., which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.


How can neural networks that extract many features be fooled by adversarial images?

The link in DuttA's comment provides a good answer to the more general question of how adversarial images are generated. As to why it's possible at all, the key is that these specialized feature detectors really are just picking up features, and it's possible to construct an image that has those features, without resembling the target object much at all.

For example, we might imagine that the key features that predict an image of a gun are regions like the trigger guard and perhaps the muzzle. The features that are extracted really will just be "round-ish thing with a curved bit in it" and "perfectly circular opening". These might predict a gun really well, but it's also easy to imagine that you could place shapes satisfying those criteria the correct distance apart without including any of the other components that we'd expect. This is what you actually see in the adversarial images that are generated. Consider the bagel or crossword puzzle images below. While they are not actually pictures of the objects in question, it is fairly easy to see what the important features were for the network: a bagel is a round shape that's shadowed in the center and lighter between the center and rim. It's a bit orange. A crossword puzzle is a bunch of black and white squares distributed in a rough grid. Possibly the copyright symbol or the letter C was also important (it is found on many crosswords!).


How do policy gradients compute an infinite probability distribution from a neural network

Do neural networks compute the probability distribution for policy gradient methods.

In short, yes. It does not have to be neural networks, any trainable parametric function approximator based on gradients will do. Neural networks are a common choice, as are linear function approximators using selected basis functions.

  If so, how do they compute an infinite probability distribution?

For background, this is an issue for generating stochastic policies in continuous action spaces only. In discrete action spaces, it is usually possible to compute an arbitrary probability density function for the whole action space, and sample from it to model the policy. It is also possible to compute a deterministic policy simply enough in continuous spaces - the input is the current state and the output is the action to take. The issue then is that this does not allow an agent to learn through exploration of the environment. To do that requires a stochastic policy.

If you want to generate a stochastic policy in continuous action spaces, you could discretise the space and sample from that using e.g. softmax to generate the action probabilities. Or you could have the approximation function do something more indirect: Output the parameters of a probability distribution that can be sampled from.

  How do you represent a continuous action policy with a neural network?

Typically by having state features as input and the parameters of a PDF that can be sampled as the output. For instance, the network could output mean $\mu$ and standard deviation $\sigma$ of a normal distribution for an action value, and the policy is given by $\pi(a|s) = a \sim \mathbb{N}(\mu, \sigma)$.

This distribution can be sampled (there are simple methods to generate a sample from a normal distribution), and returns from following this policy used as feedback to the neural network using the policy gradient theorem. Assuming that there is an optimal deterministic policy to be found, the neural network can learn over time to home in on a specific mean with low standard deviation.

In some cases, the standard deviation can be treated as a hyper-parameter, similar to $\epsilon$ in $\epsilon$-greedy action selection, and might be decayed over time. In that case, a neural network can just output the mean action.

It is also possible to learn a deterministic policy through off-policy learning, adding a noise function to support exploration. This is what Deep Deterministic Policy Gradient does.


Is there a way of path reconstruction using only the history of belief state?

The belief state in a POMDP is a distribution over the hidden state given all past actions and observations, i.e., at time $k$, the belief state is $b_k(s_k) \triangleq P(s_k \mid a_{0:k-1}, z_{1:k})$, where $a$ and $z$ are the actions and observations, respectively.
What you are asking about and calling "backtracking" boils down to the question: "given that at time $k$, I know the history of actions $a_{0:k-1}$ and observations $z_{1:k}$, what is the distribution over states at some past time step $t<k$, that is, $P(s_t\mid a_{0:k-1},z_{1:k})$"?
This is commonly known as Bayesian smoothing. You might be familiar with Bayesian filtering, which aims to recover the belief state as defined in the first paragraph - an estimate of the current state given actions and observations up to the current time. Smoothing also uses information gained after time $t$ to estimate the state at time $t$, and hence it is only possible after those actions and observations become known. There is also prediction, where you estimate the distribution over the state in the future.
Smoothing is too broad of a topic to go into detail here, but below is a picture from the book "Bayesian Filtering and Smoothing" by Simo Särkkä that illustrates the concepts. You could start there for further information on the topic.


Where or for what could genetic algorithms be used in the context of project management?

GA is well suited to optimize a non-linear fitness function with a lot of variables. Each vector that is a possible solution is evaluated with the fitness function that we want to optimize.
So, GA is well suited to optimize schedules for a lot of people, optimizing resources, etc.
Your mission is to define all variables that you can move-tune and that affect your objectives, and assign an estimated weight on every variable.
For example, create a total project cost as the fitness function, to minimize. One variable will be the number of programmers, multiplied by cost and total months. If you don't want delays over 3 months, add a correction factor: +10000*max(0, months-3), penalizing the fitness function's results where months > 3, etc.
When you apply your GA algorithm to the fitness function, you will find some minimum for your function, if any exists, because the solution may not exist if you added too many constraints.


What method to identify markers in data series via machine learning

Thresholding based on variance is one method for anomaly detection in time-series data.


How to use and update a shared/global policy between Reinforcement Learning Agents

This sounds like distributed RL, and most of the work goes in building the distributed system; the actual RL part is just a DQN (+some tricks from Rainbow DQN).
NB: multi-agent RL arises when the agents interact with each other in the same environment (like Hanabi the card game), while in this case we have multiple agents that collect experiences in parallel.
Here's a possible design from the Ape-X paper:


Calculating accuracy for cross validation

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all of them to get the average accuracy of your model.

Your first option doesn't seem great because you take the highest accuracy among folds. If for some reason, the variance between accuracies is high for a fold, this would bias your numbers. Taking mean or maybe median of accuracies might be more reasonable.

Does that help?


What is the type of problem requiring to rate images on a scale?

The main distinction between tasks is 'classification' vs 'regression'. In classification you would try to identify the presence of a cloud or not in an image, if you want to predict the level of 'cloudness' with continuous values you are then performing a regression task.

I'm not aware about state-of-the models specific for images, but you can potentially use whatever architecture you desire to perform regression, CNN, RNN, the only thing to pay attention to is the loss function you will use. There are specific functions for classifications (which usually use the argmax function to turn probabilities into labels) and for regression (the most used is the Mean Squared Error, the model try to approximate the continuous values directly).

For a quick overview of loss functions I suggest this easy tutorial 5 Regression Loss Functions. Hope it might be of use.


Why do we also need to normalize the action's values on continuous action spaces?

Notably, these two tips/tricks are useful because we are assuming the context of deep reinforcement learning here, as you pointed out. In DRL, the RL algorithm is guided in some fashion by a deep neural network, and the reasons for normalizing stem from the gradient descent algorithm and the architecture of the network.

  How does this affect training?

An observation from the observation space is often used as an input to a neural network in DRL algorithms, and normalizing the input to neural networks is beneficial for many reasons (e.g. increases convergence speed, aids computer precision, prevents divergence of parameters, allows for easier hyperparameter tuning, etc.). These are standard results in DL theory and practice, so I won't provide details here.

  And more specifically, why on continuous action spaces we need to normalize also the action's values?

Most popular discrete action space DRL algorithms (e.g. DQN) have one output node for each possible action in the neural net. The value of the output node may be a q-value (value-based algorithm) or a probability of taking that action (policy-based algorithm).

In contrast, a continuous action space DRL algorithm simply cannot have an output node for each possible action, as the action space is continuous. The output is usually the actual action to be taken by the agent or some parameters that could be used to construct the action (e.g. PPO outputs a mean and standard deviation and then an action is sampled from the corresponding Gaussian distribution - this phenomenon is mentioned in your linked reference). Therefore, normalizing the action space of a DRL algorithm is analogous to normalizing the outputs of the corresponding neural network, which is known to increase training speed and prevent divergence. Again, a quick search will yield some good resources if you are interested in these results.


Several AI systems will come up with a level of confidence to the solution found. For example, neural networks can indicate how relatable is the input problem to the ones it was trained with. Similarly, genetic algorithms work with evaluation functions, that are used to select best results, but depending on how they're built (the functions), they can indicate how close the algorithms are to an optimal solution.

In this case, the limit to when this is acceptable or not will depend on a threshold set beforehand. Is 50% confidence good enough? Maybe it's ok for OCR apps (spoiler: it's not), but is it for medical applications?

So yes, AI systems do currently have the capacity of determining if they're performing well or not, but how acceptable that is is currently based on the domain of the problem, which currently stands outside of what is built into an AI.


Which neural networks are suitable for visual place recognition?

Neural networks construct increasingly complex representations of data on each of their layers , so you are free to choose any neural network architechture for this purpose . since the lower layers of neural network (layers near the input layer) mostly compute low level representations of the image (like gabor filters etc) most architechtures won't have much difference at this level. so you can use VGGnet if you want with proper fine-tuning from layer 3 itself.


Is the summation of consistent heuristic functions also consistent?

No, it will not necessary be consistent or admissible. Consider this example, where $s$ is the start, $g$ is the goal, and the distance between them is 1.

s --1-- g

Assume that $h_0$ and $h_1$ are perfect heuristics. Then $h_0(s) = 1$ and $h_1(s) = 1$. In this case the heuristic is inadmissible because $h_0(s)+h_1(s) = 2 > d(s, g)$. Similarly, as an undirected graph the heuristic will be inconsistent because $|h(s)-h(g)| > d(s, g)$.

If you'd like to understand the conditions for the sum of heuristics to be consistent and admissible, I would look at the work on additive PDB heuristics.


What is the difference between A2C and running an agent in an environment vector?

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.


To provide a good answer would fill several pages.  To keep it very simple try many different loss functions on your model.  Your goal is to have the highest performance based on some desired prediction metric (e.g., RMSE, MAE, MAPE, etc.).  You almost always have plenty of time to try many loss functions so you don't need to have a full understanding, and few people do, to start your project.
I recommend you read the following to learn more:

5 Regression Loss Functions All Machine Learners Should Know by Prince Grover
Loss functions: Why, what, where or when? by Phuc Truong
Loss Functions in Deep Learning: An Overview


How to fine tune BERT for question answering?

The answer is yes but 'lightweight' will require a 'lightweight' model.
Your application for 'domain one' is called open domain question answering (ODQA).  Here is a demonstration of ODQA using BERT: https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
Your application for 'domain two' is a little different.  It is about learning sequences from sequences.  More specifically these are called sequence to sequence models.  Here is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
In both applications, the resources required are going to rely on the performance you require.  There are many sizes of BERT models.  Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.).  For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.
This article, NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.
There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones.  Here is an article on using that with SQuAD: https://www.tensorflow.org/lite/models/bert_qa/overview
cdQA-suite is a good package.  The following should help in fine-tuning on your own corpus:

How to create your own Question-Answering system easily with python
cdQA examples


Predict best price using neural network?

This is possible.... but there's no reason to use a neural network! Your best bet on a problem like this is likely to use a logistical regression for the yes/no aspect of the question and a linear regression (or combination of linear regressions) to answer the pricing question - there are also ways of simply using linear regressions and setting up cutoffs to answer the yes/no question.

The reality is that the accuracy of such a model/series of models would depend entirely on the quality and quantity of the data, but it's unlikely that in this case a neural network would provide a better result than smart usage of simpler models.


What is the difference between goal-based and utility-based agents?

Utility is a fundamental to Artificial Intelligence because it is the means by which we evaluate an agent's performance in relation to a problem. To distinguish between the concept of economic utility and utility-based computing functions, the term "performance measure" is utilized.

The simplest way to distinguish between a goal-based agent and a utility-based agent is that a goal is specifically defined, where maximization of utility is general.  (Maximizing utility is itself a form of goal, but generalized as opposed to specific.)

A goal-based navigation agent is tasked with getting from point A to point B.  If the agent succeeds, the goal has been satisfied.
A utility-based navigation agent could seek to get from point A to point B in the shortest amount of time, with the minimum expenditure of fuel, or both.

In the above example, the utility agent is also goal based, but where the performance measure for the goal agent is a binary [succeed/fail], the utility agent can use real numbers and measure performance by degree. The utility agent allows more granularity in evaluation.

For an example of a non-goal based utility agent consider a form of a partisan sudoku in which players compete to control regions on the gameboard by placement of weighted integers.

In a game with 9 regions, the goal based agent seeks to control a specific number of regions at the end of play.  If the agent is conservative, the goal might be 5 regions. If the agent is hyper-aggressive, the goal might be 9 regions.  When evaluating the environment (gameboard), if the agent dominates the desired number of regions, it could choose to consolidate (reinforce); if the agent does not dominate the desired number of regions, it could choose to expand (attack).

The above strategy can be effective, but is limited by the specificity of the goal.  A hyper-aggressive goal would work well against a weak opponent, but against a strong opponent it might prove disastrous.  If the agent is sophisticated, where performance has been poor, it might alter it's goal by switching to a "turtling strategy" and seek to control fewer regions, but, because the new goal is still specific, the agent may miss opportunities to improve it's final status beyond the adjusted goal.

The utility-based agent can approach the game with no specific goal beyond improving it's status. Rather than seeking to control a set number of regions, the utility-agent evaluates whether a given choice improves or worsens it's status. ("Do I dominate more or less regions if I take this position?") The utility agent can distinguish between sets of beneficial choices ("which choice maximizes my expected benefit?") and, where no benefit can be obtained, distinguish among the set of choices with the least downside ("among the set of bad choices, which is the least bad choice?")

In this example, the utility-agent doesn't even need to understand the victory condition (controlling the more regions than the opponent at the end of play.)  Instead, the utility-agent merely seeks to maximize the number of controlled regions over the course of play, which will result in victory if the agent makes more optimal choices than the opponent.


What is the difference between "mutation" and "crossover"?

The mutation is an operation that is applied to a single individual in the population. It can e.g. introduce some noise in the chromosome. For example, if the chromosomes are binary, a mutation may simply be the flip of a bit (or gene).
The crossover is an operation which takes as input two individuals (often called the "parents") and somehow combines their chromosomes, so as to produce usually two other chromosomes (the "children"), which inherit, in a certain way, the genes of both parents.
For more details about these operations, you can use the book Genetic Algorithms in Search, Optimization, and Machine Learning by David E. Goldberg (who is an expert in genetic algorithms and was just advised by John H. Holland). You can also take a look at the book Computational Intelligence: An Introduction (2nd edition, 2007) by Andries P. Engelbrecht.


How does training for DQN work if messing up in the environment in costly?

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.
With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.


Why didn't champion of the Go game manage to win the last game against AlphaGo, after winning the 4th one?

The technique used by AlphaGo is "Monte Carlo Tree Search", combined with a very well trained neural network. The network's job is to estimate the quality of different board states and moves. This estimation is deterministic. If you show AlphaGo the same board on two different occasions, it thinks it is exactly as good (or bad) on both occasions.

Monte Carlo Tree Search however, is a randomized algorithm. So as a simplified explanation, the way AlphaGo decides which moves to make is:

Look at the current board.
Pick a random move, and imagine what the board would look like if you made that move.
Pick a random move for your opponent, and imagine what the board would look like if they made that move.
Keep doing steps 2 & 3 for a while, so that we're imagining being at some board many moves in the future along a random line of play.
Ask the neural network how good this board state is.
Repeat steps 1-5 many times. Then make whichever move led to the best lines of play on average, to make right now.

What this means is, AlphaGo won't always play the same way, because it doesn't actually consider every move explicitly. It just thinks about enough lines of play to be pretty confident about whether one move is better than another. This is actually not so far removed from how humans play most of the middle of games like these.

So, in game 4, essentially, Sedol got lucky. The random lines of play that AlphaGo chose to look at did not capture some critical facts about one or more board states. This led it to make a mistake. If you asked it to play the same game through again, it might not make the same mistakes (it might think about different random lines of play, that do capture the critical facts it missed in the first game). Further, it might choose to play slightly differently on other moves, which could have a big impact on the rest of the game. These two factors prevented Sedol from simply playing game 4 over again.


Extract product information from email receipt HTML

It depends on the data. If it is structured like form data, then you might not need AI at all — simple regular expression patterns might be fine. This would apply for example to address data. If you have the word street followed by a colon, followed by some text, it seems fairly obvious that this is the name of a street, and possibly also a house number.

If, however, you have free text, eg the answer to "describe any medical conditions you have", or "which companies have you worked at before?", then you might need to look at named entity extraction (NER) to identify names of medical conditions or companies.

So, some Natural Language Processing (NLP) and information extraction might be required apart from simple pattern matching.


How do you pass the image from one convolutional layer to another in a CNN?

The application of 1 kernel (aka filter) to an input (with a 2d convolution) is a matrix (a 2d array), which is often known as a feature map (aka activation map). The application of $k$ kernels to the same input is a 3d array (sometimes called tensor, though this may not be exactly correct, or 3d volume) with depth $k$, i.e. you have $k$ concatenated feature maps. This 3d array is the input to the next convolutional layer, which needs to have 3d kernels of depth $k$: this is the main requirement in the case of 2d convolutions (in the case of 3d convolutions, the kernels can have a different depth than the input 3d volume, but you can ignore this for now!).
Another thing that you need to take care of is the padding that you can add around this 3d volume. This, along with the stride (i.e. the step with which you move the kernel), determines the height and width of the output of the convolutional layer.
So, to recapitulate, what you have to do is set the depth of the kernels in the convolutional layer $l+1$ to be equal to the number of kernels that you use in the convolutional layer $l$. So, let's say that the number of kernels is a hyper-parameter (note that this is usually the case!), and that the user sets the number of kernels $k=8$ in the convolutional layer $l$, and, for simplicity, assume that all kernels are $3 \times 3$, then, in layer $l$, you will perform $8$ convolutions, one for each kernel. These $8$ convolutions produce a 3d array of dimensions $w \times h \times 8$ (where $w$ and $h$ depend on the dimensions of the input to layer $l$, the stride, and padding). Consequently, in the convolutional layer $l+1$, the kernels need to be $3 \times 3 \times 8$.
If it's the pooling layer that produces the 3d array to be passed to the next convolutional layer, you should still set the depth of the kernels in the convolutional layer $l+1$ to the number of kernels in the convolutional layer $l$ (provided that, in the pooling layer, you only redimension the width and height of the 3d array, which is usually the case).


The sold date is a feature like any other. You can do this as follow. I am assuming the features are in a pandas data frame  called df where the column date is called date. Easiest way is to use the pandas to_datetime function. Documentation is here.
def encode_dates(df, column):
    df = df.copy()
    df[column] = pd.to_datetime(df[column] )
    df[column + '_year'] = df[column].apply(lambda x: x.year)
    df[column + '_month'] = df[column].apply(lambda x: x.month)
    df[column + '_day'] = df[column].apply(lambda x: x.day)
    df = df.drop(column, axis=1)
    return df
df=encode_dates(df. 'date')

This function will modify the df data frame. It will create 3 new columns labeled
date year, date month and date day and it will remove the date column from the data frame.
Now these new columns along with the other features can be used to train your model.


In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (3x3x3=27 values). So, three different 2D filters of size 3x3 can be concatenated to form this one 3D filter of size 3x3x3.

The 3x3x3 RGB chunk from the picture is multiplied elementwise by a 3D filter (shown as grey). In this case, the filter has 3x3x3=27 weights. When these weights are multiplied element wise and then summed, it gives one value.

So, is there a separate filter for each input channel?

YES, there are as many 2D filters as number of input channels in the image. However, it helps if you think that for input matrices with more than one channel, there is only one 3D filter (as shown in the image above).

Then why is this called 2D convolution (if filter is 3D and input matrix is 3D)?

This is 2D convolution because the strides of the filter is along the height and width dimensions only (NOT depth) and therefore, the output produced by this convolution is also a 2D matrix. The number of movement directions of the filter determine the dimensions of convolution.

Note: If you build up your understanding by visualizing a single 3D filter instead of multiple 2D filters (one for each layer), then you will have an easy time understanding advanced CNN architectures like Resnet, InceptionV3, etc.


Does it make sense to apply softmax on top of relu?

Does it make sense?

In general, yes it is interpretable, back propagation will work, and the NN can be optimised.

By using ReLU, the default network has a minimum logit of $0$ for the softmax input, which means at least initially that there will be higher minimum probabilities associated with all classes (compared to allowing negative logits which would happen randomly with usual weight initialisation). The network will need to learn to produce higher logit values for correct answers, because it has no ability to produce lower logit values for incorrect answers. This is like training a network to produce the highest regression value on one output, whilst clipping all values to be 0 or above, so it does not have the option of making one output e.g. $-1.0$ and the rest $-100.0$

It can probably be thought of as a type of regularisation, as it puts constraints on activation values that will work.

Is it needed?

That is less clear. You can try training with and without the line, and using cross-validation or a test set to see if there is a significant difference.

If the network has been designed well, then I'd expect to see a slight improvement with the added ReLU.

If it is a mistake, then I'd expect to see no difference, or better performance without the ReLU.


Why does PCA work well while the total variance retained is small?

Because it selects both Xtrain and Xtest from the space of two selected principal components. Hence, the 90% accuracy is in that 2-D selected space.

This fact that the ratio in PCA stands the information, depends on the distribution of the data and it's not true at all.


Reinforcement Learning with asynchronous feedback

I have been looking for a while into pretty much precisely the problem you describe (including the same application domain), but haven't been able to find much.

The most obvious, mathematically "correct" solution would be to simply delay your standard Reinforcement Learning update rule (of whatever algorithm you choose to implement) by 45 days; if it still wasn't reported as a fraud then, assume it was genuine. This leads to some problems though;

Need lots of memory to store experiences that were not yet used for updates
Learning only starts after a significant delay, in which you don't learn anything at all yet and likely therefore run a suboptimal policy for a long time
Very slow to adapt to new strategies of the fraudsters
What to do with people who already report fraud cases earlier, like after 10 days? Delay them for the full 45 days anyway, or trigger updates immediately (and potentially mess up the ordering in which experiences actually occurred)?

A quick and dirty "solution" is the following;

When a transaction occurs, immediately trigger a learning update under the assumption that it was a genuine transaction (for example, with a reward of R = +1).
If that transaction is later reported as a fraud, trigger an additional update (with same (state, action) pair), but with the negation of the reward that was previously assigned erroneously on top of the normal negative reward for a fraudulent case. For example, if you would normally give R = +1 for genuines, and R = -100 for frauds, give a reward of R = -101 now. This reward will not correct for the previously assigned wrong reward in completely the right way (potentially wrong position in sequence of updates, discounting due to gamma and maybe lambda depending on algorithm used, etc.), but it should be somewhat close (especially if gamma and lambda are close to 1.0).

This is certainly not ideal, has very little theoretical basis and probably breaks quite a bit of Reinforcement Learning theory, but at least it is efficient in terms of computation and memory and in my experience it works alright in practice.

If you're using off-policy RL algorithms, you can use Experience Replay buffers (very popular in DQN-style things in Deep RL these days, but can also be used in tabular RL / RL with linear function approximation etc.). If you already have historical data generated through some non-RL policy in the past (which is typically the case in fraud detection / banking applications, they do have lots of data even if they don't always share it), you can use this to fill your experience replay buffer. In the case of the first solution (at the top of this answer), this can be used for training during the initial delay of 45 days.

Since you expect there to be concept drift though (fraudsters adapting their behaviour over time), you'll want to be careful with experience replay. Old data will become less useful.

A very different style of solution is to assume that you have a team of human experts available who can investigate a very small portion of incoming transactions relatively quickly. This tends to be true for large companies in practice ("investigating" often means a phone-call to a card holder). This enables you to generate accurate feedback for a small portion of your data more quickly, so that you can also do Reinforcement Learning with much less of a delay (albeit only on a small percentage of your experience).

You can read more about this idea in the following paper (disclaimer: I'm an author on it):

Dennis J.N.J. Soemers, Tim Brys, Kurt Driessens, Mark H.M. Winands, and Ann Nowé (2018). “Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees”. In Thirtieth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-18), pp. 7831-7836. AAAI Press.

Apart from that idea you might furthermore find it interesting for references to other related work, links to data you could use, etc.

I feel like it should be possible to extend the existing Reinforcement Learning theory with proper algorithms that can properly;

Take immediate learning steps with an assumed, default, potentially incorrect reward, and
Retroactively correct for previous incorrect updates if the reward turns out to be something else than previously assumed in hindsight.

I'm not aware of existing literature in which this is done though, and it certainly doesn't seem trivial; it will require starting pretty much from "first principles" (e.g., Bellman operator).

Intuitively, I also expect doing this completely correctly will always require a significant amount of memory (memory of all previous transactions of a card holder, such that state-action pairs can be re-generated if necessary). Banks likely already store that kind of data anyway for every customer, so it may not be a problem in practice.

If anyone's planning to work on this, feel free to contact me, I'll likely be happy to collaborate :D


Is CNN capable of extracting the descriptive statistics features

CNNs learn convolutional filters that get trained on finding local, recurring patterns in some kind of image/volume data. 1D convolution is actually a thing, but I think what would be more suitable for your case is using Recurrent Neural Nets. They are specifically designed for working on time series-es of heterogeneous data.
Update:
I would like to reconsider the answer I gave earlier.
First of all, in case of dealing with time series data where it is uncertain over which time span a given event to be detected lasts, I'd generally consider using Recurrent Neural Networks (RNNs) rather than CNNs, since RNNs maintain so-called hidden states that can carry potentially useful information over time, i.e. past inputs may influence present outputs. So, either if you know that the event to be detected spans many time steps (which might not even be consecutive time steps) or if you are uncertain about how long exactly the time span of an event to be detected is, then I'd suggest going for RNNs. If you decide to use RNNs, adding extra statistics to your data might make the learning task easier for the network, but is not strictly required since you would expect an RNN to learn relevant time-series statistics on its own.
However, there is a case where using CNNs might actually suffice and yield good results as well. Consider the case of the figure shown below, where the white tiles are time series inputs and the blueish area is the area that a CNN filter/kernel (or multiple ones) is working on right now.

As you can see, there is a time series of data points and some statistics derived from that time series that have been appended to the input data stream. So, in the figure, the augmented time series input consists of the measurements themselves as well as mean, variance, and skewness values derived from the time series data (over consecutive time steps). Under the assumption that the type of event you are interested in is bounded, in terms of time steps, to the width of the CNN filter(s)/kernel(s) that you slide over your input data, then a simple CNN might suffice to detect the events you are interested in. In other words, if the width of your filter spans over at least as many time steps as an event of interest lasts, then your CNN will be able to learn to detect the event (given proper training). In that case, adding further statistics to your input data might indeed help or even be necessary to detect certain events, since it implicitly widens the time window that your CNN filter can observe at any given time step, since a mean (or other statistics) may also considers past data that the filter itself (or multiple filters...) might not span over at a given time step where you want to detect an event.


Does a solution for Wumpus World with neural networks exist?

Yes! If you read ahead to the chapters in reinforcement learning in the same book, you'll see that the wompus world appears again there. Techniques like Q-learning can be used to solve it, and since Q-learning involves learning the shape of a function, a neural network can be employed as a function approximator.

The basic idea is to treat this problem as an input/output mapping (states -> actions), and to learn which actions produce the greatest rewards.

Note however, that these approaches rely on trial and error. The logic based approach reasons about the rules of the game, and can play reasonably well right away. The learning approach will need to try and fail many times before playing well.


Can we just switch off a malicious artificial intelligence?

Malware viruses are a very simple form of AI.  It is not difficult to conceive of a form of malware that A) can't be detected easily, B) is redundantly distributed across thousands of computers that occasionally connect to the internet, C) is capable of detecting some kinds of threats to itself and mutating to avoid the threats.

So, simply "turning off" a malicious AI will not always be possible.


Can a machine learning model predict the pattern of given sequence?

Those all fit into a single quadratic, auto-correlated model.

$$ x_0 = a \\ x_i = b i^2 + c x_{i-1} + d i + e $$

The sequences can be curve fitted producing a set of $n$ perfect fits of the form $(a, b, c, d, e)$ given the above model. A rules engine given the correct parameterized rules can produce the most desirable verbal description from among the $n$ fits in the set. The rules can also be prioritized by a simple feed forward network trained to simulate the most natural selection of string descriptions from any set of fits where $n > 1$.

This will work well for the examples in the question and many more, however, if the sequence $\{1, 4, 1, 5, 9\}$ is fed into the system, it will produce some weird description based on the quadratic, auto-correlated model it was given rather than, "digits of $\pi$ to the right of the decimal place."

The only way to produce the most common response a university freshman math student would produce would be to extend the boundaries of AI engineering first. For example, once an AI system is developed that can handle natural language and cognition like a child, several of them can be separately trained in a simulation of primary and secondary school mathematics. The median response for each sequence given to the class of AI students (class made up of artificial students studying math, not class of humans studying AI) will then be a reasonable prediction of what human university freshmen would produce as a median response.


What is the role of the 'fuzzifier' w in Fuzzy Clustering?

Its not required, you can have $m=1$, actually it can be any number $\geq 1$.

Now the better question is why to have it? The answer is that it adds a smoothing effect. Lets look at it in each of the limits ($\lim m \rightarrow 1$ and $\lim m \rightarrow \infty$)

Towards $\infty$, it makes $u_{ij}$ equal to $\frac{1}{c}$, making each point have equal membership of each class regardless of $m$. From the optimization perspective, its saying how can we achieve finding clusters that are closest to all points, therefore by definition it has already achieved that, and so the Loss will always be 0. (at its global minimum)

Now in the other limit, the constants are inversely proportional to the square of the normalized euclidean distance. This makes intuitive sense, the membership is high if they are close, and the membership is low if they are not (relatively)

So why do we have the $m$, its for control. It allows us to choose and experiment with how heavy each distance should hold weight in the membership. An example where a larger $m$ may be useful is when the data isnt clean, and you dont want to rely so heavily on euclidean distance as the membership, so you forcibly add in a smoothing effect


What is the difference between an episode, a trajectory and a rollout?

I don't really think there are fixed, different definitions for all those terms that everyone agrees upon. In most contexts they're going to be quite interchangeable, and if anyone is really using them in a context where they are supposed to have crucially important, different meanings, they should probably precisely define them right there.

  I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s,a,r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.

Agreed with this.

  With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.

I can't really think of cases where it's sensible to talk about trajectories with tuples shuffled into an arbitrary order. I'd still think of trajectories as having to be in the "correct" order in which they were experienced. But I do agree that trajectories can be little samples (for instance, little sequences of experience that we store in an experience replay buffer). So, every full episode would be a (long) trajectory, but not every trajectory is a full episode (a trajectory can just be a small part of an episode).

  I think rollout is somewhere in between, since I commonly see it used to refer to a sampled sequence of $(s,a,r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.

I'd say that... often a rollout should have a "terminal" state as ending, but maybe not a true "initial" state of an episode as start. We might be in the middle of an episode, and then say that we "roll out", which to me implies that we keep going until the end of an episode. I don't think this term is as common as the other two in Reinforcement Learning, but more common in search / planning literature (in particular, Monte Caro Tree Search).

That said, when I'm working with MCTS I often like to put a limit on my rollouts where I cut them off if no terminal state was reached yet... so that isn't exactly a crisp definition either.

Due to how commonly-used this term is specifically in MCTS, and other Monte-Carlo-based algorithm, I also associate a greater degree of randomness with the term "rollout". When I hear "episode" or "trajectory", I can envision a highly sophisticated, "intelligent" policy being used to select actions, but when I hear "rollout" I am inclined to think of a greater degree of randomness being incorporated in the action selection (maybe uniformly random, or maybe with some cheap-to-compute, simple policy for biasing away from uniformity). Again, that's really just an association I have in my mind with the term, and not a crisp definition.


Who are thought leaders in the chatbot space?

One of them is certainly Doctor Richard Wallace.  Doctor Wallace was the original author of the Artificial Intelligence Markup Language spec and is Chief Science Officer at PandoraBots.


What is batch / batch size in neural networks?

You are confusing two concepts, batch size and window size.

Batch Size

The concept of a batch is related to your data and not the number of neurons.  A batch is just a subset of your training data.  The size of the batch affects computational speed, quality, and accuracy.  Large batch sizes will train faster than smaller ones but the model's accuracy can suffer.

There is a rule of thumb that a batch size should be a power of two (e.g. 32, 64, 128, etc.).

Generally speaking larger batch sizes do not generalize as well as smaller batch sizes.  You will need to experiment with the batch size to achieve optimal performance.  Batch size is considered a hyperparameter.

The following video, Batch Size in a Neural Network explained should help clarify things.

Window Size

Window size applies to time series and sequences when using recurrent style networks.  For your case, 365 is the window size.  You did not mentioned how far in the future you are trying to predict.  If we assume that you are predicting one day in the future your output would be the 366th measurement.

Your Application

A batch for you would be a number of training sets.  For example, if we used 32 as a batch size, you would have 32 sets of input windows and outputs.  The first two batches might look like the following:

Batch 1, size 32
1 Data from days 1 to 365 as the input, day 366 as the output
2 Data from days 2 to 366 as the input, day 367 as the output
...
32 Data from days 32 to 396 as the input, day 397 as the output

Batch 2, size 32
1 Data from days 33 to 397 as the input, day 398 as the output
2 Data from days 34 to 398 as the input, day 399 as the output
...


Is the test time the phase when the model's accuracy is calculated with test data set?

If it is not defined otherwise, testing is the phase where the model is passed with new data instances to derive the score of the test set. It should not be confused with validation set.

A validation dataset is a sample of data held back from training your model that is used to give an estimate of model skill while tuning model’s hyperparameters during training. There are a lot of methods of validation with k-fold cross-validation being one of the most popular.

In k-fold cross-validation, the original training set is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for validating the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data for the each epoch. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling is that all observations are used for both training and validation, and each observation is used for validation exactly once.

The validation dataset is different from the test set that is also held back from the training of the model, but is instead used to give an unbiased estimate of the skill of the final tuned model when comparing or selecting between final models.


What is the difference between model and data distributions?

Yes. In Machine Learning we consider that the samples in your training set are sampled from an underlying distribution called the data generating distribution.

Generative models classify the samples by trying to learn the distribution of the data. In most cases, either the model is incapable of doing so, or the training samples aren't enough to properly describe the data-generating distribution, so the model learns an approximation of this. This is what you call the model's distribution.

You can find more info about these concepts in a more detailed answer I wrote. If you're familiar with GANs, you can also read this post, to see where these two concepts come into play when training the two networks.


English language robots which you mentioned, are called "chatter bots". Chatter Bots are used to communicate with a human and undergo conversations in such a way that the human which is communicating will think that he/she is talking to another human.

There are two types of chatter bots: One is which uses certain rules and pattern matching techniques and the other one is which uses actual artificial intelligence techniques. Of course, the latter one is the most difficult to implement.

This can be understood with an example. The former type of bots which uses pattern matching and rules consist of questions stored in the form,

if question matches 'WHO (IS/ARE) (MEMBERS OF PARLIAMENT/PRESIDENT)?' then
     RETURN Y ... where Y is some predefined answer.

The latter part which uses actual AI techniques, uses various mechanisms to actually understand the question, extract information out of it, process the information into some standard normalized form and then do some inference w.r.t. the facts in the knowledge base. Such methods may use learning algorithms to actually learn question patterns.

To make simple bots (probably of the former type), you can use Artificial Intelligence Markup Language (AIML) which is a XML based language for developing chatter bots like ALICE bot.

If you want to focus more towards latter type of chatter bots, you may have to learn about various AI techniques, like searching, logic, knowledge and inference, learning, etc or sub-domains of AI like Natural Language Processing (NLP). There are various tool kits for NLP like for developing in Python there is nltk, for developing in Java there is Stanford's CoreNLP, and so on.


How to generalise over multiple simultaneous dependent actions in Reinforcement Learning

If you want to treat the problem as a full Reinforcement Learning problem, I'd recommend to try avoiding the combinatorial explosion of the action space by treating every sub-action as a separate decision point, a separate full action. If you have, for example, already selected 4 sub-actions for a particular customer, you can try to include those in some way in the state representation / input when moving on to the 5th sub-action. By including already-selected sub-actions in the state space, your algorithm can learn to take into account that optimal prices for some seat types will depend on what prices were already selected for others.

I do suspect such a full RL formulation will still be a difficult problem to learn though, will require huge amounts of experience. It may be worth considering to simplify it anyway, treat it as a Contextual + Combinatorial Multi-Armed Bandit Problem. That way you won't be able to learn long-term effects across multiple different customers as you described in the comments, but you will likely at least be able to learn something that works decently well with less experience. Recently, an interesting new book appeared on MAB problems, which is available for free here. You will find many chapters on Contextual MABs there, and also one chapter (chapter 30) on Combinatorial MABs.

Note that with either of these "combinatorial" approaches, you can also try to play around with the order in which you select sub-actions. For example, the sub-action / price point selected for seat type A might have a significant influence on what the optimal remaining policy would be for other seat types, whereas the sub-action for seat type B might have no influence on other seat types. It would then be useful to always prioritize selecting sub-actions for seat type A. You can try to identify these kinds of effects by keeping track of (co)variances in observed returns.

  Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?

The solutions I proposed above do not do this explicitly, they circumvent the issue by taking multiple sequential decisions. There does appear to be some research in RL with vector-valued actions. For example, the paper Clipped Action Policy Gradient briefly mentions vector-valued actions in Subsection 3.2. I am not personally familiar enough with RL + vector-valued actions to make a direct recommendation as to what approaches do or don't work well, but maybe this can at least help you find more relevant literature if this is a direction you'd like to pursue.

  Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
  Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?

This kind of generalization should again come naturally from using function approximation with either of the solutions proposed above.


Why does the Atari Gym Amidar environment only move after a certain number of episodes?

I reviewed some videos of Amidar on YouTube, and it seems that the game screen is fixed for a few seconds before the start of each level. This is probably to give the human player a chance to rest between levels and maybe even pause the game to take a break, as the gameplay looks very intense.

How do I get this environment to move immediately?

I suggest creating a gym wrapper to change the reset function of the environment to produce a different start state. Since the initial 85 frames are not influenced by the agent's actions, these frames are unhelpful and unnecessary for your agent's training. You could consider calling the step function 85 times in your new implementation of the reset function using a random action; then return the resultant state. When calling your newly implemented reset function, the agent's start state will be frame 85 instead of frame 0. As you noted, frame 85 is immediately followed with many frames of meaningful movement, which will provide higher-quality training data. In contrast, the original implementation of the reset function starts at frame 0, which is followed by 85 unhelpful fixed frames.

Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything.

ATARI environments are difficult even for some of the more powerful RL algorithms. For example, it is one of the harder environments for DQN to solve (see the Nature paper, pg. 531).  Don't worry if your agent isn't learning right away. It could take hours or even days to learn enough to noticeably improve, even with a correct implementation of a powerful algorithm. Good luck, and welcome to AI Stack Exchange!


Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)?

A natural policy to act in an environment with discrete action space would be a softmax.
This paper describes a method that uses the idea of centralized training, and I believe could be used in your implementation.
With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each action to a specific agent (assuming we have three different action spaces). Then you would have a cooperation game with 3 agents, where all of them have a common reward function. In theory, this 3 agents represents an individual agent that interacts with the environment.


How to implement a Continuous Control of a quadruped robot with Deep Reinforcement Learning in Pybullet and OpenAI Gym?

Proposed Approach

  The expected output from the policy is an array in the range of (-1, 1) for each joint.
          The input of the policy is the position of each joint from the past X frames in the environment
          (replay memory like DeepQ Net),
          the center of mass of the body,
          the difference in height between the floor and the body to see if it has fallen and the movement in the x-axis.

  The code below is just a test of the environment with a set of movements hard coded in the robot just to test how it could walk later.
          The environment is set to real time, but I assume it needs to be in a frame by frame lapse during the policy training.

Initial Considerations

It is advisable to first list and analyze the use cases, system requirements, and their design consequences. The following details can be addressed after a high level theory and design have been established.

Algorithm details
Models according to schemas such as URDF
Physics simulators, such as Bullet
Challenge frameworks, such as Gym
Goals in terms of Cartesian direction of travel
Number of limbs
Program variable names
Initialization

Some requirements are generally assumed as part of any useful robotic controller and need not be mentioned explicitly.

Continuous control (discontinuous control and lack of control are poor options)
Hinged movement always has a range (min and max)

Listing more relevant system design constraints such as these can guide analysis and design. This is an abridged list to consider as an example.

No precision in the control of angular speed and velocity at joints.
No vision, audio, radar, sonar, or lidar sensory information available for either position sensing or predictive path planning
Orientation is not directly sensed
Surface coordinates are sensed via triangulation, and that subsystem is already designed and tested
Clearance to ground is known and must be maintained above a configurable constant value
The use of Deep Reinforcement Learning is expected (which, given the mechanical design, implies the maintenance of a walking policy)
The goal is to maintain a particular direction of robot travel
Each limb has two radial degrees of freedom, controlled by an angular position command input to the motion control sub-system
The nominal radial acceleration and maximum radial velocity of angular positions are not known

Extensions of Traditional Control

The AI system must produce stability. An established method is closed loop control. The stability performance of the AI system must at least exceed standard PID control.

$$ k_p \epsilon(t) + k_i \int_0^t \epsilon(t) dt + k_d \frac {d\epsilon(t)} {dt} $$

The integral term can have an inverse exponential memory decay as such.

$$ k_p \epsilon(t) + k_i \frac {\int_0^t e^t \epsilon(t) dt} {\int_0^t e^t dt} + k_d \frac {d\epsilon(t)} {dt} $$

Nonetheless, all terms are first degree in $k$ parameters. With a feed forward network, some competition winning robotics systems have reached a higher level of performance in terms of stability and efficiency of motion.

$$ f_p(P) \epsilon(t) + f_i(P) \frac {\int_0^t e^t \epsilon(t) dt} {\int_0^t e^t dt} + f_d(P) \frac {d\epsilon(t)} {dt} $$

This is not yet tractable, since offline learning will not take advantage of terrain knowledge gained during a robot's walk. If the terrain is a constant, for instance, if it is always a flat plane, and the positioning system is stable and reliable, reinforcement would not be needed. Such an ideal environment for the walk is rare in practice.

Most servo controllers with radial encoder inputs provide positional and dynamic stability using a model similar to the second of the above three equations.

Deep Reinforcement Learning

The development of reinforced learning policy updating is desirable for real world walker control. Although a policy update function can theoretically be learned from walk examples, the distribution of terrain is not easily representable in example terrain, even if the terrains are generated by genetic-algorithm-like designs modified to simulate planetary geological processes over cosmic time frames.

The approaches that show early indications of success involve reward $r$, dynamics $p$, and policy $\pi$ that depend on both on state and action.

$$ \pi (u_t | x_t) \\
 p(x_t | x_t, u_t) \\
 r(x_t, u_t) $$

The study of variants of NAF that leverage replay memory is outlined in Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates, Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine, 2016, using this model.

$$ \mu_{n+1} (x_t) := {argmax}_u{Q_{{\pi}_n} (x_t, u_t)} \\
 f() := \tanh() \; \text{,} \\
 \mu(x) = f(k + K_x) \\
 Q(x,u) = \frac 1 2 {\big(u − \mu(x)\big)}^T P(u − \mu(x)) + x^T (B x + b) + c $$

where P, k, K, B, b, and c are learnable tensors. Gu et. al. explains.

  If f() [was instead] identity, then the expression corresponds to a globally quadratic Q-function
          and a linear feedback policy, though due to the Tanh non-linearity,
          the Q-function is not linear with respect to state-action features.

Reward

Because of the above considerations, defining reward solely on the basis of one axis of position will likely be insufficient to guide effective learning. Various actions have costs in terms of both time and energy consumption and each may affect future move choices in significant ways. Violation of the  ground clearance constraint must produce a penalty. The reward function must guide policy development to properly take these factors into consideration.

To enhance the tractability of the learning challenge, a process block might best be inserted between the deep reinforcement learning component and the controller to perform a trigonometric and Newtonian translation from desired foot positions relative to the center of robot mass and the radial positions on all control axes to achieve those positions. That removes the burden of that translation from the walker's learner.


How to predict the best from a set of messages - best practice

One way you can definitely approach the problem is by using (Deep) Reinforcement Learning (DRL).
YouTube is actually using DRL as well to suggest videos to users in order to maximize users' engagement with their website. For more information (and further references to papers explaining how other major companies implement their recommendation systems), see this paper. Just as a bit of motivation.
Other research into that direction could be found here.
Actually, there are different DRL algorithms available and each might be suitable for approaching the problem you described above from a slightly different angle.
The way how I would roughly approach your problem is as follows:
As I understand it, the goal is the following.
You want to show either of four messages to a user (only one at a time) and always display the message, which is expected to reduce some metric (=conversion time), to the user.
In that case, an DRL algorithm, or agent, could directly be trained on minimizing the conversion time without having to explicitly produce conversion time estimates per message & user. That might make phrasing the problem at hand much easier (in terms of modelling the learning task).
The outcome of training an DRL agent consists of a so-called optimal policy network, which dictates the agent which message to display to a given user, upon observing the provided user data, such that conversion time gets minimized for that user. The aforementioned network is a simple artificial Neural Network (NN) or Recurrent NN (RNN).
Training the DRL agent will consist of two steps being repeated many times in alternating order.
One step will include the sampling/generation of training data.
For this step, the algorithm (initially being untrained) will be applied to the problem at hand and predict messages for given users. Then, (in addition to the provided user data that was fed as input to the agent) each user's response (i.e. the conversion time) is recorded. This can later be used as the reward that the agent receives for having selected the selected message given the provided user data. Here, a low conversion time translates to a high reward.
In the second step, the agent's policy network will be trained in order to make the agent's predictions more accurate in the next round of generating training data. Here, the data recorded during the previous step will be used.
How the policy network is optimized largely depends on which DRL algorithm you choose. The range of different methods varies vastly. Some algorithms try to estimate utilities (i.e. measures of goodness) per action given a certain state (here, state = provided user data used to generate a prediction). In this case, an action would be choosing message A or choosing message B etc. A popular algorithm implementing this procedure is Deep Q-Learning.
However, if you rather want to predict probabilities per action, then Proximal Policy Optimization might rather be what you are looking for.
Upon convergence, i.e. stabilization of the goal metric (=convergence time per user), the algorithm has arrived at its optimal policy (ideally speaking).
The only drawback with this approach is that DRL usually requires quite a lot of training. But maybe you are lucky and pre-trained recommendation system models exist.
If there is still anything unclear, feel free to ask in the comments.


Does the input layer have bias and are there bias neurons?

The purpose of the input layer is just to conceptually represent the input and, in case it is necessary, define the dimensions of the input that the neural network expects. In fact, some neural networks, such as multi-layer perceptrons, expect a fixed-size input, but not all of them: fully convolutional networks can deal with inputs of different dimensions.
The input layer doesn't contain neurons (although in the diagrams that you will come across they are usually represented as circles, like the neurons, and that's probably why you are confused!), so it also does not contain biases, linear transformations, and non-linearities. In fact, in the context of neural networks, you could define a neuron as some unit/entity that performs a linear or non-linear transformation (to which you can add a bias). Note that the hidden and output layers can contain biases because they contain neurons that perform a linear or non-linear transformation.
However, although I have never seen it (or I don't recall having seen it), I would not exclude the existence of an input layer that transforms or augments the inputs before passing them to the next layer. For example, one could implement a neural network that first scales the input to a certain range, and the input layer could do this, although, in practice, this is typically done by some object/class that does not belong to the neural network (e.g. tf.data.Dataset).


When should I use a linear activation instead of ReLU?

The activation function you choose depends on the application you are building/data that you have got to work with. It is hard to recommend one over the other, without taking this into account.

Here is a short-summary of the advantages and disadvantages of some common activation functions:
https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/

  What does the author mean with ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.

ReLU is good for inputs > 0, since ReLU = 0 if input < 0(which would kill the neuron, if the gradient is = 0)

To remedy this, you could look into using a Leaky-ReLU instead.
(Which avoids killing the neuron by returning a non-zero value in the cases of input <= 0)


How does using ASIC for the acceleration of AI work?

Tensor operations

The major work in most ML applications is simply a set of (very large) tensor operations e.g. matrix multiplication. You can do that easily in an ASIC, and all the other algorithms can just run on top of that.


How can one find / collect data for, and come up with ideas for, using Deep Learning / AI to improve one's everyday life?

Speaking to the "collecting data" part of the question, I'll say this:

Keep in mind that not everything requires massive amounts of data. Consider also that large amounts of data about all sorts of things are available in the OpenData / LinkedData realms. Governments in particular are sources of massive amounts of data.  See, for example, data.gov or Google around and see if your (state|city|county|whatever) has an "open data" portal. You might be surprised.

Finally consider that inexpensive sensor systems and distributed sensor networks can be constructed fairly easily / inexpensively using things like Arduino / Raspberry Pi and communications protocols and software often associated with the term Internet of Things.  You may find that you can actually collect your own data, especially if you crowd-source the effort and get a bunch of other people involved.

See also:

http://opendata.reddit.com

http://datasets.reddit.com

http://www.wikidata.org

http://www.dbpedia.org

etc.


Are there Python packages for recent Bayesian optimization methods?

Apart from the Scikit-Optimize package related to Scikit-Learn, following are some of the packages related to Bayesian optimization:

GPyOpt
pyGPGO
Hyperopt
bayesian-optimization
safeopt
RoBO


What is the difference between reinforcement learning and AutoML?

Automated machine learning (AutoML) is an umbrella term that encompasses a collection of techniques (such as hyper-parameter optimization or automated feature engineering) to automate the design and application of machine learning algorithms and models.

Reinforcement learning (RL) is a sub-field of machine learning concerned with the task of making decisions and taking actions in an environment so as to maximize (long-term) reward (which is the goal of the so-called RL agent). RL is (at least partially) based on the way animals (including humans) learn. For example, the usual way of training a dog to perform a certain task is to reward it with food whenever it takes the correct action (for example, jumping, if you want the dog to jump whenever you make a certain gesture with your hand). In this case, the RL agent is the dog, the task the dog needs to perform (e.g. jumping) is the environment, food is the reward and the goal is to get food.

Given that reinforcement learning (RL) is a sub-field of machine learning, then, in principle, AutoML can also be used to automate the design of RL algorithms, models or agents. For example, if you use a neural network to represent the policy (the function the determines which action to take in the environment), then you can potentially use AutoML to find the most appropriate architecture (for example, the most appropriate number of layers) for this neural network.


How to get good results with GAN and some thousands of images?

With a Google Cloud V100 GPU the GAN would run a week to two with
  default parameters. Does this sound realistic time for this kind of
  dataset? It's definitely not feasible for me.

Yes, V100s are quite beefy. You shouldn't even need a week. Obviously this is based on my experience with various problems, rather than a concrete calculation.

  Is 4000 images enough to train a GAN from scratch?

For the size you want to generate, it is still on the edge of what would constitute a decent training set. You will get some results(depending on architecture) but will probably want to grab more data if at all possible.

  Is there any implementation with pytorch/keras that would be good to
  get nice results with?

I would check out this link: https://github.com/eriklindernoren/Keras-GAN. It has some nice implementations in Keras. As far as the particular GAN to use, I would start out with a vanilla GAN that fits your purposes and focus on toying with hyperparameters, and if that doesn't work, look into one of the other variations that correlate well with your problem.


Finding the right questions to increase accuracy in classification

The problem you're trying to address can, in some sense, be viewed as a Feature Selection problem. If you look for literature using only those words, you're not going to find what you're looking for though. In general, "Feature Selection" simply refers to the problem where you already have a large amount of features, and you're simply deciding to select which ones to keep and which ones to throw away (because they're not informative or you don't have the processing power to try training with all features for example).

I'd recommend looking around for a combination of "Feature Selection" and "Cost-Sensitive". This is because, in your case, there are costs associated with selecting features; values may be costly to obtain for some features. Searching for this combination leads to publications which look to be interesting for you, such as:

Cost-sensitive feature selection using random forest: Selecting low-cost subsets of informative features
Cost-sensitive Dynamic Feature Selection
Cost-Effective Feature Selection and
Ordering for Personalized Energy Estimates
and probably much more...

I cannot personally vouch for any of those techniques since I've never used them, but those papers certainly look relevant for your problem.

When you're looking around for more literature, terms like "cost", "cost-based", maybe "budgeted" are crucial to include. If you don't include those, you're just going to get papers on problems like:

Feature Selection: given a set of features/columns, which ones am I going to use across all samples/instances/rows?
Feature Extraction: given data (typically without clear human-defined features, like images, sound, etc.), how am I going to extract relevant features from this?
Active Learning: given a bunch of samples without labels but feature values already assigned, which one would I like an oracle/human expert/etc. to have a look at so that they can tell me what the true label is?

Those kinds of problems all do not really appear to be relevant in your case. Active Learning may be somewhat interesting in that it is about trying to figure out which rows would be valuable to learn from, whereas your problem is about which columns would be valuable to learn from. There does seem to be a connection there, Active Learning techniques might to some extent be able to inspire techniques for your problem, but just that; inspire, they likely won't be 100% directly applicable without additional work.


What if there's a game where all the AI people actually lived?

Yes, a baby can be considered an AI. Will this be our future? I don't know. That is exactly what some people are looking for, to create an AI that can live. We have several AIs that each time more surprises us. But none of them question their own existence, none of them wants to know or require attention from their god (developer) because of their reflections. I believe that what we create today can be equated with a mosquito of our real life: a brainless "being" that performs simple actions.

The cool thing about believing in such a future with Artificial Intelligence is that we do not have to repeat our mistakes. We do not need an AI evolving according to age, it can be born with a superintelligence, knowing everything, learning only new things that are not foreseen.

Some humans (and I) have a passion for the human, for every detail and imperfection, and dream of re-creating something as mysterious and complex as through an AI. It seems something impossible, some believe that it would be impossible only because who created us would be a perfect being (god). But others (me) do not stop at this issue, and continue the quest to create an AI with some characteristics of the human, such as: feelings, reflecting on one's own acts, ...

But all this, as far as I know, is surreal and only projection of our mind. It is already something interesting, for creating new challenges for humanity.


Reason for issues with correlation in the dataset in DQN

I wanted to understand what does this instability and divergence mean ?

These are with reference to learning curves for the neural network. If a neural network is stable and converges, it means that the value of the error or cost function reduces consistently over time and reaches a stable point of minimum error.

In practice this is often a noisy process and not smooth. There are degrees of noise that are expected and acceptable when solving real problems. However, an instable learning curve will oscillate wildly, and a divergent learning curve will get consistently worse error function during training.

A typical stable, converging learning curve, cost function versus training data consumed, might look like this:

Whilst an instable, diverging learning curve might look like this:

These are not using the same vertical scale - the lowest point of the unstable curve will typically be higher than most or even all of the stable curve.

  And why correlated data causes this instability.

This is because for gradient descent to work, it needs gradient samples that it uses on every weight update step to be unbiased estimates of a true gradient. In RL you have either an online learning process or non-stationary targets (and usually both), so you must use stochastic or mini-batch gradient descent, working with a few samples at a time. You need those samples to be independent, not related to each other other than by random chance, otherwise the gradient value will be biased and gradient descent will consistently make updates in the wrong overall direction.

A good way to illustrate the difference is to use a really simple example, using gradient descent updates to estimate a mean value (this is roughly equivalent to training a neural network with one neuron, with weight fixed to zero and learning a bias value to represent the mean of the target- no actual input is required).

Say we have an array of values from 0 to 200 inclusive (example code is in Python):

import numpy as np
train_y = np.arange(0, 201)

If this array is kept sorted, then sequential values are highly correlated. If you plot consecutive pairs against each other, you will get a straight line.

We can estimate the mean value by setting a "bias" value to some arbitrary number, and running an update rule based on MSE (between current bias and observed value):

mean_estimate = 0.0
alpha = 0.1
for y in train_y:
    mean_estimate += alpha * (y - mean_estimate)
print(mean_estimate)

This prints (roughly) $191$ as the estimate for the mean, almost double.

However, if we shuffle the array first, it removes the correlation. If you plot consecutive pairs against each other, you will get a scatter graph with no apparent pattern. Adding the one line np.random.shuffle(x) to do this changes the results radically:

mean_estimate = 0.0
alpha = 0.1
np.random.shuffle(x)
for y in train_y:
    mean_estimate += alpha * (y - mean_estimate)
print(mean_estimate)

We get much better estimates (typically between $90$ and $110$), closer to the true value of $100$, and not biased to be higher or lower (run it enough times and you would find the expected result of this algorithm is very close to the true value).

This is because in the first version of the code, the gradient of the error was not sampled fairly - it kept pointing "up" even for high estimates due to the correlation. In the shuffled version, gradients are likely to be in either direction depending only on the current estimate, and will appear roughly in the ratios necessary to find the correct value.

As an exercise you could extend this simple example with mini-batches and repeated "epochs" to show that the effect persists with those changes, and the shuffling is the most important change here for better estimates.


Are Convolutional Neural Networks better than existing image recognition libraries that don't use CNNs?

It would not be wise to say that CNNs are better objectively than traditional approaches to solve computer vision problems as there are many problems for which the traditional methods works just fine. CNNs do have an inherent advantage over traditional methods which is the same advantage that deep learning has over other traditional methods i.e learning hierarchical features i.e what features are useful and how to compute them.

The traditional way to approach a CV problem is to figure out the features that are relevant to the problem, figure out how to compute those features and then use those features to compute the final result. Whereas in CNN case the training process will figure out all the 3 points for you given that you have huge number of training examples.


How to calculate the advantage in policy gradient functions?

First let us note the definition of the advantage function:

$$A(s,a) = Q(s,a) - V(s) \; ,$$

where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could represent these by two different function approximators, but this would be quite inefficient. However, note that
$$Q(s,a) = \sum_{s',r} \mathbb{P}(s',r|s,a)(r + V(s') = \mathbb{E}[r + V(s')|a,s]\;,$$
so we can actually use a single function approximation, for $V(s)$, to completely represent the advantage function. To optimise this function approximator you would use the returns at each step of the episode as in e.g. the REINFORCE algorithm like you mentioned.


How many iterations are required for iterative-lengthening search when step costs are drawing from a continuos range [ϵ, 1]?

In the title of the question, you write (emphasis mine):

  step costs are drawn from a continuous range $[\epsilon, 1]$

If step costs are drawn from that range, it means that every step has a cost of at least $\epsilon$, and at most $1$. This leads back to the case that described in the question that you already understand.

Note that the book (at least, the third edition of the book) does state in the question that $0 < \epsilon < 1$. Of course, the question would not make much sense in the first place if negative costs were allowed.


How do you measure multi-label classification accuracy?

Your intuition is correct. We do use other metrics for multi-label classification. The meaning of evaluation itself changes. Apart from grading the classifier on whether it classifies correctly or not, we also have to penalize it, if it chooses the wrong class appropriately. You could use the following metrics:

micro/macro averaging Recall/Precision, etc.
Hamming Loss
Subset Accuracy