What is the difference between image processing and computer vision?

The Wikipedia article related to computer vision gives, in my opinion, a good description of the field and its relation to image processing. Below, I will only cite the most relevant parts of the article.

Computer vision is an interdisciplinary scientific field that deals with how computers can be made to gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to automate tasks that the human visual system can do.
The image data can take many forms, such as video sequences, views from multiple cameras, or multi-dimensional data from a medical scanner.
Sub-domains of computer vision include scene reconstruction, event detection, video tracking, object recognition, 3D pose estimation, learning, indexing, motion estimation, and image restoration.
The fields most closely related to computer vision are image processing, image analysis and machine vision. There is a significant overlap in the range of techniques and applications that these cover. This implies that the basic techniques that are used and developed in these fields are similar, something which can be interpreted as there is only one field with different names. On the other hand, it appears to be necessary for research groups, scientific journals, conferences and companies to present or market themselves as belonging specifically to one of these fields and, hence, various characterizations which distinguish each of the fields from the others have been presented.
Image processing and image analysis tend to focus on 2D images, how to transform one image to another, e.g., by pixel-wise operations such as contrast enhancement, local operations such as edge extraction or noise removal, or geometrical transformations such as rotating the image. This characterization implies that image processing/analysis neither require assumptions nor produce interpretations about the image content.

What is the difference between computer vision and image processing?
Computer vision is about gaining high-level understanding from images or videos. For example, object recognition, which is the task of identifying the type of objects (e.g. apples or humans) in an image, is a computer vision problem. Of course, this task requires a high-level understanding of the image, that is, an understanding of the image similar to the way humans understand visual inputs, given that an apple is a high-level object that is composed of atoms, can be green, etc. For example, a neural network that attempts to classify the type of object in an image (assuming, for simplicity, there is just one type of object) would be a computer vision technique. In computer vision, you receive an image as input and you can produce an image as output or some other type of information (e.g. the type of objects in the image).
On the other hand, image processing does not necessarily imply a high-level understanding of the image. Image processing is a subfield of signal processing but applied to images, which are $2$d signals (or functions of a fixed domain). So, for example, if you have a blurred or noisy image, the task of deblurring or denoising it is part of image processing. The typical tasks in image processing are filtering (e.g. using the Gaussian filter or the mean filter), noise removal, edge detection and color processing. In image processing, you receive an image as input and you produce another image as output.
However, note that, in many cases, to gain a high-level understanding of the images, you first need to e.g. denoise them, so you could use an image processing technique to partially solve a computer vision task. In this sense, computer vision is an interdisciplinary field.
To conclude, computer vision is not a subfield of image processing, given that image processing does not necessarily involve a high-level understanding of images. On the other hand, computer vision can use image processing techniques to gain a high-level understanding of images.


What is a weighted average in a non-stationary k-armed bandit problem?

The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.

More specifically, if you denote the rewards by a vector $X$, the weighted average will be taking the dot product between $X$ and a vector $W$ such that $0 \le W_i \le 1$ and the sum of all $W_i$ is 1.

If each $W_i = 1/n$ it will be a weighted average (a.k.a the mean). Using the exponential decay $W_i = \alpha^i/ (\sum W_i)$ is also a weighted average.

Then both strategies to compute the Q value use weighted average of previous reward.


Why does adding another network help in double DQN?

As the authors of this paper state it:

In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation.  A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.

If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.


Algorithm For Making Balanced Weapons In A Game?

I'm going to start by trying to restate your problem as I understand it.

You have a game which contains weapons.
Weapons are characterized by 5 different numbers, which can range over different values (1-5 in your examples?).
You have a way to simulate combat involving the two weapons.
The combat is random, but can be repeated many times. An average win rate can be determined.
You are looking for an AI algorithm that would take in a lot of pairs of statistics, along with the average win rates for one over the other, and give you insight into how to make the average win rate as close to 50% as possible.

If this sounds right, then fundamentally your problem is a form of regression, which something you could use AI for, but probably don't need to. However, your problem is probably not linear, so you need the interactions between the features. Here's what I suggest:

For each pair of weapons, store a comma separated list consisting of the stats for each weapon (one by one), followed by wins1 - wins2. At the top, list out the names of each attribute, separated by commas, (e.g. weapon1Str, weapon1Range, ... ,weapon1-weapon2 Then use a language like R that has simple support for complex forms of regression.

In R, this is then as simple as:

data <- read.csv(file="Myfile.csv")
lm(formula = dist ~ .*., data = data)

This should produce a list of "coefficients", one for each of the attributes, and one for the interaction between each pair of attributes, which form a lengthy quadratic equation in 10 variables.

Any zero of that equation should be a pair of weapons that minimizes this difference.

That's probably the place to start. If it doesn't work out, maybe come post a different question and we can help more.


GPT-2: (Hardware) requirements for fine-tuning the 774M model

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 easily takes more than 10GB of VRAM. You might be able to run GPT-2 774M if you run it in FP16.
Alternatively, you can use Google Collab TPUs which provide at 11GB+ VRAM. Here's a good source listing a few posts about fine tuning GTP-2 1.5B on Google Collab TPUs: https://news.ycombinator.com/item?id=21456025
And here's the notebook itself demonstrating the process: https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a


What is the meaning of test data set in naive bayes classifier or decision trees?

Your assumption about the test data is not correct completely. Maybe you use the test data to tune your learning algorithm to work better on the test data, but it's not the whole thing. Sometimes you need to know that the ML method is working or not and have a sense about how much does it work!

You have other scenarios that you want to evaluate your method:

Compare the result of the leaner with other techniques. For example, you are considering DT versus an SVM classifier over a data set. If you want to compare them, you need a value to found such a sense about the comparison.
Sometimes you are using an ensemble method and you want to tune some parameters to balance between using different ML methods. Hence, you need to evaluate these learning methods (DT, Naive Bayes) to improve the ensemble method.


DQN stuck at suboptimal policy in Atari Pong task

Incase you still haven't been able to resolve the problem, here's a link to the answer to my own question, which has the step-wise changes I made to achieve a +18 average score saturation using just a 10000 replay buffer size and a normal Double DQN (DDQN), trained for about 700-800 episodes. The updated code can also be found here.

No fancy changes like Prioritized Replay Buffer or any secret hyperparameter changes are required. It's usually something simple, like a small problem with the input preprocessing step.


How are vectors and matrices multiplied in supervised machine learning?

In general, when people do not explicitly state it, a vector $v \in \mathbb{R}^n$ is usually considered a "column vector", that is, you can think of it as the matrix $v \in \mathbb{R}^{n \times 1}$ (that is, a matrix with $n$ rows and $1$ column).
If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "row vector" (because it might be the case that the author is implicitly considering the vectors as row vectors).

Having said that, you can multiply a vector $v \in \mathbb{R}^n$ by a matrix $A \in \mathbb{R}^{n \times m}$ from the left, that is, you can do $v^T A$. Here, I considered $v$ has a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), even though I have not explicitly stated it: you can and you often need to deduce this from the context! If I transpose $v$, I obtain $v^T \in \mathbb{R}^{1 \times n}$, and, thus, you can see that you can indeed perform the operation $v^T A = u \in \mathbb{R}^{1 \times m}$. Note that, at this point, $u$, the vector resulting from the operation $v^T A$, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply $v$ from the right of $A$, because there is no way of making the dimensions match. Have a look at this question, if you do not know how to multiply a vector by a matrix from the left.

Similarly, you can multiply $v \in \mathbb{R}^n$ by the matrix $B \in \mathbb{R}^{m \times n}$ only from the right. If $v$ is a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), you need to do $B v \in \mathbb{R}^{m \times 1}$, but, if $v$ is a row vector (that is, $v \in \mathbb{R}^{1 \times n}$), you will first need to transpose it, so that you can perform the operation: $B v^T\in \mathbb{R}^{m \times 1}$.

Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.

To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.


So you want your network to represent those 3 values at each step as single composite value? I can't think of any better way than utilizing 3 LSTM units but attaching them to same write and read nodes of the enclosing network. In other words your assumption that it makes sense to keep all those 3 values together, gets hardcoded into your network by making 6 connections (3 connected to read and 3 to write) share their weights. Usually researchers leave it to the network to decide whether to keep such composite values together or separate (by learning read and write weights through backpropagation) and sometimes network chooses to keep them together all on its own like in https://youtu.be/93rzMHtYT_0?t=531 (it can be seen how network resets whole bunch of LSTM units simultaneously ).


Why does self-playing TicTacToe not become perfect?

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have given in the question and comments, I can make a few observations and guesses about where to look:

Your metric of average reward against a random player is sensible. In this case, you could also use a perfect player (that ideally randomised choice of any optimal move), where you would see a maximum averaged return of zero - this would be helpful to know if your agent had learned a fully optimal behaviour, because it would consistently score zero. In general for more complex games a perfect player is not available to test with, but as you are learning here it might help you.
Your DQNs might be unable to fit the value function. You can test that in this case by getting the value function from an optimal self-play player (all the values will be -1, 0, or 1) and using a supervised learning approach, separately from your agent. You should be able to get a loss very close to zero - if you cannot do that, then something could be wrong with your network architecture.
Whilst you are training, even though you are using a variation of Q-learning (which learns an optimal policy even whilst exploring other actions), your DQNs are not learning optimal play. That is because you have used two agents. In DQN, the algorithm is not aware that there are other learning agents, and it will treat any other agents as if they were part of the environment. Which means that the agents will spend some effort trying to set the game up for each other to make an exploration mistake. That could lead to non-optimal choices and a little bit of instability. Your decay of epsilon should help with that, although you are caught between a rock and a hard place here. You want to learn off-policy and explore, but are forced to reduce exploration. There are a couple of ways to resolve that, I will explain a bit further down . . .
10,000 games may not be enough. In the experiments I have done with TicTacToe agents, it seems between 20,000 and 50,000 games are required for a naive learner. More may be required if you have done something that makes learning inefficient. In addition, I found when adding more sophisticated learning approaches (in my case using eligibility traces) the agents appear to become close to optimal very quickly, but actually have flaws which take a long time to shake out, just as long as running a more naive algorithm. When the flaws got found and fixed, it upset the value function for a while and I saw fluctuations in my metrics similar to yours.
Q-learning with NNs is inherently unstable. DQN implements some ideas to fix that, but it is not perfect. It is not uncommon to need to adjust the batch size and/or time steps between taking frozen copy of network for the TD target calculation. The initial stability followed by poor performance looks a lot like that instability too.

Regarding your use of two opposing agents, I can see two possible improvements:

Alternately train one or other agent in each game, don't train both at once. That will mean each agent is learning to play against the other agent playing its best without exploratory moves.
Combine networks into single agent description. As this is a zero-sum game, you can take player A's network for calculating values, and just have player B try to minimise the action value on its turn. That means use min and argmin functions for steps that represent player B's turn wherever player A would use min or argmin, including in the Q-value updates - this is typically easy to add to the inner loop of Q-learning, and should improve learning efficiency (essentially you are hard-coding knowledge that this is a zero-sum game and taking advantage of that symmetry).

Both of these ideas will free you up from caring about the value of epsilon, or decaying it - you can probably just leave it fixed at e.g. 0.1

Finally, as a test of whether your agent can cope with learning optimal play in general, you could have it learn against an already optimal agent. That is obviously not something you can do for more complex games, but might help you debug agent code and hyper-parameters of the network - it divides your problem up into "can it learn this at all" and "can it learn through self-play".


How exactly does SGD with momentum works in keras

I noticed the same issue. Also, it seems that the equation for applying momentum does not exactly match up with the standard one:

$$ v_{d\theta} := \beta v_{d\theta} + (1 - \beta)d\theta \\
 \theta := \theta - \alpha v_{d\theta}$$


deep learning with kfold cross validation with epochs

Here's a good resource showing you the exact code for what you're trying to do.

Here's the particular snippet:

# evaluate a model using k-fold cross-validation
def evaluate_model(dataX, dataY, n_folds=5):
    scores, histories = list(), list()
    # prepare cross validation
    kfold = KFold(n_folds, shuffle=True, random_state=1)
    # enumerate splits
    for train_ix, test_ix in kfold.split(dataX):
        # define model
        model = define_model()
        # select rows for train and test
        trainX, trainY, testX, testY = dataX[train_ix], dataY[train_ix], dataX[test_ix], dataY[test_ix]
        # fit model
        history = model.fit(trainX, trainY, epochs=10, batch_size=32, validation_data=(testX, testY), verbose=0)
        # evaluate model
        _, acc = model.evaluate(testX, testY, verbose=0)
        print('> %.3f' % (acc * 100.0))
        # stores scores
        scores.append(acc)
        histories.append(history)
    return scores, histories

So you're using .fit in a loop over the folds set up by sklearn.model_selection.KFold.


Is there any good source for when the pole actually starts all the way at the bottom, in the cartpole problem?

It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.

A very similar environment that does have a lot more written about it is Acrobot, which does have a OpenAI Gym version. Instead of a cart on a track with forces applied and a free join to the pole, there is a longer pole fixed to a free joint, with an active joint in the middle (that the agent can apply forces to). It can be thought of as a very simple model of an acrobat on a trapeze swing (with poles instead of chains, so the swing is stable balanced upside-down)

The degrees of freedom and difficulty of the Acrobot task are similar to CartPole - I would rate it as a harder problem overall, but if you started CartPole with the pole hanging down in fact the two problems are very similar. Usually the joint motor is made too weak to achieve the goal of balancing in a single clean action, and the agent must learn to build momentum over a few swings, before moving to the balance point. That also makes both your alternate start position CartPole and Acrobot similar to MountainCar.

For a low-effort look at a very similar problem you could try Acrobot. Otherwise you will likely need to do some custom work on CartPole.


The Control Problem is, in short, the idea that AI will eventually be much better decision-makers than humans. If we don't set things up correctly beforehand, we won't get a chance to fix it afterwards, because AI will have effective control.

There are three main areas of discussion with regards to the Control Problem:

Whether or not the problem is urgent. Many AI experts, cognizant of the difficulty of getting simple systems to work effectively today, think that AI able to take control is not urgent, and as detail-minded engineers, they think it will be profoundly difficult to do any useful work today. (Andrew Ng, for example, famously called these sorts of worries like worrying about overpopulation on Mars.) Given radical uncertainty among AI experts as to when this will become an issue, however, this means we can't rule out rapid AI timescales, and should do at least some work in anticipation of those timescales.
Whether or not the problem is hard. Many people give short, simple, and wrong solutions to the control problem. Probably the most famous is the idea that intelligence and morality are inherently interlinked, and thus a more intelligent machine, by definition, will be more moral. The Orthogonality Thesis is the claim of the opposite, that intelligence and morality (or, more specifically, goal alignment) are unrelated things.
What foundations we can lay now. There are a bunch of open problems (see, for example, MIRI's technical agenda) that deal with mathematical logic of the sort that would be useful for ensuring robust value alignment, or on how to effectively do value learning (without giving an incentive to distort values), or on how to build value functions and goals such that they are fixable if they turn out to be mistaken. Those look like problems that we can do useful work on now, even without knowing what the actual structure of a future AI will look like.


Understanding notation of Goodfellow's GAN objective function

To understand this equation first you need to understand the context in which it is first introduced. We have two neural networks (i.e. $D$ and $G$) that are playing a minimax game. This means that they have competing goals. Let's look at each one separately:

Generator

Before we start, you should note that throughout the whole paper the notion of the data-generating distribution is used; in short the authors will refer to the samples through their underlying distributions, i.e. if a sample $a$ is drawn from a distribution $p_a$, we'll denote this as $a \sim p_a$. Another way to look at this is that $a$ follows distribution $p_a$.

The generator ($G$) is a neural network that produces samples from a distribution $p_g$. It is trained so that it can bring $p_g$ as close to $p_{data}$ as possible so that samples from $p_g$ become indistinguishable to samples from $p_{data}$. The catch is that it never gets to actually see $p_{data}$. Its inputs are samples $z$ from a noise distribution $p_z$.

Discriminator

The discriminator ($D$) is a simple binary classifier that tries to identify which class a sample $x$ belongs to. There are two possible classes, which we'll refer to as the fake and the real. Their respective distributions are $p_{data}$ for the real samples and $p_g$ for the fake ones (note that $p_g$ is actually the distribution of the outputs of the generator, but we'll get back to this later).

Since it is a simple binary classification task, the discriminator is trained on a binary cross-entropy error:

$$
J^{(D)} = H(y, \hat y) = H(y, D(x))
$$

where $H$ is the cross-entropy $x$ is sampled either from $p_{data}$ or from $p_g$ with a probability of $50\%$. More formally:

$$
x \sim
\begin{cases}
p_{data} \rightarrow & y = 1, & \text{with prob 0.5}\\
p_g \;\;\;\,\rightarrow & y = 0, & \text{otherwise}
\end{cases}
$$

We consider $y$ to be $1$ if $x$ is sampled from the real distribution and $0$ if it is sampled from the fake one. Finally, $D(x)$ represents the probability with which $D$ thinks that $x$ belongs to $p_{data}$. By writing the cross-entropy formula we get:

$$
H(y, D(x)) = \mathbb{E}_y[-log \; D(x)] = \frac{1}{N} \sum_{i=1}^{N}{ \; y_i \; log(D(x_i))}
$$

where $N$ is the size of the dataset. Since each class has $N/2$ samples we can split this sum into two parts:
$$
= - \left[ \frac{1}{N} \sum_{i=1}^{N/2}{ \; y_i \; log(D(x_i))} +  \frac{1}{N} \sum_{i=N/2}^{N} \; (1 - y_i) \; log((1 - D(x_i))) \right]
$$

The first of the two terms represents the the samples from the $p_{data}$ distribution, while the second one the samples from the $p_g$ distribution. Since all $y_i$ are equally likely to occur, we can convert the sums into expectations:

$$
= - \left[ \frac{1}{2} \; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \frac{1}{2} \; \mathbb{E}_{x \sim p_{g}}[log \; (1 - D(x))] \right]
$$

At this point, we'll ignore $2$ from the equations since it's constant and thus irrelevant when optimizing this equation. Now, remember that samples that were drawn from $p_g$ were actually outputs from the generator (obviously this affects only the second term). If we substitute $D(x), x \sim p_g$ with $D(G(z)), z \sim p_z$ we'll get:

$$
L_D = - \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

This is the final form of the discriminator loss.

Zero-sum game setting

The discriminator's goal, through training, is to minimize its loss $L_D$. Equivalently, we can think of it as trying to maximize the opposite of the loss:

$$
\max_D{[-J^{(D)}]} = \max_D \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

The generator however, wants to maximize the discriminator's uncertainty (i.e. $J^{(D)}$), or equivalently minimize $-J^{(D)}$.

$$
J^{(G)} = - J^{(D)}
$$

Because the two are tied, we can summarize the whole game through a value function $V(D, G) = -J^{(D)}$. At this point I like to think of it like we are seeing the whole game through the eyes of the generator. Knowing that $D$ tries to maximize the aforementioned quantity, the goal of $G$ is:

$$
\min_G\max_D{V(D, G)} = \min_G\max_D \left[\; \mathbb{E}_{x \sim p_{data}}[log \; D(x)] + \; \mathbb{E}_{z \sim p_{z}}[log \; (1 - D(G(z)))] \right]
$$

Disclaimer:

This whole endeavor (on both my part and the authors' part) was to provide a mathematical formulation to training GANs. In practice there are many tricks that are invoked to effectively train a GAN, that are not depicted in the above equations.


Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.
We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.
We can control the height of the tower with the weights.
First in 2d:

Unfortunately the closer we put this towers together the more they will start two influence each other.

But we can counter that with a negative tower between them.

Now in 3d:

This Answer is work in progress I will update it when i find out something new.


What weights should I use while back-propagating?

First I will assume you notate $y$ as the models output and $z$ as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:

$
\begin{align*}
\frac{dE}{dw_{ij}^1} &= \frac{dE}{dy}\frac{dy}{dw_{ij}^1} \\
&= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3}\frac{dn_{k}^3}{dw_{ij}^1} \\
&= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3} \frac{\partial n_{k}^3}{ \partial n_{j}^2} \frac{dn_{j}^2}{ dw_{ij}^1}  \\
&= -2*(z-y)\sum_kw_{kl}^3w_{jk}^2 x_i  \\
\end{align*}
$

So the reason you are having trouble to figure out which $k$ index to use, the answer is because you need to use both and sum over them. The $l$ index is just the only $l$ index that exists because you only have one node in that layer.


Which movies have the most realistic artificial intelligence?

Just A Rather Very Intelligent System (J.A.R.V.I.S.) in Iron Man (and related films, such as The Avengers) is something (a personal assistant) that people are already trying to develop, so JARVIS is a quite realistic artificial intelligence. Examples of existing personal assistants are Google Assistant (integrated into Google Home devices), Cortana, Siri and Alexa. There are other virtual assistants, but, unfortunately, there aren't many reliable open-source ones. Note that JARVIS is way more intelligent and capable than the other mentioned personal assistants.

Similarly, HAL 9000, in 2001: A Space Odyssey, is a sentient artificial intelligence which can be considered a personal assistant.


What is the equation of the separation line for this neuron with identity activation?

I found the answer, the output u is not related to plotting x2 against x1 here, coz u is like z axis and seeing the separation line is looking perpendicularly to the x1x2 plane, and whatever u value is, it's just a dot.

Put u as zero for the solving steps in the question.

So the separation line equation is this (with any activation function):

$x_2 = \dfrac{-w_1}{w_2}x_1 + \dfrac{-b}{w_2}$


Do you need to store prevous values of weights and layers on recurrent layer while BPTT?

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

Notation of my explanation is different from notation in the figure of question.
My notation is as below:

V - Hidden Layer - Output Layer (gradients of V are independent of
previous time steps)
U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed slightly different notation than in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/enter link description here

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.


In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.


The min and max input size should be the min and max image size of the input images. The numbers represent pixels in both axis of the image.

The anchors represents the size of the anchor box. Anchor box does not have coordinates, only have size.

Hope I can help you


How many trees should be generated in a random forest?

The number of estimators in Random Forest is a hyper-parameter. If you are using SKLearn's Random Classifier you can use one of the following techniques to find a (near) optimal hyperparameter settings (Note:You can tweak other hyperparameters like min_leaf_size etc as well with this approach);

GridSearchCV You can specify a grid of all the hyperparameters and a scoring criteria. This function will then evaluate all combinations of these parameters for you and return the setting which performed best on the validation set.

RandomSearchCV You can specify a grid of all the hyperparameters and a scoring criteria. This function will then evaluate n randomly choosen combinations of these parameters for you and return the setting which performed best on the validation set.

Bayesian Optimization You can treat the hyperparameter settings and corresponding score as a black box function and use exploitation-exploration paradigm using bayesian optimization.


Are there any anthropological studies involving AI right now?

As far as I can tell (I've been doing searches here and there on and off since I saw this question a few hours ago) the closest we've gotten to 'simulations' on this is video-games, and to a degree movies, interestingly enough. I.e. entertainment media.

Games like Portal, System Shock (with the AI 'Shodan'), and others give interpretations of what AI systems could be capable of themselves. Mass Effect is more or less entirely based around existential concepts regarding extra-terrestrial, almost primordial AI beings that threaten the earth.

But there's even more to it than the whole 'evil robots taking over the world' aspect. There's the actual implementation of AI in video games, which is where much of this technology first makes contact with the general public.

We have facial-scanners that put you into NBA games, cities full of realistically reacting people (inFamous, Assassin's Creed), and games that learn how you play and adjust the game accordingly (Metal Gear Solid does some of that stuff, as well as being thematically AI-heavy).

Ultimately, we only get things like the iPhone or VR headsets or other major proof-of-concept material only so often, but games are much more frequently implementing the most recent AI advances.

Thus, even though many AI systems are being put into place in the general workforce (many Hospitals now turning to cloud and AI health services, as recent as this week), I don't think you can really go further than video games or movies for 'simulations' or extrapolations like the ones you seem to want.

Analyzing the response to AI developments in games might be the closest thing currently possible. In terms of economic models or anything of that sort, I can find naught.


How does Monte Carlo Exploring Starts work?

each episode you will calculate the return, you will then update the action value or $Q(s,a)$  as the average each episode. Using the blackjack example from open AI gym and using a discount factor of 1, you get the following

episode 1
[{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((22, 10, False),0)=-1$

$Q((17, 10, False),1)=-1$

$Q((12, 10, False),1)=-1$

episode 2
[{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((21, 10, False),0)=1$

$Q((17, 10, False),1)=0$

$Q((12, 10, False),1)=0$

For $Q((17, 10, False),1)$ and $Q((12, 10, False),1)$ is the average return
i.e -1 for the first episode and 1 for the second.


How do we know if GPT-2 is a better language model?

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.

I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:

Prompt:

  I want to generate beautiful music using machine learning. Do you
  think a computer will ever be able to compose something as novel as
  creations of Mozart, Bach, or Chopin?

Results (first four samples, not cherry picked):

  Yes, computers will probably be able to do it by 2020. I think the key
  for us will be to make music that is interesting and beautiful, so we
  will never really know. My hope would be that computers will become
  far more efficient over time – something that is already happening in
  a lot of ways like in big data – while still being able to do more
  things that they are trained on, like find patterns. I think that's
  where I'd want to use neural networks in the future.

  We don't know, but it will probably happen. That's an interesting
  question to ask. Of course we have AI, which can do interesting things
  without human input. It can recognize the world, find objects and
  images, it understands language, we can make games for it. But if you
  imagine an AI being able to write music, will it even be good enough?
  How much better than human players? How much better can someone else
  do? AI will never really be at its best because it will always have to
  come up with creative ways to use and play it. To ask a question like
  it is good enough is to ignore that it is human. You are only allowed
  to ask good questions

  The software will help, as it already can do. But it will also allow
  more human creators, which is what I think it should do more of to
  bring music to any genre. In terms of composing, it's a matter of
  creativity in humans in general. But how does it achieve that? Let's
  just say that there are fewer humans composing today than ever before.
  Do you remember the day when, by chance, a computer could write lyrics
  to Mozart's Sonata No. 4, or the day when a human wrote lyrics to
  Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's
  just not true anymore. As we move forward, we are not going to create
  music that human musicians cannot compose. What is your goal as a
  computer music creator?

  At the core of this idea is that music is a collection of sounds – it
  was the very first art form, it was the first music that humans
  evolved, a collection of sounds. Music is not an abstract thing of
  sound, but a collection of sounds. You can't write music the way we
  write and compose books and films, because we write and compose with
  sounds in mind. This is where machine learning comes in. You can read
  some books about how to get better music writers or how to improve the
  quality of their music, but you cannot make a difference with this
  kind of information at all. What does music teach you? To write and
  perform beautifully, to convey emotion accurately, and to create
  lasting and rich relationships with


Are there any computational models of mirror neurons?

This article gives a description of mirror neurons in terms of Hebbian learning, a mechanism that has been widely used in AI. I don't know whether the formulation given in the article has ever actually been implemented computationally.


There are different algorithms, each with their advantages and disadvantages.

Gazetteers: these have lists of the entities to be recognised, eg list of countries, cities, people, companies, whatever is required. They typically use a fuzzy matching algorithm to capture cases where the entity is not written in exactly the same way as in the list. For example, USA or U.S.A., United States, United States of America, US of A, etc. Advantage: generally good precision, ie can identify known entities. Disadvantage: can only find known entities
Contextual Clues: here you have patterns that you find in the text, eg [PERSON], chairman of [COMPANY]. In this case, sentences like Jeff Bezos, chairman of Amazon, will match, even if you have never come across either Bezos or Amazon. Advantage: can find entities you didn't know about. Disadvantages: could end up with false positives, might be quite labour-intensive to come up with patterns; patterns depend on the domain (newspapers vs textbooks vs novels etc_
Structural description: this is basically a 'grammar' describing what your entities look like, eg (in some kind of pseudo-regex): title? [first_name|initial] [middle_name|initial]? surname would match "Mr J. R. Ewing" or "Bob Smith". Similar descriptions could match typical company names; you'd probably still need lists of possible surnames or firstnames. Advantages: some flexibility, and potentially good precision. Disadvantages: patterns need to be developed and maintained.

Ideally you would want to combine all three approaches for a hybrid one to get the advantages of recognising unknown entities while keeping excess false positives in check.

There might also be other machine-learning approaches, but I'm not too familiar with those. The main problem is that they are hard to fine-tune or work out why they do what they do.

UPDATE: A good starting point would be to use a gazetteer-based approach to annotate some training data, and use that to identify contextual patterns. You can then use that data to train a machine learning approach (see OmG's answer on CRF) to broaden the approach; and then you add newly recognised entities to your list.

Ideally you would want to have a gazette as your main database to avoid false positives, and use machine-learning or contextual patterns to just capture previously unseen entities.


Given a dataset with no noisy examples, is the training error for the ID3 algorithm always 0?

Yes, if you can assume that your data is separable on the features given then ID3 will find a decision tree for it (Note: this will not necessarily be an optimal tree, or even a good tree). To understand why let's look at a proof.

Assume we have one feature left and some number of examples in a leaf that does not have separated data points then, either:

This feature perfectly splits the data into their desired categories. In this case ID3 will split over this feature and be done.
This feature does not split the data into their desired categories. In this case, we have at least 2 examples with the same feature value, but they do not share the same class. This means either these two examples have the same feature value for all features (a contradiction to our assumptions) or we have created a leaf with non-separable data points (which we prove is impossible later on).

Now for the inductive step.

Assume that ID3 used some number of features to somewhat separate our examples already. Thus, we have some number of features not used and some number of examples not separated on any current leaf, and one of the following is true:

All our leafs contain data points of only one class. In which case, ID3 is done and returns.
All leafs contain data points that are separable by the remaining features. Then ID3 splits over the local optimum and continues.
At least one leaf contains data points that are not separable by the remaining features of each own's branch. Then there exists two examples p and q such that p and q share all the same remaining feature values, but do not share the same class. Thus, one of the following is true:

There is a feature that we already used that differs between p and q.
This is a contradiction as we said ID3 already separated over this feature, and thus p and q must be on different branches.
p and q have the same feature value for all features previously used. But we already said p and q share all the same remaining feature values. Therefore, they share all feature values which is a contradiction to our assumptions.

Therefore, at no point in the creation of the decision tree is ID3 allowed to create a leaf that has data points that are of different classes, but can't be separated on remaining features. And thus, ID3 must create a solution that perfectly separates the data and has 0 training error.

Something to note however, is that allowing ID3 to grow the decision tree in this way will likely cause us to overfit the training set. To combat this we set a maximum depth that we allow ID3 to grow to.


What are the models that have the potential to replace neural networks in the near future?

This is going backwards, but it kind of follows the logic of the arguments.
In terms of efficiency, I can see a few major problems with classical neural networks.
Data collection and preprocessing overhead
Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a rule of thumb it is usually proportional to the number of weights. For some supervised learning tasks, there simply isn't enough high-quality labelled data. Collecting large amounts of specialised training data can take months or even years, and labelling can be cumbersome and unreliable. This can be partially mitigated by data augmentation, which means "synthesising" more examples from the ones you already have, but it is not a panacea.
Training time vs. energy tradeoff
The learning rate is usually pretty small, so the training progress is slow. A large model that could take weeks to train on a desktop CPU can be trained in, say, two hours by using a GPU cluster which consumes several kW of power. This is a fundamental tradeoff due to the nature of the training procedure. That said, GPUs are getting increasingly efficient - for example, the new nVidia Volta GPU architecture allows for 15.7 TFLOPs while consuming less than 300 W.
Non-transferrability
Right now, virtually every different problem requires a custom neural network to be designed, trained and deployed. While the solution often works, it is kind of locked into that problem. For example, AlphaGo is brilliant at Go, but it would be hopeless at driving a car or providing music recommendations - it was just not designed for such tasks. This overwhelming redundancy is a major drawback of neural networks in my view, and it is also a major impediment to the progress of neural network research in general. There is a whole research area called transfer learning which deals with finding ways of applying a network trained on one task to a different task. Often this relates to the fact that there might not be enough data to train a network from scratch on the second task, so being able to use a pre-trained model with some extra tuning is very appealing.

The first part of the question is more tricky. Leaving purely statistical models aside, I haven't seen any prominent approaches to machine learning that are radically different from neural networks. However, there are some interesting developments that are worth mentioning because they address some of the above inefficiencies.
Neuromorphic chips
A bit of background first.
Spiking neural networks have enormous potential in terms of computational power. In fact, it has been proven that they are strictly more powerful than classical neural networks with sigmoid activations.
Added to that, spiking neural networks have an intrinsic grasp of time - something that has been a major hurdle for classical networks since their inception. Not only that, but spiking networks are event-driven, which means that neurons operate only if there is an incoming signal. This is in contrast to classical networks, where each neuron is evaluated regardless of its input (again, this is just a consequence of the evaluation procedure usually being implemented as a multiplication of two dense matrices). So spiking networks employ a sparse encoding scheme, which means that only a small fraction of the neurons are active at any given time.
Now, the sparse spike-based encoding and event-driven operation are suitable for hardware-based implementations of spiking networks called neuromorphic chips. For example, IBM's TrueNorth chip can simulate 1 million neurons and 256 million connections while drawing only about 100 mW of power on average. This is orders of magnitude more efficient than the current nVidia GPUs. Neuromorphic chips may be the solution the training time / energy tradeoff I mentioned above.
Also, memristors are a relatively new but very promising development. Basically, a memristor is a fundamental circuit element very similar to a resistor but with variable resistance proportional to the total amount of current that has passed through it over its entire lifetime. Essentially, this means that it maintains a "memory" of the amount of current that has passed through it. One of the exciting potential applications of memristors is modelling synapses in hardware extremely efficiently.
Reinforcement learning and evolution
I think these are worth mentioning because they are promising candidates for addressing the problem of non-transferrability. These are not restricted to neural networks - being reward-driven, RL and evolution are theoretically applicable in a generic setting to any task where it is possible to define a reward or a goal for an agent to attain. This is not necessarily trivial to do, but it is much more generic than the usual error-driven approach, where the learning agent tries to minimise the difference between its output and a ground truth. The main point here is about transfer learning: ideally, applying a trained agent to a different task should be as simple as changing the goal or reward (they are not quite at that level yet, though...).


An introduction to the Haar features is provided in the youTube video.  The video indicates the VJ face detector leverages a selected combination of Haar features (convolutional kernels) to detect facial features (weak classifiers), such as the nose bridge.  The binary presence of the weak classifiers are summed to determine if the window contains a face.

The ability for a VJ algorithm to detect emotion would rely on the ability to assign a set of Haar features (kernels) to recognize features associated with a particular emotion label (surprise, anger, content, fear).

It is conceivable that the initial stage of an  emotion classifier could use a the VJ algorithm to identify a face for additional stages to classify emotion.


What does the formula $1-\sum_i(e_i-a_i)^2$ mean in this NEAT Python API?

$$1-\sum_i(e_i-a_i)^2$$
$\sum$ - there just means sum. It is the greek letter for S.  You can rewrite the above formula as
$$1 -[(e_1 - a_1)^2+(e_2-a_2)^2+(e_3-a_3)^2+\ldots ]$$
$\sum$ just helps us avoid writing dozens of $+$ signs. Read more here.
What they are doing here is taking the difference of expected value $e_1$ and the actual value $a_1$ for the 1st example, and so on. The difference can be positive ($e_1 > a_1$) or negative ($e_1 < a_1$), so usually we square the difference to make it positive number.
The rest is there in the docs. Try putting in concrete imagined values for $a_i$ and $e_i$.


How should I handle action selection in the terminal state when implementing SARSA?

The value $Q(s', ~\cdot~)$ should always be implemented to simply be equal to $0$ for any terminal state $s'$ (the dot instead of an action as second argument there indicates that what I just wrote should hold for any action, as long as $s'$ is terminal).

It is easier to understand why this should be the case by dissecting what the different terms in the update rule mean:

$$Q(s, a) \gets \color{red}{Q(s, a)} + \alpha \left[ \color{blue}{r + \gamma Q(s', a')} - \color{red}{Q(s, a)} \right]$$

In this update, the red term $\color{red}{Q(s, a)}$ (which appears twice) is our old estimate of the value $Q(s, a)$ of being in state $s$ and executing action $a$. The blue term $\color{blue}{r + \gamma Q(s', a')}$ is a different version of estimating the same quantity $Q(s, a)$. This second version is assumed to be slightly more accurate, because it is not "just" a prediction, but it's a combination of:

something that we really observed: $r$, plus
a prediction: $\gamma Q(s', a')$

Here, the $r$ component is the immediate reward that we observed after executing $a$ in $s$, and then $Q(s', a')$ is everything we expect to still be collecting afterwards (i.e., after executing $a$ in $s$ and transitioning to $s'$).

Now, suppose that $s'$ is a terminal state, what rewards do we still expect to be collecting in the future within that same episode? Since $s'$ is terminal, and the episode has ended, there can only be one correct answer; we expect to collect exactly $0$ rewards in the future.


Why RNNs often use just one hidden layer?

Definitely you can have multiple hidden layers in RNN. One the most common approaches to determine the hidden units is to start with a very small network (one hidden unit) and apply the K-fold cross validation ( k over 30 will give very good accuracy) and estimate the average prediction risk. Then you will have to repeat the procedure for increasing growing networks, for example for 1 to 10 hidden units or more if needed.

However, in my experience, if you are interested to get best possible accuracy, you should start with small number of hidden layers and more simple structure, and if you are not satisfied with the corresponding accuracy, then we should go on increasing the learning rate by fixed but small small steps and each time start training fresh.


Disentangled VAE doesn't reconstruct accurate grids

First of all , this is a deepmind's paper , so the computational level they operate on is much higher.
Can you share your code files , so that I can see how you interpreted miyosuda's code.
The disentangled representation has to be scaled down so that the model trains , and gets useful results in atleast a few hours.

4GB is too less RAM for training any kind of model. The best temporary solution I can suggest is to perform the python script of distangled VAE on Colab Colab Notebooks

The more permanent solution that can making your model training much easier given your system constraints , is to use GCP or Azure.

How do you plan to customize the architecture once you finish implementing distangled VAE.


What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)?

I would say any normalization such as min-max or standard deviation is fine as far as the scaling factor is provided as a feature, since time-series of different scale might behave differently.


Artificial neural network to solve Crossword puzzles, this model might help you to get a start on evolving your own.

https://github.com/fh295/DefGen2

A brute force of word combinations would solve the matrix, but might not get the context right and might be the simplest model if you are not in for a neural network.


Recognising Noise in Simple Classification

The network classifies perfectly with validation data, however I'd like it to be able to take in random noise that looks nothing like the shapes and not be able to classify it confidently.

You need to train the network on noise well if you want it to be able to recognize it.

  or is it just best to come up with such a well-trained network that you can use some sort of confidence threshold?

You can use spot to get a confidence score for clarification. But training the network on a noise class will work way better

See:

  Gal: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning


What are the real-life applications of transfer learning?

One application I know of being used in industry is of image classification, by only training the last layer of one of the inception models released by Google,  with the desired number of classes. I can't provide specific details.

Transfer learning is useful when:

You do not have the resources (time, processing power, etc.) to train a DL  model from scratch.
You can compromise a bit on accuracy.


What are the steps to follow to learn artificial intelligence?

A really good introduction is the Berkeley CS188 class videos and projects.  You can find those materials at http://ai.berkeley.edu/home.html

You probably also want to get ahold of a copy of Artificial Intelligence: A Modern Approach by Norvig and Russell.

For more on the "machine learning" aspects of AI, including an introduction to Neural Networks, take the Andrew Ng "Machine Learning" class on Coursera.

Another book I would recommend is Introduction to Artificial Intelligence by Philip C. Jackson. It's older, which is exactly what makes it valuable.  It's a good overview of techniques and ideas that aren't "en vogue" right now, but which may still be useful to you.


Is there a way to define the boundaries of the optimal size of a training set?

For a finite value to be 'optimal,' typically you need some benefit from more paired up with some cost for more, and eventually the lines cross because the benefit decreases and the cost increases.

Most models will have a reduction in error with more training data, that asymptotically approaches the best the model can do. See this image (from here) as an example:

The costs of training data are also somewhat obvious; data is costly to obtain, to store, and to move. (Assuming model complexity stays constant, the actual cost of storing, moving, and using the model remains the same, since the weights in the model are just being tuned.)

So at some point the slope of the error-reduction curve becomes horizontal enough that more data points are costlier than they're worth, and that's the optimal amount of training data.


How to compute the number of centroids for K-means clustering algorithm given minimal distance?

Yes, the silhouette method (which is implemented in sklearn as silhouette_score) is commonly used to assess the quality of clusters produced by any clustering algorithm (including $k$-means or any hierarchical clustering algorithm). Roughly, you can compute the silhouette value for different $k$, then you would pick the $k$ with the highest silhouette value.


How to know which kind of adverb in NLP Parts of Speech (POS) tagging?

Generally POS taggers only use broad categories, especially if they work statistically: the more fine-grained the tagset is, the more training data you need. And unless there are differences in the distribution, the accuracy of the tagger would not be improved by adding more categories, eg

  I ran [there/fast/often/early].

all work -- so nothing is gained by having subcategories of adverbs.

I suggest the easiest solution for this would be to have a list of adverbs by category, and simply look them up after the tagging. Just pick all words tagged as adverbs and check if they are in your list. Unless there are ambiguities (ie some adverbs can belong to multiple categories) it should be very straight forward, and would also work with any tagger/tagset that marks adverbials.

You might have to consult a grammar book to get a list of adverbial classes, but that again should be easy to do.


Is pre-processing used in deep learning?

Yes, sure, data pre-processing is also done in deep learning. For example, we often normalize (or scale) the inputs to neural networks. If the inputs are images, we often resize them so that they all have the same dimensions. Of course, the pre-processing step that you apply depends on your data, neural network, and task.
Here or here are two examples of implementations that perform a pre-processing step (normalization in the second case). You can find more explanations and examples here and probably here too.


How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
A common example for continuous spaces is the reparameterization trick, where your policy outputs $\mu, \sigma = \pi(s)$ and the action is $a \sim \mathcal{N}(\mu, \sigma)$.


How does text classification reduce manpower costs?

There are several advantages:

Some text classification systems are much more accurate than 50%. For example, most spam classification systems are 99.9% accurate, or more. There will be little value to having employees review these labels.
Many text classification systems can output a confidence as well as a label. You can selectively have employees review only the examples the model is not confident about. Often these will be small in number.
You can usually test a text classification model by having it classify some unseen data, and then asking people to check the work. If you do this for a small number of examples, you can make sure the system is working. You can then confidently use the system on a much large set of unlabeled examples, and be reasonably sure about how accurate it is.
For text, it is also important to measure how much different people agree on the ratings. You are unlikely to do better than this, because this gives you a notion of the subjectivity of the specific problem you are working on. If people disagree 50% of the time anyway, maybe you can accept a 50% failure rate from the automated system, and not bother checking its work.


Is the minimum and maximum of a set of admissible and consistent heuristics also consistent and admissible?

Yes, in both cases. Below I give two very simple proofs that directly follow from the definitions of admissible and consistent heuristics. However, in a nutshell, the idea of the proofs is that $h_{\max}(n)$ and $h_{\min}(n)$ are, by definition (of $h_{\max}$ and $h_{\min}$), equal to one of the given admissible (or consistent) heuristics, for all nodes $n$, so $h_{\max}(n)$ and $h_{\min}(n)$ are consequently admissible (or consistent).

Definitions

Consider the graph $G=(V, E, \mathcal{G})$ representing the search space, where $V$, $E$ and $\mathcal{G} \subseteq V$ are respectively the set of nodes, edges and goal nodes, and the function $w\colon E \times E \rightarrow \mathbb{R}$, which gives you the cost of each edge $e = (u, v) \in E$, where $u, v \in V$, that is, $w(e) = w(u, v) \in \mathbb{R}$ is the cost of the edge $e$.

A heuristic $h$ is admissible if $$h(n) \leq h^*(n), \forall n \in V,$$
where $h^*(n)$ is the optimal cost to reach a goal from node $n$ (that is, it is the optimal heuristic).

On the other hand, a heuristic $h$ is consistent if

\begin{align}
h(n) &\leq w(n, s) + h(s), \forall n \in V \setminus \mathcal{G}, \text{ and} \\
h(n) &= 0, \forall n \in \mathcal{G},
\end{align}
where $s$ is a successor of $n$.

Theorem 1

Given a set of admissible heuristics $H = \{ h_1, \dots, h_N \}$, then, for every $n \in V$, the heuristics $h_{\max}(n) = \max(h_1(n), \dots, h_N(n))$ and $h_{\min}(n) = \min(h_1(n), \dots, h_N(n))$ are also admissible.

Proof

Given that, $h_i(n) \leq h^*(n), \forall n \in V$ and $\forall i \in \{ 1, \dots N \}$, then $h_{\max}(n) = h_j(n) \leq h^*(n)$ (for some $j \in \{ 1, \dots N \}$)  and $h_{\min}(n) = h_k(n) \leq h^*(n)$ (for some $k \in \{ 1, \dots N \}$), so $h_{\max}$ and $h_{\min}$ are also admissible.

$\tag*{$\blacksquare$}$

Theorem 2

Given a set of consistent heuristics $H = \{ h_1, \dots, h_N \}$, then, for every $n \in V$, the heuristics $h_{\max}(n) = \max(h_1(n), \dots, h_N(n))$ and $h_{\min}(n) = \min(h_1(n), \dots, h_N(n))$ are also consistent.

Proof

Given that, for every $i \in \{ 1, \dots, N \}$,

\begin{align}
\begin{cases}
h_i(n) \leq w(n, s) + h_i(s), & \text{ if }  n \in V \setminus \mathcal{G}\\
h_i(n) = 0, & \text{ if }  n \in \mathcal{G}
\end{cases}
\end{align}

then, if $n \in V \setminus \mathcal{G}$, $h_{\max}(n) = h_j(n) \leq w(n, s) + h_j(s)$, for some $j \in \{1, \dots, N \}$, and, similarly, $h_{\min}(n) = h_k(n) \leq w(n, s) + h_{k}(s)$, for some $k \in \{1, \dots, N \}$. Similarly, if $n \in \mathcal{G}$, $h_{\max}(n) = h_{\min}(n) = h_i(n) = 0, \forall i \in \{1, \dots, N \} $. Thus, $h_{\max}$ and $h_{\min}$ are also consistent.

$\tag*{$\blacksquare$}$

Notes

$h_{\max}$ and $h_{\min}$ have been defined in the only possible reasonable way, because, given two (admissible or consistent) heuristics, one may not always dominate the other (or vice-versa), even if both are admissible and consistent.


How to calculate the distance between the camera and an object using Computer Vision?

In general, calculation of distance between camera and object is impossible if you don't have further scene dependent information.
To my knowledge you have 3 options:
Stereo Vision
If you have 2 cameras looking at the same scene from a different point of view you can calculate the distance with classical Computer Vision algorithms. This is called stereo vision, or also multiview geometry.
Stereo Vision is the reason why humans can infer the distance to objects around them (because we have 2 eyes).
Structure from Motion
You move your camera and therefore change your viewpoint and can essentially do stereo mapping over time. Structure from Motion
Scene Understanding
Why is it then still possible for a one-eyed person to infer depth to some extent? Because humans have lots of scene dependent understanding. If you see a rubber duck that takes half of your field of view, you know it's pretty close because you know a rubber duck is not big. If you don't know the size of rubber ducks it is impossible to know whether you see a big rubber duck that is far away or a small rubber duck that is really close.
This is where Deep Learning based models come into play. A recent overview over monocular
depth estimation can be found in Zhao2020


How close we are to replacing guide dogs with AI?

Chieko Asakawa (wiki, TED, IBM) is a major researcher in this area, and the linked TED talk is probably a good introduction to the state of the art as of 2015. Here's a link to a 2016 paper on a smartphone navigation system.

Guide animals perform manipulation tasks as well as identification tasks, and so it's not clear if those could be replaced well at all. (A smartphone that reads a label is a great help, but a dog that knows which bottle to grab and deliver to you is probably a much better help.)


How long it takes to train face recognition deep neural network? (rough estimation)

Training time depends on a lot of parameters. Some of them are:

Size of each image (resolution)
Color/Monochrome image (color image has 3 times data if you consider RGB image)
Like you mentioned on the type of DNN.
No. of layers of DNN.
No. of neurons in each layer.
Total no. of images in the dataset. (2.6 million here)
GPU you are using (you didn't mention which GPU you are using. There are GPUs with wide range of capabilities, to predict the time, you need to know the exact specs of GPU).
Training time also depends on RAM on your machine and also how fast host PC can transfer data to GPU for processing.
Since you mentioned face recognition, i am assuming you are using CNN, but if you again use fully connected network, training time will change and will obviously increase manifold.

Your task of classification and your database is mostly similar to ILSVRC that uses imagenet database.

Making some reasonable assumptions for the parameters you didn't mention, i feel your task is similar to ILSVRC and i am predicting the training time will be a few days.

Below are the links which mention time of training for ILSVRC.

https://mxnet-tqchen.readthedocs.io/en/latest/tutorials/imagenet_full.html

Below are the details of imagenet database for your comparison

https://en.wikipedia.org/wiki/ImageNet


Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function?

As you stated, it's popular to have some form of a rectified linear unit (ReLU) activation in hidden layers and the output layer is often a softmax or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution.
You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.


Why should we study causation in artificial intelligence?

There are two intermixed elements in your question:

Why have people already studied causality in AI?
Why would people like to continue studying causality in AI?

tl;dr: AI systems can't function in the real world without some way of understanding uncertainty. The best ways we have to understand uncertainty don't work well unless we also understand causal structure. Getting most of the way towards this understanding was a huge accomplishment for AI research, but there's still a big problem waiting to be solved here.

For part 1, let's think about where research into causality comes from. Pearl originally set out to solve a major problem plaguing AI researchers in the 1980s: reasoning under uncertainty. This area is so central to AI systems that it has its own, large, conference: UAI. The need for reasoning under uncertainty arose because the real world is filled with uncertainty. Consider even a simple problem, like have a robot navigate an empty room. Even if the robot knows its initial position exactly, it is not likely to know its true position for long. The robots' wheels might have a certain specification, but wheels stick and slip (friction), and they do so unpredictably (e.g. maybe one part of the floor didn't get polished as much as the others). The robot might have sensors, but sensors are imperfect. Does a light sensor value of 0.25 mean a wall is near, or just that the sun is coming through the window at an unfortunate angle? Or maybe just that the shade of paint is slightly different there? Does our acoustic sensor reading of 1.35 mean that we're 1.35 meters from the wall, or did the signal get reflected, and return to us by a different path?

One of Pearl's major contributions was showing how we can use the rules of probability to reason about these events correctly. Although others had done the basics of this long ago, Pearl proposed the idea of Bayesian Networks. Thrun and many others used these techniques to solve problems like the robot navigation task discussed above.

A problem with Bayesian Networks is that, if built incorrectly, they lose much of their efficiency benefits. The correct construction is usually the one that best captures known causal relations between factors. Further, algorithms for inference in a Bayesian Network can not easily answer questions about counterfactuals. They cannot tell our robot what would happen in a world where certain actions were taken. They can only say whether certain behaviors are likely co-occur.

These outstanding issues led Pearl to work on causality and causal networks. It is important for our systems to be able to answer counterfactual questions, because often the answers to those questions determine how the system aught to act. Pearl's do-calculus (nicely summarized here, and in Pearl's The Book of Why) has solved this problem, and shown how causes can often be inferred from observation alone. It's super exciting!

Hopefully you're now convinced that Causality was worth studying within AI. You might now wonder why it's still worth studying.

The main open problem right now is that to do causal reasoning, we need to already have a network that describes causal relations. These networks are hard to construct by hand, so we'd like to be able to learn the structures of these networks. Unfortunately, this looks like a very hard problem, both to solve exactly (it's #P-hard if I recall correctly), and to approximate. There are no known general algorithms, at least as of 2018. Getting this would be a major breakthrough in AI research, and more generally, in statistics, philosophy of science, and epistemology. Even getting something that mostly worked, most of the time, would be huge. Peter Van Beek has been doing some work on this recently, which could be a good starting place if you want to read more.


Why do all states appear identical under the function approximation in the Short Corridor task?

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative.
This is why a stochastic policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a deterministic policy that is guaranteed to be optimal.


How to incentivise snake to go straight to apple?

Your network finds the infinite loop and notices that this has the best reward (0). This is probably because it hasn't found a path to eating the apple (through exploration).

Reducing the discount factor will only make the long term rewards less valuable. So it will learn eating the apple even slower.

I don't know what you are using as inputs for your network, but maybe changing your reward system could help. For example, you could give your network a reward if it advances in the direction of the apple. This way, your network will be encouraged to find the apple more than it is now.


Using CNN to identify buildings from aerial images

You need to know the basic mechanics of CNN , you can't simply finetune a pretrained net (VGGNET) everytime. VGGnet is trained on imagenet dataset so most of the features in it are not so relevant for your task (recognising buildings from aerial view). so it is much better to train a cnn from scratch. You can't simply say that it fails to classify residential , non-residential . It is best to find what could be done by trying,

My issue is that non-residential buildings are mostly larger than residential houses, now I want to use this property to distinguish between residential and non residential.

Most of the times it is better to observe the features recognised by the net and then come with an approach , rather than enforcing your views.

You may find this helpful https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Thesis.pdf


Why does the accuracy drop while the loss decrease, as the number of epochs increases?

Decrease of loss does not essentially lead to increase of accuracy (most of the time it happens but sometime it may not happen). To know why, you can have a look at this question. The network cares about decreasing the loss and it does not care about the accuracy at all. So it's no surprise to see what you presented.
Additional note: If you use batch approaches to teach your network, or if you choose a big step size you may also see that the loss increases sometimes but in the case of batching, the hope is that its trend is to be decreased.


Training an RNN to answer simple quesitons

I have thought of various ways of doing this. Such as, as well as having 26 outputs for letters of the alphabet have about 20 more for "repeat the character that is 14 characters to the left" and so on

Creating a system of rules like your example above is the very opposite of training an RNN to perform this task.

If you would like to train an RNN to answer simple questions you would not need to come up with ingenious rules, but with enough training data of the form

question -> answer

Then you could use one of many popular sequence-to-sequence NLP tools to try and learn this behaviour, effectively treating the problem as if it were machine translation between languages.

More broadly, yes, question answering has been done before, it is very popular in fact. It is an active subfield of NLP research and many methods have been developed, some of them involving RNN networks.


Approach 1: One way would be to switch from nodes representing programs to nodes representing sets of programs. The root node would represent the empty set ${}$, to which expansions could be applied only if they can generate a program from scratch. The first such expansion would produce some program $p$, so the root would now have child ${p}$. The second expansion would produce $p'$, so the root would now also have child ${p'}$ which would itself also have the child ${p,p'}.

One downside to this approach is that, even assuming reasonable restrictions (e.g. moves can use at most 2 programs, pairs cannot have identical elements, element order doesn't matter), the branching factor will grow combinatorially, because each new program can be combined with all (or most) of the previously generated programs.

Approach 2: Another way would be to switch from nodes representing programs to nodes representing meta-programs describing an initial program and a series of revisions to that program. For example, MCTS might search for expressions in something like this grammar:

Program  -> Empty
          | Program > OneProgramRevision
          | Program Program >> TwoProgramRevision
OneProgramRevision -> AddDatumAsException
                    | SampleStatement
                    | RegenerateSubtree
                    | Delete
                    | ...
TwoProgramRevision -> Concatenate
                    | Crossover
                    | ...

This is just a cartoon: one would need to add details parameterizing each of the various revisions.


I spent some time on the Future of Work in UK government. If you want to look at the impact of AI on work you need to have good definitions of soft skills that you can measure and you can track progress against.

So the definitions sometimes need to change and decomposed. Take "Coaching" from that list: there are now bots (such as Wysa.ai) that coach you. You can measure coaching as a skill by the effectiveness of the participants achieving goals.

Things really get murky when you talk about "Creativity" -- I wrote a piece on AI impact of creativity in a similar vein.


Using three image datasets with different image sizes to train a CNN

Resizing the images will work.But if you significantly reduce the image dimensions information IS lost. Here is a simple example. Suppose you have a 500 X 500 image. That has 250,000 pixels. Assume in these images the object of interest( lets say a bird in a forest) only occupies 10% of the pixels in the image(25,000 pixels). Now assume you reduce the image to 100 X 100 and thus have 10,000 pixels. Your object of interest (the bird) now only occupies 1000 pixels. These are the pixels the neural net will learn from.
Now a better way to do this is to first crop your original 500 X 500 images to maximize the percentage of bird pixels in the cropped image. For example assume the resulting cropped image comes out to be say a 200 X 200 image but in that image the subject of interest(the bird) occupies 50% of the pixels (20,000 pixels). Now if you reduce the cropped image down to a 100 X 100 the object of interest will occupy(5000 pixels).
Cropping images is of course a hassle. For some types like images for example images of people, and you are just interested in the faces there are routines available that will automatically crop the image to just the face. In general you will have to crop the images yourself.
If you want to build high accuracy classifiers you have to start with a "robust" data set.
The larger the percentage the object of interest takes up in the images the better your classifier will perform.


Does anybody know what would happen if I changed input shape of pytorch models?

In all pre-trained models, the input image has to be the same shape; the transform object resizes the image when you add it as a parameter to your dataset object
transform = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor()])
dataset = datasets.ImageNet(".", split="train", transform=transform)

T.Resize(256) changes the image shape then T.CenterCrop(224) takes a random crop to help with overfitting in the training data.
The mean and std are applied to each image before they are used as an input to the model. They were determined when the model was trained, I assume they were hyperparamters.


Equation not satisfied in Policy Iteration Algorithm

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not related to the policy update step (which uses the notation $\pi'$ and does not mention $k$).

Policy improvement consists of multiple sweeps through states to fully evaluate the current policy and estimate the value function for it. After that, it updates the policy in a separate policy improvement step. There are two loops - an inner loop indexed by $k$ in the equations and diagram, plus an outer loop which is not given an index notation.

The diagram is not showing incremental $\pi'$ policies from outer loops over policy iteration. Instead it is showing "Greedy Policy w.r.t. $v_k$" steps in the inner loop - you can think of that as the policy $\pi'$ that you would get inside the first outer loop, if you terminated the policy evaluation stage after that iteration $k$ of the inner loop.

The diagram only shows behaviour of policy iteration for a single outer loop. It demonstrates at least two interesting things:

In the case of this very simple environment, if you ran a single outer loop with long enough policy evaluation stage ($k \ge 3$) you would find the optimal policy.
Even before the value function estimate is close to convergence (with a high $k$), the policy that could be derived from the new estimates could be used to improve the policy. That leads to the idea of the value iteration method.


In this implementation of the Information Set Monte Carlo Tree Search, why can't the players see the cards of each other?

All the methods in the GameState class that is used to represent state, are stubs, and without these, the MCTS algorithm won't do anything at all. In particular, the DoMove method just changes who's turn it is, without actually taking any action.

Probably the reason the players can't see each other's cards is that this is not a completed implementation. Someone is either still working on this, or gave up half way.


How can I avoid overfitting when doing parameter tuning?

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, because to "get lucky" and overfit, a parameter combination will need to work very well on many variations of the problem, which is less likely than working well on just one set of data.


When would bias regularisation and activation regularisation be necessary?

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large.  I have primarily used kernel regularizers. First I try to control over fitting using dropout layers. If that does not do the job or leads to poor training accuracy I try the Kernel regularizer. I usually stop at that point. I think activity regularization would be my next option to prevent outputs from becoming to large. I suspect weight regularization effectively can pretty much achieve the same result.


How is the variational lower bound for hard attention derived in Show, Attend and Tell

This is Jensen's inequality at work.
First of all, note that the first line can be rewritten as an expectation
$$\sum_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})]$$
Then Jensen's inequality gives (Note that a log function is a concave function so gives the opposite inequality to what is normally given when explaining Jensen's inequality with respect to convex functions):
$$\mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})] \leq \log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] $$
and then finally you can rewrite the Expectation as a summation.
$$\log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] = \log \sum_{s} p(s \mid \mathbf{a}) p(\mathbf{y} \mid s, \mathbf{a})$$


What are the ethical and legal issues of self driving cars being released in the UK?

LEGAL
Legislative:
http://cyberlaw.stanford.edu/wiki/index.php/Automated_Driving:_Legislative_and_Regulatory_Action
http://www.ncsl.org/research/transportation/autonomous-vehicles-self-driving-vehicles-enacted-legislation.aspx
http://spectrum.ieee.org/transportation/advanced-cars/selfdriving-cars-will-be-ready-before-our-laws-are
Liability:
https://www.usatoday.com/story/money/cars/2017/04/03/tesla-mishap-raises-issues-self-driving-liability/99880620/
https://www.nytimes.com/2014/05/14/upshot/when-driverless-cars-break-the-law.html?_r=0
Regulation:
https://www.theverge.com/2016/9/19/12981448/self-driving-car-guidelines-obama-foxx-dot-nhtsa
Law Review
https://www.americanbar.org/publications/youraba/2015/october-2015/are-we-there-yet--the-legal-aspects-of-driverless-cars.html
ETHICAL
https://www.technologyreview.com/s/542626/why-self-driving-cars-must-be-programmed-to-kill/
https://www.scientificamerican.com/article/driverless-cars-will-face-moral-dilemmas/
http://news.stanford.edu/2017/05/22/stanford-scholars-researchers-discuss-key-ethical-questions-self-driving-cars-present/

This should be enough to get you started.  (In terms of how it may apply in the UK, I leave that for you to determine;)


How can fuzzy logic be used in creating AI?

A classical example of fuzzy logic in an AI is the expert system Mycin.

Fuzzy logic can be used to deal with probabilities and uncertainties.

If one looks at, for example, predicate logic, then every statement is either true or false. In reality, we don't have this mathematical certainty.

For example, let's say a physician (or expert system) sees a symptom that can be attributed to a few different diseases (say A, B and C). The physician will now attribute a higher likelihood to the possibility of the patient having any of these three diseases. There is no definite true or false statement, but there is a change of weights. This can be reflected in fuzzy logic, but not so easily in symbolic logic.


Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?

The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not affect whether the state has the Markov property. All that matters is that knowing the current state is enough to be determine the next state and reward in principle. You do not need to explicitly know the state transition model, or the reward function.
An example of a non-Markov state for CartPole would be if one of the features was missing - e.g. the current position of the cart, or the angular velocity of the pole. It is still possible to have agents attempt to solve CartPole with such missing data, but the state would no longer be Markov, and it would be a harder challenge.

For me, there is something weird in updating a Q value based on the max Q for a state and a reward value that was not from the action taken? How does this make learning better and makes you learn the optimal policy?

To recap, you are referring to this update equation, or perhaps some variation of it:
$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha(R_{t+1} + \gamma\text{max}_{a'}Q(S_{t+1},a') - Q(S_t, A_t))$$
Another correction here, the reward value is related to the action taken and the experience that happened. This is important - the values of $R_{t+1}$ and $S_{t+1}$ are from the experience that was observed after taking action $A_t$, they do not come from anywhere else. In fact this is the only data from experience that is inserted into the update, everything else is based on action value estimates in a somewhat self-referential way (a.k.a. "bootstrapping").
How does this converge to an optimal policy? It follows a process known as generalised policy iteration (GPI), which works as follows:

The target policy is to always take the best action according to your current estimates

In off policy methods only (i.e. does not apply to all GPI), there is a conversion needed from experience in a behaviour policy to value estimates for the target policy. In single-step Q learning, this is where the $\text{max}_{a'}Q(S_{t+1},a')$ comes from. It is calculating the future value from following the target policy*. No interim values are needed, because you are estimating a new value for $Q(S_t,A_t)$, there are no other time steps where you need to care about difference between behaviour and target policy (there is just "now" and "all the future, which is bootstrapped")

Once you update the action value estimates in Q table, that automatically updates the target policy because it always takes the best estimated action.

Each update improves your value estimates, due to injecting some real experienced data in the form of $R_{t+1}$ and $S_{t+1}$.

Each time the estimates improve, the target policy can also improve and become closer to the optimal policy.

When the target policy changes, this makes the estimates less accurate, because they were made for the previous policy. However, the next time the same states and actions are visited they will be updated based on the new improved policy.

With a stochastic environment, it is possible for estimates to get worse due good or bad luck for the agent when it tries different actions. However, over time, the law of large numbers will win, and the value estimates will converge to close to their true values for the current policy, which will make policy improvement inevitable if it is possible (caveat, it is only close to inevitable for the tabular Q-learning method which can be proven to converge eventually on an optimal policy).
There is proof that always taking the maximising action is a strict improvement to a policy when estimates are accurate. It is called the Policy Improvement Theorem.

* To be clear, there is nothing particularly clever about taking the max here. It is literally what the target policy has been chosen to do, so the term $\text{max}_{a'}Q(S_{t+1},a')$ is just the same as $V_{\pi}(S_{t+1})$ - the value of being in state $S_{t+1}$ for the current target policy $\pi$.


How to plot Loss Landscape with more than 2 weights in the network

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.

And also, the 3D space can't be used to plot more than 3 dimensions.

However, with a deep network with lots of weights, these can be plotted:

Loss value against any pair of 2 weights
Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):


How could I convolve a 4D image and a 4D filter with stride?

I think this should work for you:
scipy.signal.correlate | SciPy
I used it myself while I was writing a CNN in numpy.


What are some of the possible future applications of intelligent agents?

At the moment, what I can think of are the following applications, but there are potentially a lot more.

Decision Maker: If you have any problem making a decision, intelligent agents can be used to weight evidence and give you statistics to rule out bad decisions.
Online Teacher: In the far future, intelligent agents may acquire human-like skills, they maybe used to teach different students (from different backgrounds) at once, but the advantage here is that the intelligent agent can evaluate every student's level (skills, personality, intelligence, etc), and use this data to decide how to convey information to him (differently). It would be like a virtual world were children receive information in parallel, while taking different data about them into consideration: They can be used to better education.
Social Media: I think intelligent agents will be integrated into social media, to analyze messages and comments, and warn users that some friends will dislike this or that or find it offensive, to give him time for making a decision about what to say. This would help limit bullying on social media.
Language Learning: in the future (like in 10 - 20 years), intelligent agents will have the ability to generate human-level language, that would make it possible for them to generate infinite sentences and structures just like humans can do, and again they can evaluate user's level to generate listening and reading material. For a human, it would be just like talking to a native speaker of that language (or even better, since this "native speaker" would know everything about your level).


When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s_t, a_t, r_t, s_{t+1})$ and then, for every visit MC, you update
$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (G_t - Q(s_t, a_t))\;;$$
where $G_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$, for each $t$ in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.


Is it possible to use AI for detecting the volume of a cup

This could be possible, providing you have the right dataset to train it on.

The volume of a cup consist of width, height and depth. You can probably detect all three of those given the bounding box or the pixels of the cup. However detecting the dimensions of an object require a reference object, like a penny or your finger and you have to specify the exact dimension of that. If that method fits your problem which I assume not, this is a good resource to look through: Open CV measuring dimensions

However to do the task, you need a way to measure the volume without the reference object. To fo this you need a deep learning based approach. Depth detection using fully residual convolutional neural network maybe a good start for your project as you are doing something similar. You may use one of the pretrained model and apply transfer learning on it with the image of the cup and the surroundings, and get the output and feed it through another feed forward neural network to predict the volume.


The basic idea behind mini-batch training is rooted in the exploration / exploitation tradeoff in local search and optimization algorithms.

You can view training of an ANN as a local search through the space of possible parameters. The most common search method is to move all the parameters in the direction that reduces error the most (gradient decent).

However, ANN parameter spaces do not usually have a smooth topology. There are many shallow local optima. Following the global gradient will usually cause the search to become trapped in one of these optima, preventing convergence to a good solution.

Stochastic gradient decent solves this problem in much the same way as older algorithms like simulated annealing: you can escape from a shallow local optima because you will eventually (with high probability) pick a sequence of updates based on a single point that "bubbles" you out. The problem is that you'll also tend to waste a lot of time moving in wrong directions.

Mini-batch training sits between these two extremes. Basically you average the gradient across enough examples that you still have some global error signal, but not so many that you'll get trapped in a shallow local optima for long.

Recent research by Masters and Luschi suggests that in fact, most of the time you'd want to use smaller batch sizes than what's being done now. If you set the learning rate carefully enough, you can use a big batch size to complete training faster, but the difficulty of picking the correct learning rate increases with the size of the batch.


How to keep track of the subject/entity in a sentence?

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-established solution.
You mentioned Hugging Face in your question. The neuralcoref module in spaCy is itself from Hugging Face (note the reflexive anaphor used for emphasis in this sentence). If you're not a spaCy kind of person, then there's also Stanford's CoreNLP in Java that has coreference resolution. A Python wrapper is also available.
I also wanted to address a couple other topics you mentioned. You are right in that they're all somewhat connected. But you need to scope down your goal/research question because what you're aiming to achieve is too difficult for a first task. Named entity recognition, constituency parsing, and sentiment analysis. Pick just one to focus on.


From nvidia www (https://developer.nvidia.com/discover/lstm):

Accelerating Long Short-Term Memory using GPUs

The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared to CPU implementations. cuDNN is a GPU-accelerated deep neural network library that supports training of LSTM recurrent neural networks for sequence learning. TensorRT is a deep learning model optimizer and runtime that supports inference of LSTM recurrent neural networks on GPUs. Both cuDNN and TensorRT are part of the NVIDIA Deep Learning SDK.