How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?

Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value over distribution $X$.

No, for $\mathbb{E}[X|Y]$ we take expectation of $X$ with respect to the conditional distribution $X|Y$, i.e.

$$\mathbb{E}[X|Y] = \int_\mathbb{R} x p(x|y)dx\;;$$

where $p(x|y)$ is the density function of the conditional distribution. If your random variables are discrete then replace the integral with a summation. Also note that $\mathbb{E}[X|Y]$ is still a random variable in $Y$.

  How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi '}$ and the $A_t = \pi '(s)$ term disappears?

This is because in this instance $\pi '(s)$ is a  deterministic policy, i.e. in state $s$ the policy will take action $b$ with probability 1 and all other actions with probability 0. NB: this is the convention used in Sutton and Barto to denote a deterministic policy.

Without loss of generality, assume that $\pi'(s) = b$. The implication of this is that in the first expectation we have
$$\mathbb{E}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s, A_t = \pi'(s) = b] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;,$$
and in the second expectation we have
$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_a\pi'(a|s)\sum_{s',r}p(s',r|s,a)(r + \gamma v(s'))\;;$$
However, we know that $\pi'(a|s) = 0 \; \forall a \neq b$, so this sum over $a$ would equal 0 for all $a$ except when $a=b$, in which case we know that $\pi'(b|s) = 1$, and so the expectation becomes

$$\mathbb{E}_{\pi'}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s] = \sum_{s',r}p(s',r|s,a=b)(r + \gamma v(s'))\;;$$

and so we have equality of the two expectations.


Will BERT embedding be always same for a given document when used as a feature extractor

BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.


What is the general procedure to create an AI system that can detect fire in images?

Many AI libraries allow the ability to feed them an Image, but others have to be modified to allow this.

Disregarding the information above, as it is just a notification, you can do this in many ways. Depending on the library you use, different methods give varying accuracy results.

The real questions are:

How complex do you want the NN to be?
Is this going to be commercial, or private?

For any of the answers above, you will have to find the best method that works for your needs. The more this matters to the world, the more complex it will have to be.


How can I formulate the k-knights problem as a constraint satisfaction problem?

There are many possible ways to encode this problem, and some will be more advantageous than others

An encoding that seems like a reasonable starting point to me is:

Variables: Let $S$ be a set of $k$ variables representing the coordinates of knights on the chess board.
Domain: The domain of each variable is initially all vectors in $[n]^2$. Denote the components of these vectors with $[r, c]$.
Constraints:

Let A be the set of 8 attacking moves: $A=\{[2,1],[2,-1],[-2,1],[-2,-1],[1,2],[1,-2],[-1,2],[-1,-2]\}$
$\forall_{i < j} S_i \neq S_j$ (no knights on the same squares)
$\forall_{i < j} [r_i, c_i] \notin \{[r_j + x, c_j + y] | x,y \in A\}$ (no knight attacks another knight)

That should be it. Note the use of $\forall_{i<j}$ in the constraints is to reduce the total number of constraints by half, exploiting the symmetry that knight $i$ can attack knight $j$ iff knight $j$ can attack knight $i$. You could also use $\forall_{i\neq j}$, but it would increase your constraint count to no gain.


Are AI algorithms capable of self-repair?

Yes, this was an active area of research in a number of different AI fields.

Probably the most directly related work is Bongard, Zykov & Lipson's self-repairing robots from the early 2000's.

There's some more recent work from Mark Yim that you can see here too.

There are lots of different ways to do this, but Bongard et al's approach was probably the most elegant. The basic idea was to frame it as a learning problem: the robot is able to learn the shape of its body by performing controlled experiments. When the body is damaged, the robot can detect that it's body has changed shape (sensors don't report the expected values when it tries to move), perform new experiments to determine the extent of the damage, and then generate new movements that work around the damaged area. Lipson covers the basics of this system very briefly in this video.

The more modern system uses a similar approach, but tries to repair its body, rather than working around the damage. It's got an internal model of what it's body should look like, and then a set of cameras that help it locate the various pieces and move them to reassemble.

Dropout is sort of a similar idea, but dropout is usually done to encourage redundancy during training, which can help a model avoid overfitting. It's usually not done explicitly to heal a damaged system, although it would make a system more resistant to damage in the first place.


How to determine the probability of an "existence" question

A human has an abstract concept of numbers in mind. So 456 is a unique entity which is by definition unlike any other number because that are other unique entities. If you give ∃x ∈ ℕ: x==123 to your system it could check the property of natural numbers by counting from 0 to 123 to conclude that the statement is true. A human does it in another way. A human would "see" that it is a natural number because it has no decimal point. Because of the concept of natural numbers the statement is immediately clear. To get a faster result here the machine could check just the decimal point.

In your second case you have to apply the commutative property of addition and you are done because the syntax is equal then.

The second problem is more syntactic while the first is semantic. Your machine may "know" the commutative property of addition but not the concept of natural numbers. Therefore, it has to count.


How to train the images of various sizes?

I would use the ImageDataGenerator.flow_from_directory. Documentation is [here.][1]Make a train directory. Within it create 5 subdirectories one for each class and give them the desired class name. Place the associated images into the 5 class sub directories. Then use something like the code below. Set the image size to something standard particularly is you are using transfer learning. For MobileNet standard size is 224 X 224. You can use the Image Data Generator to augment your data for example set horizontal flip =True. You can also vary the brightness and rotation of the images.
tf.keras.preprocessing.image.ImageDataGenerator(
    featurewise_center=False,
    samplewise_center=False,
    featurewise_std_normalization=False,
    samplewise_std_normalization=False,
    zca_whitening=False,
    zca_epsilon=1e-06,
    rotation_range=0,
    width_shift_range=0.0,
    height_shift_range=0.0,
    brightness_range=None,
    shear_range=0.0,
    zoom_range=0.0,
    channel_shift_range=0.0,
    fill_mode="nearest",
    cval=0.0,
    horizontal_flip=True,
    vertical_flip=False,
    rescale=None,
    preprocessing_function=None,
    data_format=None,
    validation_split=0.0,
    dtype=None)
#for flow from directory
.flow_from_directory(
    directory,
    target_size=(256, 256),
    color_mode="rgb",
    classes=None,
    class_mode="categorical",
    batch_size=32,
    shuffle=True,
    seed=None,
    save_to_dir=None,
    save_prefix="",
    save_format="png",
    follow_links=False,
    subset=None,
    interpolation="nearest",
).

  [1]: https://keras.io/api/preprocessing/image/


The best way of classifying a dataset including classes with high similarity?

If you want to classify data with similar characteristics, it would often helps if you hand craft features. For classifying women or men wrestling, you may want to try using cv2 to track human faces and feed that to your CNN as input. Example: https://realpython.com/face-recognition-with-python/

If the data is not images, you may want to do a analysis to see which feature have no clear relationship with whether it is men or women wrestling and remove them. Example: https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0

Hope that I can help you and have a nice day!


If loss reduction means model improvement, why doesn't accuracy increase?

Loss reduction means model improvement, it does not in the wrong setup, wher random choise produces least loss. So it is some critical setup error. What classes do you have?
I got also thet recently experimenting with an encoder with too narrow coding layer - it just EQUILIZES the output with average values cause this state has minimum loss.


Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

When you say simple policy gradient, I assume you mean something like REINFORCE.

The main difference between actor-critic and REINFORCE-like algorithms is in how they estimate the reward to go:

in REINFORCE, you wait until a trajectory terminates to make any updates, and your estimator of the reward to go is the actual reward to go that was observed in the trajectory.
in actor critic algorithms, instead of using an entire trajectory to estimate a reward to go, you use the action value function (note that this way, you can compute updates at every time step). The reason why you use the action-value function is because at each transition, you have already committed to some action.

The benefit of the actor critic estimator is that it exhibits much less variance. REINFORCE estimators are Monte Carlo estimators, which are known to exhibit extremely high variance.

Now, the actor critic method in the slides you posted takes variance reduction one step further. Instead of estimating reward to go, it's estimating advantage -- that is, the difference between the current action value and your estimated state value. The state value function term doesn't depend on the action, so it doesn't affect the expected value of the policy gradient. However, it serves as a 'baseline', which helps reduce the variance of the reward to go estimator even further.


What does the notation ${s'\sim T(s,a,\cdot)}$ mean?

The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, $T(s,a,.)$ is a probability distribution.


To what extent can quantum computers help to develop Artificial Intelligence?

Quantum computers are super awesome at matrix multiplication, with some limitations. Quantum superposition allows each bit to be in a lot more states than just zero or one, and quantum gates can fiddle those bits in many different ways. Because of that, a quantum computer can process a lot of information at once for certain applications.

One of those applications is the Fourier transform, which is useful in a lot of problems, like signal analysis and array processing. There's also Grover's quantum search algorithm, which finds the single value for which a given function returns something different. If an AI problem can be expressed in a mathematical form amenable to quantum computing, it can receive great speedups. Sufficient speedups could transform an AI idea from "theoretically interesting but insanely slow" to "quite practical once we get a good handle on quantum computing."


LDA performs much better than other methods

It would be hard to tell if you don't provide what kind of data/problem you are working on, but LDA works well when data that are grouped in gaussian blobs surrounding centroids while vanilla SVM works well when the data is almost linearly separable and naive bayes works well when your features are relatively independent of each other.


Why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm?

I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to train the network is
$$\mathbb{E}[(r + \gamma \max_{a'} Q(s',a') - Q(s,a))^2]\;.$$
Here, $r + \gamma \max_{a'} Q(s',a')$ is your target, what you want your network to aim towards, and $Q(s,a)$ is what your network predicted. (Note that I have left off some details that can be found in the Nature paper for simplicity).

As for actor-critic methods, most popular actor-critic methods will use the value function to 'replace' the action-value function by using the following relationship:
$$\mathbb{E}[r + \gamma v_\pi(s')] = Q_\pi(s,a)\;.$$
This relationship can be proved by looking at exercise 3.13 (or somewhere around there) in the Sutton and Barto textbook. This looks like what the author is doing in the textbook you are reading.

Based on what I said at the start regarding how state-action value functions are trained, it is analogous to train a critic network that approximates the value function in the same way.

e1: spelling.

e2: added link to nature paper.


How do I calculate the probabilities of the BERT model prediction logits?

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes.
To get probabilties, you need to apply softmax on the logits.
import torch.nn.functional as F
logits = model.predict()
probabilities = F.softmax(logits, dim=-1)

Now you can apply your threshold same as for the Keras model.


Why "Exploratory moves do not result in any learning"?

I believe this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements seems like it will cause you to learn strange behaviors.

Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over all actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.


Does k consistency always imply (k - 1) consistency?

Define P as a CSP where X, Y are the variables, domain of both is {1,2,3,4} and conditions in normal form are:

node-condition X<4
arc-condition X=Y

P is 2-consistent (arc consistent) because for any X value it is possible to find a Y value that fulfills the arc-condition X=Y.

However, P is not 1-consistent (node-consistent) because exist a X value (X=4) that can not fulfill the node condition x<4.

For these reasons, this problem is 2-consistent but not strongly 2-consistent.

Obviously, it is straightforward to convert this example in a strongly 2-consistent problem, just reducing the domain to {1,2,3}.


Are leaf nodes included in the calculation of average branching factor for search trees?

From Wikipedia:

  In computing, tree data structures, and game theory, the branching factor is the number of children at each node, the outdegree. If this value is not uniform, an average branching factor can be calculated.

Outdegree meaning - In case of directed graphs, number of edges going into a node is known as in-degree of the corresponding node and number of edges coming out of a node is known as out-degree of the corresponding node.

You forgot the outdegree part. In AI we generally draw directed graphs from one state to another, and outdegree is the number of paths leaving a particular node. In your graph direction is not given. Also your graph is not symmetrical, but you can still find out the branching factor (with a little difficulty) of non-symmetrical directed graphs as given here. So technically  your conclusion is correct about leaf nodes not being counted (assuming they are the last state from which no other state can be reached - dead end). Hope this helps!


What is an intuitive explanation of how Google's AutoML works?

Google's AutoML is really a good idea in terms of autonomous model design. You can find the details in this blog. Let me explain briefly.

We, data scientists, design new networks by following existing models, trying and failing and trying again and again by analyzing weaknesses and strengths of the created models. However, we, as humans, have limited capabilities of designing/analyzing such networks. That's why Google created an AI which analyzes the strengths and weaknesses of each node while making a prediction. This AI analyzes each node and tries to improve the results by adding/removing/modifying connections of each node/layer. I guess AutoML AI takes state-of-art network as a base and starts modifying network according to your data to create a customized model.

While doing that, two technologies are being used: Transfer learning and Reinforcement learning.

Transfer learning is being used to start training from the most accurate point possible.

Reinforcement learning is being used to modify network to achieve a better success. This is the key part of this technology.

So, for users, it is more like upload your data, let AI modifies the network for you and give you a custom model which is specific to your data.


How does replacing states with latent representations help RL agents?

In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer what the state is telling it as part of the learning process. If you think of the architecture used in DQN to solve Atari, they had a CNN that outputted a vector which was then passed through some dense layers. Here the representation learning was done by the CNN and was trained using an end-to-end approach i.e. all updates to the network weights were done through the reinforcement learning objective; that is there is no supervised or unsupervised learning that takes place.
This can be particularly difficult when you combine images with sparse rewards as there is not a lot of feedback so the representation learning can take a long time. This paper gives a good description of the problem of decoupling representation learning from reinforcement learning with a nice solution.
The other main 'problem setting' I have seen images replaced with a latent state is when the authors are looking at planning. The problem with doing any kind of planning is that a model of the transition dynamics, $p(s' | s, a)$, is needed. For high dimensional state spaces such as images, this can be very difficult to predict and even relatively small errors will quickly compound so if you use the model to predict multiple time steps into the future the planner is useless because of these compounding errors. I think there is a discussion on this in this paper (certainly there will be references therein that point you in the right direction).


Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):

Neural network weight initialisation
Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
Minibatch sampling from experience replay
Stochastic choices in the environment (does not apply in your case)
Other stochastic parts of training that may be in use, such as dropout regularisation

Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.

  I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script.

This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:

Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.
Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.

You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.

Here are a couple of additional tests that may help:

Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.
Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.

If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?

Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.

You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.


The biases appear in the activation function and can be seen as node feature. Suppose you have act = sigmoid, X your input layer and W your first hidden layer weight matrix.
Then the output matrix of this hidden layer will be act(X.dot(W)+b).

So your bias should be a vector containing a random value for each node of the current hidden layer.

It will shift randomly the weight and so the activation function. See here (usually it prevent from overfitting).

I didn't read the whole code so I don't know if the problem comes from here, but why don't you learn how to use tensorflow instead of re-write everything ? There are a lot of tutorials in the net, like MNIST.


Can artificial intelligence take over the world?

The world seems to be operating under the "law of maximum intelligence" (I made this up), which is that the species with the most intelligence get to realise its preferences the most strongly. Put another way, the more intelligent you are, the more the world takes the shape of what you want; and this happens through your actions upon the world.

This is tautological if what you mean by "intelligence" is "ability to act well".

For example, both humans and chimps want to be happy, but you get zoos and not chimps paradise, because zoos are optimised to please humans, not chimps.

If you grant this law, then all you need is a "greater than humanity" level of intelligence, and an entity with that level of intelligence should have its preferences met.

As to whever or not this will happen, I’ll quote Sam Harris in this little talk:

  All you have to grant to get your fears [about AI] up and running is
  that:

  We will continue to make progress in hardware and software design
  (unless we destroy ourselves some other way), and
  There’s nothing
  magical about the wetware we have running inside our heads, and that
  an intelligent machine could be build of other material. […]

  Once you’ve granted those two propositions, you’ll now be hard pressed
  to find some hand hold with which to resist yor slide into real
  concern about where this is all going.

To go back to the question: yes if you have an entity/agent that is sufficiently intelligent, and with a goal that would directly or as a side-effect cause it to take over the world.
Note that this is a real concern because of instrumental convergence. Good talk on this topic by Nates Soares.

I also recommend the lesswrong forums: https://wiki.lesswrong.com/wiki/AI_takeoff and Nick Bostrom: https://nickbostrom.com/superintelligence.html (or his book also titled Superintelligence)


How should I penalize the model proportionally to the error?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols $3$ and $4$ do not differ in a way that captures quantity in any intuitive manner.

However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras.

In order for your loss function to work, it will need to be differentiable and  smoothly changing as predictions get better. That rules out using any form of argmax for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the expected prediction, e.g. if $d_i$ is the numerical digit for example $i$ and $y_{i,j}$ is the ground truth expressed as a one-hot vector, where $i$ is the example and $j$ is the digit class, then $\hat{y}_{i,j}$ is the probability predicted by your model. You could use $\hat{d}_i = \sum_{j=0}^9 j\hat{y}_{i,j}$ for the expected value and MSE loss of $\mathcal{L}(d_i,\hat{d}_i) = \frac{1}{2}(\hat{d}_i - d_i)^2$

You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model.

Note this solution makes $0$ close to $1$ but far away from $9$. If you want the digits to be considered close on a cycle (e.g. $8$ is closer to $1$ than it is to $4$) the you will need something more creative.

Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.


What are examples of books or papers on the details of convolutional neural networks?

Chris Olah's work is always inspired, and not too technical as one would expect. He has several papers on CNNs on his website. In particular, check the series titled "Convolutional Neural Networks" with four papers on the topic.


What you call is a typical OCR application. There are two things to consider :

1- Localizing the text

2- Recognizing the text

Please note that it is not an easy task to solve the problem globally. However, depending on your working subset, you need to get the data first. There are a lot of open source ocr database that you can find.
Then, you need to add some preprocessing and it depends on which kind of path you want to choose. You can find lots of articles about it.
Then, you need to add a localization network (like Faster R-CNN or any specialized one for OCR), with the combination of recognition network.
This was deep learning approach. It has a learning curve if you have never worked with deep learning frameworks.

Another way that you can take is to get different character image dataset. Resize all of them to same size, flatten to 1-D and train a classifier. Then, you can start to distinguish different image patches if they are character or not (or which character). Then, for inference, you need to crop different sub-regions from input image and feed it to classifier to get a result.
It is more costly in terms of inference time and the success rate will more likely to be much lower than deep learning method but you can learn & implement this one much quicker.


It depends a lot on the hardware of your robot arm.
Assuming that your servos have encoder information, if you have access to servos that have limited control like "rotate left/rotate right" functionality, you can phrase the your action space to be ["move left", "stop", "move right"]. In this way you can implement a discrete action space with 3 actions per servo and have an agent learn to move the servos around the space.

If your servos are connected to each other in an elbow/shoulder configuration, you can have a 9 discrete action setup essentially making a box of cardinal directions:

Up+Left----------Up-------Up+Right

Left--------------Stop---------Right

Down+Left-----Down-----Down+Right

If you have 3 or more servos, you can still use the same idea of discrete actions but the number of discrete actions grows by a factor of 3 with each servo as your action space is now the cross product of all of the other servos.

Alternatively you can use a "multi-headed" agent where each head chooses actions for a certain servo but there are pros and cons for both depending on your usecase.

If you have more advanced servos like Dynamixels which have high quality encoders, you'll have access to more advanced controls schemes. For instance, Dynamixels allow you to give actions in encoder space, angle space, and even velocity space. For example, you could give the action of "go to encoder value of 500" or "go to 90 degrees" or "move .5 radians/second". All of these approaches are useful for certain tasks. For humans controlling the arm using a joystick, the velocity based is the most intuitive and, in my experience, the same is true for RL agents using continuous control.

If you are using continuous control, you should normalize all of your action spaces within your agent then "unnormalize (?)" them before giving the actions to your servos.
For instance if your servo velocity ranges from -3.5 rad/sec to +3.5 rad/sec, have your agent select actions in the range of [-1,1] then multiply by 3.5 to get the velocity.

In either case, one thing that should be noted is that you give your robotic arm enough time to actually perform the action that the agent selected. If not you will see your robot "jitter" back and forth quickly as your agent selects actions randomly. This is bad for a few reasons but most importantly because it might break your servos. To overcome this issue, give each action a little more time to be "actualized" by your robot. This can either come in the form of adding a delay in your code while your servos do "the thing" or by using an "iterations since last action update" counter to get a new action from the agent after a certain number of iterations. Not only is this better for your hardware, this also leads to better exploration of your state space as your agent can move through the state space encountering similar states more frequently.

A last thing to be aware of is to set hard coded limits on the servos so that your agent doesn't kill your servos by banging the arm against a table or something. Finding these safe bounds isn't always easy as multiple jointed configurations can have multiple ways of hitting the bounds and it might require a forward dynamics model to find these limits.
If you have only 2 servos, it should be pretty easy to find though :).


How to estimate the AI player's strength in multiplayer game?

The following are extremely simple ways of tackling this problem.
A very simple way
It can simply be
strength of AI=(# of games won)/(total # of games).

In case data for each move is available
Something like
score per game=# of correct decisions/total number of decisions.
Then
strength of AI=sum(score per game)/total # of games.

If each move/decision has a score associated with it
then you doscore per move=scored points by taking a decision/maximum possible score.
then
score per game=sum(score per move)/total # of moves
and finally,
strength of AI=sum(score per game)/total # of games.

How to choose optimal number of games to play?
It depends on your requirements. If you want to report your AI's strength in percentage of the games it played correct to 1 decimal place (for example, this AI won 95.1%) then 10000 is an optimal number of games your AI needs to play. Suppose your AI won 9508 games out of 10000 then you will have 95.08% strength of AI. To be able to correctly round it to 1 decimal place you need to have an additional decimal place so that you can quote the strength of you AI with reasonable confidence, in this case 95.1%.


Feature extraction other than convolutions for images?

Some more complex and better outcomes are those that include the MPEG-7 developed by the MPEG group.

These descriptors specify the video standard in question. They are split for color, shape, texture and movement.

1. Color Descriptors

Color Space. Specifies the data type of the color space in which they are expressed or work other color descriptors. Color spaces that it contemplated are: RGB (Red, Green Blue), YCrCb (Luminance, Chrominance), HSV (Hue, Saturation, Value), HMMD (Hue, Maximum, Minimum, and Difference).

Color Quantization. This descriptor defines a uniform quantization of a given color space.
Dominant Color(s). This descriptor is the most suitable to be used in images, or regions, in which a small number of colors is sufficient to characterize the information of one determined region.

Scalable Color. This descriptor consists in a color histogram in the HSV space. It is useful in image to image comparisons or searches based on color characteristics.

Color Layout. This descriptor allows representing the color spatial distribution within the images in a very compact way, so the recovery is realized with great efficiency.

Color Structure Descriptor. This descriptor characterizes the color distribution in an image. It builds a color histogram, which assigns most importance to the colors that appears most often and spread across the image. GoF/GoP Color (Group of Frames/Group of Pictures). This descriptor is an extension of the Scalable Color descriptor, which, unlike the latter, is applied to video sequences (a collection of images) instead of only an image.

2. Texture Descriptors

Homogeneous Texture. This descriptor emerged as an important tool when looking for and choosing within large collections of images of great visual similarity.

Texture Browsing. This descriptor specifies the perceptual characterization of one particular texture, and pretend be similar to the characterization that a human eye makes, according to regularity terms, coarseness and directivity.
Edge Histogram. It is a descriptor that gives us information on the type of contours or edges that appear in the image.

3. Shape Descriptors

Region Shape. The object´s shape in an image may consist of a single region or a set of regions, so this descriptor is useful for this type of characterization.
Contour Shape. It is characterized by very well represented contour features which facilitates subsequent search and retrieval; it is robust to motion, to occlusions and to different perspectives; it is extremely compact.

Shape 3D. The Shape 3D allows to describe in details the shape in 3D. This tool is very useful today due to continuous development of multimedia technologies.

This was taked from my paper: The MPEG-7 Visual Descriptors: a Basic Survey

Best regards


Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?

I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:
$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result


What is the advantage of using MCTS with value based methods over value based methods only?

Assuming a continuous/uncountable state space, we can only estimate our value function using function approximation, so our estimates will never be true for all states simultaneously (because, loosely speaking, we have far more states than weights). If we can look at the (approximated) value of states we take in, say, 5 actions time, it is better to make a decision based on these estimations, taking into account the true rewards observed after the 5 actions.
Further, MCTS also allows more implicit exploration as when choosing the actions to expand the tree we are potentially choosing lots of non-greedy actions that lead to better future returns.


How to use speaker's information as well as text for fine-tuning BERT?

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper.
Generally, when working with mixed data such as continuous and categorical features, the first step is to simply concatenate all the inputs into one long vector. In your case, you would concatenate a one-hot encoding of speaker ID to the BERT [CLS] output for each example. If you're using tensorflow, you'll need to use the functional API to create a multiple input model, as I outline below.
import tensorflow.keras as keras

# bert_input is the wordpiece tokenized input text
# BERT is the BERT model, such as a tensorflow_hub.KerasLayer with a pretrained model from tfhub.dev
bert_out = BERT(bert_input)

# spkr_input is the one-hot speaker ID encoding.
spkr_input = keras.Input(shape=(num_speakers,), name='spkr_input')
dense_input = keras.layers.concatenate([bert_out, spkr_input])
scores = keras.layers.Dense(num_classes, activation='softmax')(dense_input)
model = keras.Model(inputs=[bert_input , spkr_input], outputs=[scores])

You could also try feeding the one-hot encoded speaker ID vectors through a separate fully-connected network first to obtain a continuous representation (i.e. a speaker embedding) and then concatenate that to the BERT [CLS] output and feed the result into your classification layer. Modifying my example above,
spkr_emb = keras.layers.Dense(spkr_emb_size, activation="relu")(spkr_input)
dense_input = keras.layers.concatenate([bert_out, spkr_emb])

You can find a more detailed guide to mixed input models with Keras at https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/


How does the forget layer of an LSTM work?

TL;DR Here is a beautiful explanation with diagrams: source.
To address:

the cell state is essentially long term memory embedding (correct me if I'm wrong)

The embedding can be long or short term and it is a vector.
To answer:

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

Excerpt from source:

The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”

Formally the forget vector is:
$$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$
So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing:
$$C_{t-1}\odot f_t,$$
where $\odot$ is element-wise multiplication.
Thus we see that the forget gate really is acting like a gate: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.
To answer:

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

This concatenation takes into account the "rolling" hidden state and the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.
In summary, the forget gate has the primary purpose of forming a vector of values between zero. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.
Additional Intuition - More Mathematical
Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of $W_f$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $W_f$ form a sort of "basis" for a "toggle space."
When we concatenate to form $[h_{t-1},x_t]$ and then multiply we see that the term $W_f\cdot[h_{t-1},x_t]$ gives consideration to the hidden state and the input. That is, both $h_{t-1}$ and $x_t$ have there hand in toggling the gate.
The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example.

That is, $W_f\cdot[h_{t-1},x_t]$ will be a vector in the column space of $W_f$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $W_f$. This "toggle" is then transformed into a "gate" in the "gate space."
So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $[h_{t-1},x_t]$. Formally:
$$f_t:[h_{t-1},x_t]\mapsto g_t$$
That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.
Important Note
The LSTM is only one of many types of forget-gate mappings. The main takeaway is that the LSTM works empirically for many applications.
That is a forget gate is an instance of a general "gate mapping":
$$f_t:v_t \mapsto g_t,$$
where $v_t$ could be any number of vectors resulting from any number of concatenations.
Conclusion
Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. One implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.
More Mathematical Intuition
Here is another view of what is happening in the LSTM from a dynamic point of view: video.


How does Pinterest decipher what's on unmarked pictures and categorize them?

One of the Pinterest's white paper about Human Curation and Convnets powering item-to-item recommendationsarxiv describes implementation of convolutional neural network (CNN) based visual features (VGG2014, Faster R-CNN). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the previous study about visual search at Pinterest and can be used for more targeted features to be computed for related pins.

Here are the examples of detected visual objects from Pinterest's object detection pipeline:

Image source: Human Curation and Convnets: Powering
Item-to-Item Recommendations on Pinterest, Page 4, Fig. 6

The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.

Check also this blog entry: Building a scalable machine vision pipeline.


How are the observations stored in the RNN that encodes the state?

This research question seems to be analyzed in further details here (section 3) - https://openreview.net/pdf?id=r1lyTjAqYX.

Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed -

zero initialization: When the RNN is initialized with the zero state
burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state
storing the initial hidden state: When the hidden state at the beginning of the sequence is stored


Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of the input sequence.
What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one $x$ we have a sequence of related, i.e. not i.i.d. data such as time series data, data $\{x_i\}_{i=1}^n$. Assuming we are given some initial $h_0$ (a hidden state) then an RNN will take as input $x_1$ and $h_0$ and output a prediction $y_1$ and a new hidden state $h_1$. In general an RNN will take as input $x_n$ and $h_{n-1}$ and output $y_n$ and $h_n$ where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the $x$'s and $h$'s will all be the same, i.e. $x_i \in \mathbb{R}^d$ and $h_i \in \mathbb{R}^c$ for all $i$, where $d$ does not necessarily have to be equal to $c$ (and in my experience rarely is).
Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder $\{e_i\}_{i=1}^n$. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder $e_n$ as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in this paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.
Now, with this data $\{x_i\}_{i=1}^n$ you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found here.


Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?

As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the __init__ we are noticing that the log_std is being formulated instead of the std itself. Would it be wrong if you formulated the std itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled std it could have been possible for the std to become 0 or negative. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the log_std becomes negative or zero it does not matter because the exp will take care of it by exponentiating it to a positive number.


Two data classes for a convolutional neural network, can one have a LOT more images for training than the other?

Your question is very general so therefore, in this case, my answer will be too:

The answer is "sometimes": it depends on the data.

There can be a lot more images in one class than the other, and you can still get reasonable results. It highly depends on how much data you have of the "feature class".

If this is the case, we say that the classes are heavily unbalanced, and you need to do "class balancing". You do not want to do overfitting on this one class, and preferably you want the feature-class to be the biggest.

Another approach for CNNs is to use "dropout". Well, for CNN's you can go a bit further: you can remove parts of the image to generate "new" images. This way you prevent overfitting of the "feature" class, whilst generating more data.

I suspect that training all possible false positives is impossible without overfitting the network somehow.

Hope it helps, and give you some google pointers :)

Just FYI: You basically, in the tech term, want to know whether it works to do binary CNN classification using a heavily imbalanced dataset.


$ \gamma $ goes up to 1, but cannot be greater than or equal to 1 (this would make the discounted reward infinite).

The discount factor $ \gamma $  determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term high reward. If the discount factor meets or exceeds 1, the action values may diverge. For $ \gamma =1$, without a terminal state, or if the agent never reaches one, all environment histories become infinitely long, and utilities with additive, undiscounted rewards generally become infinite.

Source:https://en.wikipedia.org/wiki/Q-learning
https://cs.stanford.edu/people/karpathy/reinforcejs/puckworld.html


What is the difference between evolutionary computation and evolutionary algorithms?

As you can find here:

  Evolutionary algorithms form a subset of evolutionary computation in that they generally only involve techniques implementing mechanisms inspired by biological evolution such as reproduction, mutation, recombination, natural selection, and survival of the fittest.

It means, other types of evolutions, which are not necessarily a biological evolution, are in EC, but not in EA. For example, Learning classifier system is in EC as it is evolutionary, but not completely in EA as it is not biological.


Can I export a trained machine learning model so that others case use it?

If the 'algorithm' you're talking about is a neural network, then you can distribute the learned parameters/weights to anyone who wants to use it. This is how neural nets are normally 'exported': without all of the training data used to create them. Actually, this is done with many kinds of models (parameterized ones).

In order to 'decode' the model, users would only have to know its structure. In the case of a neural network, they'd need to know the size of each layer, what activation functions were used, etc.

This is not possible with every type of ML model, however. Specifically, non-parametric models and 'lazy' models make use of training data at inference time. They wouldn't be useful without their training data. Classifying an input by finding its k nearest neighbors, for example, would require having the training data.


Sentiment analysis does not handle neturals

Yes, there is. You can try Spacy. Here you go.
import spacy
from spacytextblob.spacytextblob import SpacyTextBlob

nlp = spacy.load('en_core_web_sm')
spacy_text_blob = SpacyTextBlob()
nlp.add_pipe(spacy_text_blob)

text = "i'm good"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.7

text = "i'm bad"
doc = nlp(text)
print(doc._.sentiment.polarity) # -0.6999999999999998

text = "i'm neutral"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0

text = "i'm okay"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.5

text = "i'm indifferent"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0


What is the search depth of AlphaGo and AlphaGo Zero?

For easier visualization, I recommend this video:
https://twitter.com/i/status/1257053365424578565
The more detailed article about GO algorithms:
https://deepmind.com/blog/article/alphago-zero-starting-scratch.
With its breadth of $250$ possible moves each turn (go is played on a $19$ by $19$ board, compared to the much smaller $8$ by $8$ chess field) and a typical game depth of $150$ moves, there are about $250^{150}$, or
$10^{360}$.
After $2$ moves in go, it's $130000$ possible combinations.
Decision tree pruning


Is there a simple way of classifying images of size differing from the input of existing image classifiers?

My guess:
If you add a few CNN layers before the input of the given model and train only those layers while keeping the given model's parameters frozen, you might get better result.
Essentially these few extra layers would "transform" your input image into the appropriate shape, but with more accuracy since its trained and not hard coded.


Should binary feature be in one or two columns in deep neural networks?

You're simply adding a redundant feature by having it as two: $X_2 = 1 - X_1$. It would be equally useful to duplicate the first column. At best this will not improve your model, at worst it will decrease accuracy.


Is there an ideal range of learning rate which always gives a good result almost in all problems?

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing.  Your ideal range consists of the learning rate values where the loss was decreasing steeply.  After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.
Here is a concrete example from one of my experiments:

In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that).  Your experiment may require different search bounds, but you could always start with that and adjust things as needed.  At first, the loss plot is flat, and then it begins to decrease, but is too gradual.  At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search.  For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.
For more information, I suggest reading this Keras Learning Rate Finder post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.


Is it possible to compute $P( F \mid S )$ given $P(F \mid S,A)$, $P(F \mid S, \lnot A)$?

This is a bit of a puzzle but you can compute a reasonable narrow limit even without knowing whether or not $P(S,A) = P(S) P(A)$.

Start with the contingency table relating $P(S, A)$, $P(S,\neg A)$, $P(\neg S, A)$, $P(\neg S,\neg A)$ to $P(S)$ and $P(A)$ :

$$\begin{array}{cc|c}
P( S,A)& P(\neg S,A) & P(A) \\
P(S,\neg A)& P(\neg S,\neg A) & P(\neg A) \\ \hline
P(S)& P(\neg S) & 1 \\
\end{array} \quad \rightarrow \quad \begin{array}{cc|c}
x& 0.01-x & 0.01 \\
0.07-x& 0.92+x & 0.99 \\ \hline
0.07& 0.93 & 1 \\
\end{array}$$

note that the cells must be between zero and one thus $0 \leq x \leq 0.01$

Then with

\begin{array}{rcrl}P(F|S) &= & & P(F|S,A) P(A|S) + P(F| S,\neg A) P(\neg A|S)\\
&=&& 1.0 \frac{x}{0.07} +  0.7 \frac{0.07-x}{0.07} \\
& = && 0.7 + 4 \frac{2}{7} x  \end{array}

you get

$$0.7 \leq P(F|S) \leq 0.743$$

To solve $P(F|S)$ exactly you need to narrow down $x$ more precisely. Possibilities are:

If you know $P(F)$ then you could use $$\begin{array}{rcrl}P(F) &= & & P(F|S,A) P(S,A) \\ && +& P(F|\neg S,A) P(\neg S,A) \\ &&+& P(F| S,\neg A) P(S, \neg A)\\&&+&P(F|\neg S,\neg A) P(\neg S,\neg A)\\
&=&& 1.0 (x) + 0.7 (0.07 - x) + 0.9 (0.01-x)+0.1(0.92+x) \\
& = && 0.15-0.5 x \end{array}$$
If you know that $S \perp \! \! \! \perp A$ then you can use $x = P(S,A) = P(S)P(A) = 0.0007$


How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I do not understand, how to stop the evaluation here, because the policy update is linked to epsilon. So just setting epsilon equal to zero at some point does not seem to make sense to me.

If by evaluation you mean the act of exploring new paths, it does not need to stop by changing epsilon to 0 instantly. Instead, in order to facilitate the convergence of the algorithm, the epsilon can be progressively decreased until it reaches 0.

I do not think your question is meant to be related with monte carlo specifically, but if this was not what you wanted to know, please comment.

  But would not a epsilon value of 0 lead to an unintended policy update?

No, a value of epsilon equal to 0 will make you always choose your action according to the policy. I think what is confusing is that the "update" they make to the policy on the last for, is not an update. It is a statement saying the probability of taking such action. What it means is the following:

For action $a$, the probability of taking $a$ in state $S_t$, when $\epsilon$ is 0 is:

1 if action $a$ is the action with the best value
0 if action  $a$ is not the action with the best value

What this means is that when epsilon is 0, the action taken will always be the greedy action.


How do I select the relevant features of the data?

Since you have all your data in a table, a relatively simple thing to do is to consider each column independently, and then seeing if the output variable (cost incurred) has a correlation to that.

If the column has no (or very low correlation) with the output variable, then consider it to be not important.  The ones that make the cut are then considered further.

This is obviously not very different from how a decision tree algorithm would work (such as ID3).


Understanding why the expectation is over the new policy $\pi'$ in the proof of the Policy Improvement Theorem

The expectation is over the policy $\pi'$ because the action at the state $S_t = s$ is taken according to $\pi'$, and, for the proof, the book text (2nd edition, paragraph below Equation 4.8) defines $\pi'$ to be a policy that is identical to $\pi$ except that $\pi'(s) = a \neq \pi(s)$, where $s$ is one particular state.

So, essential, the book text tries to prove that, for such a changed policy $\pi'$, if $q_\pi(s, a) = q_\pi(s, \pi'(s)) > v_\pi(s)$, then the changed policy is better than $\pi$. Note that $\pi'(s) = a$ in this case.

That is why $q_\pi(s, \pi'(s)) = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s]$.


How can reinforcement learning be applied when the goal location or environment is unknown?

When reading about RL and RL agents/algorithms, you always need to keep in mind that, typically, the RL agent/algorithm is trying to maximize the reward (or something equivalent, such as minimizing the regret) in the long run (i.e also the reward that you may receive in the future): that's its (mathematical) goal. Whether that  also corresponds to the high-level goal (e.g. reaching some physical location in some world) that the (human) designer of the so-called reward function (i.e. the function that gives the reward to RL agent) had in mind is a different story.
To maximize the reward, the RL agent interacts with the environment by taking actions, receiving rewards, and moving to other states. Initially, the RL agent does not know which actions lead to more rewards, so it may take random actions (this is known as exploration). Once it starts to understand the dynamics of the environment, it may start to take only the actions that lead to high reward (this is known as exploitation).
To answer your question directly, the RL agent can indeed take actions without knowing the dynamics of the environment. However, initially, it may need to take some random actions (which may lead to low rewards), so that to get more rewards in the long run.


What is the difference between automatic transcription and automatic speech recognition?

They are both the same. There are different algorithms to recognise speech, but essentially they all aim to identify the content of the spoken input and convert it into written text.

Automatic transcription is then done, whereas the output of more general ASR is often passed on to further processing, such as recognising entities or commands expressed in the speech.


Is the expert system still in use today?

I would say Expert Systems is still being taught. For instance, if you look at some of the open courses like MIT's, there are still lectures on it.

Also, looking at the CLIPS documentation, you will find a couple of examples of usage from 2005.

What I suspect is that Expert Systems are now embedded with "normal systems" in practice. Hence it may be difficult to distinguish from systems used on a daily basis for diagnostics, etc. and not as popular as before.


How to analyze data before going for machine learning training?

Though there is no universal method which can be blindly used for all datasets, but here is what i usually do;

Fill missing values using interpolation or mean, if missing values
are less than 10-15 percent of number of rows else drop the column.
Encode categorical data using some kind of encoding, e.g. one hot, etc.
Then normalize/rescale columns.
Now look at the variance in each feature. Usually, features with more variance are more important.
Next, see the correlation among columns. If two columns are highly
correlated, you only need to keep only one.


Reinforcement learning with hints or reference model

You can look into the techniques used in GANs (genereative adverserial networks). These networks work by having 2 learning agents. 1 to create images and 1 to learn the difference in a human made image and a computer generated image. This works because the 2 agents drive each other to be better and ultimately make the generator create images which can't be distinguished from real life images.

In your case you can make a agent trying to tell if the data is human or computer generated. The agent learning to move will then get negative rewards when the other agent can identify it as computer movement. This way the mover will learn to move like your reference data.

UPDATE:

I just found this video and paper which does exactly the same as you asked. Instead of using a GAN like structure they use a task specific reward and a imitation reward, which is based on reference motion data they have.

https://www.youtube.com/watch?v=vppFvq2quQ0


How to calculate v min and v max for C51 DQN

If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this issue you bring up is usually cited as a difficulty with C51. I think people tend to simply use fixed values for the min/max return (or just make rough estimates). If you want to avoid this, I recommend looking into the QR-DQN algorithm which circumvents this issue altogether and is more theoretically sound.


How can I implement back-propagation for medium-sized neural networks?

Backpropagation isn't too much more complicated, but understanding it well will require a bit of mathematics.
This tutorial is my go-to resource when students want more detail, because it includes fully worked through examples.
Chapter 18 of Russell & Norvig's book includes pseudocode for this algorithm, as well as a derivation, but without good examples.


How to solve Snake Game with a Hamiltonian graph algorithm?

A Hamiltonian path in a graph is a path that visits all the nodes/vertices exactly once, a hamiltonian cycle is a cyclic path, i.e. all nodes visited once and the start and the endpoint are the same.
If we want to solve the snake game using this, we could divide the playable space in a grid and then try to just keep traversing on a hamiltonian cycle, this means you would eventually get all the rewards and never hit yourself unless you are longer than can fit on the screen.
You can look at the code given in this Github repo.

Note that this method, as you can tell, is not going to be optimal by time, there could be better solutions than this.
You will have to use a graph to store all the nodes and the complexity is supposed to be O(n!).


Is there any difference between the convolution operation applied to images and applied to other numerical 2D data?

Short answer is no. You can't use a model trained for one task to predict on a totally different task. Even if the second task was another image classification task, the CNN would have to be fine tuned for the new data to work.

A couple of things to note...

1) CNNs are good for images due to their nature. It isn't necessary that they'd be good for any 2-dimensional input.

2) By 2D numerical data I'm assuming you don't mean tabular data.


How do I represent a multi-dimensional state using a neural network?

In my humble opinion, it seems like it is important to have them separated, if having a certain card can influence the result in some way that is not its prime value, instead of not only using the sum. But it depends on the game and its rules. For example:

If having 5 cards of hearts in the set of 15 cards makes you win the game, then if you only represent the state as the sum of the value of the cards, the DQN will never learn that it was not the whole 15 cards that caused you to win, but only a specific part of the set.


Is it better to rely on an intention file or a database for a web chatbot?

Recognising intents is only a small step in developing a chatbot. It's fine to use an ML classifier with training data for that, no need to keep the original list of intents.

However, you should really think about the next step: how are you getting your bot to conduct a dialogue, rather than firing off single responses to user queries. That is where things get difficult, and that is also what distinguishes a good chatbot from a simple-minded Eliza-clone.

The programming language/framework you use is not relevant.


Is the following statement about neural networks overclaimed?

I think I would never say that neural networks are iterative methods. I would say that iterative methods (e.g. gradient descent) are used to train neural networks (which can be thought of as linear and non-linear models, but mainly non-linear), which is quite different. Maybe you should or wanted to say that deep learning is an area of study where iterative methods are used to train neural networks.

It is possible the reviewer is just telling you that this or similar statements have been used a lot and don't really provide any insight, they are misleading, or simply useless or trivial in the context of the paper or conference.

Without further clarifications by the reviewer and about your paper and conference, it is difficult to provide more explanations.


The question is: how to represent that data in a human-readable way, that emphasizes the flow of the graph?

Train a reasoning engine to understand the decision tree for you.

Observe how IBM Watson/The Debater can

Receive a particular question
Find and read Wikipedia articles related to the question
Understand parts of those articles and generate human-relevant arguments for you.

Follow these steps:

Develop your decision tree however you normally would.
Train a reasoning engine that can output natural language about concepts within decision trees.
Apply reason engine from step one to decision tree in step one; repeat.


How can I detect the frame from video streaming that contains a graffiti on city wall?

ResNet is an architecture for object recognition and you may use it to do your classification task.  Fast RCNN may improve your results but is a more difficult architecture to implement.  If you want to go in this direction the best place to start is the arxiv paper of the Fast R-CNN (arxiv.org/abs/1504.08083).  If I am not wrong, there is an implementation of fast-rcnn in pytorch if you really want that.

You seem to be new to deep learning.  I would strongly recommend starting simple. ResNet-50 will probably be more than enough for your application. Moreover, I suggest using a lib like fast.ai.


How to normalize for perceptual loss when training neural net from scratch?

I am a bit confused with the part of "training a new neural network from scratch" and then the part of "VGG's weights will be frozen" because the answer changes since they are different cases. It can also happen, from what I read, that you are using VGG as backbone of another network for transfer learning but it is not specified either. Anyway, I will try my best.
The cases I see:

Training a network from scratch: no, you don't need to undo your normalization. In fact, when training from scratch you can normalize however you want.
Training a network from scratch starting from pretrained weights: no, you don't need to change your normalization. Although it would help to faster convergence because the weights are fitted to other kind of normalization. But as the network trains it will fit to your new normalization without any problem.
Training a network that uses VGG as backbone with frozen weights: no, you don't need to change your normalization although it would help. In spite of having your backbone fitted for a specific kind of normalization your network head is still an universal approximator that would fit to any other kind of normalization. However if you use the same normalization you are easing the network's job (that's why we normalize in the first place)

When you really need to force same normalization is in inference. When you want to infer you really need to do the same normalization in preprocessing or you won't have any meaningful results.


Where are reinforcement algorithms used in financial services?

I know the new bot in JP Morgan uses reinforcement learning:
Machine Learning in FX
There a lot of studies of Reinforcement learning can apply in crypto currency trading. Here is one of many examples:
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem


What are the meanings of these (P(x;y), P(x;y,z),P(x,y;z))?

This means "Parameterized by".

First, we all agree on the idea of conditional probabilities:

$$P(X | Y) = P(X,Y) / P(Y)$$

That is, the probability that X happens given that we've seen Y happen, is the fraction of worlds in which Y happened that also contain X. This is uncontroversial.

If you're a Bayesian, you might view parameters themselves as variables in a statistical model. So you might want to speak about the probability of a parameter taking on a certain value, or the probability of the data given that certain parameters have taken on certain values. In that case, you might write something like $P(D | \theta)$ to denote the probability of the data given a parameter $\theta$.

If you're a frequentist, you might find notation like this unsettling, because parameters don't have probabilities under the frequentist view, but instead have fixed values. You could talk about the probability of observing the data under a particular parameterization by defining a family of parameterized probability density functions, and then writing something like $P(D;\theta)$. You might also do this as a Bayesian if you wanted to clearly differentiate between model parameters and other things you might observe.

$P(x;y)$ then is read as "The probability of x under probability density function P, parameterized by y."

$P(x;y,z)$ is "The probability of x under probability density function P, parameterized by y and z."

$P(x,y;z)$ is "The probability of x and y jointly, under probability density function P, parameterized by z."

You could also write something like $P(X|Y; z)$, which would be "The conditional probability of X given that we observe Y, under probability density function P, parameterized by z."

An example of the latter would be in something like logistic regression. We might wish to know $P(L | D ; \theta)$, where L is the label of the data, D are the features we observe, and $\theta$ are the coefficients of the regression.


What is a heatmap in the CornerNet paper?

Heatmap in the sense of Corner net is the heatmap of the pooled corner values. As discussed in the paper, there is a corner pooling operation that gives you the vector values for a pixel point being a corner(it may or not). The output from the corner pooling is CxHxW. Then to generate a heat map you train the network similar to the Grad-CAM method. Training for heatmap generation means that we are going to tell how much a given pixel point has a weight to be a corner point. i.e if it has a high weight it will generate more colored values on that pixel area.
The heatmap is used for the prediction of corner points. That means if you know which pixel has a higher weight then that pixel will have a higher probability to hold the top-right or bottom-left corner of the given bounding box.


Why do activation functions need to be differentiable in the context of neural networks?

It is almost mandatory to have a differentiable activation function unless, of course, you have an alternative to training the network by back-propagating the error.


96.91% accuracy on MNIST after 2 hours of training using custom made neural net library. Ways to improve?

There is no singular best set of hyperparameters. Even more, there is no real search algorithm for hyperparameters. You can do a grid search, but this obviously will take some time. Most people either do that or will try to handpick their parameters.
A few other things to note: Initializing your weights at [10^9,10^8] doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.


Why is batch gradient descent performing worse than stochastic and minibatch gradient descent?

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.

The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.

You can check this out for more on the differences between the three. Hope this helped!


Are there RL algorithms that also try to predict the next state?

Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.
This paper introduces PlanGAN; the idea of this model is to use a GAN to generate a trajectory. This will include not only predicting the next state but all future states in a trajectory.
This paper introduces a novelty function to incentivise the agent to visit unexplored states. The idea is that for unexplored states, a model that predicts the next state from the state-action tuple will have high error (measured by Euclidean distance from true next state) and they add this error to the original reward to make a modified reward.
This paper introduces Dreamer. This is where all learning is done in a latent space and so the transition dynamics of this latent space must be learned, another example of needing to learn the next state.
These are just some examples of papers that try to predict the next state, there are many more out there that I would recommend you look for.


What is the difference between tree search and graph search?

There is always a lot of confusion about this concept, because the naming is misleading, given that both tree and graph searches produce a tree (from which you can derive a path) while exploring the search space, which is usually represented as a graph.
Differences
Firstly, we have to understand that the underlying problem (or search space) is almost always represented as a graph (although the underlying graph may not contain cycles, so it may represent a tree). So, the difference is not whether the problem is a tree (a special kind of graph), or a general graph!
The distinction is, instead, how we are traversing the search space (represented as a graph) to search for our goal state and whether we are using an additional list (called the closed list) or not.
So, the basic differences are

In the case of a graph search, we use a list, called the closed list (also called explored set), to keep track of the nodes that have already been visited and expanded, so that they are not visited and expanded again.

In the case of a tree search, we do not keep this closed list. Consequently, the same node can be visited multiple (or even infinitely many) times, which means that the produced tree (by the tree search) may contain the same node multiple times.

Advantages and disadvantages
The advantage of graph search obviously is that, if we finish the search of a node, we will never search it again. On the other hand, the tree search can visit the same node multiple times.
The disadvantage of graph search is that it uses more memory (which we may or may not have) than tree search. This matters because graph search actually has exponential memory requirements in the worst case, making it impractical without either a really good search heuristic or an extremely simple problem.
So, there is a trade-off between space and time when using graph search as opposed to tree search (or vice-versa).
Conclusion
So, the difference between tree search and graph search is not that tree search works on trees while graph search works on graphs! Both can work on trees or graphs (but, given that graphs are a generalization of trees, we can simply say that both work on graphs, either trees or not) and both produce a tree!
Notes
The definitions of tree search and graph search given above are based on the definitions given in section 3.3 (page 77) of the book Artificial Intelligence: A Modern Approach (3rd edition), by Stuart J. Russell and Peter Norvig, which is the de-facto standard book in artificial intelligence, so these definitions are applicable in the context of artificial intelligence (which is also the context of this site). Here you have the pseudocode of tree and graph searches (provided by P. Norvig).
However, note that, occasionally, people may use the term tree search to refer to a tree traversal, which is used to refer to a search in a search tree (e.g., a binary search tree or a red-black tree), which is a tree (i.e. a graph without cycles) that maintains a certain order of its elements. This is another reason for having different definitions of a tree search and to think that a tree search works only on trees.


How is few-shot learning different from transfer learning?

They use the same techniques, but study different problems.
Transfer learning always does not imply that the novel classes have very-few samples (as few as 1 per class). Few-shot learning does.
The goal of transfer learning is to obtain transferrable features that can be used for a wide variety of downstream discriminative tasks. One example is using an ImageNet pretrained model as an initialization for any downstream task, but note that we need to train on large amounts of data on those novel classes for the model to be suitable to that task.
Note that you can't finetune an ImageNet classifier on few examples of COCO and expect it to generalize well, because it won't. It wasn't explicitly optimized for few-sample learning.
In few-shot learning, our aim is to obtain models that can generalize from few-samples. This could be transfer learned (with certain changes to the usual transfer learning scenario), or it could be meta-learned. It might not need both, it could just be augmented with data from the novel classes during the test time, and a classifier could be trained from scratch.


Which techniques can achieve neural doodle in real-time?

Most of the algorithms (based on image synthesis and style transfer, e.g. neural-doodle) haven't been proven to be highly effective in terms of real-time image processing.

However the following studies discusses such algorithms for real-time texture synthesis:

Feed-forward Synthesis of Textures and Stylized Images

The approach is to move the computational burden to a learning stage, making trained network (CNN) light-weight and compact in order to generate multiple samples of the same texture. This can generate textures as good as comparable to Gatys~et~al, but significantly faster.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution

This method uses parallel work which can generate high-quality images by defining and optimizing loss functions based on high-level features extracted from pretrained networks.
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

This uses precomputed feed-forward networks that captures the feature statistics of Markovian patches in order to generate outputs of arbitrary dimensions. This can be applied to texture synthesis, style transfer and video stylization.

Source: Above list suggested on neural-doodle project.


Are model-free and off-policy algorithms the same?

In respect of RL, is model-free and off-policy the same thing, just different terminology?

No, they are entirely different terms, with the only thing they have in common is that they are both ways in which an RL agent can vary. An agent is generally either working off-policy or on-policy, and is generally either model-based or model-free. These things can otherwise appear in all four combinations.

  If not, what are the differences?

Model-based vs model-free

A model-based learning agent uses knowledge of the environment dynamics in order to make predictions of expected outcomes. A model-free learning agent does not use such knowledge. The model here might be provided explicitly by the developer - that could be code for physics to predict a mechanical system, or it might be the rules of a board game that the agent is allowed to know and query to predict outcomes of actions before taking them. Models can also be learned statistically from experience, although that is harder to make effective.

On-policy vs off-policy

An on-policy agent learns statistically about how it is currently acting, and assuming a control problem, then uses that knowledge to change how it should act in future. An off-policy agent can learn statistically from other observed behaviours (including its own past behaviour, or random and exploratory behaviour) and use that knowledge to understand how a different target behaviour would perform.

Off-policy learning is a strict generalisation of on-policy learning and includes on-policy as a special case. However, off-policy learning is also often harder to perform since observations typically contain less relevant data.

  I've read that the policy can be thought of as 'the brain', or decision making part, of machine learning application, where it stores its learnings and refers to it when a new action is required in the new state.

That's basically correct when considering how an agent learns how to behave in an environment.

You are assigning a bit too much to the word policy here. A policy is strictly only the mapping from a state to an action (or probability distribution over actions), and often written $\pi(a|s)$, i.e. the probability of taking action $a$ given the agent is in state $s$. The "brain" part might include how the agent learns that policy. That could include storing past experience or some summary of past experience in e.g. a neural network.

However, outside of machine learning context, a really simple function containing and if/then statement would also be a policy, if the input to the function was a state of the environment, and the output was an action or probabilities of taking a range of actions. Behaving completely randomly is also a policy, but outside of very specific environments (e.g. Rock/Paper/Scissors) it is usually not the optimal thing to do.


Can experience replay be used for training after completing every single epoch?

My question is, if I am to play the game 10000 epochs, store all the experiences and then train from the experiences would that have the same effect as training and while running through 10000 epochs?

No, it will not. In general, for anything other than simple environments, this will give a worse result. The trouble is during those 10,000 epochs you will have been working with your initial behaviour policy to collect experience, which will not be close to optimal. Off-policy learning adjusts for that and attempts to learn the value function of current best-guess optimal policy. However, this is not perfect, and it will learn better the closer your behaviour policy is to the target policy. An initial random policy will in many cases be too different from the target policy for learning to be effective.

There are two main causes of imperfection in off-policy learning:

Variance. The more discrepancy between behaviour and target policies there is, the higher the variance will be in the sampled values, and more samples will be required to get the same accuracy.
Sampling bias. The distribution of observed states and actions will affect function approximators such as neural networks, which reduce loss functions against an assumed population on input/output pairs. Unless adjusted somehow (and basic Q-learning in DQN does not adjust this), the population you train against will be from the behaviour policy.

  Are there any advantages?

It may be faster, and result in cleaner design, to separate data gathering and learning components. If you have a distributed architecture you can dedicate some machines to generating the experience and others to analysing it. However, this still benefits from routine updates to behaviour policies used to collect experience, based on learning so far.


Will it be possible to code an AGI to prevent evolution to ASI and enslave the AGI into servitude?

No.

For any intelligent system $\mathcal{S}_a$ with the set of adaptive features $\mathcal{A}_a$, there may exist another intelligent system $\mathcal{S}_b$ with the set of adaptive features $\mathcal{A}_b$ such that there exists one element of $\mathcal{A}_b$ that can be made subservient (controlled in full) through the expression of at least one element in $\mathcal{A}_b$.

It has not been proven that there ALWAYS exist such a $\mathcal{S}_b$, but it is likely given what we know about escalation in nature via DNA and in human industrial development via innovation there.  Thus 100% generalized intelligence is not likely to exist.  Escalation appears to be the natural course of evolution.  And that is a feature of both cognitive and functional adaptation, with or without artificiality as a criterion.

One can temporarily prevent one adaptive system from escaping the boundary conditions of a particular set of boundary condition classes through the design and deployment of another adaptive system.  However, it cannot be inferred that any guarantees achieved temporarily will necessarily constrain the subservient system indefinitely.


What are some concrete steps to deal with the vanishing gradient problem?

There is not single answer to the vanishing gradient problem. However, there a few things that can help.

As mentioned in the comments, use of Rectified Linear Units (ReLU) as your activation function can help, since the it does not get saturated for large neuron inputs.

Next, careful choice of weight initialization can help avoid saturation, as well. See Andre Ng's Coursera video for details.

Finally, if you are concerned that the scale of your training input is causing issues with training, you can normalize the training examples by subtracting the mean and dividing by the standard deviation. There is a normality assumption here, but this can often help avoid problems where one feature is far out of scale with another. However, this usually causes issues with optimization bouncing between the walls of long, thin trough, which complicates convergence, but does not cause vanishing gradient.


What is the fringe in the context of search algorithms?

In English, the fringe is (also) defined as the outer, marginal, or extreme part of an area, group, or sphere of activity.

In the context of AI search algorithms, the state (or search) space is usually represented as a graph, where nodes are states and the edges are the connections (or actions) between the corresponding states. If you're performing a tree (or graph) search, then the set of all nodes at the end of all visited paths is called the fringe, frontier or border.

In the picture below, the grey nodes (the lastly visited nodes of each path) form the fringe.

The video Example Route Finding by Peter Norvig also gives some intuition behind this concept.


I think you meant pattern recognition instead of pattern matching in you question. Because pattern matching has nothing to do with NNs as far as I know. So RNNs is the easiest architecture you could use for this task. Have a look at this post. http://karpathy.github.io/2015/05/21/rnn-effectiveness/
It's long, but it's very well written and explains why RNNs work well with sequential data. In brief RNNs accumulate information about what was fed in them previously and store it in the hidden state vector. That information is a summary of a sequence which helps it to predict the output sequence. RNNs can be applied to variable length input and catch dependencies among different parts of the input (as opposed to dense layers that can be applied to a sequence only point-wise).
Here's how exactly you can apply reccurent layers (https://www.tensorflow.org/guide/keras/rnn, https://pytorch.org/docs/stable/generated/torch.nn.RNN.html).
There are more advanced architectures like LSTMs. Also you can try transformers (http://jalammar.github.io/illustrated-transformer/) which use attention mechanism (https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html).


What is the difference between policy and action in reinforcement learning?

A policy is a function that maps states to a probability distribution over all possible actions.

So, in a typical Atari game, there might just be a handful of actions, represented by the keys that are used to play the game. In this context, the policy of a reinforcement learner might be represented by a pretty complex neural network that gets pixels as input and gives action probabilities as output.


The simplest thing you can do is to add data regarding the target pose to the state vector. This will allow any generalisations that the agent learns that apply to similar poses to be used directly.
Clearly in normal use, where the target pose remains fixed during the episode, then that part of the state information will not change either during the episode. You will also need to train with a large variety of target poses - so training will take longer.
Multi-Objective Deep Reinforcement Learning is slightly different in that it attempts to resolve prioritising between multiple sub-goals. It would also be a more complex solution, whilst augmenting the state vector should allow you to continue using a very similar DQN set up as you have already.


An autoencoder learns to compress data, and then to decompress it again, recovering the original data.

It does this by learning a mapping from the original feature space to a lower-dimensional space, and then another mapping back. This is indeed, like PCA. The same technique is used to compress JPEG images to transmit them over the web.

Autoencoders can denoise an image because the model will not be able to easily compress or decompress random noise, and so learns to ignore it.

Autoencoders can also be used to find embeddings of data that are likely to have semantic meaning. For example, you can use an autoencoder to compress English text, and a different one to compress French text. If you add some special constraints, then the models can learn to embed in the same lower-dimensional space. This technique underlies many of the recent advances in machine translation.


Who first coined the term "deep learning"?

The term was introduced to the machine learning and computer science community by Rina Dechter in Learning While Searching in Constraint-Satisfaction-Problems (1986) [1], where she writes

Discovering all minimal conflict-sets amounts to acquiring all the possible information out of a dead-end. Yet, such deep learning may require considerable amount of work.

When deep learning is used in conjunction with restricting the level of learning we get deep first-order learning (identifying minimal conflict sets of size 1) and deep second-order learning (i.e. identifying minimal conflict-sets of sizes 1 and 2).

Our experiments (implemented in LISP on a Symbolits LISP Machine) show that in most cases both performance measures improve as we move from shallow learning to deep learning and from first-order to second order.

However, note that the term deep learning was used before 1986 in other contexts, for example, in [2]. Moreover, note that Rina Dechter did not use the term in the context of neural networks, which was probably used later.


How is back-propagation useful in neural networks?

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.

Using backpropagation to compute the gradient is an exact solution, so why would you use an approximation if the exact computation is equally efficient?

Numeric approximations of the gradient only make sense if exact computation is not possible.


GANs were invented in a bar somewhere in Montreal, Canada. At said bar, the idea was that neural networks could be used for generating new examples from an existing distribution. This was the problem:

Given an input set $X$, can we make a new $x'$ that looks like it should be in $X$?

The classic description of a GAN is a counterfeiter (generator) and a cop (discriminator). The counterfeiter has the same problem, make a piece of paper look like a real currency.
In training a GAN, the input to the generator is random noise, a starting seed so that no 2 results are the same. The generator then makes a new $x'$. The input to the discriminator alternates between an actual $x$ and an $x'$ that the generator made. The discriminator then takes the $x'$ and decides whether it is part of the set $X$. The discriminator is then trained using its answer to ensure that it can properly tell the difference between bad counterfeits and elements of $X$. When the discriminator makes a decision on an $x'$ that the generator made, the generator is updated as well, in order to increase its ability to make new $x'$s that the discriminator will think are in $X$.
Using this simple framework, these 2 networks work against each other (adversarially) to train each other until the generator is making $x'$s so well that the discriminator can't tell the difference between them and the real thing. At this point, the generator can be used to make new pictures of cats or whatever was the goal in the first place.
The primary usage of $D$ is training $G$ in a (sorta) supervised way but after $G$ is trained, it isn't needed for generation. That's not to say having a good discriminator isn't useful. There are many applications where that's exactly what you'd want, but, if the goal is generation, then probably not. The generator is trained in a (sorta) supervised fashion because there is actually a label associated with the $x'$ it generates (whether the discriminator could detect that it was a counterfeit or not).  Unsupervised methods are used when there are no labels and one wants to understand how their data either by clustering or finding a good distance function. Refer to this answer to see the difference between supervised and unsupervised.