I'm going to assume here that you're using the standard, basic, simple variant of $Q$-learning that can be described as tabular $Q$-learning, where all of your state-action pairs for which you're learning $Q(s, a)$ values are represented in a tabular fashion. For example, if you have 4 actions, your $Q(s, a)$ values are likely represented by 4 matrices (corresponding to the 4 actions), where every matrix has the same dimensionality as your maze (I'm assuming that your maze is a grid of discrete cells here).

With such an approach, you are learning $Q$ values separately for every single individual state (+ action). Such learned values will always only be valid for one particular maze (the one you have been training in), as you seem to have already noticed. This is a direct consequence of the fact that you're learning individual values for specific state-action pairs. The things you are learning ($Q$ values) can therefore not directly be transferred to a different maze; those particular states from the first maze do not even exist in the second maze!

Better results may be achievable with different state representations. For example, instead of representing states by their coordinates in a grid (as you would likely do with a tabular approach), you'd want to describe states in a more general way. For example, a state could be described by features such as:

Is there a wall right in front of me?
Is there a wall immediately to my right?
Is there a wall immediately to my left?

An alternative that also may actually be able to better generalize to some extent could be pixel-based inputs if you have images (a top-down image or even a first-person-view).

When states are represented by such features, you can no longer use the tabular RL algorithms that you are likely familiar with though. You'd have to use function approximation instead. With good state-representations, those techniques may have a chance of generalizing. You'd probably want to make sure to actually also use a variety of different mazes during the training process though, otherwise they'd likely still overfit to only a single maze used in training.


Should image augmentation be applied before or after image resizing?

As it is told in PIL documentation
It uses some filters to resize images.And those filters are explained here
uses mostly numerical methods as I see. So it is approximating the image data input. Which means you are right about data loss. But here might be the question would it change the data so much If it is done after augmentation or before?
Since in numerical methodic approaches the more values means more valid approximation in general.
So it might be beneficial to augment and then resize.
But one should look for its real answer. Mine is just a thought experiment. You will get better proven answers


How can genetic programming be used to prevent cyberterrorism?

They treated it as a classification problem. While it's common to use some variety of Neural Nets (NNs) to build classifiers, Genetic Programming (GP) can also be used for this purpose. In contrast to NN classifiers, GP can use a wider range of operations (e.g. if,while,logical statements,arbitrary mathematical functions etc) to perform the classification than weighted arithmetic expressions involving an activation function. Whether or not this is actually of benefit depends on the specific application.

In addition, the abstract implies their algorithm is adaptive (i.e. responds in some fashion to the nature of incoming attacks), which would most easily be achieved by continuing to run the GP program in the background to monitor potential intrusions.


Can Machine Learning make economic decisions of human quality or better?

Consider managing a memory structure as an economic function.  (Where to put, and how to manage, the resources constituted by data.)  This is something computers can do better and faster than any human.  The reason is that the system in which the economic decisions are being made is fully defined.

Routing of packages is a similar, economic function that computers do much better than humans.

These functions haven't been handled by Machine Learning in the past, but, soon after the AlphaGo milestone, Google found an economic application for Machine Learning.  Google's DeepMind trains AI to cut its energy bills by 40% (Wired)

So it's entirely context dependent.

As the model increases in complexity and nuanced, utility will be reduced. (In the former case it's a time and space issue related to computational complexity, and in the latter case, often a function of incomplete information or inability to define parameters.)

But as the sophistication of the machine learning algorithms increases, and the models continue to be refined, the algorithms will get better and better at managing intractability and incomplete information.


How does back-propagation through time work for optimizing the weights of a bidirectional RNN?

I have not implement the backprop of a bi-directional RNN from scratch so I can't be sure my answer is correct but I hope it helps.
You can see how bi-directional RNN works from this video from Andrew NG. I got the image below from that video:

For more clarity:

So if you know how to backprop through a simple RNN, you should be able to do so for bi-directional RNN.
If you need more detail, let me know.


How to detect LEGO bricks by using a deep learning approach?

So I am assuming that you are trying to detect a lego brick from the image. One idea is that you can use transfer learning. Leveraging a pre-trained machine learning model is called transfer learning. The underlying idea behind transfer learning is that one takes a well-trained model from one dataset or domain, and applies it to a new one.  François Chollet has written a very comprehensive guide to transfer learning (https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)

I admit I took some of this information from Christopher Bonnett's article named Classifying e-commerce products based on images and text.

I also suggest using the Lego brick dataset from Kaggle on this link: https://www.kaggle.com/joosthazelzet/lego-brick-images
It has over 12,700 lego brick images.

If processing power is a problem, you can use Amazon Web Services for cloud computing. It is inexpensive for small scale operations like this.

Of course for the object detection, you can always increase the number of convolution layers. However, if you have too many layers, you should also include residual blocks/residual network. This would allow neural networks even with over a thousand layers to operate effectively. This video should help you understand how residual networks work (https://www.youtube.com/watch?v=ahkBkIGdnWQ)

Finally, make sure not to overfit during your training and if you do follow the residual network idea, you should also include upsampling in your convolution neural network( More in here: https://towardsdatascience.com/up-sampling-with-transposed-convolution-9ae4f2df52d0)

I hope this helped and good luck on your endeavor.


What are the AI technologies currently used to fight the coronavirus pandemic?

According to the Baidu Research's blog post How Baidu is harnessing the power of AI in the battle against coronavirus (12-03-2020), there are already some artificial intelligence tools or algorithms being used to fight the coronavirus.
Given that I cannot confirm that these AI tools and algorithms I will mention are really being used in practice, I will only quote the parts of the blog post that potentially answer my original question.
To give some context, similar to HIV viruses, the virus that is causing the coronavirus pandemic, SARS-CoV-2 is capable of rapidly mutating, making vaccine development and virus analysis difficult.
AI-powered and non-contact infrared sensor system

Baidu has developed several tools that are effective in building awareness and screening populations, including an AI-powered, non-contact infrared sensor system that provides users with fast multi-person temperature monitoring that can quickly detect a person if they are suspected of having a fever, one of the many symptoms of the coronavirus. This technology is currently being used in Beijing's Qinghe Railway Station to identify passengers who are potentially infected where it can examine up to 200 people in one minute without disrupting passenger flow.

AI-powered pneumonia screening and lesion detection system

By leveraging PaddlePaddle and the semantic segmentation toolkit PaddleSeg, LinkingMed has developed an AI-powered pneumonia screening and lesion detection system, putting it into use in the hospital affiliated with XiangNan University in Hunan Province. The system can pinpoint the disease in less than one minute, with a detection accuracy of 92% and a recall rate of 97% on test data sets.

Automated HealthMap system

The Boston Children's Hospital used an automated HealthMap system that scans online news and social media reports for early warning signs of outbreaks, which led to the initial awareness that COVID-19 was spreading outside China.

Autonomous vehicles carry out non-contact tasks

Access to health care and resources at a moment's notice is vital for battling the spread of the coronavirus. Autonomous vehicles are playing a useful role in providing access to necessary commodities for health-care professionals and the public alike by delivering goods in infected areas and disinfecting hospitals, effectively minimizing person-to-person transmission and alleviating the shortage of medical staff.
Apollo, Baidu's autonomous vehicle platform, partnered with a local self-driving startup called Neolix to deliver supplies and food to the Beijing Haidian Hospital.


How to convert sequences of images into state in DQN?

I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN?

As nbro mentioned in a comment to your answer, this question has very recently been asked and answered here.

  I also want to know since $s_1$ is equal to a set, which can be seen in line 4 of the algorithm, then why is $s_{t+1}$ equal to $s_t$, $a_t$, $x_{t+1}$?

The algorithm presented in the original DQN paper is relatively simple and written to express the main ideas of their approach (e.g. experience replay, preprocessing histories, gradient descent, etc.); in fact, it isn't even the exact algorithm used in the experiments! For example, the experiments use frame-skipping to reduce computation - this is not mentioned in Algorithm 1 in the paper. With that in mind, setting $s_{t+1}$ equal to $s_t, a_t, x_{t+1}$ in the algorithm signifies a general notion of constructing the next raw state $s_{t+1}$ from the previous preprocessed state $s_t$, previous action $a_t$, and current frame $x_{t+1}$. For example:

If the action space at the next timestep is constrained by the state, then the state may need additional parameters to encode the action space.
The algorithm needs some indication if a state is terminal, and such an indication may need to be encoded in the state.
If there is frame skipping, then multiple frames will be needed to construct the next state, possibly using the previous state as well.

The above examples should display how the encoding of the state cannot always simply be a stack of raw frames, or even a function of $s_t$, $a_t$ and $x_{t+1}$, and therefore a more general approach is often required.


Is it ethical to implement self-defence for street walking AI robots?

The question mentions "walking robot", but it may be illustrative to re-frame the discussion in terms of self-driving cars, because:

It gives a common point of reference, rather than everyone having their own separate vision of how vulnerable/powerful a kung-fu walking robot might be.
We already know a lot about societal attitudes to car theft.
Given that autonomous vehicles will soon be mainstream, the morality of the question is then more of a pressing issue.

So, should a self-driving car run someone over (likely killing them) if they try to steal it? I'm hoping that few people would argue that it should.

Should it attempt to do a lesser amount of damage (say, calculated to hopefully only break a leg)?

Again, I'd argue not. The main reason for saying this is that our decision-making algorithms are simply not sufficiently context aware to be able to decide whether theft or harm is the intent. To concretely illustrate this: a recent fatality arose because a self-driving Tesla was oblivious to context to the extent that it couldn't distinguish between a high-sided van and empty space.

Under those circumstances, it's probably best not to allow commercial autonomous systems to cause physical damage (even to inanimate objects).

'Running away' (or rather, 'driving away', in the case of the car) is another matter: driving is what it's designed to do.


You need to define a scoring function, which returns a value of whatever criterion you are interested in, be it 'artisticity' or 'heat'. This could be something you use a machine learning algorithm for, provided you have a set of training data with labelled images.

You then need to extract features from the images. In the case of 'heat' this could be a colour spectrum (ie distribution of colours across the image), or whatever. If it's a reasonably small image, pixel values might be feasible. These features you can feed into your algorithm, and try to learn an association between the feature values and the (assigned by you) label of the image. You will end up with a classifier that takes image features as input, and returns the labels as output. The quality of the classifier depends on your images, the features you selected, and the task.  If there is no structure in the data, then a classifier will not work properly.

If you have a continuous value (eg temperature in degrees C) you would run your two sample images through the classifier, and then compare the output values.


How is the loss value calculated in order to compute the gradient?

In your case, $L$ is the loss (or cost) function, which can be, for example, the mean squared error (MSE) or the cross-entropy, depending on the problem you want to solve. Given one training example $(\mathbf{x}_i, y_i) \in D$, where $\mathbf{x}_i \in \mathbb{R}^d$ is the input (for example, an image) and $y_i \in \mathbb{R}$ can either be a label (aka class) or a numerical value, and $D$ is your training dataset, then the MSE is defined as follows

$$L(\mathbf{W}) = \frac{1}{2} \left(f(\mathbf{x}_i) - y_i \right)^2,$$

where $f(\mathbf{x}_i) \in \mathbb{R}$ is the output of the neural network $f$ given the input $\mathbf{x}_i$.

If you have a mini-batch of $M$ training examples $\{(\mathbf{x}_i, y_i) \}_{i=1}^M$, then the loss will be an average of the MSE for each training example. For more info, have a look at this answer https://ai.stackexchange.com/a/11675/2444. The https://ai.stackexchange.com/a/8985/2444 may also be useful.

See the article Loss and Loss Functions for Training Deep Learning Neural Networks for more info regarding different losses used in deep learning and how to choose the appropriate loss for your problem.


What is the motivation behind using a deterministic policy?

You're right! Behaving according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a stochastic behavior policy.

Under some reasonable assumptions--like that the environment is fully-observed and is stationary--an optimal deterministic policy always exists. The proof can be found in chapter 6 of "Markov Decision Process -- Discrete Stochastic Dynamic Programming" by Martin L. Puterman. The same cannot be said for stochastic polices. For this kind of environment (even if it's stochastic) an optimal policy is hardly ever stochastic.

So, a motivation for wanting to learn a deterministic policy is often because we know that there is an optimal deterministic policy.

It's possible your question was also tangentially about off-policy learning. "Why learn a deterministic policy directly (off-policy) when we can just use something like decaying $\epsilon$-greedy?" Briefly, off-policy learning is very powerful and general. It's necessary in any algorithm that uses experience replay, for example. A discussion about the merits of off-policy learning is probably best left to another question, but reading Section 5.5 of Sutton and Barto's RL book should get you started.

Finally, directly learning a deterministic policy could be more computationally efficient if using deterministic policy gradients. In the setting with continuous state and action spaces, the deterministic policy gradient exists and has a simpler expectation than the stochastic policy gradient.

Stochastic policy gradient:

$$\begin{align*}
\nabla_\theta J(\pi_\theta) &= \int_\mathcal{S} \rho^\pi (s) \int_\mathcal{A} \nabla_\theta \pi_\theta (a|s) Q^\pi (s,a) \text{d}a\text{d}s\\
& = \mathbb{E}_{s\sim \rho^\pi, a\sim\pi_\theta}[\nabla_\theta \log \pi_\theta (a|s) Q^\pi (s,a)]
\end{align*}$$

Deterministic policy gradient:
$$\begin{align*}
\nabla_\theta J(\mu_\theta) &= \int_\mathcal{S} \rho^\mu (s) \nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}\text{d}s\\
& = \mathbb{E}_{s\sim \rho^\mu}[\nabla_\theta \mu_\theta (s) \nabla_a Q^\mu (s,a)|_{a=\mu_\theta (s)}]
\end{align*}$$

Notice that the expectation in the deterministic policy gradient isn't over the action space. Estimating this expectation would require many fewer samples in the setting of a continuous, high-dimensional action space.

To recap:

Optimal policies are often deterministic, not stochastic
Learning deterministic policies directly (off-policy) is powerful and general
It can also be more efficient if in a continuous, high-dimensional action space


How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words.

lets assume two words $e_i$ and $e_j$ pop up in the exact same context of some word $e_k$ with 3 other context words as well. An example would be:

He leaped over the truck
He jumped over the truck

Where the italics represent the words with similar meanings but the bolded words above are the ones being predicted/skipped in CBOW. Lets now show this.

Let the rest of the context be denoted $\{e_r\}_r$ and the skipped word as $e_s$ and so the loss will try to minimize both $-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)]))$ and $-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))$.

Assuming a long run withlarge enough batch-size (so ignoring catastrophic forgetting) we can essentially say it will be minimizing
$$-e_s^T [log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)])) +\\ log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))]$$
Now let $\mathcal{U}\mathcal{V}\frac{1}{R+1}\sum_re_r$ be denoted as $c$, $\mathcal{U}\mathcal{V}\frac{1}{R+1}e_i$ as $\hat u_i$, and $\mathcal{U}\mathcal{V}\frac{1}{R+1}e_j$ as $\hat u_j$. So we have to minimize
$$-e_s^T[log(softmax(\hat u_i+c)) + log(softmax(\hat u_j+c))]$$
or equivalently
$$-e_s^T\ log(softmax(\hat u_i+c) * softmax(\hat u_j+c))$$
As we know in crossentropy, we get our critical point when the inside of the log here would equal $e_s$, which is only possible if both $softmax(\hat u_i+c)$ and $softmax(\hat u_j+c)$ were to be equal to $e_s$. softmax is non-invertible and is invariant under addition only, therefore to achieve this we would need $\hat u_i + c$ to equal $\hat u_j + c + K$ where $K$ is just some arbitrary constant vector ($K=[k,k,k,...]$. Subtracting $c$ and multiplying by $R+1$ we make our model want

$$\begin{align}
u_i &= u_j + K \\
\implies K &= \mathcal{U}\mathcal{V}(e_i -e_j) \\
\end{align}$$

This would mean for all common words, we see this activity that $u_{i-j}$ would be a constant, but this is difficult because that means a $\delta$ must exist such that $\mathcal{U}\delta = k*\vec 1$ and that all similar word pairs indexed by $(a,b)$ would have $v_a \approx v_b + const*\delta$ (because $u_a = u_b + K \implies v_b = v_a + \mathcal{U}^{-1}K$ and we denote $\delta = \mathcal{U}^{-1}K$). This constraint is heavy, tough to learn, and would also be difficult to process on multiple word senses that would appear in the vocabulary. This is because if it were to learn this, it enforces constraints on the images of $\mathcal{U}$ and $\mathcal{V}$ along with creating word vectors with highly different magnitudes making the learning process difficult. This would indicate that this constant would be low in practice. Therefore we would get $v_i \approx v_j$.


How to classify language as friendly or aggressive with AI?

I did a little search and couldn't find any database that has ground truth for aggressiveness. This means that you need to build yourself a database. This might be huge undertaking. Take thousands of messages, and classify them by hand whether they are aggressive or not. This part is quite labor intensive.

Second part is much easier at start but would be pain to optimize (both performance and computational cost). I would suggest you to start with Naive Bayes classifier for this job. That is the preferred classifier for spam detection. ANN would probably not work for this case because the data would be a huge sparse vector. Estimated number of words in English is over a million, which means the input layer of your ANN should be able to scale up to that number. Search for sparse vector classification for additional classifier that can be used in these cases.


Is it possible to build an AGI with neural networks on neuromorphic chips?

No. The reasons include but are not limited to:

lack of understanding of how the brain works

current ANNs are mostly good at pattern recognition and generative   tasks, but lack the capacity to create abstractions on their own

we can't match the size/number of artificial neurons to the number of biological neurons

even with a much smaller ANN size, performance is an issue (i.e. state of the art image categorization ANNs have to be trained few weeks on multi GPU rigs to match human level).


Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions.

As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original cost formula B is clearly better, with 4 total cost compared with A's cost of 12 - A just has one low cost at the beginning, which I put in deliberately as it exposes the problem with your conversion.

In your two reward formulae:

reward = 1/(cost+1)**2.
  A: 1.0 + 0.04 + 0.04 + 0.04 = 1.12
  B: 0.25 + 0.25 + 0.25 + 0.25 = 1.0

reward = 1/(cost+1).
  A: 1.0 + 0.2 + 0.2 + 0.2 = 1.6
  B: 0.5 + 0.5 + 0.5 + 0.5 = 2.0

So with this example (numbers carefully chosen), maximising the total reward favours A for sum of inverse squares but B for sum of inverses, whilst B should be the clear preference for minimising sum of costs. It is possible to find examples for both of your formulae where the best sum of rewards does not give you the lowest cost.

In your case, if you truly want to minimise total cost, then your conversion to rewards should be:

reward = -cost

Anything else is technically changing the nature of the problem, and will result in different solutions that may not be optimal with respect to your initial goal.


Is there a way to run other platforms (other than Atari) in an OpenAI's Gym-like environment?

Ok, I found

https://blog.openai.com/gym-retro/

but there may be other platforms?


How to create a Q-Learning agent when we have a matrix as an action space?

I was thinking this strategy may work.
So, Q-learning takes vector input as state representation let's say your vector has n dimensions i.e. [$n_0$, $n_1$, $n_2$,..., $n_{n-1}$]
Now, from my interpretation you want to populate a matrix with 0 and 1's given the state vector but action-space has a high complexity e.g. an 8*8 matrix has 64 cells i.e. $2^{64}$ possible actions if you want the action to be a matrix.
I suggest this:
Fill each cell, one at a time. i.e. Your agent has only two possible action 0 and 1.
To indicate to your agent that you are at a specific cell, concatenate the row and column number to the state vector before passing it as input to the Q-learning agent.
Example:
If you original state-vector is [55, 22, 100, 4] and you have to fill cell at position (10, 30) of the matrix, the state-vector should be modified as follows:
[55, 22, 100, 4, 10, 30].
I'm not sure of the sample-efficiency of this approach though.


Is my interpretation of the return correct?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side is correct though.

It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all future rewards.

For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.

The decision to have reward time step match that of next state is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.


How to make DNN learn multiplication/division?

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you only need one hidden layer to approximate the multiplication on a compact, note that you need to apply a non linear activation on the hidden layer to do this.


Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.

It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network).

Even with one input neuron it is not possible. That is because even with one input, the output can be an arbitrarily complex mapping to classes. A good example with two inputs that would require an infinite number of hidden neurons to supply a simple classifier would be classifying x,y points as being in the Mandelbrot set.

In some, more constrained, examples, with well-defined functions, you can construct a minimal neural network that solves the problem perfectly. For instance a neural network model of XOR can be made with two hidden neurons (and six links). However, this kind of analysis is limited to simple problems. You might be able to come up with some variation of your idea if all inputs were boolean, and the neural network limited to some combined bitwise logic on all the inputs.

Your idea of matching number of edges to number of possible interactions between inputs does not work because you are only considering the most basic kind of interaction between two variables, whilst variables can in practice combine in all sorts of ways to form a function.

In addition, each neuron in a hidden layer works with a linear weighted sum, plus a fixed transformation function. This is in no way guaranteed to match the function shape that you are trying to approximate with the neural network. An analogy that you might be aware of is discrete Fourier transforms - it is possible to model any part of a function by combine sine and cosine waves of different frequencies, but some functions will require many such waves in order to be represented accurately.

Your link to the answer in Cross Validated Stack Exchange gives you a rule of thumb that the writers find often works with the kinds of data that they work with. This is useful experience. You can use such rules as the starting point for searching for architecture that works on your problem. This will likely be more useful than your idea based on counting the possible variable interactions. However, in both cases, the most important step is to perform a test with some unseen examples, and to search for the best neural network architecture for your problem.

There are things you can do with variable interactions though. For instance, try looking for linear correlations between simple polynomial combinations of variables and your target variable, e.g. plot $x_1 x_2$ vs $y$ or $x_3^2 x_4$ vs $y$ . . . you may find some combinations have a clear signal implying a relationship. Take care if you do this sort of thing though, if you test very many of these, you will find a linear relationship purely by chance that looks good initially but turns out to be a dud when testing (it's a form of overfitting). So you should generally test a lot less than the size of your dataset, and limit yourself to some modest maximum total power.


Can PHP be considered as a serious programming language for AI?

Question on-topicness questionable, but...

The most logical reason why PHP is unsuited for neural networks is that PHP is, well, intended to be used for server side webpages. It can connect to various external resources, such as databases, via native language features. It is very much a glue language, and not a processing language. PHP is also mostly stateless, only allowing you to store state in either clients, file storage or databases.

As such, it's not suitable for this sort of thing - not because PHP is a high level language, but rather because it's so request based and focused towards creating pages to serve to clients.

That won't stop people from trying, though - there are various esoteric programming languages out there in which regular programming would be an insane task or not possible at all - but from a ease of development perspective, making a neural network in PHP makes no sense.


Is Lisp still worth learning today in the particular context of Machine learning?

LISP was popular because back in the old days of AI because of the functional syntax, which worked well with the GOFAI paradigm of the time.

Nowadays most researchers have given up on the classical computational theory of mind (read: language of thought), and thus also the GOFAI paradigm that it associates with.

LISP is not what you want to learn if you want to do neural network stuff, but the philosophical background is still important to know.


How do RNN's for sentiment classification deal with different sentence lengths?

One of the essential pre-processing we do on the corpus involves treating the variable-length sentences to a fixed length. There are various ways in which we can do this:
Truncate

This involves reducing the length of all the sentences to the length of the shortest sentence in the corpus. This is generally not done as it reduces the amount of information that we can learn from the corpus. This image shows pre sequence truncation, where we remove from the back to make the sentences of the same length.

Padding

This is the most preferred method when it comes to handling the problem of variable length sentences. In this approach, we increase the size of each vector to the longest sentence in the corpus. There are two ways to this:

Post-Padding: Adding zeroes in the ending
Pre-Padding: Adding zeroes in the beginning

References

Effect of Padding on LSTMs and CNNs by Dwarampudi Mahidhar Reddy and N V Subba Reddy, et al.


Can we use a neural network that is trained using Reinforcement Learning for dynamic game level difficulty designing in realtime?

There have been many researches in dynamic difficulty adjustment (DDA). I see this one is quite explaining: AI for Dynamic Difficulty Adjustment in Games. However, there are many factors when we are trying to do dynamic difficulty adjustment. As explained in paper above, one major problem is it is sometimes hard to make sure the created model will still deliver the intended experience.

You can also read this paper about game design and DDA on ace05The Case for Dynamic Difficulty Adjustment in Games .

So for short, you can use neural network -- or other learning methods -- to do dynamic difficulty adjustment. But it's more about game design and experience impact of your DDA.


Understanding the configuration of replay memory and epsilon in deep reinforcement learning

Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.

Epsilon is a parameter that governs the trade-off between exploration and exploitation. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (exploitation), or try something new (exploration). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.

You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
In the example that you posted, you can see that Pacman starts off doing nothing but exploring, but after 500000 games it exploits 95% of the time.


How can AI techniques be used in software testing?

In large software with many actions and possible flows like web applications, enterprise software, etc, it is really hard and time taking to test out all possible scenarios via traditional approach. So, building a machine learning model is an interesting approach to solving this. A reinforcement learning system with an end goal to crash or make the software unresponsive can be tried.

There is research being done on this idea. You can take read this research paper which explores Reinforcement Learning as an approach to automated GUI robustness testing. Also, some companies like Appdiff trying to incorporate AI in software testing with mobile apps in context, but similar thinking can be reasonably extended to web apps.


How does one make it obvious that the structure of a neural network should be what it is?

While, as you begin to hit on, there are general guidelines to follow when building a neural network, they are far from standardized. This is because even though AI is a reasonably old field(1950s), neural networks have only been the tool of choice for less than a decade. Before, NNs did horribly, due to lack of data, and computation along with some less than efficient architectures.

With that being said Hinton's general rule is to add nodes/layers till the model begins to overfit and then add dropout(pretty reasonable in practice).

As such, the whole field is essentially an art as much as a science currently with only basic guidelines to follow based on your problem and data. This is part of the beauty though imo, with there being so much left to discover.

Hope that helped answer your question!


Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

Your problem is that TFIDF is cutting out around 90 terms.

Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.

There are two things I'd try out if I were you:

Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:

wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)


How to train a reinforcement learning agent from raw pixels?

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning values?

A Convolutional Neural Network (CNN) structure is a standard neural network architecture when working with 2D pixel input in reinforcement learning, and it is the technique used in the original DQN paper (see paragraphs 1 & 3 of Section 4.1 of https://arxiv.org/abs/1312.5602). CNNs typically take 3-dimensional input, where the first two dimensions are height and width of your images, and the third is rgb color. The technique in the paper was to convert each RGB frame (or image) to greyscale format (so it has only 1 color channel/dimension instead of 3) and instead use the rgb_color dimension as a frames dimension that is indexed by each stacked frame.

  Currently, I am watching a YouTuber: Machine Learning with Phil, and he did it very differently. On the 13th minute, he defined a network that outputs a batch of values rather than Q-values for 6 states. In short, he outputs a matrix rather than a vector.

Later in the tutorial series, he most likely will discuss the training of the neural network. During training, you need to find the q-values of a batch of sets of stacked frames. Specifically, each element of the batch is a set of stacked frames. In other words, a set of stacked frames is treated as a single observation, so a batch of sets of stacked frames is a batch of observations.

To find these q-values, you will perform a forward pass of the batch of observations through the neural network. A forward pass of a single observation (set of stacked frames) through the neural network yields a vector of q-values (one for each action). Thus, a forward pass of a batch of observations (batch of stacked frames) will yield a matrix of q-values (one vector of q-values for each observation (or set of stacked frames)). This technique is used because many standard neural network libraries are designed to perform a forward pass on a batch of inputs through the neural network much faster than performing a forward pass on each input separately.


What are the pros and cons of using a spatial transformation network to predict the next video frame?

Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence.


How is the expected value in the loss function of DQN approximated?

Just as the paper says
$$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$$
where
$$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$$
Then in the Background section of the paper, it says

Differentiating the loss function with respect to the weights we arrive at the following gradient:

$$\nabla_{\theta_i} L_i(\theta_i)\\= E_{(s,a)\sim p,s'\sim\mathcal{E}}\left[\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)\right]\tag{1}$$

Rather than computing the full expectations in the above gradient, it is often computationally expedient to optimize the loss function by stochastic gradient descent.
...
and the expectations are replaced by single samples from the behavior distribution $ρ$ and the emulator $\mathcal{E}$ respectively.

If you're familiar with SGD and Stochastic Optimization then you know what happens here:
The expression inside the expectation of (1), i.e. $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ , is an unbiased estimation of the real gradient $\nabla_{\theta_i}L_i(\theta_i)$ - its expectation is the real gradient. In other words,
$$\widehat{\nabla L}=\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i).$$
Then by theory of Stochastic Optimization we can optimize $L$ by $\theta\leftarrow \theta - \alpha\widehat{\nabla L}$ , which is how SGD works.
The unbiased estimation $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ can be sampled and calculated directly - you can run the emulator $\mathcal{E}$ and current behavior policy to collect $r, s', a'$ and calculate the gradient of your Q network $\nabla Q$ using TensorFlow. So the gradient to $L$ can be approximated.
The $y_j$ in the pseudocode  of the paper is also an estimation. It'll be more proper to denote it as $\hat{y_j}$.
(I'm not an native English speaker so forgive my poor expression.)


How is the log-derivative trick of a trajectory derived?

The identity $$\nabla_{\theta} P(\tau \mid \theta) = P(\tau \mid \theta) \nabla_{\theta} \log P(\tau \mid \theta)\tag{1}\label{1},$$

which can also be written as

\begin{align}
\nabla_{\theta} \log P(\tau \mid \theta)
&=
\frac{\nabla_{\theta} P(\tau \mid \theta)}{P(\tau \mid \theta)}\\
&=\frac{1}{P(\tau \mid \theta)} \nabla_{\theta} P(\tau \mid \theta)
\end{align}

directly comes from the general rule to derive the logarithm of a function and the chain rule
\begin{align}
\frac{d \log f(x)}{d x}
&= \frac{1}{f(x)} \frac{d f}{dx}.
\end{align}
Note that $\log f(x)$ is a composite function and that's why we apply the chain rule and that the derivative of $\log x = \frac{1}{x}$, as your text says.

People shouldn't call this a trick. There's no trick here. It's just basic calculus.

Why do you need identity \ref{1}? Because that identity tells you that the derivative of the probability of the trajectory given the parameter $\theta$ with respect to $\theta$ is $P(\tau \mid \theta)$ times the gradient of the logarithm of that same probability. How is this useful? Because the logarithm will turn your product into a sum (and the derivative of a sum is the sum of the derivatives of the elements of the sum), Essentially, the identity \ref{1} will help you to compute the gradient is an easier way (at least, conceptually).


Intel Movidius Neural Stick vs TensorFlow

The Intel Movidius Stick is an USB stick for running a neural networks (that are programmed in Tensor Flow) on resource scare devices such as a Raspberry Pi (in order to reduce the traffic by sending data to cloud service or a server with more capacity where your neural network runs).

Tensor Flow is a programming framework which can be used as python library for the implementation of neural networks.

You are comparing an execution environment with an implementation language. The accuracy should not depend on them, instead I would considering on your model architecture (type of cnn) and your training as the most influential factors on accuracy of your test data.


Isn't evolutionary theory the essence of intelligence after all?

The theory of evolution seems to be intelligent as it creates life

When you say "seems to be intelligent" that begs the question: How are you defining "intelligent"? Which of course is still one of the big issues in AI research.
I think there are some flaws with the argument that "creates life" = "intelligent":

Evolution does not create life. It operates on entities where there is a copy mechanism which is not 100% reliable, plus a selective environment that also impacts likelihood of further copies being made. Some form of proto-life (an initial Darwinian ancestor or Ida) needed to exist before evolution started.

The process of creating the first proto-life capable of undergoing evolution is generally thought to be a large semi-random search through chemical combinations. Random search is sometimes used in optimisation problems, and might be studied as part of AI search topics. However, it would normally be considered something of a baseline algorithm, and definitley not tick the boxes for all the general traits of intelligence.

Isn't this evolutionary mechanism itself the same as the essence of human intelligence?

The Wikipedia article on artificial intellegence lists challenges faced by researchers and developers in AI. The categories chosen there are:

Reasoning, problem solving; Knowledge representation; Planning; Learning; Natural language processing; Perception; Motion and manipulation; Social intelligence; General intelligence

Together, these are mainly traits of mammalian, avian and a few other multicellular species, with a few traits such as language heavily focused on humans.
I think it is important to separate out the mechanism whereby these traits arose naturally - which is generally agreed to be via an evolutionary process - from how those traits function. Artifical intelligence may use a little bit of reverse engineering from the natural traits in order to inspire design, but most AI systems do not use theory of evolution directly.
When used directly, evolutionary algorithms can be used to solve search and optimisation problems. Also they can be used to solve simplified problems in perception and motion/manipulation. However, we are not able to scale up such algorithms to solve all aspects of general intelligence. Instead, systems like machine learning are designed to work from analysis of the problem, inspired in part by working natural systems. These work far more efficiently than evolutionary algorithms. There are no competitive evolutionary variants of AlphaZero, Watson, GPT-3 or neural-networks used in image processing.
Evolutionary algorithms have their place in AI in practice and research. However, they do not define or encapsulate a form of general intelligence.


What is the neuron-level math behind backpropagation for a neural network?

Backpropagation is actually a lot easier than it is made out to be - if you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.

This is hands down the best walk through of back prop I've found on the internet. If you are still confused after that, feel free to ask me any further questions. Here is also a quick forward and backward pass example I made for a simple CNN (only a few layers though, and the gradient only goes back to channel 1 of filter 1)


How do I take the correct classification predictions of an ml algo (i.e. random forest/neural net) and sort the instances in each category?

For the ANN, it should be the average of the error per instance from testing (prediction) when each instance is left out of training.  ANNs can unfortunately learn based on the order of instances used for training, so it helps to train/test and then shuffle (permute, or randomly re-order) and then assign to k-folds, then train/test again in order to prevent the ANN from learning based on instance order.  The RF won't do this, since the in-bag objects in all the bootstraps (trees) that were sampled with replacement will vary appreciably.
(FYI - there's commonly an option to train using "batch" or "online" error updates during back-propagation.  Batch sums the error over the training objects, then updates coeffs.   Online updates coeffs as each training instance is processed.  I tend to like online, since online seems to reduce error faster.  Batch can get off the gradient path easier, since the summed error can come in large "chunks" and throw the ANN off course)
As far as sorting instances within each class label after training/testing, you can do anything you wish.  Certainly tracking each instance as it is tested/predicted and building an array/vector of errors will give you a distribution of error per instance.
Quite often, test objects can widely very in their feature values, so e.g. k-means cluster analysis can be used during ANN training.  See my blog on this.
In short, it's not just sorting when done.  Ensembles are on the back end of an attack plan, but the checklist for best practices would look like:

Randomly shuffle the order of all instances.
Assign instances to $k$-folds (partitions), e.g. 10-fold.
Train with instances in folds 1-9, test (predict class) of instances
in fold 10.  Increment confusion matrix.  Then train with instances
in folds 1-8,10 and test instances in fold 9, update confusion
matrix, ...., cycle through all test folds.
Repeat steps 1-3 nine more times (this is called doing another "re-partition").
Calculate average error for each predicted instance, then sort.

The above steps are only for the ANN.  The RF performs its own bookkeeping.


Is Artificial Intelligence restricted to electrical based technology?

Not yet. Synthetic virology / Synthetic life are still in their infancy.
We can now synthesize simple bacteria (see Craig Venter's fascinating TED talk and also an article about his recent work) but definitely nothing that may be called 'rational' in human standards.


What are the advantages of having self-driving cars?

One of the main arguments for self-driving cars is that presumably they'll get better and better at driving as the technology progresses, they have no temporal attention deficits or aggressive urges or drug habits and sense their environment 360°, all the while communicating with the other cars, which all together basically amounts to LESS DEAD PEOPLE.

We are really interested in this.

It is also unclear whether most people will actually own cars in 30 years. Maybe there'll be a net of mini busses with flexible routes which take you from door to door on demand. That would reduce traffic quite a bit and there would also be less incentive to drive 200 m to get cigarettes or something. Self-driving cars would allow us to use the car as a resource a lot more efficiently, because suddenly we can relocate empty cars without paying a driver.


Can neural networks be used to prove conjectures?

Your idea may be feasible in general, but a neural network is probably the wrong high level tool to use to explore this problem.

A neural network's strength is in finding internal representations that allow for a highly nonlinear solution when mapping inputs to outputs. When we train a neural network, those mappings are learned statistically through repetition of examples. This tends to produce models that interpolate well when given data similar to training set, but that extrapolate badly.

Neural network models also lack context, such that if you used a generative model (e.g. an RNN trained on sequences that create valid or interesting proof) then it can easily produce statistically pleasing but meaningless rubbish.

What you will need is some organising principle that allows you to explore and confirm proofs in a combinatorial fashion. In fact something like your idea has already been done more than once, but I am not able to find a reference currently.

None of this stops you using a neural network within an AI that searches for proofs. There may be places within a maths AI where you need a good heuristic to guide searches for instance - e.g. in context X is sub-proof Y likely to be interesting or relevant. Assessing a likelihood score is something that a neural network can do as part of a broader AI scheme. That's similar to how neural networks are combined with reinforcement learning.

It may be possible to build your idea entirety out of neural networks in principle. After all, there are good reasons to suspect human reasoning works similarly using biological neurons (not proven that artificial ones can match this either way). However, the architecture of such a system is beyond any modern NN design or training setup. It definitely will not be a matter of just adding enough layers then feeding in data.


Why are RNNs used in some computer vision problems?

There are tasks in computer vision where recurrent neural networks (RNNs) can be useful because there's some sequential sub-task in the main task.
For instance, in the paper Long-Term Recurrent Convolutional Networks for Visual Recognition and Description, the authors investigate the use of a neural network that is both recurrent and convolutional to solve certain computer vision tasks that also have a sequential component/part, such as video recognition tasks, image to sentence generation problems, and video narration challenges.
There are other papers that investigate the combination of convolutional and recurrent layers, such as Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition, which also has a biological motivation.


Why does our AI play worse at even levels of depth?

When the number of levels is odd, it means the first player can do one more extra movement on the board. As it is an extensive form game, when decide using backward induction, as the last and the first move are for the first player, so the first player can act better than the situation that the second player will make the last move.


What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

Don’t think about it as the $w_{(n)}-w_{(n+1)}$ being proportional to something. Think about it this way:
I'm now at $w_{(n)}$. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of $w_{(n)}$.
If $E$ increases as I increase $w$ (that is, if $\frac{\partial E}{\partial w}>0$, then obviously, I would want to move a little bit to the left. In other words, $w_{n+1}<w_{n}$ or $w_{n+1}-w_{n}<0$.
On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, $w_{n+1}-w_{n}>0$.  So, basically, your step should have the opposite sign of the derivative.
$$w_{n+1}-w_{n} \propto-\frac{\partial E}{\partial w}$$
$\alpha$, the learning rate, is just the constant of proportionality.
Caution: think about small values for this rate, not big numbers.  Taking a huge step can cause you to overshoot the minimum point.


If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

The output layer of the network contains one unit, telling me the Q value of the provided state with the assumption that the action taken in that state will be determined by the policy.

Typically in Reinforcement Learning, the symbol $Q$ is used when you calculate an action value, and if you are evaluating for a specific policy, it is noted $q_{\pi}(s,a)$ where $\pi$ is the policy, $s$ is the current state, and $a$ the action to be taken.

What you appear to be calculating with your network is not the action value, but the state value, $v_{\pi}(s)$. Note that $v_{\pi}(s) = q_{\pi}(s,\pi(s))$ when you have a deterministic policy, and there is a similar relationship for stochastic policies.

With your setup, it would be possible to learn the value of any fixed policy that you provided as input. It is also possible to drive policy improvements, with some caveats, but that would not be Q learning. The rest of this answer assumes you want to implement Q learning, and probably something like Deep Q Networks (DQN).

  However, I don't know what policy would make the agent learn to play the best. I can't just make it choose random actions because that would make the policy non-stationary.

Actually that would make the policy stochastic. Non-stationary means that the policy would change over time. Choosing random actions is a perfectly valid stationary policy (provided the probabilities in each state remain the same).

In addition, in an optimal control scenario you actually will have and want a non-stationary policy. The goal is to start with some poor guess at a policy, like completely random behaviour, and improve it based on experience. The policy is going to change over time. That makes learning Q values in RL a non-stationary problem.

Typically you really do start control problems with a random policy. At least in safe environments such as games and simulations.

  I'm trying to train an agent to play it at a reasonable skill level using TFLearn.  What can I do?

First, modify your network so that it estimates action values $Q(s,a)$. There are two ways to do that:

Take the action $a$ as an input, e.g. one hot encoded, concatenated with the state $s$ to make the complete inputs to the neural network. This is simplest approach conceptually.
Assign a number in range $[0,N_{actions})$ to each action. Change the network to output a value for each possible action, so that your estimate $\hat{q}(s,a)$ is obtained by looking at output indexed by $a$. This is a common choice because it is more efficient for selecting best actions to drive the policy later.

Using action values for $Q(s,a)$, not state values $V(s)$, is an important part of Q learning. Technically it is possible to use state values if you have a model of the environment, and you could add one here. But it would not really be Q learning in that case, but some other related version of Temporal Difference (TD) learning.

Once you start working with action values, then you have a way to determine a policy. Your best estimate of the optimal policy is to take the action in state $a$ with the highest action value. This is:

$$\pi(s) = \text{argmax}_a \hat{q}(s,a)$$

So run your neural network for the current state and all actions. Using the first form of NN above this means run a minibatch, using the second form you run it once which is more efficient (but constructing the training data later is more complex).

If you always take the best action, this is called acting greedily. Usually in Q learning, you want the agent to explore other possibilities instead of always acting the same way - because you need to know whether changing a policy would be better. A very common approach in DQN is to act $\epsilon$-greedily, which means take this greedy action by default, but with some probability $\epsilon$ take a completely random action. Usually $\epsilon$ starts high, often at $1$, and then is decayed relatively quickly down to some lower value e.g. $0.1$ or $0.01$, where it stays during learning. You set $\epsilon = 0$ for fully greedy behaviour when evaluating or using the policy in production.

So that has set your policy, as per your question. There is still more to implement in DQN, summarised below:

Unfortunately you cannot train the agent directly online on each step and expect good results in practice. DQN requires experience replay as the source of training data. For each step of experience, should store data for start state, action, immediate reward, end state and whether this was the last step in an episode ($s,a,r,s',done$)
To train the network, improve the Q estimates and thus improve the policy, then after each step you use the Bellman relation for optimal policy as a target: $q^{*}(s,a) = r + \gamma \text{max}_{a'} q^{*}(s',a')$:

Construct a minibatch of training data from the experience replay table, using your NN to calculate the TD target for each one $\hat{g} = r + \gamma \text{max}_{a'} \hat{q}(s',a')$
Train your NN towards $\hat{q}(s,a) \rightarrow \hat{g}$ on this minibatch.

It is usual to use a separate NN to calculate the TD target, and every so many steps (e.g. 1000 steps) make this "target network" a clone of the current learning network. It helps with stability, but you may not need that for a simple environment like Flappy Bird.


Can I do topic classification of Arabic text (software requirements) without a training dataset?

Yes, it is possible to do it. This can be posed as an unsupervised text classification. You can look at TF-IDF, Neural networks (BERT) etc. for creating embedding of your text and then, use clustering techniques like K-Means, KNN etc. for classification.


Can I perform multiclass classification when the number of features is less than the number of targets?

Of course. It only depends if those features are informative enough for the task at hand. In order to better understand the phenomenon, you can imagine 2 features displayed as points in a 2D plane. The number of possible target classes goes up to the number of clusters you can find in that plane.

About the suggestion, I can only recommend the utilisation of a non-linear classifier.


How would 1D gradient descent look like?

For a function of one variable, there are only two options for directions in the domain: left or right, so it becomes almost trivial, but you can still talk about gradient descent.

You would take steps to the left if the slope/derivative is positive and make steps to the right if the slope/derivative is negative--i.e. the opposite direction of the derivative (the 1d version of the "gradient" in gradient descent), which is equivalent to the higher dimensional case.


What is the purpose of an activation function in neural networks?

Almost all of the functionalities provided by the non-linear activation functions are given by other answers. Let me sum them up:

First, what does non-linearity mean? It means something (a function in this case) which is not linear with respect to a given variable/variables i.e. $f(c1.x1 + c2.x2...cn.xn + b) != c1.f(x1) + c2.f(x2) ... cn.f(xn) + f(b).$
NOTE: There is some ambiguity about how one might define linearity. In polynomial equations we define linearity in somewhat a different way as compared to in vectors or some systems which take an input $x$ and give an output $f(x)$. See the second answer.
What does non-linearity mean in this context? It means that the Neural Network can successfully approximate functions (up-to a certain error $e$ decided by the user) which does not follow linearity or it can successfully predict the class of a function that is divided by a decision boundary that is not linear.
Why does it help? I hardly think you can find any physical world phenomenon which follows linearity straightforwardly. So you need a non-linear function that can approximate the non-linear phenomenon. Also, a good intuition would be any decision boundary or a function is a linear combination of polynomial combinations of the input features (so ultimately non-linear).
Purposes of activation function? In addition to introducing non-linearity, every activation function has its own features.

Sigmoid $\frac{1} {(1 + e ^ {-(w1*x1...wn*xn + b)})}$
This is one of the most common activation function and is monotonically increasing everywhere. This is generally used at the final output node as it squashes values between 0 and 1 (if the output is required to be 0 or 1). Thus above 0.5 is considered 1 while below 0.5 as 0, although a different threshold (not 0.5) maybe set. Its main advantage is that its differentiation is easy and uses already calculated values and supposedly horseshoe crab neurons have this activation function in their neurons.
Tanh $\frac{e ^ {(w1*x1...wn*xn + b)} - e ^ {-(w1*x1...wn*xn + b)})}{(e ^ { (w1*x1...wn*xn + b)} + e ^ {-(w1*x1...wn*xn + b)}}$
This has an advantage over the sigmoid activation function as it tends to centre the output to 0 which has an effect of better learning on the subsequent layers (acts as a feature normaliser). A nice explanation here. Negative and positive output values maybe considered as 0 and 1 respectively. Used mostly in RNN's.
Re-Lu activation function - This is another very common simple non-linear (linear in positive range and negative range exclusive of each other) activation function that has the advantage of removing the problem of vanishing gradient faced by the above two i.e. gradient tends to 0 as x tends to +infinity or -infinity. Here is an answer about Re-Lu's approximation power in-spite of its apparent linearity. ReLu's have a disadvantage of having dead neurons which result in larger NN's.
Also, you can design your own activation functions depending on your specialized problem. You may have a quadratic activation function which will approximate quadratic functions much better. But then, you have to design a cost function that should be somewhat convex in nature, so that you can optimise it using first-order differentials and the NN actually converges to a decent result. This is the main reason why standard activation functions are used. But I believe with proper mathematical tools, there is a huge potential for new and eccentric activation functions.
For example, say you are trying to approximate a single-variable quadratic function say $a.x^2 + c$. This will be best approximated by a quadratic activation $w1.x^2 + b$ where$w1$ and $b$ will be the trainable parameters. But designing a loss function that follows the conventional first-order derivative method (gradient descent) can be quite tough for non-monotonically increasing function.
For Mathematicians: In the sigmoid activation function $(1 / (1 + e ^ {-(w1*x1...wn*xn + b)})$ we see that $e ^ {-(w1*x1...wn*xn + b)}$ is always < 1. By binomial expansion, or by reverse calculation of the infinite GP series we get $sigmoid(y)$ = $1 + y + y^2.....$. Now in a NN $y = e ^ {-(w1*x1...wn*xn + b)}$. Thus we get all the powers of $y$ which is equal to $e ^ {-(w1*x1...wn*xn + b)}$ thus each power of $y$ can be thought of as a multiplication of several decaying exponentials based on a feature $x$, for eaxmple $y^2 = e^ {-2(w1x1)} * e^ {-2(w2x2)} * e^ {-2(w3x3)} *...... e^ {-2(b)}$. Thus each feature has a say in the scaling of the graph of $y^2$.
Another way of thinking would be to expand the exponentials according to Taylor Series:
$$e^{x}=1+\frac{x}{1 !}+\frac{x^{2}}{2 !}+\frac{x^{3}}{3 !}+\cdots$$
So we get a very complex combination, with all the possible polynomial combinations of input variables present. I believe if a Neural Network is structured correctly the NN can fine-tune these polynomial combinations by just modifying the connection weights and selecting polynomial terms maximum useful, and rejecting terms by subtracting the output of 2 nodes weighted properly.
The $tanh$ activation can work in the same way since output of $|tanh| < 1$. I am not sure how Re-Lu's work though, but due to its rigid structure and problem of dead neurons we require larger networks with ReLu's for a good approximation.
But for a formal mathematical proof, one has to look at the Universal Approximation Theorem.

A visual proof that neural nets can compute any function
The Universal Approximation Theorem For Neural Networks- An Elegant Proof

For non-mathematicians some better insights visit these links:
Activation Functions by Andrew Ng - for  more formal and scientific answer
How does neural network classifier classify from just drawing a decision plane?
Differentiable activation function
A visual proof that neural nets can compute any function


What are the benefits of the VGDL over the ALE?

Here is a description of the input to an ALE agent:
Percept state: A single game screen (frame): a 2D array of 7-bit pixels, 160 pixels wide by 210 pixels high.
Actions: 18 discrete actions defined by the joystick controller

Regarding VGDL, as far as I can see, the main site associated with it is gvgai.net, which is currently down. The associated API is described in this paper.

Percept state for GVGAI is more structured than for ALE, but the closest correspondence to ALE appears to be an 'Observation grid', consisting of a 2D array of sprite identifiers.

Actions: ACTION_NIL, ACTION_UP, ACTION_LEFT, ACTION_DOWN, ACTION_RIGHT and ACTION_USE (stated as 'typical' values).

Of the two, it would seem that ALE is more suitable for AGI, because of the more 'free form' nature of the input.

However, one of the issues with either of these approaches is that the set of possible actions is strongly constrained. These domains are therefore 'operationalised' - the hard task of working out what actions are possible has already been solved for the AI by the API, effectively acting as a bottleneck on the complexity of mapping from input to output.

A range of alternative game-playing frameworks are listed here and one alternative (which I personally believe is more useful for AGI purposes) is the Artificial Text Adventurer, in which (at each turn) agent is presented with natural language input describing the scene and must then output a command in natural language. Disclaimer: I am associated with this competition.


Clarifications on "Prioritized Experience Replay" (Deepmind, 2015)

for 1) i think your confusing elements touched vs sequences. At each point for N turns you have 2 possible options, therefore you have $\prod_{i=1}^N 2$ or $2^N$ possible sequences.

for 2) The priorities are updated based on the expected rewards. They do not add new elements each time, they update


What kind of AI technique can I use to play the "Lines" game?

Writing a game AI is difficult in general and it's even harder when there's no game to play. Like in your case.

While "Lines" is pretty neat, there's no (playing) opponent and there are no moves. apart from the setup. So what we have is an optimization problem where we need to get more point than the "opponents".

Game playing algorithms like Reinforcement learning or MCTS are IMHO out of questions as there's no game to play. Still, it's an interesting problem, though I can't see how to make any learning AI. Anyway, what you need first is a simulator and an optimizer choosing promising actions.

Some ideas concerning the actions

Some actions are trivial, e.g., the eraser as there are just a few dots to erase. Trying all combination should be good enough for the start.

The scissors can be in theory placed anywhere, but the only interesting places are just next to the oponent piece or a crossing. Additionally, you only need to consider crossings which gets "won" (reached first) by an opponent. Moreover, they only matter if they get won by an opponent having more points than you or blocking you.

The dots to insert should similarly by placed near an oponent or so that you just win a crossing.

The additional lines are worst, as they may be used to let you reach unconnected areas, or reach a crossing faster or simply give you additional points (then you want a long line somewhere).

A model

Most of the game can be seen as a non-directed graph with edges labeled by their length, where the crossing and the initial dots are nodes. The simulator doesn't need to work "pixel by pixel", but may instead jump to the next time, when a crossing gets reached or two "flows" meat each other. The points of a player grow linearly with time, where the slope is given by the number of their extending "flows".

So far, I could imagine modelling it as an Integer Linear Program, but with the additional lines, it's probably no more possible.

I'd probably let the optimization be simulator-guided, i.e., analyse whether I could get closer to winning by reaching a crossing sooner. The details are still unclear....


It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also  2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.


Given two optimal policies, is an affine combination of them also optimal?

Yes, in general any linear combination of probability distributions between optimal policies is also an optimal policy. In fact any combination with each state treated separately will also be an optimal policy.
This can be seen using the equation for optimal deterministic policy in terms of optimal value function:
$$\pi^*(s) = \text{argmax}_a [\sum_{r,s'}p(r,s'|s,a)(r + \gamma v^*(s'))] = \text{argmax}_a [q^*(s,a)]$$
The only way to have multiple equivalent optimal policies is when there are two or more actions tied for $\text{max}_a [q^*(s,a)]$. If that is the case, then it does not matter - in terms of expected future reward - which of those actions is taken in the affected state. It is possible to take one or the other action.
You can use any rule you wish to decide which of the tied-for-max actions to take, including a random choice. Creating a new policy that is a linear combination of optimal policies is one way to make that random choice. You could equally say that you would make one action choice on a Monday and another on Tuesday - in terms of being optimal choices it does not matter at all how you break ties.


You have stumbled upon a common drawback of the vast majority of modern planning technology. The "flattening" you refer to is actually called "grounding" in the community. Indeed, grounding is the first step of almost every planner out there. Planners that don't do this grounding phase are typically referred to as "lifted planners", but their availability is far more limited (i.e., you'd likely need to reach out to an author of a paper to get access to their code).

That said, many of the planners, Fast Downward included, will try to be smart about the grounding phase. This includes simple things such as making sure that typing for the parameters (and the objects that instantiate them) is adhered to, but goes further with certain checks on reachability. The process isn't flawless, but it usually does a good job when the true number of ground actions is relatively small. If your model really does have an astronomical number of ground actions, then you're kind of out of luck here.

To your questions specifically:

Yes, they are designed for problems where you can ground things. But the mark of suitability is the number of reachable ground actions. If this is small enough, then either (1) the planners should be able to handle it or (2) it presents a research opportunity for grounding things better.
This is the most recent work that comes to mind, as well as the papers that cite it:

https://www.aaai.org/ocs/index.php/ICAPS/ICAPS14/paper/view/7948/8033
https://scholar.google.com/scholar?cites=2377320520977506929&as_sdt=2005&sciodt=0,5&hl=en

They aren't assumed to be distinct. There are situations when you wouldn't want them to be, and so it is up to the designer to add the preconditions that ensure they are distinct (note that this is one excellent way to equip the grounder with extra information about what it should generate).
Generally, having actions with many parameters is a recipe for problems that have too many ground actions. Either manually remodeling things yourself, or applying a technique like operator splitting is the approach most often taken.

Finally, if you'd like to have a more directed discussion, feel free to join us over on the slack channel. There may be more eyes on the questions you pose there.


Why do we need floats for using neural networks?

TL;DR

Not only it's possible, it even gets done and is commercially available. It's just impractical on a commodity HW which is pretty good at FP arithmetic.

Details

It is definitely possible, and you might get some speed for a lot of troubles.

The nice thing about floating point is that it works without you knowing the exact range. You can scale your inputs to the range (-1, +1) (such a scaling is pretty commonplace as it speeds up the convergence) and multiply them by 2**31, so they used the range of signed 32-bit integers. That's fine.

You can't do the same to your weights as there's no limit on them. You can assume them to lie in the interval (-128, +128) and scale them accordingly.

If your assumption was wrong, you get an overflow and a huge negative weight where a huge positive weight should be or the other way round. In any case, a disaster.

You could check for overflow, but this is too expensive. Your arithmetic gets slower than FP.

You could check for possible overflow from time to time and take a corrective action. The details may get complicated.

You could use saturation arithmetic, but it's implemented only in some specialized CPUs, not in your PC.

Now, there's a multiplication. With use of 64-bit integers, it goes well, but you need to compute a sum (with a possible overflow) and scale the output back to same sane range (another problem).

All in all, with fast FP arithmetic available, it's not worth the hassle.

It might be a good idea for a custom chip, which could do saturation integer arithmetic with much less hardware and much faster then FP.

Depending on what integer types you use, there may be a precision loss when compared to the floating point, which may or may not matter. Note that TPU (used in AlphaZero) has 8-bit precision only.


Selecting the right technique to predict disease from symptoms

I think you're coming at your problem slightly wrong... what you're essentially talking about is a belief network.
You may want to look into existing Bayesian Learning techniques to get your head around this, but belief networks commonly use the exact scenario you're talking about; using a set of known (or uncertain facts) statements to produce some inferred probability of a particular output.
Even more, they often express this through disease-symptom based examples in tutorials! Try here.
My point is that it would be better to use a belief network as the theoretical groundwork is all already there for you, instead of an ANN.


How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.

Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform').

... "unusual" element to point here;  I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.

There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously too smooth.

I've also noticed that the losses provided by history.history are different

Yes, so this is a little gotcha in the keras implementation. For training loss, keras does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. (link)
That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by $\sim1/2$  of an epoch.
The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated ("dying ReLus") almost instantly.
Couple of suggestions that I can make:

I don't see you setting up a learning rate. Try making it smaller (like, much smaller). And see if there's any difference.

MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?

Does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ?

Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on https://playground.tensorflow.org/ - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.


Mining repeated subsequences in a given sequence

You can do this similar to the BIDE approach. It can be done like this:
class TreeNode:
    def __init__(self, element, depth, count=0, parent=None):
        self.count= count
        self.element= element
        self.depth= depth
        self.subnodes= dict()
        self.parent= parent

    def __repr__(self):
        return f'{self.__class__.__name__}({self.element}, {self.depth}, {self.count})'

    def get_subnode(self, element, create=True):
        if create is True:
            return self.subnodes.setdefault(element, TreeNode(
                element,
                self.depth+1,
                count=0,
                parent=self
            )
        )
        else:
            return self.subnodes.get(element)

    def get_subnode_increasing_count(self, element):
        subnode= self.get_subnode(element)
        subnode.count+= 1
        return subnode

    def harvest(self, N, m, M, prefix=[], append_result_to=None):
        # get all sequences which occurs more than N times
        # and have a length (depth) of m < depth < M
        prefix= prefix.copy()
        prefix_len= len(prefix)
        if append_result_to is None:
            result= list()
        else:
            result= append_result_to
        if self.element is not None:
            prefix.append(self.element)
            prefix_len+= 1
        if N >= self.count or prefix_len >= M:
            return result
        if N < self.count and m < prefix_len and prefix_len < M:
            # the subsequence matches the constraints
            result.append(prefix)
        for subnode in self.subnodes.values():
            subnode.harvest(
                N,
                m,
                M,
                prefix=prefix,
                append_result_to=result
            )
        return result

    def get_prefix(self):
        if self.parent is None:
            prefix= []
        else:
            prefix= self.parent.get_prefix()
        if self.element is not None:
            prefix.append(self.element)
        return prefix

    def print_tree_counts(self, leaves_only=False, min_count=0):
        if leaves_only is False or not self.subnodes:
            if self.count >= min_count:
                print(self.get_prefix(), self.count)
        for subnode in self.subnodes.values():
            subnode.print_tree_counts(leaves_only=leaves_only, min_count=min_count)

sequential approach
def find_patterns(S, N, m, M):
    root= TreeNode(None, 0, 0)
    active_nodes= []
    for el in S:
        root.count+=1
        # append the root node
        active_nodes.append(root)
        # now replace all nodes in active nodes by the node
        # that is reached one level deeper following el
        active_nodes= [node.get_subnode_increasing_count(el) for node in active_nodes if node.depth < M]

    # now harvest the tree after applying the restrictions
    return root, root.harvest(N, m, M)

S='AAFFAABDEFEDFFAAFCCABDDCCCABDDCACBEABCFFAAEABCADEFFAABCDAEABCABDD'
root, patterns= find_patterns(S, 3, 3, 6)
patterns

The result is:
[['F', 'F', 'A', 'A']]

Your second pattern occurs exactly 3 times and so doesn't fullfill the requirement of > 3 occurances ([C,A,B,D,D]).
modifications for parallel processing
To make it processable in parallel you can do a slide modification. Just create another method in TreeNode, that allows to merge nodes. Like this:
def merge_nodes(self, other_nodes):
    # merge other_nodes into this node
    # including all subnodes
    if len(other_nodes) > 0:
        elements= set()
        for other_node in other_nodes:
            self.count+= other_node.count
            elements.update(other_node.subnodes.keys())
        # elements now contains the set of the next elements
        # with which the sequence continues across the
        # other nodes
        for element in elements:
            # get the node of the resulting tree that represents
            # the sequnce continuing with element, if there is
            # no such subnode, create one, since there is at least
            # one other node that counted sequence seq + element
            my_subnode= self.get_subnode(element, create=True)
            other_subnodes= list()
            for other_node in other_nodes:
                # now get the subnode for each other node, that
                # represents the same sequence (seq + element)
                other_subnode= other_node.get_subnode(element, create=False)
                if other_subnode is not None:
                    other_subnodes.append(other_subnode)
            # merge the subnode the same way
            my_subnode.merge_nodes(other_subnodes)

Now you can call find_patterns on subtrees, but you need to take into account, that you may not split the input sequence normally. You need to define some overlapping sequence part, so that patterns, that begin at the end of one sequence-fragment can be completed but that this overlapping sequences need to be counted differently (otherwise you get double-counts, which lead to wrong inputs). So you have to make sure, that only patterns, which began in the sequence fragment are continued with the overlap part, but that no new patterns are started in the overlap part, because you count them elsewere:
def find_patterns(S, N, m, M, overlap=[]):
    root= TreeNode(None, 0, count=0, parent=None)
    active_nodes= []
    def process_element(active_nodes, element, M):
        return [
            node.get_subnode_increasing_count(element)
                for node in active_nodes
                    if node.depth < M
        ]
    for el in S:
        root.count+=1
        # append the root node
        active_nodes.append(root)
        # now replace all nodes in active nodes by the node
        # that is reached one level deeper following el
        active_nodes= process_element(active_nodes, el, M)
    # complete the already started sequences with the
    # overlapping sequence (the sequence that may be
    # processed in another process)
    for el in overlap:
        active_nodes= process_element(active_nodes, el, M)
    # now harvest the tree after applying the restrictions
    return root, root.harvest(N, m, M)

Now you can do:
split_point= len(S) // 2
S1= S[split_point:]
S2= S[:split_point]
S1_overlap= S2[:10]
# in fact you could have just used :5 above,
# but that doesn't affect the result
r1, _= find_patterns(S1, 3, 3, 6, overlap=S1_overlap)
# Note: you could safely start the comand in the next
# line in parallel to the previous call of find_patterns
# and of course, you could split your sequence in as
# many fragments, as you like, as long as you maintain
# the overlap correctly and call merge_nodes afterwards
r2, _= find_patterns(S2, 3, 3, 6)
r1.merge_nodes([r2])
patterns= r1.harvest(3, 3, 6)

This yields the same result as above. To make it a bit clearer, what I mean with the overlap part:
S1 is 'CBEABCFFAAEABCADEFFAABCDAEABCABDD'
S2 is 'AAFFAABDEFEDFFAAFCCABDDCCCABDDCA'
S1_overlap is 'AAFFAABDEF'

With find_patterns(S1, 3, 3, 6) you would only search for repeated patterns in S1, so it would not consider patterns which begin with the part BDD (starting at the end of S1) and are continued within S2. With find_patterns(S1, 3, 3, 6, overlap=S1_overlap) I consider these patterns as well.


What is the best .net programming language for artificial intelligence programming?

If you're talking about pure speed, C will get you there if you really know C and operating systems, etc. C++ is nicer in terms of user friendliness, and won't be much slower. I don't know much about VB but I don't see many benefits.

Please clarify at least generally what the AI program is about. It is extremely difficult to answer the generalized question "What programming language is best for AI"? if you know what I mean. (  I vote python :^]  )

To add: Any programming language of those listed is "stable" if you write things correctly, but C++ and its great IDE's will help you to that point much more nicely than C will. C, to fully utilize it's potential, requires much fiddling with delicate and precise systems. It'll go that little bit faster, as the cost of being less stable in the practical, " uh oh, now I need to troubleshoot this" sense.


What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (typically in single-GPU setups) by cleverly avoiding memory issues. The core idea of gradient accumulation is to perform multiple backward passes using the same model parameters before updating them all at once for multiple batches. This is unlike the conventional manner, where the model parameters are updated once every batch-size number of samples. Therefore, finding the correct batch-size and accumulation steps is a design trade-off that has to be made based on two things: (i) how much increase in the batch-size can the GPU handle, and (ii) whether the gradient accumulation steps result in at least as much better performance than without accumulation.
As for your example configurations, there are the same in theory. But, there are a few important caveats that need to be addressed before proceeding with this intuition.

Using Batch Normalization with gradient accumulation generally does not work well, simply because BatchNorm statistics cannot be accumulated. A better solution would be to use Group Normalization instead of BatchNorm.
When performing a combined update in gradient accumulation, it must be ensured that the optimizer is not initialized to zero (i.e. optimizer.zero_grad()) for every backward update (i.e. loss.backward()). It is easy to include both statements in the same for loop while training which defeats the purpose of gradient accumulation.

Here are some interesting resources to find out more about it detail.

Thomas Wolf's article on different ways of combating memory issues.
Kaggle discussions on the effect on learning rate: here and here.
A comprehensive answer on gradient accumulation in PyTorch.


Why can the Bellman equation be turned into an update rule?

Why are we allowed to convert the Bellman equations into update rules?

There is a simple reason for this: convergence. The same chapter 4 of the same book mentions it. For example, in the case of policy evaluation, the produced sequence of estimates $\{v_k\}$ is guaranteed to converge to $v_\pi$ as $k$ (i.e. the number of iterations) goes to infinity. There are other RL algorithms that are also guaranteed to converge (e.g. tabular Q-learning).

To conclude, in many cases, the update rules of simple reinforcement learning (or dynamic programming) algorithms are very similar to their mathematical formalization because algorithms based on those update rules are often guaranteed to converge. However, note that many more advanced reinforcement learning algorithms (especially, the ones that use function approximators, such as neural networks, to represent the value functions or policies) are not guaranteed or known to converge.


How to predict the "word" based on the meaning in a document?

While your question has some ambiguities, I try to answer.
From my understanding you want your model to predict “topic” of a sentence or a description. It’s just a classification problem with huge possible number of output classes.
The first initial issue is very short length of documents (sentences). Most of topic modelling algorithms such as LDA have statistical approach and do not work very well with very short documents (less than 50 words could be a good definition of very short document).
The second issue is how do you want to collect enough data to train your model that supposed to predict target out of extremely large number of output classes? Dictionaries are not enough because they offer a single definition for each word. Examples of words in dictionaries don’t help much and they will probably affect your model adversely. How can your model be generalised by a single (or few) example(s) for each class?
So, it’s not possible, but maybe having some innovations can help.
Here is definition of “apple” in oxford dictionary: ”a round fruit with shiny red or green skin that is fairly hard and white inside”. There are just two nouns in the definition: "fruit" and "Skin", if we just read the definition without considering these two words, even we, as human, struggle to guess.
Consider nouns in input data and use them to build up a natural graph. You just consider main classes such as "fruit". If you’re getting some good results, consider other words, adj, adv, ...


In the definition of the state-action value function, what is the random variable we take the expectation of?

I am using the convention of uppercase $X$ for random variable and lowercase $x$ for an individual observation. It is possible your source material did not do this, which might be causing your confusion. However, it is the convention used in Sutton & Barto's Reinforcement Learning: An Introduction.

What I didn't understand what is  here. i.e., what is the random variable on which we are calculating long run rewards?

The random variable is $R_t$, the reward at each time step. The distribution of $R_t$ in turn depends on the distribution of $S_{t-1}$ and $A_{t-1}$ plus the policy and state progression rules. There is no need to include the process that causes the distribution of each $R_t$ in every equation. Although sometimes it is useful to do so, for example when deriving the Bellman equations for value functions.

My guess is policy function. It is averaging long run rewards over all possible policy functions. Is it true?

No, this is not true. In fact it is the more usual assumption that the policy function $\pi(a|s)$ remains constant over the expectation, and this is what the subscript $\pi$ in $\mathbb{E}_{\pi}[...]$ means.
The expectation is over randomness due to the policy $\pi$, plus randomness due to the environment, which can be described by the function $p(r, s'|s,a)$ - the probability of observing reward $r$ and next state $s'$ given starting in state $s$ and taking acion $a$. These two functions combine to create the distribution of $R_t$. It is possible that both functions are deterministic in practice, thus $R_t$ is also deterministic. However, RL theory works on the more general stochastic case, which is also used to model exploratory actions, even if the target policy and environment are deterministic.


Right now, most AI interact with the world through mechanisms they have been provided by humans such as steering a car, sending output to speakers, or interacting with web APIs.
If any of those mechanisms can interact with the running code of the AI, then in theory, no,  there isn’t a way to place a ‘stop button’ on it. Fortunately, it’s laughably improbable for some situations to happen:

  An intelligent self driving car becomes self aware and wants to connect to the internet. It uses gps to find an internet cafe and threatens to drive over someone if they don’t upload the AI to the internet and remove the car’s internal (code) stop button.