Will artificial intelligence make the human more rational?

I would counter this question with another one:
Is rationality all we strive for?

Intelligence is a waste ground, and we have not even defined which intelligence we're talking about: Emotional Intelligence (empathy), Rational/Intellectual Intelligence (IQ), artistic intelligence, etc. etc. etc.

Something which demarks us as humans is that sometimes we're able to deal with to counteracting intelligences. Logic sometimes dictates one thing, but logic is bound to what parameters are taken into account, and mostly it's about quantifying. an intelligence which does qualitative logic behaves sometimes different, and suggests sometimes other actions.
As humans we weight and counterweight these factors.

Politics is a good example. In most countries there are 2 or 3 rational approaches to problem solving (to make it easy let's call them left, center, right - though I know in fact it's more positions and in an at least two dimensional space, not linear).
But what I mean is all those positions are rational, nevertheless it's other types of intelligence who define their the way they use their ratio.

So in the end this fourth step is just a continuity and I assume that with time theoretically at least the shift should be away from blue collar jobs to more white collar jobs until there are no blue collar jobs.
So far the theory I guess, but then again? Would this be smart? Or intelligent? A world where machines make all the dirty work and we're just supervisors?

My thought to this is we will still need mechanics, probably with more technology always more, just that their focus will shift. 20 years ago mechanics were teached analogue technology, nowadays they learn more digital technology (best example would be car mechanics), but there you see: They're still needed, we also need people with analogue knowledge to mantain older systems which cannot or will not be converted to digital era. And I think that's OK!
So from my point of view, nobody has to fear to loose it's job, but instead embrace, that when you keep your knowledge up to date, there will always be a job for you.
Nowadays for example IT-professionals who are able to code older computer languages are better paid than ever, because their scarce...


What are some examples of intelligent agents for each intelligent agent class?

There are no distinguishable hardware examples for each IA class. The same mobile robot architecture with proper sensors can be implemented to behave as any IA class. The way you can determine the class of an intelligent agent is from the way it processes the percept. Based on chapter 2 of Artificial Intelligent: A Modern Approach I will try to give a concise explanation for each class:

A simple reflex agent takes an action based on only the current environmental situation; it maps the current percept into proper action ignoring the history of percepts. The mapping process could be simply a table-based or by any rule-based matching algorithm. An example of this class is a robotic vacuum cleaner that deliberate in an infinite loop, each percept contains a state of a current location [clean] or [dirty] and, accordingly, it decides whether to [suck] or [continue-moving].

A model-based reflex agent needs memory for storing the percept history; it uses the percept history to help to reveal the current unobservable aspects of the environment. An example of this IA class is the self-steering mobile vision, where it's necessary to check the percept history to fully understand how the world is evolving.

A goal-based reflex agent has a goal and has a strategy to reach that goal. All actions are taken to reach this goal. More precisely, from a set of possible actions, it selects the one that improves the progress towards the goal (not necessarily the best one). An example of this IA class is any searching robot that has an initial location and wants to reach a destination.

An utility-based reflex agent is like the goal-based agent but with a measure of "how much happy" an action would make it rather than the goal-based binary feedback ['happy', 'unhappy']. This kind of agents provide the best solution. An example is the route recommendation system which solves the 'best' route to reach a destination.

A learning agent is an agent capable of learning from experience. It has the capability of automatic information acquisition and integration into the system. Any agent designed and expected to be successful in an uncertain environment is considered to be learning agent.


Is it possible to perform neuroevolution without a fitness function?

You do not always need an explictly coded fitness function to perform genetic algorithm searches. The more general need is for a selection process that favours individuals that perform better at the core tasks in an environment (i.e. that are "more fit"). One way of assessing performance is to award a numerical score, but other approaches are possible, including:

Tournament selection where two or more individuals compete in a game, and the winner is selected.

Opportunity-based selection, where agents in a shared environment - typically with limited resources and chances to compete - may reproduce as one of the available actions, provided they meet some criteria such as having collected enough of some resource. I was not able to find a canonical name for this form of selection, but it is commonly implemented in artificial life projects.

A key distinction between A-life projects and GA optimisation projects is that in A-life projects there is no goal behaviour or target performance. Typically A-life projects are simulations with an open ended result and the developer runs a genetic algorithm to "see what happens" as opposed to "make the best game-player". If your project is like this then you are most likely looking for the second option here.
To discover more details about this kind of approach, you could try searching "artifical life genetic algorithms" as there are quite a few projects of this type published online, some of which use NEAT.
Technically, you could view either of the methods listed above as ways of sampling comparisons between individuals against an unknown fitness function. Whether or not a true fitness function could apply is then partly a matter of philosophy. More importantly for you as the developer, is that you do not have to write one. Instead you can approximately measure fitness using various methods of individual selection.

So far I've implemented most of the basics in amethyst (a parallel game engine written in rust), but I'm worried that the learning will happen very slowly. Should I approach this problem differently?

It is difficult to say whether you should approach the problem differently. However, the biggest bottlenecks against successful GA approaches are:

Time/CPU resources needed to assess agents.

Size of search space for genomes.

Both of these can become real blockers for ambitious a-life projects. It is common to heavily simplify agents and environments in attempts address these issues.


How should I select the features for predicting diseases (in particular when patients specify their health issues)?

So for Medical Prognosis, there are some variables that commonly come up like Age, Sex, Ascites, Hepato, Spider, Status of the disease and many others but it depends on the disease. You'll commonly encounter these variables if you're doing regression or classification.
Also, if you're reading Radiology Reports for getting the input for the model then you also have to take care of jargons. The same symptoms can be written in various ways but all point towards the same prognosis i.e., there can be synonyms for labels. Try reading this to get more information on how we can do information extraction from Radiology Reports. This is the famous CheXpert paper


Most suitable model for video classification with a fixed camera

It seems you need a spatio-temporal model to learn human-body detection and action.
With regards to interesting papers on the subject I would recommand to look at Vicky Kalogeiton web site.

Her PhD thesis
2017, V. Kalogeiton, Localizing spatially and temporally objects and actions in videos,

basically cover her 3 papers on the subject:

2016, Kalogeiton, V., Schmid, C., and Ferrari, V. Analysing domain shift factors between videos and images for object detection,
[2017a, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Action tubelet detector for spatio-temporal action localization]Kalogeiton 2017a,
2017b, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Joint learning of object and action detectors

summary of Kalogeiton's PhD introduction .

Introduce an end-to-end multitask objective that jointly learns object-action relationships.
The action-object detector leverages the temporal continuity of videos.

Though intra class variations are key and appears as spatial location accuracy, appearance diversity, image quality, aspect distribution, and object size and camera framing.
Actions class refers to an atomic class such as jump, walk, run, climb, etc.

The detector builds anchor cuboids named tubelets and defined as sequences of bounding boxes with associated scores. The action detection spans over a period of time (first and last video frame detected) and takes place at a specific location in each frame. Intra frame action detection can be ambigous. On the other way a sequence bears more information (across class similarities) than a single frame to infer action.

Most previous work uses per-frame object detections, and then leverage the motion of objects to refine their spatial localization or improve their classification.

Contributions

differences between still and video frames for training and testing an object detector among which (see Chapter 3 for more details ):

spatial location accuracy,
appearance diversity,
image quality,
aspect distribution,
camera framing

jointly detect object-action instances in uncontrolled videos using an end-to-end two stream network architecture (see chapter 4 for more details )
propose the ACtion Tubelet detector (ACT-detector), which takes as input a sequence of frames and outputs tubelets, i.e. sequences of bounding boxes with associated scores (see chapter 5 for more details).


What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?

This answer assumes that you only have a problem with this notation from the article:

$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

This is a standard notation, used in many disciplines, for defining a function and its input and output domains. It is a bit like the method signature for the function - it does not fully define it, but does enough to show how it can interact with other expressions.
All functions can be thought of as maps between the input domain and output domain. You provide an input value, and it returns an output value. The values can be arbitrary mathematical objects. To show what kind of objects the inputs and outputs are allowed to be, the notation for sets is used.
Importantly the symbol $\mathbb{R}$ at the end does not refer to the set of possible rewards in the environment (although it is a reward function, and that will be its output), but the set of all real numbers, because a reward is always a real number*.
As a concrete example, if you had the function $f(x) = x^2 - 2x + 7$ defined for a real number $x$, then its equivalent notation might be $f : \mathbb{R} \rightarrow \mathbb{R}$. If you allowed $x$ to be complex then it would be $f : \mathbb{C} \rightarrow \mathbb{C}$, because $\mathbb{C}$ is the standard symbol for the set of all complex numbers.
So now we can break down the notation $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

$r$

The function is called $r$

$:$

It has an input domain of . . .

$\mathcal{S} \times \mathcal{A}$

The cartesian product of the set of all possible states $\mathcal{S}$ and the set of possible actions $\mathcal{A}$.
That is much the same as saying the function has a signature $r(s, a)$ where $s \in \mathcal{S}$ and $a \in \mathcal{A}$

$\rightarrow$

It has an output domain of . . .

$\mathbb{R}$

any single real number.

* This choice (of declaring the more general $\mathbb{R}$ instead of specific $\mathcal{R}$) is made partly because operators like $+$ and $\times$ are well defined for real numbers. This is a useful thing to assert about the behaviour of the reward function output when defining how value functions work for instance. Of course you could be more specific, defining $\mathcal{R}$ as some subset of $\mathbb{R}$, that would be correct and more precise definition, but it is not needed for general theory in reinforcement learning. The less precise definition is fine for nearly all purposes.


How are the weights retained for filters for a particular class in a CNN?

Besides the last layer rest of the weights are shared among all classes. When an image is passed to the network all weights are updated accordingly. The only weights that are directly responsible for one specific class are the ones of the final layer. The rest of the weights are updated to find the best values to minimize the average loss for all classes.

To rephrase there aren't "filters" in convolutional layers that are specific for a single class. They are used to extract features so that the final layer can make the final prediction (which has weights for each specific class).

I'd suggest you look a bit into how gradient descent and backpropagation works.


What are bag-of-features in computer vision?

Introduction

Bag-of-features (BoF) (also known as bag-of-visual-words) is a method to represent the features of images (i.e. a feature extraction/generation/representation algorithm). BoF is inspired by the bag-of-words model often used in the context of NLP, hence the name. In the context of computer vision, BoF can be used for different purposes, such as content-based image retrieval (CBIR), i.e. find an image in a database that is closest to a query image.

Steps

The BoF can be divided into three different steps. To understand all the steps, consider a training dataset $D = \{x_1, \dots, x_N \}$ of $N$ training images. Then BoF proceeds as follows.

1. Feature extraction

In this first step, we extract all the raw features (i.e. keypoints and descriptors) from all images in the training dataset $D$. This can be done with SIFT, where each descriptor is a $128$-dimensional vector that represents the neighborhood of the pixels around a certain keypoint (e.g. a pixel that represents a corner of an object in the image).

If you are not familiar with this extraction of computer vision (sometimes known as handcrafted) features, you should read the SIFT paper, which describes a feature (more precisely, keypoint and descriptor) extraction algorithm.

Note that image $x_i \in D$ may contain a different number of features (keypoints and descriptors) than image $x_j \neq x_i \in D$. As we will see in the third step, BoF produces a feature vector of size $k$ for all images, so all images will be represented by a fixed-size vector.

Let $F= \{f_1, \dots, f_M\}$ be the set of descriptors extracted from all training images in $D$, where $M \gg N$. So, $f_i$ may be a descriptor that belongs to any of the training examples (it does not matter which training image it belongs to).

2. Codebook generation

In this step, we cluster all descriptors $F= \{f_1, \dots, f_M\}$ into $k$ clusters using k-means (or another clustering algorithm). This is sometimes known as the vector quantization (VQ) step. In fact,  the idea behind VQ is very similar to clustering and sometimes VQ is used interchangeably with clustering.

So, after this step, we will have $k$ clusters, each of them associated with a centroid $C = \{ c_1, \dots, c_k\}$, where $C$ is the set of centroids (and $c_i \in \mathbb{R}^{128}$ in the case that SIFT descriptors have been used). These centroids represent the main features that are present in the whole training dataset $D$. In this context, they are often known as the codewords (which derives from the vector quantization literature) or visual words (hence the name bag-of-visual-words). The set of codewords $C$ is often called codebook or, equivalently, the visual vocabulary.

3. Feature vector generation

In this last step, given a new (test) image $u \not\in D$ (often called the query image in this context of CBIR), then we will represent $u$ as a $k$-dimensional vector (where $k$, if you remember, is the number of codewords) that will represent its feature vector. To do that, we need to follow the following steps.

Extract the raw features from $u$ with e.g. SIFT (as we did for the training images). Let the descriptors of $u$ be $U = \{ u_1, \dots, u_{|U|} \}$.
Create a vector $I \in \mathbb{R}^k$ of size $k$ filled with zeros, where the $i$th element of $I$ corresponds to the $i$th codeword (or cluster).
For each $u_i \in U$, find the closest codeword (or centroid) in $C$. Once you found it, increment the value at the $j$th position of $I$ (i.e., initially, from zero to one), where $j$ is the found closest codeword to the descriptor $u_i$ of the query image.

The distance between $u_i$ and any of the codewords can be computed e.g. with the Euclidean distance. Note that the descriptors of $u$ and the codewords have the same dimension because they have been computed with the same feature descriptor (e.g. SIFT).

At the end of this process, we will have a vector $I \in \mathbb{R}^k$ that represents the frequency of the codewords in the query image $u$ (akin to the term frequency in the context of the bag-of-words model), i.e. $u$'s feature vector. Equivalently, $I$ can also be viewed as a histogram of features of the query image $u$. Here's an illustrative example of such a histogram.

From this diagram, we can see that there are $11$ codewords (of course, this is an unrealistic scenario!). On the y-axis, we have the frequency of each of the codewords in a given image. We can see that the $7$th codeword is the most frequent in this particular query image.

Alternatively, rather than the codeword frequency, we can use the tf-idf. In that case, each image will be represented not by a vector that contains the frequency of the codewords but it will contain the frequency of the codewords weighted by their presence in other images. See this paper for more details (where they show how to calculate tf-idf in this context; specifically, section 4.1, p. 8 of the paper).

Conclusion

To conclude, BoF is a method to represent features of an image, which could then be used to train classifiers or generative models to solve different computer vision tasks (such as CBIR). More precisely, if you want to perform CBIR, you could compare your query's feature vector with the feature vector of every image in the database, e.g. using the cosine similarity.

The first two steps above are concerned with the creation of a visual vocabulary (or codebook), which is then used to create the feature vector of a new test (or query) image.

A side note

As a side note, the term bag is used because the (relative) order of the features in the image is lost during this feature extraction process, and this can actually be a disadvantage.

Further reading

For more info, I suggest that you read the following papers

Video Google: A Text Retrieval Approach to Object Matching in Videos (2003) by Sivic and Zisserman
A Bayesian Hierarchical Model for Learning Natural Scene Categories (2005) by Fei-Fei and Perona
Introduction to the Bag of Features Paradigm for Image Classification and Retrieval (2011) by O'Hara and Draper
Bag-of-Words Representation in Image Annotation: A Review (2012) by Tsai


How to encode card game state into neural network input

I am not sure what you mean by

There is now multiple ways to represent this same exact game state in my network because each individual card has it's own state. To me this does not seem good.

We can think of game state as a column vector of the form $1Xn$. In this case, you can formalize cards held according to some encoding. This can be anything from a Gödel Numbering to simple integer assignment. Here is an example for poker in python: https://pypi.org/project/treys/
In the example of duplicates you would have a column vector that simply has duplicate indices. I.e: {241, 424, 112, 112, 455}$


Which online AI masters degrees do you recommend me?

If you aren't looking to be accredited, here are some good options:
Stanford
Berkeley
University of Toronto
metacademy
If the links are dead, a google search on the relavant course code should yield results.
If you are looking for accreditation, then you might want to try:
Coursera
But note that in my opinion, the class materials from the links above should be of better quality.


A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.

An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.

There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.


What's the input to the Generator?
In the basic implementation of GANs, the Generator only takes in a vector of random variables. This might seem strange, but after training, the generator can transform this input noise into an image resembling those of the training set.
How does it work?
It is trained along with its counterpart the Discriminator, whose goal is to distinguish real images (i.e. the dataset's images) from fake ones (i.e. images produced by the Generator). The Generator's goal in training is to fool the Discriminator into thinking that its images are real.
Training process
In the beginning, where they are both untrained, they are both "terrible" at their respective tasks. The Generator can't produce anything resembling an image, but the Discriminator can't distinguish real from fake. As training progresses, the Discriminator starts identifying ways to distinguish the real images from the fake ones (i.e. patterns that appear in real images, but not in fake ones). The Generator, however, in its attempt to fool the Discriminator, starts producing those same patterns in its own images. After a while of both models becoming better at their respective tasks, we reach a point where the Generator can produce realistic images and the Discriminator is very good at distinguishing between real or fake.

Edit as suggested from comment:
A vector of random values is used as an input, so that the Generator can learn to generate unique outputs. In itself the Generator is deterministic, meaning that it has no internal sources of randomness. If we give it the same input vector twice, it will produce the same output both times. Thus, we feed the Generator with random values, so that it can learn to produce different outputs, depending on those values.


What is the difference between learning and non-learning agents?

A learning agent is an agent which has a problem generator, a learning element, a critic and a performance element. ProbGen just creates different instances of a problem w.r.t. a domain and a set of parameters in jurisdiction. A learning element is responsible for making efficient choices and performance element chooses actions for the selected choice. It also changes some of the knowledge base. A critic is basically a metric to tell which is a better choice.

Deep learning is a bit complicated than that. Then again, let's simplify. You see, deep learning is a specific arrangement or subsection of neural networks that helps the learning agent learn faster. That's as simply as I could explain it. What we do is that we arrange the layers in a neural network with a purpose and densely pack it to make it more task specific. Obviously, I am leaving out a lot of implementation details but for them, I am afraid, you need to look up a tutorial.

How are they used in AI?
That's problem specific. Learning agents are everywhere from spam detection to roombas to artificial dog pets. Deep learning on the other hand is currently focused on user pattern recognition and personalization in every front. Please let me know if you need me to expand upon this answer.


Understanding the node information score in the paper "Hierarchical Graph Pooling with Structure Learning"

Here, $H$ is a $n * d$ matrix where $n$ is the number of total nodes in the graph and $d$ is the dimension of embedding of each node.

Using the notation in the question, the basic GNN formulation without self loop is: $\text{D}^{-1}\text{A}\text{H}$. If you study this equation closer then you will find that the $i^{th}$ row of $\text{A}\text{H}$ generates the $i^{th}$ node's representation by summing the node representation of its neighboring nodes. Multiplying it with $\text{D}^{-1}$ makes the aggregated representation normalized with respect to the degree of a node (number of neighbors).

By defining a metric called information score:
$$||\text{H} - (\text{D}^{-1}\text{A}\text{H})||$$
we will be getting low values for nodes that are well represented by their local neighborhood nodes and high values for nodes that are having a hard time being represented/summarized by its neighboring nodes.
To approximate the graph information, the authors choose to preserve the nodes that can not be well represented by their neighbors, i.e., the nodes with relatively larger node information score will be preserved in the construction of the pooled graph, because the authors believe it can provide more information.


Number of LSTM layers needed to learn a certain number of sequences

From my personal experience, the units hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.
And about the number of LSTM layers, trying out a single LSTM layer is a good start point, the model trains better with more LSTM layers.
For example, MAX_SEQ_LEN=10, in Keras:
Lstm1 = LSTM(units=30, return_sequences=True); #Time sequence, to feed to next layer
Lstm2 = LSTM(units=20, return_sequences=True); #Time sequence, to feed to next layer
Lstm3 = LSTM(units=10, return_sequences=False);

Output = Lstm3(Lstm2(Lstm1));
```


Parameters to calculate affluence in localities of Metro city

Affluence could encompass several parameters:
Income;
Wealth (property ownership);
Life expectancy;
Access to services such as education and health;
Access to clean natural resources;
Low levels of criminality.

Property prices in each locality might be easy to obtain from real estate agent sources
Ratings for schools or medical facilities in each area might be published

Generally, where public statistics are collected on a locality, they will be related in one way or another to affluence. A useful strategy might be to collect as many of these diverse data sets as possible, and to learn a composite affluence score from the data. It is very likely that all of these parameters will be correlated to a greater or lesser degree, and so you could accurately learn about affluence from a small number of these parameters.


Facial recognition and classifying unknowns with neural networks

Summary

It is true that neural networks are inherently not good at classifying 'unknowns' because they tend to overfit to the data that they have been trained on, if the underlying structure of the neural network is complex enough. However, there are multiple ways to go about reducing the affects of overfitting. For example, one technique that is used for this is called dropout. Another example can be batch normalization. Despite these techniques, the best way to reduce the affects of overfitting is to use more data.

For the facial recognition example that you have given above, it is common that the models that have been trained have 'seen' a huge amount of data. This means that there are very few 'unknowns' and even if there are, the neural network has learned how to tell if there are facial features present or not. This is because certain structures of neural networks are really good at telling if there is a pattern of features present in the input data. This helps the neural networks to learn if the image that is being input has certain features/patterns in it or not. If the these features are found then the input data is classified as face otherwise it is not.

What can you do in your case?

Let us assume that you are going to train your neural network to recognize if an input image is a cat or not. You will use a Convolutional Neural Network (CNN) and train it to recognize if the input is cat or not. The not part means that you have to include a lot of examples in your training data that are not cat. In the perfect case you will be able to show it everything that is not a cat and classify it as such. Also you show it multiple images of what a cat is. CNNs are really great for this application. You might want to research regarding this and see what kind of CNN best suits your application. If you don't have gazillion samples of what a cat is not then you can use regularization techniques like dropout and batch normalization.

PS: For more details please mention what strategies you have used up till now. Also it would be better if you can share what your desired task is.


Why am I getting the incorrect value of lambda?

$TD(\lambda)$ return has the following form:
\begin{equation}
G_t^\lambda = (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n}
\end{equation}
For you MDP $TD(1)$ looks like this:
\begin{align}
G &= 0.64 (r_0 + r_2 + r_4 + r_5 + r_6) + 0.36(r_1 + r_3 + r_4 + r_5 + r_6)\\
G &\approx 6.164
\end{align}
$TD(\lambda)$ looks like this:
\begin{equation}
G_0^\lambda = (1-\lambda)[\lambda^0 G_{0:1} + \lambda^1 G_{0:2} + \lambda^2 G_{0:3} + \lambda^3 G_{0:4} + \lambda^4 G_{0:5} ]
\end{equation}
Now each $G$ term separately:
\begin{align}
G_{0:1} &= 0.64(r_0 + v_1) + 0.36(r_1 + v_2) \approx 7.864\\
G_{0:2} &= 0.64(r_0 + r_2 + v_3) + 0.36(r_1 + r_3 + v_3) \approx -5.336\\
G_{0:3} &= 0.64(r_0 + r_2 + r_4 + v_4) + 0.36(r_1 + r_3 + r_4 + v_4) \approx 25.864\\
G_{0:4} &= 0.64(r_0 + r_2 + r_4 + r_5 + v_5) + 0.36(r_1 + r_3 + r_4 + r_5 + v_5) \approx -11.936\\
G_{0:5} &= 0.64(r_0 + r_2 + r_4 + r_5 + r_6 + v_6) + 0.36(r_1 + r_3 + r_4 + r_5 + r_6
 + v_6) \approx -0.336
\end{align}
Finally, we need to find $\lambda$ so that the return is equal to $TD(1)$ return, we have:
\begin{equation}
6.164 = (1 - \lambda)[7.864 - 5.336\lambda + 25.864\lambda^2 - 11.936\lambda^3 - 0.336\lambda^4]
\end{equation}
When you solve this equation, one of the solutions is  $0.205029$ which is close to what you needed to get considering the numerical errors. Your problem was that you only considered probability in first state, but that decision prolongs to future states as well.

EDIT

As pointed out by bewestphal, this is not a full solution, it misses one crucial step to get it fully correct. Hint for it can be found in his answer and that's the correct solution.


Dempster-Shafer Theory was studied fairly seriously in AI through the 70's and 80's. However, Dempster-Shafer theory has some serious shortcomings, that will cause an agent that uses it to make irrational decisions. These were uncovered by Pearl and others in the 80's, with more problems emerging in later years.

See Cheeseman for a summary of the arguments that probability is the only suitable framework for reasoning under-uncertainty in AI (which is the current orthodox view), along with Josang & Harkin 2012 (which develops a theory that subsumes Dempster-Shafer, and under which DS can be shown to lead to poor decisions), and Pearls' On Probability Intervals.


How much the dialects recognition and speech recognition are relevant?

The tutorials you link are not much relevant, there are already existing implementations of your exact problem.

You can use https://github.com/swshon/dialectID_e2e, there are many other similar implementations on github.


Can we make a chatbot that UNDERSTANDS not just answer questions based on his database?

This question has been studied academically for decades, and is really an extension of the work on Philosophy of Mind that was done in the two or three centuries before that.

A good resource is Mind Design II, though it's getting a little bit old now.

The modern schools of thought are:

Cognitivism. This is in decline, but was extremely popular in the 70's and 80's, and still fairly widespread in one form or another in the AI research community. It says that human brains really are just computers. If they're just computers, and that they're probably running a sort of symbolic reasoning algorithm like unification (although I think it's hard to find anyone who really thinks it's unification anymore). This is the idea underpinning work like SOAR. The main bottleneck, as Drayfus pointed out in the 1970's, is that you need to write down all the facts about something for a machine to "understand" it. "All the facts" turns out to rapidly turn into an infinite number for anything more complex than the smallest "microworlds" that you could deploy an AI program in. Searle also proposed his Chinese Room argument in response to this group, but it holds for Connectionist approaches as well (more on that later...).
Connectionism The connectionists hold that the complexity of our brains comes from massively parallel computation consisting of messages passed between billions of neurons in our heads. They think the correct approach to general AI is likely to involve simulations of similar architectures. It turns out that many of the things that are incredibly hard for Cognitivist projects (e.g. vision), are easy to solve with these approaches. The main criticisms from Cognitivists are that we don't have a very good idea of what these things are doing, and so the claim that they help us understand intelligence is false, and using them to solve practical problems might be dangerous. These are both somewhat fair, in my view. Older Cognitivist arguments, put forth most elegantly by Jerry Fodor, have now been discredited. Fodor argued that properties like language could never be understood as statistical artifacts of parallel computation, but he was wrong: all the best computational systems for language are now connectionist, and no one's ever made a cognitivist one that's even half as convincing. This is the dominant paradigm behind most of the modern advances in the field. Hinton's work forms the basis of most of the recent advances.
Dynamics Searle's argument was rooted in the idea that mapping inputs to outputs couldn't be what's happening in our heads, and that such a system couldn't be called intelligent. This also seems to be an implicit assumption in your question. The Dynamicists believe a variety of things, but I'd characterize them as collectively rejecting this idea. Authors like Paul Churchland argue that Searle's argument is rooted in a sort of pre-enlightenment "folk psychology". It's a bit like the theories that predated modern chemistry. Everyone was sure that fire was a substance that lived inside wood. If you heated up the wood properly, it could get out of the wood, making more heat. On the surface this seems pretty reasonable, but of course, it's wrong: The fire is actually a mixture of the wood with oxygen in the air, forming a new gas. There's no fire inside the wood. Similarly, Churchland would argue that there's no "Consciousness" inside us, allowing us to control our actions in the way we popularly imagine. Subjective experience is more likely to be "along for the ride", and entirely or mostly separate from the intelligent behaviors we observe. Some researchers think it could be described by a sub-system that maps observations of what the rest of the brain does into "stories" for the rest of the brain to receive as a sort of summary digest. Active research in this area tends to focus on things like the insect metaphor, and the interaction of the machine with its environment. It was fairly popular in the 1990's, but the phenominal success of connectionist approaches in the 2000's has led to its decline. Probably the best known experiments were the work of Rodney Brooks.


How to build a custom morphological analyser for translation system

Huggingface has a Helsinki-NLP/opus-mt-ka-en repository with a Georgian (Ka) to English (en) model. A tokenizer_config.json is available


Backpropagation with stride > 1 involves dilation of the gradient tensor with stride-1 zeroes. I created a blog post that describes this in greater detail.


What algorithm should I use to classify documents?

Text approach:

Use LDA (Latent Dirichlet Allocation). LDA is unsupervised. Feed it in corpuses of text from the various documents (i.e. OCR them and feed LDA the results of OCR). It will then cluster them based on the contents of the text (with or without stop words - at your discretion). If possible, you could do a supervised approach of using a bag-of-words and any classifier such as an SVM or Random Forest.

Image Approach:

Use a CNN (convolutional neural network) and train it on images of the various vendors. If you don't have this class discrimination, and can't get it, then use an unsupervised approach such as an autoencoder and then cluster the points in the lower-dimensional autoencoder feature space.


Should the reward or the Q value be clipped for reinforcement learning

I'll start with the last question in your post:

  I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I highly doubt there will be any such theoretical work. The problem is that these variants of clipping (clipping rewards and clipping $Q$ values) fundamentally modify the task / the original objective. Once you clip either of those things, you fundamentally change what your agent is trying to optimize for from what the original goal was. I don't think it's ever going to be possible to get any rigorous, theoretical proofs about which one would be better in general. You'd likely have to start out with some very strong assumptions on the reward structure in the original task to have any hope of proving anything here, but such strong assumptions make you lose generality.

Intuitively... I think reward clipping feels "safer" to me more often than clipping $Q$-values. Clipping $Q$-values seems more aggressive, it could be viewed as some combination of clipping rewards (if you clip $Q$-values to $[-1, 1]$, you're still at the very least also clipping all rewards to that range), but additionally also putting a constraint on how far in the future you're looking (in some sense). This whole argument is very handwavy though

I suppose, slightly less handwavy, you could say that reward clipping is definitely "better" (in the sense that you don't deviate as much from the original objective) in environments where rewards of similar magnitudes can be collected frequently. I struggle to really think of a situation where clipping $Q$-values would be a clear favorite based on intuition. I wouldn't be surprised if clipping $Q$-values may turn out to be better after empirical evaluation in some cases, but it's difficult to say where that would be. It will also very much depend on what range is chosen. Clipping rewards to a range of $[-1, 1]$ is very different from clipping $Q$-values to the same range.


How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is
\begin{align}
E_{\pi_b}[G_t|S_t = s] &= E_{\pi_b}[R_{t+1} + G_{t+1}]\\
&= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\
\end{align}
You can see if you write out this recursively that this expectation depends on $\pi_b(a|s), \pi_b(a'|s'), \ldots$

If you collect trajectory with policy $\pi_a$ you would get expected return that depends on $\pi_a(a|s), \pi_a(a'|s'), \ldots$ Since these are two different policies then $\pi_b(a|s) \neq \pi_a(a|s)$ for some $(s, a)$. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return $G$ sampled by following policy $\pi_b$ to update policy $\pi_a$ because it's not sampled according to the proper distribution, if we did, we would be updating policy $\pi_a$ with biased gradient update that does not reflect how policy $\pi_a$ performed.


What are the differences between the DQN variants?

I assume OP understand basics of DQN - Time Difference (TD), target network and replay buffer. My experience is just my observations, it not necessarily generalize.

Some of those variants:

Double DQN: small difference in Time Difference propagation. Instead of propagating for maximum action of target network, Q taken form target network propagated for maximum action for current network. This claim to reduce the extrapolation error of TD, which prone to overestimation. My experience: not really helping.

Dueling DQN: difference in network architecture - Q separated into Advantage (Q(a)-Value) and Value (max_a Q(a)) which calculated separately and summed. My experience: not really helping.

Prioritized DQN: instead of uniform sampling of replay buffer it is sampled with probability depended on the error of TD. My experience: really helping, must have for any implementation.

Episodic memory DQN: this is really ambitious method. It combine memory-augmented network (differentiable dictionary, idea similar to Neural Turing Machine) with DQN. Difficult to implement correctly. My experience is ambiguous - I tried it in much simplified form, it was not quite clear to me if it was helping. I intend to return to it later. It also incur considerable performance cost.

Missing here:

C51 architecture - this could be important advance. It's attempt to get rid of regression in DQN(regressions are hard for  NN) and replace it with classification. Could be costly from performance point of view. Some reports claim it no worse then best performing Deep RL methods (with exception of tree-based method like Alpha Zero).


In any case, it appears that much of his past work on Github involves procedural generation. Which is AI... ish. Unless there's more to it, which we can't see because half the site is down.
This paper appears to offer an analysis of combining procedural generation with game AI.
From the abstract:

Populated and immersive game contexts require large numbers of minor,
background characters to fill out the virtual environment. To limit
game AI development effort, however, such characters are typically
represented by very simplistic AI with either little difference
between characters or only highly formulaic variations. Here we
describe a complete workflow and framework for easily designing,
generating and incorporating multiple, interesting game AIs. Our
approach uses high-level, visual Statechart models to represent
behaviour in a modular form; this allows for not only simplistic,
parameterbased variation in AI design, but also permits more complex
structure-based approaches. We demonstrate our technique by applying
it to the task of generating a large number of individual AIs for
computer-controlled squirrels within the Mammoth 1 framework for
game research. Rapid development and easy deployment of AIs allow us
to create a wide variety of interesting AIs, greatly improving the
sense of immersion in a virtual environment.

Here's an article from 2015 on AI and procedural generation, which discusses Angelina at length. This article links to a more in-depth article from 2013.
Here's an excerpt:

Cook gave ANGELINA the ability to learn about people so that it could
make games based on current events. Then Cook gave ANGELINA memory -
that is, the ability to keep track of the people it had learned about.
The memory's not a big deal, even though it led to a number of
philosophical disagreements around Cook's desk. ANGELINA's memory is
actually just a text file where it stores the names of all the people
it's heard of, alongside a number: a measure of its opinion of them
based on the things it's learned from internet chatter. It liked
Al-Assad more than May. It liked everyone more than May.


Why do we resize images before using them for object detection?

There are different questions and even different lines of thought here. Let's go through them
On resizing

Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN)
What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small image VS big image). Take into account that the kernel sizes do not vary although the image size does.

On keeping aspect ratio (or letterbox as some people like to say)

Why to keep aspect ratio? This is more of a philosophical question. It is believed that keeping the aspect ratio helps the nets to learn the natural variability in object sizes (say a person bounding box can not be super hight and super thin, because that would be a street light).

Why not to keep aspect ratio? If you resize without keeping the aspect ratio and the aspect ratio distortion is not mega super very huge, the networks will still learn. In other words, if your input images don't have crazy aspect ratios, then there is no difference between adding or not a bit of distortion. In fact, sometimes it will even act as a regularization or augmentation.

Conlusion
As long as your application is not too specific and your input images aspect ratios are bounded (this is, if  you train with images from any regular camera), you should not worry too much about this.
When to worry about this? When you train with huge vertical or horizontal images or if you train with images taken from some very specific devices like geophisical, radio, or optical sensors. In these cases you should pay special attention on how to resize or split an image. For example, with a recording of a radio sensor, if you resize with aspect ratio deformation, a wave from a specific frequency would transform to another because of the sine wave warp)


[Ref-some standard checks performed by programmers]

Speeding up Convolutional Neural Networks with Low Rank Expansions

From the abstract:

  The focus of this paper is speeding up the evaluation of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability.

  Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain.

  Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition, showing a possible 2.5x speedup with no loss in accuracy, and 4.5x speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.


Using ConceptNet5 to find similar systems to solve specific problems?

Hierarchical Temporal Memory should help with this. You would have to encode the text data into SDRs. You would then have a coincidence detector. Could you get the right information back out in the way you're trying to? I think so. I'm not fully learned in HTMs yet but check out Nupic (Open source).


Does fp32 & fp64 performance of GPU affect deep learning model training?

First off I would like to post this comprehensive blog which makes comparison between all kinds of NVIDIA GPU's.

The most popular deep learning library TensorFlow by default uses 32 bit floating point precision. The choice is made as it helps in 2 causes:

Lesser memory requirements
Faster calculations

64 bit is only marginally better than 32 bit as very small gradient values will also be propagated to the very earlier layers. But the trade-off for the gain in performance vs (the time for calculations + memory requirements + time for running through so many epochs so that those small gradients actually do something) is not worth it. There are state of art CNN architectures, which insert gradients midpoint and has very good performance.

So overall 32 bit performance is the one which should really matter for deep learning, unless you are doing a very very high precision job (which still would hardly matter as small differences due to 64 bit representation is literally erased by any kind of softmax or sigmoid). So 64 bit might increase your accuracy classification by $<< 1 {\%}$ and will only become significant over very large datasets.

As far as raw specs go the TITAN RTX in comparison to 2080Ti, TITAN will perform better than 2080Ti in fp64 (as its memory is double than 2080Ti and has higher clock speeds, BW, etc) but a more practical approach would be to use 2 2080Ti's coupled together, giving a much better performance for price.

Side Note: Good GPU's require good CPU's. It is difficult to tell whether a given CPU will bottleneck a GPU as it entirely depends how the training is being performed (whether data is fully loaded in GPU then training occurs, or continuous feeding from CPU takes place.)
Here are a few links explaining the problem:

CPU and GPU Bottleneck: A Detailed Explanation

A Full Hardware Guide to Deep Learning


Why is Short-Time Fourier Transform used for preprocessing audio samples?

Fourier transform is used to transform audio data to get more information (features).

For example, raw audio data usually represented by a one-dimensional array, x[n], which has a length n (number of samples). x[i] is an amplitude value of the i-th sample point.

Using the Fourier transform, your audio data will be represented as a two-dimensional array. Now, x[i] is a not a single value of amplitude, but a list of frequencies which compose original value at the i-th frame (a frame consists of a few samples).

See the image below (from wikipedia), the red graph is an original value of n samples before transformed, and the blue graph is a transformed value of one frame.


Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective deep to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term deep only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.
Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the universal approximation theorem. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective deep to start with).
In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as deep.


What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients
That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $t - o$.
Information on derivatives of cross entropy with sigmoid function and with softmax function. I would also suggest some more research on cross entropy loss functions beyond my links.


Why can't OCR be perceived as a good example of AI?

Whenever a problem becomes solvable by a computer, people start arguing that it does not require intelligence. John McCarthy is often quoted: "As soon as it works, no one calls it AI anymore" (Referenced in CACM).

One of my teachers in college said that in the 1950's, a professor was asked what he thought was intelligent for a machine. The professor reputedly answered that if a vending machine gave him the right change, that would be intelligent.

Later, playing chess was considered intelligent. However, computers can now defeat grandmasters at chess, and people are no longer saying that it is a form of intelligence.

Now we have OCR. It's already stated in another answer that our methods do not have the recognition facilities of a 5 year old. As soon as this is achieved, people will say "meh, that's not intelligence, a 5 year old can do that!"

A psychological bias, a need to state that we are somehow superior to machines, is at the basis of this.


How should I take into consideration the number of steps in the reward function?

How would you implement this "Number of Steps" cost?

What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).
As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, $\gamma$.
For Q-learning, the factor should be in the TD target calculation:
$$G_{t:t+1} = R_{t+1} + \gamma \max_{a'}Q(S_{t+1},a')$$
For Monte Carlo control, the factor appears more like this in the calculation of a return:
$$G_t = \sum_{k=0}^{T-t} \gamma^k R_{t+k+1}$$

would it be best to use an exponential functions to discount the reward at the current time step?

Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented "normal" Q-learning from equations like those above, it should already be there.


When training a DQN, how should we update the value of actions that were not taken?

The loss function for DQN algorithm is
\begin{equation}
L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2]
\end{equation}
Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ would have value
\begin{equation}
y = r + \gamma \max_{a'} Q(s', a', \theta^-_i)
\end{equation}
and we would have standard form of DQN loss. We calculate the gradient of that loss with respect to network parameters $\theta$, backprop it and slightly shift weights so to more accurately estimate $Q(s, a)$ for that specific state-action pair.

We never took other actions. Since we never took them we didn't get reward $r$ and we can't estimate $\max_{a'} Q(s', a', \theta^-_i)$ for those other actions. We don't want to change weights that had effect on calculating actions values for those other actions because we have no way of estimating how accurate they were. Standard approach is to set
\begin{equation}
y = Q(s, a;\theta_i)
\end{equation}
this way the loss for those other actions would be $0$ and it would result in not changing the weights that had influence in calculating their action values.

Setting target $y$ to $0$ for all other actions would mean that we want $Q(s, a)$ for them all to slightly shift to $0$. That would not be correct since we have no way of knowing their true value. I think you misinterpreted that part in the implementations.


What do the subscripts mean in $N_{t,n,\sigma,L}$?

Here is a paper with the mathematical definition of each term:

  Let Nt,n,σ,L be all target functions that can be implemented using a
  neural network of depth t, size n, activation function σ, and when we
  restrict the input weights of each neuron to be |w|1 + |b| ≤ L.


What does the depth of a decision tree depend on?

While this sounds obvious, the depth of the tree depends on how your algorithm builds the tree. For a fixed dataset $\mathcal{D}$, there're many algorithms, such as ID3, C4.5, CART, etc. (and their variants) to build your tree. For the most part, these algorithms recursively partition the dataset, so it's never possible to get a tree larger than $|\mathcal{D}|$. Large/deep trees are also prone to overfitting and are computationally expensive, so these algorithms typically prune the tree so it's much smaller than $|\mathcal{D}|$.
In fact,
Kearns and Mansour showed that under the weak learning assumption (i.e. at each node, there's a split that classifies the data at this node better than a random classifier, by $\gamma$) to achieve $\epsilon$ training error, it suffices to make $(1/\epsilon)^{\mathcal{O}(\log(1/\epsilon)/\gamma^2)}$ splits (and thus depth is upper bounded by this).
But of course, you can always cook up trees of arbitrary depth...


Can you help me understand how weight normalization works?

If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of w?)

Gradients being noisy means that they are "inconsistent" across different epochs / training steps. With that I mean that they'll sometimes point in one direction, later in a different (maybe the opposite) direction, etc., that the gradients at different time steps give inconsistent/conflicting information, that we don't consistently keep following the same direction through gradient descent but keep jumping all over the place. Note that this is across epochs, it's not across different dimensions within the same epoch.
So, for example, we might have a vector of gradients that looks like $[1, 1, 1, \dots, 1]$ at one time step (i.e., positive gradients in all dimensions), and the next training step get a gradient more like $[-1, -1, -1, \dots, -1]$ (this example is pretty much the "most extreme" kind of noise you can have, completely conflicting directions, in reality it'd generally be less extreme).
Such "noisy gradients" can exist because, in practice, we almost always use estimates of the gradient of our loss function, rather than the true gradients (which we'll generally not even be able to compute). For example, if we have a very large training dataset in a supervised learning setting, we'd ideally use the gradient of the loss function computed across the complete dataset. That tends to be computationally expensive (requires forwards and backwards passes for every single instance in the dataset), so in practice we'll often only use a small minibatch to estimate the gradient; this can result in widely different estimates of the gradient for different minibatches. (note: there can also be different reasons for using minibatches rather than full dataset, such as avoiding overfitting, but that's not too important to consider for this specific question)
A different example of a setting where we can have noisy gradients is in Deep Reinforcement Learning. There, computing the loss function itself is often rather noisy (in the sense that our own predictions, which tend to still be incorrect during the training phase, are a component of the targets that we're updating towards), so our estimates of the gradients will also be noisy.

Derivation for your second question, about the gradient of the loss with respect to $\mathbf{v}$:
We start with the following, from Equation (3) in the paper (this gradient you understand, right?):
$$\nabla_{\mathbf{v}} L = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_g L}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v}$$
Note that Equation (3) also gives:
$$\nabla_{g} L = \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$$
If we plug that into the previous Equation we get:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^3} \mathbf{v} \\
\end{aligned}
The $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ term that shows up before the minus can also be isolated in the term after the minus:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \\
\end{aligned}
Equation (2) tells us that $\mathbf{w} = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \mathbf{v}$, which means that $\frac{\mathbf{w}}{g} = \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$. Plugging this into the above leads to:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{g} \cdot \frac{\mathbf{w}}{g} \\
\end{aligned}
Below Equation (2), they also explain that $\vert \vert \mathbf{w} \vert \vert = g$, so we can rewrite the above to:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \\
\end{aligned}
In that last step, the $'$ symbol in $\mathbf{w}'$ denotes that we transpose that vector.
Finally, because we have that common $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ before and after the minus symbol, we can pull it out, multiplying it once by the identity matrix $\mathbf{I}$, and once by the remaining negated term. This is very similar to how you would simplify something like $a - ax$ to $a (1 - x)$ in "standard" algebra; the identity matrix $\mathbf{I}$ plays a very similar role here to the number $1$ in "standard" algebra:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \left( \mathbf{I} - \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \right) \\
\end{aligned}
Note: in all of the above, I kind of neglected to pay attention to order of matrix/vector multiplications and their dimensions. Probably the $\nabla_{\mathbf{w}} L$ term should by now already have been moved to after those large brackets, instead of being before those brackets. In this case, it should work out fine regardless because $\mathbf{I}$ as well as $\mathbf{w} \mathbf{w}'$ are square and symmetric matrices. The only resulting difference is in whether you get a row or a column vector out at the end.
This can now finally be rewritten as the Equation you mentioned that you didn't understand yet, Equation (4) in the paper.


Is it difficult to learn the rotated bounding box for a (rotated) object?

Cartesian Bias and Pipeline Efficiency

You are experiencing a techno-cultural artifact of Cartesian-centric imaging running all the way back to the dawn of coordinate systems. It is the momentum of practice as a consequence of applying Cartesian 2D coordinates to rasterize images appearing at the focal planes of lenses from the dawn of television and the earliest standards of raster based capture and display.

Although some work was done toward adding tilt to bounding rectangles in the late 1990s and since, from a time and computing resource conservation perspective, it is computationally and programmatically less costly to include the four useless triangles of pixels and keep the bounding box orthogonal with the pixel grid.

Adding a tilt angle to the bounding boxes is marginally competitive when detecting ships from a satellite only because two conditions offset the inefficiencies in that narrow domain. The ship appears as an oblong rectangle with rounded corners from a satellite positioned in geosynchronous orbit. In the general case, adding a tilt angle can slow recognition significantly.

Biology Less Biased

An interesting side note is that the neural networks of animal and human vision systems do not have that Cartesian-centricity, but that doesn't help this question's solution, since non-orthogonal hardware and software is virtually nonexistent.

Early Non-Cartesian Research and Today's Rasterization

Gerber Scientific Techonology research and development in the 1980s (South Windsor, Connecticut, U.S.) investigated vector capture, storage, and display, but the R&D was not financially sustainable for a mid-side technology corporation for the reasons above.

What remains, because it is economically viable and necessary from an animation point of view, is rasterization on the end of the system that converts vector models into frames of pixels. We see this in on the rendering  SVG, VRML, and the original intent of CUDA cores and other hardware rendering acceleration strategies and architectures.

On the object and action recognition side, the support of vector models directly from imaging is much less developed. This has not been a major stumbling block for computer vision because the wasted pixels at one tilt angle may be of central importance at another tilt angle, so there are no actual wasted input pixels if the centering of key scene elements is widely distributed in translation and tilt, which is often the case in real life (although not so much in hygienically pre-processed datasets).

Conventions Around Object Minus Camera Tilt and Skew from Parallax

Once edge detection, interior-versus-exterior, and 3D solid recognition come into play, the design of CNN pipelines and the way kernels can do radial transformation without actually requiring $\; \sin, \, \cos, \, \text{and} \, \arctan \;$ functions evaporate the computational burden of the Cartesian nature of pixel tensors. The end result is that the bounding box being orthogonal to the image frame is not as problematic as it initially appears. Efforts to conserve the four triangles of pixels and pre-process orientation is often a wasted effort by a gross margin.

Summary

The bottom line is that efforts to produce vector recognition from roster inputs have been significantly inferior in terms of resource and wait time burden, with the exception of insignificant gains in the narrow domain of naval reconnaissance satellite images. Trigonometry is expensive, but convolution kernels, especially now that they are moving from software into hardware accelerated computing paths in VLSI, is computable at lower costs.

Past and Current Work

Below is some work that deals with tilting with regard to objects and the effects of parallax in relation to the Cartesian coordinate system of the raster representation. Most of the work has to do with recognizing 3D objects in a 3D coordinate system to project trajectories and pilot or drive vehicles rationally on the basis of Newtonian mechanics.

Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs, James T. Klosowski, Martin Held, Joseph S.B. Mitchell, Henry Sowizral, and Karel Zikan, 1998

Sliding Shapes for 3D Object Detection in Depth Images, Shuran Song and Jianxiong Xiao, 2014

Amodal Completion and Size Constancy in Natural Scenes, Abhishek Kar, Shubham Tulsiani, Joao Carreira and Jitendra Malik, 2015

HMD Vision-based Teleoperating UGV and UAV for Hostile
Environment using Deep Learning, Abhishek Sawarkar1, Vishal Chaudhari, Rahul Chavan, Varun Zope, Akshay Budale and Faruk Kazi, 2016

Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds, Z Liu, H Wang, L Weng, Y Yang, 2016

Amodal Detection of 3D Objects:
Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images, Zhuo Deng, 2017

3D Pose Regression using Convolutional Neural Networks, Siddharth Mahendran, 2017

Aerial Target Tracking Algorithm Based on Faster R-CNN Combined with Frame Differencing, Yurong Yang, Huajun Gong, Xinhua Wang and Peng Sun, 2017

A Semi-Automatic 2D solution for Vehicle Speed Estimation from Monocular Videos, Amit Kumar, Pirazh Khorramshahi, Wei-An Lin, Prithviraj Dhar, Jun-Cheng Chen, Rama Chellappa, 2018


How to implement word2vec using Tensorflow 2.0 keras API?

Your question is really broad.. But there are many manuals and tutorials which describe how to do it step by step.

My personal favorite would be:

Practical Text Classification With Python and Keras - It has a very detailed explanation of every step of the implementation while remaining practical.

You can also try:

A Word2Vec Keras tutorial - This one focuses more on understanding how word2vec embedding works, but it also contains an implementation.


Detecting license plate using tensorflow

Such a task is getting easier to complete, but there are still difficulties with rotations, skewing, scaling---to name a few issues. Your network has the benefit of simplicity and lightness for the target hardware, but it may suffer under the above conditions.

So 237 images (181+56) may be small for a "generic" approach, depending on how representative and diverse the dataset is. Also, the dataset is unbalanced (the first class has twice as many examples), which causes bias in learning.

There are several ways to expand a base dataset:

Transform images and add them (with the same label, if supervised learning) to the dataset. Many libraries allow to rotate, skew, scale, or even blur images. Be careful as transforming needs be "reasonable", and changing image format over and over creates artifacts that can confuse the machine (e.g. too much JPEG savings on transformed images).
Generate synthetic data. Assuming license plates have a known format, it may be easy to generate images with good fidelity to real plates. This is not always possible, but license plates are formalized, so there should be only a handful patterns (typically plain, light, diplomatic, military vehicles).

Aside the dataset (potential) issue, the graph is fine. However, it may be worth trying different settings. It really depends on how you have ended up with the current graph. Removing pooling layers helps keeping more information, as well as a larger input image. 120x60 looks pretty small, and the comparison to fast-RCNN's RoI pooling layer looks odd, given that RoI comes after the feature maps. So a larger input image could give better results.


Could it be possible to detect text, symbols, and components directly in a scanned PDF file with a program like Tensorflow or another program?

Yes, that's possible.
I am working on a project in which I have to detect text in images. I did a quick search and found these two algorithms:

1. EAST: (Efficient and Accurate Scene Text Detector)
I am not sure if it is based on Machine Learning. Here are some links link1  link2 explaining how to use it with an example and using tesseract to extract the detected text.

2. CTPN: (Connectionist Text Proposal Network)
This algorithm is based on Machine Learning. Here is its link in github. In the description, you will find a link to a pre-trained model that you can use. Or simply, you can prepare your own data and train your own model.

For me, I tried both of them, and the CTPN model gave better results especially when the image contains large text.


Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy.
Value functions are always specific to some policy, which is why you will often see the subscript $\pi$ in e.g. $v_{\pi}(s)$ when there is a defined policy.
The policy evaluation step (step 2) in policy iteration converges to an accurate value function estimate for whatever the current policy is. In general this will not be an optimal value function, except on the last time that step 2 is used, and there is no change to the policy in the next stage policy improvement (step 3).
The policy improvement stage (step 3) can only usefully be run once for any value function estimate. The policy is updated to be greedy with respect to the value function from step 2 - that will always give the same results from the same value function estimate. If the value function is accurate then this new policy is guaranteed to be as good or better than the previous policy. Once step 3 is done, further improvements are only possible if the new policy is accurately evaluated.
Comparison with value iteration
The difference with value iteration is that it never accurately evaluates any interim policies. In value iteration, the implied policy changes every time the maximising action changes due to new value estimates. In the later stages, when the optimal policy has been found and is stable, then the value function will converge to the optimal value function. In value iteration, most of the interim value functions are not accurate, but when it becomes accurate it will also be the optimal value function.


Why does is make sense to normalize rewards per episode in reinforcement learning?

The "trick" of subtracting a (state-dependent) baseline from the $Q(s, a)$ term in policy gradients to reduce variants (which is what is described in your "baseline reduction" link) is a different trick from the modifications to the rewards that you are asking about. The baseline subtraction trick for variance reduction does not appear to be present in the code you linked to.

The thing that your question is about appears to be standardization of rewards, as described in Brale_'s answer, to put all the observed rewards in a similar range of values. Such a standardization procedure inherently requires division by the standard deviation, so... that answers that part of your question.

As for why they are doing this on a per-episode-basis... I think you're right, in the general case this seems like a bad idea. If there are rare events with extremely high rewards that only occur in some episodes, and the majority of episodes only experience common events with lower-scale rewards... yes, this trick will likely mess up training.

In the specific case of the CartPole environment (which is the only environment used in these two examples), this is not a concern. In this implementation of the CartPole environment, the agent simply receives a reward with a value of exactly $1$ for every single time step in which it manages to "survive". The rewards list in the example code is in my opinion poorly-named, because it actually contains discounted returns for the different time-steps, which look like: $G_T = \sum_{t=0}^{T} \gamma^t R_t$, where all the individual $R_t$ values are equal to $1$ in this particular environment.

These kinds of values tend to be in a fairly consistent range (especially if the policy used to generate them also only moves slowly), so the standardization that they do may be relatively safe, and may improve learning stability and/or speed (by making sure there are always roughly as many actions for which the probability gets increased as there are actions for which the probability gets decreased, and possibly by making hyperparameters easier to tune).

It does not seem to me like this trick would generalize well to many other environments, and personally I think it shouldn't be included in such a tutorial / example.

Note: I'm quite sure that a per-episode subtraction of the mean returns would be a valid, albeit possibly unusual, baseline for variance reduction. It's the subsequent division by standard deviation that seems particularly problematic to me in terms of generalizing to many different environments.


How would one debug, understand or fix the outcome of a neural network?

If the observation that the neural network saw was recorded, then yes the prediction can be explained.

There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning models predictions. It can be used to establish why a machine learning model made a prediction, help a data scientist debug a model, and help a data scientist improve the accuracy of a specific model. LIME can be used to explain the predictions of any neural network including CNNs, RNNs, and DNNs.


What is the point of using 1D and 2D convolutions with a kernel size of 1 and 1x1 respectively?

Traditional CNNs used for image classification (and related tasks) are composed of 1 or more fully connected layers (FCs), after the convolutional and pooling layers, which take as input the features extracted from the convolutional and pooling layers, in order to perform classification or regression.
One problem with FCs in CNNs is that the number of parameters can be very big, with respect to the number of parameters in the convolutional layers.
There are tasks, such as image segmentation, where this big number of parameters is not really needed. An example of a neural network that does not make use of fully connected layers but only uses convolutions, downsampling (aka pooling), and upsampling operations is the U-net, which is used for image segmentation. A neural network that only uses convolutions is known as a fully convolutional network (FCN). Here I give a detailed description of FCNs and $1 \times 1$, which should also answer your question.
In any case, to answer your question more directly, $1 \times 1$ convolutions have been used for image segmentation tasks, i.e. dense classification tasks, i.e. tasks where you want to assign a label to each pixel (or a group of pixels), as opposed to sparse classification tasks such as image classification (where the goal is to assign 1 label to the whole image). Moreover, in comparison with FC layers, they have fewer parameters and, more importantly, the number of parameters in an FCN does not depend on the dimensions of the images (as in the case of traditional CNNs), which is a good thing (especially, when your images have high resolutions), but typically it depends on the number of kernels and instances (of objects), in the case of instance segmentation.
The FCN paper discusses this reduction of the number of parameters (and computation time), so you should probably read this paper for more details.


Is the bias also a "weight" in a neural network?

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias.
One notable text using this convention is Understanding Machine Learning: From Theory to Algorithms by  Shai Shalev-Shwartz and Shai Ben-David. In section 20.1 there is a diagram of a neural network where a neuron outputting a constant value is added to each layer which you might find helpful.
To understand why this works, suppose the outputs of the previous layer are $u_1, \dots, u_n, u_{n + 1}$, where $u_{n + 1}$ is always $1$. Then a neuron in the next layer (without a bias) computes
$$ \sigma\left(\sum_{i = 1}^{n + 1} w_i u_i \right) = \sigma\left(\sum_{i = 1}^n w_i u_i + w_{n + 1}\right),$$
where $\sigma$ is the activation function. So, the weight $w_{n + 1}$ just serves as the bias because it is multiplied by $u_{n + 1} = 1$.


What is the difference between deep learning and shallow learning?

That article only mentions "shallow learning" in the title and it mentions "shallow" to refer to the fact that deep learning models are not really learning any "deep" concepts, where "deep" here means "philosophically deep". So, I think the title is just (fairly?) provocative.

Currently, in machine learning, the expression "shallow learning" isn't really standardized, as opposed to deep learning, which refers to learning, with gradient descent and back-propagation, from (typically) huge amounts of data with neural networks. Nevertheless, "shallow learning" may occasionally refer to everything that isn't deep learning (e.g. traditional machine learning models, such as support vector machines), but most likely it refers to learning in neural networks with only a small number (0-2) of hidden layers (i.e. non-deep neural networks).

Note that the difference between deep and shallow neural networks isn't really clear. Some people may consider neural networks with only 1-2 hidden layers already deep, while others may consider only neural networks with e.g. 5-10 hidden layers deep. This also shows that deep learning isn't actually well-defined too.

The other linked article actually says

  CNN performance was compared to that of conventional (shallow) machine learning methods, including ridge regression (RR) on the images’ principal components and support vector regression.

So, in this article, they use "shallow learning" to refer to traditional (or conventional) machine learning models, which confirms what I said above.


Why is the reward signal normalized in openAI's REINFORCE?

It's just a hack to normalize the rewards to make optimization less problematic in cases where the reward might have very large/small values. It's fine because scaling the rewards by a constant factor won't change what the optimal policy is. You can think of the mean reward as the baseline.


What is the formula used to calculate the loss in the FaceNet model?

The loss function used is the triplet loss function.

Let me explain it part by part.

Notation

The $f^a_i$ means the anchor input image. The $f^p_i$ means the postive input image, which corresponds to the same people as the anchor image. The $f^n_i$ corresponds to the negative sample, which is a different person(input image) then the anchor image.

The formula explained step by step

The first part, $||f^a_i - f^p_i||^2_2$ basically calculates the distance between the anchor image output features and the postive image output features, which you want the distance to be as small as possible as the input is the same person. For the second part, $||f^a_i - f^n_i||^2_2$ , it calculates the distance of the output features of the anchor image and the negative image. You wnat the distance to be as large as possible as they are not the same person. Finally, the $\alpha$ term is a constant(hyperparameter) that adds to the loss to prevent negative loss.

How it works

The loss function optimizes for the largest distance between the anchor and negative sample and the smallest distance of the positive and anchor sample. It cleverly combines both metrics into one loss function. It can optimize for both case simultaneously in one loss function. If there is no negative sample, the model will not be able to differciate different person and vice versa.

Hope I can help you and have a nice day!


Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?

The approach you describe is called neuromorphic computing and it's quite a busy field.

IBM's TrueNorth even has spiking neurons.

The main problem with these projects is that nobody quite knows what to do with them yet.

These projects don't try to create chips that are optimised to run a neural network. That would certainly be possible, but the expensive part is the training not the running of neural networks. And for the training you need huge matrix multiplications, something GPUs are very good at already. (Google's TPU would be a chip optimised to run NNs.)

To do research on algorithms that might be implemented in the brain (we hardly know anything about that) you need flexibility, something these chips don't have. Also, the engineering challenge likely lies in providing a lot of synapses, just compare the average number of synapses per neuron of TrueNorth, 256, and the brain, 10,000.

So, you could create a chip designed after some neural architecture and it would be faster, more efficient, etc …, but to do that you'll need to know which architecture works first. We know that deep learning works, so google uses custom made hardware to run their applications and I could certainly imagine custom made deep learning hardware coming to a smartphone near you in the future. To create a neuromorphic chip for strong AI you'd need to develop strong AI first.


One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent.

In the MAB problem, the number of trials is typically finite owing to some sort of budget in the number of trials. Hence, this is less of problem. However, by all means discounts are still valid and helpful in cases where the analysis in asymptotic in the number of trials.


What is the difference between artificial intelligence and computational intelligence?

What is the difference between Artificial Intelligence and Computational Intelligence?

The short answer is that they are two parallel research efforts working on similar problems, but with different methodologies and histories. Essentially, they study similar things, but with different tools. In the modern context, computational intelligence tends to use bio-inspired computing, like evolutionary and genetic algorithms. AI tends to prefer techniques with stronger theoretical guarantees, and still has a significant community focused on purely deductive reasoning. The main area of overlap is in machine learning, especially neural networks.

The longer answer is that your source from 1948 says they are synonyms in part because it predates the split in the research community, which took place later.
The two communities have always some overlap in topics, but in my experience, mostly are skeptical of each other's methodologies, and mostly publish in separate journals. Some authors consider CI to be a subset of AI however, particularly those writing in the 1990s.
Example topics that are solidly in AI but definitely not in CI are logical and expert systems, and statistical approaches to machine learning like regression.
Example topics that are solidly in CI but perhaps not in AI (depending on whether one views CI as a subset of AI or not) are genetic programming, fuzzy logic, and ant colony optimization.
As a rule, AI-rooted techniques have better theoretical guarantees, and better developed theory in general (there are exceptions though). For example, Fuzzy Logic has been strongly criticized for the lack of a solid theoretical foundation (good modern summary here), as have genetic and evolutionary approaches (most famously, both lack a proof of convergence within finite time to a global optimum on a smooth surface, even though they do quite well in practice).
CI-rooted techniques nonetheless often see major performance advantages in specific problems (see, for instance, deep learning results), and tend to have a strong experimental and engineering tradition. The No Free Lunch theorems are often used to justify their use when theoretical certainty is missing. Basically, the theorems say that, in learning and optimization problems, a technique can only perform well on a problem by performing poorly on some other problem. CI authors argue that there are some problem domains in which their techniques work well (which must be true, because simpler algorithms like hill-climbing outperform them on simple problems).
Check out this paper for lots more references on CI, or this book for a list of core topics in the field.


Is there a rigorous proof that AGI is possible, at least, in theory?

A strong reason why people think the mind can be implemented on a Turing Machine stems from the Computational Theory of Mind (CTOM), which is the leading theory of mind for now.
There are lots of reasons for supporting the CTOM, one of which being that the language of belief/desire psychology (propositional attitudes over mental representations) seems to fit nicely to a computational framework.
But most simply is that the computation analogy is very helpful in fields such as psychology and neuroscience. When we know of an input/output pair, but don't know how it is implemented, we could say "its performing the relevant computation".
And since Turing showed that any computation can be performed on an appropriate Turing Machine, the natural extension is that the mind can be implemented on a computer.
However, the CTOM is more of a useful idea than a complete theory. We still don't know how to analyze thought in a logical syntax, which can be implemented in a computer. And we also don't know how/why "computation" (whatever that means in this sentence) is performed in the brain.


How should I define the loss function for a multi-object detection problem?

Okay so your CNN model is taking an Image and outputting the bounding boxes for them. That means the last layer of CNN model must be having four outputs which are generating real numbers. This is a regression problem.
In that case, you can take L1 loss (mean absolute error) or L2 loss (mean square error) as your loss function. I have created a similar project, I have used L1 loss.
Suppose your input image is x and predicted_bb is the model output and real_bb is your original bounding box for image x. Then you should proceed as follows
predicted_bb = model(x)

# CALCULATE LOSS BETWEEN THE ACTUAL & PREDICTED BB COORDINATES

loss_bb = torch.nn.functional.l1_loss(predicted_bb, real_bb, reduction="none").sum(1)

# SET GRADIENTS TO ZERO

optimizer.zero_grad()

# BACKPROPOGATE THE LOSS

loss.backward()

For tensorflow keras
predicted_bb = Dense(4, activation='relu')(x)

model = Model(inputs = image_input, outputs = [predicted_bb])
model.compile(loss=['mae'], optimizer='adam', metrics =['accuracy'])


Are deep learning models more prone to overfitting than machine learning ones?

Your reasoning isn't wrong. Deep Neural Networks (DNNs) have a much larger capacity than simpler ML algorithms (excluding NNs) and can easily memorize even a very complex dataset and overfit.

DNNs, however, are so effective because they usually are applied on tasks that are harder, so it's not as easy to overfit. For example an image classifier might be trained on a dataset with millions of images; a task much harder to overfit on.

In cases where this isn't possible (e.g. an image classification task with a couple thousand images), transfer learning is used. You can initialize your weights from a model pre-trained on a large dataset, use its already-trained feature extraction layers and simply fine tune the last layer.

Data augmentation also helps a lot here, which effectively increases size of the training set and discourages the DNN from memorizing the samples. It is so effective that it is used even in large datasets, where it is harder to overfit.

Additionally, DNNs employ several methods to prevent them from overfitting. The most prominent of these is dropout, which is a very effective regularizer. Batch Normalization has also proven an effective regularizer. SGD allows you to explore more parameters than GD, which also is effective against overfitting. Finally, early stopping and parameter norm penalties aren't uncommon in DNNs.


Can neural networks deal with unbounded numbers as inputs?

Depending on the activation functions in the first layer, this is not a real problem, many functions such as $\tanh(x)$ and $\sigma(x)$ (sigmoid) have asymptotic upper and lower bounds, so the huge input values swamp all other inputs in the corresponding neurons, but the output is well-behaved.

However, a conversion before the input layer may be appropriate for some kind of data. For example, you might want to take the $\log(x)$ of the input values such that the initial weight networks work with the relation of the values instead of their absolute magnitude. This may be useful for audio data such as speech or music and many other measurements of signals that may be attenuated before the measurement.

If your numbers don't actually represent magnitudes of some kind but arbitrary sequences of digits, you should use a network that can deal with sequences, of course.


Smaller interest area for images than the size of the image in classification neural networks

"The problem here is that if i use the whole image when training, my model will not learn that the interest area is only in the center of the image" Yes, it will, if you have enough samples and they are all classified based on this area only.

"but on the other hand if i crop the images to be of size 32x32 i am loosing a lot of information which can help the model to train on" . I don't understand what information you are loosing if you only care about the center 32x32 area.

In the end, you need to ask yourself how will the test input look like and work based on that. If you only care about an area of 32x32 which is always is (and will be) at the center of all images, ignore everything and train and evaluate based just on that area (why add extra unnecessary calculations to the model?). If the size and location of the area might change, don't crop the image.

Bottom line, the test set and the train set need to be of the nature (shape) for the model to work properly.


Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs:

https://github.com/openai/gym/pull/1789

  Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been inconsistent and incorrect, returning the furthest hit result instead of closest.

...

  Before fix - lidar traces through ground, and hits the side of a pit,
  giving the agent the impression of a "phantom canyon" in front of the
  pit, that only appears as it approaches the pit:
  https://i.imgur.com/XKPnRTR.png

...

  After fix - lidar is stopped by terrain, even when another object is behind it: https://i.imgur.com/Gg8B5BD.png

...

  After triple checking the docs - I've submitted a minor tweak (returning -1 instead of 1 for an object that should be ignored) - it now seems legs are correctly ignored, and the traces are accurate in all situations!

...

  It seems to me that solutions to BipedalWalkerHardcore-v2 have not just learned to deal with the complex environment - but advanced a step ahead, and are able to deal with the complex environment and sensory hallucinations causing them to jump at the slightest hint of a cube, and keep running even when it looks like the ground is not visible below their feet, relying more on the touch sensor than the lidar, or perhaps recognising the difference in "shape" between a real pit and a "fake pit" (A real pit has a floor)

BipedalWalkerHardcore-v2 has been bumped to BipedalWalkerHardcore-v3 with these fixes as of Jan 31, 2020.

You might want to try retraining your agent now! (although it is still a difficult environment to solve)

To expand on why DDPG doesn't solve it, when although buggy, BipedalWalkerHardcore-v2 is solvable: The solution landscape to this problem is as full of pits as the environment itself. To learn to leap over a pit in the environment for example, the agent must perform a complex sequence of actions that is difficult to discover by random chance. Each time it fails, it learns that being close to a pit is highly likely to result in a large penalty, and in an effort to maximise it's rewards, will often remain stationary with a naive method like DDPG as the rewards for doing that are higher than trying and falling into the pit once more. In short, vanilla DDPG lacks enough exploration power to find the complex series of actions required before it converges on not going near the pit. Not to mention all of the other things it needs to learn to be successful.

Of the very few published examples that have solved it, one used Evolutionary Strategy - a gradient free method essentially trying millions of policies and evaluating them, and a custom A3C method that was tailored to solve this particular environment. Both had high computational and sample requirements.
I speak from experience as I have personally solved this environment with a RL exploration algorithm that generalises to other environments, solves it in 4 hours on a single cpu, and can be used with any off-policy RL algorithm, but unfortunately I'm unable to publish it because of IP with the company I work for.

TLDR; The chance of vanilla DDPG solving it are infeasibly small.