What are the most compact Real Time-Strategy Games?

This is an interesting question. One good place to start would be the gargantuan catalogue of rather simple RTS flash games and the like. There are many sites that have 100s of such titles, and it would not be too difficult to build the framework for these types of games(there are open source tools that can help as well).

As far as "real" mainstream games, the pickings are rather slim for simple benchmark RTS titles. One that comes to mind is Rome: Total War. At least in exhibition, there is not much to strategize over besides who to attack and from where. Also, as I assume this relates to starcraft, one good simplified predecessor to starcraft would be the Dune series, and in particular Dune 2.


The problem with the Gambler's Problem in RL

The intuitive explanation is that there are many equally good "optimal" policies. This is mentioned at the end of the example problem description you posted. My gut says that the family of optimal policies would be any policy from the double/nothing family. So, for example, if you bet 25 on the first bet instead of 50, I think your overall chances of winning should be the same as if you bet 50, it'll just take longer in expectation. The resulting family of policies will look more undulating than the one in the book.

As Neil notes, for low values of $p$, the probability that you win a gamble, it is the case that there is a unique optimal policy.


Why does the machine learning algorithm need to learn a set of functions in the case of missing data?

@The Pointer the $2^n$ came from the question: How many function do we need to have if each of the $n$ inputs can be missing?
example: $f_1(\text{missing}, x_2, x_3, \dots, x_n)$ for $x_1$ missing
$f_2(x_1, x_2, \text{missing}, x_4, \text{missing}, \dots, x_n)$ for $x_3$ and $x_5$ missing.
So this problem is a combinatorial one and the event for each $x_i$ is Missing or Not.
Each function corresponds 1-by-1 with a possible set $(x_1, \text{missing}, x_2, \dots, x_n)$. So how many sets can you form $2^n$. Why?
This formula comes back to like tree developing. First variable $x_1$ missing or not (2 possbile events). Now after that, for EACH of these events 2 possbile events for $x_2$ so in total 2 \times 2 (2 for $x_1$ , 2 for $x_2$ for each $x_1$'s event) and etc., $2 \times 2 \times 2 \times \dots$ $n$ times = $2^n$.


What does the Critic network evaluate in Actor Critic?

The same argument of a state-value function int Reinforcement Learning: An Introduction Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping.

  Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated. This is a useful distinction, for only through bootstrapping do we introduce bias and an asymptotic dependence on the quality of the function approximation. As we have seen, the bias introduced through bootstrapping and reliance on the state representation is often beneficial because it reduces variance and accelerates learning.


Is PReLU superfluous with respect to ReLU?

Lets assume we have 3 Dense layers, where the activations are $x^0 \rightarrow x^1 \rightarrow x^2$, such that $x^2 = \psi PReLU(x^1) + \gamma$ and $x^1 = PReLU(Ax^0 + b)$

Now lets see what it would take to conform the PReLU into a ReLU

$\begin{align*}
PReLU(x^1) &= ReLU(x^1) + ReLU(p \odot x^1)\\
&= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\
&= ReLU(Ax^0+b) + ReLU((eye(p)A + eye(p)b)x^0)\\
&= ReLU(Ax^0+b) + ReLU(Qx^0+c) \quad s.t. \quad Q = eye(p)A, \ \ c = eye(p)b\\
&= [I, I]^T[ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
\implies x^2 &= [\psi, \psi][ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
&= V*ReLU(Sx^0 + d) \quad V=[\psi, \psi], \ \ S=[A, Q] \ \ d=[b, c]
\end{align*}$

So as you said it is possible to break the form of the intermiediary $PReLU$ into a pure $ReLU$ while keeping it as a linear model, but if you take a second look at the parameters of the model, the size increase drastically. The hidden units of S doubled meaning to keep $x^2$ the same size $V$ also doubles in size. So this means if you dont want to use the $PReLU$ you are learning double the parameters to achieve the same capability (granted it allows you to learn a wider span of functions as well), and if you enforce the constraints on $V,S$ set by the $PReLU$ the number of paramaters is the same but you are still using more memory and more operations!

I hope this example convinces you of the difference


Understanding the intuition behind Content Loss (Neural Style Transfer)

For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

  a) Relu:0 does this mean Relu activation has not yet been applied?

It has been applied.

  b) Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

Yes.

  c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

The first dimension in keras is used for representing batch dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the batch_size. When computing the content loss, though, you have a single image you want to perform style transfer to, so batch_size=1. That's why you see the unit in the beginning of the tensor's shape.

  d) Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

It outputs the tensor containing all the pixel values that are outputted from that layer.

  With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

  a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

I think you're confusing things a bit. You are getting the pixel values, just in a different shape. Instead of stacking them in a $64 \times 64 \times 515$ array you are stacking them in a $1096 \times 515$ array. Exactly the same information is stored in a different shape. Think of it like transforming a $10 \times 2$ matrix to a $5 \times 4$ one. It contains the same information, in a different shape.

  b) Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...

  Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

If you have the output image as a tensor, let's say y. You can convert it to numpy through keras.backend.eval:

import keras.backend as K

y_arr = K.eval(y)

There are many ways to save the image, including scikit-image, scipy, opencv, PIL, matplotlib, etc. I'll use the last:

import matplotlib.pyplot as plt

plt.imsave('my_image.png', y_arr)

Note that in order to work y_arr should have a shape of either (height, width, 3) (if it is an RGB image) or (height, width) (if it is grayscale).


Whats the name of the value that you add or subtract from a minimax tree node?

I think you mean the leave nodes only which are changed. The other nodes in the tree are calculated during calculating the best move with this tree. The values at the leaves are called utility values in Russel and Norvig's "Artificial intelligence: a modern approach". Some times it is called heuristic value; see https://en.wikipedia.org/wiki/Minimax.


Can abstractive summarization be achieved using neural networks?

The ability to re-frame summarization as a problem for ANN is rather dependent on what kind of output you're looking for: you mentioned 'salient parts of the text'.

One possibly is to use a deep learning approach that first chunks together words that belong in the same phrase as a single 'feature'.

Another possibility is to identify both key words and relations between them. Here is some previous work on using neural nets for relational learning.


What does it mean for AlphaZero's network to be "fully trained"

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you eventually reach optimal performance, you can't play more optimal than what optimal is (not saying that it reached optimal performance but possibly something close to optimal for it approximation abilities). So probably combination of those two leads for the performance increase to reach it's limit.


How can I use one neural network for both players in Alpha Zero (Connect 4)?

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e., wining connect4 game.

Therefore, we could say that from agent1's point of view, he is player1 and the player2 is agent2. The same way, from agent2's point of view, he is player1 and the player2 is agent1.

Here, we can use the same definition as you proposed:

"let's say player1's pieces on the board are represented as -1, empty board as 0 and player2's pieces as 1"

When it is the turn for agent1, he will receive an observation (representing the board configuration) from the environment. In this case, agent1's pieces are represented as -1, empty board as 0 and for player2's (agent2's) pieces as 1. Then, you could use the neural network to compute, e.g., the next action for agent1.

When it is the turn for agent2, he will receive another observation from the environment, but this time you represent its own pieces as -1, while the agent1's (player2's from agent2's point of view) as 1.

With this approach, the neural network will always receives as an input a representation in which the pieces 'it' controls are -1, while its rival's pieces are 1 and empty space 0. From the neural network's point of view, he does not care about which agent is playing as the actual observation is indistinguishable to calculate the next action.

Therefore, you will have to generate two different observations, in which you invert the sign of the ones on it, depending on which turn it is (agent1's or agent2's turn).

For example:


Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$?

it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.

if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer


What is the difference between latent and embedding spaces?

Embedding vs Latent Space
Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a number of different terms for the same or similar concepts.
"Latent space" and "embedding" both refer to an (often lower-dimensional) representation of high-dimensional data:

Latent space refers specifically to the space from which the low-dimensional representation is drawn.
Embedding refers to the way the low-dimensional data is mapped to ("embedded in") the original higher dimensional space.

For example, in this "Swiss roll" data, the 3d data on the left is sensibly modelled as a 2d manifold 'embedded' in 3d space. The function mapping the 'latent' 2d data to its 3d representation is the embedding, and the underlying 2d space itself is the latent space (or embedded space):

Synonyms
Depending on the specific impression you wish to give, "embedding" often goes by different terms:

Term
Context

dimensionality reduction
combating the "curse of dimensionality"

feature extraction feature projection feature embedding feature learning representation learning
extracting 'meaningful' features from raw data

embedding manifold learning latent feature representation
understanding the underlying topology of the data

However this is not a hard-and-fast rule, and they are often completely interchangeable.


Why is an expectation used instead of simple sum in GANs?

I'm assuming you mean the original loss explained in the original GAN paper. This practice was by design and for interpretation. An expectation is what your expecting to get, which is generally a good objective when your are sampling. Note this isn't just in GANs, it's in most objective functions used in a widespread of problems.
In general practice, though, dividing is good because it works as a normalization. Let's say you train one batch with 10 elements and the next with 8. Wouldn't you want each training example to be weighted equally?
Also, even if you use equal-sized batches, then the "division" you mention gets eaten up by the learning rate if you're using some form of gradient scheme.


Is an SVM the same as a neural network without a hidden layer?

First, what makes the neural network different than linear regression is the non-linearity (activation function), not the number of layers. So, a neural network with $n$ layers with no non-linearities is still the same as linear regression. Second, SVM finds the hyperplane of maximum margin. You are not guaranteed to find that hyperplane that has maximum margin with a neural network (using non-linearities).


What is the difference between a convolutional neural network and a regular neural network?

TLDR:
The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting).

See the following chart that depicts the several neural-networks architectures including deep-conventional-neural-networks: .

Neural Networks (NN), or more precisely Artificial Neural Networks (ANN), is a class of Machine Learning algorithms that recently received a lot of attention (again!) due to the availability of Big Data and fast computing facilities (most of Deep Learning algorithms are essentially different variations of ANN).

The class of ANN covers several architectures including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) eg LSTM and GRU, Autoencoders, and Deep Belief Networks. Therefore, CNN is just one kind of ANN.

Generally speaking, an ANN is a collection of connected and tunable units (a.k.a. nodes, neurons, and artificial neurons) which can pass a signal (usually a real-valued number) from a unit to another. The number of (layers of) units, their types, and the way they are connected to each other is called the network architecture.

A CNN, in specific, has one or more layers of convolution units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighborhood) share their weights.

The convolution units (as well as pooling units) are especially beneficial as:

They reduce the number of units in the network (since they are many-to-one mappings). This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network.
They consider the context/shared information in the small neighborhoods. This future is very important in many applications such as image, video, text, and speech processing/mining as the neighboring inputs (eg pixels, frames, words, etc) usually carry related information.

Read the followings for more information about (deep) CNNs:

ImageNet Classification with Deep Convolutional Neural Networks
Going Deeper with Convolutions

p.s. ANN is not "a system based loosely on the human brain" but rather a class of systems inspired by the neuron connections exist in animal brains.


How do transformers understand data and answer custom questions?

GPT-3 (and the likes) don't really have any understanding of the semantics nor pragmatics involved in the language. However, they are good at constructing text content similar to the contents created by a person (when the texts and the concepts are not too "complicated").


In sequence-to-sequence, why is the output of the decoder used as its input?

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:
$\begin{align*}
p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\
 =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i})
\end{align*}
$

This can be sampled by samping each of the conditional is ascending order. So thats exactly what theyre trying to imitate. You want the second output dependant on the sampled first output, not its distribution.

This is why the hidden state is NOT good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.

Note: In training they use ground truth as input by default because its working under the assumption the model shouldve predicted the correct word and if it didnt the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls)


How can I evaluate the performance of a system that generates text?

This is a tricky issue. I assume you are using transition probabilities to pick the next suitable word, so you could use some other corpus data, derive probabilities from it, and compare  those to your system. Not very satisfactory, though, as you might end up evaluating the system in a circular way, deriving your test data in the same way as you generate your output.

A better, if somewhat more laborious evaluation would be to present the trigger sequence to human beings (on a fairly large scale), and then measure the overlap between your system and the human choices. Of course, with the you will get a very large set of possible replies, but the round yellow would be a lot more restricted (and it would make more sense to include that kind of sequence in an evaluation anyway).

The figure you will get might be quite small (depending on the range of possible options), but that is to be expected, so your system could score well even with a small overlap only. That's how I would approach this problem.


Could we code in Natural language once we pass the Turing test?

No.  Humans pass the Turing Test but cannot express themselves with enough precision and little enough ambiguity to code well in English (natural language).  No machine will improve on that until it can greatly improve on human-level clarity in expressing their purpose when programming.

Is it possible to program in natural language?  Yes, I think so.  But it would require superhuman intelligence to anticipate all the possible confusions that might arise when a word is not sufficiently precise or accurate to represent an software activity or mechanism.  This probably would require adopting a rigorous convention in choosing a subset of unambiguous words to form a "natural" programming language (a kind of creole).  Coding consistently using only those words would require discipline that exceeds the ability of most humans, methinks.


How do I prove that the MSE is zero when all predictions are equal to the corresponding labels?

This is very easy to prove.
Let's first prove that, if $\hat{y}_k = y_k$, then the $E = 0$. I will leave all steps, so that it's super clear.
\begin{align}
E
&=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 \\
&=\frac{1}{2}\sum_k(y_k - y_k)^2\\
&=\frac{1}{2}\sum_k(0)^2\\
&=\frac{1}{2}\sum_k 0\\
&=\frac{1}{2} 0\\
&=0\\
\end{align}
To prove the other way around, i.e. if $E = 0$, then $\hat{y}_k = y_k$, you can do as follows
\begin{align}
\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2
&=E\\
&=0
\end{align}
Recall now that any number squared is non-negative (i.e. positive or zero). Given that $(\hat{y}_k - y_k)^2 $ is non-negative, then $\sum_k(\hat{y}_k - y_k)^2$ is a sum of non-negative numbers. The only way that a sum of non-negative numbers is equal to zero is if all numbers are zero, so we must have $\hat{y}_k = y_k$ (because any non-zero number squared is non-zero).
(Note that $E$ is the mean squared error, i.e. a loss function, and it's not the back-propagation algorithm, which is just the algorithm that you use to compute partial derivatives of $E$ with respect to the parameters of the model, which are not even visible in the way you wrote $E$).


Training Custom object detection network using tensor-flow object detection API?

I have found a hack to integrate keras with tensorflow object detection API. This hack works if you have trained the keras classification model with tensorflow backend. If above is the case you can extend the classification model to a object detection model by first converting the keras checkpoint to a tensorflow checkpoint then in the object detection API write new feature extractor layers using tf.keras (Keras is now part of core tensorflow starting from version 1.4). you can simply copy paste your layer definitions and pass the path to the converted checkpoint in the config file and it will work.


What is the difference between a problem representation and problem modelling?

I would say these terms are often used interchangeably in AI. When they differ, I would say that problem modeling means finding a mathematical description of the problem, while problem representation means finding a particular way to represent that mathematical formalism.

For example, a list of numbers can be stored (represented) with a linked list, and array list, a hash table, or a self-balancing tree. All of them can produce a faithful model of the list, but if what you want to do is find the order that elements were entered in, the array list or linked list is far faster and more natural. If what you want to do is determine whether certain pieces of information are present in the list, the hash table is fastest. If what you want to do is find ranges of similar elements, the tree is fastest. Essentially, representational choices are engineering problems, while modelling choices are scientific problems.


Have evolutionary algorithms been used for engineering design?

Examples of GA implementations:

NASA’s spacecraft evolved antenna
Solution for TSP with a big number of connections. Here is an example with Python and here with C#.
From my experience, GA can be even used for a training of small artificial neural networks, but with less efficiency than with traditional deep learning approaches

Also, I heard about successful projects with using GA for scheduling, packing and  logistics
Therefore, everything, that can be represented in a form of digital chromosomes (can be serialized to and deserialized  back from a sequence of bits) and we can define a function to evaluate its efficiency – fitness or reward function can be optimized by using of GA


The real approach of creating AI for a fighting robot in an MMO

I would set up a list of goals for your bot. These could be 'maintain a minimum level of health', 'knock out human player', 'block way to location X', etc. This obviously depends on the domain of your MMO.

Then you can use a planner to achieve these goals in the game. You define a set of actions with preconditions and effects, set the current goal, and the planner will work out a list of actions for the bot to achieve the goal. You can easily express your actions (and the domain) in PDDL.

Examples for actions would be 'move to location X', 'eat X', 'attack player X'. A precondition of 'attack player X' could be 'health(X) is low', and an effect could be 'health(X) is reduced by Y'. There are different ways of expressing these depending on the planner's capabilities.

The beauty of this is that you don't actually have to explicitly code any behaviour. You describe the domain, and tell the bot what it should achieve, and what capabilities it has. The actual behaviour then emerges out of that description. If the bot only attacks a player if the player has lower health, then observing the player eat (and thus up their health) could result in the bot eating (to push its own health above the player's so that it can attack) — but you have not told the bot directly to do that.

For a starting point, go to http://education.planning.domains/ for a list of resources.

If you only have a few actions available, it might appear predictable to a human user, but with a variety of goals and actions, this will quickly become more complex and seem more 'intelligent'.

Update: Here is a link to a paper, Applying Goal-Oriented Action Planning to Games, which describes how this can be applied in a game.


Strategy of using intermediate layers of a neural network as features?

This is an example of Transfer Learning, wherein instead of starting the learning process from scratch, a pre-trained model is used and appended with custom layers to achieve additional functionality.


The "objective function" is the function that you want to minimise or maximise in your problem.

The expression "objective function" is used in several different contexts (e.g. machine learning or linear programming), but it always refers to the function to be maximised or minimised in the specific (optimisation) problem. Hence, this expression is used in the context of mathematical optimisation.

For example, in machine learning, you define a model, $\mathcal{M}$. To train $\mathcal{M}$, you usually define a loss function $\mathcal{L}$ (e.g., a mean squared error), which you want to minimise. $\mathcal{L}$ is the "objective function" of your problem (which in this case is to be minimised).

In the context of search algorithms, the objective function could represent e.g. the cost of the solution. For example, in the case of the travelling salesman problem (TSP), you define a function, call it $C$, which represents the "cost" of the tour or Hamiltonian cycle, that is, a function which sums up the weights of all edges in the tour. In this case, the "objective" of your problem is to minimise this function $C$, because, essentially, you want to find an inexpensive tour, which is associated with either a local (or global) minimum of $C$. This function $C$ is the "objective function".

It should now be easy to memorise the expression "objective function", as it contains the term "objective", and the "objective" (or goal) in your (optimisation) problem is to minimise (or maximise) the corresponding function.


What type of network for a repeated experiment

So it seems any time series modeling would do the trick. If you new to neural nets and seems you want to play with keras, maybe start by throwing it into a simple LSTM.


How can we use data augmentation for creating data set for face recognition and will the inverted faces on augmented images detected?

This is the purpose of rotational augmentation. In fact you should augment your images at different angle as well than just inverting it. This will make your model angle agnostic.


How to measure the reasoning capabilities of neural networks

What do you mean by "understood the concept of a car" ? A classification algorithm need not know that cars are used to transport humans or anything else it's used for. It only needs to know what combination of features we decide to label as a car.

Or maybe you meant we could use some other metric for understanding that is not classification accuracy?

This is a kind of Catch-22 situation.

When we want to classify a car we can reformulate the task to learning what set of a combination of features do us humans decide to call a car. What labels we want to assign a combination of features is arbitrary to a degree. It preferably has to be  consistent for inference but an NN can learn a dataset assigned completely random features to a 100% accuracy.

For instance, perhaps in an alternative universe we label the front-view of a car as 'B' and the rear view as 'A' and the notion/label of 'car' is nonexistent: even though we are ostensibly referencing the same object in physical space, we can assign it different labels.

It follows that to measure if a model has properly interpolated what combination of features we like to call car, we ourselves must be able list out such features; and if we were able to do that, we would not not need the process of learning/teaching algorithms in the first place!

A much more interesting subject is testing the extrapolative powers of neural networks- something much more proximate to what we canonically refer to as 'reasoning'.

It's quite hard to measure this in static environments such as the classification of images, but the ability to extrapolate is more apparent in reinforcement learning settings where the model requires us to train it for times order of magnitudes above the time for needed for humans.

The whole notion of pristine reasoning in machine learning breaks down when it comes to induction(see sufficient reason). Thus, we resort to probabilities that can be roughly translated to "this line of reasoning yielded the correct result 99.9% of the time, so it would make sense that we keep using it.

In fact, the idea of perfect-as-possible induction has been mathematically formulated except it isn't computable :( Solomonoff Induction


Does self-supervised learning require auxiliary tasks?

Most (if not all) self-supervised learning techniques for (visual or textual) representation learning use pre-text tasks, and many pre-text tasks have been proposed in recent years.
However, as I say in my other answer (which you cite), the term SSL has also been used (at least, in robotics: for example, see this paper, which I am very familiar with) to refer to techniques that automatically (although approximately) label the unlabelled dataset for your downstream task (i.e. image recognition), i.e. they automatically create a labeled dataset of pairs $(x_i, \hat{y}_i)$, where $x_i$ is an image that contains an object and $\hat{y}_i$ is the automatically (and possibly approximately) generated label (such as "dog"). This latter use of the term SSL is closer to some weakly supervised learning (WSL) techniques. Actually, it can be considered a WSL technique.
Now, in this specific paper, they actually solve some kind of pre-text task, i.e. they exploit the relations between two different sensors to produce the labels.
To answer your question more directly: in all SSL papers that I have come across, some kind of pre-text task is always solved, i.e., in some way, you need to automatically generate the supervisory signal, and that task that we solve with the automatically generated learning signal (with the purpose of learning representations or generating a labeled dataset) can be considered the pre-text task (which may coincide with the downstream task, for example, in the case you're training an auto-encoder with an unlabelled dataset for the purposes of image denoising).
In any case, I wouldn't bother too much about it. Just keep your context in mind when reading your paper. If you're really worried about it, then you should probably read almost all SSL-related papers, but, in that case, by the end of that, you will be an expert on the topic and you will not need our help (or my help).


How can I read the .weights file that stores the weights of the pre-trained YOLO in Keras?

The .weights seems to be the extension for a framework called "darknet". You can read .h5 files with Keras.
However, if you really want to build an object detection framework, there is no necessity to stick to the darknet's .weights files. There are many pretrained models on the web. Or else you could fine-tune a pre-trained ImageNet model in Keras, which I think is the best option, although there is not much you can do, if you got the exact weights of the YOLO model. Instead, it is better to train one from scratch or at least by fine-tuning an ImageNet model. By doing so, you will learn a lot.


Could AI kill the joy of competitive sports and games?

Unlikely!

Chess has been "solved" by AI much longer than GO (chess engines even before AI are way too strong for human players) and still people are playing and competing.

Simply put competition and sports live from the human element. Humans competing against each other will still create the same joy for most people regardless of the fact that all involved players might lose against a computer.

Some select individuals on the highest level might be put off by the new reality but it won't be the end of competition.

No human is faster than a car and yet we still celebrate running competitions.

Indeed I think long-term we will gain entertainment by watching different AIs and models compete against each other in chess or Go.


Why does the hill climbing algorithm only produce a local maximum?

This part of your sentence is not always true "and not reached to final goal/solution". If you have just one maximum at all and it is finite, hill climbing (HL) can reach to it and it is a global maximum too (for example, if the function is a parabola).

To answer your question is back to the stop criteria of HL. It will be stopped when reaching to a maximum point. The maximum point means a point which the objective function has a greater value in that point than on its $\epsilon$-neighborhood of that point. Therefore, if objective function has some local maxima, depends on the initial point of HL, it might be trapped in one of the local maxima as their definition is satisfied in the explained definition.

In sum, depends on the initial point of HL, it might not reach to the global maxima and trapped in local maxima if there exists.


How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?

The master's thesis Action space representation in combinatorial multi-armed bandits (2015) seems to provide an answer to my question.
Several algorithms can be used

Naive Monte-Carlo Sampling (NMC)
Linear Side Information (LSI)
Monte Carlo Tree Search with Hierarchical Expansion (MCTS-HE)
MCTS with Dimensional Expansion

The idea is to divide and conquer.
Here's a screenshot of the HE algorithm from section 4.2.


Is it possible to apply deep dream technique for the audio streams?

As far as I can see, there's no reason why you couldn't (for example) take the convolutional inputs to deepdream from adjacent sample points, rather than adjacent spatial positions, as is the case with image input.

Given the 'self similar' nature of deep dream images, listening to this fractal granular synthesis technique might be of interest/inspiration.


Data augmentation for very small image datasets

In my opinions, you have around $25\times 20 \times 2 + 5 \times 20 = 1100$ samples, so the list of problems is:

Lack of data
Imbalance between class 1,2 and class 3

With the simple task, the model with the low capacity (small number of parameters) is more suitable for your task and to avoid overfitting because of the lack of data. Since the augmentation is nothing but regularization, based on EfficientNetV2 paper, with the small model you don't need too much regularization, select some basic affine transformation and elastic transformation then apply Rand Augment with small magnitude mix with 0.1 dropout should be enough.
The second problem should be solved with the weight class for weight, which means set the scale factor for the third class, or duplicated the samples of the third class then apply the strong affine transformation to it.


Why should weights of Neural Networks be initialized to random numbers?

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the start of learning.

Using backpropagation, the gradient at the outputs of a layer L involves a sum multiplying the gradient of the inputs to layer L+1 by the weights (and not the biases) between the layers. This will be zero if the weights are all zero.

A gradient of zero at L's output will further cause all earlier layers(L-1, L-2 etc all the way back to layer 1) to receive zero gradients, and thus not update either weights or bias at the update step. So the first time you run an update, it will only affect the last layer. Then the next time, it will affect the two layers closest to the output (but only marginally at the penultimate layer) and so on.

A related issue is that with weights all zero, or all the same, maps all inputs, no matter how they vary, onto the same output. This also can adversely affect the gradient signal that you are using to drive learning - for a balanced data set you have a good chance of starting learning close to a local minimum in the cost function.

For deep networks especially, to fight vanishing (or exploding) gradients, you should initialise weights from a distribution that has an expected magnitude (after multiplying the inputs) and gradient magnitude that neither vanishes nor explodes. Analysis of values that work best in deep networks is how Xavier/Glorot initialisation were discovered. Without careful initialisation along these lines, deep networks take much longer to learn, or in worst cases never recover from a poor start and fail to learn effectively.

Potentially to avoid these problems you could try to find a good non-zero fixed value for weights, as an alternative to Xavier initialisation, along with a good magnitude/distribution for bias initialisation. These would both vary according to size of the layer and possibly by the activation function. However, I would suspect this could suffer from other issues such sampling bias issues - there are more weights, therefore you get a better fit to desired aggregate behaviour when setting all the weight values randomly than you would for setting biases randomly.


Convolutional Sequence to Sequence Learning: Training vs Generation

I know it's too late to answer your query after 1.5 years
The inference algorithm is explained at https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference
Implementation of the algorithm is available at: https://github.com/pytorch/fairseq.
One could get a complete understanding from their source code.


In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?

First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$.

If you look at your equation more carefully

$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$

You will see that the gradient is taken with respect to $\theta$, which are the parameters (i.e. a vector) e.g. of your neural network, that is, $\nabla_{\theta}$.

In this case, it doesn't really matter whether $\pi$ represents a distribution or not (for some specific value of $\theta$), but you're right that $\pi$ often represents a probability distribution over the possible actions (given a specific state). In any case, $\pi$ is a function of the parameters $\theta$ (i.e. in the case of a distribution, $\pi_{\theta}$ is a family of distributions for all possible values of $\theta$), i.e. if you change $\theta$ the outputs of $\pi$ also change, so you can take the derivative of it with respect to $\theta$.


How do we know that now artificial intelligence will really take off?

This has happened in the past where people were really excited and saying things like we will have AI in decade or so. This is happening again. Not sure why people don't learn from history of AI. In both the cases what's happening is this - You develop a technique to solve a particular problem, you apply that technique, the technique seems to be general enough, people start to apply that same technique to various problems, people get excited that this is the silver bullet they were looking for, the technique starts to show its limitations and doesn't work for many problems, hype gets shattered, start over.


Why feeding the correct output as input during training of seq2seq models?

The reason why you would use the ground truth as input to the decoder is to lean the testing distribution. From what I've seen so far most of the papers are using scheduled sampling (Bengio et al.). Meaning that you will introduce a new term p which will be the probability of the network to feed as input its own prediction. Initially p will be very small so that the network will use the ground truth, but later the more iterations passes by, the probability will decrease and the network will start using its own predictions.


What are the current open source text-to-audio libraries?

voice-builder is an opensource text-to-speech (TTS) voice building tool from Google.


Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many Tabular methods are proven to converge, for instance the paper by Chris Watkins titled "Q-Learning" introduces and proves that Q-learning converges. Also you include tabular methods as being parameterised function approximators. This is not true. Tabular methods maintain an estimate of the value function in a look-up table for each state-action pair and there is no function approximator being used.
Now for non-tabular methods, i.e. Deep Reinforcement Learning. Here we are using SGD only to optimise the parameters of the networks (assuming of course that we are using NN's as our function approximators, but this is a fair assumption if you read the literature). This is why off-policy methods are typically preferred because they can use a replay buffer which allows the use of data from any past trajectory. When using a replay buffer we sample random past experiences which de-correlates the data and allows the i.i.d. assumption to hold when using SGD.
If we were to use an on-policy algorithm, in theory SGD may not converge to any local optima because we are violating the i.i.d. assumption. However, all this means is that we are not guaranteed to converge. For instance, I have run an experiment using REINFORCE (an on-policy learning algorithm) using NN's as function approximators and they were able to obtain an optimal policy. However, this was for a very simple environment using modestly sized networks, so this is likely why they were able to be trained using non-i.i.d. data. A more in depth question/answer as to why NN's require i.i.d. data can be found here.
To address your edit, Replay Buffers are not required but if we can use them, then why would we not? It helps to maintain the i.i.d. assumption and so it helps us obtain a local optima. If we did not use them then there would not be the guarantee that it would converge. As an aside, Replay Buffers are typically used because they make the algorithm have a greater sample efficiency - this means we can obtain an optimal policy using much less data.
If you are wondering why don't on-policy methods use a replay buffer, the answer is because they are on-policy. The actions used in updates must be taken according to our current policy that we are learning the value functions for (or are optimising the policy of in Policy Gradient Methods). This is not the case in off-policy algorithms - e.g. in Q-learning we are learning the value functions of the greedy policy but we follow a different policy that allows for exploration.


The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.
The training loop in PyTorch would look something like this:
# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()

# Forward pass
features = encoder(image)
outputs = decoder(features, captions)

# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size),
                         captions.view(-1))

# Backward pass.
loss.backward()

# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.
I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.


Can a purely policy convolution neural network based game learn to play better than its opponents?

(I'd leave this in the comments but sadly I can't.)

Here's a great paper (leveraging formal game theory) in which "players" make strategic choices based on each event ("move") and an algorithm calculates the best strategy for all players based on each player's abilities throughout the multi-stage "game". So instead of learning from winning and predicting what the next perfect move the player thinks (based off of learning) will lead to a "win", the player calculates an optimal strategy and moves to win the game itself over multiple moves.

https://users.ece.cmu.edu/~youzhib/paper/bao2017csf.pdf

Note: This paper deals with cyber war games so is heavily tilted towards that domain, but if you take away the game theoretics and concept of strategy over a multi-stage game like ti-tac-toe, you should be able to improve your overall outcomes. That being said, how the model would accommodate a random "player" with random moves would be interesting to see.


In how few updates can a multi layer neural net be trained?

In Don't Decay the Learning Rate, Increase the Batch Size, Smith et al. train ResNet-50 on ImageNet to 76.1% with only 2500 updates. Has anyone done it in less?

In The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent Sankararaman et al. present the concept of gradient confusion which slows convergence, and show that depth increases gradient confusion while overparameterization on width and skip connections reduces gradient confusion.

In Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash, Mikami et al. train ResNet-50 on ImageNet to 75.29% with some number of updates, but I can't do the math to compute out how many updates they must have used.

To summarise:

Larger mini-batches help, but give diminishing returns and cause other issues.
Deeper models tend to require more updates.
Wider layers need fewer updates.
Skip connections, label smoothing, and batch norm might help.
The best I have found so far on ImageNet/ResNet-50 to >75% is 2500 updates.


How to export Pytorch Deep Neural Networks trained model to C++ program to use it from C++?

PyTorch now has a C++ frontend. I haven't tried it, but I'm sure you could use that.

Another option, which is more production-tested, is using a message passing framework such as ZeroMQ to communicate requests and results between Python and C++ executables.


Is there any real-time computer vision system that can learn to detect new objects of new classes?

You are probably looking for incremental learning (sometimes known as lifelong learning) techniques, i.e. machine learning techniques that attempt to address the catastrophic forgetting effect of neural networks when trained incrementally, i.e. as new classes or data are added to the original training data.
There are different techniques and some of them store (or compress) the old data in order to fully or partially re-train the neural network with the new classes or data.  However, note that this is a relatively new area of research and significant progress still needs to be made to produce serious tools. If you are specifically interested in incremental class learning, maybe have a look at this paper Class-incremental Learning via Deep Model Consolidation (2020).


Is it a good idea to store the policy in a database?

You have lots of choices in how to store a policy, depending on how you have built it - using which RL algorithm, and what kind of representation for states and actions.

Tabular reinforcement learning algorithms lend themselves well to storage in a database table with an indexed state_id column and one or both action and value columns. This might be a good choice if you have a moderate sized state space, as you would avoid the need to load the whole table into memory just to compute the next move.

Whether this is feasible will depend on the complexity of your game. Even relatively simple games like checkers turn out to have too large a state space to enumerate all the states in this way.

So you are more likely to need some kind of policy function or state value function implemented using a parametric function approximator. Very often in RL this will be a neural network. In which case you would use whatever storage mechanism your neural network library supported - most will happily read and write their parameters to a file or string, allowing you a lot of flexibility on how and where to store them.

So your policy is likely to be stored in one or two files on disk as a serialised neural network. How to use that efficiently in a web service is a complex subject in its own right. You could just read the files back and instantiate the neural network each time it is needed, and this will probably be OK for a simple game and low traffic service. However, this is very inefficient.

Some neural network libraries designed around use in production will allow you to pre-load the neural network and keep it in memory between requests. How to do this depends entirely upon the frameworks you are using, so I cannot explain in more detail here. Initially I would not worry too much about this part for your project.


Algorithm that creates new images based on other images

I'm not an expert on that so you could probably get a better answer.

I'm not sure to understand what you're looking for. Are the couple of images about the same thing? Like pictures of cats and you want to generate a new cat based on these pictures? If that's what you want, you could probably take a look at Generative Adversarial Network (GAN) : Introduction.
A GAN is made up of a Generator and a Discriminator. The goal of the discriminator is to distinguish the real data from the generated data. And the goal of the generator is to improve its generated data to look similar to real data. Then, if there are different cat images in your dataset, the generator will learn to create a new cat based on that dataset.

If what you're looking for is to take different images like a cat and a dog and generate a "catdog", you can take a look at Variational AutoEncoder (VAE). For example you can train two different VAE (Encoder/Decoder). One for cats, and one for dogs. Then you take the encoder of dogs and the decoder of cats. That what I saw one day, not sure if it really works.

Correct me if I'm wrong


What are some resources regarding the complexity of training neural networks?

There are a few technical papers and books on the topic

Computational Limitations on Learning from Examples (1988) by Leonard Pitt and Leslie G. Valiant, published in Journal of the ACM
Training a 3-node neural network is NP-complete (1992) by Avrim L. Blum and Ronald L. Rivest, published in Neural Networks
Computational complexity of neural networks: a survey (1994) by Pekka Orponen, published in Nordic Journal of Computing
On the relative time complexities of standard and conjugate gradient backpropagation (1994) by J. V. Stone and R. Lister, published in IEEE
Neural Networks - A Systematic Introduction (1996), specifically chapter 10 The Complexity of Learning, a book by Raul Rojas
Complexity Analysis of Multilayer Perceptron Neural Network Embedded into a Wireless Sensor Network (2014) by Gursel Serpen and Zhenning Gao, published in Complex Adaptive Systems
On the Computational Efficiency of Training Neural Networks (2014) by Roi Livni, Shai Shalev-Shwartz and Ohad Shamir, published in NIPS 2014
Convolutional Neural Networks at Constrained Time Cost (2014) Kaiming He and  Jian Sun, published in IEEE Conference on Computer Vision and Pattern Recognition (2015)
Learning Neural Networks with Two Nonlinear Layers in Polynomial Time (2018) by Surbhi Goel and Adam Klivans

However, note that gradient descent (and other optimization algorithms) and the back-propagation algorithm are numerical algorithms (that is, they deal with numerical errors), so the time complexity is not the only factor affecting the actual performance of these algorithms and models.


Using UMAP, PCA or t-SNE to find the separating hyperplane?

All the methods are basically manifold methods which are used to squeeze hyper dimensions to two or three dimensions with a certain amount of information loss. So whatever you "see" with these methods are not real and deceiving. You may see a separation of data points in 3D but when they are mapped onto the actual dimensions it may be complete rubbish.

By definition, these methods do not provide a way to find separating hyperplane. They just tell you some condensed information about the whole data. It is generally advised to use methods like t-SNE with caution.


Can I shuffle image channel data as a form of data augmentation?

As a rule of thumb for image data augmentation, look at the augmented images:

Can you correctly classify or measure your target label from the augmented images?
Could something similar to the augmented images appear in the environment where you want to run inferences on previously unseen inputs?

For your suggested augmentation of shuffling the channels, it may pass the first test. However, the second test shows that you are probably taking a step too far.

  will this have a regularizing effect on the CNN model?

Yes, but it might not be that useful to have strong cross-channel regularisation.

If there is important information for your task in the separate colour channels, then shuffling the channels makes it harder for the neural network to use that (it is not impossible, the CNN can still learn filters that will trigger most strongly on features that tend to appear in red channel and not blue in your problem for instance).

If there is not important information for your task in the colour information, then you may find it simpler and easier to turn your images into single channel greyscale instead, and use that throughout. Although that is not completely the same, for many image types it will achieve a similar effect (and possible boost to accuracy) for a fraction of the effort.


Tabular Datasets where Deep neural networks outperforms XGBoost

In my opinion, no.
Also images could be interpreted as tabular dataset as well, where certain columns represent different rgb codes of pixels. If you seek to use neural nets opt for image datasets, with large sample size.
Neural networks generally require large sample sizes to perform, and huge dimension inputs to not be outperformed by boosting.


Which nonfictional documentaries about Artificial Intelligence are available?

AlphaGo (2017) is quite a good watch, given that it is a documentary about the AlphaGo program, how DeepMind developed it, the help they had, and doesn't get too technical.  You can watch the trailer here.

Another documentary which wasn't exactly AI but something that is an interesting watch or at least was when it came out was The Human Face of Big Data.


Combine two feature vectors for a correct input of a neural network

The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.

Assume the first sample is made of the pair $X_1$ and $Y_1$. Let the corresponding feature vectors for $X_1$ and $Y_1$ be $\textbf{v}_1$ and $\textbf{v}_2$, respectively.

$$
\textbf{v}_1 = [f_1, f_2, \ldots , f_n],\\
\textbf{v}_2 = [g_1, g_2, \ldots , g_m].
$$
Then, the first sample's feature can be defined as
$$
\textbf{v} = [f_1, f_2, \ldots , f_n, g_1, g_2, \ldots , g_m].
$$
Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).


It depends on the algorithm.

A classical algorithm would not be able to recognize the pattern of primes unless it was programmed to do so.  This pre-programming would not have to be exclusive to primes, but the algorithm would have to "understand" the nature of primes including 1 (the number can only be divided by 1 and itself.)

Because computers are good at number crunching, it should be quite possible to program an algorithm that looks for numerical relationships, such as primes, parity and factors, without being taught those specific concepts.

Extending this to alternate representations of numbers, 1=1, 11=2, 111=3 requires a type of abstract thinking (i.e. the literal representation is not the true meaning. In an 8-bit byte 11101101 = 237. In Base-2 1,11,111 are the primes 1,3,7;)

However, if the algorithm was superintelligent, figuring out the example would be trivial, and, ostensibly, correctly deriving patterns of any kind, no matter how opaque, would be possible, given enough time and memory.


How should I generate datasets for a SARSA agent when the environment is not simple?

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem

There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach.
As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you are using SARSA then you would not expect to use any record of previous experience either because SARSA is designed to work on-policy and online - which means that data needs to be generated during training. Training data for SARSA is typically stored only temporarily before being used, or is used immediately (you might keep a log of it for analysis or to help document your thesis, but that log will not be used for further training by the agent). This is different to Q-learning and DQN, which could in theory make use of longer-term stored experience.
You have two main choices for acquiring data:

Use a real environment. In your case, set up 15 mobile users and 3 edge servers. Instrument the environment to collect state and reward data for the agent. Implement the agent as the real decision maker in this environment.

Simulate the environment. Write a simulation that models user behaviour and server loading. Instrument that to provide state and reward data, and integrate your learning agent with it. Typically the agent will call the environment's step function, passing the action choice as an argument and receiving reward and state data back.

If you can simulate the environment, this is likely to be preferable to you since you will likely use less compute resources (than 3 servers and 15 mobile phones) and can run the training faster than real time. Deep reinforcement learning can use a large amount of experience to converge on near-optimal policies, and fast simulations can help because they generate experience faster than reality.
You can also do both approaches. Train an initial agent in simulation, then implement the real version once it reaches a good level of performance in simulation. You can even have the agent continue to learn and refine behaviour in production. Given that you are working with SARSA, this may be an important part of the intent of your project, that the agent continues to adapt to changes in user behaviour and server load over time. In fact this is a key advantage of SARSA over Q-learning, that it should be more reliable and safe to use in such a continuous learning scenario deployed to production.

and what the experience means.

The experience in reinforcement learning is the record of states, actions and rewards that the agent encounters during training.


After i did a bit more study on this, i found this solution.

μsmall[x]

1; -> x≤0
(20-x)/(20-0); -> 0
0; -> x≥20

μmedium[x]

0; -> x≤20 or x≥70
(x-20)/(45-20); -> 20
(70-x)/(70-45); -> 45

μlarge[x]

0; -> x≤70
(x-70)/(100-70); -> 70
1; -> x≥100

For the final result i need to add 3 more rules for the function to work. So it become 9 rules and 3 variables, with 3 rules for each input variables.

To answer both of my question:

Add a triangle graph and use the formula.
Add a triangle graph for each rules.

What i learn from this is the total rules needed to determine the output is the number of input variables rules multiplied(i.e. v1 = 3 rules, v2 3 rules, total rules = 3x3=9 rules). And i'm not really sure about this one but you can combine multiple kind of graphs.


How do I compute the variance of the return of an evaluation policy using two behaviour policies?

If you simulate many trajectories and receive many estimates of the two returns you're interested in, you could empirically compare their sample variances.

However, the variance of ordinary importance sampling is in general unbounded.
If you're wanting some theoretical bounds on the variance of importance sampling estimates, I'd start with weighted importance sampling, whose variance converges to zero (Sutton and Barto, section 5.5).


Is there a way to predict points on a map?

Leaving aside the time aspect, you could do a cluster analysis on the event coordinates. If you use an algorithm that gives you a medoid (ie centre) of the clusters, you can then look at other points, and work out how close they are to the centres of the event clusters. It might be possible from this to predict which event could happen at those coordinates (which is the closest cluster medoid), and how likely it is (distance from the medoid).

This, however, depends very much on the shape of the data. If there is no discernible structure contained in it, then this will not work. But it is definitely worth trying.


Why is the policy not a part of the MDP definition?

The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state.

The policy corresponds to a strategy that the RL agent can follow to act in that environment. Note that the MDP doesn't define what the agent does in each state. That's why you need the policy! An optimal policy for a specific MDP corresponds to the strategy that, if followed, is guaranteed to give you the highest amount of reward in that environment. However, there are multiple strategies, most of them are not optimal. This should clarify why the policy is not part of the definition of the MDP.


How to make input variable as trainable parameter in a neural network?

It's just a typical optimization problem. You want to optimize function
\begin{equation}
f(\theta, x)
\end{equation}
with fixed parameters $\theta$ (network weights) and optimization variable $x$. Typically the approach is consisted of choosing a search direction and direction step. For deciding search direction you can use the gradient of objective with respect to decision variable and have an update of form
\begin{equation}
x = x - \alpha \nabla_x f(\theta, x)
\end{equation}
where $\nabla_x f(\theta, x)$ is the gradient. You can also use Newton or Quasi-Newton methods of form
\begin{equation}
x = x - \alpha B^{-1}\nabla_x f(\theta, x)
\end{equation}
where $B$ is a Hessian or approximate Hessian of $f(\theta, x)$ with respect to $x$. Step length parameter is usually decided with line search or you can also use trust region approaches which decide step direction with constraints to it's length. For more details about optimization you can consult Numerical Optimization by Nocedal and Wright


How to implement an artificial network that outputs an integer within a range?

First of all, sigmoid does not output 0 or 1, it outputs any real number in the range between 0 and 1.

Furthermore, normal neural networks doesn't output binary values, unless the output layer uses the step function as activation function (which is rare).

I'm not really sure if you want the NN to be a classifier or regressor, but it sounds like you want a regressor.

Regression is when you are interested in the value of the output nevron(s) itself. A simple example is if you want the network to predict the sum of two input neurons.

If you want to change the network from a classifier to a regressor you should probably reduce the number of neurons in the output layer to 1, and change the activation function of that neuron from softmax to the identity function (f(x)=x; which is the same as no activation function at all).

Hope this helps. Provide some more details if this didn't answer your question.


What is meant by both papers is that we have two agents (generator and discriminator) playing a game with the value function V defined as a sum of the expectations (i.e. an expectation of the outcome value defined as a sum of two terms, or actually a logarithm of a product...). The generator uses a strategy G encoded in the parameters of its neural network (θg), the discriminator uses a strategy D encoded in the parameters of its neural network (θd). Our goal is to (hopefully) find such a pair of strategies (a pair of parameter sets θgmin and θdmax) that produce the minimax value.

While trying to find the (θgmin, θdmax) pair using gradient descent, we actually have two loss functions: one is the loss function for G, parameterized by θg, another is the loss function for D, parameterized by θd, and we train them alternatively on minibatches together.

If you look at the Algorithm 1 in the original paper, the loss function for the discriminator is -log(D(x; θd)) - log(1 - D(G(z); θd), and the loss function for the generator is log(1 - D(G(z; θg)) (in both cases, in the original paper, x is sampled from the reference data distribution and z is sampled from noise):

The ideal value for the loss function of the discriminator is 0, otherwise it's greater than 0. The "loss" function of the generator is actually negative, but, for better gradient descent behavior, can be replaced with -log(D(G(z; θg)), which also has the ideal value for the generator at 0. It is impossible to reach zero loss for both generator and discriminator in the same GAN at the same time. However, the idea of the GAN is not to reach zero loss for any of the game agents (this is actually counterproductive), but to use that "double gradient descent" to "converge" the distribution of G(z) to the distribution of x.


Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Yes, there are different ways. What I think you are looking for is under the research field of Localization and Mapping. Which divides in the following subfields:

For getting current (the robot) position and trajectory go to models for Odometry Estimation
For getting a representation of the world around the robot go to models for Mapping
If you want both of them (I am guessing you want). Go for SLAM (Simultaneous Localization and Mapping) models

Here it is the amazing survey that links you to tons of papers with different models for each category. If you want to know what are the most common architectural blocks (LSTM, ConvLSTM, RNN...) used for your problem, read the most promising papers under your target category.
References:
Survey: https://arxiv.org/abs/2006.12567


Can we detect the emotions (or feelings) of a human through conversations with an AI?

I think you are definitely on a very sensible track. No one defines right or wrong in emotion field. It's not hard science. It's all theories.

I have recently read a paper regarding emotions in Reinforcement Learning (RL). It has explained briefly emotion from 3 perspectives: psychology, neuroscience and computer science. In particular, your way of emotion definition matches one of the categorical emotion theory in the psychology perspective. The other theories in psychology perspective and componential emotion. You can try to implement them and try out which one works well. The paper has also introduced ways to measure the level of emotions (emotion elicitation).

Here is the link for the paper I have mentioned. I am sure you will receive lots of inspiration. I have also written a summary of this paper. Take a look if the original paper is too long to read.

I don't have any concrete solution for implementing. But the general idea is always trying to categorize abstract concepts and quantify them. And try some thing, and iteratively modify and improve it. All the best!


What are examples of resources that describe the basics of Spiking Neural Networks in detail?

There is a book Spiking Neuron Models: Single Neurons, Populations, Plasticity (2002) by Wulfram Gerstner and Werner M. Kistler.
I also found these slides that could be useful.
(Additional resources would be appreciated.)


Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent publication in Science (which nbro linked to in his comment). I'm not only focusing on the official version of the paper just because it is official, but also because I found it to be much better / more complete, and I would recommend reading it instead of the preprint if you are able to (I understand it's behind a paywall so it may be difficult for some to get access). The answer to this specific question would probably be identical for the arXiv version anyway.

No, AlphaZero does not use $Q$-learning.

The neural network in AlphaZero is trained to minimise the following loss function:

$$(z - \nu)^2 - \pi^{\top} \log \mathbf{p} + c \| \theta \|^2,$$

where:

$z \in \{-1, 0, +1\}$ is the real outcome observed in a game of self-play.
$\nu$ is a predicted outcome / value.
$\pi$ is a distribution over actions derived from the visit counts of MCTS.
$\mathbf{p}$ is the distribution over actions output by the network / the policy.
$c \| \theta \|^2$ is a regularisation term, not really interesting for this question/answer.

In this loss function, the term $(z - \nu)^2$ is exactly what we would have if we were performing plain Monte-Carlo updates, based on Monte-Carlo returns, in a traditional Reinforcemet Learning setting (with function approximation). Note that there is no "bootstrapping", no combination of a single-step observed reward plus predicted future rewards, as we would have in $Q$-learning. We play out all the way until the end of a game (an "episode"), and use the final outcome $z$ for our updates to our value function. That's what makes this Monte-Carlo learning, rather than $Q$-learning.

As for the $- \pi^{\top} \log \mathbf{p}$ term, this is definitely not $Q$-learning (we're directly learning a policy in this part, not a value function).

  If it's a Supervised Learning, then why is it said that AZ uses Reinforcement Learning? Is "reinforcement" part primarily a result of using Monte-Carlo Tree Search?

The policy learning looks very similar to Supervised Learning; indeed it's the cross-entropy loss which is frequently used in Supervised Learning (classification) settings. I'd still argue it is Reinforcement Learning, because the update targets are completely determined by self-play, by the agent's own experience. There are no training targets / labels that are provided externally (i.e. no learning from a database of human expert games, as was done in 2016 in AlphaGo).


Best way to train Neural Network for Voice Commands?

What Neural Network does it learn from your input and output and creates a best algorithms.

This is inaccurate. A neural network is a function approximator, so it approximates an unknown function. However, it does so in many cases by learning from your input and output(as in the case of supervised learning). The function your approximating would here for example be the function that maps a certain soundwave input to "open camera", etc.

The problem with supervised learning is that you need massive datasets to accurately approximate the target function - luckily for Google they have such datasets and resources available.

  Is Google also Hard-Coded the commands for Google Voice Assistance?

No, because that would not generalize well over multiple languages/accents/background noise, etc. It is therefor that neural networks as function approximators have had such a large impact over the last years - they generalize well.
If you remember when the first "voice assistance" products came out, they rarely functioned - this was often due to them being "hard coded", with only static measures for noise-suppression.


What are the current trends/open questions in logics for knowledge representation?

Some of the work on descriptive logics and modal logics was done within the context of artificial intelligence from a research funding perspective.  Some was part of the normal academic apparatus of mathematics departments.  Furtherance of these fields in the AI context has been hindered by historically low return on investment.

Although first order logic, propositional logic, and fuzzy logic have been investigated conceptually and developed considerably, there are no theorems of note that have been proven without human assistance.  Currently, mathematics is furthered solely by willful acts of humans, although that fact is not immutable.

There have been no proofs offered that artificial machinery cannot ever apply logic to prove theorems, develop disciplines, or assemble new logical systems.  There are also no existing proofs to the contrary.

It is an open question in academia and with governments and corporations investigating the potentials of AI.  To characterize any of the work in progress as a trend would be to exaggerate the effort level along one of many theoretical paths.

A search of academic articles using either descriptive logics and modal logics as search terms, with date constrained to 2008-2018, returns a trend of variety.  Another search of the same decade for articles that reference the twentieth century work in those disciplines produces the same wide distribution.

There is no media hype or academic, corporate, or government claims of any research results that can be translated into useful application.  All of these statistics are research inhibitory.  Investors and educational institutions like paths with clear advantages.

Changes to this condition may come from work being done on the ontological and epistomological side in neural research as it applies to human linguistic networks in the brain.  Although lower animals can appear to draw conclusions, they are largely learning patterns of behavior based on past experience without applying logic.

Dolphins, elephants, dogs, and primates can communicate and exhibit forms of intelligence and personality traits, but existing research indicates that they are unable to learn dialog or develop the ability to form new ideas.

Those investigating semantic approaches in AI seem to exhibit a single orientation, and that may be the one identifiable trend.  They suspect that there is one single causal origin responsible for three things.

The ability to develop complex languages
The ability to construct new schools of thought
The ability to develop a logical argument for or against a proposition or a theory or consider its likelihood logically

There may be something to the idea that natural language rather than formal grammars are the key to higher thinking.


We cannot do homework for students in this network, however I can suggest that several items affecting cost and several usage patterns are missing and the number of rules is shy by an order of magnitude.  I wholeheartedly agree with the educational directives you received.

Consider first developing your lists further to include peripherals like DVD burner, USB devices, and audio.  Whether the user does scientific programming, watches movies on the monitor, develops software, and other specific usage scenarios is also more specific and therefore will produce a better tailored system than the answer to the question of whether the user is a professional.

It is not the metric of the number of rules that is of most importance.  It is the number of operations contained in the rule set that is the guiding metric.  This is because rules in Prolog can be aggregated.  The rough estimate of rule operator count to complete a system is sqrt(i*o)/4, where i is the number of input permutations and o is the number of output permutations.

(This is the application of Shannon Information Theory, that number of bits n = log2 (P'/P), where P' and P are the a posteriorii and a priori probabilities respectively.  The divisor of four is because there are about 16 = 24 operators normally used.)

You may end up with thirty or forty rules.

Create some use cases that exercise the extremes as well as some of the typical cases from among the permutations in inputs and outputs.  Run your system on those cases and observe the system behavior.  Learn how to debug by outputting intermediate results or stepping through rule execution.

There are no shortcuts to researching and developing other than not wasting time worrying about how much time it will take.  You can also optimize your homework time by learning the tools and then stepping back, taking a deep breath, and saying, "I can do this!"


What is the importance of the endocannabinoid system for cognitive function?

The release of Adenosine, Dopamine, Endorphin, Endocannabinoids, GABA, Glutamate, Norepinephrine, Oxytocin, Serotonin, and many others into specific regions of the brain are very likely an essential part of both activation tuning of single neurons and neuroplasticity, two essential aspects of organic learning researchers have been and will continue to work to understand.

Most of those I've met in that sector of research are curious about the larger questions of what intelligence and consciousness are, and all of them appear to be interested in discovering how learning systems may be valuable in software engineering contexts.  These overarching questions are difficult to answer and the dive into the detail of learning has resulted in the expansion of ideas presented by Dr. Norbert Wiener in the mid 20th Century at MIT.

How chemical feedback in regions of the brain are secreted, how they disseminate geometrically into organic structures, how they interact with receptors, and what that does to the cell metabolism to produce change in the cell is almost definitely part of the DNA driven design of higher animal learning.  There does not appear to be anything pointless or arbitrary about it.  Adaptation to improve survival is evident and, yes, study is well underway.

Narcotics that interfere with the natural functioning of these organic signaling systems can lead to the inability to adapt in the individuals addicted to them.  That fact is a strong form of evidence that learning depends on these systems.

Oxytocin is another neurotransmitter of interest because its release is associated with what would fit into the higher levels of human thought and motivation on Abraham Maslow's famous hierarchy of needs.  Oxytocin seems to be part of reward signaling for modes of thought like authenticity, compassion, intimacy, wisdom, spirituality, and other human mental capacities and patterns of thought that transcend mere rationalism.  Why is that important?  Because the ability to lay down selfish goals for the good of the community seem to depend largely on the oxytocin system's preemptive ability over mere survival mode neural activity in mammals.

Regarding the cannabinoid receptors, there is a large enough body of media that show a correlation between a pool of successful artists and marijuana use to legitimately wonder whether there is any tie between creativity and the endocannabinoid system.  However in science (and hopefully in technology too), we are careful not to draw conclusions rashly.

It is also possible that this apparent correlation is simply a social phenomenon where the popularity of the artistic products or live performances is mainly because of potential audiences similarly stimulating their receptors with cannabinoids.  For instance, those engaging in LSD trips on stage attracted those also engaging in LSD trips into their audiences two generations ago.  Whether the work of those artists was more creative because of the impact on the serotonin receptors is largely subjective.  How can researchers come to conclusions about what is good performance?

One could analyze the audio to produce a table of notes in a performance and then develop a system to attach a numerical value to several positive quality metrics, study the price of tickets, or count downloads, but the decisions of what to measure and how to aggregate it into a final judgment of excellence is itself necessarily a subjective choice.

Nonetheless, if we place the many misconceptions of popular psychology and the drug culture aside, there is much research into the endocannabinoid system as part of the learning signaling, what machine learning researchers are currently calling reinforcement.  The more general term is, "A non-linear control system's feedback signal," already developed in detail in the 1940s (Behavior, Purpose and Teleology — A Rosenblueth, N Wiener, J Bigelow — Philosophy of Science, 1943 — U Chicago).  Some new trendy name will probably appear in the 2020s.

This article claims that the mammals under test, "exhibited enhanced learning."

Memory in Monoacylglycerol Lipase Knock-out Mice, by Bin Pan, Wei Wang, Peng Zhong, Jacqueline L. Blankman, Benjamin F. Cravatt and Qing-song Liu; Journal of Neuroscience 21 September 2011, 31 (38) 13420-13430; DOI: https://doi.org/10.1523/JNEUROSCI.2075-11.2011

To find many others: https://scholar.google.com/scholar?q=endocannabinoid+learning


What is the definition of the hinge loss function?

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here.
The hinge loss function is defined as follows
$$
\ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1},
$$
where

$t = \{-1, 1\}$ is the label (so, if your labels are in the set $\{0, 1 \}$, you will have to first map them to $\{-1, 1\}$)
$y$ is the output of the classifier (e.g. in the context of the linear SVM, $y=\mathbf{w} \cdot \mathbf{x}+b$, where $\mathbf{w}$ and $b$ are the parameter of the hyper-plane)

This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the ReLU, this loss should look familiar to you. In fact, their plots are very similar.
For more details, you probably should start with the related Wikipedia article, then maybe one of the many machine learning books that covers support vector machines, for example, Pattern Recognition and Machine Learning (2006) by  Christopher Bishop, chapter 7 (page 325).


What are options in reinforcement learning?

An option is a generalization of the concept of action. The concept of an option (or macro-action) was introduced in the context of reinforcement learning in the paper Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning (1998) by Richard S. Sutton, Doina Precup and Satinder Singh, so that to capture the idea that certain actions are composed of other sub-actions. Section 2 of the mentioned paper formally defines the concept of an option (which is actually composed of a policy, a termination condition, and an initiation set).

The authors of the mentioned paper give examples of options

  Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques.

The option picking up an object, going to lunch, and traveling to a distant city is composed of other sub-actions (e.g. picking up an object), but is itself an action (or macro-action). A primitive action (e.g. joint torques) is itself an option.

A set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), which are MDPs where the time between actions is not constant but it is variable. In other words, a semi-MDP (SMDP) is an extension of the concept of MDP that is used to deal with problems where there are actions of different levels of abstraction.  For example, consider a footballer that needs to take a freekick. The action "take a freekick" involves a sequence of other actions, like "run towards the ball", "look at the wall", etc. The action "take a freekick" takes a variable number of time steps (which depends on the other sub-actions).

Semi-MDPs are thus used to deal with such problems that involve actions of different levels of abstraction. Hierarchical reinforcement learning (HRL) is a generalization (or extension) of reinforcement learning where the environment is modeled as a semi-MDP.

Curiously, certain models that have won the RoboCup (the famous AI football) context are based on the concept of semi-MDPs, options and HRL. See e.g. WrightEagleBASE, which use the MAXQ-OP (MAXQ online planning) algorithm.

Semi-MDPs can be converted to MDPs. The picture below (which is a screenshot of figure 1 of the mentioned paper that introduces the "options framework" in RL) illustrates the relationship between semi-MDPs and MDPs.

The empty circles (in the middle) are options, while the black circles (at the top) are primitive actions (which are themselves options).

In the paper Reinforcement learning of motor skills with policy gradients mentioned in the question, apparently, the term skill is not formally defined. However, I suppose that skills can be represented as options.


What are the steps that I need to follow to build a neural network for face recognition?

My implementation to increase detection on video is using object tracking algorithms.
More specifically, first, I detect the object using a trained classifier. Then I track the object with the KCF algorithm. If the object tracker misses the object, again I call for the classifier.


Are these two versions of back-propagation equivalent?

The two examples present essentially the same operation:

In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output.
Both examples use the logistic function for node activation (the derivative of the logistic function $s$ is $s(1 - s)$. This derivative is obviously very easy to compute, and this is part of the reason why it was so widely used (these days the ReLU activation function is more popular, especially with convolutional networks).
The first method also uses momentum.

The main difference I can see is that in the first case backpropagation is iterative while in the second example it is performed in batch mode.

The last video in the series by Welch Labs introduces a quasi-Newtonian method which offers the advantage of finding the minimum of the cost function by computing the Hessian (matrix of second-order derivatives of the error with respect to the weights). However, this feels like comparing apples and oranges - the vanilla gradient descent does not use second-order information.


Why do current models use multiple normalization layers?

One issue is that a normalized set of initial weights may not stay normalized as learning progresses; so given that we adjust weights proportionately according to their relative values and also when working on a subset of the learning data the model may become convinced that one subset of features is important and others not, this can result in the weights becoming unbalanced again. In the learning curve we may see this as a plateau, where the learning becomes convinced that a few features are much more important than they really are and fails to find new features that can contribute even more because tiny proportionate changes did not move them far, or quickly, enough into a noticeable range.

So when we re-sample from the database to get the next batch we need to be fully open to the possibility that the previous batch learning was too heavily biased towards its own favoured set. In effect we are trading accuracy for openness to new feature combinations, and at the same time assisting generalization.


What makes learned feature detectors specialize in CNN?

You are thinking along the correct line when you consider intuitivity when examining the strange self-specialization aspect of the layers in any deep network that has been tuned, through toil, mental machinations, and late nights, to work.

Do AI experimenters manipulated parameters, topology, and algorithmic hierarchies to force circuit evolution that produces data flows through specialized functions like this one.

Noisy, normalized, cubes (horizontal, vertical, depth) of integers straight from video hardware or the video channel of a multimedia file
Indications of edges
Indications of corners, ends, and bends
Indications of relative angles (in the case of size independent recognition)
Indications of shapes
Indications of 2D topology
Indications of object forms
Indication of scene
Indication of action (when a hypercube is used and frame is another dimension)

Of course, one could design a solution and then tune layers, one by one, to follow the design, but it is not necessary because of the concept of entropy removal, the elimination of information that is redundant or irrelevant.

The principle in information science is this, in general: As redundancy is removed from the information, the number of bits representing the information decreases and the level of abstraction increases.

In minimalist art, a single word an identical twin might say to the other twin as the train rolls away, or a good Pictionary player, given a small window of information passing opportunity, adaptations can be made to pass large amounts of information base on previously agreed upon conventions.

When you think of it this way, it makes sense that by narrowing the bit width of the aggregate data representation from layer to layer that only certain parametric optimizations will provide an end-to-end indication of convergence, given a goal such as fully categorizing objects based on a small number of features.

When Leibniz envisioned a world tied together by mathematical certainty, he was overly optimistic, however he was correct in at least one important respect, mathematics had a reality all its own.  The success of Newtonian mechanics is one early example of a huge reality that has become apparent and has been so widely applicable, deduced from a thought experiment involving a cannon ball and the moon.

There are few demonstrations of discoveries that originally appear to be wild inventions than the amazingly accurate prediction of the previously anomalous orbit of Mercury.  But Einstein was not surprised by his success.  He had deduced general relativity from the challenge of Ernst Mach and the undeniable results from light and gravity experiments.  He discovered how it must be before it was found.

In the same sense, the above sequence of events can be proven mathematically to be the efficient strategy for vision.  The sequence of processes to transform incoming visual signals to collision avoidance control was there in mathematics before there were visual receptors on microbes.  That is why the narrowing of the visual pathways of a Pterodactyl have similarities to those of a shark shark even though their common ancestor may not have been able to see.

The incentive to specialize is the incentive to converge on the objective even after redundancy is stripped by force by limiting the information channel aperture.  Even if the aperture is in multiple dimensions, the same principle applies.


Current research on indoor localization and navigation in changing environment?

I would start with:

PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Image-based localization using LSTMs for structured feature correlation
Image-based Localization using Hourglass Networks
UAN: Unified Attention Network for Convolutional Neural Networks
Scene Coordinate and Correspondence Learning for Image-Based
Localization

You can also search for articles using the 7-scenes dataset (Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images) which is pretty standard


Can artificial intelligence classify textual records?

AI can categorize documents very accurately.  It is not a new application but in the last year the accuracy of the underlying algorithms such as text classifiers, and language models in general, has significantly improved.  There are applications of language models which now surpass human performance. Microsoft is one of the leaders in this area.  For example, see their Turing Natural Language Generation (T-NLG) language model: Turing-NLG: A 17-billion-parameter language model by Microsoft. The T-NLG was developed in a research group called Project Turing which works on AI tools for Microsoft's products including Office-365.  My guess is this same research group makes the AI for Project Cortex.


What is a "trajectory" in reinforcement learning?

In answer that you linked, I may have used an informal definition of "trajectory", but essentially the same thing as the quote. A "trajectory" is the sequence of what has happened (in terms of state, action, reward) over a set of contiguous timestamps, from a single episode, or a single part of a continuous problem.

So $(s_3, a_3, r_4, s_4, a_4, r_5, s_5, a_5, r_6, s_6)$ taken from any scenario where an agent was used in the problem environment would be a trajectory - at least as I intended it in the answer. This could be from real-world data, or a simulation. It could involve a totally random or untrained agent, or a fully-optimised policy.

In the other definition that you have found, the focus on states and a horizon could make it slightly different, but actually I suspect that it is the same thing, as it is not that useful to only know the states. The Quora answer is probably just using "path of the agent through the state space" as shorthand to describe the same data.

A "horizon" in reinforcement learning is a future point relative to a time step, beyond which you do not care about reward (so you sum the rewards from time $t$ to $t+H$). Fixed horizons can be used as an alternative to a discount factor for limiting sums of reward in continuous problems. They may also be used in other approaches, but basically mean the same thing - a time step beyond which you don't account for what happens.


Three step threshold in Facenet model of face recogniton

The problem originated because of the nature of the code.

Code:
https://github.com/AISangam/Facenet-Real-time-face-recognition-using-deep-learning-Tensorflow/blob/master/classifier.py

                model = SVC(kernel='linear', probability=True)
                model.fit(emb_array, label)

                class_names = [cls.name.replace('_', ' ') for cls in img_data]

As you see the code uses a SVC (Support Vector Classifier) to classify the classes. The SVC (or SVM) does not have an extra class for unknown class.

For the threshold variable, it is used in face detection, aka drawing a bounding box around the face for FaceNet to classify it.

Code:

https://github.com/AISangam/Facenet-Real-time-face-recognition-using-deep-learning-Tensorflow/blob/master/identify_face_image.py

            frame = frame[:, :, 0:3]
            bounding_boxes, _ = detect_face.detect_face(frame, minsize, pnet, rnet, onet, threshold, factor)
            nrof_faces = bounding_boxes.shape[0]

As you can see, the threshold variable is only used in detecting the bounding box.

Code for getting class name:

predictions = model.predict_proba(emb_array)
                    print(predictions)
                    best_class_indices = np.argmax(predictions, axis=1)
                    # print(best_class_indices)
                    best_class_probabilities = predictions[np.arange(len(best_class_indices)), best_class_indices]
                    print(best_class_probabilities)
                    cv2.rectangle(frame, (bb[i][0], bb[i][1]), (bb[i][2], bb[i][3]), (0, 255, 0), 2)    #boxing face

                    #plot result idx under box
                    text_x = bb[i][0]
                    text_y = bb[i][3] + 20
                    print('Result Indices: ', best_class_indices[0])
                    print(HumanNames)

You can see that no unknown class can be found.

Solution

You can try adding another threshold value and check if the predictions maximum value is lower than the threshold value. I have little experience in tensor flow so this is just a proof of concept, not sure if it will work.

best_class_probabilities = predictions[np.arange(len(best_class_indices)), best_class_indices] #original code
if(best_class_probabilities < threshold_2):
    best_class_indices = -1
HumanNames = "unknown"

By the way, because of the nature of triplet loss, you don't have to add and extra class to the SVC/SVM as the embedding model is locked and not trained, so unknown class embeddings will be very different to the known class. However you can try either approach.

Hope it can help you can have a nice day!


Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

word2vec and BERT are completely different things. You don't need to use BERT Tokenizer for training word2vec model. But, if you want to go ahead. Here's a link which can help you.