What are good action outputs for reinforcement learning agents acting in a trading environment?

Working Backward

Working backward from the trading interface available to you Note 1, you will need two things for each Exchange-Traded Fund (ETF) or other tradable commodity of another class.

An operation to perform
An associated monetary amount

The system can have an output structure Note 2 for each ETF like this (depending of course on the trading interface available to you to you and your banks primary monetary system).

Ternary operation indicator (buy, sell, hold)
Trade amount in USD

Why Not Just One Number per ETF?

A few corroborating reasons exist for why a single positive, zero, or negative trade amount is not likely the optimal architectural choice.

The difference between holding and trading an amount of 1.0 monetary unit is not equivalent to the difference between trading 1.0 and 2.0 monetary units.  Stated mathematically, the function of profitability to trade amount is not smooth and probably not even continuous.
When you implement the ternary output as two binary outputs {buy, sell}, training against the Boolean expression (buy AND sell) is likely to improve your initial performance and possibly your ongoing performance Note 3.

Limitations on Real Trades to Consider

Because you have the limit of the assets in the liquid account from which you can buy, you will need to train against breaking the bank or using a stage after the NN outputs imposing rules or a formula based on gain and loss probabilities.  This financial constraint muddies your question because there are several ways to ensure you do not break the bank (get an insufficient funds response from your trade operation).

Optimizing a Deeper Architecture

Let's first assume you use probabilistic calculus to produce a closed form (formula) for how to trade based on predictions from the NN architecture you design.  Then the NN outputs might be continuous values representing the distribution of outcomes for each ETF.  Projections will almost always be dependent on investment duration.

In such an architecture the NN output activation scheme would be a continuous function (not necessarily linear) producing something like this Note 4.

Mean expected delta value in one day
Std deviation in one day expectation
Mean expected delta four weeks
Std deviation in two week expectation
Mean expected delta two years
Std deviation in two years expectation

Any NN optimization of an investment portfolio that does not inherently deal with probability is nonsense.  Optimizing for maximum gain will introduce great risk.  Optimizing for minimum risk could result in losses.  The goal of optimization must be some representation of the balance between the desire to win and the fear of loosing, to put it in anthropological terms.

Mean and standard deviation are obvious starting choices, and the traditional categories of short, medium, and longer term investment is also reasonable to begin with in the temporal domain Note 5.

Pure NN

Now let's assume you replace the calculus with another NN scheme trained to maximize portfolio total assets Note 6.  Such a replacement NN scheme must also take as an input your available liquid asset amount along with the above probabilistic projections.  You must also train to ensure the aggregation of buys and sells do not break your bank, that your liquid asset account never drops below zero.

The trade amount activation should be a continuous function, but not necessarily linear and probably not tanh either because the asymptotes of that function would be counterproductive unless it is made proportional to your available liquid assets to train by aggregating your options in your architecture.  However, that's not optimal because you may find a better deal to use those assets a minute later.

The odd roots (third root or fifth root or both with coefficients), when used along with training to not break the bank and to maximize the rate of portfolio growth will produce a better environment for learning in the earlier layers because of the probabilistic and aggregation realities of liquid asset limitations.

NOTES

Note 1 — Preferably a secure RESTful API, however experimentation can employ a web browser the HTTPS transactions you can control using https://github.com/SeleniumHQ/selenium or https://github.com/watir/watir.

Note 2 — Input would be things like number of companies, exposure ratings provided by investment firm(s), short options, inception date, and a sequence of events, each event containing fixed point numbers and flags like these.

Price per share
Closing price flag
Expense ratio
Dividend per share

Note 3 — The binary output vector {buy, sell} value of {1, 0} causes a buy.  The value of {0, 1} causes a sell.  The value {0, 0} is ignored, and {1, 1} may trigger another training operation to correct this illegal output, which is likely to be indicative of the staleness of the last round of training.  If the NN is re-entrant (reinforced), the feedback vector could include this anomalous flag or weight it heavily in an aggregation of feedback sources.  In summary, the ternary scheme can be expected to augment the training speed and resulting accuracy.  More importantly, it opens additional options for continuous optimization.

Note 4 — Delta is an aggregation of price increase or loss, dividends, and holding and trading expenses because that is the proper metric related to profitability of the portfolio.

Note 5 — Four weeks and two years have been chosen for the middle and longer range projections so that the time ratios are 26.09 and 28 between the three durations.  Common choices are temporally skewed.  If 1 day, 1 week, and 1 year had been chosen, the ratios would have been 7.0 and 52.18.  If 1 day, 1 month, and 1 year had been chosen, they would have been 30.44 and 12.

Note 6 — Do not assume that even a well trained NN will ever outperform formulae properly derived from probabilistic calculus for the last stage in a trading profitability architecture.


Is there any measure of separability of classes?

Are you thinking something like Information Gain?

Information Gain basically uses the concept of information entropy to determine if splitting a variable is useful.


What are some information processing models besides feedforward or multi-layered neural networks?

Neural Network equivalents that is not (vanilla) feed forward Neural Nets:

Neural net structures such as Recurrent Neural Nets (RNNs) and Convolutional Neural Nets (CNNs), and different architectures within those are good examples.

Examples of different architectures within RNNs would would be: Long Short Term Memory (LSTM) or Gated Recurrent Unit (GRU). Both of these are well described in Colah's blog post on Understanding LSTMs

What are some alternative information processing system beside neural network

There are sooo many structures. From the top of my head: (Restricted) Boltzmann machine, auto encoders, monte carlo method and radial basis networks to name a few.

You can check out Goodfellow's Deep learning-book that is free online and get the gist of all the structures I mentioned here (most parts requires a bit of math knowledge, but he also writes about them quite intuitively).

For Recurrent Neural Nets I recommend Colah's blog post on Understanding LSTMs

Is there any system in which the topology of a neural network is variable?

Depends on what you mean with the topology of a neural network:

I think in the common meaning of topology when talking about Neural Networks is the way in which neurons are connected to form a network, varying in structure as it runs and learns. If this is what you men then the answer, in short, is yes. In multiple ways actually. On the other hand, if you mean in the mathematical sense, this answer would become a book that I wouldn't feel confortable writing. So I'll assume you mean the first.

We often do "regularization", both on vanilla NN and other structures. One of these regularization techniques are called dropout, which would randomly remove connections from the network as it is training (to prevent something called overfitting, which I'm not gonna go into in this post).

Another example for another way would be on the Recurrent Neural Network. They deal with time series, and are equipped for dealing with timeseries of different lengths (thus, "varying structure").

Does it exist neural net systems where complex numbers are used?

Yes, there are many papers on complex number machine learning structures. A quick google should give you loads of results. For example: DeepMind has a paper on
Associative Long Short-Term Memory which explores the use of complex values for an "associative memory".

Links:

Goodfellow's Deep Learning-book: deeplearningbook.org

Colah's blogpost on RNN's: colah.github.io

Paper on DeepMinds Associative LSTM: arxiv:1602.03032


How do I optimize a specific function using a genetic algorithm?

Yours is a really nice and easy question: you've seen how to use GAs in a complex problem but you're missing how to apply them to the most basic of all. I'll show you:

In a real-world implementation we have to structure our problem to be solved with GAs; we need to modify it, finding an equivalent representation that accepts individuals, each one of these is built from a genome and must be evaluated using a fitness function.

If you see this graphically you'll discover that the fitness function is just describing an area (or line or volume, depending on the grade) on our space and we are randomly dropping individuals on it, awarding the ones that fell higher than the others. We then try to modify the genomes of these guys to move them towards peaks of this fitness function.

In practice the fitness function is our world, our ground truth and we are exploring it.

Now to the function optimization part: well, we do not need any abstraction here, any strange individual representation or transformation of the problem; we just want to find a maximum, and this is exactly what GAs are for!

So, let's have a look at the elements you need to solve your problem with GAs:

Fitness function => you have it already! It's the function you want to find the max of.
Individual => as the maximum is enough, so the individual will be just a point in space.
Genome => like every respective point in space, the genome will just be a collection of real numbers (one for each dimension).

Now, like in any GA, you will instantiate randomly an initial population (random points in the domain). The evaluation of course is the easy part, just put them into the function and you'll se the value of the individuals.
What about crossover and mutation though? You'll have to use techniques that work with real-number genes, like BLXalpha, BLXalphabeta.
These are just defining ranges between values and picking random values inside these ranges, I wrote a pretty detailed answer about this, you can check it out at: https://ai.stackexchange.com/a/6323/15530


How do big companies, like Facebook, model individuals and their interaction?

This article may shed some light on this question:

Facebook Doesn’t Tell Users Everything It Really Knows About Them


What is the use of the $\epsilon$ term in this back-propagation equation?

The change in nomenclature from what you expected is Lisa Meeden's choice, for unknown reasons. Those with whom she published in the past used $\epsilon$ to represent error, the result of a loss function. Why she did not use $\alpha$ may be because the letter $a$ was used elsewhere in the formula, but, if that was the reasons, that wasn't a great one.

Changes in nomenclature between articles is common, for instance between Carnegie Mellon literature and Cal Tech. Robotics control systems people use different nomenclature from those who write game competition systems. The online mappings between choice of Greek letters and their meaning cannot be relied upon. This is not particular to AI. I've seen it in science and engineering literature, even between text books in courses that are part of the same university curriculum.


How to deal with huge branching factors in real-time?

As you mentioned in the question, you cannot solve all problems with decision trees.  Decision trees usually works well in a turn-based game with a good heuristic function, but in RTS games takes a different approach.

In the case of a very complex RTS game, one could implemented a rule-based AI. For example

given it is the early game use all units to scout for resources
if a certain criteria is met build the base a certain way
if another criteria is met build an army
if the army is big enough, attack
if being attacked by the enemy, bring the units back to the base to defend

Each of these rules could implement various other AI technique, for example use A-star to find the optimal path between a unit's current location and destination.

Further optimization could be done by "grouping" similar units to act like one unit. e.g. calculate the path for the entire group instead of each individual unit.

You could also add finer grained rules, like if enemy is a certain distance from a unit, move closer and attack or retreat to the base (depending on health, ammo, abilities, etc).

The benefit of this approach is that a rule-based system executes very fast as no training or decision trees are necessary and this frees up a lot of system resources for visuals like physics and graphics.

The disadvantage is that if the rule system is not complex enough, the player will easily recognize the pattern and the game will become predictable and boring.  You will also notice that the more different units you add to the game, the exponentially more complex the rule system becomes as you have to cater and test interaction between each type of unit in the game otherwise players might find a weakness in the game design and exploit to complete missions in ways it was not designed to be completed.

One of the reasons why multi-player games are so popular is that you do not play against set rules, but against creative people who have the ability to comes up with new strategies you had never seen before.


What is the actual learning algorithm: back-propagation or gradient descent?

You can run gradient descent without back propagation, in some cases:

Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost function value.
In "black box" gradient-based learning algorithms where you don't know how (or don't want to) calculate gradient analytically, so you choose to measure it. This is a slow process, similar to gradient checking of neural networks, i.e. adjust each parameter in turn by a small $+\epsilon$, measure your loss function, and divide difference by $\epsilon$ to get gradient estimate for that parameter.

In those cases, you can calculate the gradient, and then use the optimisation function of your choice - e.g. SGD, Adam, Nesterov Momentum - to make changes to the learned parameters. The model will be changed and should learn its task based on the cost function.

You can also run back propagation without gradient descent. In that case, the model will not change and nothing is learned.

That makes the different flavours of gradient descent technically the "core" learning algorithm in my opinion, and back propagation the preferred method for calculating gradients in neural networks. These gradients are most often estimates of an unknown "true gradient" due to training in batches of small samples of training data.

In practice, the two terms back propagation and gradient descent are rarely separated when discussing neural network training. So a lot of people will say that they train a network "using back propagation" or that they are using a gradient descent method without mentioning how they are calculating the gradients using back propagation (because that's pretty much a given). Or they might just say they are "using the Adam optimiser" which in the library they are using automatically runs an NN forward on a mini-batch of data, performs back propagation*, and then changes weight parameters based on the gradients, but heavily adjusted by previous gradient measurements.

* With auto-differentiating frameworks such as TensorFlow, PyTorch etc, it is not really clear that the algorithms are performing "classic back propagation" - although they work using the same chain rule, they may well not perform calculations in the exact same groups that you learned when you studied neural networks and had to write the code for calculating gradients by hand. Instead of back propagating gradients layer-by-layer, they are propagated through an abstract computation graph that includes the learnable parameters.


Which Reinforcement Learning algorithms are efficient for episodic problems?

When applying techniques like SARSA(which are on-policy), one needs to have control over a simulator. If one is able to access only the episodic dataset, then the only choice is to opt for Q-learning or Off-policy Monte-Carlo(or off-policy methods in general).

  Can a reward of zero
  be given in preliminary states before termination state of each
  episodes at which it will be rewarded with a positive/negative
  (win/loss) value?

With regards to the above question, the answer is yes. The task would be a sparse reward task with a reward occurring only at the last transition. The issue that one faces in a sparse reward task is that of slow convergence(or even lack of convergence).

Some guidelines for tackling sparse reward tasks are as follows :

Monte-Carlo, and n-step Q-learning are preferred over Q-learning/SARSA.

Consider the 10-step chain MDP, where the only reward is +1 when the transition from s10 to END occurs.

Let us consider the first episode in training to be start->s1->s2->s3->....->s10->end. The Q-learning update would result in no sane updates to states s1, s2, s3, s9 because the Q-value of the next state is random-value. The only state with a sane update is s10

However, if we use n-step Q-learning or a Monte-Carlo based update. The Q-values of all the states are updated in a logical manner since the reward from the end of the episode propagates to all states in the episode.

n-step Q-learning would be ideal since by adjusting the value of n, one can trade-off the benefits of Monte-Carlo methods(described above) and Q-learning(low variance).
The use of pseudo/auxiliary rewards.

This is not something necessarily recommended since the addition of new reward-structures can cause unintended behaviour. On the flip-side, it could lead to faster convergence.

A simple example is as follows: Consider a game of chess, where the only reward is at the end of the game. Since the game of chess has very long episodes, one can introduce the following reward-structure instead :

+100 for winning the game
+1 for capturing any piece on the board

Hence, the pseudo reward may provide some direction for learning. Note the different scales in the two rewards(100:1). This is necessary because the primary goal of the task should not shift to capturing pieces, but to win the match.


Is word embedding a form of feature extraction?

Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc, on a large corpus). Word Embedding techniques help extract information from the pattern and occurence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.

However, from another standpoint, word-embedding models were not developed aiming to solve a particular feature extraction problem, but rather, to generalize and model the language used in a corpus to gain a semantic understanding of the words and the relationships between them. Such that, all the various corpus-specific tasks can then employ the same "library" of information which was collectively & exhaustively learnt by the embedding model. Meaning, the word embedding model learns a language model that is task-agnostic for all tasks on that corpus unlike feature extraction methods which are specifically task-oriented.

Hence, the similarity is - word-embeddings can effectively aid in feature extraction; the dissimilarity is - they're not primarily meant to extract features more than they are for modeling a language which might be an "overkill" for a particular feature extraction task on a dataset.


What does AI software look like, and how is it different from other software?

Code in AI is not in principle different from any other computer code. After all, you encode algorithms in a way that computers can process them. Having said that, there are a few points where your typical "AI Code" might be different:

A lot of (especially early) AI code was more research based and exploratory, so certain programming languages were favoured that were not mainstream for, say, business applications. For example, much work in early AI has been coded in Lisp, and probably not much in Fortran or Cobol, which were more suited to engineering or business. Special languages were developed to make it easy to program with symbols and logic (eg Prolog).

The emphasis was more on algorithms than clever/complex programming. If you look at the source code for ELIZA (there are multiple implementations in many different languages), it's really very simple.

Before the advent of neural networks and (statistical) machine learning, most AI programming was symbolic, so there hasn't been much emphasis on numerical computing. This changed as probabilities and fuzziness were increasingly used, but even if using general purpose languages there would be fewer numerical calculations.

Self-modifying code is inherently complex; while eg Lisp made no difference between code and data (at least not in the same way as eg C or Pascal), this would just complicate development without much gain. Perhaps in the early days this was necessary when computers had precious little memory and power and you had to work around those constraints. But these days I don't think anybody would use such techniques anymore.

As modern programming languages evolved, Lisp and Prolog (which were the dominant AI languages until probably 20 to 30 years ago) have been slowly replaced by eg Python; probably because it is easier to find programmers comfortable in an imperative paradigm rather than a functional one. In general, interpreted languages would be preferred over compiled ones due to speed of development, unless performance is important.

The move to deep learning has of course shifted this a lot. Now the core processing is all numeric, so you would want languages that are better with calculations than symbol handling. Interpreted languages would now mainly make up the 'glue' code to interface between compiled modules, and be used for data pre-processing. So current AI code is probably not really that different from code used in scientific computing these days.
There is of course still a difference between R&D and production code. You might explore a subject using an interpreted language, and then re-code your algorithm for production in a compiled language to gain better performance. This depends on how established the area is; there will for example be ready-made libraries available for neural networks or genetic algorithms which are well-established algorithms (where performance matters).
In conclusion: I don't think AI code is any more complex than any other code. Of course, that's not very exciting to portray in a film, so artistic licence is used to make it more interesting. I guess self-modifying code also enables the machines to develop their own conscience and take over the world, which is even more gripping as a story element. However, given that a lot of behaviour is nowadays in the (training/model/configuration) data rather than the algorithm, this might even be more straight forward to modify.
Note: this is a fairly simplified summary based on my own experience of working in AI; other people's views might vary, without either being 'wrong'.
Update 2021: I now work at a company that extracts business information/events from news data on a large scale using NLP methods. And we're using Lisp... so it's still in active, commercial use in AI.


What is the Bellman operator in reinforcement learning?

The notation I'll be using is from two different lectures by David Silver and is also informed by these slides.
The expected Bellman equation is
$$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$
If we let
$$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$
and
$$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$
then we can rewrite $(1)$ as
$$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$
This can be written in matrix form
$$\left.
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
\cal{R}_1^\pi \\
\vdots \\
\cal{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
\cal{P}_{11}^\pi & \dots & \cal{P}_{1n}^\pi\\
\vdots & \ddots & \vdots\\
\cal{P}_{n1}^\pi & \dots & \cal{P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\right. \tag 5$$
Or, more compactly,
$$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$
Notice that both sides of $(6)$ are $n$-dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as
$$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$
for any $v\in \mathbb{R}^n$. This is the expected Bellman operator.
Similarly, you can rewrite the Bellman optimality equation
$$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$
as the Bellman optimality operator
$$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$
The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$.
Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.
Specifically, the fact that the Bellman operators are contractions gives the useful results that, for any policy $\pi$ and any initial vector $v$,
$$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$
$$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$
where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$. The proof is due to the contraction mapping theorem.


How to deal with the addition of a new state to the environment during training?

There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)
In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in Exploration by Random Network Distillation. They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.


Data scan not making sense for coco dataset

What is strange is that all 9 nets are getting similar loss at the end (~7.5). I must be doing something wrong, right?

Yes, Please check the annotations file. Did you remove all the other classes except Person? What's the mAP of all those models?

  "how much accuracy can I expect to gain as I increase the data?"

It's hard to tell but more the data(variation), better the model.


What are the main AI technologies behind the Watson platform?

As a person who works with people who work on Watson, perhaps I can give some insight.

The name Watson is casually thrown around a lot whilst many people aren't aware of its evolution into a larger suite of systems and services. We now have Chef Watson, Watson Health, and many other developing projects along the "cognitive" route. Watson is really an amalgamation and varied application of the different cognitive computing routes IBM is pursuing.

So, there are many forms of NLP that Watson conducts and has conducted, developed by different teams to fit different processes, interconnected in different ways. Additionally, much (probably all) of it is proprietary/classified since, as one would imagine, ongoing research is constantly being conducted and added to Watson. This is likely your largest obstacle. The precise workings of the NLP of the Jeopardy flavor of Watson are probably themselves still classified (I can't find anything in the time I've just spent looking myself)

There are, thus, many answers to this question; many outdated, and others not always applicable. The full answer is very complicated and, by the time you find out what the answer is today, it's probably already been advanced. The researches I know are always working on new, cutting-edge algorithms and processes for text classification and the related NLP topics. See, for example, this article http://www.aaai.org/Magazine/Watson/watson.php, where it is stated

  For the Jeopardy Challenge, we use more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses.

and

  Watson's metalearner uses multiple trained models to handle different question classes as, for instance, certain scores that may be crucial to identifying the correct answer for a factoid question may not be as useful on puzzle questions

Most of these techniques have likely already been modified, adapted, or dropped altogether for other methods.

To point you to more information, take a look at the links below.

General

The DeepQA Research Team, specifically, have a look at their publications
IBM Research AI

Articles

Building Watson: An Overview of the DeepQA Project (2010) written by the IBM Watson Research Team, led by David Ferucci, and published in AI Magazine Fall, 2010.

Videos

How does IBM Watson work? (2018), by IBM Watson, which states that Watson uses deep learning and transfer learning

Other

Natural language processing: an introduction (2011), a good but technical paper on natural language understanding (that acknowledges Watson)


What is the difference between Kaldi and DeepSpeech speech recognition systems in their approach?

Both of them using the end-to-end approach for speech recognition. However, because of the code complexity in DeepSpeech, you can't tune the model for your work. Kaldi could be configured in a different manner and you have access to the details of the models and indeed it is a modular tool. I think Kaldi could be a better tool academically and also commercially. But, Deepspeech is a BlackBox and could be a proper tool if your work is near to the work of DeepSpeech.

Moreover, if you are working in a language that there is not enough data for learning, you have a dozen of tools such as grapheme-to-phoneme to establish your dataset to start the learning process. Also, you can start with other models of Kaldi to work on which do not depend on the big data to be learned. However, DeepSpeech requires many hours of samples that is not a common asset for everyone in every language.

Also, you can know more about the traditional technique of speech processing vs the contemporary technique (deep learning) in this medium post.


What can be done to correct for sampling bias introduced from (noisy) training data while training a DNN?

I feel like from the information your giving (some sort of biased data) you can't get an answer as robust as you'd like (what algorithmic changes can be made).
In general, the reason these methods like DNN's work is that they learn from the data. What you train it to do is what it is capable of, and there's little one can do to 'balance' it to classes of data it just never sees. It's like training someone to do algebra then giving them a trigonometry test. It's all math, sure, but you just never can expect much without the proper learning.
That being said, you should perhaps look at other methods to work with this data, or to approach the problem. Given that you cannot collect unbiased data and that you can't explain more due to confidentiality, I really doubt anyone here can help you that much.
I can at the most point you to this article : "Classification on Data with Biased Class Distribution".
And suggest that perhaps your current approach may not be the most appropriate given the unfortunate circumstances.


Regression For Elliptical Curve Public Key Generation Possible?

I don't know of any neural network that can do cryptography well, so you would have to do a little experimenting yourself. The main thing that sticks out to me is that doing operations in the elliptical curve requires the modulus operator since it works in finite field, and I think neural networks have a hard time learning the modulus operator in general. So I would focus on that first. Some things to help the network learn the modulus operator:

I would try to increase the hidden layers to a number 4x-10x bigger than the input dimension, which maps it to a higher dimension to hopefully learn more complex behavior.
I would use less layers, maybe only 2-4 hidden layers, to speed up the development time.
Most importantly, I would train with a LOT of data points (25% of the total possible finite field possibilities). I don't think the neural network can learn unless the number of data is this high.
For reference, someone got a network to learn the modulus operator using these points.

For rapid iterating, I would test with a much smaller finite field. E.g. use a 8 bit security and see if the neural network can do well with that first before moving on to the full 256-bit security key (or whatever your end goal is).
Taking this a step further, I would first test to see if the neural network can even perform a point addition in the elliptical curve well, because if it doesn't then it definitely can't do point multiplication which is needed to compute the public key.


How to reduce variance of the model loss during training?

There are a few things you can play with:

Try reducing the learning rate, or increasing decay.
Try using regularization(L1/L2 or dropout)
Try using momentum(your model may be stuck in a local minima)
Adjust other hyperparams(nodes, layers, batch size, etc.)

Unless you have some knowledge about the specific cause of high loss variance, the above steps in some amount should get you where you need to go.


Is there an alternative to the use of target network?

I have done some research and would like to share.

Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient.

There are two approaches that might work:

Experience reweighting
Constrained optimization

Experience reweighting

Probably the simplest idea is to use importance sampling ratio (Precup 2001) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown (Sutton 2016) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.

Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called Emphatic TD (Sutton 2016). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.

Constrained optimization

The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, Durugkar suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. Achiam 2019 provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence).

Sidenote

Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).

Works like:

Q($\lambda$) (Harutyunyan 2016), which augments the value function with a correction term
Retrace($\lambda$) (Munos 2016), which extends the former by truncating the importance sampling ratio

are shown to work only in tabular case.


Can we stop training as soon as epsilon is small?

How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give rise to a large TD error -- that is a large difference between the returns you expected from taking this action as to what you actually observed. How much the $Q$-value changes depends on the magnitude of this TD error.

$Q$-learning is not guaranteed to converge if there is no exploration. Part of the convergence criteria assumes that each state-action pair will be visited infinitely often in an infinite number of episodes, and so if there is no exploration then this will not happen.


What are the current big challenges in natural language processing and understanding?

According to a nice article by Sebastian Ruder https://ruder.io/4-biggest-open-problems-in-nlp/ based on answers from top NLP researchers https://docs.google.com/document/d/18NoNdArdzDLJFQGBMVMsQ-iLOowP1XXDaSVRmYN0IyM/edit

Natural language understanding
NLP for low-resource scenarios
Reasoning about large or multiple documents
Datasets, problems, and evaluation

I recommend having a look at the article. More details in the slides https://drive.google.com/file/d/15ehMIJ7wY9A7RSmyJPNmrBMuC7se0PMP/view


What would happen to an agent trained using Markov Decision Process if the goal node changes?

It is possible, at design time for a reinforcement learning problem, to allow for changes within an environment. You can make any element into a variable property of the state, that the agent can realistically be told at the start or sense from the environment.
If you do add new variable to model the possibility of change:

It allows the agent to learn to solve a more general problem where the chosen property can vary.

It increases the size of the state space.

It requires training to include variations of the new variable.

Usually this also increases the time taken to train.
It is not always possible to use a state variable for the task - perhaps a goal state is effectively hidden from the agent and the purpose of training is for it to be discovered. In which case, you will require at least some re-training. It may be faster to start with the existing trained agent if the difference is not large.
If you cannot simply extend the state representation, and the environment changes in a small enough way, then it may also be possible to use an agent which continuously explores which will re-train itself over time in repsonse to changes in the environment. The DynaQ+ algorithm is an example of a method which is designed to explore and find changes in the agent's environment to allow for this kind of online retraining when things change.


How difficult is this sound classification?

I think if you got the dataset, then a standard 1d convolutional neural network would work to some extent. It's not that there is some property of nearby sounds that it would pick up on. It would just memorize all the sounds that tend to come from your desk. I think the coding part would be pretty standard stuff. But collecting the data will be hard. You have to get a really big labeled dataset of sounds coming from your desk and sounds coming beyond a 3 foot radius. This dataset has to be realistic and representative of the real world. Getting that dataset would be pretty tricky but it is doable if you put multiple microphones in your house in order to triangulate the exact positions of all sounds. It would be like GPS but using sounds waves instead of light waves.


A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate Modelling: A Practical Guide

In the context of Bayesian optimization, one wants to optimize a function $y=f(x)$ which is expensive (very time consuming) to evaluate, therefore one optimizes the surrogate model $y_S=f_S(x)$ which is cheaper (faster) to evaluate.


What is the difference between 2d vs 3d convolutions?

Looking at it from the perspective of input to output in that fashion is probably not the best. So lets start with our goal and how these ND convolutions accomplish that (Note these are in my own words, and may not be best stated).

Assumption: There exists highly correlative local associations

Goal: Have a linear model that takes advantage of these local associations

Solution: The ND Convolution

Explanation: ND convolutions take advantage of our locality assumption by connecting only local nodes/neurons. The fact that its a sliding window allows us to learn filters for any location along with ones that can be reusable.

Your Question:  Where does the N in ND convolution matter?:
When using an ND convolution we are working off the assumption that there exists this locality in N dimensions and nothing more. So we connect all other components of the input in a dense matter because we have no assumptions to work on in this space. So now going to the shapes you mentioned such as the input and output of the 2D convolution. We are convolving a filter of size $(C_{in}, k_h, k_w)$ with a $(C_{in},H,W)$ activation shaped map (the $N$ just refers to the number of activation maps, and there is no association between them). We use $C_{in}$ channels on the kernel because we are not making any assumptions about locality between channels.  On the other hand in a 3D convolution we make locality assumption is 3 dimensions, so our kernel will be ($C_{in}, k_h, k_w, k_r$). These kernel sizes are actually determined by the input size, the amount of dimensions of the kernel will actually match the inputs (minus the batch) because it needs to densely match each one.

You may be thinking now, that in torch this is not the case: This is because its rare to want a 2D Convolution with an inputs that's different than the one you mention, so they only implemented it for a singular shape. I hope this clears up the convolutions and helps you understand not just for 2 and 3 dimensional convolutions, but for all N.


How to normalize images before training?

tl;dr subtracting the mean and dividing by the standard deviation is theoretically more sound, but is impractical compared to dividing by $255$.

Explanation
As you know neural networks perform better when their input is scaled. The 2 most common ways to perform scaling are:

normalization, where you want to scale the image to the $[0, 1]$ range.
standardization, where you want to bring the mean to $0$ and the standard deviation to $1$.

Theoretically, I think, the latter has some advantages, however in practice there is not a significant difference, especially if the network has normalization layers.
Images are usually stored in 8-bit color mode, meaning that they take integer values from 0 to 255. Because you know the minimum, and maximum values that each pixel can take, it is extremely easy to normalize an image (i.e. you simply have to divide each pixel value by $255$).
On the other hand it is much less practical to compute the mean and standard deviation of the pixel intensities of the training set. Keep in mind that in most cases the training set doesn't even fit into memory, and you have to approximate the true mean/std by moving averages (i.e. load a few images, compute their mean/std, close them, load some more, add their mean to the MA, ...). But why bother if you can similar results by simply dividing each image by $255$.


For such time-series data that has a significant amount of periodicity, I would recommend converting data to the frequency domain and performing various spectral analysis methods as @firion has already mentioned. For example, you could perform Fourier Analysis and study the individual components and identify patterns there.

Also, it generally not recommended to perform the normal pattern extraction approaches to time-series data as they fail to understand the temporal relationship between subsequent data points.

Hope this helps!


Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ and $\sum\limits_{n=1}^\infty \alpha_n^2(a) < \infty$. These conditions imply that in the limit each state-action pair will have been visited an infinite number of times, thus we will have walked around the whole table, so there are no out-of-sample state-action pairs.

However, in the case of function approximation, then convergence is no longer guaranteed. Generalisability is possible though - assuming we have an infinite state or action space then we will only ever visit the same state-action pair once, so the role of a function approximator is to allow us to generalise the state/action space.

NB that the convergence conditions I mentioned are only required in some proofs of convergence, depending on what type of convergence you are looking to prove. See this answer for more details.


Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?

It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.


Which nodes are expanded in the expansion phase of MCTS?

By far the most common (and likely also the most simple / straightforward) implementation is to expand exactly one node in the Expansion Phase; specifically, the node corresponding to the very first state selected (semi-)randomly by the Play-Out Phase. This is also pretty much the bare minimum you have to do if you want any form of tree growing at all (which you do).

Other variants are possible too, but are much less common. The variant you suggest in the question is to expand all the children of the final node encountered during the Selection Phase, and run a Play-Out for all of them. I am not familiar with any literature on such a strategy really, never tried that myself. Intuitively, I would expect it to perform very similarly, perhaps slightly worse. Essentially what this would do is that it moves the "behaviour" of the search algorithm slightly more towards Breadth-First Search behaviour, rather than Best-First search behaviour. You spend a bit less of your computation time in the Selection Phase, because every Selection Phase is followed up by for example 6 (or whatever branching factor you have) Play-Outs instead of just a single one. On average I'd expect this to be slightly worse, because the Selection Phase is the primary source of the "Best-First Search" behaviour of the algorithm. I certainly don't expect a change like this to cause a large difference in performance though, if any. It will also likely be domain-dependent; worse in some cases, better in other cases.

A different variant that I did once use myself is to expand every node in the complete line of play followed by the Play-Out phase. You can visualize this as a very "thin", but "deep" expansion, whereas your suggestion discussed above would be visualized as a "shallow" but "wide" expansion (and the conventional expansion strategy of a single node would be "thin" and "shallow").

For this strategy, it is much easier to clearly define the advantages and disadvantages that it has in comparison to the standard strategy. The main advantage is that you retain more information from your Play-Outs, you throw less information away. This is because the Backpropagation phase, after the Play-Out terminates, can only store information in nodes that exist. If you immediately expand the complete line of play followed in the Play-Out Phase, you can store the result (the evaluation in the terminal state) in all of those nodes. If you don't expand the complete Play-Out (e.g. only expand the very first node), you'll have to "skip" all of those nodes which you didn't expand yet (they don't exist), and you can't store results in there yet.

The main disadvantage of this approach is that it requires more memory, the tree grows a lot more quickly.

I would personally recommend this approach if you have very strict limitations on computation time, if you expect to be able to only run very few iterations of the MCTS algorithm. For example, I personally used this in my General Video Game AI agent; this is a real-time game where you have to make decisions every ~40 milliseconds. In such a low amount of time, you cannot run many MCTS simulations. This means that:

You do not expect to run out of memory, even if you grow your tree very quickly, so the increased memory requirements become a non-issue.
Due to the low expected number of iterations, it is extremely important to retain as much information as possible, not throw any information away. If we can't run many simulations, we want to make sure to squeeze every little bit of information we can out of each of them.

For contrast, if you're developing an agent to play a board game, and it has in the order of multiple minutes of thinking time per turn, the standard approach of only expanding a single node per Expansion Phase becomes a lot more appealing. If you're capable of running tens of thousands of iterations or more, it really doesn't hurt if you "forget" about a little bit of information deep down in the tree. The risk of running out of memory also becomes a lot more serious if you're running many iterations, so you don't want to grow the tree too quickly.


Can most of the basic machine learning models be easily represented as simple neural network architectures?

I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point.
I don't think there is a relation between this argument and universal approximation theorems, which state that certain classes of neural networks can approximate any function in certain domains, rather than any learning algorithm.


What does it mean to do multi-dimensional processing with tensors in tensor cores?

Both PR and Theoretically Valid

Although Anima Anandkumar's presentation is a puff piece for NVidia, her representation is not contrary to theory.

  ... next level [above NVidia's GPU success] ... means new algorithmic research. So, if you think about the current computations in our deep learning systems, they are all based on linear algebra. Can we come up with better paradigms to do multi-dimensional processing. Can we do truly tensor algebraic techniques in our tensor cores, and what kind of new architectures will this realize?

Tensors

A tensor is the extension of the idea of functions and vectors. Vectors emerged allow the encapsulation of multiple variables of the same type into a single unit to express multidimensional properties such as position or force. Variables that are not encapsulations of multiple variables are then referred to as scalars. When such multidimensional properties are compositions of functions rather than variables because their values are dependent upon other variables, they are vector fields.

To more cleanly model electromagnetism and gravity in mathematics, higher levels of variable and functional encapsulation were required. Scalar variables and functions that return scalar variables are thus tensors of rank one (1). Vectors and vector fields are tensors of rank two (2), and higher ranks are extensions of the pattern created by scalars and vectors. A matrix containing functions can be represented as a tensor of rank three (3), encapsulating its elements into a single unit.

Digital Processing and Integration Trend

CUDA cores, which are the signal processing units used in current generation NVidia GPUs (graphical processing units) can be used for production of two or three dimensional rendering or leveraged to produce parallel processing of signals through an artificial network. This is following the trend to delegate to digital circuitry the most significant bottlenecks in algorithm execution. VLSI (very large scale integration) technology is the logical result of this trend.

Industrial and military calculators with tests and branching running on rows of racks of relays
Add speed with tubes and load programs with punch cards
Reduce power and improve reliability and speed with transistors, magnetic core memory, and paper tape
Integrated circuits on motherboards with vinyl tape
Handheld programmable calculators for students
Microprocessors with magnetic disks
Floating point, digital signal, and graphics processors to extend microprocessors
Multiple cores (repeating large scale circuit patterns on a single substrate)
Re-purposing and extending graphics rendering circuits to offload artificial network computations to GPUs

With all these, speed, size, energy thrift, and convenience are not really a shift in the original paradigm that started with Norbert Wiener, Claude Shannon, Alan Turing, John von Neumann, and others. In fact, computing is still catching up in several ways with the perspectives of these pioneers and a long way off from producing in VLSI common concepts in science. Software is the solution, which is why it is called soft, meaning flexible, not necessarily weak. However, flexibility sacrifices speed and capability, thus the trend above.

The Parallel Computing Challenge

Part of what Anima Anandkumar is stating that the signal paths in current VLSI processors are still a much lower level of abstraction than the ideas of mathematicians, physicists, and AI engineers. Scientific theory describes probability, statistical distribution, expectation, force, loss, gain, pain, reward, memory, momentum, semantics, combination, and correlation at a much higher level than current digital circuitry.

The use of sequential algorithms is the splaying out in time what could be a massively parallel operation. The serial algorithm limits the rate of processing. The finding of ways of dealing with things in parallel in mathematics can be done with a pencil. In computers, paralleling algorithms and finding parallel processing structures in VLSI form that are as flexible as software in some respects is much more challenging and thus far behind.

This is one problem that has been the focus of research in at least twenty large corporate and government labs since for half a century and has been the intention of VLSI from the onset. The work is not specific to NVidia. It is not a new problem and the approach to solving it has been along this paradigm.

Add abstraction and encapsulation in the mind.
Express it in mathematics.
Write it as a serial algorithm, leveraging whatever parallel constructs are supported by programming languages and libraries that can leverage VLSI parallelism or computing clusters.

The last seventy years of computing machinery development has been bringing hardware, operating systems, and software closer to the level of mathematical expressions that were twenty to two hundred years old. That may change, and everyone wants to ride the new wave.

Anima Anandkumar and her counterparts in IBM, Intel, Google, Microsoft, the U.S. Navy, Amazon, Alibaba, and the other corporate and government laboratories do not state (because it is either classified or company confidential) what they intend to do to further the parallelization of computing. Whatever they do along those particular lines would not be a paradigm shift but rather a next step along the current paradigm.

Enters Corporate Strategy

They would also not state what they may be doing that is not in that paradigm. They are constrained to only give hints without theoretical substance. If they were working on a chip that exhibited what the human brain exhibits when neurons grow and connect according to DNA based propensity, they wouldn't say that in a technically precise way. Corporate secrecy is part of the global economic game play, not tipping the hand. They have their game face on.

The idea of moving from linear to non-linear is a good public relations theme and when used for puff pieces is not conclusively technical. The pitch usually goes along this line.

  What they did was very linear. We are moving into a non-linear space.

It is an attempt to claim that what was done was primitive and the impending game changing advancement is coming from the speaker and their people. Sometimes it works to create a temporary hike in the value of tradable securities, which that company needs as of this writing. If there is a true change in the game, it will be known when it is released. Those who have worked in laboratories for years know to wait until something is released that demonstrates, when the example code is configured and ran, what game change actually occurred, if any. Or they develop the game changer themselves, which is why it is a lab.

Ambiguity of 'Linear' Even in Mathematics

Also note that curved lines are still lines and the term linear can mean two things depending on the context.

Conformant to the linear equation $\vec{Y} = V \vec{X}$, such that it graphs as a line, plane, or higher dimensional flat surface, with constant gradient and without curvature.
Conformant to the principles of linear algebra, which would include spaces, eigenvalues, regression with high-order polynomials, and a number of constructs that involve varying gradient and curvature.

Predicting Paradigm Shifts

The skepticism developed after watching tech company PR for some time does not necessarily dismiss game changing technology advancements of the past and potentials of the future. Any open source project team, individual, corporation, or government lab might do something that shifts the paradigm, usually over a period of years. Classic examples:

Rope
Wheel
$F = m a$
Oxygen (that air is not an element)
Electromagnetism
Alternating current energy transmission
Relativity
Internal combustion
Radio

Computing examples:

Information theory
Cybernetics
First transistor
LISP and FORTRAN
C and UNIX
TCP/IP

Shifts do not need to be as far reaching and game changing as these to have an impact. Who will achieve the next with regard to AI concepts, information structure, algorithm, execution environment is not something very easy to predict. Consider those that lived before any of the above and try to imagine them trying to predict that the projects of Isaac Newton, Antoine Lavoisier, Michael Faraday, Nicola Tesla, Norbert Wiener, Claude Shannon, Ken Thompson, Dennis Ritchie, or any of the others were the seeds of the next paradigm shift.

There is research into analog artificial networks, neuromorphic hardware, semantic modelling, graph algorithms, and other potential game changers, each of which has impressive conceptual foundations and is discussed in some of the Q&A here. These are a few.

What topologies are largely unexplored in machine learning?
Novelty Search Mutation Algorithm
Can deep networks be trained to prove theorems?
If digital values are mere estimates, why not return to analog for AI?
Are commercially available neural ICs digital?

What of these may point to the beginnings of paradigm shift cannot be known, and even if an idea first posted here or referenced from here is the seed, it may not be known later. The multilayer perceptron may be the seed of future streets and highways being dominated by automated vehicles in 2090, but no one in a million people will realize in seventy years that the trend toward AV research was seeded by MLP enthusiasm from the first decade of this century.

Gaps to be Filled and NVidia is a Contendere

All this aside, we use NVidia hardware every day for robotics and analysis, so they have credibility resulting from past success. If they produced a chip that does something remarkably clever before Intel or the IBM-MIT collaboration, it would be a small surprise, but not it is not a completely unbelievable possibility. Certainly the comprehension of Hilbert spaces, semantics, and topology are limited in the computer science field, and a paradigm shift toward greater comprehension of them or some new thing that is not even part of mathematical thought today would add some needed diversity to the computing industry.


Can a large discrete action space be represented using Gaussian distributions?

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value $Q(s,a_{n})$ and $Q(s,a_{n+1})$ is not correlated for any $n$, then it will not be possible to learn with the approximation at all.
You may have some sense of how similar actions $a_{n}$ and $a_{n+1}$ are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. $a_{900}$ and $a_{901}$. If this holds in general, then that is a good indicator that you can treat $n$ as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).
It might not matter if for a small fraction of cases the difference in outcomes between $a_{n}$ and $a_{n+1}$ is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean $\mu(s)$ and standard deviation $\sigma(s)$.
There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action $a_n$ is optimal for a given state $s$, but $a_{n-1}$ and $a_{n+1}$ are a lot worse, then the training process of a typical policy gradient approach may never settle upon $\mu(s) = n$ and $\sigma(s) \approx 0$, because any intermediate values between the starting policy and the optimal one will perform badly.
For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.


This sounds like a great project, although this exact setup limits your options somewhat.  Supervised machine learning approaches are effectively ruled out because you don't have the necessary training data to develop a model (i.e. the dependent variable: whether it is raining or not).  You could look at accessing similar data (from what source depends on your geographical location) that includes precipitation information, and attempt to train a model based on that, before applying it to your own data to 'predict' the current conditions.

If you want a model to determine whether it's raining or not (rather than attempting to define the type, rate or other characteristics of precipitation) then you need a binary classification model.  If you can get the data to train a supervised model, there are many such models available.  Which one to choose depends on a number of factors – enough factors that it is as much an art as a science (and often the best way to choose is just to try several and pick the best one).  XGBoost is a popular choice for a wide range of problems, including binary classification.  State Vector Classifiers (SVCs, a subtype of State Vector Machine, SVM) are also common.  Of course there are neural networks also.  Depending on how the dependent variable correlates with the independent ones, you may find a simpler model to be more effective.  If you choose to try a neural network, I suggest keeping it simple and starting with a Multi-Layer Perceptron (MLP).

Alternatively, you could try an unsupervised approach, which commonly takes the form of clustering.  In this instance, you would look to try clustering algorithms, such as DBSCAN, to see if the independent variables naturally form groups (clusters) which differentiate between when it is raining or not.  Of course you can cluster the data using an unsupervised model, but you would still need to then correlate that with rainfall in order to make sense of the model.

There are a number of challenges you may need overcome.  For example, depending on the frequency with which the data is sampled and stored, you may need to downsample it to a manageable size.  Presumably you also have the time at which the data was collected.  If not, you would also need to make use of the brightness sensor to differentiate between night and day, which may confound or over-power any cooling effect from rainfall.  (You could look at clustering the data, based on temperature and brightness, in order to create a new feature which indicates night and day.)

To summarise, there are numerous approaches you could take, all of which will, at some point, require data on whether it is/was raining or not in order to train a model.  If you have suitably localised weather data available, you may be able to train a model using that, synchronised via time and/or the other data you'll be collecting.  The problem, as stated, is a binary classification one (as opposed to a time series prediction one, i.e. weather forecasting, for example).  So look into classification and clustering models for supervised and unsupervised machine learning respectively.  Scikit Learn for python or Caret for R are commonly used packages for the types of models I've described.


Facial Recognition + Database + Compare & Identify - is it complicated?

Depending on the level of accuracy/speed you want your system to have, you could try using an existing python API. First thing I found on google, the README looks adapted to your project. Otherwise ask your question with more accurate wanted specifications perhaps.


Where can we use the Bayes' theorem in artificial intelligence?

A toy example:

Suppose an agent has information about the reliability of fire alarms. It may know how likely it is that an alarm will work if there is a fire i.e. $P(alarm | fire)$. But, if the problem is tweaked a bit and if the agent must know the probability that there is a fire, given that there is an alarm, it can use Bayes' rule:

$P(fire | alarm) = [P(alarm | fire) ×P(fire )] / P(alarm )$

where:

$P(alarm | fire)$ is the probability that the alarm worked, assuming that there was a fire. It is a measure of the alarm's reliability.

The expression $P(fire)$ is the probability of a fire given no other information. It is a measure of how fire-prone the building is.

$P(alarm)$ is the probability of the alarm sounding, given no other information.


Early success on prime number testing via artificial networks is presented in A Compositional Neural-network Solution to Prime-number Testing, László Egri, Thomas R. Shultz, 2006.
The knowledge-based cascade-correlation (KBCC) network approach showed the most promise, although the practicality of this approach is eclipsed by other prime detection algorithms that usually begin by checking the least significant bit, immediately reducing the search by half, and then searching based other theorems and heuristics up to $floor(\sqrt{x})$.  However the work was continued with Knowledge Based Learning with KBCC, Shultz et. al. 2006
There are actually multiple sub-questions in this question.  First, let's write a more formal version of the question: "Can an artificial network of some type converge during training to a behavior that will accurately test whether the input ranging from $0$ to $2^n-1$, where $n$ is the number of bits in the integer representation, represents a prime number?"

Can it by simply memorizing the primes over the range of integers?
Can it by learning to factor and apply the definition of a prime?
Can it by learning a known algorithm?
Can it by developing a novel algorithm of its own during training?

The direct answer is yes, and it has already been done according to 1. above, but it was done by over-fitting, not learning a prime number detection method.  We know the human brain contains a neural network that can accomplish 2., 3., and 4., so if artificial networks are developed to the degree most think they can be, then the answer is yes for those.  There exists no counter-proof to exclude any of them from the range of possibilities as of this answer's writing.
It is not surprising that work has been done to train artificial networks on prime number testing because of the importance of primes in discrete mathematics, its application to cryptography, and, more specifically, to cryptanalysis.  We can identify the importance of digital network detection of prime numbers in the research and development of intelligent digital security in works like A First Study of the Neural Network Approach in the RSA Cryptosystem, G.c. Meletius et. al., 2002.  The tie of cryptography to the security of our respective nations is also the reason why not all of the current research in this area will be public.  Those of us that may have the clearance and exposure can only speak of what is not classified.
On the civilian end, ongoing work in what is called novelty detection is an important direction of research.  Those like Markos Markou and Sameer Singh are approaching novelty detection from the signal processing side, and it is obvious to those that understand that artificial networks are essentially digital signal processors that have multi-point self tuning capabilities can see how their work applies directly to this question.  Markou and Singh write, "There are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics."
On the cognitive mathematics side, the development of a mathematics of surprise, such as Learning with Surprise: Theory and Applications (thesis), Mohammadjavad Faraji, 2016 may further what Ergi and Shultz began.


What is layer freezing in transfer learning?

Why is this layer freezing required?
It's not.
What are the effects of layer freezing?
The consequences are:
(1) Should be faster to train (the gradient will have far less components)
(2) Should require less data to train on
If you do unfreeze the weights, I'd think your performance would be better because you are adjusting (i.e., fine-tuning) the parameters to your specific problem at hand. I am not sure what the marginal improvements are in practice, as I have not experiemented much with fine-tuning (like are the improvements typically a 0.01% reduction in error rate? Not sure.)


It is likely converging to a far worse local optimum from which it can't recover, so yes I would guess if all else is the same, that is where the issue would be. I would first try to adjust hyperparameters from a static seed for weight initialization.  ε and α, are likely good places to start.

If that fails, adjust some architecture params like the number of units or layers in the network as well as the starting seed value.

You seemingly have run into a core problem at the heart of ML research which is reproducibility, which is often predicated by initialization.


Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes word vectors as input and produces estimates for the probability of the next word as outputs but it is auto-regressive as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.
BERT, by contrast, is not auto-regressive. It uses the entire surrounding context all-at-once. GPT-2 the context vector is zero-initialized for the first word embedding.


How could human behavior and relationships be implemented?

In general, what you are describing implies a hierarchical sequence model, in which mannerisms adapt to the regime or paradigm in effect.  Expressive modalities are how we recognize the operative context from the behaviors of other agents.  For an artificial agent, avoiding un-canniness would involve clustering the factors underlying the classification of discourse contexts, tagging them with corresponding factor models for manner in a way which closely aligned to the classifications which human agents make.  This involves developing an adequate latent representation for both spaces, which is likely to involve quite a lot of training data, or some clever transfer learning in conjunction with one-shot techniques (generally, taking samples as modes).

When the context becomes one of friendship, for example, the style factors of expression should extend to factors of trust, collaboration, disclosure, sympathy.  In order to implement these, layered abstractions in the representation of behaviors will need to be crafted, presumably by training.  In order to align these learned categories to human categories, which I suppose to be essential to emotional fluency, exploiting the structure of corpus semantics - including distributional characteristics of the linguistic labels and inferences - when estimating loss gradients seems a natural strategy, which exploits cultural learning.

These comments are necessarily speculative, as no such systems are current, to my knowledge.  At least, not in any significant degree of maturity or application.  Certainly other approaches are conceivable. I am just extrapolating plausible strategies, in the context of current technologies, strategies for implementing the kinds of behaviors you describe in a useful way.

More, specifically, I am considering the question as a manifold-learning problem, and hence requiring a representation model in which the inputs and outputs vary over state-spaces, with the behavior being learned as a mapping between those spaces.  Each of those components (the input space representation and its natural topology, the output space representation likewise, the mapping between them, and the sequence model hierarchy which extrapolates from the manifold to a control process emsemble) has its own peculiar challenges, and the challenges to creating an adequate implementation are a confound of those.  Still, it seems more a question of time and money than of notional feasibility.


Why do my rewards fall using tabular Q-learning as I perform more episodes?

The problem here is likely related to the state approximations you are using.

Unfortunately, OpenAI's gym does not always give reasonable bounds when using env.observation_space, and that seems to be the case for CartPole:

>>> env = gym.make('CartPole-v0')
>>> env.observation_space.high
array([4.8000002e+00, 3.4028235e+38, 4.1887903e-01, 3.4028235e+38],
      dtype=float32)
>>> env.observation_space.low
array([-4.8000002e+00, -3.4028235e+38, -4.1887903e-01, -3.4028235e+38],
      dtype=float32)

Processing this, similarly to your code:

>>> discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE
__main__:1: RuntimeWarning: overflow encountered in subtract
>>> discrete_os_win_size
array([0.48000002,        inf, 0.0418879 ,        inf])

>>> discrete_state = (state - env.observation_space.low)/discrete_os_win_size
>>> discrete_state
array([11.27318768,  0.        , 19.50682776,  0.        ])

That means that all the velocities will get squashed down to $0$ in your approximation. Your agent cannot tell the difference between a nearly static balancing position (generally the goal) and transitioning through it really fast - it will think that both are equally good. It is also not able to tell difference between moving towards balance point, or moving away from it.

I suggest you check for what reasonable bounds are on the space (a quick look suggests +/- 2.0 might be a reasonable starting point) and use that instead.

The approximation approach of discrete grid is also very crude, although it does allow you do use tabular approaches. If you want to stick with a linear system (and avoid trying neural networks and DQN) then the next step up would be some form of tile coding, which uses multiple offset grids to obtain smoother interpolation between states.


Lego minifigure facial recognition: where to start?

Since you are mainly interested in starter libraries and packages to read up on here are some pointers:

Image recognition as well as general spatial classification mainly consists of two major tasks.

Translating images to data that can be fed into a ML model of any kind.
Building a model that is able to use this data to complete tasks like classification (which lego figure is this?), spatial clustering (which pixels form the "face"), etc.

For the first task you will find a lot of python libraries that suit the task but `` PIL/Pillow``` should be the main library for this. So start by reading some documentation and transferring the images you do have (this includes standardizing and cleaning the images to improve the results of step 2).

For the second task the actual model you will use depends on your task but generally speaking some form of neural net is a good place to start and if you prefer python then you should look into tensorflow and keras (easy to use interface to tensorflow).

Where to start

Start with the classical MNIST numbers recognition case and if you have grokked that, you will now where to go from there or at least be able to ask a more specific question here.

References

https://pillow.readthedocs.io/en/stable/
https://keras.io/
https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-dataset-54c35b77a38d


An example of a unique value function which is associated with multiple optimal policies

Consider a very simple grid-world, consisting of 4 cells, where an agent starts in the bottom-left corner, has actions to move North/East/South/West, and receives a reward $R = 1$ for reaching the top-right corner, which is a terminal state. We'll name the four cells $NW$, $NE$, $SW$ and $SE$ (for north-west, north-east, south-west and south-east). We'll take a discount factor $\gamma = 0.9$.

The initial position is $SW$, and the goal is $NE$, which an optimal policy should reach as quickly as possible. However, there are two optimal policies for the starting state $SW$: we can either go north first, and then east (i.e., $SW \rightarrow NW \rightarrow NE$), or we can go east first, and then north (i.e., $SW \rightarrow SE \rightarrow NE$). Both of those policies are optimal, both reach the goal state in two steps and receive a return of $\gamma \times 1 = 0.9$, but they are clearly different policies, they choose different actions in for the initial state.

Note that my language was slightly informal above when talking about "policies for the starting state". Formally, I should have said that there are two optimal policies that select different actions in the starting state (and the same actions in all other states).


How should one standardize input when transfer learning

You should use the same transformation (i.e same mean and std in case of z-score normalization) on the new data while using any pretrained model.
If the pretrained model was trained without any normalization then it does not matter if you use normalization or not as long as range of input data is fixed (0-255 for the 8-bit input images). You can use pretrained models with batch normalization which takes care of internal covariance shift and thus there is no need to normalize data.


What are the most important components in the algorithm of a minimum AI?

Some good places to start would be cognitive architectures and as mentioned in another answer intelligent agents. The question is broad but you definitely want to look into planning & decision making. You might also want to check out the L5 and L6 layers of Hierarchical Temporal Memory (As in Nupic) as it relates to feedback, behavior and attention.

If I were you I'd aim for more cognitive solutions (I realize that term is a bit ambiguous itself when we talk about machines). There's also new AI initiative going on involving probabilistic programming. See Probabilistic Models of Cognition made by Goodman (Stanford University) and Tenenbaum (MIT) or Anglican made by Wood (University of Oxford) et al.


If certain moves are compulsory, will there still be a need for a quiescence search?

I understand your question to be:

  If some moves are compulsory, and my agent has no choice about which move to make next, do I need to perform a search, or can I just return the compulsory move?

The answer depends on what your goal is.

If your goal is to make an interactive agent that will play the game against you, then you are correct: there's no need to perform a search. Just return the compulsory move, and then run the search next time your agent has a choice about what to do.

If your goal is to determine the optimal way to play a game, or the expected payoff from a certain game position (another common use of search techniques), then you should run the search as normal, since the forced move won't necessarily lead to a particular end state.

Tangentially, if you're interested in ways to speed up search for checkers, check out the Chinook papers. There's a popularized account here, and more technical ones here and here by Schaeffer et al.


Feature scaling strategy for many feature with very large variation between them?

You should check the distribution of each feature and scale them accordingly, but in any case you should aim to roughly the same interval of values for every feature.
For example, if f1 has the standard distribution and f2 is close to the uniform one, then you can scale f1 to N(0,1) and f2 to U(-1,1).
In other words, try to have maximum, minimum and mean values for all features as close as possible, while keeping their original distributions.


Can BERT be used for sentence generating tasks?

For newbies, NO.

Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.

For advanced researchers, YES.

You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition). Though the text generation quality is hard to control.

Here's the technical report BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model, its errata and the source code.

In summary:

If you would like to do some research in the area of decoding with
BERT, there is a huge space to explore
If you would like to generate
high quality texts, personally I recommend you to check GPT-2.


What are some intermediate or advanced books on neural networks?

Neural Network Design (2nd edition) by Hagan et al. is one resource you could look at. It's a huge tome, weighing in at over 1000 pages in pdf form, but it is freely available (you can also buy a dead-tree version if you really want one).


What are some tactics for recognizing artificially made media?

Digital Media Forensics (DMF) field aims to develop technologies for the automated assessment of the integrity of an image or video, so DMF is the field you are looking for. There are several approaches in DMF: for example, those based on machine learning (ML) techniques, in particular, convolutional neural networks (CNNs).

For example, in the paper Deepfake Video Detection Using Recurrent Neural Networks (2018), David Güera and Edward J. Delp propose a two-stage analysis composed of a CNN to extract features at the frame level followed by a temporally-aware RNN to capture temporal inconsistencies between frames introduced by the deepfake tool. More specifically, they use a convolutional LSTM architecture (CNN combined with an LSTM), which is trained end-to-end, so that the CNN learns the features in the videos, which are passed to the RNN, which attempts to predict the likelihood of those features belonging to a fake video or not. Section 3 explains the creation of deepfake videos, which leads to inconsistencies between video frames (which are exploited in the proposed method) because of the use of images with different viewing and illumination conditions.

Other similar works have been proposed. See this curated list https://github.com/aerophile/awesome-deepfakes for more related papers.


why the sigmoid function will be 1 and 0 if we use a fully connected layer that produce a big enough positive(res negative )output

In general it's better to not use sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use sigmoid-like function, then go with Tanh function, at least it has ~0 mean.


Open-source tools or libraries to experiment with neural networks and evolutionary algorithms

https://github.com/wagenaartje/neataptic

As this is written in Javascript and does not (yet) offer GPU support, it is quite slow. However, it is very nice to fiddle around with flexible network architectures. The only visualisation that it offers right now is a map of network architecture, but graphs could easily be implemented.


Using the book's random walk example, If you have a state space with $1000$ states and you divide them into $10$ groups, each of those groups will have $100$ neighboring states. The function for approximation will be
\begin{equation}
v(\mathbf w) = x_1w_1 + x_2w_2 + ... + x_{10}w_{10}
\end{equation}
Now, when you pick a state, the feature vector will be a one-hot encoded vector with $1$ that is placed in a position that depends on in which group does the chosen state belong. For example if you have state $990$ that state belongs in group $10$ so the feature vector will be
\begin{equation}
\mathbf x_t = [0, 0, ..., 0, 1]^T
\end{equation}
what this means is that the only weight that will be updated is weight $w_{10}$ because gradients for all other weights will be $0$ (that's because features for those weights are $0$)


Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is:

Add the previous action choice to the state representation.

It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.
By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, which you had otherwise lost by making the reward dependent on a variable that was both systematically changing and hidden from the agent.
The state in a MDP is often not just the current observations that the environment provides, but can include any relevant knowledge that the agent has. At the extreme that can include a complete history of all observations and actions taken to date. It is common practice to derive the state as a summary of recent history of observations and actions taken so far. In your case, all you need do is concatenate the previous action to the observation, because you know about the constraint and how it affects optimisation.


Why are weights not initialized with mean=1?

Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :

It may be easier for the network to learn identity function, but we may have a similar issue about not being able to learn comparison, comparison is quite an important reasoning in my opinion, this is why having negative weight values is important, and initializing all weights around 1 may make it difficult for the network to get negative weights.
Maybe it causes an issue with the non-linearity of the activation functions. If we use ReLU function for example, it only is non linear if there are negative values in the network. It is not uncommon to have positive inputs for a network, so initializing all weights around 1 may lead to have only positives values on the network, which makes ReLU function linear.

Anyway weight initialization has always been a complicated topic and it seems there is no definitive answer to what should be done about initializing weights. I am not an expert of the topic, just giving you my thoughts about it.


How do I choose the number of neurons in the fully-connected layer before the softmax layer?

How can I decide the number of neurons in the FC layer before the softmax layer?

Train different network architectures (with different numbers of neurons in the last FC layer). Use cross-validation - a set of data you have not trained on - to measure the performance of the network, using a metric that you have decided beforehand is a good proxy for your experiment goals. For instance, you might choose getting the highest classification accuracy as your goal - but might choose something different for an unbalanced dataset, because 90% accuracy is not meaningful.

There is not usually a good reason to over-tune your network. Trying some variations with 1.5 x or 2 x geometric series (e.g. 5, 10, 20, 40 neurons in layer) is probably enough to find a good hidden layer size.

  Is it $N * 5$, where $N$ is the number of classes?

No, in general. That may work just fine for your problem though.

  Is there any documentation for deciding the number of neurons in the FC layer (before the softmax layer)

Not really. The most direct thing to do is find solutions that work well for similar problems to yours, and adapt as necessary. So you can search for any similar problem domain and see what the researchers used there.

There is not much theory to guide you here. However, if it helps with your intuition, more neurons can make a better fit to higher frequency variations in the target function you are learning, whilst more layers will mean better handling of complex function spaces (e.g. where rules about the mapping between input and output can be made according to combining simpler rules) - provided in both cases that you have enough training data that it is possible for the NN to learn the function approximately. These are not strictly defined traits of functions for machine learning, so many NN users will work with this intuitive view.

If in doubt, assuming this is not an image, NLP or other well-studied problem, then I might just guess at e.g. 64 neurons per hidden layer, and try 1, 2, and 3 hidden layers as a starting point (all the same size). I cannot say if that will work for you and your problem, but it might help get past the "blank page effect" and start you training and testing some variations.


How can we process the data from both the true distribution and the generator?

The Focus of This Question

"How can ... we process the data from the true distribution and the data from the generative model in the same iteration?

Analyzing the Foundational Publication

In the referenced page, Understanding Generative Adversarial Networks (2017), doctoral candidate Daniel Sieta correctly references Generative Adversarial Networks, Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio, June 2014.  It's abstract states, "We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models ..." This original paper defines two models defined as MLPs (multilayer perceptrons).

Generative model, G
Discriminative model, D

These two models are controlled in a way where one provides a form of negative feedback toward the other, therefore the term adversarial.

G is trained to capture the data distribution of a set of examples well enough to fool D.
D is trained to discover whether its input are G's mocks or the set of examples for the GAN system.

(The set of examples for the GAN system are sometimes referred to as the real samples, but they may be no more real than the generated ones. Both are numerical arrays in a computer, one set with an internal origin and the other with an external origin.  Whether the external ones are from a camera pointed at some physical scene is not relevant to GAN operation.)

Probabilistically, fooling D is synonymous to maximizing the probability that D will generate as many false positives and false negatives as it does correct categorizations, 50% each.  In information science, this is to say that the limit of information D has of G approaches 0 as t approaches infinity.  It is a process of maximizing the entropy of G from D's perspective, thus the term cross-entropy.

How Convergence is Accomplished

Because the loss function reproduced from Sieta's 2017 writing in the question is that of D, designed to minimize the cross entropy (or correlation) between the two distributions when applied to the full set of points for a given training state.

  $H((x_1, y_1), D) = 1 \, D(x_1)$

There is a separate loss function for G, designed to maximize the cross entropy.  Notice that there are TWO levels of training granularity in the system.

That of game moves in a two-player game
That of the training samples

These produce nested iteration with the outer iteration as follows.

Training of G proceeds using the loss function of G.
Mock input patterns are generated from G at its current state of training.
Training of D proceeds using the loss function of D.
Repeat if the cross entropy is not yet sufficiently maximized, D can still discriminate.

When D finally loses the game, we have achieved our goal.

G recovered the training data distribution
D has been reduced to ineffectiveness ("1/2 probability everywhere")

Why Concurrent Training is Necessary

If the two models were not trained in a back and forth manner to simulate concurrency, convergence in the adversarial plane (the outer iteration) would not occur on the unique solution claimed in the 2014 paper.

More Information

Beyond the question, the next item of interest in Sieta's paper is that, "Poor design of the generator's loss function," can lead to insufficient gradient values to guide descent and produce what is sometimes called saturation.  Saturation is simply the reduction of the feedback signal that guides descent in back-propagation to chaotic noise arising from floating point rounding.  The term comes from signal theory.

I suggest studying the 2014 paper by Goodfellow et alia (the seasoned researchers) to learn about GAN technology rather than the 2017 page.


Why is dot product attention faster than additive attention?

The additive attention method that the researchers are comparing to corresponds to a neural network with 3 layers (it is not actually straight addition). Computing this will involve one multiplication of the input vector by a matrix, then by another matrix, and then the computation of something like a softmax. Smart implementation of a dot-product will not break out the whole matrix multiplication algorithm for it, and it will basically be a tight, easily parallelized loop.


What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next).
So attention mask would be like this
[0, 1, 1, 1, 1]
[0, 0, 1, 1, 1]
[0, 0, 0, 1, 1]
[0, 0, 0, 0, 1]
[0, 0, 0, 0, 0]
For example: If you are translating English to Spanish
Input: How are you ?
Target: < start > Como estas ? < end >
Then decoder will predict something like this
< start > (it will be given to decoder as initial token)
< start > Como
< start > Como estas
< start > Como estas ?
< start > Como estas ? < end >
Now compare this step by step prediction sequences to attention mask given above, It would make sense now to you


What does the "number of channels" correspond to in U-Net?

In this example you have a gray scale image of size 572x572 and 1 (gray) channel. The first convolution operation consists of 64 filters of size 3x3 and 1 channel per filter. The channel of the filters always fits the channel size of the previous layer (here: the Input).
In the second convolution step of this explicit architecture, you again use 64 filters of size 3x3. In this case, each of these filters consists of 64 channels according to the previous output (64 feature maps/channels). The output of the second convolution consists of 64 feature maps according to the amount of 64 filters in the second convolution. This video from Andrew Ng visualizes it perfectly.


Can someone explain me what does this loss curve says?

Follow this answer from StackOverflow, I think your problem related to the second cases where your loss gets Nan value.
Maybe you should try to use the larger datatype (for example float 16 -> float 32)