How fast does Monte Carlo tree search converge?

Yes, Monte Carlo tree search (MCTS) has been proven to converge to optimal solutions, under assumptions of infinite memory and computation time. That is, at least for the case of perfect-information, deterministic games / MDPs.

Maybe some other problems were covered too by some proofs (I could intuitively imagine the proofs holding up for non-deterministic games as well, depending on implementation details)... but the classes of problems I mentioned above are what I'm sure about. The initial, classic proofs can be found in:

Bandit based Monte-Carlo Planning by Levente Kocsis, Csaba Szepesvári
Improved Monte-Carlo Search by Levente Kocsis, Csaba Szepesvári, Jan Willemson

Much more recently the paper On Reinforcement Learning Using Monte Carlo Tree Search with Supervised Learning: Non-Asymptotic Analysis appeared on arXiv, in which I saw it is mentioned that there may have been some flaws in those original papers, but they also seem to be able to fix it and add more theory for the more "modern" variants which combine (deep) learning approaches inside MCTS.

It should be noted that, as is typically the case, all those convergence proofs are for the case where you spend an infinite amount of time running your algorithm. In the case of MCTS, you can intuitively think of the proofs only starting to hold once your algorithm has manage to build up the complete search tree, and then on top of that had sufficient time to run through all the possible paths in the tree sufficiently often for the correct values to backpropagate. This is unlikely to be realistic for most interesting problems (and if it is feasible, a simpler breadth-first search algorithm may be a better choice).

  How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step is a bit slow)?

If you're thinking of a standard, tabular TD learning approach like Sarsa... such approaches actually turn out to be very closely related to MCTS. In terms of convergence speed, I'd say the important differences are:

MCTS focusses on "learning" for a single state, the root state; all efforts are put towards obtaining an accurate value estimate for that node (and its direct children), whereas typical TD implementations are about learning immediately for the complete state-space. I suppose the "focus" of MCTS could improve its convergence speed for that particular state...
but the fact that the search tree (which can be viewed as its "table" of $Q$-values as you'd see in Sarsa or $Q$-learning) only slowly grows can also be a disadvantage, in comparison to tabular TD learning approaches which start out with a complete table that covers the complete state space.

Note that papers such as the last one I linked above show how MCTS can also actually use Temporal Difference learning for its backing up of values through the tree... so looking at it from a "MCTS vs TD learning" angle doesn't really make too much sense when you consider that TD learning can be used inside MCTS.

  Is there a way to exploit the information gathered during the simulation phase to accelerate MCTS?

There are lots and lots of ideas like that tend to improve performance empirically. It will be difficult to say much about them in theory though. Some examples off the top of my head:

All Moves As First (AMAF)
Rapid Action Value Estimation (RAVE, also see GRAVE)
Move Average Sampling Technique (MAST)
N-Gram Selection Technique (NST)
Last-Good-Reply policy
...

Many of them can be found in this survey paper, but it is somewhat old now (from 2012), so it doesn't include all the latest stuff.


Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?

this is maybe not a paper but this article is dope.

I got 2 link in below to help you understand about policy gradient algorithms especially Reinforce Learning. Both article have a good explanation about that algorithms and not only an explanation, also contain a good example about that. They really put their effort into it so i'm sure if you've read it and really understand it, you can code from scratch. Hope this is help you.

If you think this answer is helping you, i will grateful if you make my answer as accepted. Just kidding, have a nice day....

(no i'm not kidding)

https://towardsdatascience.com/policy-gradients-in-a-nutshell-8b72f9743c5d by Sanyam Kapoor on Towardsdatascience.com

https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 by Jonathan Hui on Medium.com


Does DQN generalise to unseen states in the case of discrete state-spaces?

An environment is said to have a discrete state-space, when the number of all possible states of the environment is finite. For example, $3\times3$ Tic-tac-toe game has a discrete state-space, since there are 9 cells on the board and only so many different ways to arrange Os and Xs.
A state-space can be discrete regardless of whether integers or non-integers are used to describe it. For example, consider an environment where a state is represented with a single number. If the set of all possible states is $ \{0, 0.3, 0.5, 1\}$, your state-space is discrete, because there are only $4$ states. However, if all possible states is the set of real numbers from $0$ to $1$, than it's not discrete anymore - because there are infinitely many of them. State-space can still be discrete, even if possible states are represented with multiple numbers. For example, our environment could be $10\times10\times10$ cube, where the agent is only allowed to stand on integer coordinates. In this scenario, there are $1000$ different places where the agent can be and hence the state-space is discrete.
The Deep Q-Network can be designed to accept any type of input; just like a regular ANN, it's not restricted to only one integer. An input to DQN is the state of the environment, regardless of how it's represented. For the previous example, you can setup the DQN to have an input layer with $3$ neurons, each one accepting an integer that describes agent's position along the $x, y, z$ axes.
One downfall of Q-learning is that when the environment has a very large number of states and actions, representing each state-action pair becomes impractical in terms of memory. Think of chess, where there are so many different possible positions, and multiple available moves in each of them. Moreover, for an agent to learn properly, every state-action pair value must be visited (agent needs to determine Q-values), which can impractical in terms of training time.
DQN algorithm takes care of these problems: it only needs to store the neural network (also few other things if you use a variation of DQN), and it doesn't need to visit every state-action pair to learn. They way it learns is by adjusting the weights and biases in the network in order to approximate the optimal policy. Given the algorithm is implemented correctly, the agent should be able to pick up some useful patterns (or solve the environment).
I used this paper as a reference for one of my projects. It implements the DQN algorithm to learn to play Sungka (a game similar to Mancala), which has finite number of possible states and actions.


How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?

Let's add a step index to your expression
$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$
We can expand it one step further
$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$
And further
$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$
So, I guess, we can write a general formula for $m$ steps behind like:
$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary} $$
For $n-m$ large enough $(1-\tau)^{n-m}$ should be close to 0 and you should be able to approximately reconstruct your $Q_{target}^n$ using only the history of $Q_{primary}$es
Edit: I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.


How can I build an AI with NLP that read stories

This is quite an ambitious project, and IMHO well beyond the scope of what a single individual can do (within a reasonable time span) at present.

You need to first analyse the story text to identify the characters. This can already be quite a tricky task, as pronouns and other reference expressions are generally used to make a text less monotonous. If a character is referred to by name, say Jane, then you can assume that a follow-up the young woman will refer to her and not a male character mentioned in the same paragraph. But what about the young scientist? Such expressions can be very opaque, and you'd need a lot of world-knowledge to decode them correctly, as they can refer to any distinctive attribute of the character.

Identifying speech is a bit easier, unless you're talking about indirect speech. Jane was thinking aloud. She wasn't going to be able to do that. It was too hard. -- is that speech or not? Compare to Jane was thinking aloud: "I am not going to be able to do that. It is too hard.", which is the direct speech equivalent. Also, unless you're dealing with a play, most of the text will probably not be speech. For the audio version you will probably only want to deal with direct speech, which is usually (but not always) indicated by quote marks.

Analysing emotions seems to be comparatively easy if you have reached this stage, though if it is just based on keywords in the speech it probably won't be very accurate. If you can assign any descriptive statements to characters, that might be more successful, though by no means trivial.

Generating the text as audio should be straight forward. Most operating systems nowadays have speech synthesis integrated, and you can generally choose different voices, so if your text is marked up properly with which voice should speak which part it would be trivial.

To summarise: The NLP part is the hardest bit of it. As has been mentioned in the comments already, I don't think it's a problem that machine learning can help with, and I would stick to traditional methods of parsing the text into a structural representation and then applying rules to identify the bits you are interested in. The recognition of emotion might be a subtask that is suitable for ML, but in the past I have only applied pattern matching to similar tasks, so I can't really say much about that.

From my own experience in text analysis I would think that you might be able to get decent results with a few simple heuristics, but those will likely fail when it becomes a bit more complicated. A lot hinges on the type of story: children's fairy tales might be easier than War and Peace in that respect.


Can deep learning be used to help mathematical research?

Mathematical equations are generally expressed in a sequential form known as 'infix notation'.  It is characterised by the placement of operators between operands. To make the order of the operations in the Infix notation unambiguous, a lot of parenthesis are needed. Infix notation is more difficult to parse by computers than prefix notation (e.g. + 2 2) or postfix notation (e.g. 2 2 +).

There is a deep learning approach to symbolic mathematics recommended in the research paper by Guillaume Lample and François Charton. They have found an interesting approach to use deep neural networks for symbolic integration and differentiation equations. This paper proposes a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models.

Deep Learning for Symbolic Mathematics

This approach is essentially representing mathematical problems in prefix notation. First a symbolic syntax tree is constructed that captures the order and values of the operations in the expression. Second, the tree is traversed from top to bottom and from left to right. If the current node is a primitive value (a number), add it to the sequence string. If the current node is a binary operation, add the operations symbol to the sequence string. Then, add the representation of the left child node (could be recursive). Then, add the representation of the right child node. This procedure resulted in the following expression.

We can expect further more advances in this area with the emergence of better symbolic learning models leveraging attention based transformers and other neural symbolic learning models. Recent work by MIT, DeepMind and IBM has shown the power of combining connectionist techniques like deep neural networks with symbolic reasoning. Please find the details in the following article.

The Neuro-Symbolic Concept Learner


What should the action space for the card game Crib be?

I've actually implemented this game before using deep reinforcement learning. You are dealing with a dynamic action space here, where the action space may change at each time step of the game (or more generally the MDP). First, let's discuss the actual action spaces in each one of the two phases of Crib (or Cribbage) and formalize the question.

Phase 1: The Discard:In this phase, you are concurrently discarding  2 cards without respect to order. Therefore, you have a fixed discrete action space of size ${{6}\choose{2}} = 15$.

Phase 2: The Play: In this phase, you and your opponent are sequentially playing one of each of your remaining 4 cards (the original 6 minus the 2 discarded from phase 1). Therefore, you have a discrete action space of size $4! = 24$. Here's the catch - not every one of these actions is legal. The current state of the game restricts which cards you are allowed to play (the sum of all cards currently played must not be greater than 31). Since you do not know your opponent's cards and/or policy, you do not know which of these 24 actions are valid. To remedy this, the action space should dictate which of your remaining cards may be played at the current time step. Thus, you have a dynamic discrete action space of size 1, 2, 3, or 4 at each time step.

  How can I make an action space for these actions?

Since you didn't specify any implementation standard (e.g. OpenAI Gym), there are multiple paths to take, and they usually depend on your implementation of the state feature vector. In my own implementation, I tinkered with two possible state representations, which are fairly simple to describe.

Possibility 1: Separate State Representations for each Phase: In phase 1, you need to know the cards in your hand and the scores; i.e., the state feature vector could be encoded as a list of [card0, card1, card2, card3, card4, card5, your score, opponent score]. This state represents all information known about the game during phase 1 from a single player's viewpoint; after each time step, the current player may change, and the state must be updated according to the current player's viewpoint. Each card can be encoded as an integer from 1 to 52 (not starting from 0, as we will see in the next paragraph), and the score is an integer from 0 to 120 (tip: sort your cards for a reduced state space and faster convergence). Your action space can be the set of integers from 0 to 14 that maps to a 2-card combination in your hand. Alternatively, you could have a dynamic action space that sequentially asks for 1 of your 6 cards to discard and then for 1 of your remaining 5 cards to discard. The action space can be a subset of integers from 0 to 5 that maps to a single card in your hand. Be careful here - when choosing the second card to discard, your algorithm must know which card was discarded first. You can solve this by adding another component to your state vector that represents the first card that was discarded (set to 0 at the beginning of the phase), and therefore, make sure to update the state after the first discard.

In phase 2, you need to know the 4 cards in your hand, the cut card, the scores, and the cards currently played. Another helpful but unnecessary feature for learning is the current sum of played cards. A possible representation is a list of [card0, card1, card2, card3, card played0, card played 1, card played 2, …,  card played 7, your score, opponent score, cut card, current sum of played cards]. The values of played cards should be initialized to 0 at the beginning of the phase. The state can be updated so that any card in your hand that you have played is set to 0, and any card played by your opponent can be set to its negative value. This will correctly encode which cards have been played from your hand, which cards played are your from your opponent, and all other available information. Consequently, the action space is dynamic and is a subset of integers from 0 to 3 that maps to a single card in your hand.

Possibility 2: Identical State Representations for each Phase Another possibility is to have all of the above information for each phase encoded into a single state representation along with the phase number. State features that are unique to phase 2 that aren't relevant for phase 1 can be set to 0 during phase 1 and vice versa. With this representation, the state feature vector is of the same length at all time steps of the game, and the action space will change as described above. The important ideas for the encoding are exactly the same as above and will change based on your particular implementation, so I won't include the details here.

  How do I model this?

If you are going to implement something similar to Possibility 1, then you may need two agents that each learn a policy for a separate phase. For starters, you could use Q-learning or DQN and take the action with greatest q-value at each timestep, making sure that the chosen action is always a member of the current action space.

If you are going to implement something similar to Possibility 2, then you may only need a single agent that learns a policy for each phase, simply because the phase is a feature of the state. Essentially you are trading off a more complicated state representation for a simpler learning algorithm.

  I've read this post on using MultiDiscrete spaces, but I'm not sure how to define this space based on the previous chosen action. Is this even the right approach to be taking?

After reading the OpenAI Gym documentation, it looks like the MultiDiscrete space is a product of Discrete spaces. Therefore, it is a fixed action space and inherently not what you want here (a dynamic action space). I don't believe that OpenAI Gym standards will support dynamic action spaces natively. You would need to do some extra work such as providing a method that returns the current action space of the environment. Alternatively, if you want to follow the (state, reward, done, info) signal paradigm from OpenAI Gym, you may provide the current action space in the info dictionary. Finally, another idea is to allow the agent to always choose an action from a larger fixed action space (e.g. the set of integers from 0 to 3 for phase 2) and then penalize the agent through the reward signal whenever it chooses an action that is not a member of the current action space (e.g. if the chosen card was already played in phase 2). Afterward, you would return the current game state as the next state and make the agent try again.

My advice is to first determine the state representation, and the rest of your implementation should follow, using the ideas above.


What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).

Tabular Q-learning (TQL)
Deep Q-learning (DQL)
Deep Q-network (DQN)

Is it an RL algorithm?
Yes
Yes
No (unless you use DQN to refer to DQL, which is done often!)

Does it use neural networks?
No. It uses a table.
Yes
No. DQN is the neural network.

Is it a model?
No
No
Yes (but usually not in the RL sense)

Can it deal with continuous state spaces?
No (unless you discretize them)
Yes
Yes (in the sense that it can get real-valued inputs for the states)

Can it deal with continuous action spaces?
Yes (but maybe not a good idea)
Yes (but maybe not a good idea)
Yes (but only the sense that it can produce real-valued outputs for actions).

Does it converge?
Yes
Not necessarily
Not necessarily

Is it an online learning algorithm?
Yes
No, if you use experience replay
No, but it can be used in an online learning setting


What is 3D face recognition? and how we can check liveness of a face image?

1. What is typically meant by 3D-face recognition? We are usually extracting the face encoding from 2D-images, right?
Yes. The goal is to reconstruct the three-dimensional shape, as well as the texture of a face from a single or multiple images of that person.
In recent years, "the performance of 2D face recognition algorithms has significantly increased with the use of deep neural networks and the use of large-scale labeled training data." Deep 3D Face Ifentification with the latter still being a notable problem because there are so few of them and they are usually needed to better measure and hence train the deep networks. (The referenced paper gives you a nice overview.) Even with enough data, occlusions, lens distortions or possible variations of expressions, etc. make the problem tough. So many other tricks and augmentation techniques are needed.
2. How can 3D face recognition used for liveness detection?
3D face recognition is used to differentiate 2D spoofs from actual present people. Using multiple time-delayed images one could possibly track if the face moves like a living person as well, or if it is static or stretching weird like a mask, puppet, or moving display.
Nonetheless, often even advanced 2D image-based techniques fall flat in real-world applications because of the used cameras. They might not have enough resolution to recognize if the input is a video of a face on a high-resolution display as explained here.


Is there an efficient way to implement a random crossover of individuals stored in a matrix?

Firstly, before we commence I will recommend that you refer to a similar questions on the network https://stackoverflow.com/questions/828486/neural-net-optimize-w-genetic-algorithm

The majority of ML studies focus on gradient algorithms, usually a variation of back-propagation to obtain the weights of the model.

However since genetic algorithms are powerful searching algorithms they can be utilized to tune and optimize the structure and parameters of neural networks.

Genetic algorithms are stochastic search techniques that guide a population of solutions towards an optimum using the principles of evolution and natural genetics.

Genetic algorithms are especially capable of handling problems in which the objective function is discontinuous/non differentiable, nonconvex, multimodal or noisy. (Dharmistha D)

Algorithms, which combine genetic algorithms and error back-propagation, have been shown to exhibit better convergence properties than pure back-propagation.

To facilitate the crossover operation for the exchange of information between two parents we need to define four components within a genetic algorithm.

A way of coding solutions to the problem on chromosomes;
A fitness function which returns a value for each chromosome given to it;
A way of initializing the population of chromosomes;
Operators that may be applied to parents  to alter their genetic composition i.e. mutation.

In these hybrid systems weights in different layers of the network are optimized using a genetic algorithm. Experimental results demonstrate that the genetic algorithms can in some cases optimize the hyper-parameters of an ANN better than a hand tuned model.

Regarding an efficient way to implement a random crossover, here is something to get you started
(Code courtesy of Dan Golding 2013)

%Randomly choose 2 individuals

n = size(new_pop, 1);
l = size(new_pop, 2);

breeders = new_pop(randperm(n,2),:);

%Choose a crossover point

cp = randperm(l, 1);

%Crossover

b1 = [breeders(1, 1:cp), breeders(2, cp+1:end)];
b2 = [breeders(2, 1:cp), breeders(1, cp+1:end)];

For further reference I recommend that you kindly look at the below links

Matlab crossover genetic algorithm

https://stackoverflow.com/questions/17696323/matlab-crossover-genetic-algorithm

Single point ordered crossover in matlab

https://stackoverflow.com/questions/16302382/single-point-ordered-crossover-in-matlab

To optimize a neural network of multiple inputs using a genetic algorithm.

https://www.mathworks.com/matlabcentral/answers/180513-to-optimize-a-neural-network-of-multiple-inputs-using-a-genetic-algorithm?s_tid=gn_loc_drop

How can I use the Genetic Algorithm (GA) to train a Neural Network in Neural Network Toolbox

https://www.mathworks.com/matlabcentral/answers/100323-how-can-i-use-the-genetic-algorithm-ga-to-train-a-neural-network-in-neural-network-toolbox


For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic tasks. In the case of continuing tasks, $T = \infty$ and $\gamma = 1$ cannot be true at the same time, because the return may not be finite in that case (as I think you already understood).
Moreover, note that, in that specific example of the book, they assume that the agent ends up in an absorbing state, so this specific sum is finite, no matter whether $T$ is finite or $\infty$, given that, once you enter the absorbing state, you will always get a reward of $0$. Of course, if you discount those specific rewards, the sum will still be finite. However, in general, if you had a different MDP where the absorbing state is not reachable (i.e. the episode never ends), then the return could not be finite.


How does the DQN loss from td_targets against q_values make sense?

First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.


Streaming time series data to detect fraud?

Elasticsearch with X-pack is a tool that would come in handy. The suite has an visualization engine, Kibana and logstash that collect, parse and transform the data telemetry.

https://www.elastic.co/products/x-pack/machine-learning

Modeling can take a while to learn, luckily there are tools like Elasticsearch that can help.

Grafana also has tools but does not have a specific AI engine to it.But has a lot of integration with third party fraud detection tools.

https://grafana.com/

Sridhar


What is the best resources to learn Graph Convolutional Neural Networks?

I believe Graph Representation Learning book by William L. Hamilton is a great resource to start


What interests us in this problem are only the intervals for 1 person. Lets say that we want to train a neural network on recognizing the simple pattern in date differences. This would mean that we could train the neural network on series of purchase histories of multiple people. That means that one possible input is the previous intervals: in your case the previous 2 intervals (10 days, 15 days). That is a very small number and it will be hard to recognise a pattern for 3 consecutive purchases, but lets disregard that. After all, you can always experiment with different input sizes.

For training, you can take multiple customers, where of all of them you know when 4 purchases took place. You calculate the intervals, thus for each customer you get 3 intervals. As input you give the network 2, you let the network predict the 3rd one, and then you let the network learn off its mistake (you can do that because you know what the 3rd one was in reality, that's how you train the network).

Provided that the network isn't too complex or too simple (by that I mean that a hidden layer could be necessary, and you need to give it sufficient neurons) than the network might give a decent approximation. The only problem is, as Manuel Rodriguez said, that the network would indeed only learn to make predictions based on the corectness of its previous ones on a large group of people. Perfect predictions thus won't be possible, but you can minimize the error by making your dataset and input larger, so the network has more information to work on.


Is there bidirection sequence-to-sequence neural machine translation?

Bidirectional Mapping of Arbitrary Functions

Interchange between what properties are considered labels and what properties are considered features can support bidirectional map learning.

Unidirectional training represents the objective of functional approximation $f_p$ in terms of features $\mathcal{X}$ and labels $\mathcal{Y}$ through convergence of parameters $p$, ideally toward an accurate and reliable approximation.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$

Because only a small proportion of arbitrary functions are monotonic (have only one value in the domain for any given value in the range), the existence of an inversion of $f_p$ is unlikely. This is certainly true of natural language processing functions. This fact will be further illuminated below.

Bidirectional training of a non-monotonic translation function $f_p$ and a reverse translation $g_p$ can be represented as two adjacent training objectives.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$
 $$ g_q(\mathcal{Y}) \Rightarrow \mathcal{X} $$

Function $g_q$ is not a guaranteed inversion of $f_p$ and may rarely be. Parameters resulting from convergence $q$ may not be derivable from parameters resulting from convergence $p$ or vice versa either.

Temporal Bidirectionality to Minimize Round Trip Inaccuracy

For reasons further illuminated below, temporal bidirectionality may be important in some use cases. For this reason, use two B-LSTM networks.

There may be some efficiency gains by training both B-LSTM networks in the same training phase, especially if more than one host, GPU, parallel signal processor, or AI accelerator is available at training time. The same is true of test and validation.

The ontology of network types is as follows.

RNNs (recurrent artificial networks) are a stateful extension of MLPs (multi-layer perceptrons).
LSTM (long short term memory) networks are an extension of RNNs, and use gating within each cell to adaptively regulate retention of state and its dissipation.
B-LSTMs provide bidirectionality in the temporal direction, which allows more variety in what causality and reactance can be modeled.

Importance of Contemporary Concepts in Linguistics

Since the natural-language-processing tag was included with the posting of the question and the term, "Machine translation," was mentioned, it appears that bidirectional natural language translation using AI is the project goal. If that is the case, a contemporary linguist's understanding of words sequences in text and the sequence of phonetic elements in speech will help with developing an effective solution using AI. An engineer should be aware of the interplay between expectations placed on project results and technical feasibility at the current state of NLP technology.

Human translation experts rely on cognition to accurately represent text that may contain cultural references, analogies, colloquialisms, and other literary or oratory devices.
Because of rhyme, alliteration, cadence and other phonetic devices, prose and poetry may not have an equivalent in the target language.
Linguistic elements are not strictly words. A single word may have multiple linguistic elements because of tenses, plurality, gender, prefixes, suffixes, and other modifiers. For example, "Multicolored," is an adjective made up of three linguistic elements: {"multi-", "color", "-ed"}. A single idea is sometimes represented as multiple words, the individual meanings and juxtaposition of which does not imply the meaning of the word group. For example, "I don't wish to hitch my apple-cart to that wobbly wagon," has nothing to do with apples, carts, mechanical stability, or wagons.
The translation $\mathcal{A} \Rightarrow \mathcal{B} \Rightarrow \mathcal{C}$ may not lead to $\mathcal{A} = \mathcal{C}$.
When a series of linguistic elements are communicated, the semantics represented are not often a series. For instance, "Flying birds can also jump," is equivalent to, "If a bird is able to fly, it can jump."  This works in natural language because the associations and modifying aspects between linguistic elements is a semantic network that correlates with a cognitive ontology. The serialization of this network and its reconstitution in another mind cannot be formalized in a way that eliminates all ambiguity. Communication $\mathcal{K}_o \Rightarrow \mathcal{L} \Rightarrow \mathcal{K}_r$, where $\mathcal{K}$ are cognitions at originating intelligence $o$ and target intelligence $t$ respectively and $\mathcal{L}$ is a language expression in between, may not lead to $\mathcal{K}_t = \mathcal{K}_o$.
For the above reasons, formal grammar has been discarded as a set of principles that can facilitate NLP. Humans neither write nor speak in conformance with any formal grammar on a regular basis.

Expected Dominance of Recursive Network Technology

For this reason, LSTM networks may not be the direction of NLP technology development. Recursive network designs (often using RNN as an acronym but referring to a distinct subset of recurrent networks) can train semantic network models based on serial streams of linguistic elements. That appears to bear closest resemblance to language centers in the human mind, and may emerge as a dominant NLP component in emerging AI systems.


What is the main contribution of the paper Disentangling by Factorising?

One of the core contributions presented in the paper consists of understanding at a deeper level the objective function used in Beta VAE and improving it

More specifically, the authors started from Beta VAE OF

$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-\beta K L\left(q\left(z | x^{(i)}\right) \| p(z)\right)\right]$$

which consists of the classical reconstruction loss and the regularization term which steers the Latent Code PDF towards a target PDF.

Developing the second term

$$\mathbb{E}_{p_{\text{data}}(x)}[K L(q(z | x) \| p(z))]=I(x ; z)+K L(q(z) \| p(z))$$

they observed there are 2 terms which push in different directions:

$I(x,z)$ is the mutual information between the input data and the code, and we do not want to penalize this as in fact we want this to be as high as possible, so that the decode contains as much information as possible about the input
$KL(q(z) || p(z))$ is the KL divergence term which if penalized pushes the latent code PDF $q(z)$ towards the prior $p(z)$ which we choose as factorized
So penalizing the second term in Beta VAE OF means also penalizing the mutual information hence ultimately loosing reconstruction capability

The authors then propose a new OF which fixes this

$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-K L\left(q\left(z | x^{(i)}\right)|| p(z)\right)\right] -\gamma K L(q(z) \| \bar{q}(z))$$

In the rest of the paper they run experiments and show they have improved the disentanglement of the representation (according to the new metric they propose in the paper) while preserving reconstruction hence improving the reconstruction vs disentanglement tradeoff with respect to the Beta VAE paper


Why can a neural network use more than one activation function?

First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):
$$ \begin{array}{rcl}
y_1 & = & \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\
y_2 & = & \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\
y_3 & = & \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\
y_4 & = & \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\
\end{array}
$$
If that is how you understand it, then you've missed the point of "deep" neural networks. (Sequential) deep neural network are structured in several layers:
$$
 l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\
 l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\
 \ddots\\
 l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\
 y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\
$$
The activation function $\sigma^l$ of a single layer stays the same.

To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.

To follow this analogy - you are mistaken by thinking that we are simultaneously trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we first decompose over $\sin$ functions and then decompose over hermitian polynomials. This actually can makes sense from practical point of view.

Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.


Why am I getting spikes in the values of the loss function during training?

The spikes could be caused by many reasons: insufficient model capacity, incorrect label, buggy input parsing, ... Finding out the culprit requires some detective work. For instance, you could apply the learned model to the whole train set and manually examine the datapoints which result in the highest loss. Alternatively, you could compare the learning outcomes of different models (both weaker and stronger).


What are the advantages and disadvantages of using LISP for constraint satisfaction in 3D space

This is actually a question which will only receive opinion based answer's. A question you should ask yourself is if the constraint part is really that complex that it is worth to use a different programming language. Unity itself offers a C# API [1] and I would therefore stick with that.

[1] https://unity3d.com/programming-in-unity


Influence of location on a Neural Network trained for parking detection occupancy

My question is: is it possible to deploy a NN model trained with images all coming from a single parking area in another parking area? Is not such a model for parking detection occupancy supposed to generalize independently from the location where training images have been taken?

Generalisation in machine learning has limits, and for complex systems such as computer vision, it is very easy to exceed them. This is why the state-of-the-art systems use huge amounts of training data which is often augmented with automated variations, plus trained on for many days.

If you want some factor in your model to generalise well, you should:

a) Train with variations in that factor

b) Test with variations in that factor that have not already been seen in the training set

For the factor you wish to have the model generalise in, then it looks like (a) has not been done, and that you have just done (b) and found that the first attempt has failed an important test that you care about.

I suspect the trained model you have used will generalise well by time of day, type of vehicle and some other details. It does not appear to generalise to a new location because it was trained on only one location.

If you have limited training data, you may want to look at approaches to fine-tuning against more general image classifiers (such as the many ImageNet-trained classifiers, which are available to download). However, even then you should train with a few varieties of parking space, or at the very least with the spaces that you care about (if you do the latter then it probably won't generalise to other spaces, but you won't care).

It may also be worth seeing what you can do to transform the input that you are working with to have similar view over the new parking area that the training images have. Using a transform to align the outline of the new area with the old one, and possibly some colour correction could help a lot.


How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question.

The gradient is just the generalization of the derivative to multivariable functions. The gradient of a function at a certain point is a vector that points in the direction of the steepest increase of that function.

Usually, we take a derivative/gradient of some loss function $\mathcal{L}$ because we want to minimize that loss. So we update our parameters in the direction opposite the direction of the gradient.

$$\theta_{t+1} = \theta_{t} - \alpha\nabla_{\theta_t} \mathcal{L} \tag{1}$$

In policy gradient methods, we're not trying to minimize a loss function. Actually, we're trying to maximize some measure $J$ of the performance of our agent. So now we want to update parameters in the same direction as the gradient.

$$\theta_{t+1} = \theta_{t} + \alpha\nabla_{\theta_t} J \tag{2}$$

In the episodic case, $J$ is the value of the starting state. In the continuing case, $J$ is the average reward. It just so happens that a nice theorem called the Policy Gradient Theorem applies to both cases. This theorem states that

$$\begin{align}
\nabla_{\theta_t}J(\theta_t) &\propto \sum_s \mu(s)\sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\\
&=\mathbb{E}_\mu \left[ \sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\right].
\end{align}\tag{3}
$$

The rest of the derivation is in your question, so let's skip to the end.

$$\begin{align}
\theta_{t+1} &= \theta_{t} + \alpha G_t \frac{\nabla_{\theta_t}\pi(A_t|S_t,\theta_t)}{\pi(A_t|S_t,\theta_t)}\\
&= \theta_{t} + \alpha G_t \nabla_{\theta_t} \ln \pi(A_t|S_t,\theta_t)
\end{align}\tag{4}$$

Remember, $(4)$ says exactly the same thing as $(2)$, so REINFORCE just updates parameters in the direction that will most increase $J$. (Because we sample from an expectation in the derivation, the parameter step in REINFORCE is actually an unbiased estimate of the maximizing step.)

Alright, but how do we actually get this gradient? Well, you use the chain rule of derivatives (backpropagation). Practically, though, both Tensorflow and PyTorch can take all the derivatives for you.

Tensorflow, for example, has a minimize() method in its Optimizer class that takes a loss function as an input. Given a function of the parameters of the network, it will do the calculus for you to determine which way to update the parameters in order to minimize that function. But we don't want to minimize. We want to maximize! So just include a negative sign.

In our case, the function we want to minimize is
$$-G_t\ln \pi(A_t|S_t,\theta_t).$$

This corresponds to stochastic gradient descent ($G_t$ is not a function of $\theta_t$).

You might want to do minibatch gradient descent on each episode of experience in order to get a better (lower variance) estimate of $\nabla_{\theta_t} J$. If so, you would instead minimize
$$-\sum_t G_t\ln \pi(A_t|S_t,\theta_t),$$
where $\theta_t$ would be constant for different values of $t$ within the same episode. Technically, minibatch gradient descent updates parameters in the average estimated maximizing direction, but the scaling factor $1/N$ can be absorbed into the learning rate.


How can I generate What, why, who types of questions from "Gap-fill-in" type of questions?

This seems tricky. It seems that any "surface level" transformation wouldn't give adequate results and any working solution would need to properly capture the sentence structure, and generate a gramatically correct transformed sentence.

One possible option is to use a "traditional pipeline" - e.g. you run a NLP pipeline up to syntactic parsing, which for general domain english is quite accurate (you'd need some special handling for the gap "____" part though), then implement some heuristic rules to transform the syntax tree, and regenerate a sentence from the transformed tree. There are a lot of publications about similar transformations in machine translation domain, used as a way to preprocess data before running statistical machine translation for language pairs with very different word(or sentence part) ordering.

A second option that may work is to look into the field of controlled natural languages, or something like http://www.grammaticalframework.org/ that can be used as a toolkit to help generating new sentences.

Current fashion also suggests a very different option that might work - you could train a character-level recurrent neural network with an attention mechanism (look into recent neural machine translation publications for details) to do this transformation, but I'm not sure of how much training data it will need for decent accuracy.


How can I train an AI to find known and new patterns in a log file?

Correlation Between Entries

The first recommendation is to ensure that appropriate warning and informational entries in the log file are presented along with errors into the machine learning components of the solution.  All log entries are potentially useful input data if it is possible that there are correlations between informational messages, warnings, and errors.  Sometimes the correlation is strong and therefore critical to maximizing the learning rate.

System administrators often experience this as a series of warnings followed by an error caused by the condition indicated in the warnings.  The information in the warnings is more indicative of the root cause of failure than the error entry created as the system or a subsystem critically fails.

If one is building a system health dashboard for a piece of equipment or an array of machines that inter-operate, which appears to be the case in this question, the root cause of problems and some early warning capability is key information to display.

Furthermore, not all poor system health conditions end in failure.

The only log entries that should be eliminated by filtration prior to presentation to the learning mechanism are ones that are surely irrelevant and uncorrelated.  This may be the case when the log file is an aggregation of logging from several systems.  In such a case, entries for the independent system being analyzed should be extracted as an isolate from entries that could not possibly correlate to the phenomena being analyzed.

It is important to note that limiting analysis to one entry at a time vastly limits the usefulness of the dashboard.  The health of a system is not equal to the health indications of the most recent log entry.  It is not even the linear sum of the health indications of the most recent N entries.

System health has a very nonlinear and very temporally dependent relationships with many entries.  Patterns can emerge gradually over the course of days on many types of systems.  The base (or a base) neural net in the system must be trained to identify these nonlinear indications of health, impending dangers, and risk conditions if a highly useful dashboard is desired.  To display the likelihood of an impending failure or quality control issue, an entire time window of log entries of considerable length must enter this neural net.

Distinction Between Known and Unknown Patterns

Notice that the identification of known patterns is different in one important respect than the identification of new patterns.  The idiosyncrasies of the entry syntax of known errors has already been identified, considerably reducing the learning burden in input normalization stages of processing for those entries.  The syntactic idiosyncrasies of new error types must be discovered first.

The entries of a known type can also be separated from those that are unknown, enabling the use of known entry types as training data to help in the learning of new syntactic patterns.  The goal is to present syntactically normalized information to semantic analysis.

First Stage of Normalization Specific to Log Files

If the time stamp is always in the same place in entries, converting it to relative milliseconds and perhaps removing any 0x0d characters before 0x0a characters can be done before anything else as a first step in normalization.  Stack traces can also be folded up into tab delimited arrays of trace levels so that there is a one-to-one correspondence between log entries and log lines.

The syntactically normalized information arising out of both known and unknown entries of error and non-error type entries can then be presented to unsupervised nets for the naive identification of categories of a semantic structure.  We do not want to categorize numbers or text variables such as user names or part serial numbers.

If the syntactically normalized information is appropriately marked to indicate highly variable symbols such as counts, capacities, metrics, and time stamps, feature extraction may be applied to learn the expression patterns in a way that maintains the distinction between semantic structure and variables.  Maintaining that distinction permits the tracking of more continuous (less discrete) trends in system metrics.  Each entry may have zero or more such variables, whether known a priori or recently acquired through feature extraction.

Trends can be graphed against time or against the number of instances of a particular kind.  Such graphics can assist in the identification of mechanical fatigue, the approach of over capacity conditions, or other risks that escalate to a failure point.  Further neural nets can be trained to produce warning indicators when the trends indicate such conditions are impending.

Lazy Logging

All of this log analysis would be moot if software architects and technology officers stopped leaving the storage format of important system information to the varying convenient whims of software developers.  Log files are generally a mess, and the extraction of statistical information about patterns in them is one of the most common challenges in software quality control.  The likelihood that rigor will ever be universally applied to logging is small since none of the popular logging frameworks encourage rigor.  That is most likely why this question has been viewed frequently.

Requirements Section of This Specific Question

In the specific case presented in this question, requirement #1 indicates a preference to run the analysis in the browser, which is possible but not recommended.  Even though ECMA is a wonderful scripting language and the regular expression machinery that can be a help in learning parsers is built into ECMA (which complies with the other part of requirement #1, not requiring additional installations) un-compiled languages are not nearly as efficient as Java.  And even Java is not as efficient as C because of garbage collection and inefficiencies that occur by delegating the mapping of byte code to machine code to run time.

Many experimentation in machine learning employs Python, another wonderful language, but most of the work I've done in Python was then ported to computationally efficient C++ for nearly 1,000 to one gains in speed in many cases.  Even the C++ method lookup was a bottleneck, so the ports use very little inheritance, in ECMA style, but much faster.  In typical kernel code traditional, C structures and function pointer use eliminates vtable overhead.

The second requirement of modular handlers is reasonable and implies a triggered rule environment that many may be tempted to think is incompatible with NN architectures, but it is not.  Once pattern categories have been identified, looking for the most common ones first in further input data is already implied in the known/unknown distinction already embedded into the process above.  There is a challenge with this modular approach however.

Because system health is often indicated by trends and not single entries (as discussed above) and because system health is not a linear sum of the health value of individual entries, the modular approach to handling entries should not just be piped to the display without further analysis.  This is in fact where neural nets will provide the greatest functional gains in health monitoring.  The outputs of the modules must enter a neural net that can be trained to identify these nonlinear indications of health, impending dangers, and risk conditions.

Furthermore, the temporal aspect of pre-failure behavior implies that an entire time window of log entries of considerable length must enter this net.  This further implies the inappropriateness of ECMA or Python as a choice for the computationally intensive portion of the solution.  (Note that the trend in Python is to do what I do with C++:  Use object oriented design, encapsulation, and easy to follow design patterns for supervisory code and very computationally efficient kernel-like code for actual learning and other computationally intensive or data intensive functions.)

Picking Algorithms

It is not recommendable to pick algorithms in the initial stages of architecture (as was implied at the end of the question).  Architect the process first.  Determine learning components, the type of them needed, their goal state after training, where reinforcement can be used, and how the wellness/error signal will be generated to reinforce/correct desired network behavior.  Base these determinations not only on desired display content but on expected throughput, computing resource requirements, and minimal effective learning rate.  Algorithms, language, and capacity planning for the system can only be meaningfully selected after all of those things are at least roughly defined.

Similar Work in Production

Simple adaptive parsing is running in the lab here as a part of social networking automation, but only for limited sets of symbols and sequential patterns.  It does scale without reconfiguration to an arbitrarily large base linguistic units, prefixes, endings, and suffixes, limited only by our hardware capacities and throughput.  The existence of regular expression libraries was helpful to keep the design simple.  We use the PCRE version 8 series library fed by a ansiotropic form of DCNN for feature extraction from a window moving through the input text with a configurable windows size and move increment size.  Heuristics applied to input text statistics gathered in a first pass produce a set of hypothetical PCREs arranged in two layers.

Optimization occurs to apply higher probabilistic weights to the best PCREs in a chaotically perturbed text search.  It uses the same gradient descent convergence strategies used in NN back propagation in training.  It is a naive approach that does not make assumptions like the existence of back-traces, files, or errors.  It would adapt equally to Arabic messages and Spanish ones.

The output is an arbitrary directed graph in memory, which is similar to a dump of an object oriented database.

قنبلة -> dangereux -> 4anlyss
bomba -> dangereux
ambiguïté -> 4anlyss -> préemption -> قنبلة

Although a re-entrant algorithm for a reinforcement version is stubbed out and the wellness signal is already available, other work preempted furthering the adaptive parser or working toward the next step to use the work for natural language: Matching the directed graphs to persisted directed graph filters representing ideas, which would mimic the idea recollection aspect of language comprehension.

Final Comments

The system has components and process architecture similar to the log analysis problem and prove the concepts listed above.  Of course, the more disorganization in the way logging is done between developers of the system doing the logging, the more difficult it is for a human or artificial agent to disambiguate the entries.  Some system logging has been so poorly quality control for so long that the log is nearly useless.


How should I design this LSTM network to perform stock prediction?

I can't say for all cases, but I can certainly give you an example of how an LSTM could be used to predict stock prices.

An LSTM is temporal, meaning you can feed in one input, get an output and the network will remember this interaction (if it's important), which will effect the outcome of future predictions. As such, it is reasonable to feed in all the data for the day (Open, Close etc), obtain an output for the predicted stock price of the following day (or a prediction for all the inputs you feed in, the output can be as large or small as you want), then repeat for as long as you want, each time only feeding in the data for that particular day as the network will remember previous days.

In your example, 60 timesteps would mean doing a forward pass of the LSTM 60 times, for 60 days. Each of these forward passes will produce an output, which you can compare to the next actual stock price for verification, until you reach the current day, where the prediction is for the future.


What would be the reason behind using plots (such as box-plots or histograms) for ML development?

At a basic level, these kinds of low-dimensional plots where you look at one or two variables at a time can help to give you a sense of what types of relationships you might expect to see, such as linear, non-linear, or periodic relationships, which can steer you toward an appropriate family of models. You wouldn't want to use a linear model to predict data that has highly non-linear relationships, for example, nor would you want to use a monotonic non-linear model to predict a periodic function like a sine wave. Knowing about the general distribution of certain variables can also give you a sense of what statistical assumptions might or might not be met - if a model assumes that data is normally distributed, it might not be appropriate if your histograms suggest otherwise. Statistical analysis can help you determine if the underlying assumptions for certain model classes are or are not met.


Reasoning behind performance improvement with hopfield networks

Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue
On eq(6) of the relevant blog post, they identify the weight matrix of a discrete, binary Hopfield network as
$$ \boldsymbol{W} = \sum_i^N x_i x_i^T $$
with N raw stored entries, that are retrieved by iterating the initial guess $\xi$ with the following update rule
$$ \xi_{t+1} = \text{sgn}( \boldsymbol{W} \xi_{t} - b ) $$
Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):
$$ \xi_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi_{t} ) $$
Where $\boldsymbol{X} = (x_0, x_1, \dots , x_N ) $
The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix $\boldsymbol{W}$, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the $\boldsymbol{X}$ matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a "best fit" among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here


What techniques can be used to predict future attendance of students for a particular subject lecture session?

I suggest you should use AI Regression Model for future predictions for an attendance of students. Because of this technique or model design for future predictions.

Follow this to get more information about regression type and methodology


In computer vision, the adjectives dense and sparse are used in a variety of tasks (e.g. optimal flow), but they are commonly used in the context of the correspondence problem, which is the problem of finding a map (or correspondence) between pixels of two images (e.g. two successive frames of a video). In this context, these adjectives thus refer to the number of pixels of the image that are used to solve this specific task. A dense correspondence is thus a correspondence between two images using all (or, at least, many) pixels. In other words, a dense correspondence attempts to map all (or many) pixels of an image to all (or many) pixels of another image.

In the case of DensePose, the correspondences are between RGB images and surface-based representation of the human body (the figures of the paper illustrates this).


Does the bag-of-visual-words method improve the classification accuracy?

Bag-of-visual words (BOVW) was classicly used in computer vision before the introduction of neural networks or some more advanced classical techniques, such us VLAD or Fisher Vectors. In any case, it is a good technique to use, but it is not the state-of-the-art today, and I won't recommend you use it for a real-life project.


Attempting to solve a optical character recognition task using a feed-forward network

If you are serious about OCR systems and not just trying to learn CNN, start with a known good OSS product like tesseract. OCR some 1200 DPI grayscale documents, reverse engineer it to some degree so that you understand the design, join the development team, and pick up a ticket or two. Once you get into pull requests for a real system that is in constant use for a while, you will be an expert. Starting from scratch would make sense after you are an expert.


What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.

The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25.

The backpropigation algorithm, which is used to train a neural network, will propagate an error signal backwards. At each layer, the error signal will be multiplied by, among other things, the derivative of the activation function.

It is therefore the case that by the 4th layer your signal is at most $0.25^4 = \frac{1}{256}$ the size that it was at the start of the network. In fact, it is likely much smaller than this. With a smaller signal, your learning rates at the bottom of the nextwork will effectively be much smaller than the learning rates at the top, which will make it very difficult to pick a learning rate that is effective overall.

This problem is known as the vanishing gradient.

To fix this, if you want to use a deep architecture, consider using an activation function that does not suffer from a vanishing gradient. The Rectified Linear activation function, used in so-called "ReLU" units, is a non-linear activation that does not have a vanishing gradient. It is common to use ReLUs for the earlier layers in a network, and a sigmoid at the output layer, if you need outputs to be bounded between 0 and 1.


How to avoid being stuck local optima in q-learning and q-network

I found out the problem why the optimisation process got stuck and never moved closer to global optimum. It's because of the rate between 'explore' or 'exploit'.
Basically, in RL, agent 'explore's by doing a random action and to find new solutions, 'exploit's the existing so-called known max future rewards to do the max action.
Initially, I put the agent to explore when $random() < 1/(replay\_index+1)$, exploration rate reduces too quick (<10% after 10 iterations), and when the number of replays (number of times to play again from start) is not enough, the explore rate at the end of the loop is almost zero, and nothing new learnt.
The solution opted is allowing 'explore' and 'exploit' have the same rate (or lower exploration a bit is also ok), pseudo-code:
# Part 1 in a step: Choose action
if random() < 0.5: # 0.25 is also good, 25% for exploration
    action = random_action()
else:
    action = choose_best_known_action()

Explore rate can be reduced correctly this way:
if random() < 1-i/NUM_REPLAYS: # i is current train step index
    action = random_action()
else:
    ...

With the half-explore/half-exploit scheme above, the agent will learn to infinity, so, it is kinda sure that global optimum would be reached. When knowing from practice the number of iterations should be used, 'exploit' may be utilised more for faster convergence.
Note that the 'explore' and 'exploit' rates are put equal above, the but q-table or q-network is still better and better due to having another 'exploit'-kind when updating q-table or fitting q-network with Bellman equation, there's another 'exploit' here, the 'max' in Bellman equation:
Pseudo-code:
# Part 2 in a step: Update q-table or q-network
q[s][a] += learning_rate * (reward + max(q[sNext][aNext]) - q[s][a])

# Q-network
# target = r + max(...
```


I believe it would be more accurate to say that (some) search engines use AI.  Broadly saying "search engines are AI" is not really correct. At the core, most search engines are nothing more than an inverted text index using something like tf–idf scoring. That's a very mechanical/simple thing that nobody would really call AI.

But more sophisticated search engines may use AI or AI techniques to do things like semantic analysis - so they can actually "answer questions" instead of just looking up words in an index.


What are the latest 'hot' research topics for deep learning and AI?

The hot topics of today might be the cold, wet ashes of tomorrow.  For instance, the convergence speed of CNN and LSTM approaches, especially in combination, have diverted considerable attention away from basic RNN designs.

Similarly, the cold topics of today might be the burning embers of tomorrow. Of course, some of the cold topics will stay cold.  The sweet spot may be to identify those that are getting warmer and are likely to be sustainable building blocks future technology.

Residual Attention Networks

Residual attention networks, like LSTM networks, are an improvement over RNNs using a different approach.  Because attention networks are designed to conserve resources, they either converge faster or with less demand on hardware and network to support parallel execution.

Residual Attention Network for Image Classification, Fei Wang et al, 2017
Look Closer to See Better: Recurrent Attention Convolutional Neural
Network for Fine-grained Image Recognition, Jianlong Fu et all, 2017
VideoWhisper: Toward Discriminative Unsupervised Video Feature Learning With Attention-Based Recurrent Neural Networks, Na Zhao, 2017
Hierarchical Recurrent Attention Network for Response Generation
Chen Xing, Wei Wu et al, 2017

Automated Development of Non-Cartesian Models

Research into the automation of modelling is key to many AI applications.  Some of the algorithms under development do not simply extract tensors of features (arrays, matrices, cubes, and hyper-cubes), but develop graph models, directed or associative, with or without cycles permitted.

Hierarchical Self-organizing Maps System for Action Classification
Z Gharaee, P Gärdenfors, M Johnsson, ICAART, 2017
Runtime Modelling for User-Centric Smart Cyber-Physical-Human Applications, Lorena Castañeda Bueno, 2017
Probing the Topological Properties of Complex Networks Modeling Short Written Texts, Diego R. Amancio, 2015
Schema-summarization in Linked-Data-based feature selection for recommender systems, Azzurra Ragone et. al., 2017

Signal Topologies That Support Equilibria

Many ignore the importance of GANs, not because they can do interesting things with images but because of how they deviate from the simple topology of signal path where convergence on a trained set of parameters is achieved over a one-dimensional array of layers and blocks of layers.

The discriminative and generative components in GAN design are described in some detail in another AI Stack Exchange question on *Understanding the GAN Loss FUnction.  Although the generation of images from the GAN approach and its conceptual children demonstrate a new capability in the artificial network space, the breadth of this multi-network significance may not be immediately obvious.  It is not a stack in depth of layers, but a stack of two deep networks in a figure-eight topology, conceptually much like a Möbius strip.

This topology creates a balance between two networks, the generative (G) and the discriminative (D).  Its designer termed it an adversarial relationship because G and D play opposing roles.  However, their action in the system is actually collaborative, creating a balance that is much like a chemical equilibrium or symbiosis in biology, so that a specific objective is achieved.  This may reveal the most promising direction in AI today.

Designing signal topologies that support additional forms of collaboration and symbiosis between networks, where each network is a component that learns its roll in conjunction with other component networks, so that the aggregate system learns its function can synthesize forms of artificial intelligence that DNNs cannot.

Rules based systems and deep networks are one dimensional in terms of signal flow.  By themselves may never approximate the most notable features of the human brain.

Parallel Processing Using GPUs as DSPs

VLSI implementations of spiking networks is important, and there are now implementations such as https://github.com/Hananel-Hazan/bindsnet that leverage GPU hardware acceleration to investigate them without access to the VLSI chips being developed by large corporations.

Speech Recognition and Synthesis for End-to-End TTS

The recent emergence of excellence in synthesis using systems such as Google's WaveNet have opened the door to more accurate TTS (text to sound) applications, such that it is probably a good time to become an expert in voice recording for use in training example sets but a bad time to start a custom speech production house using live speakers.

Automated Vehicles

Automated vehicles of various types need experts in vehicle physics, automotive manufacture, aeronautics, and consumer products for a wide range of vehicle types with strong economic and safety incentives driving semi-automation and full automation.

Mars landers
Consumer drones
Industrial drones
Military drones
Passenger aircraft
Passenger automobiles
Limos
Trains
Wheel chairs
Delivery vehicles
Automated food distribution
Nuclear plant repair robots
Electrical distribution repair robots

Summary

It may be difficult to discover in advance what of hot technologies in AI will remain dominant in five years or which of the warming technologies will be blazing hot then, but the above are solid technologies showing significant early promise and for which there are high business, industrial, and consumer demands.


Can ConvNets be used for real-time object recognition from video feed?

We are getting there, with as usual some trade-off between quality and speed.
For example Lecture 8: Spatial Localization and Detection lecture shows some benchmarks (mAP = Mean Average Precision, higher is better; FPS = frame per second):


How can I systematically learn about the theory of neural networks?

Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.

Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.

Each optimisation problem has a landscape or a fitness landscape. Computer science has some nice toy landscapes.

There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's work.

Once you can identify the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research.

For example some researchers are working on how different evolutionary algorithms handle different landscapes, or neural networks handle different landscapes.

Start by exploring the types of machine learning problems.
Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces.


Ensure convergence of DDQN if true Q-values are very close

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.

The problem you describe is basically that you have situations where $Q^*(s, a_1) = Q^*(s, a_2) + \epsilon$ for some very small value $\epsilon$, where $a_1 \neq a_2$.

An important thing to note is that the "true" $Q^*$ values are directly determined by the reward function, which essentially encodes your objective. If your reward function is constructed in such a way that it results in the situation described above, that means the reward function is essentially saying: "it doesn't really matter much whether we pick action $a_1$ or $a_2$, there is some difference in returns but it's a tiny difference, we hardly care." The objective implied by the reward function in some sense contradicts what you describe in the question that you want. You say that you care (quite a lot) about a tiny difference in $Q^*$-values, but normally the "extent to which we care" is proportional to the difference in $Q^*$-values. So, if there is only a tiny difference in $Q^*$-values, we should really only care a tiny bit.

Now, with tabular RL algorithms (like standard $Q$-learning), we do expect to be able to eventually converge to the truly correct solutions, we expect to eventually be capable of correctly ranking different actions even when the differences in $Q^*$-values are small. When you enter function approximation (especially neural networks), this story changes. It's pretty much in the name already; function approximation, we cannot guarantee being able to do better than just approximating. So, 100% ensuring that you'll be able to learn the correct rankings is not going to be feasible. There may be some things that can help though:

If you are "in control" of your environment and its reward function (meaning, you implemented it yourself and/or can modify it if you like), modifying the reward function would probably be the single most effective solution to your problem. As I described above, a reward function that results in tiny differences in $Q^*$-values essentially encodes that you only care about those difference to a tiny (negligible) extent. If that's not the case, if you actually do care about those tiny differences... well, you can try changing the reward function to accentuate those tiny differences. For example, you could try simply multiplying all rewards by $100$, then all differences in returns are also multipled by $100$. Note that this does mean you get larger values everywhere (larger losses, larger gradients, etc.), so it would require changing hyperparameters (like learning rate) accordingly, and could destabilize things. But at least it will emphasize those small differences in $Q^*$-values. (probably won't work, see Neil's comment)
Increasing batch size. You already mentioned this yourself, but I'll also explain why it can help. A major issue in your problem is that you described the environment to be highly stochastic. This means you get a large amount of variance in your observed returns, and therefore also in your updates. If the true differences in $Q^*$-values are very small, these differences will get dominated and become completely invisible due to high variance in observations. You can smooth out this variance by increasing the batch size.
Decreasing learning rate. You also mentioned this in the question, and that it didn't really help. I suppose a high learning rate early on can help to learn more quickly, but decreasing it later on can help to take more "careful", small update steps that don't take "jumps" of a magnitude greater than the true difference in $Q^*$-values.
Distributional Reinforcement Learning: algorithms like $Q$-learning, DQN, Double DQN, etc., they all learn what we can call values, or scalars, or point estimates. Given a state-action pair $(s, a)$, such a $Q$-value represents the expected returns obtained by executing $a$ in $s$ and continuing with a greedy policy afterwards. In stochastic environments, it is possible that the true expected value $Q^*(s, a)$ never coincides with any conrete observation at the end of an actual trajectory. For example, suppose that in some state $s$, an action $a$ has a probability of $0.5$ of giving a reward of $1$ and instantly terminating afterwards, but also has a probability of $0.5$ of giving a reward of $0$ and instantly terminating. Then, we have $Q^*(s, a) = 0.5$, but we never actually observe a return of $0.5$; we always observe returns of either $0$ or $1$. When comparing the output of the learned $Q$-function (which should hopefully be around $0.5$ in such a situation) to any observed trajectories, we'll almost always have a non-zero error and take update steps that keep bouncing around the optimal solution, rather than converging to the optimal solution. Since you mentioned that you see $Q$-values fluctuating around, you may have observed this. Excessive fluctuations around $Q$-value estimates will likely make it difficult to get a consistent, stable ranking when the optimal $Q$-values that you are fluctuating around are themselves very close to each other. There is a class of algorithms (extensions of DQN mostly) referred to as distributional RL, which aim to address this issue by learning a full distribution of returns we expect to be capable of observing, rather than the single-point estimates that $Q$-values essentially are. I believe the first major paper on this topic was the following: https://arxiv.org/abs/1707.06887. There have been multiple follow-up papers too, the most recent of which I believe (but may be wrong here) to be this one: https://arxiv.org/abs/1806.06923. In general, I do know Marc Bellemare and Rémi Munos at least have done quite some work on that, so you could search for those names too.


What is the equation to update the weights in the perceptron algorithm?

I will tell you my knowledge, correct me if I am wrong.
Perceptron Learning Algorithm (PLA) is a simple method to solve the binary classification problem.
Define a function:
$$
f_w(x) = w^Tx + b
$$
where $x \in \mathbb{R}^n$ is an input vector that contains data points and $w$ is a vector with the same dimension as $x$ which present for the parameters of our model.
Call $y=label(x)=\{1,-1\}$ where $1$ and $-1$ are the label of each $x$ vector.
The PLA will predict a class like this:
$$
y=label(x)=sgn(f_w(x))=sgn(w^Tx+b)
$$
(The definition of sgn function can be found in this wiki)
We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.
Now, let define the cost function first, you can see that if a data point lies in the correct area, $y$ and $f(x)$ have the same sign, which means $y(w^Tx+b) > 0$ and otherwise. Similar to your example, I will define:
$$
g(x)=y(w^Tx+b)
$$
We ignore all the points in the safe zone ($g(x)>0$), only update to rotate or move the line to adapt with the misclassified points ($g(x)\le 0$), here, you can understand why we only update if $g(x)\le0$.
We need to define a cost function to minimize it, so our cost function will become:
$$
L(w)=\displaystyle\sum_{x_i\in U}(-y_i(w^Tx_i+b))
$$
where

$U$is the set of the misclassified points
$y_i$ is the label of data point $i$-th
$x_i$ is the $i$-th data vector
$w$ and $b$ is parameters of our model

For each data point, we have the derivative is
$$
\frac{\partial L}{\partial w} = -y_ix_i \\
\frac{\partial L}{\partial b} = -y_i
$$
Finally, update them by Stochastic gradient descent (SGD), we get:
$$
w = w - \frac{\partial L}{\partial w} = w + y_ix_i \\
b = b - \frac{\partial L}{\partial b} = b + y_i
$$
For your last question, notice that the weight and bias changed from $4$-th updated, so we have:
$$
y_5 = 1, x_5 = (4,8), w = (-2,-2), b = -2 \\
\Rightarrow g_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
$$


Why state-action value function as an expected value of the return and state value function, does not need to follow policy?

Let's first write the state-value function as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E,a_t \sim \pi}[r(s_t,a_t) + \gamma G_{t+1} | S_t = s, A_t = a]\; ;$$
where $r(s_t,a_t)$ is written to show that the reward gained at time $t+1$ is a function of the state and action tuple we have at time $t$ (note here that $G_{t+1}$ would be just the sum of future reward signals). This allows us to show that the expectation is taken under the joint distribution of $s,r\sim E$ where $E$ is the environment and actions are taken from our policy distribution.
As we have conditioned on knowing $a_t$ then the only random variable in the expectation that is dependent on our policy $\pi$ is $G_{t+1}$ because this is the  sum of future reward signals and so will depend on future state-action values. Thus, we can rewrite again as
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma \mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] | S_t = s, A_t = a]\;,$$
where the inner expectation (coupled with the fact its inside an expectation over the state and reward distributions) should look familiar to you as the state value function, i.e.
$$\mathbb{E}_{a_t\sim \pi}[ G_{t+1} |S_{t+1} = s'] = v_{\pi}(s')\;.$$
This leads us to get what you have
$$q_{\pi}(s,a) = \mathbb{E}_{s_{t},r_{t} \sim E}[r(s_t,a_t) + \gamma v_{\pi}(s') | S_t = s, A_t = a]\;,$$
where the only difference is that we have made clear what our expectation is taken with respect to.
The expectation is taken with respect to the conditional distribution $S_{t+1},R_{t+1}|A_t,S_t$, and we usually include the $\pi$ subscript to denote that they are also taking the expectation with respect to the policy, but here this does not effect the first term as we have conditioned on knowing $A_t$ and only applies to the future reward signals.


Appropriate convolutional neural network architecture when the input consists of two distinct signals

I don't know what you mean by desctized signals but if I understand your question correctly, separating two signal and passing them through same architecture of CNN (even with different parameters) is not a good idea. Because when they are together (as different channels) they will be treated differently by the CNN (each channel has its own parameters) and even this way the network is able to combine these two signals and get better results by information extracted from this combination.


Has any research been done on DNN Music?

The first thing is to define what is a «good» and a «bad» sound. This is an extremely tricky issue, since the networks need numeric inputs. And music is whole bunch of numbers.

I know from people doing research in identifying how similar two sounds are, and imitation, say: you hear a sound and try to make another that sounds like it. Like when you hum a song or similar. That is by no means easy. These guys are using something similar to feature extraction, with Fourier transforms and energy and such things. They feed the networks with the (selected) features and... Train.

Now, to return to your original question: *What do you present as target during training?* You can present different types of music as categories and classify (I couldn't help but think on this research with fish). Or you define categories of music you like and see if the network can classify them ;)

One basic decision here is how long you get a piece of sound. Since it is needed to analyse frequency, this is a key issue. Since you talked about DNN, I was wondering if you wanted to do it online, as a stream, in which case I don't have the slightest idea where to begin, other than do it after a little while.

Other idea: I remember a little sketch in this series about a researcher that makes use of the relations between peaks in the Fourier spectrum in order to differentiate noise from music.


Training network with 4 GPUs performance is not exactly 4 times over one GPU why?

Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A large batch size also would help as each GPU will have 1/4 the batch size. a batch size of 64-128 is good for 4 GPU. See the following example code fro CIFAR-10 for multi gpu code. It have dataloaders and dataparallel.

import os
import time
import datetime

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import torch.backends.cudnn as cudnn

import torchvision
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader

from model import pyramidnet
import argparse
from tensorboardX import SummaryWriter

parser = argparse.ArgumentParser(description='cifar10 classification models')
parser.add_argument('--lr', default=0.1, help='')
parser.add_argument('--resume', default=None, help='')
parser.add_argument('--batch_size', type=int, default=768, help='')
parser.add_argument('--num_worker', type=int, default=4, help='')
parser.add_argument("--gpu_devices", type=int, nargs='+', default=None, help="")
args = parser.parse_args()

gpu_devices = ','.join([str(id) for id in args.gpu_devices])
os.environ["CUDA_VISIBLE_DEVICES"] = gpu_devices

def main():
    best_acc = 0

    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    print('==> Preparing data..')
    transforms_train = transforms.Compose([
        transforms.RandomCrop(32, padding=4),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

    dataset_train = CIFAR10(root='../data', train=True, download=True,
                            transform=transforms_train)

    train_loader = DataLoader(dataset_train, batch_size=args.batch_size,
                              shuffle=True, num_workers=args.num_worker)

    # there are 10 classes so the dataset name is cifar-10
    classes = ('plane', 'car', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck')

    print('==> Making model..')

    net = pyramidnet()
    net = nn.DataParallel(net)
    net = net.to(device)
    num_params = sum(p.numel() for p in net.parameters() if p.requires_grad)
    print('The number of parameters of model is', num_params)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(net.parameters(), lr=args.lr)
    # optimizer = optim.SGD(net.parameters(), lr=args.lr,
    #                       momentum=0.9, weight_decay=1e-4)

    train(net, criterion, optimizer, train_loader, device)

def train(net, criterion, optimizer, train_loader, device):
    net.train()

    train_loss = 0
    correct = 0
    total = 0

    epoch_start = time.time()
    for batch_idx, (inputs, targets) in enumerate(train_loader):
        start = time.time()

        inputs = inputs.to(device)
        targets = targets.to(device)
        outputs = net(inputs)
        loss = criterion(outputs, targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

        acc = 100 * correct / total

        batch_time = time.time() - start

        if batch_idx % 20 == 0:
            print('Epoch: [{}/{}]| loss: {:.3f} | acc: {:.3f} | batch time: {:.3f}s '.format(
                batch_idx, len(train_loader), train_loss/(batch_idx+1), acc, batch_time))

    elapse_time = time.time() - epoch_start
    elapse_time = datetime.timedelta(seconds=elapse_time)
    print("Training time {}".format(elapse_time))

if __name__=='__main__':
    main()

Source: https://github.com/dnddnjs/pytorch-multigpu/blob/master/data_parallel/train.py

Hope I can help you and have a nice day!


Are there any known models/techniques to determine whether a person in a store is a customer or a store representative?

Access to the back of the store, basically if you can track them entering and leaving an employees only area.
But this requires a really good tracking algorithm and if you lose track you're going to have to fall back on some other properties.


Should one rescale (normalize) image before or after data augmentation?

So far, it seems this is more a software "integration" issue. One great tip from http://karpathy.github.io/2019/04/25/recipe/ is to visualize everything as often as you can during development. For data augmentation, try to visualize the image right before it enters your convnet.
What I found is a bug can happen if your particular image transform library try to detect if you have rescaled or not by just checking the data type. i.e. if it sees float, it will assume [0, 1] or if an integer, it will assume [0, 255].
This can end badly if you resize (without rescaling at the same time) using tf.image.resize(...). Resizing an image (e.g. to 224x224x3) has the "side effect" of converting the pixel value to a float, while still having an (approx.) range of 0.0-255.0. This is such that a downstream library will see a float and make the wrong assumption of [0, 1] range and totally mess up the transform. If you use any color related transforms, definitely be careful, they are usually dependent on pixel value range assumption.
Because of this, if you use tf.image, you may find it safer to do resize and rescale at the same time to ensure that float => [0, 1] such that your downstream color augmentation library may most likely work. Or you just have to manage the type carefully and do tf.cast(...) wherever necessary to satisfy any library assumption.
At the end, a human eye will still have to verify them.


Are decision tree learning algorithms deterministic? Given a fixed dataset, do they always produce a tree with the same structure?

Generally, yes. Most decision tree learners, like the common ID3 and C4.5/C5.0 algorithms, are deterministic. At each step, the learners consider all possible features that have not yet been used to split the data, and find the splits that maximize some function (e.g. information gain). There is no randomness (or pseudo-randomness) in this process.
The exceptions to this would be if you used randomness to break ties (rather than, say, using the index of each feature, as is common), but this would be an unusual modification.
What about the random forest?
Like the name suggests, random forests do make use of randomness, or at least, pseudo-randomness. If we're only concerned about whether or not the algorithm is deterministic in the usual sense of the word (at least, within computer science), the answer is no.
If you start the same random forest learning algorithm, with the same datasets, at two different times (or, using two different seeds for your pseudorandom number generator), you will get two different forests. This is because the algorithm selects random subsets of the features and/or datapoints to learn on, and, if different seeds are used, the subsets will be different each time.


Should I always start from the same start state in reinforcement learning?

It depends on the task the agent is trying to learn and of course on the environment constrains.

In an Atari game agents have a pre-fixed starting point because that's part of the games rules, so I would say that this is enough of a justification to make each simulation start from that starting point. Moreover, you have to pay attention to the kind of reward function you're using, for example (a really dumb one just to give a grasp of the concept) if you're rewarding the agent depending on how close it gets to the ending point rather than how far it goes from the starting point, the agent might end up having huge rewards only because it respawned close to end point, which would be an artefact and not a fair reward for a good action choice.
Other situations in which it does not really make sense to select a random starting point might be dialogue systems, in which you know that a conversation start with greetings, therefore it would not be logical to make an agent starting with a random question or sentence (the space in these case would be made of different dialogue acts).

There are anyway situations in which the environment allows a random selection of the start location of the agent. In this paper for example, an agent was trained to escape from a maze, and both, the exit point and the initial location of the agent where randomly selected at each iteration. This was partially due to research reasons, the authors were analysing the possibility to train policies more complex than a simple 'space memorisation' one but anyway, conceptually there's nothing wrong in this situation in selecting a random starting point. Other tasks could be following or escaping from a specific object, again, as long as the reward function is properly designed, even with a random starting location at the end the agent would anyway learn to move faster when close to the target (toward or in the opposite direction depending on the task). Actually, in this situation I think that a random initial location would have potential benefits, like avoiding the agents to learn biases due to external biases of the environment (like parts of space with less random obstacles than others).


How can active learning be used in the case of complex models that require a lot of data?

As I found this case backs to the sequence labeling. Sequence labeling has some classic solution such as conditional random fields (CRFs) and hidden Markov model (HMM). Also, have some solution in Active learning (AL) which use from algorihtms such as struct SVM ($\text{SVM}^{\text{strcut}}$) like this paper.

Also, some NLP solutions in active learning could help to solve these kind of problems such as this paper which is about active learning in named entity recognition (NER).

Besides, the combination of active learning with Deep networks such as CNN happens. For example, this paper explains more about the idea.


Deep Reinforcement Learning Atari: how does the agent understand motion?

In the article Playing Atari with Deep Reinforcement Learning, Mnih et al, 2013, which was a major outbreak in Deep Reinforcement learning (especially in Deep Q learning), they don't feed only the last image to the network. They stack the 4 last images :

For the experiments in this paper, the function φ from algorithm 1
applies this preprocessing to the last 4 frames of a history and stacks
them to produce the input to the Q-function

So they add the motion through sequentiality. From various articles and own coding experiences, this seems to me to be the main common approach. I don't know if other techniques have been implemented.
One thing we could imagine would be to compute the Cross-correlation between a previous frame and the last one, and then feed the cross correlation product to the net.
Another idea would be to train previously a CNN to extract motion features from a sequence of frames, and feed these extracted features to your net. This article (Performing Particle Image Velocimetry using Artificial Neural Networks: a proof-of-concept), Rabault et al, 2017 is an example of a CNN to extract motion features.


Create an AI to solve a puzzle (by deduction)

What you are trying to achieve sounds a lot like inductive logic programming:

  Given an encoding of the known background knowledge and a set of examples represented as a logical database of facts, an ILP system will derive a hypothesised logic program which entails all the positive and none of the negative examples.


Can we use MCTS without a generative model?

You either need a generative model or an emulator of the environment. In the later case you don't calculate your transitions and rewards using the model but feed your actions and states to the emulator and work with the results.

The emulator can be a black box as long as it returns the next state and the reward when provided with the current state and an action. You also need a way to identify all legal actions in a given state to build the tree.


In neural networks, what does the term depth generally mean?

Very good question, I have heard it refered to all the 3 ways you described. I am not sure if there is a purely objective answer so I will answer with what I understand by "depth" generally:

If speaking of model architecture: the number of layers of the model.
If speaking of CNN layers: the number of filters (not to be confused with the size of the kernel).
If speaking of RNN layers: the number of time steps (temporal "depth").

On the other hand I have never heard about depth regarding to FC layers. As I said the term "depth" is very subjective depending on the speakers. It should not be this way, and maybe theoretically is not, but I speak from what I heard in my experience.