What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the difference between prediction and control in Reinforcement Learning (RL).

Online vs Offline

These concepts are not specific to RL, many learning systems can be categorised as online or offline (or somewhere in-between).

Online learning algorithms work with data as it is made available. Strictly online algorithms improve incrementally from each piece of new data as it arrives, then discard that data and do not use it again. It is not a requirement, but it is commonly desirable for an online algorithm to forget older examples over time, so that it can adapt to non-stationary populations. Stochastic gradient descent with back-propagation - as used in neural networks - is an example.
Offline learning algorithms work with data in bulk, from a dataset. Strictly offline learning algorithms need to be re-run from scratch in order to learn from changed data. Support vector machines and random forests are strictly offline algorithms (although researchers have constructed online variants of them).

Of the two types, online algorithms are more general in that you can easily construct an offline algorithm from a strictly online one plus a stored dataset, but the opposite is not true for a strictly offline algorithm. However, this does not necessarily make them superior - often compromises are made in terms of sample efficiency, CPU cost or accuracy when using an online algorithm. Approaches such as mini-batches in neural network training can be viewed as attempts to find a middle ground between online and offline algorithms.

Experience replay, a common RL technique, used in Deep Q Networks amongst others, is another in-between approach. Although you could store all the experience necessary to fully train an agent in theory, typically you store a rolling history and sample from it. It's possible to argue semantics about this, but I view the approach as being a kind of "buffered online", as it requires low-level components that can work online (e.g. neural networks for DQN).

On-policy vs Off-Policy

These are more specific to control systems and RL. Despite the similarities in name between these concepts and online/offline, they refer to a different part of the problem.

On-policy algorithms work with a single policy, often symbolised as $\pi$, and require any observations (state, action, reward, next state) to have been generated using that policy.
Off-policy algorithms work with two policies (sometimes effectively more, though never more than two per step). These are a policy being learned, called the target policy (usually shown as $\pi$), and the policy being followed that generates the observations, called the behaviour policy (called various things in the literature - $\mu$, $\beta$, Sutton and Barto call it $b$ in the latest edition).

A very common scenario for off-policy learning is to learn about best guess at optimal policy from an exploring policy, but that is not the definition of off-policy.
The primary difference between observations generated by $b$ and the target policy $\pi$ is which actions are selected on each time step. There is also a secondary difference which can be important: The population distribution of both states and actions in the observations can be different between $b$ and $\pi$ - this can have an impact for function approximation, as cost functions (for e.g. NNs) are usually optimised over a population of data.

In both cases, there is no requirement for the observations to be processed strictly online or offline.

In contrast to the relationship between online and offline learning, off-policy is always a strict generalisation of on-policy. You can make any off-policy algorithm into an equivalent on-policy one by setting $\pi = b$. There is a sense in which you can do this by degrees, by making $b$ closer to $\pi$ (for instance, reducing $\epsilon$ in an $\epsilon$-greedy behaviour policy for $b$ where $\pi$ is the fully greedy policy). This can be desirable, as off-policy agents do still need to observe states and actions that occur under the target policy - if that happens rarely because of differences between $b$ and $\pi$, then learning about the target policy will happen slowly.

Prediction vs Control

This can get forgotten due to the focus on search for optimal policies.

The prediction problem in RL is to estimate the value of a particular state or state/action pair, given an environment and a policy.
The (optimal) control problem in RL is to find the best policy given an environment.

Solving the control problem when using value-based methods involves both estimating the value of being in a certain state (i.e. solving the prediction problem), and adjusting the policy to make higher value choices based on those estimates. This is called generalised policy iteration.

The main thing to note here is that the prediction problem is stationary (all long-term expected distributions are the same over time), whilst the control problem adds non-stationary target for the prediction component (the policy changes, so does the expected return, distribution of states etc)

Combinations That Work

Note this is not about choice of algorithms. The strongest driver for algorithm choice is on-policy (e.g. SARSA) vs off-policy (e.g. Q-learning). The same core learning algorithms can often be used online or offline, for prediction or for control.

Online, on-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs), and learns from observation data as it arrives. It should always act the same way (it may be observing some other control system with a fixed, and maybe unknown policy).
Online, off-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs) from the perspective of an arbitrary fixed target policy $\pi$ (which must be defined to the agent), and learns from observation data as it arrives. The observations can be from any behaviour policy $b$ - depending on the algorithm being used, it may be necessary to have $b$ defined as well as $\pi$.
Offline, on-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs), and is given a dataset of observations from the environment of an agent acting using some fixed policy.
Offline, off-policy prediction. A learning agent is set the task of evaluating certain states (or state/action pairs) from the perspective of an arbitrary fixed target policy $\pi$ (which must be defined to the agent), and is given a dataset of observations from the environment of an agent acting using some other policy $b$.
Online, on-policy control. A learning agent is set the task of behaving optimally in an environment, and learns from each observations as it arrives. It will adapt its own policy as it learns, making this a non-stationary problem, and also importantly making its own history of observations off-policy data.
Online, off-policy control. A learning agent is set the task of behaving optimally in an environment. It may behave and gain observations from a behaviour policy $b$, but learns a separate optimal target policy $\pi$. It is common to link $b$ and $\pi$ - e.g. for $\pi$ to be deterministic greedy policy with respect to estimated action values, and for $b$ to be $\epsilon$-greedy policy with respect to the same action values.
Offline, on-policy control. This not really possible in general, as an on-policy agent needs to be able to observe data about its current policy. As soon as it has learned a policy different to that in the stored dataset, then all the data becomes off-policy to it, and the agent has no valid source data. In some cases you might still be able to get something to work.
Offline, off-policy control. A learning agent is set the task of learning an optimal policy from a store dataset of observations. The observations can be from any behaviour policy $b$ - depending on the algorithm being used, it may be necessary to have $b$ defined as well as $\pi$.

As you can see above, only one combination offline, on-policy control, causes a clash.

There is a strong skew towards online learning in RL. Approaches that buffer some data, such as Monte Carlo control, experience replay or Dyna-Q do mix-in some of the traits of offline learning, but still require a constant supply of new observations plus forget older ones. Control algorithms imply non-stationary data, and these require online forgetting behaviour from the estimators - another online learning trait.

However, mixing in a little "offline" data in experience replay can cause some complications for on-policy control algorithms. The experience buffer can contain things that are technically off-policy to the latest iteration of the agent. How much of a problem this is in practice will vary.


What are some examples of LSTM architectures?

In the paper, LSTM: A Search Space Odyssey (2017), by Klaus Greff et al., eight LSTM variants on three representative tasks (speech recognition, handwriting recognition, and polyphonic music modeling) are compared.

The compared variants are

Vanilla LSTM features three gates (input, forget, output), block input, a single cell, an output activation function, and peephole connections (connections from the cell to the gates). The output of the block is recurrently connected back to the block input and all of the gates. The vanilla LSTM is trained using gradient descent and back-propagation through time (BPTT). The original LSTM (which is not the vanilla LSTM) does not contain, for example, the forget gate or the peephole connections (but the cell possesses a constant error carousel, a constant weight of $1$).
LSTM trained based on the decoupled extended Kalman filtering (DEKF-LSTM), which enables the LSTM to be trained on some pathological cases at the cost of high computational complexity.
Vanilla LSTM trained with an evolution-based method (called evolino), instead of BPTT.
LSTM block architectures evolved with a multi-objective evolutionary algorithm, so that to maximize fitness on context-sensitive grammar.
LSTM architectures for large scale acoustic modeling, which introduces a linear projection layer that projects the output of the LSTM layer down before recurrent and forward connections in order to reduce the number of parameters for LSTM networks with many blocks.
An LSTM architecture with a trainable scaling parameter for the slope of the gate activation functions, which improves the performance of LSTM on an offline handwriting recognition dataset.
Dynamic Cortex Memory, an LSTM composed of recurrent connections between the gates of a single block, but not between different blocks, which improves the convergence speed of LSTM.
Gated Recurrent Unit (GRU), which simplifies the architecture of the LSTM by combining the input and forget gate into an update gate.

There are other related neural network architectures, such as the neural Turing machine (NTM) or differentiable neural computer (DNC). In general, there are several architectures that use LSTM blocks, even though they are not just recurrent neural networks. Other examples are the neural programmer-interpreter (NPI) or the meta-controller.


How do we prove the n-step return error reduction property?

Let's start by looking at:

$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$

We can rewrite this by plugging in the definition of $G_{t:t+n}$:

\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert,
\end{aligned}

where $R_{t:t+n} \doteq R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n}$.

If you go all the way back to page 58 of the book, you can see the definition of $v_{\pi}(s)$:

\begin{aligned}
v_{\pi}(s) &\doteq \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} \mid S_t = s \right] + \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right]
\end{aligned}

Using this, we can continue rewriting where we left off above:

\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert \\
%
=& \gamma^n \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert
\end{aligned}

Because the absolute value function is convex, we can use Jensen's inequality to show that the absolute value of an expectation is less than or equal to the expectation of the corresponding absolute value:

$$\left| \mathbb{E} \left[ X \right] \right| \leq \mathbb{E} \left[ \left| X \right| \right].$$

This means that:

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
\end{aligned}

Now, the important trick here is to see that:

$$\max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \leq \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right]$$

I'm skipping the formal steps to show that this is the case to save space, but the intuition is that:

The left-hand side of this inequality involves finding an $S_t = s$ such that some function of $S_{t + n}$ is maximized, whereas the right-hand side involves finding an $S_t = s$ such that exactly the same function of $S_{t}$ is maximized.
In the left-hand side, selecting an $S_t = s$ implicitly induces a probability distribution over multiple possible states $S_{t + n}$, given by $S_t$, the environment's transition dynamics, and the policy $\pi$. Intuitively, this is more "restrictive" for the $\max$ operator, it does not have the "freedom" to directly select a single state $S_{t + n}$ such that the function of $S_{t + n}$ is maximized. The right-hand side is free to choose any single state $S_t = s$ such that $S_t$ in the right-hand side were equal to an "optimal" $S_{t+n}$ on the left-hand side, but it is also free to make even better choices which might never be uniquely reachable after $n$ steps on the left-hand side.

We can use this to rewrite the previous inequality we had (where we might be making the right-hand side a bit bigger than it was, but that's fine, it already was an upper bound anyway so that inequality will still hold):

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
&= \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - v_{\pi}(s) \Bigr\rvert \mid S_t = s \right].
\end{aligned}

After this rewriting we've got a hidden $\mathbb{E}_{\pi}$ "inside" another $\mathbb{E}_{\pi}$ (because the definition of $v_{\pi}(s)$ contains an $\mathbb{E}_{\pi}$), which I suppose is kind of ugly... but mathematically meaningless.

The maximum of a random variable is an upper bound on the expectation of that random variable, so we can get rid of the expectation in the right-hand side (again potentially increasing the right-hand side, which again is still fine since it's already an upper bound anyway):

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \Bigl\lvert V_{t + n - 1}(s) - v_{\pi}(s) \Bigr\rvert,
\end{aligned}

which we can finally rewrite to Equation (7.3) in the book by moving the subtraction of $v_{\pi}(s)$ outside of the expectation on the left-hand side of the inequality (which is fine because, as I already mentioned above, the definition of $v_{\pi}(s)$ itself contains another $\mathbb{E}_{\pi}$ anyway).


How to interpret this learning curve plot

Note the X index is training set size. For the first and second case, teh training set size starts at 0(or 1). The model will overfit certainly at that data size. When data size increases, the model overfits less and less and eventually the model have enough data samples that it won't overfit. The data size continue to increase and the model performance increase as well. To a certain point, the validation loss increase start to diminish and the model slightly overfits the samples.  For the third graph, it seems like originally teh loss is low and started to increase.
Hopes it help


How to improve the reward signal when the rewards are sparse?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of his PhD thesis, that potential-based reward shaping (PBRS) is the way to shape the natural/correct sparse reward function (RF) without changing the optimal policy, i.e. if you apply PBRS to your sparse RF, the optimal policy associated with the shaped, denser RF is equal to the optimal policy associated with the original unshaped and sparse RF. This means that PBRS creates an equivalence class of RFs associated with the same optimal policy, i.e. there are multiple RFs associated with the same optimal policy. So, PBRS is the first technique that you can use to deal with sparse reward functions.
To give you more details, let $R\left(s, a, s^{\prime}\right)$ be your original (possibly sparse) RF for your MDP $M$, then
$$R\left(s, a, s^{\prime}\right)+F\left(s, a, s^{\prime}\right)\tag{1}\label{1}$$
is the shaped, denser RF for a new MDP $M'$.
In PBRS, we then define $F$ (the shaping function) as follows
$$
F\left(s, a, s^{\prime}\right)=\gamma \Phi\left(s^{\prime}\right)-\Phi(s),
\tag{2}\label{2}
$$
where $\Phi: S \mapsto \mathbb{R}$ is a real-valued function that indicates the desirability of being in a specific state. So, if we define $F$ as defined in \ref{2}, we call it a potential-based shaping function.
The intuition of $F$ in \ref{2} is that it avoids the agent to "go in circles" to get more and more reward. To be more precise, let's say that there's a state $s^* = s_t$ that is desirable but not the goal state. If you shaped the reward function by adding a positive reward (e.g. 5) to the agent whenever it got to that state $s^*$, it could just go back and forth to that state in order to get the reward without ever reaching the goal state (i.e. reward hacking). That's, of course, not desirable. So, if your current state is $s^*$ and your previous state was $s_{t-1}$, and whenever you get to $s_{t-1}$ you just shaped the reward function by adding a zero reward, to avoid going back to $s_{t-1}$ (i.e. to avoid that the next state $s_{t+1} = s_{t-1}$, and then go back to $s^*$ again, $s_{t+2} = s^*$, i.e. going in circles), if you use \ref{2}, you will add to your original sparse reward function the following
$$
F\left(s, a, s^{\prime}\right)=\gamma 0 - 5 = -5,
\tag{3}\label{3}
$$
In other words, going back to $s_{t+1}$ (to later try to go back to $s^*$) is punished because this could again lead you to go back to $s^*$. So, if you define $F$ as in equation \ref{2}, you avoid reward hacking, which can arise if you shape your RF in an ad-hoc manner (i.e. "as it seems good to you").
One of the first papers that reported this "going in circles" behaviour was Learning to Drive a Bicycle using Reinforcement Learning and Shaping

We agree with Mataric [Mataric, 1994] that these heterogeneous reinforcement functions have to be designed with great care. In our first experiments we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Consequently the agent drove in circles with a radius of 20–50 meters around the starting point. Such behavior was actually rewarded by the reinforcement function

You probably also want to watch this video.
Another approach to solving the sparse rewards problem is to learn a reward function from an expert/optimal policy or from demonstrations (inverse reinforcement learning) or to completely avoid using the reward function and simply learn the policy directly from demonstrations (imitation learning).
Note that I'm not saying that any of these solutions fully solves all your problems. They have advantages (e.g. if done correctly, PBRS can speed the learning process) but also disadvantages (e.g., in the case of IRL, if your expert demonstrations are scarce, then your learned RF may also not be good), which I will not discuss further here (also because I don't currently have much practical experience with none of these techniques).


What is the difference between continuous domains and discrete combinatorial optimization?

A continous domain can be imagined as a space in which the axes of the coordinate systems are the parameters of the continous domain. If we take 2D Cartesian space as an example, it is a continous domain where there is an infininite amount of possibilities of placing an object in this space. Let's call the position of the object its state. Constraining the space to e.g. a sheet of paper does not change its continuity. There is still an infinte amount of posibilites of placing a dot (or any object) on that sheet of paper or with other words the state space of the dot is still infietly large.

Computational methods are much better in dealing with descretet spaces (mainly because the possiblities to take the next step is limited). In a discrete space the possibilities are limited. If we take the example above and discretize the sheet of paper, e.g. use a grid on the paper and say that we are not interested in the exact location of the point, just in which grid it is in we have succesfully defined a discrete space.

A different example for a problem in descrete domain can be navigation on a map. If we assume that the exact rajectory is not important, then we can make descrete decisions of turning left, right or moving forwards or backwards by one step. Even if add further possibilites (like moving up one step or down) until the number of possibilities in one state is not infinite the domain is descret.


Constraining the output value range of a CNN independent of the loss function

You can limit the absolute values of the outputs by "punishing" large values in the loss function. This is done by adding an extra term to the loss function.

For example, if your existing loss function is L(yhat, y) where yhat is the output and y the correct value, create a new loss function

L'(yhat, y) = L(yhat,y) + max(0, ||yhat||1 - k)

When the outputs are "small" the L1 norm of your output yhat is less than k and imposes no extra loss (keeping everything as before). However once the norm grows above k it starts to impose a loss that guides the training to reduce the absolute values of the outputs values.

You can also limit the values in the network itself by selecting a bounded activation function for the output, for example tanh or sigmoid. However, you need to pick an activation function that suits your problem.


What is the goal of a constraint solver?

You want Chapter 6 of Russell & Norvig's AI: A Modern Approach, for a starting place.

The goal of a constraint solver is to find an assignment of values to variables such that every variable is assigned a value, but none of a list of constraints are violated.

An example problem that a constraint solver can solve is the problem of assigning students to groups for a project. The variables are the names of the students. Each variable has a set of possible values it could take on that correspond to the names of the groups. The constraints prohibit assigning certain students to the same group (perhaps they hate each other), or require certain students to be assigned to the same group (perhaps they like each other). More complex constraints are also possible (e.g. at least 2 of these 5 students need to be in the same group).

Constraint propagation is the idea that several constraints might logically imply a stronger constraint. As a simple example, suppose that student $S$ must be assigned to group $1$. That's one constraint. Suppose further that student $T$ must be assigned to the same group as $S$. That's another constraint. Together, however, they imply that $T$ must be assigned to group $1$.

Most constraint solvers operate by iteratively trying to assign values to variables and then backtracking when a constraint is violated. When assignments are made, different constraint propagation methods make different decisions about how much computational effort to spend deriving the logical consequences of constraints, as opposed to simply trying more assignments of values to variables until the consequences become apparent (one of the existing constraints gets violated).

The simplest form of constraint propagation is called forward checking. After each speculative assignment of a value to a variable, forward checking looks at all the constraints the variable is involved in. It then derives the logical consequences of the assignments. For example, continuing with the student problem from above, if $S$ was assigned value $1$, then forward checking would notice the constraint $S=T$, and would, effectively, add the constraint $T=1$ (it actually does something a little different, but it has this effect). As another example, if the assignment $T=2$ were made, forward checking would notice that $S=T$, and would add the constraint $S=2$. It would also detect the contradiction that $S=1$ and $S=2$ are both constraints, and would declare the assignment $T=2$ invalid.

More advanced constraint propagation methods can reason about these kinds of logical consequences before assignment are made. For example, they could derive that $T=1$ before the program starts. Wikipedia has a good summary of these, starting with arc consistency.


Many of the best probabilistic models represent probability distributions only implicitly

The probabilistic models that represent distributions implicitly are, for example, the GANs. (Goodfellow is one of the authors of the original GAN model).
In the paper Variational Inference using Implicit Distributions (2017), the authors write

Implicit distributions are probability models whose probability density function may be intractable, but there is a way to

sample from them exactly and/or calculate and approximate expectations under them, and

calculate or estimate gradients of such expectations with respect to model parameters.

Implicit models have been successfully applied to generative modelling in generative adversarial networks (GANs)
A popular example of implicit models are stochastic generative networks: samples from a simple distribution - such as uniform or Gaussian - are transformed nonlinearly and non-invertably by a deep neural network

They also provide a table (table 1) that shows some probabilistic models that use an implicit distribution. Here $I$ denotes inference only. VI stands for variational inference.

See this blog post Variational Inference using Implicit Models, Part I: Bayesian Logistic Regression. See also the paper Learning in Implicit Generative Models (2017).


Looking to build, compile, and/or find dataset for serial-parallelized code examples

A generic way to start with under such circumstances is to try find an "oracle".

Serial to parallel converters exist for quite some times, and some are open source (e.g. PIPS). The idea is to get serial code from step (1), use the "oracle" to produce parallel code, and that's it: Each conversion makes en entry in the dataset.

Ensuring the quality of the dataset is critical here. A script generating the dataset shoud ensure (1) the serial code compiles and runs properlly, (2) the parallelized code compiles and runs properly too, (3) serial and parallel programs produce the same result, (4) some metrics state objectively how the parallel version does against the serial version, and (5) keep track of the actual hardware configuration.

Point (4) is critical to the quality of such a dataset. Parallel programs are not always faster than a serial version: A 1000-iteration loop dumped over 1000 workers on a 8-core CPU may not do so well compared to 8 workers---we need to check what the converter is doing. And point (5) ensures we know under which conditions the data is valid.

Using several "oracles" would be even better, to bring diversity, and to hopefully get the learning algorithm discover the best conversion tradeoffs---perhaps better than what the carefully hand-crafted converters are able to do on their own.


Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values?

The basic idea of gradient descent is:

Calculate the gradient of some score with respect to parameters that you can control

Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)

The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.
If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.
There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.
It is also possible to calculate gradients and then only update some of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.


Use mobile device camera for moving pattern recognition

Firstly, before we commence I will recommend that you refer to similar questions on the network i.e. https://stackoverflow.com/questions/6499880/ios-gesture-recognition-utilizing-accelerometer-and-gyroscope and https://stackoverflow.com/questions/6368618/store-orientation-to-an-array-and-compare
Your problem can be divided into three parts.
How to gather sensor data.
How to use the gathered data to train a model
How to use the trained model to make a prediction.
A modern smartphone contains around six sensors packed into one device. To implement your application I recommend that you use raw sensor data from either the gyroscope or the accelerometer.
On the android platform, you can access these sensors and acquire the raw sensor data by using the Android sensor framework. The Android sensor framework is part of the android.hardware package.

To capture the raw sensor data, take regular samples (> 20 Hz) and save the maximum values of x, y and z in an array each (to recognize in all 3 planes). If you want the gesture to be 5 seconds long, keep 100 samples (at 20 Hz). Then analyze if any of the three arrays has values which change sinusoidally. If it does, you've got a gesture.
You could store these values into an array, if the user is in 'record mode'. And when the user tries to replicate that movement, the model could predict the replicated movement array from the recorded one's. The thing is, how can you compare the arrays in a smart way? (Randy M 2011)

This leads us to the next step which is applying ML.
To train your model, you can choose to either train the model on the cloud or train it locally on the device. In most cases the problem of training a model on a mobile device is about computational limitations. Machine Learning algorithms running on a mobile device need to be carefully designed and implemented since most mobile devices have weak processors and small RAM's.
Its quite a challenge to squeeze a large neural network (NN) into the small RAM's that smartphones's have since NN's require that the model is fully loaded into memory. In many cases it is advisable to slim the model down and set some weights near zero to zero. (Chomba B 2017)
Incase you decide to utilize the cloud. Your mobile app is required to simply send an HTTPS request to a cloud web service along with the required raw sensor data, and within seconds the service replies with the prediction results. Though there are several cloud services i.e. Microsoft Azure Cognitive Services, Clarifai and Google Cloud Cognition that you can leverage to host the server side of your application, personally I recommend that you consider using reality.ai which is specifically an AI tool for engineers working with signals and sensors.
The next step will be to select an appropriate algorithm to be used in classifying the gesture. Here you can employ either Logistic Regression, Support Vector Machines, Random Forest or Extremely Randomized Trees depending on your app's use case. In order to train your model we then provide the learning algorithm with 'labeled examples'. The ML algorithm then extracts the features and constructs a mathematical model that can accurately describe gestures i.e. roll, pan from raw sensor data.


How should I deal with variable input sizes for a neural network classifier?

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each instance belonging to an interval with a function that you like. Most common choice is obviously averaging the values, but also summing then could be a choice or a moving average.

Don't think that in this way you will loose information, preprocessing is also called data cleaning for a reason, not aways raw data are better.

As a last note, I would suggest you to look into 'Machine learning for the quantified self', there is growing literature going on about using sensors to train predictive models about movements and body measures like heart-rate. You might find something about preprocessing for this specific sensor-applications.


How to work on different models for a given problem?

it seems to me that you are talking about hyperparameter tuning and effect of hyperparameters on the network in general. If you are working with tensorflow, I recommend you to look into tensorboard.

Hands-on TensorBoard can be a good starting point.


Tensorflow-gpu cannot use Nvidia GPU with CUDA

Seems like you have a CUDA version conflict. Remove the existing CUDA 10.2 and install CUDA 10.0 (going by your missing libraries, it requires a v10.0).

You can find the archived releases here:
https://developer.nvidia.com/cuda-toolkit-archive


Can prior knowledge be encoded in deep neural networks?

Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.


Which problems in AI are not machine learning?

This is a common question and it has already been answered in many different places:

What is the difference between artificial intelligence and machine learning?

Artificial Intelligence vs. Machine Learning vs. Deep Learning

  Artificial intelligence is a broader concept than machine learning, which addresses the use of computers to mimic the cognitive functions of humans. When machines carry out tasks based on algorithms in an "intelligent" manner, that is AI. Machine learning is a subset of AI and focuses on the ability of machines to receive a set of data and learn for themselves, changing algorithms as they learn more about the information they are processing.

More specific to your question:

AI without machine learning

  If you insert a small amount of knowledge into a machine, you can call it an engineering product. But if you instill a sufficiently large amount of knowledge such that the machine makes better decisions than a human, that can be could AI. For example, if you take hundreds of medical doctors and each spends hundreds of hours detailing correlations between symptoms and disease, if you then pack that knowledge into an easy-to-use machine, that's AI without machine-learning


When is bias values updated in back propagation?

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?

Yes.

  How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?

It differentiates the loss function (like MSE) with respect to the weights and biases, that is, it finds the partial derivative of the loss function with respect to each of the parameters. You can think of the partial derivative of the loss function with respect to one of the parameters of the model as representing the "contribution" of that parameter to the loss of the model.

The partial derivatives of the loss function with respect to each of the parameters (weights and biases) is collectively called the gradient, which is thus a vector of $N$ partial derivatives, where $N$ is the number of parameters of the model.

  Will the learning rate same for bias and weights?

It is usually the same, but, in theory, nobody prevents you from updating the biases using a different learning rate, so you could use a different learning rate to update the biases and weights.


Using 3D Points as Inputs to a Neural Net

A few thoughts :

10x3 matrix for each example is really a small amount of data. FCNN could do a good job on that.
As a result, I'm not sure CNN is appropriated. The smallest dimension is 3, so it'll force you to have really small kernels.
Have you thought about LSTM? since that your data is sequential, LSTM may be useful. However, I'm not sure they could be really effective on this little amount of data. but it could be nice to try.


No, there is no difference.

Of course, you are likely not able to extrapolate results obtained from synthetic data to expect identical or similar results in real life to unless you have very compelling reasons to do so.

Without a more specific question, I'm afraid a more specific answer is not possible.


What are advantages of using meta-heuristic algorithms on optimization problems?

Meta-heuristics are particularly suited for combinatorial optimization problems, given that, although they are not usually guaranteed to find the optimal global solution, they can often find a sufficiently good solution in a decent amount of time. So, they are an alternative to exhaustive search, which would take exponential time. For example, ant colony optimization algorithms have been used to approximately (or exactly, in the case of small or medium-size instances) solve the travelling salesman problem, whose decision version is an NP-complete problem (which means that, unless P=NP, there is no polynomial-time solution to solve it).

Meta-heuristics can also be easily applied to many problems, given that they are not problem-specific. For example, in the case of genetic algorithms, you just need to encode the possible solutions, but, in principle, you can apply genetic algorithms to a wide range of problems, although they may not always be the best solution to each of these problems. Moreover, as opposed to gradient-based optimisation algorithms, there's no need for the gradient of the objective function. For instance, in the case of genetic algorithms, you just need a way of evaluating the solutions (e.g. the fitness or the novelty).

Meta-heuristics often incorporate some form of randomness in order to escape from local minima. Ant-colony optimization algorithms or simulated annealing are two good examples of this approach.

If you are still interested in meta-heuristics, the book Clever Algorithms: Nature-Inspired Programming Recipes (by Jason Brownlee) is a very good resource for learning about them. There's also a Github repository with the implementation of the algorithms described in this book.


Decide Number of input Parameters and Output Parameters - ANN

This should be possible given the fact that ANNs have the ability to do the feature engineering and feature selection tasks by themselves.

This means that given a lesser number of input parameters, the model will be able to generate and select additional features by itself. You will obviously not be able to understand or model these features manually.

The only thing to keep in mind is that you will need a large dataset and a number of iterations before you are able to achieve a decent accuracy.

For example, there are networks that can generate image from classes.
Give this a read and here is an example where the output layer is larger than the input layer.


CNN High Variance across multiple trained models, what does it mean?

Firstly if you're doing spectrogram classification, you'd probably want to use a loss function like cross entropy. That would give you less "high-variance" like results when you're calculating your evaluation metrics.

About your ideas:

Dropout and early stopping would be enough
Data augmentation with speech is quite helpful, and there are a few libraries to help you with volume/speed/noise augmentation. Volume/speed augmentation is actually really important and you'd want to do it anyway for decent results in production. Speech processing is sometimes a bit tricky, in the sense that what we hear may not be what ends up in the spectrogram and parameters like noise, recording setup, encoding, accent end up causing a domain-mismatch (covariate shift)
In general, a smaller model (less parameters) helps with high variance. Start with small and simple models first, make sure eval metrics look ok, and then try larger models.
Random initialization doesn't really affect the results in a systematic way. You can indeed ensemble predictions of models trained with different initialization; that does help with variance but don't expect a lot of difference. The gain is not worth the computation cost for models applied in practice (ie not kaggle)

Other ideas:

Check that you shuffle data properly in train/test, and between folds
Do some error analysis on the mis-predicted test samples, do they have something in common. Check the label distribution in your dev/test set.
If the test set was captured in a process different from your dev set (total blind), don't expect a robust performance. In general, speech recognition is really data hungry with a decent "generic* model needing hundreds of hours of recording from different domains (tv, news, movies, telephone, lecture, echo, etc.).


What are some simple open problems in multi-agent RL that would be suited for a bachelor's thesis?

I have several undergraduates working on multiagent deep RL problems for their theses, but most of them have been working for 8-9 months. 2 might be a stretch.
Good multiagent deep RL problems for a bachelor's thesis might look something like:

Pick an older video game, which has been studied using Deep RL, but not in depth. Right now my students have been liking Nintendo 64 games.
Read the papers that study this game already.
Pick one of the described approaches and reproduce the paper's results in your own system.
Pick one of the parameters that the paper does not explore changing, and see what happens as you change it.

This probably does not lead to a publishable result, but it is real science and can make for a fine undergraduate thesis.
A slightly harder project, which may require more time, would be to examine the "future work" sections of these papers, and perform one of the experiments suggested there. These experiments often lead to small publishable results.


What are the minimum computing resources needed to train a machine learning algorithm?

It might be hard to implement deep reinforcement learning algorithms, especially considering your previous experience and the computing resources you have. They require almost the same (even more) GPU power. Deep reinforcement learning algorithms use deep neural networks for learning the optimal policy. Even if you are given appropriate resources, it would be tough to replicate the results of the paper, if you are a novice.


Optimal mixed strategy in two player zero sum games

I will answer the first question question based on information I have gathered so far. The probability of each action for the $\textbf{two player zero sum game}$ need not be the same for both players. It turns out that in the game of Morra, the probability vectors just turn out to be the same value.
In general to determine $\textbf{optimal mixed strategies}$ for two player two action game: Suppose we have 2 actions $a_1$ and $a_2$. In a mixed strategy, we let the probability of the row player taking action $a_1$ be $p$. The probability of the row player taking $a_2$ is then $1-p$. Now, we will try to find the probability $p$ such that the column player is indifferent to either action $a_1$ or $a_2$. (I.e the expected payoff for column player if row player plays $a_1$ with probability $p$ is the same for either actions).
Solving for $p$, we know that the column player can play any mixed strategy since any strategy played by the column player will yield the same payoff (since expected payoff is the same for either $a_1$ and $a_2$). Since column player can play any mixed strategy, we want to find a probability distribution over $a_1$ and $a_2$ for the column player such that the row player is indifferent to either action $a_1$ or $a_2$. We let the column player take action $a_1$ with probability $q$. It turns out we can repeat the same process for the column player and find the values of $p$. The game value for the row player can be computed from $x^TAy$ and this turns out to be the same value for the column player as well. (Only in zero sum games)
Doing this process yields a $\textbf{mixed strategy Nash Equilibrium}$, using such an approach does not always work however in the scenario when one action choice always dominates another action choice. (I.e in the case of the prisoner's dilemma of Alice and Bob, no matter what mixed strategy Bob tries to use, there will never be a mixed strategy such that Alice is indifferent to refusing or testifying). She will always testify.
I am not too sure about how to solve zero sum games involving more than 2 actions. I think linear programming would have to be used but i am not to familiar with how it can be applied.


Is Lisp still being used to tackle AI problems?

Overall, the answer is no, but the current paradigms owe a lot to LISP. The language most commonly used today is python.

Relevant answers:

Stack Overflow thread explaining why LISP was thought of as the AI language: Why is Lisp used for AI
Quora answer by Peter Norvig, who wrote a popular textbook on the subject and is currently Director of Research at Google: Is it true that Lisp is highly used programming language in AI?

LISP pioneered many important concepts in what we now call functional programming, with a key attraction being how close the programs were to math. Many of these features have since been incorporated into modern languages (see the Wikipedia page). LISP is very expressive: it has very little syntax (just lists and some elementary operations on them) but you can write short succinct programs that represent complex ideas. This amazes newcomers and has sold it as the language for AI. However, this is a property of programs in general. Short programs can represent complex concepts. And while you can write powerful code in LISP, any beginner will tell you that it is also very hard to read anyone else's LISP code or to debug your own LISP code. Initially, there were also performance considerations with functional programming and it fell out of favor to be replaced by low level imperative languages like C. (For example, functional programming requires that no object ever be changed ("mutated"), so every operation requires a new object to be created. Without good garbage collection, this can get unwieldy). Today, we've learned that a mix of functional and imperative programming is needed to write good code and modern languages like python, ruby and scala support both. At this point, and this is just my opinion, there is no reason to prefer LISP over python.

The paradigm for AI that currently receives the most attention is Machine Learning, where we learn from data, as opposed to previous approaches like Expert Systems (in the 80s) where experts wrote rules for the AI to follow. Python is currently the most widely used language for machine learning and has many libraries, e.g. Tensorflow and Pytorch, and an active community. To process the massive amounts of data, we need systems like Hadoop, Hive or Spark. Code for these is written in python, java or scala. Often, the core time-intensive subroutines are written in C.

The AI Winter of the 80s was not because we did not have the right language, but because we did not have the right algorithms, enough computational power and enough data. If you're trying to learn AI, spend your time studying algorithms and not languages.


Optimal RL function approximation for TicTacToe game

I think you can break this problem down into two parts to try and find the solution.

1. Can the neural network model the desired function?

Take the tabular function you have learned in the exact agent, and treat it as training data for the neural network model, using the same loss function and other hyperparameters as you intend to use when the NN is being used online inside the RL inner loops.

You can answer two related questions with this:

Does the loss reduce down to a low value after a suitable number of epochs? If so, then the NN has capacity to learn and can learn fast enough. If not, you need to look at the hyperparameters of the NN.
Does the trained NN play well against the exact agent? Ideally it plays the same, but it is possible that even though the loss is low, one or two key values in the function are compromised, meaning it still loses. I am not entirely sure what you would do in this case, but either try changing the hyperparameters to increase the capacity of the NN, or try augmenting the data so that there are more examples of the "difficult" action values to learn, to see if the issue is something that can be solved in learning.

Probably you will find your NN architecture is good, or only requires minor changes to become useful. The more likely issues are in next section.

2. Is the RL framework set up correctly for function approximation?

It is quite hard to get this right. Bootstrapping value based methods can easily become unstable if converted naively from tabular to function approximation approaches. Some variants are moderately stable - most stable would probably be Monte Carlo approaches.

If you don't want to use Monte Carlo control, then the answer here would be to take ideas from DQN used originally to play Atari games:

Don't learn online. Store transitions in an experience replay table - store $(s, a, r, s', done)$ tuples where $done$ is true if $s'$ is a terminal state - and sample a minibatch from it on every step. Reconstruct the bootstrap estimates of value functions to train from each time you sample, don't store and re-use the estimate from the time the action was taken.
Optionally use two value estimators - the current learning one, used to select plays and which is updated on each step, and a "target" one used to calculate TD targets. Update the target network by cloning the learning network every N steps (e.g. every 100 games).
To avoid figuring out hyperparameters for SARSA epsilon decay, I suggest use one-step Q learning. Also one issue you may be facing with SARSA combined epsilon decay is "catastrophic forgetting" where the agent gets good, starts to train itself only on play examples by good players, and forgets the values of states that it has not seen in the training data for many time steps. With Q learning you can avoid that by having a relatively high minimum epsilon e.g. 0.1.

In fact with TicTacToe learning through self-play, you can get away with $\epsilon = 1$ and Q learning should still work - i.e. it can learn optimal play by observing random self-play. This should apply equally tabular and function approximation agents. It doesn't scale to more complex games where random play would take too long to discover optimal strategies.


Can rewards be decomposed into components?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however  that many scientist will often follow the reward hypothesis (Sutton and Barto) which says that

All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

The argument for a scalar reward could be that even if you define your policy using some objective vector (as in MORL) - you will find a pareto bound of optimal policies, some of which favour one component of the objective over the other - leaving you (the scientist) responsible for making the ultimate decision concerning the objectives' tradeoff - thus eventually degenerating the reward objective into scalar.
In your example there might be two different "optimal" policies - one which results in a very high value of distance_x but relatively poor distance_y and a one that favours distance_y instead. It'll be up to you to find the sweet spot and collapse a reward function back to a scalar.


Why did Distributional Q Learning go out of popularity?

Actually, distributional RL is a well-studied field in deep RL. Generally speaking, distributional RL needs more computing sources (1.1X), because of the quantile head. We can also find new distributional RL literature in NeurIPS2020 :).


Why is dropout favoured compared to reducing the number of units in hidden layers?

Dropout only ignores a portion of units during a single training batch update.  Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.
Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for.  Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)


What is a "generalized" machine learning algorithm?

Actually you have used 2 terminologies there:

The first one is that  Machine Learning algorithm results can be "generalised". This refers to how well your trained Machine Learning model will perform on previously unseen data (test set or implemented on field). This is particularly not easy as data trends may change over time resulting in loss of accuracy. There are various methods to implement this like (having a cross validation set and a test set, which comes under the broad scheme of k-fold cross validation)
The second you mentioned is '"generalised" algorithms, routines or functions'. Most Machine Learning algorithms can be applied to a broad range of problems. For example the training of a NN is generally done by backprop which is universally applied to all NN's. Similarly, you can use CNN to find features of local interest (i.e. local dependencies) in anything that can be represented in a pictorial form (strings of DNA). Also combinations of CNN and RNN are being used to solve many problems. Thus, only a basic generalised algorithm is being applied to a lot of problems. NOTE: I have never seen any one use it in this context, but practically it happens.

Here are a few resources for general reading purposes (not mathematical):

Overfitting and Underfitting With Machine Learning Algorithms
Generalization and Overfitting


Is there anything that ensures that convolutional filters end up different from one another?

Yes, your thought experiment is correct, and the concept is known as breaking the symmetry. This is why biases can be initialized to $0$ (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has $n-1$ filters (or however many filters that are unique) instead of the full $n$ filters.
As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.
Consider the backprop equations as defined by this online book:

The gradient of the current layer's weights depends on

The future layers' weights, errors, and activation function's derivatives
The current layer's activation function's derivative, and
The previous layer's outputs.

Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by $j$ and $k$ in equation $BP4$). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.


Is it required that taking an action updates the state?

A very vague question. What's the objective?

Reinforcement Learning (RL) typically uses the Markov Decision Process framework, which is a sequential decision making framework. In this framework, actions influence the state transitions. In other words, RL deals with controlling (via actions) a Markov chain. The objective in RL is figure out how to take actions in an optimal (in some sense) way!

If, in the application you mentioned, the actions don't influence the state transitions and the objective is to predict states, RL is not required. It's just a regression/ time-series problem.


Solving a planning if finding the goal state is part of the problem

Your question is still nearly perfectly unclear, but let's make a few guesses in order to make some progress.

The biggest mystery are your time slices:

How many of them you need? With more granular time, you get more unknowns and more constraints and the complexity grows quickly.
When someone gets a car, they probably use it to travel somewhere, so it may be necessary to assign the driver to the car for multiple subsequent slices depending in the distance. That's something completely missing from your description (so I'll ignore it).

As already said, you need a cost function, i.e., something to minimize. Even before that, you need to define your variables. Let's say, you need binary variables

x[t, c, e]

such that x[t, c, e] = 1 when at time slice t the car c is assigned to the employee e (otherwise, it's zero). Now, we can specify some constraints

x[t, c, e] = 0 for each t, when the employee e can't drive the car c
for each t and c, the sum of x[t, c, e] over all e is at most one (as no employee can use two cars at the same time)
for each t and e, the sum of x[t, c, e] over all c is at most one (as no car can be use twice at the same time)

I may have missed some constraints, but that's no big deal; I hope you've got the idea.

Your cost function will be a sum of a few terms.

What you wrote about priorities is unclear to me, too, but you can define a weight w[c, e] (or maybe just w[c]), so let the sum of w[c, e] * x[t, c, e] over all t, c and e be the term describing these priorities.
Let u[e] be the cost of giving the employee e no car. The expression y[t, e] = 1 - sum over all c of x[t, c, e] equals to one, when the employee e has no car at time slice t. Now, the corresponding cost term can be expressed as the sum of u[e] * y[t, e] over all e and t.

Again, there may be more of this.

What I described so far is a integer linear programming formulation of your problem. Convert my text into ILP generating code, get an ILP solver, feed it and run it....

When you problem is small, you may even get the optimum. Otherwise, you'll get some solution (this is guaranteed) and a bound on the optimum, so you can know how far you might improve.

That's more than any heuristic solver (e.g., simulated annealing) can offer. OTOH arriving at a good solution may be faster using some heuristic solver. But that's all irrelevant until you get a clear problem definition.


When should I use simulated annealing as opposed to a genetic algorithm?

Simulated Annealing vs genetic algorithm?

Simulated annealing is a materials science analogy and involves the introduction of noise to avoid search failure due to local minima. See images below. To improve the odds of finding the global minimum rather than a sub-optimal local one, a stochastic element is introduced by simulating Brownian (thermal) motion. Variants of simulated annealing include injection of random numbers with various distributions and the averaging effect of mini-batching (dividing a batch into segments and performing network parameter adjustment after each).

Genetic algorithms are search methods based on principles of mutation, meiosis, symbiosis, test, elimination of inadequacy, and recursion. The advantages of such approaches is the simulation of sexual reproduction, where the possibility of dominant genetic features from two individuals producing a child individual containing the best of both exists. In a population of children, the probability of such a hybrid emerging is higher. Over several generations, even higher than that.

  What kind of problems does simulated annealing perform better than genetic algorithms if any?

This question can only be answered well if comparing original, pure versions of both, not the variants that have developed since their introduction.

Simulated annealing or other stochastic gradient descent methods usually work better with continuous function approximation requiring high accuracy, since pure genetic algorithms can only select one of two genes at any given position.

  From my experience, genetic algorithm seems to perform better than simulated annealing for most problems.

Those performance results would be of value to others and should be published as a paper, presented as an open source project, or published creative commons in the appropriate AI venue. At the very minimum, the results could be placed in the above question or an answer to it.

References

[1] Optimization by Simulated Annealing, Science S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi, Science New Series, Vol. 220, No. 4598, May 1983, pp. 671-680

[2] N. Metropolis, A. Rosenbluth, M. Rosenbluth., A. Teller. E. Teller, J. Chem. Phys. 21. 1087 110511

Optimised simulated annealing for Ising spin glasses, 2015, S.V. Isakov et. al.
A parallel simulated annealing method for the vehicle routing problem with simultaneous pickup–delivery and time windows, 2014, Chao Wang et. al.
Feature-based tuning of simulated annealing applied to the curriculum-based course timetabling problem, 2015, Ruggero Bellio et. al.

Images from Other Answers

If digital values are mere estimates, why not return to analog for AI?
Method to check goodness of combinatorial optimization algorithm implementation, plotted from https://github.com/facebookresearch/nevergrad/blob/master/nevergrad/functions/corefuncs.py


Is it possible to use self-supervised learning on different images for the pretext and downstream tasks?

No, the images do not need to be same. You can use different images for downstream task however you need to do some changes in model definition while loading model state_dict as CNN architecture used for pretext task 'relative patch location', assume Alexnet will expect 2 patches as input.


This looks like it could be a homework problem, so consider updating it with the homework tag if so.

The second model has the same precision, but worse recall, than model 1. Therefore we would rather have model 1 than model 2.

The third model has worse recall than model 1, and worse precision than model 1, therefore we would rather have model 1 than model 3.

Thus, model 1 is the best object detection model.


How to show temporal difference methods converge to MLE?

The convergence and optimality proofs of (linear) temporal-difference methods (under batch training, so not online learning) can be found in the paper Learning to predict by the methods of temporal differences (1988) by Richard Sutton, specifically section 4 (p. 23). In this paper, Sutton uses a different notation than the notation used in the famous book Reinforcement Learning: An Introduction (2nd ed.), by Sutton and Barto, so I suggest you get familiar with the notation before attempting to understand the theorem and the proof. For example, Sutton uses letters such as $i$ and $j$ to denote states (rather than $s$), $z$ to denote (scalar) outcomes and $x$ to denote (vector) observations (see section 3.2 for example of the usage of this notation).

In the paper The Convergence of TD($\lambda$) for General $\lambda$ (1992), Peter Dayan, apart from recapitulating the convergence proof provided by Sutton, he also shows the convergence properties of TD($\lambda$) and he extends Watkins' Q-learning convergence theorem, whose sketch is presented in his PhD thesis Learning from Delayed Rewards (1989), and defined in detail in Technical Note: Q-learning (1992), by Dayan and Watkins, to provide the first strongest guarantee or convergence proof for TD(0).

There is much more research work on the convergence properties of TD methods, such as Q-learning and SARSA. For example, in the paper On the Convergence of Stochastic Iterative Dynamic Programming Algorithms (1994), where Q-learning is presented as a stochastic form of dynamic programming methods, the authors provide a proof of convergence for Q-learning by making direct use of stochastic approximation theory. See also Convergence of Q-learning: a simple proof by Francisco S. Melo. In the paper Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms, the authors provide a proof of the convergence properties of on-line temporal difference methods (e.g. SARSA).


In Faster R-CNN, how can I get the predicted bounding box given the neural network's output?

If I understood well you have 2 questions.

How to get the bounding box given the network output
What Smooth L1 loss is

The answer to your first question lies in the equation (2) in the section 3.2.1 from the Faster R-CNN paper. As all anchor based object detector (Faster RCNN, YOLOv3, EfficientNets, FPN...) the regression output from the network are not the bounding box coordinates. The regression output predicts the shift of the predicted bounding box with respect to the selected anchor (all of these networks, use more than 1 anchor per location, check section 3.1.1 from the paper).
So basically what your network predict is $t_x, t_y, t_w, t_h$:

And the bounding box coordinates are $x, y, w, h$, and the anchor coordinates are $x_a, y_a, w_a, h_a$. So in order to compute $x, y, w, h$ from $t_x, t_y, t_w, t_h$, you just have to invert the equations above. However I think you could gain more intuition about it if you take your time and read the whole section 3.1 from the paper. I know sometimes is a pain, but you will grasp the high level concept.
With regard to your second question. Yes the loss is computed with the output from the network and the "coded" ground truth, meaning you compute the loss with the paramters $t$ (predicted) against $t^*$ (coded ground truth) instead for computing loss with the real coordinates of the bounding boxes (decoded output from the network). For the equation on Smooth L1 loss check this wonderful documentation.


How is the depth of filters of hidden layers determined?

Does the next convolutional filter have a depth of 40? So, would the filter dimensions be 3x3x40?

Yes. The depth of the next layer $l$ (which corresponds to the number of feature maps) will be 40. If you apply $8$ kernels with a $3\times 3$ window to $l$, then the number of features maps (or the depth) of layer $l+1$ will be $8$. Each of these $8$ kernels has an actual shape of $3 \times 3 \times 40$. Bear in mind that the details of the implementations may change across different libraries.
The following simple TensorFlow (version 2.1) and Keras program
import tensorflow as tf

def get_model(input_shape, num_classes=10):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(40, kernel_size=3))
    model.add(tf.keras.layers.Conv2D(8, kernel_size=3))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(num_classes))

    model.summary()

    return model

if __name__ == '__main__':
    input_shape = (28, 28, 1)  # MNIST digits have usually this shape.
    get_model(input_shape)

outputs the following
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 40)        400
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 24, 24, 8)         2888
_________________________________________________________________
flatten (Flatten)            (None, 4608)              0
_________________________________________________________________
dense (Dense)                (None, 10)                46090
=================================================================
Total params: 49,378
Trainable params: 49,378
Non-trainable params: 0
_________________________________________________________________

where conv2d has the output shape (None, 26, 26, 40) because there are 40 filters, each of which will have a $3\times 3 \times 40$ shape.
The documentation of the first argument (i.e. filters) of the Conv2D says

filters – Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

and the documentation of the kernel_size parameter states

kernel_size – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

It doesn't actually say anything about the depth of the kernels, but this is implied from the depth of the layers.
Note that the first layer has $(40*(3*3*1))+40 = 400$ parameters. Where do these numbers come from? Note also that the second Conv2D layer has $(8*(3*3*40))+8 = 2888$ parameters. Try to set the parameter use_bias of the first Conv2D layer to False and see the number of parameters again.
Finally, note that this reasoning applies to 2d convolutions. In the case of 3d convolutions, the depth of the kernels could be different than the depth of the input. Check this answer for more details about 3d convolutions.


Number of states in taxi environment (Dietterich 2000)

This is more of a combinatorics than AI question but regradless, the full state information for the environment is:

$(taxi \space position, passenger \space position, destination \space position)$

There are 25 possible taxi positions, 5 passenger positions and 4 destination positions making it $25 \cdot 5 \cdot 4 = 500$, so the paper is correct.

You are also correct but you divided 1 objective into 2 objectives and you have 2 separate policies, a pickup policy and a dropoff policy. So your state information would be for each policy:

$(taxi \space position, destination \space position)$

There are 25 possible taxi positions and 4 possible destination positions making it $25 \cdot 4 = 100$. You have 2 policies so you have $200$ states.

EDIT

Actually in the second case, I think you could get away with only 1 policy where you would simply change the destination position once you pick up the passenger so you dont need 2 separate policies and you would have only $100$ states


Using an 'operation ID' as a neural network input

Great question.

The operational ID is actually a category, or categorical input. Each category requires its own input signal into the neural network. Each signal is actually an activation which is multiplied by a weight and added to all the other weights and inputs. This means that if you use the same input signal for all your operational IDs then 101 is more important than 50. As you have mentioned these inputs are very different, and therefore must have a binary input for each operation id. If you add more IDs then you have to retrain the neural network.

You also mentioned you will be using scikit-learn, which has built in functions for category data. Have a look at this link on the one hot encoder.


REINFORCE algorithm for portfolio optimization - problem while training

One thing you could try to simplify the output logic would be to use a softmax output and then with your outputs set a var to = (max_output - min_output)/2 then treat that number as your long/short "threshold" and this ensures that your ouput always sums to 1 while still allowing the net to learn to output short signals. I would also check to make sure you have 1.0 bias nuerons since I imagine at times (especially first epoch) you are passing in zeros for the portfolio weights.


How do weak learners become strong in boosting?

As @desertnaut mentioned in the comment

No weak learner becomes strong; it is the ensemble of the weak learners that turns out to be strong

Boosting is an ensemble method that integrates multiple models(called as weak learners) to produce a supermodel (Strong learner).

Basically boosting is to train weak learners sequentially,  each trying to correct its predecessor. For boosting, we need to specify a weak model (e.g. regression, shallow decision trees, etc.), and then we try to improve each weak learner to learn something from the data.
AdaBoost is a boosting algorithm where a decision tree with a single split is used as a weak learner. Also, we have gradient boosting and XG boosting.


How should I define the action space for a card game like Magic: The Gathering?

There are several different ways you can model the state and action spaces in such sequential (extensive-form) environments/games. For environments with small action spaces or those typically introduced to beginning-RL students, the state space and action space remains constant along an agent's trajectory (termed normal form games when there are multiple agents). In sequential games which can be illustrated as trees, a "state" is analogous to "information set" which is defined as the sequence (tuple) of actions and observations since the beginning of the game's episode. Terminal states (leaf nodes) exist, and the action space $\mathcal{A}[x]$ at an information set $x$ can be defined as a union of action sequences that can be taken to each terminal state, not counting terminal states that cannot be reached from the current information set.
In the above examples, I discussed games such as the examples you stated where more than one agent can interact with the environment, but this is a generalization over RL and can be applied to when only one agent is maximizing its reward.


Which algorithm can be used for extracting text patterns in tabular data?

You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the different field names you have and the comment for rejection. Whichever field gets the highest score, you increase the rejection count for that field. In the end, you will have a tally of who is your biggest enemy.
You can create some rules which prevent injection of wrong data (like your password should be 7 characters long or something along these lines) or post-process your entries to match a uniform format.


Why is the hypothesis function $h_{\theta}(x)$ equivalent to $E[y | x; \theta]$ in generalised linear models?

In generalised Linear models, each output variable $y_i$ is modelled as a distribution from the exponential family, with the hypothesis function $h_{\theta}(x)$ for a given $\theta$ as the expected value of $y_i$ and maximum likelihood estimation is usually the method used to solve GLM's.


What is the difference between EER and DCF

Regarding DCF, Carbonell et al. (2007) state "Although it is an excellent evaluation tool, the DCF has the limitation that it has parameters that imply a particular application of the speaker detection technology."
For EER the authors make the following statement: "The EER is a concise summary of the discrimination capability of the detector. As such it is a very powerful indicator of the discrimination ability of the detector, across a wide range of applications. However, it does not measure calibration, the ability to set good decision thresholds."
References
Carbonell, J. G., Siekmann, J., Müller, C., van Leeuwen, D. A., & Brümmer, N. (2007). An Introduction to Application-Independent Evaluation of Speaker Recognition Systems. In Speaker Classification I (pp. 330–353).


How do I model the blocked N queens problem as a search problem?

In general, the process of modelling a problem as a search problem consists in creating a graph which contains nodes, which represent the possible states in your problem, and edges, which represent the relations between these states (that is, you will have an edge between nodes $A$ and $B$ if it is possible to go from state $A$ to state $B$, and vice-versa, in your problem).

In the case of the 8 (or, in general, N) queens problem (or blocked 8 queens problem), the states are all possible configurations of the board (that is, all possible combinations of the positions of all the 8 queens). In the graph that represents this problem, you will have a node for all possible valid configurations, where a valid configuration is one that is allowed by the rules of the game. You will also have an edge between configurations (or states) $A$ and $B$ if you can go from state $A$ to state $B$. For example, configurations where you have two queens on the same square or where you have a queen in a blocked area will not be represented in your graph as a node (or state).

In the case of the 8 queens problem, you actually have more than one solution, so you will have more than one goal state (or configuration of the board).

When using the BFS, the goal will then be to find a path from the initial configuration of the queens (that is, the initial state) to one of these 18 (or 12) goal states.

In the case of the 8 queens problem, there are 4,426,165,368 possible configurations (so you will likely not be able to draw this graph).


How can I incrementally train a Yolo model without catastrophic forgetting?

There's something called Elastic Weight Consolidation to prevent neural networks from forgetting previous tasks as they train on new tasks. It might be helpful for your case too.

The main idea is to quantify the importance of parameters for task $t$ and penalize the model in proportion when it changes its parameters as it trains to learn task $t+1$. As you can see, this incentivizes model to change parameters that are less important for task $t$ which prevents the model from forgetting it.


Proof of Maximization Bias in Q-learning?

Your bias formula is a bit incorrect. You should subtract a true value, not an estimator.

$\mathrm{b}(\hat{\theta}) \stackrel{\text { def }}{=} \mathbb{E}[\hat{\theta}]-\theta$

$\mathrm{b}\left(\max _{a} Q\right)=\mathbb{E}\left[\max _{a} Q\right]-\max _{a} q=\mathbb{E}\left[\max _{a} Q\right]-\max _{a} \mathbb{E}[Q]$

$\mathbb{E}\left[\max _{a} Q\right] \geq \max _{a} \mathbb{E}[Q] \Rightarrow \mathrm{b}\left(\max _{a} Q\right) \geq 0$


Does feature scaling have any benefits if all features are on the same scale?

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.


In AlphaZero, which policy is saved in the dataset, and how is the move chosen?

However, I’m not sure which policy is saved

The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).

Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?

Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.


For forecasting and trading control, given limited data, what AI approaches are well matched?

This is an interesting problem, the answer of which is highly coveted for obvious reasons.  The production of an answer in this public space is appropriate, provided one believes in a more level distribution of assets across those alive and is comfortable with rewarding technical people for the abilities they developed while others might have been engaging in less technical pursuits.  I only ask that those who may successfully employ it consider using at least some of any windfall generated in a philanthropic manner, to benefit the less fortunate.

Here we go.

Abstract Algebra

One of the fundamental principles presented in Morgenstern and von Neumann's Theory of Games and Economic Behavior is the idea that the value of a monetary unit may vary based on actions of those that control the monetary system, but the underlying value itself has abstract algebraic properties favorable to prediction.  These properties, the book proposes in early chapters, permit additive modelling.

Explained another way, imagine a pile of paper bills of a particular denomination.  The selection of any one of the bills from anywhere in the deque is identical in value to the selector as any other selection of one bill.  The selection of which particular bill is also not a factor in the value remaining in the deque after the selection.

This is also true of shares, which is why serial numbers are not a trading parameter at the investor-trade interface.

For more theory behind this, the interested parties can study group theory and rings.

Critique of M&VN

This convenient additive view of value beneath monetary systems has been critiqued.  The value of assets, products, and services may be demonstrated to vary with life expectancy and philosophic divergence.  While some people fall into an investment mode, others may be divesting.

Fortunately, similar to Isaac Asimov's fictional presentation of psycho-history, the predictive mechanisms that may be developed are, in fact, not fettered by this investment versus divestment mode variance.  The volume of trading and number of traders typically hide the variance through cancellations across either side of mean behavior.

Those with considerable assets can exploit this caveat and a large number of other caveats in the additive nature, but most traders cannot.

Patterns in Stock Values

The phrase that may have started a number of investment firms down the path toward the use of machine learning in high speed training, leading to a conspicuous growth in the number of billionaires in the world, is a simple phrase from Darren Aronofsky's Pi, a 1998 movie in black and white.  The protagonist, Maximillian Cohen, looks at the scrolling LED version of a ticker tape and says, "I know these numbers," and then begins to verbally state each price just before it appears.

The patterns in stock prices have intrigued those who watch trends since the emergence of the New York Stock Exchange.  But they are not cyclic, in spite of the naive belief that they run in cycles.  They only appear to run in cycles, just as one might find a pattern they think they understand in a graphic representation of the Mendelbrot Set, but when the renderer of the set zooms in or out, the greater detail or the generality respectively reveal that the understanding was an illusion.

Attempts to use Fourier Analysis as part of a predictive strategy for anything but extremely narrowly timed speculation have been of limited value.  The reliability of the prediction of advantageous trading choice, expressed as a probability $P(C)$, dwindles to pure entropy (zero bits of actual predictive information) as the clock time of the prediction increases relative to that of the last time in the complete vector of state historic states $\vec{S}$ used for prediction.

$$\lim_{(t - t_\emptyset) \rightarrow \inf} P(A|\vec{S}) \rightarrow 0$$

In plain terms, the risk of trading based on a predictive model from past price trend increases with the staleness of trend information.

The increase in entropy is likely to be a feature of the prediction of any system as complex as the stock market even as prediction tools mature.  It is a consequence of initial input sensitivity combined with measurement inaccuracy and the inevitable unsatisfactory distributed model of decisioning.  These constraints may not be inescapable, but they are certainly resistant to brute force attacks, such as adding artificial network layers.

There are tools that can be predictive in ways better than trying to identify a spectrum using Fourier Analysis.  But let's first consider a spectrum, which is how music listeners identify the pitch, loudness, timbre, and envelope of notes.
Unlike the playing and listening to music, the distribution of pitches in the spectrum from moment to moment are not purely a representation of the spectral distribution in which value fluctuations reside.

The spectral analysis of stock prices reflects artificial bounds.  The spectrum is bounded by the market sampling rate and Nyquest's criteria on the high frequency side.  It is bound by the accumulation of additive effects that tends price fluctuation toward a Gaussian distribution on the low frequency end of the spectrum.

Three-fold Challenge of Freshness, Accuracy, and Complexity

Some may note that the intention of the question, to control (with or without human moderation) a stock portfolio to maximize profit is a planning problem that appears to have the Markov Property, provided the trader is not wealthy or trading in penny stocks, since the impact of previous trades seems unlikely to have significantly impacted market trends.  As we shall see, this appearance may not be objectively true.

Initial input sensitivity is a nearly universal quality of complex systems.  In the case of the stock market, the complexity is high.  Consider a cursory examination of the magnitude of market complexity, understanding that a single price is affected by neighboring prices in that sector and business and in the major market indicators that people watch.  A stock is almost completely isolated in terms of trading actions but inherently tied to all traders and other stocks through an interconnected network of IT pipelines, trading interfaces, news events, and brains, each containing a few billion neurons with complexities much greater than artificial cells.

A popular name for initial input sensitivity is the Buterfly Effect, which, properly explained is this.

  Whether a butterfly flaps its wings in Brazil may determine the path of a major storm in Peking a month later.

Sometimes the initial word, "Whether," is not included, which renders the explanation incorrect.  It is a statement of the effect of whether or not the perturbation occurs.  Once cannot say that the air movement from the wing flap causes the storm but rather the presence of a small perturbation may be grossly amplified over time in a complex system.  Or it may not.

Studies of chaotic systems have proven this potential of amplified effect as a characteristic of many if not most complex systems of differential equations with sufficient curvature in their behavioral surface to cause a system phenomena that could most descriptively be called, "Radically increasing bifurcations."

For those seeking greater depth of understanding, the work of Fourier, Nyquist, Cantor; G. Julia and P. Fatou; and Aleksandr Lyapunov may be of interest.

Because of initial input sensitivity, the Markov Property cannot be guaranteed for stock trading, even at low levels of investment.  Even though, technically, the Markov Property is not guaranteed, its assumption at low trading levels and in the leveraging of fresh data for short term gains, it may lead to reasonable approximations with sufficient reliability to win at the negative sum game of trading.

Trading as a Below Zero Sum Game

In a zero sum game, those trading with stale information and trying to apply ineffective predictive strategies will, over any longer period lose. Their monetary assets will pass to those with fresher information using effective predictive strategies.  With trading fees, the condition is more challenging, because the sum of all interactions is below zero by the sum of the fees.

Gaining Advantage

To gain advantage in any practical way above those who may have already employed machine learning in their trading operations or hired a funds manager that already did, there will have to be some clever application of more advanced principles in a way that drives effectiveness even higher than those already in game play.

These are some key elements.

Since the windowing over the price sequence of any LTS (liquid tradable security) to derive a spectrum will produce spectra with no particularly improved predictive power than the price sequence in the time domain, the goal must be to discover the nature of the patterns beyond spectral features.
For very short duration investments, spectral features may provide some advantage but perhaps not enough to overcome others using similar predictive strategies.
Some stocks are predictors of others in that there is a strong unidirectional or bidirectional causal relationship.  For instance, the cost of a barrel of oil impacts the future cost of toys, since plastic is a refinery product.  The development of a semantic net with causal relationships modeled and tuned with each epoch of training can produce predictive power that may exceed the market norm.  In this context, something like convolution may be advantageous, but not in the frame-horizontal-vertical-pixel-layer sense, since semantic networks have a non-cartesian topology.  The edges and vertices in a semantic net of LTSs don't make nice 90 degree angles that lend themselves to trivial applications of loops and arrays.
Attractors, another analytical concept from the world of chaos, are an analytical tool that lend themselves to trends in stock groups and possibly individual stocks, however diversification principles of investment also apply well to attractors.  The thing to study is auto-correlation in $\mathbb{R}^{(m n)}$, where $m$ is the number of features in the stock statistics for which there is a fresh historical record and $n$ is the number of LTSs in the group.  The group is not a trench, with varying risk pooled together to provide a favorable curve of return versus stability.  The group is a set of LTSs that correlated positively or negatively and with or without relative temporal shift (delay).
Leverage factors outside the exchange.  For instance, the result of a close election is a key event that impacts markets.  Terrorism is.  Experts who may know nothing about where the market is going but sound good can vastly sway trends.  Some billionaire somewhere may have rode a wave in a temporary correlation between the price of oranges and the price of gold, floating down the eastern seaboard in a yacht, not having the faintest idea why oranges correlated to gold, but analysis showed too many correlative effects to deny the tie, and they rode it until the model broke.

Project Strategy

It is recommended NOT to test with actual trades.  Build a system and create a simulated portfolio.  Do not use retirement funds and college funds to test a system pitted against some of the best trading automation in the world funded by a few hundred billionaires.  You will end up in Gamblers Anonymous that way.  Also understand that, as time passes, the bar will rise, and more people will be trading using predictive systems so that what is a profitable system today may consume all your time and energy and eventually lose everything you gained as other systems gain advantage over yours.

Also remember that asset management, in the final summary, is a zero sum game.  Trading gains are inescapably stuck inside the larger game where the pieces are placed back in the box and the box back in the closet.  This is true even if you have children or grandchildren.  Eventually, chaos has its sway.  Our great, great, great, great grandchildren, who probably won't care to know our name, might end up poorer because we were wealthy or wealthier because we were poor.  It's the Butterfly Effect again.

There are infinitely more important things than asset accumulation.  Ask any honest billionaire with an advanced terminal condition.

Layer Depth?

Yes depth.  There is no way to meaningfully model even the most banal aspects of the complexity of markets with a few layers of perceptrons.

Real Time Learning?

Reinforcement, yes, because the Q-learning algorithm class and variants are a powerful tool for re-entrant, real time learning system design.  However, be careful to not necessarily assuming the Markov Property.  Whether you can use an algorithm, right out of the box, wired directly to inputs from market data with no normalization and prevail in a sub-zero sum game is unlikely.

Innovation will be a requirement to any earned and even short term sustainable success.  Always, with the application of these powerful AI components, thoughtfully apply what is known to the design, initialization, and further development of the system into which the component fits.

Another network type worthy of consideration with time series is the LSTM network design.  This is where layer depth makes sense, however systems with multiple AI components are really the work horses of the industry today.  Very few problems give way easily to the deployment of some code generated from one algorithm specified in one paper and nothing added.

System Topology

There is likely to be multiple pipelines containing information in a system that sustainably provides growth-positive automated or semi-automated portfolio management, even among groupings of LTSs that seem to display patterned and more easily predictable trends.

If one wishes to get a general sense of the complexity of system design that may be required to beat those at the forefront of the investment community, visit a nuclear power plant or disassemble the control system of the jet engine of a passenger jet.  Still, someone seems to be getting it right.  A suspiciously large number of new billionaires have emerged since 1998 with investing given as the industry in their Forbes entry.

Additional technical information is provided in this answer.


Does Monte Carlo tree search qualify as machine learning?

Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but the patterns are not very general), but really MCTS is not a suitable algorithm for most learning problems.

AlphaZero was a combination of several algorithms. One was MCTS, but MCTS needs a function to tell it how good different states of the game might be (or else, it needs to simulate entire games). One way to handle this function in a game like chess or Go is to approximate it by training a neural network, which is what the Deep Mind researchers did. This is the learning component of AlphaZero.


Metrics of quality of parameter space exploration

It is somewhat common to measure the number of operations involved in the search. This is only really useful if you're doing scientific work, because the computational cost of measuring it accurately is quite high. For example, if you were using a GA that used tournament-based fitness selection, and wanted to compare it to one that used round-robin selection, counting the number of evaluations of individuals would be a good measure of total computational effort.

In Genetic Programming, it is fairly common to build the measurement of the total op count into the interpreter you write for the programs. You can then compare that directly to the number of evals times the length of the genomes for something like a GA.