Help with Novelty Recognition and Binary Classification for Emotion Recognition

The Project

It appears from the question that emotional detection and response is the longer term goal of the project and that recognizing potential emotional manifestations in easily detectable physiological metrics is an initial R&D objective.

Mobile device applications are already available to do this, but biometric monitoring via a wristband, excellence in AI design, and marketing excellence could overtake these apps in the marketplace and provide emotional regulation to improve productivity and reduce health and wellness risks.

The (at least initial) goal seems to be basic binary classification, which simplifies the output, but not the detection, which seems to have the following two criteria.

Acquire biological information related to emotion
Avoid drawing the user's attention to perform unnecessary tasks

The Biometric Device Challenge

There is definitely a challenge to classifying biometric trends as meaningful in this context.

  What do you think about my reasoning of the problem? Do you have other perspectives on how to handle these problems? I have been told this could also be considered an anomaly detection problem, for example.

Novelty detection is not the first milestone in this challenge. Detection of useful features is the prerequisite. Novelty comes into play once changes in the organism can be characterized with some reliability and accuracy (few false positives and false negatives) and the particular user has provided subjective reporting to correspond with some change in the organism.

Biological systems have forms of stasis that can be indirectly sensed, including these.

Blood pressure
Metabolic rate
Multi-ionic salinity
Cognitive attention
Blood sugar levels

These affect the dermis at the wrists through perturbations in externally detectable metrics, the inclusion of which depends on the capabilities of the wrist device.

Circumference
Coefficient of electrical resistance
Surface temperature
Multi-ionic salinity
Surface moisture

With blood and urine lab tests, conditions can be controlled, however sampling of those fluids are usually infrequent, leading to the use of ranges to detect anomalies that may be indicative of disease, conditions, or other health risks. There are major challenges to using ranges of metrics at the wrist.

Sensitivity to the general and ongoing physical condition of the individual
Unique individual responses to emotional states
Interference from changes in the temperature, pressure, air movement, and humidity of the environment

One inroad to detection of the internal metrics of the organisms is the fact that biological stasis is not the same thing as perfect regulation. Biological systems are chaotic. Normal signals from time series sampling of biological metrics are neither constant nor periodic. When changes in the attractors and spectra characterizing the chaotic fluctuation of direct metrics (like electrical resistance) or indirect metrics (like systolic blood pressure) occur, a request for subjective information may be asked from the wearer.

These are some chaotic principles to consider.

Spectra can provide much information about the intensity of the force driving the chaotic fluctuation in the time series, which is likely to correlate with stress.
The various ways to evaluate exponents, including the Lyapunov exponent, may be of significant value in correlating patterns in the external, indirect time series to emotional states.
Autocorrelation at the primary frequencies involved may reveal changes indicative of significant mental transitions.

Once a correlation is established, then the novelty detection makes sense, except for an occasional re-acquisition of the correlation, since the wearer may become more internally connected to their physical and emotional states as they use the app, and the answers to inquiries may mature correspondingly.

Additional Questions Within the Question

These additional questions were also asked.

  What algorithms would you use to separate normal from more meaningful physiological data? Support Vector Machines perhaps? Maybe some decision theory?

A recurrent neural network such as B-LSTM or GRU are commonly excellent at characterizing time series, but there may be a need for more than one network.

One to extract the features of the chaotic patterns in the raw data
Another to train to correlate changes in the features of the chaos with subjective reporting

This second training must be done during in situ field use of the first already trained network. Since training is not generally viable on a mobile device, there will need to be a client server arrangement where the training data and its resulting trained network can be communicated through secure RestFUL services so the training processes can be asynchronous and leverage GPUs or other hardware acceleration.

  Do you know any books or papers on similar matters? Even if I have found some after hours and hours of research, you may be able to point me to something different than those I have.

The terms above in a scholarly search will bring up all kinds of study materials. Specifically these

Lyapunov exponent
Strange attractors
Spectral analysis of biometrics
Chaos autocorrelation
Phase space

One good book for all but the middle item is Chaos Theory Tamed, 1997
by Garnett P. Williams.

  It is worth noting that data collection is supposed to be done when no other factors are messing with signal readings, such as sport.

This is where higher level detection involving chaotic analysis is much more reliable than ranges. The chaotic features of athletic activity will be distinctly different from those of fear states or sexual arousal.


What is the "thing" which is trained in AI model training

This answer applies to Machine Learning (ML) part of AI, as that seems to be what you are asking about. Please bear in mind that AI is still a broad church, including many other techniques than ML. ML, including neural networks for deep learning, and Reinforcement Learning (RL) is only a subset of AI - some AI techniques are more focused on the algorithm than parameters.

  What are the trained models? are they algorithms or a collection of parameters in a file?

In ML, the usual process is to feed data into parametric function (e.g. a neural network) and alter the parameters of it to "fit" the data. The main output of this is a collection of parameters and hyperparameters that describe the parametric function. So 90% of the time, when discussing the "trained model", it means the same thing as the collection of paramaters.

However, those parameters are of limited use without a library that can re-create the function from them. Parameters will be saved from a specific library and can be loaded back into that library easily. It is also possible for libraries to read or convert from models saved from other libraries, much like how different spreadsheet programs can read each others' files.

  What do they look like? e.g. file extensions

This varies a lot, depending on which library was used. It is not possible to make a general statement. For example, Tensorflow can save variables to a "Checkpoint" file with .ckpt extension, but can get more sophisticated depending on how much of the model you want to export, and full models with whole structure will contain more than just the variables and have the .pb extension.

  Especially, I want to find the trained models for detecting birds (the bird types do not matter). Are there any platforms for open-source/free online trained AI models??

There are a few places where you can find selections of pre-trained models. One such place is Tensorflow's Model Zoo and you might be interested in Tensorflow detection model zoo.

Other frameworks may also provide example code. For instance Caffe also has a "Model Zoo" (searching Model Zoo is a good starting strategy).

If you are working at the level of collecting model parameters and want to run these models yourself, you will need to learn a bit about each library, what language is used to work with it, and maybe follow some tutorial about how to use it. A few models will be packaged up with working scripts to use from the command line, but many are not and may take sometime and effort to get working.

When you have a specific detection target, you may be disappointed to find models that don't quite match what you want. For image classifying, it is common if you have a specialist need to take a pre-existing general model that has been trained on large dataset for weeks, and then "fine tune" it with your own image dataset for your purpose. Most NN libraries will have tutorials and examples of this fine tuning process.


Does a trajectory in reinforcement learning contain the last action?

If we use $T$ as the notation for the terminal state, then the last action is $a_{T-1}$. This is because when you reach state $s_T$ you don't take another action, which would be $a_T$, because the episode is finished upon reaching the terminal state.


First of all, you need to realise that you will not be able to do it. Google is a multi-billion dollar company, with a large number of very bright and well-funded researchers. That tells me that it is not something a single person can do by themselves.

Then, you already have some pre-conceptions about it. You want to use a machine learning approach, using neural networks. I would start with a blank slate and think about the problem first, before considering the tools I would want to use.

If you don't know were to start, I would suggest you have a look at Eliza, which is old but still very hard to improve on. There are many open source implementations of it available, and it wouldn't be too hard to implement your own from scratch.

Eliza uses a symbolic approach (ie no neural networks), which has the big advantage that you can see what is going on, and you can actually understand how it works. Have a play around with it, create some chatbots, and see how you can make the conversations sound more human-like. Then, when you have a clearer idea about this, proceed to other frameworks such as wit.ai or Microsoft's LUIS. There you trade in control for ease of use; but I suggest you don't do this without understanding what you're actually doing first.


NLP: What is expected from the output of a perfect coreference system?

The clusters you mentioned at the beginning seem right. A "perfect" coreference system should be able to find all words/phrases that refer to the same object.
However, in language, there will always be some ambiguity. For example, consider the sentences "Bob and John like building. He has fun doing it.". It is ambiguous whether "He" refers to "Bob" or "John". In that sense, a perfect coreference system may not even exist theoretically. If it did, it would either avoid classifying "He" or recognize that it could go with either "Bob" or "John".


What is an appropriate fitness function for a simulated self-driving car?

You're correct that it is related to the fitness function, but only indirectly.  Recall that a function is the mathematical embodiment of a conceptual relationship.  The model is a motion model.  Proximity is not simply applying the distance formula $D = \sqrt{(x_2 - x_2)^2 + (y_2 - y_2)^2}$.

Proximity as we "feel it" when driving a car is based on the risks we see in the trajectories of ourselves and the other moving objects along with the stationary objects.  Mathematically, this is an integral over time of a distribution of probable locations.

Look for work in the aeronautics industry about collision avoidance.  The mathematics and algorithms are well developed for antiaircraft weaponry (where the fitness function is the inverse of the one you want) and air traffic control, where the fitness is like yours except with an altitude dimension.


Generalization error: Inputs drawn from distributions

The language used here is confusing me, because it is discussing a "distribution", as in a "probability distribution", but then refers to inputs, which are data gathered from outside of any probability distribution. Based on the limited information my studying of machine learning has taught me so far, my understanding is that the machine learning algorithm (or, rather, some machine learning algorithms) uses training data to implicitly construct some probability distribution, right? So is this what it is referring to here?

They're not referring to probability distributions of training data that ML algorithms (implicitly) construct here. The main point of confusion seems to be where you state this:

  but then refers to inputs, which are data gathered from outside of any probability distribution

Any data / inputs ever collected always originate from some distribution. We will typically not exactly know what that distribution is, we are often not able to provide a clean expression for it, and it might not even be a nice "smooth" distribution, but that doesn't mean it doesn't exist.

If I collect a large number of photographs of $H \times W$ pixels of streets for the purpose of training a self-driving car, then this collection of training data was collected from some distribution. For each of the pixels in the $H \times W$ plane, there exists some probability distribution that tells us how likely it is for such a pixel to have a certain colour under the data collection procedure that was used to generate our data. This is a largely unknown distribution, for which we don't have a nice mathematical expression, but it does exist. I assume that, in this distribution, it's relatively likely for pixels in the centre to be gray (because streets tend to be gray and we collected data by taking photographs of streets). I also guess it's relatively likely for pixels at the top of the images to be blue, because of the sky. Other than that, we can't say much about the distribution, but it does exist.

  Is the "distribution of inputs we expect the system to encounter in practice" the so called "test set"?

Kind of, yeah. Although I suppose the "test set" is mostly a thing in academic settings, where we use a test set to evaluate how well an approach performs on data that it did not observe during training. In the "real world", the distribution of inputs we expect the system to encounter in practice refers to the distribution that generates samples we encounter after "deployment" of the model. For example, this could be the distribution over all images that a self-driving car may encounter when driving anywhere in the world.

Continuing with the self-driving car example, we may get a large generalization error if we only train it on images of streets in one particular city or country, but then afterwards have it drive in many different cities or countries around the world (which may look very different).


Is it a good practice to pad signal before feature extraction?

Padding is a common practice both in image-processing (typically via CNNs) and in sequence-processing tasks (RNNs, Transformers).
For CNNs all the standard convolutional layers - Conv1D, Conv2D and Conv3D,- have the padding argument. The padding values can be valid or same for 2d and 3d convolutions. And extra causal type of padding is possible for 1d convolutions and the documentation refers to this paper: WaveNet: A Generative Model for Raw Audio -  which sounds quite close to what you are interested in.
This animations might be useful to get a bit more intuition about the convolutions and strides/padding. The general consensus is that using same padding is advantageous for model performance - your network gets more information about the borders of your inputs (and deeper networks are possible).
For sequential models padding is even more important. Training samples are usually  of unequal lengths, so you have to pad them with a special token (usually called [PAD] that gets encoded as 0). Here are some examples of this mentioned in tensorflow docs, huggingface.transformers or BERT tutorial.


Is the bias supposed to be updated in the perceptron learning algorithm?

I found the answer to my question.

Treat $\theta$ as a normal weight, associated with an input that always equals $-1$.


Can a trained object detection model deal with variations of the input?

Not necessarily. Supposing your data is from the distribution of possible images containing an upright person close to the camera. Something like a neural network would perform poorly on the new data since it comes from a distribution other than on what it was trained.

You could try augmenting the dataset to try to get some synthetic "far away upside down people" but there are no guarantees here.

I'll look for the source but A. Ng cite's an experiment where a team trained a neural network on a large dataset of vehicle images. They did not realize that the cars were all facing the same direction and their model performed very poorly on images that were very similar with the primary difference being a horizontal flip.


Successful methods for optical character recognition?

This is not meant to be a direct answer, as OCR is not an area I have experience in, but I want to present a deeper problem, that of mathematical notation.

Mathematics has "exploded" in the last few centuries, and notation is not standard across various sub-fields.  Check out this question on notational change with integrals. In some cases, individual sub-fields have alternate forms of notation.

My suspicion is that this task will be enormously difficult without a great deal of human guidance, so I'd point you towards AI methods that utilize "human-assisted machine learning".


Can machine learning be used to pass the Turing test?

Essentially yes.

The Turing Test is essentially a benchmark or challenge problem. It is a task that AI researchers would like to be able to solve.

Machine learning is a technique. It is a tool developed by AI researchers to solve various problems. Some kinds of machine learning are applicable to the Turing Test, but others are not. Machine learning is also applicable to a wide range of other problems.


Ensemble Learning using Convolutional Neural Networks

I am new to AI but this is something I can think of. There might be other much better ways. Or even functions in scikit learn to do it

1) create a list of all the 22 models

2) iterate over the models one by one and use model.predict() for each model and store the hotencoded output to another list or numpy array.

3) Take average of the output list or numpy array.

4) since the output is a vector of 0,1 it is possible to get decimals in the output after taking avg. Round it off using the basic rules to either 0 or 1.

5) If your CNNs are very deep like inception v3, densenet etc. You might run into memory issues while loading all the 22 model's weights into the memory at once. So you can iteratively load the models in small fixed batches and use model.predict() on each model in the batch append the outputs to a list. and then clear them off the memory before loading another batch of models.

This is just my idea. I might be completely wrong also.


What is the difference between the concepts "known environment" and "deterministic environment"?

What's the difference between the two terms? Don't they mean the same thing?

They mean different things, and can occur in any combination.

A known, deterministic environment

This is an environment where the researcher knows how to calculate all the transitions in advance of observing them, and the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Any classic board game against a fixed opponent (where any situation allows choice of action, the opponent always picks the same choice in the same situation)

An unknown, deterministic environment

This is an environment where the researcher does not have the knowledge to calculate all the transitions in advance of observing them, but any observation the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Simple mechanical physics environments, where initial measurements are unknown, imprecise or the researcher does not want to code knowledge of them into the agent. E.g. pole balancing.

A known, stochastic environment

This is an environment where the researcher knows all the rules about transitions, but those rules include transitions with random elements. Transition from state $s$ given action $a$ varies according to some probability function $p(s'|s,a)$ possibly so does the reward $p(r|s,a)$ - sometimes combined into joint probability function $p(r, s'|s,a)$.

Example: Any board game involving dice, e.g. Backgammon.

An unknown, stochastic environment

This is an environment where the researcher does not know all the rules, or can only calculate expected results with difficulty or to a low level of precision. Transition from state $s$ given action $a$ varies according to some unknown probability function $p(r, s'|s,a)$. Learning the transition function may require many samples of $s, a$, and this will be a statistical approximation.

Example: In practice, most complex environments, including real-world physics with friction, fluids, non-perfect measurements.

It is quite common for purpose of experiments to have a simulation where the environment is technically known (because the researcher wrote it, or has access to the code and underlying models), but that agents are written to treat it like this last, more challenging case. Agents that can figure out how to act without prior knowledge of the environment are often of interest.


What are some suitable positive functions as activations of neural networks?

First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say $\bf a$ activation function of the neural net and not $\bf the$ activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.


Which deep learning models are suitable for image-to-image mapping?

Since you have already tried U-Net. You may look into Siamese Networks (with CNNs for images), they are very well known for computing similarity via deep learning. This is a central idea and can be performed with both text and images. As a tip, you may be able to leverage a lot of architecture from U-Net to Siamese.

Hope it helps, Some useful links to start with :

https://medium.com/@prabhnoor0212/siamese-network-keras-31a3a8f37d04
https://www.aclweb.org/anthology/W16-1617.pdf


When do SARSA and Q-Learning converge to optimal Q values?

The true answers are 1 and 3. 1 because the required conditions for tabular Q-learning to converge is that each state action pair will be visited infinitely often, and Q-learning learns directly about the greedy policy, $\pi(a|s) := \arg \max_a Q_\pi(s,a)$,  and because Q-learning converges to the optimal Q-value function we know that the policy will be optimal (because the optimal policy is the greedy policy wrt the optimal Q-function).
3 is true because Q-learning is by definition an off-policy algorithm, because we learn about the greedy policy whilst following some arbitrary policy.
2 is false because SARSA is on-policy, so it will be learning the Q-function under the random policy, and 3 is false because SARSA is strictly on-policy, for reasons analogous to why Q-learning is off-policy.


What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.

The key to the understanding is that in on-policy algorithms you have to use actions (and generally speaking trajectories) generated by the policy in the updating steps (to improve the policy itself). This means that in the on-policy case $a_i = \mu(s_{i})$ (in equations 11-13).

Whereas in the off-policy case you can use any trajectory to improve your value/action-value functions, which means that the actions $a_t$ can be generated by any distribution $a_t~\pi(s_t,a_t)$. In (16) the algorithm explicitly states however that the action-value function ($Q^w$) has to be evaluated at $\mu_{s_{t+1}}$ (just like in the on-policy case) and not at $a_t$ which was the actual action in the trajectory generated by policy $\pi$.


Are standard deviation, variance, skew good features for ML?

I would say it is useful if you have an extensive knowledge in the domain you want to apply your model in. You also need more data for it to yield reasonable results.
As for real world uses l can only think of trading at the moment.


In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$?

The correct interpretation (based on the comment by the author of the question below):
Yep, you are right. It is actually only a single cell per object that contributes to the loss with $\mathbb{1}^\text{obj}_{ij}$ factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.
My original (incorrect) interpretation of the paper:
From what I read in the paper it is not a single bounding box for all cells in the grid for a particular object (your original suggestion).
Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates $B$ bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.
Every cell $i$, which is not a background cell, will have exactly one box $j$, such that $\mathbb{1}^\text{obj}_{ij} = 1$
This is based on the following paragraph from the paper:

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box (i.e. has the highest IOU of any predictor in that grid cell).


What is the optimal exploration-exploitation trade-off in Q*bert?

I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.
More exploration in long term
I think you have correctly identified that exploration could be an issue. In off-policy learning (which Q-learning is an instance of), it is usual to set a minimum exploration rate. It is a hyperparameter that you need to manage. Set too high, the agent will never experience the best rewards as it will make too many mistakes. Set too low, the agent will not explore enough to find the correct alternative actions when the opportunity to learn them occurs.
I would suggest for you something like:
epsilon = max(min(1.0, 1000 / (frames + 1)), 0.01)

You can choose numbers other than 0.01, but I think that is a reasonable start for many Atari games. You could try higher, up to 0.1 in games which are more forgiving of mistakes.
Remove dropout
I am not sure why, but I always have problems with dropout in RL neural networks. Try removing the dropout layer.
More convolutional layers
Convolutional layers are very efficient generalisers for vision and grid-based problems. You won't really benefit much from having a single layer though. I would add another two, increase the number of output channels.
Maybe state representation?
It is not clear from your description whether you are using a single colour frame for the state representation, or stacked greyscale frames for the last 3 inputs. It should be the latter, and if you want to more closely replicate the orginal DQN Atari paper, you should take the previous 4 frames as input.
In addition, you should be normalising the input into range $[0,1]$ or $[-1,1]$. The native image range $[0,255]$ is tricky for neural networks to process, and quite common for value functions to get stuck if you don't normalise.


Why is Standard Deviation based on L2 Variance and not L1 Variance

I've found the answer, the L2 is Standard Deviation, and the L1 is Mean Deviation. Standard deviation describes the variation better and the values are always different on different sets of X while the Mean Deviation gives the same values sometimes.

  *Footnote: Why square the differences? If we just add up the differences from the mean ... the negatives cancel the positives:

  standard deviation why a       (4 + 4 − 4 − 4) / 4  = 0 So that won't work.
  How about we use absolute values?

  standard deviation why a       (|4| + |4| + |−4| + |−4|) / 4  =  (4 + 4 + 4 +
  4) / 4 = 4 That looks good (and is the Mean Deviation), but what about
  this case:

  standard deviation why b       (|7| + |1| + |−6| + |−2|) / 4  =  (7 + 1 + 6 +
  2) / 4 = 4 Oh No! It also gives a value of 4, Even though the differences
  are more spread out.

  So let us try squaring each difference (and taking the square root at
  the end):

  standard deviation why a      √( (42 + 42 + 42 + 42) / 4 ) = √( 644) = 4
  standard deviation why b      √( (72 + 12 + 62 + 22) / 4 ) = √( 904) = 4.74...

Reference:
https://www.mathsisfun.com/data/standard-deviation.html


OpenAI Retro is an extension to OpenAI Gym.

As such, it does not support multiple agents in multi-player environments. Instead, the environment is always presented from the perspective of a single agent that needs to solve a control problem.

When there is more than one agent in a Gym problem, everything other than the first agent is considered part of the environment. Typically player 2 in 2-player problems is controlled by whatever game AI already exists.

You can work around this in a couple of ways, both involve you modifying existing environments, requiring a solid understanding of the integration and wrapper layers in Gym so that you can make correct edits. However you might find environments where either of these are done for you.

In fact, within the Retro environments, some multiplayer environments are supported using the second option below.

1. Supply additional player(s) as part of the environment

You could modify the environment code so that it accepts a parameter which is the agent used to control other players. Typically such an agent would not be set up to learn, it could just be some earlier version of the agent that is learning. You would need to code the interface between the supplied agent and the game engine - taking care to allow for its potentially different view of success.

If a Gym environment has been set up for you (or any human) to compete against your own agent using the keyboard, that is already halfway there to this solution. You could take a look at where the Gym environment has been set up for this, and modify the code to allow for automated input as opposed to action selection by keyboard.

This approach probably works best for competitive games where you can use a random selection of previous agents inserted into the environment to train a next "generation".

2. Present actions for multiple players as if there was a single player

Provided the game emulator allows you to do this, you can edit the code so that multiple players are controlled at once. The Gym environment does not specify how you construct your action choices, so you are then free to split the action vector up so that parts of it are chosen by one trained agent and parts of it by another.

You may need to work on different views of the environment and/or reward signal in this case. For instance in a competitive game (or a co-operative game with competitive elements such as personal scores) then you will need to add custom code to re-interpret reward values compared to single player game. For something simple like Pong, that could just be that Player 2 receives the negative of Player 1's reward signal.

This approach probably works best for cooperative games where success is combined. In that case, you have the choice of writing any number of separate agents that control different aspects of the action space - that would include having agents that separately controlled P1 and P2 game controller inputs.

Each agent would still effectively treat its partner as a part of the environment, so you may want to mix training scenarios so that agents for each controller did not get too co-dependent. For instance, if you hope to step in to control P1, you probably don't want the AI running P2 to have too specific expectations of P1's role and behaviour in the game.

This second option is what OpenAI Retro interface does. For instance Pong supports this:

env = retro.make(game='Pong-Atari2600', players=2)

I verified this using the sample code in the documentation, which controls both paddles, and receives the reward signal as a Python list mostly [0.0, 0.0] but whenever a player scores it becomes [-1.0, 1.0] or [1.0, -1.0] depending on which player got the point.

This will only be supported on retro games where someone has made the effort to make it work as above. If you have a game which does not do this, but in theory supports multiplayer on the ROM, you will need to look at and alter either the AI retro wrapper or the emulator or both.


How to decrease accuracy from 99% to 80%~85% using keras for training a model

There are many things affecting accuracy. I'm gonna assume a lot here because you don't say anything about the model, what you're trying to achieve or how many classes you have. You're not even saying whether you're classifying or not. Also, you're not saying which accuracy you're using (classification, AUC, F1 etc.).

I'm gonna assume here that you have some classification problem.

Accuracy is the measure of how many classifications you got correct. In my experience 99% is a warning sign because it's too good to be true, and a result like that is often due to overfitting. Since this, in my experience, is the main reason you'd actually want the accuracy down, this is what I'm going to assume is your problem.

Overfitting occurs when you train "too much" and the model only learns things that are within your training set, and fails on everything else. That is: it generalized bad.

To prevent this there are a number of things you could do;

1) Data segmentation

The most common is to split your data into three bulks: training (~70% of the data), validation(~20%) test (~10%). These percentages are indications and would vary depending on how much data you have, and the balancing.

The idea is that you train on the training data, you run the validation through the network, and calculate the accuracy. When this accuracy, call it validation accuracy, is satisfactory, then you stop the training and run the test data through it. The latter accuracy (test accuracy) is the one that most papers publish (combined with AUC and F1 score.

Important: When you have split the data into these bulks you should put away the test, and not use it during training at all. You only use this at the very end to do an extra check that you haven't overfitted.

2) Regularization
There are many types of regularization. Two very popular regularization methods for preventing overfitting are the L2-regularization (see previous link) and the dropout methods.

Without going into detail, these methods prevent the model weights from becoming too large. This is a good thing since the model won't rely too much on one feature, which in turn attenuates overfitting.

I hope you learned something, and the most important lesson is that you should know what you're doing. If not, you could end up with a model that is not behaving like you thought. In the case of overfitting, you'd end up with a model that only works on your training data, which doesn't really do much good.

I really recommend the book by Goodfellow: deeplearningbook.org.


Why does the value of state change depending on the policy used to get to that state?

I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there

It doesn't.

The value of a state changes depending on what the agent will do next. That is where the dependency on the policy comes in, not in past behaviour, but expectations of future behaviour. The future behaviour depends on the state transitions and rewards presented by the environment, plus it depends on the distribution of actions chosen by the policy.

More formally, the value function of a state is not just a relative and arbitrary scoring system, but equals the expected (discounted) sum of rewards, assuming the MDP follows the given dynamics, including action selection:

$$v_{\pi}(s) = \mathbb{E}_{A \sim \pi}[\sum_{k=0}^\infty \gamma^k R_{t+k+1} | S_t = s]$$

Without identifying a policy, it is not possible to assess a value function. In value-based control methods, the policy to evaluate can be implied, somewhat self-referentially, as the policy that acts greedily (or maybe $\epsilon$-greedily) according to the current estimates of the value function.

  If there can be a concrete example (perhaps on a GridWorld or on a chess board), that might make it clear why that might be the case

A very simple deterministic MDP with a start state and two terminal states illustrates this:

Start in state B. Taking the left action is followed by a transition (with $p=1$) to terminal state A, and a reward of $0$. Taking the right action is followed by a transition (with $p=1$) to terminal state C, and a reward of $3$.

What is the value of state B? It depends on what the policy chooses. A deterministic left policy $\pi_1$ has $v_{\pi_1}(B) = 0$, a random policy $\pi_2$ choosing left and right with $p=0.5$ has $v_{\pi_2}(B) = 1.5$. The optimal policy chooses action right always and has $v_{\pi_3}(B) = v^*(B) = 3.0$


It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.

Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit.
However, this regularisation is not a magic bullet. The degree of regularisation that you achieve will vary, it may depend on hyper-parameters in the regularisation technique. In your case the relative weightings of objective functions for the accuracy and simplicity of the model will have a sweet spot that minimises overfit without compromising too much and under-fitting instead.

It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

Typically using regularisation requires cross-validation in order to find good values for your regularisation hyperparameters. If the regularisation technique you have chosen is a good fit for your problem, then you may not have to search too much - there may be a broad set of values that work well for you.
In turn, a good choice of regularisation may mean that your model's accuracy is less sensitive to other hyperparameters of your model, so searching for an accurate model becomes a little easier.
However:

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

No. Assuming you want to find the best performing model, you still have to perform cross-validation. At best you may have to perform a little less than without the added objective because it has stabilised your model against other factors that can affect generalisation. However, you might have to perform more cross-validation, at least initially, in order to establish useful values of the new relative weighting hyperparameters you added with the secondary objectives. In addition the new simplicity objective function will likely change the best choices for other hyperparameters, such as number of free parameters, learning rate and length of training time.
If you were previously performing cross-validation after every epoch of training and picking the best model after the epoch that gave the best accuracy on the cv set, then that is often considered a different form of regularisation called early stopping. You may find you could relax this and test less often during training, because with regularisation based on complexity objectives, the training will tend towards a more stable end point and be less likely to overfit through additional training epochs. Although in my experience cross-validation during training is usually left on by default, in order to plot learning curves and ensure this stability really holds.


The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (concat/sum/...) the outputs of forward/backward at a matching timestep t.

You can check how Keras implements it here. There are distinct self.forward_layer and self.backward_layer that are initialized separately.

Your for loop doesn't look ok though. To calculate the output at time step 0 you'd have to calculate forward(0) and backward(n), which means you have to run backward on all the samples first. In practice, each direction is calculated separately and the results are merged afterwards. Check the implementaion in Keras here.


CNNs: What happens from one neuron volume to the next?

Short answer: One to many
Long answer:

The point is that you use a 3D convolution in a CNN. Each kernel has the size of n*m*C (C is the numbers of feature maps) and every feature map has its own kernel(=weights) and bias.

An example:

The size of the layer 2 is: 9x9x10 (stride 1,no padding), the kernel size is 3x3x10

the dimension of the next layer would be 3x3xn (n are the number of kernels that layer 2 has)

here I found a very clear explanation:
https://www.quora.com/How-does-the-second-convolution-layer-work-on-multiple-feature-maps

I hope this helps.


Is my Neural Network program fully connected?

Is figure = code?

No.
Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.

Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

Why size 86?

The input is concatenated with the hidden state:
n_inputs + n_neurons = 36 + 50 = 86.
Why size 100 in the second layer?

The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50).
Therefore you get 50 + 50 = 100.
Why size 200?

There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (layer0/kernel).
Why size 50?

The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.


How to get a binary output from a Siamese Neural Network

I am working on something rather similar in the sense that I get a continuous value (between 0 and 1) as output that I want to be binary. I think you should be using the ROC curve (Receiver-Operating-Characteristic) instead of the accuracy. If you want a single value take the area-under-the-curve. The ROC curve is basically true-prositive-rate vs false-positive-rate as you test for different values of threshold. You could also (or instead) use precision vs recall.

Also, the optimal threshold does not depend as much on the dataset as it depends on your neural network. As you said this value will change as your network learns. Simply use the outputted value and test different thresholds.

If do want to keep the accuracy, simply pick the threshold for which is it best (at the given iteration). Keep in mind that you should not define the threshold from the validation set as you would be cheating.

ps: from rereading it seems like you were implying to learn the threshold from inside the network. I believe it should just be a post-process


How is REINFORCE used instead of Backpropagation?

REINFORCE is called a gradient estimator because it doesn't work on the true gradient, that comes from a loss function and the whole data, but makes up a heuristic loss, so that the gradient it ends up with isn't the true one. Let's see that with the REINFORCE equation:

$$
{\huge
\Delta \mathbf{\theta}_t = \alpha \nabla_{\mathbf{\theta}} \log \pi_{\mathbf{\theta}} (a_t \mid s_t) v_t
}%
$$

As this shows, the gradient is still there ($\nabla_\theta$). But the policy corresponds to the network's output, so we can use backpropagation to compute the gradient of that heuristic loss with respect to the weights. The real gradient is unknown to us, but this estimation will do the job.


Understanding the loss function in deep Q-learning

Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, look up TD-learning to have a better grasp of the concept.


Why don't those developing AI Deepfake detectors use two detectors at once so as to catch deepfakes in one or the other?

Because it is possible to fool many different models at once.
See table 2 in this paper, for an example using adversarial perturbations: https://arxiv.org/pdf/1610.08401.pdf
That being said, there is no reason to think that using two detectors at once will not increase chance to detect deepfakes. It will just not resolve the problem completely.


Is there a rigorous proof for finding Hopfield minima?

See the paper On the Convergence Properties of the Hopfield Model (1990), by Jehoshua Bruck.

In the first section of the paper, J. Bruck describes the Hopfield network (popularized by J. J. Hopfield in 1982 in his paper Neural networks and physical systems with emergent collective computational abilities, hence the name of the network), then he describes the notation that is used throughout the paper and he gives some examples where a simple Hopfield network (with two nodes) converges to stable states (of the network) and cycles (of which the author also gives a definition).

The usual proofs of the convergence properties of Hopfield networks involve the concept of an energy function, but, in this paper, J. Bruck uses an approach (based on an equivalent formulation of the Hopfield network as an undirected graph) that does not involve an energy function, and he unifies three apparently different convergence properties related to Hopfield networks (described in part $C$ of the section Introduction). More specifically, finding the global maximum of the energy function associated with the Hopfield network operating in a serial mode (which is defined in part $A$ of the Introduction section of the paper) is equivalent to find a minimum cut in the undirected graph associated with this Hopfield network.

Furthermore, note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network (more specifically, its weight matrix). For example, if the weight matrix $W \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the network) associated with the Hopfield network is a symmetric matrix with the elements of the diagonal being non-negative, then the network will always converge to a stable state.

See also the chapter 13 The Hopfield Model of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas.


How does adding a small change to an neuron's weighted input affect the overall cost?

I believe he's just saying that:

$$
\frac{\partial C}{\partial z_j^l} \Delta z_j^l \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C
$$

so that the change in cost function can be arrived at simply for a small enough perturbation $\Delta z_j^l$.

Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just:
$$
\partial C \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx  \frac{\partial C}{\partial z_j^l} \Delta z_j^l
$$


How to create an AI snake for a video game?

A relatively simple option which uses AI techniques that are 'traditional' for adversarial games (and which is therefore less of a 'research project' than the use of Machine Learning) is Minimax.

The ingredients for this are:

A list of all the actions that a snake can immediately perform from its current position.
A measure of quality (a.k.a. 'fitness') for the resulting world state.

Traditionally specified for two opponents, the minimax algorithm looks a specified number of moves ahead (alternating between opponents at each turn) and attempts to find the world state that maximizes the quality measure for one opponent whilst minimizing it for the other.

An extension of the two-player algorithm to n opponents (as seemingly required by the OP) is given in this paper.


Why MLP cannot approximate a closed shape function?

In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, tanh or hyperbolic tangent).

Assuming at least one hidden layer, then the universal approximation theorem applies. How closely you can approximate any given function is limited by the number of neurons, and not strongly by the choice of activation function. However, the choice of activation function is still relevant to how good an approximation is. When you get to low numbers of neurons in a hidden layer, then the approximations are more strongly tied to the nature of the activation function.

With one neuron in a hidden layer, you can only approximate some affine transformation of the activation function. Other low numbers, such as 2, 3 etc, will still show strong tendencies for certain families of shapes. This is very similar conceptually to using limited number of frequencies in a Fourier transform - if you limit yourself to only $a_1 \text{sin}(x) + a_2 \text{sin}(2x)$ to approximate a function, then you will definitely notice the sinusoidal building blocks in any output.

I suspect that if you changed the activation function in the first hidden layer to $f(x) = x^2$ then you could get a good result with two neurons per layer. If you then took that network and tried to train it on a simple linear split, it would fail, always producing some curved closed surface that covered the training examples as best that it could - kind of the opposite problem as you are seeing with your circular pattern fitted to NN with tanh activations throughout.

One interesting thing about using $f(x) = x^2$ is that this is a deliberate choice (given knowledge about how you constructed the example) to map your input space to a new space where examples can be linearly separated. In fact this appears to be what layers in multi-layer NNs learn - each layer incrementally and progressively maps its input space to a new space where examples can be better separated in a linear fashion.


What is the “Hello World” problem of Unsupervised Learning?

I disagree with the context that MNIST is the "hello world" of supervised learning. It is definitely, though, the "hello world" of image classification, which is a very specific sub-field of supervised learning.
I'd consider the Iris dataset a better candidate for the "hello world" of supervised learning, with other close candidates such as the Wine, Wisconsin breast cancer or Pima indians datasets. However, as an even simpler and more alternative choice, a lot of people prefer generating their own 2-dimensional datasets so that can more intuitively understand what the different algorithms are doing. An example of this is TensorFlow playground.
Equivalently, in unsupervised learning there are a lot of different tasks. I personally think that clustering is probably the task that is easier for people to understand and as such the most common intro to unsupervised learning. Here there are, as well, two options:

Using an already established dataset, e.g. Iris (without the labels).
Generating your own synthetic 2-dimensional data, to better understand how the algorithms work. An example is this.


Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I see at least 3 issues with your DQN code that need to be fixed:

You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.
Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.
Your batch_target is incorrect. You need to train on returns and not just rewards. In your train function, compute the 1-step return $r + \gamma \cdot max_{a'} Q(s',a')$, remembering to set $max_{a'} Q(s',a') = 0$ if $s'$ is terminal, and then pass it to model.fit() as your prediction target.


How did the OpenAI 5 for Dota concatenate units?

Tl;dr max-pool
You can see in the diagram, everywhere there are a variable number of inputs (pickups, units, hero modifiers/abilities/items), a max-pool follows, though I don't know the specifics of the max-pool implementation.
From https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five :

Notice that while the number of modifiers, abilities and items is variable, the network max-pools over each of those lists. This means that only the highest value in all those dimensions actually gets through. At first it does not seem to make sense – it might give the impression that you have an ability that is combination of all existing abilities, e.g. ranged passive heal. But it seems to work for them.

Above processing is done separately for each of the nearby units, the results from general attributes, hero modifiers, abilities and items are all concatenated together. Then different post-processing is applied depending if it was enemy non-hero, allied non-hero, neutral, allied hero or enemy hero.

Finally the results of post-processing are max-pooled over all units of that type. Again this seems to be questionable at the first sight, because different qualities of nearby units would be combined, e.g. if one of the dimensions would represent the health of a unit, then the networks sees only the maximum health over the same type of units. But, again, it seems to work fine.


Do I need LSTM units everywhere in the network?

For question 1) I dont understand what you are getting at... LSTM cells will work on a contiguous block of inputs, where it sequentially uses the states from a previous time step and the new input to generate the next ones.

question 2) Please look into the LSTM atchitecture
As you see, biases are already there, is there somewhere specific you want it, that it isnt?

question 3) Generally yes, but the normalization step can be expensive (such as softmax), so if you want to get clever, you can use negative sampling or hierarchial softmax-- but generally, you you predict a probability over all possible words given the previous text


Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0?

A common concept in AI is "recursive self-improvement." That is, the AI 1.0 would build a version 1.01, which would build a version 1.02, and so on.

This is probably not going to be thought of as the newer version 'destroying' the older version; if an AI can self-modify, it's probably going to be more like going to sleep and waking up smarter, or learning a new mental technique, or so on.

One important point is that even if the AI is not allowed to self-modify, maybe because of a block put in by its programmers, that won't necessarily prevent it from constructing another AI out in the wild, and so an important problem is to figure out how to best generalize the concept of "don't improve yourself" so that we can make AIs that have bounded scope and impact.


The connection between number of layer of DNN and computational complexity of it

The computational complexity of DNNs is based on 3 main factors.

Matrix Multiplication
Non linear transformation
Weight sharing

Matrix multiplication is the fundamental operation when computing the forward and backward passed in DNNs if using back-propagation. As the complexity of matrix multiplications becomes more expensive as the size of matrices become larger, understanding how to construct networks with effectively sized later which balance the time complexity with accuracy becomes imperative.

Nonlinear transformations allow DNNs to learn nonlinear functions. It is a very important aspect and has been studied rigorously. Classically, the functionf(x) = 1(1+exp(x)) was used to squash outputs from linear layers to be a nonlinear output. However this function has recently been replaced in many applications with the linear rectifier unit (ReLU) f(x)=max(x,0). The relu is much much faster to compute and doesn’t seem to affect the end performance substantially or even noticeably in some situations.

Weight sharing is the idea that some weights in a DNN must share the same value. Beyond the theoretical reasons why this is chosen, it also decreases the number of values that must be updated when performing back-propagation. This is the reason why Convolutional NNs are orders of magnitude faster than their non-convolutional counter parts for image recognition tasks.

There are other things to be aware of when trying to analyze the computational complexity of DNNs but they usually relate to one of the 3 items above.

To estimate the complexity, counting the number of matrix multiplications and their matrix sizes, add time for nonlinearities and you should have a pretty good estimate.


Comparing a large/general CNN to a smaller more specialized one?

After reading your question I can relate it to the Representation Learning papers such as SimCLR and SwAV. These models use a "Big Task agnostic CNN" to obtain smaller representations of the images and then they train another CNN for classification. I suggest you read Big Self-Supervised Models are Strong Semi-Supervised Learners by Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi and Geoffrey Hinton. The code for the following can be found here. But I feel that training such a model would take up a lot of computational resources.


Any interesting ways to combine Monte Carlo tree search with the minimax algorithm?

There has indeed been some research towards combining MCTS and minimax-like algorithms. For example, the following two publications:

Monte-Carlo Tree Search and minimax hybrids
Monte-Carlo Tree Search and Minimax Hybrids with Heuristic Evaluation Functions

The basic intuition behind such combinations tends to be to use small minimax-like searches inside a larger MCTS search, and/or backing up proven wins/losses (proven in the sense that minimax-like search has established that a certain state is a certain win or loss when following perfect play) through the MCTS tree, in addition to the more standard value estimates.

  but if possible I'd like to somehow import MC search into the engine I've already built.

I'm not sure how easy that would be. As I described above, I think it is more common that small minimax searches are used inside a larger MCTS search. I think that would mean that you'd first want to build an MCTS, and then try to import your already implemented minimax into it. That's kind of the opposite / other way around of what you write that you'd like to do.

I don't think integrating small MCTS searches inside a "larger" minimax would work well. The value estimates of small MCTS searches would be too unreliable for subsequent use in an "outer" minimax-like algorithm.


What are some resources for coding some artificial intelligence techniques in the context of games?

One of the simplest games you could solve with an AI technique is tic-tac-toe, which is a very simple game. To solve it, you could use minimax or alpha-beta pruning (an extension of minimax), which are very basic but fundamental search techniques for two-player games, so minimax can be applied not only to tic-tac-toe but any two-player game.

The notes CS 161 Recitation Notes - Minimax with Alpha-Beta Pruning provide a decent overview (with a concrete example) of alpha-beta pruning, which may be a little bit confusing at the beginning, but it is a relatively simple search technique that you can grasp with a few reading iterations (but take this with a grain of salt because the time and effort that it takes to learn something strongly depends on your knowledge and experience). There are several tutorials online that show how to implement minimax and alpha-beta pruning for tic-tac-toe. I could list them all, but I think it's better you look for them and choose your favorite. Maybe have a look at this implementation.

Of course, when it comes to games, one cannot forget about reinforcement learning. I cannot currently list any good tutorials, but there are many resources online. You can even apply RL to tic-tac-toe, so this may be your next step. As someone mentioned in the comments, OpenAI's gym is an RL library for implementing RL agents, which can be used to solve many games, so you should definitely have a look at it. OpenAI's gym comes with a nice introduciton, especially if you are already familiar with RL, which, of course, you probably should before trying to use it. In principle, you can apply RL to many games, including chess, snake, etc.

There's also a book entitled Clever Algorithms: Nature-Inspired Programming Recipes by Jason Brownlee that describes numerous AI techniques, some of them could, in principle, be applied to games too. The book is concise and relatively clear. It also comes with the implementation (in Ruby, which is a language similar to Python) of all of the algorithms presented in it.

To conclude, although this site is not appropriate for recommendations, I recommend you first get familiar with fundamental search techniques, such as minimax and alpha-beta pruning, then you could start learning RL, which can be applied to many games. There are many resources online for both of them.


What is the impact of scaling the features on the performance of the model?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. We do feature scaling to make our model robust to outliers and make an initial impact of every feature on the model will be roughly similar

Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling, but, still, it might be a good idea to rescale/standardize your data.

You should explore your data more carefully, find the outliers, apply transformation if needed.
Not sure if it is a good idea
You can apply different preprocessing techniques like MinMaxScaller, Rank,  Log transform, Extracting square root, StandartScaller and etc.


What is the double sample problem in reinforcement learning?

The double sampling problem is referenced in Chaper 11.5 Gradient Descent in the Bellman Error in Reinforcement Learning: An Introduction (2nd edition).
From the book, this is the full gradient descent (as opposed to semi-gradient descent) update rule for weights of an estimator that should converge to a minimal distance from the Bellman error:

$$w_{t+1} = w_t + \alpha[\mathbb{E}_b[\rho_t[R_{t+1} + \gamma\hat{v}(S_{t+1},\mathbf{w})] - \hat{v}(S_{t},\mathbf{w})][\nabla\hat{v}(S_{t},\mathbf{w})- \gamma\mathbb{E}_b[\rho_t\nabla\hat{v}(S_{t+1},\mathbf{w})]]$$
[...] But this is naive, because the equation above involves the next state, $S_{t+1}$, appearing in two
expectations that are multiplied together. To get an unbiased sample of the product,
two independent samples of the next state are required, but during normal interaction
with an external environment only one is obtained. One expectation or the other can be
sampled, but not both.

Basically, unless you have an environment that you can routinely re-wind and re-sample to get two independent estimates (for $\hat{v}(S_{t+1},\mathbf{w})$ and $\nabla\hat{v}(S_{t+1},\mathbf{w})$) then the update rule that naturally arises from gradient descent on the Bellman error does will work any better than other approaches, such as semi-gradient methods. If you can do this rewind process on every step, then it may be worth it because of the guarantees of convergence, even in off-policy with non-linear approximators.
The paper proposes a workaround for this issue, keeping the robust convergence guarantees, but dropping the need to collect two independent samples of the same estimate on each step.


If deep learning is what you are trying to use here, you should keep in mind that the real intent behind deep learning is to learn a probability distribution, which means that if you were to use a deep learning model to "rotate" images, you can only do it on a specific class of images (e.g. faces, cats, etc...).
If that's your goal, generative models are the way to go:
Autoencoders
You can train an Autoencoders to slightly change the angle. Autoencoders are a special type of neural network that is trained to output the same input you feed into them with a few imposed restrictions to prevent it from learning a trivial identity function. In your case, you could use a variation of a de-noising autoencoder. A de-noising autoencoder, as the name suggests, generates the same input image minus an artificial stochastic noise. The way this is achieved is by feeding a corrupted version of the image and then evaluating the loss on the non-corrupted version.
How can this be adapted in your case?
In your case, you could feed the original images into your autoencoder and evaluate them based on the rotated images. This will result in your autoencoder effectively learning the inner distribution that generates the images in order to generate a slightly "rotated" version of it. For more info on de-noising autoencoders, see the original paper.
Generative adversarial networks
For a more sophisticated approach, you can use generative adversarial networks. GANs are relatively harder to manipulate, but usually perform better than other generative models when it comes to images.
How can this be adapted in your case?
In general, GANs generate images from noise. However, in your case, you can use the original (non rotated) images as input for the generator. The generator can be a convolutional autoencoder for example. And the "real images" dataset will be your rotated images. This way, your model will learn to generate slightly rotated images by being fed an equivalent of noise in traditional GANs which in your case will be the original images. For more info on GANs, I suggest this and the original paper.
I should point out that one of the flaws of GANs is distorted perspectives so it's probably a shot in the dark; however, I think it won't be a problem here because you would be using real images as input instead of complete noise.
Related work
Now, as far as the literature goes, I don't think this has been done before except for this one (kinda) for faces representation learning. The only way this is similar to yours is that you can modify the implementation to only generate the faces it has been fed from different perspectives instead of an average of everything it has learned.


Representing inputs and outputs for a card game neural network

Assuming there's no ordering to the hand (i.e. it doesn't matter what order cards were added to it), then a reasonable approach is to use one input neuron for the number of each kind of card that is present in a player's hand.

You don't describe how the game is played, but a common approach for extracting actions is to have one output neuron for each possible action. To select an action, you would pick the one corresponding to the neuron with the highest output response to a given input.


In Monte Carlo learning, what do you do when an end state is reached, after having recorded the previously visited states and taken actions?

I am assuming you are asking about Monte Carlo simulation for value estimates, perhaps as part of a Monte Carlo control learning agent.
The basic approach of all value-based methods is to estimate an expected return, often the action value $Q(s,a)$ which is a sum of expected future reward from taking action $a$ in state $s$. Monte Carlo methods take a direct and simple approach to this, which is to run the environment to the end of an episode and measure the return. This return is a sample out of all possible returns, so it can just be averaged with other observed returns to obtain an estimate. A minor complication is that the return depends on the current policy, and in control scenarios that will change, so the average needs to be recency-weighted for control e.g. using a fixed learning rate $\alpha$ in an update like $Q(s,a) \leftarrow Q(s,a) + \alpha(G - Q(s,a))$
Given this, you can run pretty much any approach that calculates the returns from observed state/action pairs. You will find that the "bubble up" approach is used commonly - the process usually termed backing up - working backwards from the end of the episode.
If you have an episode from $t=0$ to $t=T$ and records of states, actions, rewards $s_0, a_0, r_1, s_1, a_1, r_2, s_2, a_2, r_3 . . . s_{T-1}, a_{T-1}, r_T, s_T$ (note indexing, reward follows state/action, there is no $r_0$ and no $a_T$), then the following algorithm could be used to calculate individual returns $g_t$:

$g \leftarrow 0$
for $t = T-1$ down to $0$:
$\qquad g \leftarrow r_{t+1} + \gamma g$
$\qquad Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha(g - Q(s_t,a_t))$

This working backwards is an efficient way to process rewards and assign them with discounting to action values for all state, action pairs observed in the episode.

Or perhaps these are two different approaches?

It would be valid to calculate only the return for the first state/action, and randomly select state/actions to start from (called exploring starts). Or in fact take any arbitrary set of estimates generated this way. You don't have to use all return estimates, but you do need to have an algorithm that is guaranteed to update values of all state/action pairs in the long term.

Is there a situation where you'd one rather than the other?

Most usually you will see the backed up return estimates to all observed state/action pairs, as this is more sample efficient, and Monte Carlo is already a high variance method that requires lots of samples to get good estimates (especially for early state/action pairs at the start of long episodes).
However, if you work with function approximation such as neural networks, you need to avoid feeding in correlated data to learn from. A sequence of state/action pairs from a single episode are going to be correlated. There are a few ways to avoid that, but one simple approach could be to take just one sample from each rollout. You might do this if rollouts could be run very fast, possibly simulated on computer. But other alternatives may be better - for instance put all the state, action, return values into a data set, shuffle it after N episodes and learn from everything.


Understanding relation between VC Symmetrization Lemma and Generalization Bounds

Let $\varepsilon$ in (17) is equal to $\sqrt{\frac{4}{n}\left(\log{(2\mathsf{N}(\mathcal{F},n))}-\log{\delta}\right)}$. We have:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) e^{\frac{-n}{4}\left(\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)\right)} =
2\mathcal{N}(\mathcal{F},n) e^{\log{\delta} - \log{(2\mathcal{N}(\mathcal{F},n))}}
$$

As we know that $e^{\log{n}} = n$ (suppose that base of $\log$ here is $e$), we can write:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) \left(\frac{\delta}{2\mathcal{N}(\mathcal{F},n)}\right)
$$

Hence:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant \delta
$$
As we know that if we have $P(x > a) \leqslant c$, we can conlude that $P( x<a) \geqslant 1-c$, we will have:

$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| < \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \geqslant 1- \delta
$$

Now, as this inequality is true for supremum of set $\mathcal{F}$ and $R(f) -R_{emp}(f) \leqslant \varepsilon$ is subset of  $|R(f) -R_{emp}(f)| \leqslant \varepsilon$ (in terms of probability state space), we can say the following inequality is correct for any function $f$ with at least probability of $1-\delta$:
$$
R(f) \leqslant R_{emp}(f) + \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}
$$


Why are CNN binary classifier output probability distributions often skewed?

Yes, due to this issue, you should use temperature scaling after training your model. It will calibrate your probability and you will start to get the same kind of distributions. Here are a good article along with implementation on it.


How does an unsupervised learning model learn?

Supervised learning is typically an attempt to learn a mathematical function, $f(\bf X)=\bf y$. For this, you need both the input vector $\bf X$ and the output vector $\bf y$. The model outputs have whatever dimensionality that the target values have.

Unsupervised learning models instead learn a structure from the data. A clustering model, for example, is learning both how many clusters exist in the data (a number that's not the same type as the inputs) and where those clusters are located (which is also a different type from the inputs). The output of running this model on a new datapoint $x$ is not the same type as $x$, but instead a classification label.

Similarly, time series models learn parameters that symbolize how vectors in the input relate to each other, rather than raw inputs themselves.

As for how they learn, the structures are mathematical objects whose fitness is determined by the input data. The simplest possible unstructured unsupervised learning problem is probably "what's the mean of the data?", and it should be clear how that's 'learned' through processing the input. More sophisticated models are just adding more pieces to that calculation.


How many ways are there to perform image segmentation?

Apart from the multitudes of traditional image segmentation techniques (Watershed, Clustering or Variational methods), newer Segmentation schemes using Deep Learning are actively being used, which provide better results and are better for real-time applications, owing to minimum computation overheads involved.

The following blog provides a detailed review of recent advancements in this field: Review of Deep Learning Algorithms for Image Semantic Segmentation

For the traditional methods, this Wikipedia article provides a nice summary:
Image Segmentation


In Artificial Intelligence, what are semantic analyses used for?

In Natural Language Processing (NLP), you are dealing with knowledge/intents/... expressed in linguistic structures. For example, if you have a text understanding system, you would read the text, usually sentence by sentence, analyse the syntactic structure, and then the meaning (composed of the word meaning and the structural meaning, plus the pragmatic meaning (based on the situation/context)); in other words, looking at the semantic content of a sentence helps you understand what it is about. An AI system can then process the content towards the goal of understanding the text.

In natural language generation, a semantic representation is turned into text. Here the process in reversed: the meaning of the sentence is known, but the appropriate words and sentence structures have to be chosen by the generator.

These are two examples where semantic analysis is used in NLP; there are plenty of others, as meaning is central to the processing of natural language.


What functionality, does control look like in autonomous vehicles levels 4 and 5?

Automation Levels

Most cars have some Level 1 automation, such as cruise control and various skid/flip probability reduction systems. Most high volume passenger vehicles have higher levels. Some military and private air, land, and sea equipment are already at Level 5.

Level 4 requires that driving be automated during normal driving conditions, with manual override.  However, to my knowledge, no one has published a mathematically terse and comprehensive distinction between normal and abnormal to aid in testing Levels 4 vehicles, so testing to a standard is probably not yet possible.

For legal and political reasons, Level 5 is essentially a statistical criteria. For fully automatic to be viable as a market product for general public use, the safety data for passengers and pedestrians must indicate at least the level of safety for manually driven vehicles. Although this will likely suffice from a law and public relations purpose, it is inadequate as a quality standard for automation engineering and testing.  The ambiguities are numerous.

Statistical criteria required to pass the test (i.e. sample size, duration, randomization, and double or single blindness)
Mathematically terse and comprehensive scenarios for the test
Allowable proportion of level 2, 3, or 4 vehicles in the control group.
Probably others

There will be no vehicle driver in what they are calling Level 5 — only passengers.  The idea is to give no power to the occupants of the AV other than destination or change in destination.

This has been the safety norm in other sub-sectors of transportation for a century.  Passengers cannot talk to the pilot of a jet or the engineer of a train.  In the majority of cases, safety is compromised whenever a person that has not undergone the discipline of intense safety training has control over any aspect of the vehicle's operation.

That is the primary impetus behind AV from the forward thinkers in government and academia.

Specificity and Insight

It is of paramount importance that researchers define system criteria more specifically and scientifically.  Systems architecture, software engineering, safety evaluation, and quality control policies and procedures of the automated system driving the AV requires such.

With a billion plus lives at stake, the design should progress with the diligence and care as if designing a human-occupied drone aircraft or a civilian Mars lander from the ground up, even if your first phase is to only achieve what is being called Level 4 for basic passenger cars.

Target Reliability and Safety

Humans eat food, have emotional conversations, text each other, ingest mind altering substances, and fall asleep while driving.  Judging the safety statistics of an AV by comparing with those of humans driving may sound practical, but it is absurd.  It will become clear just how absurd the popularized Level 5 criteria is as the parameters of design are enumerated.

Design should instead minimize the possibility of any accident ever.  The goal should be zero mistakes, both at point of sale (the dealership) and at later points in the product life-cycle as the AV learns.

Defining a Mistake

A mistake should be defined as follows:

  Any less than optimal state indicated by the correction signal used to direct reinforced learning in any of the system's re-entrant or coincident training mechanisms

The adaptive (i.e. machine learning) portion of the system must permit re-entrant or coincident training (reinforced learning) because there is no possible way to predict the common routes of the buyer at the vehicle's point of sale.

To comprehend the complexity of the problem space for AVs and begin to simplify it, consider the dimension of conditions, controls, and priorities (embodied in feedback signals) related to driving on roads for any vehicles that use roads.

Control Channels

Starters (there are two in the case of most hybrids)
Engine stop
Breaking controls (there are two in the case of regenerative breaking)
Steering shaft or hydraulic control position
Break control position (three or four depending on emergency/parking break design)
Transmission planetary gear clutch state, traditional automatic transmission control state, or clutch and traditional manual transmission control state
External lighting switch positions (Widely variable, but at least six for headlight, high beam, break light, left signal, right signal, and tail light)
The content of any messages broadcasted, multicasted, or sent specifically to any other vehicles with compatible reception (if birds do it, so can electronic systems) either via light, sound, or RF (this will require the development of layers of inter-vehicle communications protocols)
Horn
Probably others

Data Acquisition Channels

Wheel positions (there may be 2 or 4 positions to read with an encoder since one cannot assume perfect alignment)
Break torques (4 of them, which can be read by 16 redundant strain gauges)
Break metal temperatures (4)
Torques and temperatures for any independent emergency breaks
Accelerometers (two devices x three dimensions per device to detect acceleration/deceleration, centripetal force, and, with some math, tire skid velocity for all four tires in two dimensions)
Tachometers (one before the transmission and one on each wheel)
Engine and coolant temperature detectors
Cameras (must be high resolution to recognize animals, humans, shopping carts, curbs, road signs, train signals, speed bumps/humps, and hazards, which can be IR or visible and the more angles covered the better)
Wind turbulence resistant external microphones to detect horns and sirens (at least four to detect likely orientation of audio source)
Suspension strain gauges (to detect vertical road force on each tire)
The content of any incoming messages from compatible systems
Battery voltages and currents (two batteries for regenerative breaking or hybrid startup and motive assist, and possibly several currents)
Fluid levels, pressures, viscosity, and transparency (fuel, oil, steering, transmission, break hydraulic, and possibly others)
Probably others

A system can be operational and possibly reach what most would consider Level 5 with fewer channels of acquisition and control than above, but it would be poor technology planning to start designing systems with unnecessary limitations.  Such limitations will also very likely increase the cost of engineering and the effectiveness of training while saving nothing.

Why Vehicles Have Very Little Instrumentation Today

A human cannot make use of all of all the information above.  Nor could a human control all the channels listed above without a high frequency of mistakes.  A properly designed electromechanical learning system can.

It would be lazy systems architecture not to capitalize on the positive impact the additional instrumentation would have on safety, total cost of ownership, and other quality criteria for the AV buyer who can afford the extra sensors and computing power.  Furthermore, after a few years of manufacturing for a mass market, the cost of the additional components will become small in comparison with the cost of metal and plastic.

Operational Criteria and the Formalization of a Mistake

The problem space contains at least nineteen dependent variables (output channels) and forty-six independent variables (input channels).  Some are binary, some are floating point, some are streamed data, some are streamed audio, and some are streamed video.

Together they form a space in sixty-five dimensions.  That is what must be optimized according to some predetermined and possibly re-programmable formalization of what is optimal.

Let's consider this idea of optimum safety, thrift, and comfort as quality control criteria.  Real time quality control should follow TQM ideals, continuously ensuring quality in multiple dimensions and at multiple points in the total system.

Maximal distance from other vehicles
Maximal distance from stationary objects (bridge abutments, buildings)
Maximal distance from pedestrians on foot, bike, wheelchair, ...
Maximal distance from edges of pavement
Minimal lane switching
Minimal loads and torques on wheels
Minimal fuel consumption
Within operation parameters for tires, breaks, engine RPM, and dozens of other parts and subsystems
Shortest distance for route to destination
Shortest time distribution mean for route to destination
Safest route to destination
Least stops on route to destination
Probably others

Most of this must be balanced, so optimization criteria must be aggregated.  Such aggregation must go beyond the simplicity of a loss function.  Summing squares will not work at all, so give up aggregating in such a simplistic way.  A multivariate extension of the Pythagorean Theorem is find for calculating distance in linear space, but driving cars is very non-linear.  This kind of robotics system will require more thought in the formulation of balances, priorities, and the concept of emergency.

Further expanding on the above definition of a mistake, any real time control that does not optimize the sixty-five-plus dimensional surface is faulty.  Now what is optimal must be defined.  Consider the following quality control criteria, roughly in correct order.

Pedestrian safety
Passenger safety
Mechanical system integrity
Fuel conservation
Vehicle external coating integrity
Mechanical system wear
Passenger comfort
Time conservation in reaching destinations
Probably others

Applying Optimization in This Context

Aggregation of the incoming signals acquired are not only based on multiple criteria but the prioritization is not always constant, implying the need for a vector of correction signals rather than a single floating point number.

A single dimension of signaling to feed the disparity between ideal operation and the current system behavior (called a loss function in gradient descent) will not suffice.  There will, out of necessity, be a need for training and reinforcement with a complexity that involves the idea of preemption.  Evolution has declared preemption the design of choice for nervous systems with brains.

For instance, the pedestrian safety feedback signaling must always preempt the fuel conservation feedback no matter how much fuel would be consumed in staying clear from pedestrians, in planning a route where pedestrian density is lower, steering the vehicle clear of people, choosing speed, and applying breaking.

All biological systems have these preemption mechanisms — even bacteria.  A turtle doesn't balance the transportation aspect with safety when it retracts under its shell.  The behavioral interest in the turtle's destination is shelved (stored and temporarily forgotten) until the preemptive system that detected danger indicates the danger has passed.

Humans Should be Passengers on the Streets of the Future

The reason humans are generally driving in a continuous state of mistaken control is because the priorities that maximize the parameters of transportation for society (above) are inconsistently followed by humans.  Birds fly smarter than humans drive.  The priorities of an emotional being that wants to get somewhere fast and while talking, texting, eating, and possibly getting high will often be mistaken.

Future people may look back at the period between the advent of Model-T market penetration and the complete transition to AV as a period of strange inequality.  Stepping back, the worldwide interest in domestic security, airline safety, train and subway safety, and building code contrasts strongly against the cultural insistence of every household have instant access to drive anywhere, any time, and in any mental condition.


Could a CNN hear the difference between sound of a pet moving, and a person?

It is an interesting application. It is possible. You can interpret sound as histogram (2D image) and apply same image processing techniques (CNN) to extract information. Alternatively, you can keep them as phase / intensity values and train a network on top of them (RNN). That is a great idea. Go for it!


Is it possible to train a CNN to predict the dimensions of primitive objects from point clouds?

I'm working on a similar problem. I'm using a 2D point cloud of an object, for example, X and Y coordinates for height, and with that more simple data set I will train a regression model (currently working on that). In my opinion, this approach with dissecting complex point cloud into cross sections that contain wanted dimension and feeding that to the model will be more simple and easier for the training.


Introduction to statistical learning is sort of a resource for the tools that go into AI programming, with some exercises in R.

Elements of statistical learning is the more scientific and math / theory heavy version of ISL. I’m reading that one now.

They are both freely available from Stanford online at the links above.


Dataset too small for multiclass text classification

For pretrained models in NLP, look at BERT and RoBERTa. If you can find a language model trained on your data's superset on Huggingface, then, use that pretrained model.
In order to multiclass classification, since your data is less, look at augmentations in NLP (most notably, backtranslation amongst others). Use focal loss (to handle class imbalance).
Since, you are going to finetune use small learning, 1e-5. But, you will be adding your own layers also, so keep 1e-5 for the pretrained model and 1e-3 for the new layer you put.


Optimum Discriminator for label smoothed GAN

The equation most likely comes from one of the following references:

David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015.

I was not able to retrieve the book but Szegedy et al. discuss on page 6 (see "Model Regularization via Label Smoothing").
Ian Goodfellow discusses this topic on page 31 of his tutorial:

I. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. ArXiv e-prints April 2017.

A similar question was asked and answered in the Data Science community: https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans


A few of us have spent quite a bit of time thinking about this. I summarised our work in a Medium article here: https://towardsdatascience.com/deep-learning-vs-puzzle-games-e996feb76162
Would love to hear what you think.
Spoiler: so far, good old SAT seems to beat fancy AI algorithms!


Is the policy gradient expression in Fundamentals of Deep Learning wrong?

There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
$$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim\pi, s\sim\mu}\left[ G_t \nabla_\theta \log\pi(A_t|S_t)\right]\;;$$
where $\mu$ is our state distribution induced by the policy $\pi$ and $G_t$ are the discounted returns.
Now, as we want to maximise our objective $J(\theta)$, we want to perform gradient ascent. That is, our parameters should be updated according to
$$\theta_{t+1} = \theta_t + \alpha \nabla_\theta J(\theta)\;.$$
The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of $f$ is just the same as finding the minimum of $-f$. Using this, we can then say our parameters can be updated using gradient descent on the negative of our objective, i.e. we get our parameter update rule to be
$$\theta_{t+1} = \theta_t - \alpha \nabla_\theta (-J(\theta))\;;$$
which is exactly what you have in the book.


How do I calculate the policy in the Proximal Policy Optimization algorithm?

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).

However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi_{\theta old}$ is different to $\pi_\theta$.

Here's a great explanation of the algorithm: https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl


Viola-Jones algorithm: Haar-like features, how are the features extracted?

I would look at table 1 of the original paper. While you're reading the alogorithm, try to really focus on Step 2 when you get to it.

In summary, each feature is used to train it's own classifier. So in your example, the calculated features X1, X2, ... Xn you describe coorespond to apply some set of feature transforms f_1, f_2, ... f_n to a single image. This is a bit backwards from what actually happens. What the method really does is train a classifier for each feature. So if you had n features, you would have n classifiers. Then in an adaboost fashion, you upweight the classifier that performed the best. I.e, you are upweighting the classifier based soley on th best performing feature. You then repeat and re-weight all the classifiers until you reach convergence.


How do I handle large images when training a CNN?

How do I handle such large image sizes without downsampling?

I assume that by downsampling you mean scaling down the input before passing it into CNN. Convolutional layer allows to downsample the image within a network, by picking a large stride, which is going to save resources for the next layers. In fact, that's what it has to do, otherwise your model won't fit in GPU.

  Are there any techniques to handle such large images which are to be trained?

Commonly researches scale the images to a resonable size. But if that's not an option for you, you'll need to restrict your CNN. In addition to downsampling in early layers, I would recommend you to get rid of FC layer (which normally takes most of parameters) in favor of convolutional layer. Also you will have to stream your data in each epoch, because it won't fit into your GPU.

Note that none of this will prevent heavy computational load in the early layers, exactly because the input is so large: convolution is an expensive operation and the first layers will perform a lot of them in each forward and backward pass. In short, training will be slow.

  What batch size is reasonable to use?

Here's another problem. A single image takes 2400x2400x3x4 (3 channels and 4 bytes per pixel) which is ~70Mb, so you can hardly afford even a batch size 10. More realistically would be 5. Note that most of the memory will be taken by CNN parameters. I think in this case it makes sense reduce the size by using 16-bit values rather than 32-bit - this way you'll be able to double the batches.

  Are there any precautions to take, or any increase and decrease in hardware resources that I can do?

Your bottleneck is GPU memory. If you can afford another GPU, get it and split the network across them. Everything else is insignificant compared to GPU memory.


Recommended Time serie forecasting model for Fibonacci levels classification

It seems like you are trying to produce two outputs here. This usually makes the models more complex. What if you only predicted the Fibonacci levels for each time step? Then count how many time steps it stays at that level.
As for producing the Fibonacci levels themselves, you can look into categorical time series where the values are categories instead of real numbers.
You have a many-to-one problem. So, yes the model takes two inputs, Duration and Level, but only produces one output, the Level.
The times septs in a model will always be the same. You have to choose a range that is appropriate and all your predictions will be based on that. It's important to note that finding the best time step size is a trial and error process. For example, you could first try using 10 time steps and if you find the model needs more you retrain it with 20 and compare the results.