What is "planning" in the context of reinforcement learning, and how is it different from RL and SL?

The concept of "planning" is not just related to RL. In general (as the name suggests), planning consists in creating a "plan" which you will use to reach a "goal". The goal depends on the context or problem. For example, in robotics, you can use a "planning algorithm" (e.g. Dijkstra's algorithm) in order to find the path between two points on a map (given e.g. the map as a graph).
In RL, planning usually refers to the use of a model of the environment in order to find a policy that hopefully will help the agent to behave optimally (that is, obtain the highest amount of return or "future cumulative discounted reward"). In RL, the problem (or environment) is usually represented as a Markov Decision Process (MDP). The "model" of the environment (or MDP) refers to the transition probability distribution (and reward function) associated with the MDP. If the transition model (and reward function) is known, you can use an algorithm that exploits it to (directly or indirectly) find a policy. This is the usual meaning of planning in RL. A common planning algorithm in RL is e.g. value iteration (which is a dynamic programming algorithm).

Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision- making, or the question of where the predictive models necessary for planning would come from.

Planing is often performed "offline", that is, you "plan" before executing. While you're executing the "plan", you often do not change it. However, often this is not desirable, given that you might need to change the plan because the environment might also have changed. Furthermore, the authors also point out that planning algorithms often have a few limitations: in the case of RL, a "model" of the environment is required to plan.

For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful.

I think the authors simply want to say that supervised learning is usually used to solve specific problems. The solutions to supervised problems often are not directly applicable to other problems, so this makes them limited.

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment.

In RL, there is the explicit notion of a "goal": there is an agent that interacts with an environment in order to achieve its goal. The goal is often to maximize the "return" (or "future cumulative discounted reward", or, simply, the reward in the long run).

How is RL different from planning and supervised learning?

RL and planning (in RL) are quite related. In RL, the problem is similar to the one in planning (in RL). However, in RL, the transition model and reward function of the MDP (which represents the environment) are usually unknown. Therefore, the only way of finding or estimating an optimal policy that will allow the agent to (near-optimally) behave in this environment is to interact with the environment and gather some info regarding its "dynamics".
RL and supervised learning (SL) are quite different. In SL, there isn't usually the explicit concept of "agent" or "environment" (and their interaction), even though it might be possible to describe supervised learning in that way (see this question). In supervised learning, during the training or learning phase, a set of inputs and the associated expected outputs is often provided. Then the "objective" is to find a map between inputs and outputs, which generalizes to inputs (and corresponding outputs) that have not been observed during the learning phase. In RL, there isn't such a set of inputs and associated expected outputs. In RL, there is just a scalar signal emitted by the environment, at each time step, which roughly indicates how well the agent is currently performing. However, the goal of the agent is not just to obtain rewards, but to behave optimally (in the long run).
In short, in RL, there is the explicit notion of agent, environment and goal, and the reward is the only signal which tells the agent how well it is performing, but the reward does not tell the agent which actions it should take at each time step. In supervised learning, the objective is to find a function that maps inputs to the corresponding outputs, and this function is learned by providing explicit examples of such mappings during the training phase.
There are some RL algorithms (like the temporal-difference ones), which could roughly be thought of as self-supervised learning algorithms, where the agent learns from itself (or from the experience it has gained by interacting with the environment). However, even in these cases, the actions that the agent needs to take are not explicitly taught.


Why does Q-learning converge under 100% exploration rate?

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random policy (i.e. 100% exploration) is guaranteeing this and the other conditions are met (which they probably are) then Q-learning will converge.
The reason that different state-action pairs have longer arrows, i.e. higher Q-values, is simply because the value of being in that state-action pair is higher. An example would be the arrow pointing down right above the trophy -- obviously this has the highest Q-value as the return is 1. For all other states it will be $\gamma^k$ for some $k$ -- to see this remember that a Q-value is defined as
$$Q(s, a) = \mathbb{E}_\pi \left[\sum_{j=0}^\infty \gamma^j R_{t+j+1} |S_t = s, A_t = a \right]\;;$$
so for any state-action pair that is not the block above the trophy with the down arrow $\sum_{j=0}^\infty \gamma^j R_{t+j+1}$ will be a sum of $0$'s plus $\gamma^T$ where $T$ is the time that you finally reach the trophy (assuming you give a reward of 1 for reaching the trophy).


In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation.  The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture.  $W$ has dimension $d_{model} \times d_{model}$, which means that we are adding $d_{model}^2$ parameters to our architecture.  $W^Q$ and $W^K$ each have dimension $d_{model} \times d_k$, and $d_k=\frac{d_{model}}{h}$. If we use these two matrices, we only add $2\frac{d_{model}^2}{h}$ parameters to our architecture, even though their multiplication (with the transpose) allows us to have the correct dimensions for matrix multiplication with $Q$ and $K$.
We do use $h$ attention heads, which then brings our number of parameters back up, but the multiple heads let the model attend to different pieces of information in our data.


Any explanation why multiple linear layers work better than a single linear layer in practice?

The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x > 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.


Can I use augmented data in the validation set?

You should not use augmented data in the validation nor in the test set.
Validation and test set are purely used for hyperparameter tuning and estimating the final performance, i.e. estimating the generalization error. These two data sets should be as close as possible to other data, which you could have acquired, but you actually haven not, i.e. your true data distribution.
It's fine to augment training data, since it mimics other samples from the true distribution by applying transformations, noise, etc. and thus helps to increase generalization performance (assuming your augmentation assumptions are right). But evaluation should always be performed on original data.


What makes a machine learning algorithm a low variance one or a high variance one?

What this is talking about is how much a machine learning algorithm is good at "memorizing" the data. Decision trees, for their nature, tend to overfit very easily, this is because they can separate the space along very non-linear curves, especially if you get a very deep tree. Simpler algorithms, on the other hand, tend to separate the space along linear hyper surfaces, and therefore tend to under-fit the data and may not give very good prediction, but may behave better on new unseen data which is very different from the training data.


Fuzzy Logic Controller: Choosing Membership Function

First I need to note that there is no prescribed/best way to choose the shape of membership function in fuzzy systems, that's the fuzziness in it. One could argue that the best way is to ask an expert in the field where you are going to apply your fuzzy solution but those are not always available.

With that said, fuzzy membership functions are used to describe the distribution of probabilities in real world for the variable you are trying to use in your fuzzy controller. That means you go out in the real world, you look at the system you are trying to control, you try your best to understand how it works and reacts to different outside changes and based on your findings you choose the shape that best fits. Or if you want you may call this process heuristic choice (something like that, I do not like theory very much).

On top of that you need to realize one important thing, the shape of membership function does not have big impact on the resulting controller behavior. The most influencing parts are the Fuzzy Rules and their inference methods you use in your controller, but that is a different topic. So no matter what you choose it will not make a big difference.

Gaussian functions are most commonly used because of the character of the world we are living in. Many people argue that everything in the world has Gaussian distribution (everyone is entitled to their opinion). And the triangular functions are used because they are the simplest alternative that is somewhat similar to Gaussian function.

But if you absolutely need to chose the best one for your particular problem. There always are Simulation tools that exists precisely for this purpose. One of them is Matlab Simulink as you mentioned, but there are others if you don't like the price of the Matlab.

My suggestion is, go with your gut, test it in safe environment and if it works then deploy it to the real world.


What is the typical AI approach for solving blackjack?

Blackjack is usually modelled using Monte Carlo (MC) Methods. There is a lot of literature on MC methods which is interesting on its own right but here is a paper describing how MC is applied to Blackjack. There is also a good description on page 110 of the Introduction to Reinforcement Learning.

Good luck!


How is an architecture composed of a second model that validates the first one called in machine learning?

Not in terms of models, but there is a terminology called 'Hierarchical learning', wherein if your model has a task to classify disease, then, If it detects a presence of a disease (disease/ no disease), then it proceeds to further classify a disease(class A/B/C/...). Else it does not proceed.
This technique of hierarchical learning is very common amongst supervised learning tasks.

Now according to your question, you have two models and I assume that they have different tasks and provide a binary outcome(yes/no). Here, you can call it as 'Multitask learning', where the output of task1 is given to task2 for processing.
If task1 detect the presence of disease, then task2 classifies disease into various classes / or segment it / localize it etc.


When we use a neural network to approximate the Q values, is the Q target a single value?

When we use our network to approximate our Q values,is the Q target a single value?

Yes, the target Q value is a single value if you are just updating a single training example. The loss function of a vanilla DQN for a single experience tuple $(s_t,a_t,r_t,s_{t+1})$ is calculated as $$L(\theta) = [r_t + \gamma \,max\,Q_{a_{t+1}}(s_{t+1},a_{t+1};\theta) - Q(s_t,a_t;\theta)]^2$$ where $r_t + \gamma \,max\,Q(s_{t+1},a_{t+1};\theta)$ is the target Q value. However, when using mini-batch gradient descent, you would have to compute multiple target Q values equivalent to the batch size

During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?

During backpropagation of the loss function, the weights $\theta$ are automatically updated. You do not need to pass in the state again. Because in the first place, you would have computed $Q(s_t,a_t;\theta)$ by passing in the state as input to the neural network. That is how backpropagation works for Deep Q networks.
Training for the DQN is as follows:

Collect experience tuples of $(s_t,a_t,r_t,s_{t+1})$ and store them in a replay buffer.
Sample mini-batch of experiences from the replay buffer.
From these sampled batch of experiences, compute $Q(s_{t+1}.a_{t+1};\theta)$ by passing $s_{t+1}$ into the network and take the Q value with the maximum values
Compute $Q(s_t,a_t;\theta)$ by passing $s_t$ into the network.
Compute the Loss for this experience and propagate the loss back to the network, hence updating the weights.

Also, since the weights have changed after backpropagation, the Q values for the same state would also be updated if you pass the same state in to the network again. Check out this paper as it explains how Deep Q Network works.


What does the notation $[m]=\{1, \ldots, m\}$ mean in the equation of the empirical error?

This is a commonly used notation in theoretical computer science.

$[m]$ is not the variable $m$, but is instead the set of integers from $1$ to $m$ inclusive. The empirical error equation thus reads in English:

The cardinality of a set consisting of the elements $i$ of the set of integers $[m]$ such that the hypothesis given input $x_i$ disagrees with label $y_i$, normalized by $m$.


Why is loss displayed as a parabola in mean squared error with gradient descent?

Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take $o_{expected}=20$ and $o_{net}=40$ as example. Your MSE is then 400, because
$MSE = (o_{expected}-o_{net})^2$.
Just imagine $y = x^2$ with x being the output of your network.
If you want to shift the parabola with optimum at $20$ the formula you get is $y = (20-x)^2$.
For every new case you train the  net on, you get a different parabola with different parameters.


Choosing an instance in AWS or GCP for training a CNN

I can't do a better job at explaining than Jeremy Howard's team did.

Follow carefully and systematically all the steps put together here.

You can read more about the different types of machines here, about the possible choices of GPUs here and about the regions and zones here.

Follow the fastai course too.
I can't emphasize how good is it.
You can find the lessons in the left panel.

I would recommend you use Colab for experiments as it has a Tesla-K80 and the cloud computing for something that requires more power.

Hope it helped!


What is the best approach for multivariable and multivariate regression?

You create a model like this very easily with keras. Follow these steps.
Data Exploring
Before building a machine learning model, you must explore your data first. An approach is to use libraries to visualize the data as graphs. You can use the tool pandas-profiling
https://towardsdatascience.com/exploring-your-data-with-just-1-line-of-python-4b35ce21a82d
Points to look for when exploring the data
You should look for the following:

Distribution of labels and input features
Range of input features and labels
Missing label or input feature
Nan on value
Outliers

You should look out for them and use appropriate data cleaning or filtering method to clean them. For example, Nan or missing value may be substituted with 0 and outliers will be removed. Input features may also be normalized.
Building the model
Keras is a easy tool for building machinea learning model. For how to build a basic MLP ( Multi-Layer Perceptron), you can refer to the example code and the resource.
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
Experimenting
After creating the basic model, you can start experimenting with stuff like:

Different model structure like adding layers and changing hidden layer size
Different Optimizer
Different hyperparameters like learning rate and batch size
Different input features

You will gain more experience as you go. There is a lot of ways to improve the performance of the model.
Bewares
Here are things to beware:

If your model doesn't work out, don't worry. Try another method or tune parameters until it work.
If the testing loss become very high while training loss is low, you can add dropout to prevent this from happening. This is called overfitting.

Hope I can help you and have fun!


How to design a recommendation system for shift swapping?

You might not even need a classifier.

I would devise a scoring function, based on analysis of the previous data you have. Each user gets a score based on features like

how many times in the past has this user swapped a shift with somebody else?
how many times has the user swapped with the current user?
how many times has the user swapped this particular shift?

For each criterion you add a number of points to the score; the second one might be weighted higher that the first one. Then the person with the highest score is most likely to switch shifts with your current user.

The main question is the design of the scoring function, but I don't think you'd need to go into all the overkill of setting up a classifier; just think which criteria would make someone more likely to swap, and encode them directly. This has the advantage that it's transparent, ie you can always see why someone got recommended, and you can tweak your scoring method if the results are not quite what you'd want. This is often hard to do with ML classifiers.


In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew.
The point of regret as a metric therefore is to compare your choices with the ideal choices. This makes sense in MABs, because although the primary goal is to gain the most reward, the learning part of the goal is to calculate from experience what are the best choices - usually whilst sacrificing as little as possible in the process.
The formula captures that concept, so does not concern itself with individual rewards in the past that could have been due to good or bad luck. Hence it uses expected (or mean) rewards.


Can a deep neural network be trained to classify an integer N1 as being divisible by another integer N2?

There is a recent development in research that was looking into effectiveness of neural networks on arithmetic. Interestingly, feed-forward neural networks (MLPs) with various activation functions as well as LSTMs (RNNs which are Turing-complete) are not able to model simple arithmetic operations (e.g. addition/multiplication), they designed a new logic unit which can solve all of the simple arithmetic problems.

See: Neural Arithmetic Logic Units

More recently, DL can solve symbolic maths: Deep Learning for Symbolic Mathematics


(1) Yes this is the diagram for a classical LSTM unit. Of cause there are some variants and those diagrams would look slightly different.

(2) It is very common for researchers to use more than one layers of LSTM and achieves better performance than a single layer one. A common way to "stack" LSTMs is to use the previous layer's output ($h_t$ in your diagram) as the input to the next layer ($x_t$). However, I have seldom seen any successful application of 5+ layers of LSTMs, while for CNNs it is common to use tens or even hundreds of layers.


Detect observations under certain conditions

Since my comment would not fit, I'll answer the question.

I think this problem will be even more better solved if you have the web data as well. Say after adding the required necessity in the cart the customer will check the total amount. And if it is below 50$, the person will add some more items. So checking this can give you a better clue.
Another data you must have for better guessing is the order in which the person has added items to his cart. This will also provide you an important clue about the cheap items the customer is adding to cart to try to cross the minimum threshold.
You will also have to segregate cheap items from pricey ones for a customer. This data with the above 2 approaches is a definite giveaway.
This might be a personal opinion, but whenever I need to cross the shipping fee threshold I usually add items which I have already bought the previous time to escape shipping fee. You will need more data to see if this is true in most cases.
And if you don't have any of the above data, the pure Machine Learning approach would be to pick an item which is costly and somewhat popular and an item which will string together people with likewise interests. Say, a great book on ML is this item.
You find out all the customers who bought this book. Check all the similarities between all the other things bought by the customer, like say one customer adds a book on AI after this while another buys a book on python. So these are related by computer science. Go on until you find all the things that are generally dissimilar between all the customers, since now they are buying to cross the threshold and each will buy according to his own requirements without a common interest in mind. There you have all the data you need.
Or you can use the converse of this approach, find a non-significant thing like sugar. String together all the customers who bought this item. Check their interests to see if it has anything to do with sugar. If not send them to one group. Now in this group, match the interests of the customer on other items, if dissimilar you get a good idea that this item was bought to cross threshold.

I understand this is a kind of opinionated answer. But I think these are trade secrets not revealed by companies. so you have to figure out algorithms yourself. Also, simple machine learning won't work, lots of logical programming is also required. And I also think you need to have a good understanding of human psychology works. You can interview your friends and family about what they would do and get a general intuition. Hope this helps!


What is the name of a human-inspired machine learning approach?

If it was based on how the human brain learns, it might have used hebbian learning.

One example for such a network would be HTM.


Are there any discount-factors based on branching factors?

First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the optimal policy in the case where a discount factor is absent. This means that "artificially" adding a discount factor is harmful if we expect to be capable of learning / finding an optimal policy. Generally, we don't expect to be capable of doing that though, except for the case where we have an infinite amount of processing time (which is never in practice). In that other answer which you linked to in your question, I describe that it can still be useful, that it may help to find a good (not optimal, just good) policy more quickly, but it does not come "for free".

I'm not aware of any research evaluating ideas like the one in your question. I am not 100% sure why, I suspect it could be an interesting idea in some situations, but would have to be investigated carefully due to my point above; if not evaluated properly, it could also unexpectedly be harmful.

One thing to note is that the use of discounting factors $\gamma < 1$ is extremely common (basically ubiquitous) in Reinforcement Learning (RL) literature, but rather rare in literature on tree search algorithms like MCTS (though not non-existant; for example, it's used in the original UCT paper from 2006). For the concept of "branching factors", we have the opposite; in RL literature it is very common to consistently have the same action space regardless of states ("constant branching factor"), whereas this is very uncommon in literature on tree search algorithms. So, the combination of discount factors + branching factors is actually somewhat rare in existing literature (which of course doesn't mean that the idea couldn't work or be relevant, it just might explain why the idea doesn't appear to have been properly investigated yet).

One important concern I do have with your idea is that it seems like it could be somewhat of an "anti-heuristic" in some situations (with which I mean, a heuristic that is detrimental to performance). In many games, it is advantageous to be in game states where you have many available moves (a large branching factor), this can mean that you are in a strong position. Consider, for example, chess, where a player who is in a convincingly winning position likely has more moves available than their opponent. I suspect your idea, when applied to chess, would simply promote an aggressive playing style where both players capture as many pieces as possible in an effort to quickly reduce the branching factors across the entire tree.

  If not, I'd welcome thoughts on how this might be applied.

  (I might divide 1 by the number of branches and add that value to the goodness of a given state, but that's a "value-add" as opposed to a discount.)

Such an additive change would be more closely related to the idea of reward shaping, rather than discounting (which is again a thing that, if not done carefully, can significantly alter the task that you're optimizing for). Intuitively I also suspect it might not do anything at all since you'd always be adding the same constant value regardless of which move you took (regardless of which move you took, your parent state will always have had the same branching factor). I might be missing out on some details here, but I think you'd have to have a multiplicative effect on your regular observed rewards.

I suppose one example that could be worth a try would be something like maximizing the following return;

$$\sum_{t = 0}^{T} \gamma^t \beta^{b(S_{t}) - 1} R_{t + 1},$$

where:

$0 \leq \gamma \leq 1$ is the normal time-based discount factor (can set it to $1$ if you like)
$0 \leq \beta \leq 1$ is a new branching-factor-based discount factor
$b(S_t)$ is the branching factor (the number of available moves) in state $S_t$
$R_{t + 1}$ is the immediate reward for transitioning from state $S_t$ to $S_{t + 1}$

I put the $-1$ in the power of $\beta$ because I suspect you wouldn't want to do any discounting in situations where only one move is available. Intuitively, I do suspect this $\beta$ would require very careful tuning though. It is already quite common to choose $\gamma = 0.9$ or $\gamma = 0.99$, with $\beta$ you may want to stay even closer to $1$.


Are Asimov's Laws flawed by design, or are they feasible in practice?

Asimov's laws are not strong enough to be used in practice. Strength isn't even a consideration, when considering that since they're written in English words would first have to be interpreted subjectively to have any meaning at all. You can find a good discussion of this here.

To transcribe an excerpt:

  How do you define these things? How do you define "human", without first having to take a stand on almost every issue. And if "human" wasn't hard enough, you then have to define "harm", and you've got the same problem again. Almost any really solid unambiguous definitions you give for those words—that don't rely on human intuition—result in weird quirks of philosophy, leading to your AI doing something you really don't want it to do.

One can easily imagine that Asimov was smart enough to know this and was more interested in story-writing than designing real-world AI control protocols.

In the novel Neuromancer, it was suggested that AIs could possibly serve as checks against each other. Ray Kurzweil's impending Singularity, or the possibility of hyperintelligent AGIs otherwise, might not leave much of a possibility for humans to control AIs at all, leaving peer-regulation as the only feasible possibility.

It's worth noting that Eliezer Yudkowsky and others ran an experiment wherein Yudkowsky played the role of a superintelligent AI with the ability to speak, but no other connection outside of a locked box. The challengers were tasked simply with keeping the AI in the box at all costs. Yudkowsky escaped both times.


It is called a Latent variable because you cannot access it during train time (which means manipulate it), In a normal Feed Forward NN you cannot manipulate the values output by hidden layers. Similarly the case here.

The term originally came from RBM's (they used term hidden variables). The interpretation of hidden variables in the context of RBM was that these hidden nodes helped to model the interaction between 2 input features (if both activate together, then the hidden unit will also activate). This principle can be traced to Hebb's rule which states "Neurons that fire together, wire together." Thus RBM's were used to find representation of models, in a space (generally lower dimensional than than the original). This is the principal used in Auto Encoder's also. Thus as you can see we are explicitly, not modelling the interaction between 2 features, how the process is occurring is "hidden" from us.

So, the term latent basically can be attributed to the following ideas:

We map higher dimensional data to a lower dimensional data with no prior convictions of how the mapping will be done. The NN trains itself for the best configuration.
We cannot manipulate this lower dimensional data. Thus it is "hidden from us.
As we do not know what each dimension means, it is "hidden" from us.


Does elitism cause premature convergence in genetic algorithms?

There are many ideas to escape from local optima in GA. One solution is selecting the population for the next iteration based on the probability that is defined based on the individual score. In that case, you have a chance to select a bad score individual to escape from the local optima.
Another efficient solution is playing with the mutation rate to get rid of local optima. In that way, you can increase the rate smoothly, to find a proper rate.


How can I improve this word-prediction AI?

Seems like recurrent neural networks (RNN) should work for your use case. An excellent introduction is available at: The Unreasonable Effectiveness of Recurrent Neural Networks


Why can't LSTMs keep track of the "important parts" of a sequence?

The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.

To prove this point, I recommend reading Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons".

Addressing specifically this comment in your question:

  I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).

If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.

Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements

Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.

LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important

For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another view of the input may be most helpful.

This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.


What is the difference between a machine learning engineer and deep learning engineer?

Job titles in IT are somewhat arbitrary, and not very consistent even within the same country or general industry. Job titles in software development, especially newer forms of software development such as mobile apps, even more so. The latest trends in Data Science, Machine Learning and "AI" are at the sharp end of that.

There are some common themes - e.g. an AI Researcher is likely to be expected to be up to date with scientific papers and might be expected to produce some (or summaries/applications of same within the work environment) whilst a Deep Learning Engineer might
be expected to be familiar with the ins and outs of provisioning compute resources in Amazon (AWS) or Google (GCP), moving data around for training or inference, with enough coding experience to integrate a TensforFlow model with some already-existing web service. However, none of this is strictly separated and a job advertised as "Data Scientist" for a smaller company might be expected to do both those things and more.

All this means that your best bet is to not take much notice of the job title. Read the job specification carefully. Approach each company with questions about daily tasks, responsibilities and assumed technical skills. Expect in some cases that the companies will be unsure themselves of what the job entails and expecting you to take a mature role in defining your own working relationship with the rest of the company - the more senior and experienced the role being looked for, the more likely this is. Both sides  - prospective employee and prospective employer - may have incorrect assumptions about what is realistic and possible.

TL;DR: It's a bit Wild West out there at the moment, and likely will remain so over the next decade. To reduce variance in your own experience, research job posts carefully and talk in detail to employers about expectations in advance.


Deep Q Learning Algorithm for Simple Python Game makes player stuck

This is probably the most major factor:

model.compile(loss='categorical_crossentropy', optimizer='adam')

you have set the loss function for a multiclass classifier. It is going to have some weird results when values - either predicted or target - are outside of range 0..1

You should use this instead:

model.compile(loss='mean_squared_error', optimizer='adam')

because your Q network outputs the expected future return on each action. This could easily be outside of the range that 'categorical_crossentropy' is designed for.

In addition, you really need to look into experience replay. It is not an optional extra when using neural networks with Q learning - it is pretty much required for anything but the most trivial environments. It is very likely your agent will still fail to learn without experience replay, if you correct all other problems with your code.

  I am not sure if the NN architecture (units and hidden layers) is appropriate for given complexity.

It looks more complex than it needs to be, assuming that your 4 inputs represent paddle x position, particle x,y position and colour. I would suggest making the network simpler (maybe just 40 neurons per layer at a guess), to speed things up a little.

Check your input scaling. Neural networks like to train on inputs that have mean 0, standard deviation 1, and it is worth scaling them so that they fit roughly into -1..1 or similar. Your feature engineering is not shown in your code, so it might be an issue.

  Also, is it possible that this is failing due to the rewards being very delayed?

This can be a factor that makes learning harder.

  It can take 100+ frames to get to the food, so maybe this isn't registering well with the neural network.

100 time steps delay between rewards is not much for DQN. It should be correctly predicting Q values - it will just take more episodes to learn to predict the best movement when the food is further away.


Could new training pictures destroy the trained weights of the neural network?

Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?

This can happen, and happen to various degrees depending on how the neural network is set up, but it is usually something you want to avoid.
Provided a neural network has enough capacity (in terms of number of free parameters), then the function it generates can be flexible enough to approximate the "true" function of the problem you are trying to match to data. Then as long as the learning steps are small enough, adjustments to learn one point on the function will take a larger step towards correct answer on that point than they adjust away from correct answer on other points (intuitively this is about the mutually good solutions being in part orthogonal and/or correlated, so a step towards A does not necessarily move the same distance away from B). Repeating the points again and again will allow the NN to match all the points over time.
Often this is possible to do almost exactly for a large amount of data at the same time. However, this is the opposite problem to your worries in the question, and is considered a bad thing because it can cause overfitting. Overfitting is very easy to achieve with neural networks because they are so flexible and can have large numbers of parameters.
Most often, the goal of training a supervised learner, such as a neural network, is not to exactly match the training data, but to generalise to new data from the same population distribution. You will typically spend far more time and effort combatting overfitting when training a NN, than worrying about lack of capacity to handle multiple data points. Methods to combat overfitting are:

Regularisation. Techniques that adjust a learning algorithm, typically by adding some constraint.
Cross validation. Measuring results of training against unseen data, and picking the best generalised model.


Multiple GPUs one expensive GPU, which gpu to buy for real time processing (not training)

Here's a link to some benchmarks that should give you some insights.

In my experience (I've used systems with both 1080s and V100s) I've found that as of about a year ago, a lot of the common tools couldn't use the V100s well.  Until we started doing some manual optimization, the 1080s were comparable if not better on common tasks.  Of course, once we put in the time to optimize, recompile drivers, etc., the V100's really came to life and outperformed everything else by several times.


What are the best algorithms for image segmentation tasks?

U-Net and U-Net inspired architectures have been quite popular in the medical image-related tasks ever since it was first introduced. There have been several improved versions of U-Net designed for specific tasks that followed. One such example is Attention U-Net, extremely popular for Pancreas Segmentation.

Other examples of architectures that have achieved state-of-the-art results in image segmentation tasks in recent years include Multi-Scale 3DCNN + CRF, popular for Brain and Lesion images, Multi-Scale Attention for MRIs, etc. A recent paper that describes an interesting 3D FCNN architecture is HyperDense-Net, widely used for multi-modal tasks in medical image segmentation.


How can I stabilise a recurrent neural network used for binary classification?

Firstly, dealing with the issue that the program gives different answers every time without making any changes can be due to a couple of things.

Assigning random values to weights and bias. This can be solved by setting a seed manually at the start of the program.
Make sure you have set the model to the testing mode after training. For some frameworks, this has to be done manually.

Secondly, regarding your expected results.

To generate a proper accuracy metric, you will have to sample your dataset into training and testing data, making sure there is now overlap between them. This might be an issue as you have stated training on data till 2017 and then again training on data of 2018.

Lastly, don't expect that the model will know that the output is wrong and directly change it because it's binary classification. This is not how neural networks work. The model fits the solution better by gradually updating its weights and biases over a number of iterations. So it will take a number of epochs to learn new trends in the data for 2018.


Is value iteration stopped after one update of each state?

Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the pseudocode for Value Iteration, which consists of iterating between policy evaluation and policy improvement.
In normal policy evaluation, you would apply the update $v_{k+1}(s) = \mathbb{E}_\pi[R_{t+1} + \gamma v_k(S_{t+1})|S_t = s]$ until convergence. In the policy iteration algorithm, you perform policy evaluation until the value functions converge in each state, then apply policy improvement, and repeat. Value iteration will perform policy evaluation for one update, i.e. not until convergence, and then improve the policy, and repeat this until the value functions converge.
The line
$$V(s) \leftarrow \max_a \sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]$$
perform both the early stopping policy evaluation and policy improvement. Lets examine how:
The $\sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]$ is the same as the expectation I wrote earlier, so we can see clearly that is policy evaluation for just one iteration. Then, we take a max over the actions -- this is policy improvement. Policy improvement is defined as (for a deterministic policy)
\begin{align}
\pi'(s) &= \arg\max_a q_\pi(s,a) \\ &= \arg\max_a \sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]\;.
\end{align}
Here, we assign the action that satisfies the $\mbox{argmax}$ to the improved policy in state $s$. This is essentially what we are doing in the   line from your pseudo when we take the max. We are evaluating our value function for a policy that is greedy with respect to said value function.
If you keep applying the line from the pseudocode of value iteration it will eventually converge to the optimal value function as it will end up satisfying the Bellman Optimality Equation.


Object detection: combine many classes into one?

I think there is no absolute answer for this. Often its kind of trial and error. In general the CNN tries to generalize the problem, so using all logos with different augmentations and ground truths can maybe lead to some feature maps, which are so general that the CNN can find logos.

But if your logos are so various, and embedded in colorful websites, the tasks seems quite difficult, also if they vary in shape and form.Like you said, I think you definitely need an FPN (Feature Pyrimad network) to get the different sizes, scales and so on combined with an RPN (region-proposal network) to find the logos multiple times in the websites (if thats nessecary). For that you can use Mask-RCNN (https://github.com/matterport/Mask_RCNN). You can try to transfer-train it on Imagenet-Backbone for example to reduce training time. Just tried to use Mask-RCNN to segment colored cells from medical images. Worked out quite good.

I used ImageLabel for labeling, worked out quite intiutive and saved all in a JSON file.Imglabel

How large is your dataset and how complex are the logos ? Do you have examples which you can demonstrate?


Wouldn't convolutional neural network models work better without flattening the input in any stages?

I have had similar thoughts about neural networks before. Convolution layers are layers of two dimensional nodes effectively passing the spacial data so why don't we use two dimensional hidden layers to receive information out of them.

I'm sure someone has used this type of implementation before. I believe the papers bellow are using this. Part of the point of neural networks is that the weights are trained in order to optimize finding the best solution so regardless of the spacial information it learns to 'focus'/increase weight on locations that are associated with deciding the solution.

Think of the problem where your neural network examines an image and gives true or false. Training images are True if the center is red and one of the corners is blue or if the center is blue and one of the corners are red. Flattening the layers or not should have basically no effect on this model. In other circumstances like object detection or labeling outlines yes I believe not flattening will benefit the model. With that said flattening the data does not erase spacial relationship each layer will still be trained to detect the spacial information that gives a correct answer the flattened layers just won't have the benefit of neighbors when the layers are one dimensional instead of two.

In a a CNN with multi class detecting as the task you could allow each class to have its own CNN like hidden layers that narrow to a decision node and decide if they match that class or not. Imagine a palm tree shape where the palm trunk is the image convolutions and each leaf on the top are the two dimensional hidden layers that narrow to an output layer.

 Multi-dimensional NN    and
 Three dimensional Neural Network

I know I spoke in a lot of abstraction so if any part doesn't make sense, I'll make an edit to clarify.


Yes, this method of training a model is commonly known as online learning and specific learning algorithms have been designed for this purpose, such as, Stochastic Gradient Descent(SGD). As opposed to Batch Gradient descent, which computes gradients over the entire training set at each step, the SGD algorithm computes gradients for individual samples and updates the model's parameters.
The online learning strategy is not specific to Reinforcement Learning(RL) methods but is also suitable for the broader classification of Machine Learning algorithms. Suppose, we are given a simple model pretrained on a large Face Recognition dataset, and wish the model to continuously adapt to new faces, we may keep feeding in the new input images with their correct labels (obtained from feedback), as they appear, and apply SGD to incrementally update the model. To go about this using an RL based approach, we may penalize the model with negative rewards for every label misclassified and update it.


Is a genetic algorithm efficient for a snake game?

Q-Learning and Genetic algorithm are both good algorithms to create an IA that plays Snake.
The one you use depends mostly on how you understand and model your IA environment.

Q-Learning algorithm is an algorithm that needs a State (give by
the environment), Actions it can take, and Rewards to give
him according to how it performs.
Genetic algorithm needs to have intrinsic parameters (could be caracteristics of the network taking the decisions, or more simple things like leg size / muscle strenght if you want to make an IA that runs), how to merge 2 parents to create children, and a Metric to evaluate how the network performed.

I assume your snake is make of a grid (like old snake where you only have 4 directions, you can't go in diagonal directions).
In snake example, here is how I would define it in both case (careful, this is how I would model the problem, there might be more efficient models) :
Q-Learning

State : what is in each block of the grid (snake tail, snake head, food)
Actions : Up, Down, Right, Left
Rewards : Not dying : +0.01; dying : -10; eating piece of food: +1.

Genetic Algorithm

Intrinsic Parameters : Parameters of the network that takes decisions (inputs of the network could be whole state of the environment as described in Q-Table).
How to merge : ?
Metric : Lenght of the snake or time lived.

I can't find exactly how to make a genetic algorithm, but it should be possible.
Hope it helps. Q-Learning is usually easier so this is what I would use, but feel free to use whatever you want.


Test batch time is referenced in the paper on page 33, Section 3.6, Inference Time and in Table 6. When an example set is used, a portion of it is used for training and another portion is used for testing. In mini-batch configurations the mini batch sizes are also independently configurable between train and test phases in many implementations.

References

SSD: Single Shot MultiBox Detector, 2016, Wei Liu


After re-reading Jang's original (1993?) paper on ANFIS, I learned he recommended simply squaring the b-parameter to deal with the notion that b could be changed to a negative value when using back-propagation. While this solves the negative domain issue, the issue still remains that if b is a non-integer, then the bell function loses its intended shape. I suppose after squaring one could round to the nearest integer, but I suspect that doing so might hinder convergence. Perhaps a consideration is only tuning the a and c parameters using something like a genetic algorithm.


RL: What should be the output of the NN for an agent trying to learn how to play a game?

It depends on whether the action is part of the input or output of a neural network estimating the Q-value(state, action).

The network on the left has the state as input and outputs one scalar value for each of the categorical actions. It has the advantage of being easy to setup and only needs one network evaluation to predict the Q-value for all actions. If the action space is categorical and single-dimensional I would use it.

The network on the right has both the state and a representation of the action as input and outputs one single scalar value. This architecture also allows to compute the Q-value for multi-dimensional and continuous action spaces.

The action space of tic-tac-toe can be easily represented by a vector of length 9, thus I would recommend the left NN-architecture. However, if your game has continuous-valued variables in the action space (e.g. the position of your mouse pointer), you should use the NN-architecture on the right.

The approach to prevent illegal moves is only partially dependent on the choice of the Q-function architecture and covered by another question: How to forbid actions


Is there any opensource 2d open-world simulation with python API?

You could try Mesa.

It has various examples that are commonly-used in agent-based modelling, like Epstein's model, a wolf/sheep predator/prey model, and many more.

There is also an introductory tutorial.


Typically the ramification of overfitting is poor performance on unseen data. If you're confident that overfitting on your dataset will not cause problems for situations not described by the dataset, or the dataset contains every possible scenario then overfitting may be good for the performance of the NN.


Possible model to use to find pixel locations of objects

from what I understand you are building you own model for this specific use case. From my perspective I would try not to reinvent the wheel, as it is said, and use an already proven and working model such as the YOLOs (v1, v2 and v3).

YOLO does not tell you the center pixel of the image directly but it tells you the center cell, with respect to the predicted object, of a grid (which is built on top of the image) responsible for predicting each object. See on the left image how the grid is built on top of the input image, then YOLO computes a probability map, and then the bounding boxes are predicted from the cell in the object's center. I have highlighted which cells would be responsible for predicting each object in the rightmost image. (Image from YOLOv1 paper)

The grid can have different resolutions but if you make it equal to the image size, then you would have the center pixel of each object predicted. This is because YOLO predicts objects in each cell of the grid, so if the grid is equal to the image size, YOLO will predict objects in each pixel.

As an example, imagine you have an input image of $[H \times W] = [416 \times 416]$ then YOLO would compute a grid of $[S_1 \times S_2]=[52 \times 52]$ on top of it. And predict objects in the center cell of the $[S_1 \times S_2]$. So, if you tune YOLO for computing a grid such as $[S_1 \times S_2] = [H \times W]$, then YOLO would output objects prediction with respect to the image pixels, in other words, YOLO would predict bounding boxes centered in the image pixel on the object's center.

This is how I would proceed for this use case, I hope it helps you or at least give you some clues about how to proceed further! Cheers! :)

NOTE: I chose the image size and grid size with numbers I usually see at work. Specifically, using YOLOv3. In YOLOv3, for aninput image of $[H \times W] = [416 \times 416]$, 3 grids are built, with different resolutions (for predicting big and small objects), with the following grids sizes are: $[13 \times 13], [26 \times 26], [52 \times 52]$


DQN Breakout adding an extra negative reward to help training?

In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called Reward Shaping.

Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as part of the state. So from its perspective, a negative reward for "timing out" actually looks like a random chance of getting that same reward in the kind of states that happen later in the game. The effect may be small enough chance that it doesn't really matter (after all it still gets positive rewards for hitting bricks). However, it may appear as high variance on certain game positions, making the agent learn to avoid some of them for no good reason.

If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from $0.999$ to $0.99$). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.

If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't need to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always $0$ anyway.

If you have more time, then there are also several extensions to basic DQN that require less sampled experience to learn an environment, or improve the resulting policies. A good source for those ideas is the paper Rainbow: Combining Improvements in Deep Reinforcement Learning.


How can I learn a graph given nodes with features in a supervised fashion?

It's perfectly reasonable to apply 'traditional' Deep Learning approaches to try and learn an adjacency matrix (a matrix is just a vector of vectors, which can be flattened into a single output vector) but you might need a lot of training data as N gets larger.

Your outputs could certainly have the form of an adjacency matrix, as you describe. Whether it's more useful to have 'boolean' (either 0 or 1)
or 'probabalistic' entries in the matrix depends both on the data and the specifics of your end application.


Can an AI learn how to play chess without instructions?

It's possible for an AI to learn chess without even knowing how to move the pieces. Google's AlphaZero didn't do that as their programmers coded the chess rules, but it's possible.

One can learn the rules from human played chess games. Once the rules are known, we could use reinforcement learning to improve playing strength (and other board games).


What is meant by degrees of freedom of latent variables?

A good example is the degree of freedom in Student's distribution:

 The degrees of freedom refers to the number of independent observations in a set of data.

For example:

When estimating a mean score or a proportion from a single sample, the number of independent observations is equal to the sample size minus one.

e.g, if we have 100 observation $X_1, \ldots, X_{100}$ and we want to estimate their mean $\bar{X}$, as $\bar{X} = \frac{X_1 + \cdots + X_{100}}{100}$, if we know the mean, just we need to find the value of 99 variables from $X_1, \ldots, X_{100}$. Hence, here the degree of freedom is 99.
Your referenced paragraph is a general explanation in the paper as well. However, base on the above example, the degree of freedom in the paragraph depends on the likelihood function and number of observations that we have from spectrograms.
Now, as the DoF of latent variables is not high, using Gibbs sampling we will approximate some observations, and then using them we will compute the value of the latent variables.


Are neural networks the only way to reach "true" artificial intelligence?

If by true AI, you mean 'like human beings', the answer is - no-one knows what the appropriate computational mechanisms (neural or otherwise) are or indeed whether we are capable of constructing them.

What Artificial Neural Nets (ANNs) do is essentially 'nonlinear regression' - perhaps this is not a sufficiently strong model to express humanlike behaviour.

Despite the 'Universal function approximation' property of ANNs, what if human intelligence depends on some as-yet-unguessed mechanism of the physical world?

With respect to your question about "the only way":
Even if (physical) neural mechanisms somehow actually were the only route to intelligence (e.g. via Penrose's quantum microtubules), how could that be proved?

Even in the formal world of mathematics, there's a saying that "Proofs of non-existence are hard". It scarcely seems conceivable that, in the physical world, it would be possible to demonstrate that intelligence could not arise by any other mechanism.

Moving back to computational systems, note that Stephen Wolfram made the interesting observation in his book 'A New Kind of Science' that many of the apparently distinct mechanisms he observed seem to be capable of 'Universal Computation', so in that sense there's nothing very particular about ANNs.


How is a neural network where the majority of inputs are 0 trained?

It's the loss function.

I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why:

From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But this only occurs if the output nodes are actually trained when their desired outputs are 0, which is what happens in the case of the squared sum/mean error. However, in the case of cross-entropy loss, which is explained excellently here, you can see that the only node that receives gradients is the node that should be trained towards 1. This removes the massive bias towards 0, and punishes confident false positives, making it perfect for a classification problem.

As to how you would achieve something like this for regression I do not know, but at least this solves the issue for classification problems.


When does the selection phase exactly end in MCTS?

If we choose to only expand a single node, the nodes that were not expanded will never be considered in future selections as we only select child nodes until a leaf is reached during the Selection Phase. Is this correct?

No this is not correct.

  If not, what am I misunderstanding about the Selection Phase?

The selection phase does not end only when you reach a node that has no expanded nodes. It ends when you reach a node that has any unexpanded nodes. At which point you typically pick one or more nodes you have not yet expanded at that point in the tree, expand them and collect one or more rollout results for them. Variations are possible such as choosing to whether to expand or continue selecting stochastically, or expanding all child nodes at the same time using value estimates to initialise them - the latter is what AlphaZero does.


Are Relational DBs and SQL used in Expert Systems?

A recent research example is the "Grind" system. Take a look at the paper Computing FO-Rewritings in $\mathcal{E} \mathcal{L}$ in Practice: from Atomic to Conjunctive Queries (2018) by Peter Hansen and Carsten Lutz. Here's the abstract.

A prominent approach to implementing ontology-mediated queries (OMQs) is to rewrite into a first-order query, which is then executed using a conventional SQL database system. We consider the case where the ontology is formulated in the description logic $\mathcal{E} \mathcal{L}$ and the actual query is a conjunctive query and show that rewritings of such OMQs can be efficiently computed in practice, in a sound and complete way. Our approach combines a reduction with a decomposed backwards chaining algorithm for OMQs that are based on the simpler atomic queries, also illuminating the relationship between first-order rewritings of OMQs based on conjunctive and on atomic queries. Experiments with real-world ontologies show promising results.


Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b

Reinforcement Learning is a method for learning to perform beneficial actions in an environment. One way this is accomplished is by learning to predict useful actions as a function of the observed state of the environment. Another is by learning to predict the expected utility gain of doing an action in a particular observed state. Usually the fact that the agent executes a sequence of actions is exploited to learn more rapidly.

In contrast, the problem you describe is to solve a linear system of equations, which is to say, to learn some hidden values $x$ such that $Ax = b$ for known $A$ and $b$. Gradient decent is a natural way to solve this problem because it is easy to calculate the gradient, and, since the matrices and vectors have the same dimensionality, it is reasonable to expect that the optimization surface is smooth with a single global minima.

While RL techniques could be applied to solve this problem (I guess by predicting values of X in response to inputs b?), none of the usual features that privilege the RL approach over a standard supervised learning approach are present (there's no sequential relationship between datapoints, there's just a list of constraints essentially).


Is a subset of a problem solution, used to recreate complete solution considered a heuristic?

Whether or not a label fits any particular instance depends on what you're using the label for. If something specific is riding on whether this approach is a 'heuristic' or not, that context is important.

But I wouldn't call this a heuristic, because I think of that as a shortcut for solving a problem, not either storing a solution or reformulating the problem (which is how I'd think of this).


Firstly, before we commence I will recommend that you refer to similar questions on the network i.e https://stackoverflow.com/questions/39386936/machine-learning-with-incomplete-data , https://stats.stackexchange.com/questions/103500/machine-learning-algorithms-to-handle-missing-data

Row Deletion

If a particular row has more than 70% missing values, you can delete the row to handle the null values. This method is advised only when there are enough samples in the data set. The major disadvantage of this method is that it reduces the power of the model because it reduces the sample size.

Replacing With Mean/Median/Mode

We can calculate the mean, median or mode of the feature and replace the missing values with it. Another approach is to approximate it with the deviation of neighbouring values.

Although this approach adds variance to the data set, it yields better results compared to removing rows and columns.

KNN or Random Forest imputation

In this approach, the missing values of an attribute are imputed using existing attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.

The advantage of this approach is that k-nearest neighbour can predict both qualitative and quantitative attributes. Additionally you do not need to create a prediction model for each attribute with missing data in the dataset.

Predicting the Missing Values

Prediction is one of the more sophisticated methods for handling missing data.
Using the features which do not have missing values, we can predict the null values with the help of a machine learning algorithm.

In this case, we divide our data set into two. One set with no missing values and another set with missing values. The first data set becomes the training data set of the model while the second data set with missing values is the test data set.

We then create a model to predict target variables based on other attributes of the training data set and populate the missing values of the test data set.(Sayali S 2016)

Caret or randomForestSRC packages in R

The R package randomForestSRC can handle missing data for a wide class of analyses i.e. regression, classification, unsupervised and multivariate (Ankur C 2014).
Additionally, the Caret R package can be used to predict missing data.

Reference : https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/


What is meant by the number of channels of a network?

It is exactly that - the number of color channels or any other analogue to color that you use.


Can alpha-beta pruning be used for applications apart from games?

Thinking about this more, the answer is in fact yes, but not for the application you mention.

You cannot use alpha-beta pruning to learn a model to predict customer outcomes, because it is only useful for domains where you are concerned about an adversary. In finding a customer model, there is no reason to worry about someone coming in and forcing you to make bad decisions about the optimization of the model. Consequentially, there is no reason to use minimax search, and thus, to use alpha-beta pruning.

There are applications other than (video) games where you could use these techniques though. For example, there are security games. In these "games" we want to use AI to find a strategy to protect an airport. It is reasonable to try and design our model under the assumption that someone else wants to break it. You could use Alpha-Beta pruning here (although in practice, more sophisticated algorithms are used).


What language is the GPT-3 engine written in?

The GPT-3 source code hasn’t been released but the creators say it uses the “same model and architecture as GPT-2” (source) with some exceptions.
The GPT-2 source code is written in 100% Python. The model is based on Tensorflow and NumPy which are written using C and C++. My best guess is that GPT-3 is also written in Python using libraries based on C.


Is fuzzy logic connected to neural networks?

They are unrelated.

There is a possibility of interpreting fuzzy values as probabilities, but strictly speaking they are different: fuzzy values are vague, while probabilities reflect likelihood (see Wikipedia entry for Fuzzy Logic)

While rolling a particular number on a six-side die has a probability of $1 \over 6$, a roll can actually only ever have one outcome.

A fuzzy value "quite old" can simultaneously be member of a number of fuzzy sets with different degrees of membership, eg "young" with 0.001, "adolescent" with 0.1, "old" with 0.4, "ancient" with 0.7. Unless it is "defuzzified", it is simultaneously contained in all the sets.

Defuzzyfication is a way of interpreting the result of a series of fuzzy operations and finding the set that best matches, but it is not a clearly defined process such as picking a random number according to a set of probabilities (or rolling the die).

I am not sure that the sum of all fuzzy set membership values of any given fuzzy value has to add up to 1.0; whereas this condition has to hold for probabilities.

[EDIT: to clarify - probabilities are not a set; I refer here to all possible outcomes of a random event which have a certain probability of being realised. The sum of all possible event probabilities has to be 1.0]

One alternative interpretation for your application could be the confidence that the input set is identical to the training set. Which could be a fuzzy value if you wanted to do something else with it, eg by combining it with other fuzzy variables.


What algorithms are considered reinforcement learning algorithms?

The dynamic programming algorithms (like policy iteration and value iteration) are often presented in the context of reinforcement learning (in particular, in the book Reinforcement Learning: An Introduction by Barto and Sutton) because they are very related to reinforcement learning algorithms, like $Q$-learning. They are all based on the assumption that the environment can be modelled as an MDP.
However, dynamic programming algorithms require that the transition model and reward functions of the underlying MDP are known. Hence, they are often referred to as planning algorithms, because they can be used to find a policy (which can be thought of as plan) given the "dynamics" of the environment (which is represented by the MDP). They just exploit the given "physical rules" of the environment, in order to find a policy. This "exploitation" is referred to as a planning algorithm.
On the other hand, $Q$-learning and similar algorithms do not require that the MDP is known. They attempt to find a policy (or value function) by interacting with the environment. They eventually infer the "dynamics" of the underlying MDP from experience (that is, the interaction with the environment).
If the MDP is not given, the problem is often referred to as the full reinforcement learning problem. So, algorithms like $Q$-learning or SARSA are often considered reinforcement learning algorithms. The dynamic programming algorithms (like policy iteration) do not solve the "full RL problem", hence they are not always considered RL algorithms, but just planning algorithms.
There are several categories of RL algorithms. There are temporal-difference, Monte-Carlo, actor-critic, model-free, model-based, on-policy, off-policy, prediction, control, policy-based or value-based algorithms. These categories can overlap. For example, $Q$-learning is a temporal-difference (TD), model-free, off-policy, control and value-based algorithm: it is based on an temporal-difference (TD) update rule, it doesn't use a model of the environment (model-free), it uses a behavioural policy that is different than the policy it learns (off-policy), it is used to find a policy (control) and it attempts to approximate a value function rather than directly the policy (value-based).


Are 1D CNNs really the appropriate model for human activity recognition?

Human activity recognition, for example as done with the Sports-1M Dataset, is a classification task. Given a video, say which activity / action was executed.

So it is a special case about video classification.

  human activities like swinging the arm are sequential actions and they require LSTMs

This is wrong. As a simple example, think of handwriting recognition as done on write-math.com: I receive a sequence of points (x, y) -- the pen-tip --  and classify which symbol is written. So it is a sequence, but still there a plenty of ways to use a simple multi-layer perceptron (MLP) for it. Two big groups of approaches:

Feature-Engineering: Instead of giving an arbitrary number of coordinates, one can normalize it to a fixed amount. This is what is currently done on write-math.com and works pretty well (see my Bachelors thesis)
Space Transformation: Similar to the feature engineering, but more extreme. Instead of looking at (variants of) coordinates, one can look at the image which is there in the end. This way one can apply CNNs.

You can do similar things for movies:

Multiple Classifications: Classify based on single frames and average the results
Feature-Engineering: Normalize the length to a fixed number; interpret the fixed-length video as an image with more channels.
Use CNNs as feature extractors / for dimensionality reduction and LSTM for the sequence classification.

You might be interested in Beyond Short Snippets: Deep Networks for Video Classification

Efficiency vs Effectiveness

Model A is more effective than model B, if its optimizarion criterion is better.

Model A is more efficient than model B, if it uses less resources (e.g. memory) to achieve the same optimization value (e.g. same accuracy / MSE)


What is the process of inventing deep neural network models? How researchers deal with long training times?

Part of the answer is that new architectures are often variants of existing architectures. There are some rough heuristics people follow (e.g. using powers of 2 in layer sizes, changing layer sizes according to a schedule, adding normalization layers, etc.). However, even when building off an existing architecture, there can still be a lot of toying around with different configurations and hyperparameters to require copious training time.
Researchers working for a company will likely have the luxury of some serious compute, e.g. GPU clusters or cloud services like AWS. This can really make a huge difference over training a model on your PC GPU. Further, many models can be trained at once. Many academic institutions also employ cluster computing on dedicated GPU clusters - for example, here at the University of Utah our department has a cluster with 8 really excellent GPUs.
One last thing I will note, is that it's not always necessary to train the model to completion on the full training set to get a sense of which network configuration and hyperparameter settings are working better than others. If I have a massive dataset that will take days to train on, I may take a smaller subset of the training data and do some quick comparisons between different models to get a feel for what changes have the greatest impact on test performance. It's not completely reliable, but it can guide intuition to narrow down the number of models you want to train on the full dataset to convergence. For example, using this technique I was able to find that predictions from my model were fairly invariant to the exact layer sizes I was using and was far more sensitive to a hyperparameter in my optimization solver.


The Hidden Agenda User Simulation Model (Schatzmann/Young) describes a chatbot training design in which a user simulator assembles and executes a conversational agenda, with the direct goal being to train the target chatbot.

Perhaps you can add specificity to this design by casting the user simulator as the teacher, and creating an agenda in which it is communicating information to the (student) chatbot. The trained behavior expected is, perhaps, correct responses to topical questioning by the teacher.


How to reproduce neural network training with keras

As @codeblooded said, you should set random seed for numpy and keras, and also set pythonhashseed. The seeds set the state of the random number generator which makes the results different.

This method only works when you train the network on CPU. The problem with getting same result on GPU every single time is that cuDNN is not deterministic. Specifically, the reduce_sum() function in tensorflow is known to have different results even if a random seed is set. You can see this for more details. But simply said, reduce sum is not deterministic in tensorflow, but however it is used in bias gradient calculation. This can be avoided by using matmul in place of reduce sum, and you can see the implementation in the article mentioned.

To get reproducible results, you can try using different framework, like pytorch. I have done experiments on pytorch and it seems to be reproducible. See this colab notebook. All you need to do is add a couple lines to the code. Theano based keras seems to work as well but I haven't tested it.

However for hyperparameter testing and searching 0.3% should not affect the result, and if you really want very accurate result, average the accuracy of 30 or more tries to get an accurate result. If you do this you don't need to set the random seed.


How to create and train (with mutation and selection) a neural network to predict the next state of a board?

Re: Should I also have it randomly delete nodes and connections, in case they also make improvements?
You may read about "dropout". In this case it's not preferred since overfitting is actually a benefit.

Re: Should the nodes in the hidden layer perform a logical AND on all of their inputs or an OR?
There are no logical operations in Neural Networks. You need to multiply the input by the weights. At the end you will arrive to a fraction number that you need to convert back to 1 or 0, here I suggest to look into "softmax" that is an activation function exactly for this.

Re:how do I know if it will ever get above 90% accuracy, or even just 50%?
As Jaden suggested you will need a benchmark. You may simply take my results from your other post. (i.e 100% in 20 epochs)

I agree with Jaden, you have a lot on your table. Perhaps it would make sense to first develop a DNN to learn the GOL, you will need the activations, the loss calculation and at least one optimizer. Once this is tested and working you could start with the second model for the genetic algorithm that will need a different loss function and also the genetic operations, population selection etc. Nice big project, good luck!


Validation Accuracy remains constant while training VGG?

Ok, I solved this problem
The simple thing was that learning rate was too big
I changed the code to this
LR = batch_size/((z+1)*100000)
LR=LR/3

instead of
LR = batch_size/((z+1)*1000)
LR=LR/3

and it seems to work well


How to train an LSTM with varying length input?

You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the pytorch LSTM implementation).
For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.


Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?

It will depend on the goal of the work:

If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.

Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.

Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.

For comparison with other methods for sample efficiency whilst learning and measuring regret (i.e. how much the exploration is costing you).

When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.

If you do really have an ideal agent, then of course you could just stop and say "job done". In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.


Feeding YOLOv4 image data into LSTM layer?

You can save the extracted features coming out of YOLOv4 and save it to pickle file and use them for later.
You can also find related information in this project made by Jason Brownlee
https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/


What are some resources on continuous state and action spaces MDPs for reinforcement learning?

There is a small survey of continuous states, actions and time in reinforcement learning in my thesis proposal.

Regarding books, Reinforcement Learning: State-of-the-Art seems to be pretty up-to-date from the excerpts I've read.


Can a crossover result in a node with no outgoing connections?

The disabled/enabled bit in a connection gene indicates whether or not it should be expressed in the calculation of the network.

Here's an example:

This is a neural network and its corresponding connection genes which represent the layout of the network among other things.
The top connection gene going from 1 -> 3 that has a weight of 0.1 is expressed in the calculation of the network.
The bottom connection gene going from 2 -> 3 that has a weight of 0.4 is not expressed in the calculation of the network.

Calculating the network:

  1633 * 0.1 = 163.3

  Given the weights in the example the output of this network is 163.3

Hypothetically, if both connection genes had their enabled bit set to true (which can happen in the future, to quote the paper: "The disabled genes may become enabled again in future generations") then the output of the network would be:

  (1633 * 0.1) + (10 * 0.4) = 167.3

To answer your question, the connection between two nodes still exists in the network regardless of whether the bit in the gene is enabled or disabled, but, if the bit in the gene is disabled it is not used when the output of the network is being calculated as I've shown above.


Is it possible to control asymptotic behaviour of neural network models?

For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.
Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on two novel technical constructs:

A Multi-Dimensional Spline Interpolator with Prescribed Asymptotic Behavior
and A Custom NN layer that guarantees zero asymptotics in chosen directions.

Let $f(x)$  be the Function that we want to approximate, while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function $S(x)$ that has the same asymptotics as $f(x)$.

Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions, or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function $S(x)$.

For step one, they show how to construct a universal control variate, a multi-dimensional spline $S(x)$ that has the same asymptotics as $f(x)$; and that can be used in all situations. In specific applications, of course, if more is known about the function $f$; a better choice of the control variate could be available.

For step two, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function $R(x) = S(x) - f(x)$ with a special NN that has vanishing (zero) asymptotics in all, or some, directions.

One-Dimensional Spline with Asymptotics Control:
The paper discussed that Instead of using the natural boundary conditions, i.e. setting second derivatives
to zero at the boundaries, they have fixed the first derivatives at boundary points to
arbitrary values. $\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$.
Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. $f(x) \simeq f_{−}(x)$ for $x<h_0$ and $f(x) \simeq f_{+}(x)$  for $x>h_{N+1}$
for large negative $h_0$ and large positive $h_{N+1}$. First, we
expand the set of spline nodes to include points $h_0$ and $h_{N+1}$ and make sure
the spline passes through them, $S(h_0) = f_{−}(h_0)$ and $S(h_{N+1}) = f_{+}(h_{N+1})$. To
finish, they have specifies first order derivatives at these new boundary points.
$$S^{\prime}_0(h_0) = f^{\prime}_{−}(h_0), S(h_{N+1}) = f^{\prime}_{+}(h_{N+1})$$
This fully specifies the approximating function $\tilde{S}(x)$,
$$\tilde{S}(x)=\left\{\begin{array}{ll}f_{-}(x), & x \leq h_{0} \\ S(x), & h_{0}<x<h_{N+1} \\ f_{+}(x), & h_{N+1} \leq x\end{array}\right.$$
One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives
where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between $h_0$ and $h_1$, say $h_{\frac{1}{2}}$ , and find, analytically, the value $S(h_{\frac{1}{2}})$ such that the second derivative at $h_{0}, S^{\prime \prime}\left(h_{0}\right)= f_{-}^{\prime \prime}\left(h_{0}\right)$ is matched.

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems. This paper deals with studying the asymptotical properties of multilayer neural networks models used for the adaptive identification of wide class of nonlinearly parameterized systems in stochastic environment.

A study of the asymptotic behavior of neural networks. where they have studied of neural networks modeled as a set of nonlinear differential equations of the form $TX+X=Wf(X)+b$, where $X$ is the neural membrane potential vector, $W$ is the network connectivity matrix, and $F(X)$ is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity $W$ need not be symmetric. The simulated behavior of typical neural networks is presented

Refrence
Neural Networks with Asymptotics Control
Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems
A study of the asymptotic behavior of neural networks


Doesn't the number of explored nodes with IDA* increase linearly?

No misunderstanding. IDA* is indeed n^2 vs n for A* in certain situations.
See
https://pdfs.semanticscholar.org/388c/0a934934a9e60da1c22c050566dbcd995702.pdf
for a reference.
IDA* only works in certain (not uncommon) situations. If there are many states with the same costs, and this number grows exponentially with the cost, then IDA* will be O(n). An example of such a problem is the 8 sliding piece puzzle.
IDA* will however, as noted, become exponential when all costs differ. Examples include continuous costs, or discrete values spanning a large range.
If you really wanted to apply IDA* to e.g. a map, a workable approach would likely be to round all values - or perform a log+floor operation.


Can someone direct me to a sites and/or videos that can bring an absolute beginner up to speed with AI?

I think the best approach would be to pick up a good book that lays out the fundamentals very well.

Artificial Intelligence: A Modern Approach (AIMA) is a great place to begin with! It is used as a reference textbook for many university level AI courses. It has helped me quite a lot as a student.


Markov property in maze solving problem in reinforcement learning

Hi Hunnam and welcome to our community!

  By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states.

No this is not exactly correct.
We can use RL to solve problems with the Markov Property exactly because the current state is a sufficient statistic of the future. In other words, the state encodes the distribution of future states.

Note that the state isn't necessarily the observations. As you point out in the next paragraph:

  However, I saw that in some case we can define a state to be the history of observations and actions taken so far, such as =ℎ=11…−1−1 .

At times we can use the history to represent the state. The history can be a series of observations.

  I think maze solving can be of that case since the current state, or the current place in a maze, clearly depends on which places the agent has been and which ways the agent has taken so far.

This isn't correct in the general case. Given a maze which you know how to solve, regardless of where you start, you know how to reach the exit. This is the markov property. Given the current position, you have enough information to make a certain and optimal decision.

Perhaps an example of a situation where the history is necessary will help illustrate the differences.

Suppose you are playing Pong. If you take a single frame, it doesn't contain enough information to know the direction of the ball. Therefore the observations alone are insufficient. What if you remember the previous frame? Then combining the two observations gives you all the information you need in order to make an optimal move.


Feasibility of an AI assistant to expedite game development?

In the history of programming the productivity of the programmer has increased. Early MS-DOS base games were programmed in Pascal and Assembly language direct for the CGA graphics card adapter. With the upraising of the C/C++ language and standardized operating systems, it become possible to use software libraries. (1) In early MS-DOS period, a team of programmers was needed for a simple jump'n'run game, while nowadays a single programmer can create such a game with existing game-engines in one weekend.

It is obvious to think this development forward and imagine a much advanced technique which support automatic programming. Creating games in a point&click fashion is possible with so called game construction sets. That are games, which comes with a level editor and a Lua scripting engine for modification of existing content. This is not directly related to Artificial Intelligence but goes into the same direction. The idea is, to use computer programs for increase the productivity. In the context of programming such AI-related support is hard to implement. Because programming and game-design contains lots of domain specific knowledge which has to be formalized. If an AI system should support the programming itself, the AI has to parse existing Stackoverflow threads and give hints to the programmer how to implements this in his software. So called “agent-based software engineering” is researched since the 2000s.

The most advanced example in semi-autonomous game design is perhaps the RPG adventure generator described in Barros, Gabriella Alves Bulhoes, et al. "Who killed albert einstein? From open data to murder mystery games." IEEE Transactions on Games (2018) It can parse Wikipedia article to generate a playable game from scratch.


Why don't we use trigonometric functions for the output neurons?

The main reason why the sigmoid function is used is because it 'does not blow up' since it is between 0 and 1 always. As for the relu it is used because it is computationally cheap and even resolves the problem of vanishing gradient(and hence used more often than sigmoid).
So a function like tan(x) will blow up for certain values of x. This can cause a problem of exploding gradients. So, I believe for this reason tan(x) cannot be a good non-linearity to be used.
As for any other function, it is more because of the results that we have gotten over the years and sigmoid and relu have been promising.


To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.


From comments, you say there is no "outer" goal for picking an adversary other than  scoring highly in an individual episode.
You could potentially model the initial adversary choice as a partially separate Markov Decision Process (MDP), where choosing the opponent is a single-step episode with return equal to whatever reward the secondary MDP - which played the game - obtains. However, this "outer" MDP is not much of an MDP at all, it is more like a contextual bandit. In addition, the performance of the inner game-playing agent will vary both with the choice of opponent, and over time as it learns to play better against each opponent. This makes the outer MDP non-stationary. It also requires the inner MDP to know what opponent it is facing in order to correctly predict correct choices and/or future rewards.
That last part - the need for any "inner" agent to be aware of the opponent it is playing against - is likely to be necessary whatever structure you chooose. That choice of opponent needs to be part of the state for this inner agent, because it will have an impact on likely future rewards. A characterisation of the opponents also needs to be part of whatever predictive models you could use for the outer agent.
A more natural, and probably more useful, MDP model for your problem is to have a single MDP where the first action $a_0$ is to select the opponent. This matches the language you use to describe the problem, and resolves your issue about trying to run a hierarchy of agents. Hierarchical reinforcement learning is a real thing, and very interesting for solving problems which can be broken down into meaningful sub-goals that an agent could discover autonomously, but it does not appear to apply for your problem.
This leaves you with a practical problem of creating a model that can switch between choosing between two sets of radically different actions. The select an opponent action only occurs at the first state of the game, and the two sets of actions do not overlap at all. However, in terms of the theoretical MDP model this is not an issue at all. It is only a practical issue of how you get to fit your Q function approximator to the two radically different action types. There a a few ways around that. Here are a couple that might work for you:
One shared network
Always predict for all kinds of action choice, so the agent still makes predictions for switching opponents all the way to the end of the game. Then filter the action choices down to only those available at any time step. When $t=0$ only use the predictions for actions for selecting an opponent, for $t \ge 1$ only use predictions relating to moves in the game.
Two separate approximators
Have two function approximators in your agent, use one for predicting reward at $t=0$ that covers different opponent choices, and use the other for the rest of the game. If $n$ is small and there is no generalisation between opponents (i.e. no opponent "stats" that give some kind of clue towards the end results), then for the first approximator, you could even use a Q table.
For update steps you need to know whether any particular action value was modelled in one or other of the Q functions - and this will naturally lead you to bootstrap
$$\hat{q}_{o}(s_0, a_0, \theta_0) \leftarrow r_1 + \gamma \text{max}_{a'}\hat{q}_{p}(s_1, a', \theta_1)$$
where $\hat{q}_{o}$ is your approximate model for action values of selecting opponents (and $a_0$ must be an opponent choice) at the start of the game, and $\hat{q}_{p}$ is the nodel you use for the rest of it (and $a'$ must be a position play in the game). I've misused $\leftarrow$ here to stand in for whatever process used to update the action value towards the new estimate - a tabular method that would be a rolling average with current estimate, in neural networks of course that is gradient descent using backpropagation.