How to recognize with just name and last name if the person is a political exposed person

But if they enter a name that is not in any of their data sources, how would I train a model to decide if its pep or not?

Based on just a person's name and nothing else, the accuracy of this model is going to be very low. Consider that most first name, surname combinations in Europe are going to be repeated across the population.

However, the accuracy might still be slightly better than guessing. Some families and social classes could be more likely to be involved in political work, and a statistical model would pick up on that.

To train the model, take your positive names, and combine with a random selection of names of people that are known to be "not political" or similar enough. It doesn't matter if some of the names are the same provided you are confident in your data. Probably you could just take a phone directory or the electoral register or some other list of general names. Provided your "political" people are a small fraction of all people, this will work well enough even if you have some of them in the negative class.

Ideally you mix those name groups in the rough proportion that the bank expects to see "political" and "non-political" customers, so that your data set is a good representation of the target population.

Then you train a classifier on the names. As this is text and sequence data, you will need a solution for that. Possibly LSTM would be suitable architecture, but so might some feature selection from the names in a more simple ML model.

Remember to hold back some data (both positive and negative cases) for cross-validating and testing the model.

Expect the accuracy of your model when testing to be low. Very low. I would not at all be surprised to find the end result unusable by itself.

If this is seriously to be part of some bank's account setup process, there needs to be additional data used later in the process. A gate for additional checks based purely on someone's name will perform very poorly in my opinion.


How to implement RAM versions of Atari games

To my knowledge from reading about model-based and model-free reinforcement learning,

DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):

https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1

You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."

Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.

Also there are a lot of states: there are (256^128) states. That is a really big number (I'm just emphasizing that training may take a long while)

Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf . I am assuming they used image data as well because Double DQN was compared to DQN.


Is there a way to understand neural networks without using the concept of brain?

tl;dr I always like to think of Neural Networks as a generalization of logistic regression.

I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks.

A perceptron is essentially a simple binary logistic regressor (if you threshold the output). If you have many perceptrons that share the same input (i.e. a layer in a neural network), you can think of it as a multi-class logistic regressor. Now, by stacking one such layer after an other, you create a Multi-Layer Perceptron (MLP), which is a Neural Network with two layers. There is equivalent to two multi-class logistic regressors stacked one after the other. One notable thing that changes is the training technique here, i.e. backpropagation (because you don't have direct access to the targets from the hidden layer). Another thing that can change is the activation function (it's not always sigmoid in Neural Networks)

Introduce sparse connectivity and weight sharing and you get a Convolutional Neural Network. Add a connection from a layer to its self (for the next timestep) and you get a Recurrent Neural Network. Likewise, you can reproduce any Neural Network through this reasoning.

I know this is an over-simplified way of presenting them, but I think you get the point.


How do I determine the most appropriate classifier for a certain problem?

This is one of the main skills that separates someone with a deep understanding of, and experience in, machine learning learning, from a neophyte. There are several approaches:

Try several methods, perhaps with automated hyperparameter optimization, and see if there's a big difference in typical model quality. This is pretty common if you don't have a lot of experience, but also something experts may try in a more targeted way.
Visualize the shape of your problem, perhaps by using a dimensionality reduction technique like PCA or tSNE, or maybe an auto-encoder. If you compress the data to 2d, are there clear linear patterns? Maybe try a linear model like logistic regression. Are there several distinct groups? How many lines would you need to draw to separate them? If it's a lot, maybe you're going to need a very non-linear model. If it's just a few, maybe a small multi-layer perceptron can help. If there are spiral bands or circular shapes, maybe and SVM with a non-linear kernel. Knowing how to translate the visualizations into intuition about the kinds of model that can help is an advanced skill. You need to understand what shapes of patterns each kind of model can learn to fit, and how these do or do not translate into a higher dimensional space.
Read the literature. If you're working in computer vision, you should try a CNN. Why? Well, everyone else is using them. They work great on most computer vision problems. They hold most competition records. It'd be silly not to try them. If you're working on spam classification though, CNNs are a bad choice. People use compression classifiers, Bayesian models, and sometimes multi-stage models to build complex features. If you look at the recent literature in your area, you can tell what to use. Reading and understanding this literature well enough to interpret it usually requires more advanced scientific training in ML and/or the application area.

As a final note, the No Free Lunch theorems for ML tell us that this is always going to be an art (at least, that's one interpretation experts argue for), and not a science, so the more practice you get with it, the better you'll become.


How to understand mapping function of kernel?

A kernel function $f : \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is a valid support vector kernel if it is a Mercer kernel. Mercer's condition essentially ensures that the Gram matrix of the kernel is positive semi-definite. Interestingly, this ensures that the SVM objective is convex.
The Euclidean distance function does not satisfy Mercer's condition since it's Gram matrix is not necessary positive semi-definite. Thus, it is not a valid kernel.


Could an AI be built to learn based of interaction with a human?

In principle, yes, what you are proposing can be done.  The exact details of how to do it are an open research question.  The details would also depend on exactly what your goals for the system are. If you're just trying to build some domain specific system that learns a very specific kind of knowledge, then that's probably going to be easier than building an AGI that learns like a child does.

What I will suggest, although this is not proven, is that building a really powerful system of this sort will probably require more than deep learning.  I would also caution anybody interested in AI against thinking that deep learning is the "be all, end all" of AI techniques. My guess is that doing this well will ultimately require a multi-agent system, maybe something like Minsky's "Society of Mind" approach, or a Blackboard model, with collaborating agents, each specialized for various aspects of intelligence.  My feeling is that you will, indeed, need deep learning for classification / pattern matching, but possibly also things like Case Based Reasoning, K-Lines, a Semantic Network, Rule Learning, BDI, and other techniques, working together.


Neural network for detecting individual human face

I am assuming that you are new to all this. You can start with making a basic human face detection. Train the program to detect a human face with very good accuracy. This will help you to get familiar with the coding ground related to image processing and basic machine learning.
After that train your program to identify faces of only 2-3 people. Trying for too many in the beginning won't be a good idea. Test your program's accuracy in different situation, with a different number of crowd, etc.
If it's working fine then you can train your program for more individuals. In addition to this leave room for learning from experience in your code. Some codes only learn once and they implement the same thing for their whole life. A nice example of this kind is OCR. If a face is detected wrongly OR your program detects a new face about which it doesn't know anything. Then you should be able to tell the program and it should include that in its database. I think some form of reinforcement learning will help. Not much sure about it though.
Now the Implementation
I will highly recommend you to learn python and get familiar with OpenCV. You can think of OpenCV as a collection of libraries. I find them very helpful for image processing and machine learning. Another good thing about it is that you can import OpenCV in python, Java or C++ according to your need.
OpenCV has an inbuilt function that allows it train a neural network for positive and negative images. The success of your program depend highly on your choice of positive and negative images, so choose them wisely. The result of the training is stored as a haar cascade file. This cascade file can be used in your program to use the trained data and function accordingly.
For basic human face detection, you can find the cascade file online and implement a code like this

  face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml');

For detecting individual faces you will need to make different classes. The number of classes will represent the number of individual faces you want to detect in the beginning.
You can find OpenCV tutorials here.


How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose.

The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:

Train a large, densely connected, network with a standard training
algorithm
Examine the trained network to assess the relative importance of the
weights
Remove the least important weight(s)
retrain the pruned network
Repeat steps 2-4 until satisfied

However, there are several optimized methods for pruning neural nets, and it is also a very active area of research.


Can models get 100% accuracy on solved games?

So can we perfectly learn a solved game as well?

The short answer is yes. If your model has enough complexity it can theoretically learn any behavior you want.

So in theory our model could learn tic-tac-toe

Tic Tac Toe has already been solved. Another popular game that has been solved is Checkers, by the algorithm Chinook.
To be more specific, in Reinforcement Learning we make the assumption that any decision making process can be modeled as an MDP (Markov Decision Process). Once there, there are a host of different methods like Q-Learning and TD that theoretically converge towards the optimal policy - the one the plays perfectly.
Now, just because it is theoretically possible doesn't mean it will always empirically work. In games that are very complex and have a large state space it is extremely difficult to perfectly solve. This is because the only feasible way to go about them is to approximate and getting perfect play even in small edge cases becomes much more difficult as a result.
If you want to learn more about this topic I would highly recommend this series RL Course by David Silver


What does an oscillating validation error curve represent?

Without more information the best diagnostic is:
You potentially have a bug in your code.
Justification
Neural network systems are capable of failing silently. The system can still appear to "learn" even in the presence of said bug - a while back I had a very similar issue with a toy CNN project. My network could get 99% accuracy on the training set but always achieved 8-13% on the validation set. This looked like overfit but none of the methods solved the issue. Finally, I found that I wasn't feeding the data correctly into the network during train time but I was feeding the data correctly during validation time.
Conclusion
Provide the following for better diagnostic:

Is this a custom CNN you hardcoded?
are you using python?
what's the objective function?
can you provide the loss curve for the training set?
show us some code too (this will be very helpful for working toward a solution)

I hope this helps, and best of luck!


Do you need single or multiple networks to detect multiple faces?

AFAIK, normally detection algorithms work in a sub-window of the image and not the whole of it. For example, for a specific size and orientation you slide a sub-window on the image and extract sub-images. Then you apply your algorithm on every sub-image for detection and report the size-and-orientations with positive results.

You can have a single neural network for face detection in this case or you might want to have different detectors for different orientation or any other feature, that is your decision.

There is also the technique of Combining Classifiers by which you can improve the decision of single classifiers by combining them.

Ensemble Learning is another way in which your classifiers are not trained independently but rather together. In fact, the well-known object detector of Viola and Jones uses such a technique.


What is machine learning?
Machine learning (ML) has been defined by multiple people in similar (or related) ways.
Tom Mitchell, in his book Machine Learning (1997), defines an ML algorithm/program (or machine learner) as follows.

A computer program is said to learn from experience $E$ with respect to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

This is a quite reasonable definition, given that it describes algorithms such as gradient descent, Q-learning, etc.
In his book Machine Learning: A Probabilistic Perspective (2012), Kevin P. Murphy defines the machine learning field/area as follows.

a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty (such as planning how to collect more data!)

Without referring to algorithms or the field, Shai Shalev-Shwartz and Shai Ben-David define machine learning as follows

The term machine learning refers to the automated detection of meaningful patterns in data.

In all these definitions, the core concept is data or experience. So, any algorithm that automatically detects patterns in data (of any form, such as textual, numerical, or categorical) to solve some task/problem (which often involves more data) is a (machine) learning algorithm.
The tricky part of this definition, which often causes a lot of misconceptions about what ML is or can do, is probably automatically: this does not mean that the learning algorithm is completely autonomous or independent from the human, given that the human, in most cases, still needs to define a performance measure (and other parameters, including the learning algorithm itself) that guides the learning algorithm towards a set of solutions to the problem being solved.
As a field, ML could be defined as the study and application of ML algorithms (as defined by Mitchell's definition).
Sub-categories
Murphy and many others often divide machine learning into three main sub-categories

supervised learning (or predictive), where the goal is to learn a mapping from inputs $\textbf{x}$ to outputs $y$, given a labeled set of input-output pairs

unsupervised learning (or descriptive), where the goal is to find "interesting patterns" in the data

reinforcement learning, which is useful for learning how to act or behave when given an occasional reward or punishment signals

However, there are many other possible sub-categories (or taxonomies) of machine learning techniques, such as

deep learning (i.e. the use of neural networks to approximate functions and related learning algorithms, such as gradient descent) or
probabilistic machine learning (machine learning techniques that provide uncertainty estimation)
weakly supervised learning (i.e. SL where labeling information may not be completely accurate)
online learning (i.e. learning from a single data point at a time rather than from a dataset of multiple data points)

These sub-categories can also be combined. For example, deep learning can be performed online or offline.
Related fields
There is also a related field known as computational (or statistical) learning theory, which concerned with the theory of learning (from a computational and statistical point of view). So, in this field, we are interested in questions like "How many samples do we need to approximately compute this function with a certain error?".
Of course, given that machine learning is a set of algorithms and techniques that are data- or experience-driven, one may wonder what the difference between machine learning and statistics is. In fact, in many cases, they are very similar and ML adopts many statistical concepts, and you may even read on the web that machine learning is just glorified statistics. ML and statistics often tackle the same problem, but from a different perspective or with slightly different approaches (and the terminology may slightly change from one field to the other). If you are interested in a more detailed explanation of their difference, you could read Statistics versus machine learning (2018) by Danilo Bzdok et al.
What is machine learning good for?
ML can potentially be used to (at least partially) automate tasks that involve data and pattern recognition, which were previously performed only by humans (e.g. translation from one human language, such as English, to another, such as Italian). However, machine learning cannot automate all tasks: for example, it cannot infer causal relations from the data (which often must be done by humans), unless you include causal inference as part of machine learning. If you are interested in causal inference, you could take a look at the paper Causal Inference by Judea Pearl (Turing Award for his work in causal inference!).


Convert LSTM univariate Autoencoder to multivariate Autoencoder

If each of your three features is a scalar then my first attempt would be to combine them into a vector for each step in the sequence. So instead of LSTM(128, input_shape=(30,1)) for a length-30 univariate sequence you would say LSTM(128, input_shape=(30,3)) for a multivariate (3) sequence. Similarly your output would become TimeDistributed(Dense(3, activation='linear')).
If your input features are each a vector then you should consider switching from the sequential API to the functional API. You would need three separate Input objects followed by a Concatenate layer before the LSTM layer. You would have three Dense layers to be your corresponding outputs. I tried adapting your code:
in0 = Input(shape=(trainX0.shape[1], trainX0.shape[2]))
in1 = Input(shape=(trainX1.shape[1], trainX1.shape[2]))
in2 = Input(shape=(trainX2.shape[1], trainX2.shape[2]))
concat = Concatenate()([in0, in1, in2])
lstm_enc = LSTM(128)(concat)
lstm_enc_drop = Dropout(rate=0.2)(lstm_enc)
rep = RepeatVector(trainX0.shape[1])(lstm_enc_drop)
lstm_dec = LSTM(128, return_sequences=True)(rep)
lstm_dec_drop = Dropout(rate=0.2)(lstm_dec)
out0 = TimeDistributed(Dense(trainX0.shape[2], activation='linear'))(lstm_dec_drop)
out1 = TimeDistributed(Dense(trainX1.shape[2], activation='linear'))(lstm_dec_drop)
out2 = TimeDistributed(Dense(trainX2.shape[2], activation='linear'))(lstm_dec_drop)
model = Model(inputs=[in0, in1, in2], outputs=[out0, out1, out2])
model.compile(optimizer='adam', loss='mae')
model.summary()


Is it necessarry to create theory/infrastructure to prevent people from creating AI incompatible with a safe model (if we create one)?

It seems impossible to prevent that. If someone can make a safe AI from scratch in the near future, then someone else can make a dangerous AI from scratch as well. If all that's needed is a computer (or eventually a robot) it will be really hard to stop people from creating one.

Banning computers? Maybe it could prevent it, but that comes with quite a few negative sides as well.

A law against creating AIs? Would be really hard to follow-up, and what about the AIs people want to make to be used for something "good"?

And even if we could come up with a good set of laws they probably wouldn't be introduced in all countries, and also be really hard to follow up.

I guess we just need to make our own defence mechanisms which can fight unsafe AIs (physically or virtually) when that time comes.


Why is the hyperbolic tangent with MSE better than the sigmoid with cross-entropy?

See the blog post Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training (2013) by James D. McCaffrey.

It should give you an intuition of why the average cross-entropy (ACE) is more appropriate than MSE (but MSE is also applicable).

In a few words, $\tanh$ + MSE is like sigmoid + MSE, but with labels for classes $-1$ and $1$ instead of $0$ and $1$. If you look at the shape of $\tanh$ function, it has the same flat tails where changes of the argument don't change the result.


Is it possible for value-based methods to learn stochastic policies?

Yes, but only in a limited sense, due to the ways it is possible to generate stochastic policies from a value function. For instance, the simplest exploratory policy used by SARSA and Monte Carlo Control, $\epsilon$-greedy, is stochastic.

SARSA natually learns the optimal $\epsilon$-greedy policy for any fixed value of $\epsilon$. That is not quite the same as learning the optimal policy, but might still be useful in a non-stationary environment where exploration is always required and the algorithm is forever learning online.

You can also use other functions to generate stochastic policies from value functions. For instance, sampling from the Boltzmann distribution over action values using a temperature parameter to decide relative priorities between actions with different action values.

However, all these approaches share the problem that they cannot converge towards an optimal stochastic policy. The policies are useful for mangaging exploration, but will only be optimal in the limited sense of optimal given the fixed policy generator or by chance. There is no way for a purely value-based method to learn a conversion from values to an optimal balance of probabilities for action choice.

For strict MDPs this is not an issue. If the MDP has the Markov property in the state representation, then there will always be a deterministic optimal policy, and value-based methods can converge towards it. That may include reducing $\epsilon$ in $\epsilon$-greedy approaches or the temperature in Gibbs sampling, when using an on-policy method.

  I started to think about settings where the optimal policy is stochastic(POMDP), and if it is possible to learn this policy for the "traditional" value-based methods

It isn't.

To resolve this you need to add some kind of policy function and a mechanism to search for better policies directly by modifying that function. Policy Gradient methods are one approach, but you could include genetic algorithms or other search methods too under this idea.

It may still be useful to use a value-based method as part of a policy search, to help evaluate changes to the policy. This is how Actor-Critic works.


Does fitness proportionate selection select multiple individuals?

Yes, you select the same individual multiple times, according to the distribution of fitness values. Typically, you have at least as many offspring as parents, so if you didn't replace parents in the pool to be potentially selected again, you'd just have every parent being selected once, which defeats the purpose of selection. You want fitter parents to be selected more often so that you drive the average fitness higher across generations.

That said, there are no iron clad rules. There are algorithms that create selection pressure in other ways, and you can do whatever works for you. But in the canonical version of a GA, you do selection with replacement. Though, many people find other selection schemes like tournament selection perform better across a pretty wide range of problems than roulette wheel anyway.


Analyzing vibration using machine learning

If you want to use machine learning for such a project, you can use vibrations data directly, and treat the problem as a regular audio classification problem.

A simple approach would be to use a Neural Network with Convolutions. This would take care of features extraction for you. And maybe follow these by dense layers at the end.

Given that, it would be easier to make suggestions if you posted samples of your data.

Edit: Also, keep in mind that machine learning usually requires large datasets so if you are collecting data all by yourself in the way you describe, you might not have enough sample to run a good model. In such a case when number of samples is limited one can use transfer learning - using a pre-trained model - but I am not aware of any such pre-trained models for wave data.


How can I perform lossless compression of images so that they can be stored to train a CNN?

Your input image size and memory are not directly related. While using CNN's, there are multiple hyperparameters that effect the video memory(if you are using GPU) or physical memory(if you are using CPU). All the frameworks these days uses a simplified data-loaders, for instance in Tensorflow or PyTorch, you are required to write a data-loader that takes in multiple hyper-parameters that are mentioned below and fit the data into VRAM/RAM, and this is strictly dependent upon you batch size - memory occupied on VRAM has direct relation to the batch size.

Whatever may be your image size, while you are writing the data-loader you have to mention the transformation parameters to your data-loader, during the training phase the data-loader will automatically load required images into your memory according to the batch size you have mentioned. As you have mentioned about image compression, this is an irrelevant parameter at-least for most of the generic use-cases, the most relevant hyperparameters are

Scaling
Cropping
Random flip
Normalization of the RGB values
ColorJitter
Padding
RandomAffine

And many more.

PyTorch provides really good transformers in data-loader, please do check https://pytorch.org/docs/stable/torchvision/transforms.html.

For Tensorflow, have a look at https://keras.io/preprocessing/image/.


Why do we use a last-in-first-out queue in depth-first search?

We use the LIFO  queue, i.e. stack, for implementation of the depth-first search algorithm because depth-first search always expands the deepest node in the current frontier of the search tree. The search proceeds immediately to the deepest level of the search tree, where the nodes have no successors. As those nodes are expanded, they are dropped from the frontier, so then the search "backs up" to the next deepest node that still has unexplored successors.

A LIFO queue means that the most recently generated node is chosen for expansion. This must be the deepest unexpanded node because it is one deeper than its parent, which, in turn, was the deepest unexpanded node when it was selected.


Why is this variable in equation 2 of the SQAIR paper a random vector of $n$ ones followed by a zero?

From my understanding of the paper, $Z^{pres}$ keeps track of the objects in the scene. For every step of the sequential inference, $z^{pres,i}$ takes either a 0 or a 1. A 1 represents that an object is present and it has to be explained by the remaining latent variables. 0 indicates that all the objects have been explained and inference is complete.


What type of neural network would be most feasible for playing a realtime game?

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays the game; Deep Reinforcement Learning: Pong from Pixels.

As far as the choice of layers go, again, you should read up on deep reinforcement learning, perhaps study some existing models that work, for a similar task and take their architecture as your starting point. The above link is a nice introduction into such thinking.

As a continuation, I would recomend you employ one of the well known, and developed, libraries, such as TensorFlow or pyTorch. They will allow you to run your models on GPU, like you desire.

For a toolkit, that specializes to reinforcement learning, see openAI gym.


What are the domains where SVMs are still state-of-the-art?

State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask when you might prefer to try an SVM.

SVMs have several advantages:

Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear combinations of features, rather than the original feature set. In contrast, a more modern approach like a deep neural network will need to get deeper or wider to model the same patterns, which will increase its training time.
SVMs have an inherent bias towards picking "conservative" hypotheses, that are less likely to overfit the data, because they try to find maximum margin hypotheses. In some sense, they "bake-in" Occam's razor.
SVMs have only two hyperparameters (the choice of kernel and the regularization constant), so they are very easy to tune to specific problems. It is usually sufficient to tune them by performing a simple grid-search through the parameter space, which can be done automatically.

SVMs also have some disadvantages:

SVMs have a runtime that scales cubically in the number of datapoints you want to train on (i.e. $O(n^3)$ runtime)1. This does not compare well with, say, a typical training approach for a deep neural network which runs in $O(w*n*e)$ time, where $n$ is the number of data points, $e$ is the number of training epochs, and $w$ is the number of weights in the network. Generally $w, e << n$.
To make use of the Kernel trick, SVMs cache a value for the kernelized "distance" between any two pairs of points. This means they need $O(n^2)$ memory. This is far, far, more trouble than the cubic runtime on most real-world sets. More than a few thousand datapoints will leave most modern servers thrashing, which increases effective runtime by several orders of magnitude. Together with point 1, this means SVMs will tend to become unworkably slow for sets beyond maybe 5,000-10,000 datapoints, at the upper limit.

All of these factors point to SVMs being relevant for exactly one use case: small datasets where the target pattern is thought, apriori, to be some regular, but highly non-linear, function of a large number of features. This use case actually arises fairly often. A recent example application where I found SVMs to be a natural approach was building predictive models for a target function that was known to be the result of interactions between pairs of features (specifically, communications between pairs of agents). An SVM with a quadratic kernel could therefore efficiently learn conservative, reasonable, guesses.

1 There are approximate algorithms that will solve the SVM faster than this, as noted in the other answers.


Is Deep Q Neural Network (DQN) applicable only with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?

No, DQN (that is the commonly-used abbreviation by the way, there also is a dqn tag which you may wish to use) is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kinds of inputs.

The DQN algorithm should be viewed separately from the Neural Network Architecture though. DQN can be used with any kind of Neural Network architecture. Yes, the most commonly-used type of architecture used with DQN is probably architectures that start with a bunch of convolutional layers, and those kinds of architectures are best suited for image-based inputs. This is not a requirement though. If you have other kinds of features where convolutional layers don't make a lot of sense, you can, for example, simply start directly with some ReLU layers.

Here are two examples of DQN being used for problems without image-based inputs from the OpenAI baselines repository:

DQN + Mountain Car
DQN + Cartpole

In both cases, they're using deepq.models.mlp() to construct a relatively straightforward Multi-Layered Perceptron architecture, without any convolutional layers. In this example for Atari games, they do have image-based input and therefore also construct an architecture with convolutional layers using deepq.models.cnn_to_mlp().

Note that, if you have relatively straightforward features, it may often not be necessary to use Deep RL approaches; tabular RL approaches of RL with Linear Function Approximation may work just as well in cases where you already have good features. If you have image-based inputs, those kinds of approaches are much less likely to work well. The disadvantage of Deep RL approaches like DQN is that they tend to require much more experience / data than simpler approaches.

So, generally the interesting question isn't "can DQN (or another Deep RL approach) handle my inputs?", because the answer is probably yes. The more important question would often be "do I have to use DQN (or another Deep RL approach?". The answer to that question will almost always be yes if you have image-based inputs, but relatively often be no if you already have good features as inputs.

Another class of problems where Deep RL is really popular nowadays is continuous control problems (e.g. robot simulators, MuJoCo, etc.). These don't have image-based inputs, but still generally require Deep RL (not DQN though; DQN doesn't handle continuous-valued outputs very well, it generates discrete outputs).


How does the decision tree implicitly do feature selection?

Consider a dataset $S \in \mathbb{R}^{N \times (M + 1)}$ with $N$ observations (or examples), where each observation $S_i \in \mathbb{R}^{M + 1}$ is composed of $M$ elements, one value for each of the $M$ features (or independent variables), $f_1, \dots f_M$, and the corresponding target value $t_i$.

A decision tree algorithm (DTA), such as the ID3 algorithm, constructs a tree, such that each internal node of this tree corresponds to one of the $M$ features, each edge corresponds to one value (or range of values) that such a feature can take on and each leaf node corresponds to a target. There are different ways of building this tree, based on different metrics to choose the features for each internal node and based on whether the problem is classification or regression (so based on whether the targets are classes or numeric values).

For example, let's assume that the features and the target are binary, so each $f_k$ can take on only one of two possible values, $f_{k} \in \{0, 1\}, \forall k$, and $t_i \in \{0, 1\}$ (where the index $i$ correspond to the $i$th observation, while the index $k$ correspond to the $k$th column or feature of $S$). In this case, a DTA first chooses (based on some metric, for example, the information gain) one of the $M$ features, for example, $f_j$, to associate it with the root node of the tree. Let's call this root node $f_j$. Then $f_j$ will have two branches, one for each of the binary values of $f_j$. If $f_j$ were a ternary variable, then the node corresponding to $f_j$ would have three branches, and so on. The DTA recursively chooses one of the remaining features for each node of the child branches. The DTA does this until all features have already been selected. In that case, we will have reached a leaf node, which will correspond to one value of the target variable. When the DTA chooses a feature for a node, all observations of the dataset $S$ that take on the first binary value of that feature will go in the branch corresponding to that value and all other observations will go in the other branch. So, in this way, the DTA splits the dataset based on the features.

The following diagram represents a final decision tree built by a DTA.

You can see that the first feature selected (for the root node) by the DTA is "Is it male?", which is a binary variable. If yes, then, on the left branch, we have another internal node, which corresponds to another feature and, at the same time, to all observations associated with a male. However, on the right branch, we have a leaf node, which corresponds to one value of the target, which, in this case, is a probability (or, equivalently, a numerical value in the range $[0, 1]$). The shape of the tree depends on the dataset and DTA algorithm. Therefore, different datasets and algorithms might result in different decision trees.

So, yes, you can view a decision tree algorithm as a feature selection or, more precisely, feature splitting algorithm.


Are there any rules for choosing batch size?

Data that has a size of a factor of $2$ (aka $2^n$ for some integer $n$) allows for easier memory management because data can be organized contiguously (without gaps). This allows for faster memory reading and thus faster iteration time in general. From a computational point of view, this is important as it can be taken advantage of by the compiler and speed up iteration loops. This is why batch sizes are chosen as such in practice. However, this does not necessarily imply better training results.
In regards to your question of "Can I use other values for batch size?":
Yes, you can use different values, and, for the most part, you probably won't see the difference in computational performance because of the speed of modern training APIs. So, until training a large model with a lot of compute where this optimization has a larger impact, feel free to experiment :)


What is an identity recurrent neural network?

An identity recurrent neural network (IRNN) is a vanilla recurrent neural network (as opposed to e.g. LSTMs) whose recurrent weight matrices are initialized with the identity matrix, the biases are initialized to zero, and the hidden units (or neurons) use the rectified linear unit (ReLU).

An IRNN can be trained more easily using gradient descent (as opposed to a vanilla RNN that is not an IRNN), given that it behaves similarly to an LSTM-based RNN, that is, an IRNN does not suffer (much) from the vanishing gradient problem.

The IRNN achieves a performance similar to LSTM-based RNNs in certain tasks, including the adding problem (a standard problem that is used to examine the power of recurrent models in learning long-term dependencies). In terms of architecture, the vanilla RNNs are much simpler than LSTM-based RNNs, so this is an advantage.

For more details, see the paper A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015), by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. See also this Keras implementation of the MNIST experiment described in the linked paper.


Finding anomaly detection by pattern matching in a set of continous data

Without knowing the kind of data and the process generating it it's hard to give a definite answer.
In general, I would attempt a network that has as inputs the actual sensor readings, and outputs the expected readings. You train this network by presenting data with errors added as inputs, and correct readings as outputs.  It should learn to guess the correct (non-anomalous) readings from a set of actual readings, and you can find the sensors with anomalies by taking the difference between actual and predicted readings.
Depending on the kind of anomalies you expect (misreadings such as fluke zero or max values, or actually anomalous states of the system being measured) your training data should be set up to have samples of such anomalies.
If there are temporal correlations (for example, temperature readings that change slowly) using a RNN might be helpful but its dimensioning heavily depends on the nature of your measured system.
External factors influencing the system (state of a heating mechanism, time of day/year, etc.) could be added at inputs for better predictions.
At the end of the day, trial and error is your friend. Start with a simple network and see how well it behaves. Go for more complex networks when you see where the limitations of the simple solution are.


How to train a model to predict the number of people at a certain bus stop before they cumulate in large numbers?

Maybe you can use a recurrent neural network on saved data to train a predictive model based on past data.


Are commercially available neural ICs digital?

Intel's Movidius uses digital circuitry for signal paths.   It is a parallel processing VLSI chip that can host a sequence of convolution layers and other types of parallel operations, just as CUDA cores in NVidia products do.  These all are extensions of the idea of DSPs (digital signal processors).  They are adept as front ends to real time vision systems in robots and vehicle automation.  They are also stuffed into racks and used for massively parallel operations in data centers, leveraged for large batch or mini-batch jobs related to indexing or supporting transaction based services.

There are three broad signal theory characteristics referenced conceptually in this question which may benefit from some disambiguation.

Analog versus digital
Deterministic versus stochastic
Predictable versus unpredictable outcome

Of the eight permutations, any of these five can be realized or reasonably accurately approximated.  Under what conditions they are repeatable is in parentheses.

Analog, deterministic, predictable outcome — Yes, within the bounds of specifications such as signal to noise ratio
Analog, deterministic, unpredictable outcome — Usually no
Analog, stochastic, unpredictable — No
Digital, deterministic, predictable outcome — Yes
Digital, stochastic, unpredictable — No

The commercially available ICs used for artificial neural networks mentioned above are digital, but they may not be deterministic.  For instance, the seed of random number generators can be the system time.  When embedded into larger computing systems entropy can also be collected digitally from mouse movement, microphones, camera frames, or secure network sources and then used in the digital signal processing algorithms, including artificial network ones.

Future chips may actually return to analog processing for some of the workload.  The accuracy of digital circuitry may be worth sacrificing in exchange for increase complexity and throughput per unit area on the VLSI substrate.  The storage and retrieval advantage of flip flops and other digital structures can still be maintained by interfacing the storage to the forward signal paths with A-to-D and D-to-A conversion. It is difficult to predict whether the trade-off will gain traction and, if so, when.

The term, "Balanced edge case," references what mathematicians familiar with chaos theory sometimes call initial input sensitivity, colloquially known as, "The Butterfly Effect."  Chaos can occur in nature where the only discrete operations are at the quantum level.  Chaos can also occur in digital systems with no greater mathematical complexity than the Logistic Equation: $x_{t+1} = r \, x_t (1 - x_t)$, where interesting behavior is observed for values of $r > 2.0$.

In electrical power engineering, much of what appears on an oscilloscope in triphase circuits are not sine waves but chaos that exhibits a strong peak in the power spectrum at 50 or 60 Hertz.


How is the error calculated with multiple output neurons in the neural network?

Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation
Edit:


Variational Autoencoder task for better feature extraction

I have not worked on this but I think I can give you a theoretical perspective of using VAE's. Regression is a Supervised Learning task and is basically a mapping from Input to Output where the Neural Net will approximate the function $f(input) = output$.

VAE's on the other hand are good for finding how a latent variable affects the output. For example, if you have a task of training on a persons facial emotions, and if your latent space contains 2 variables $z_1$ and $z_2$ then you might find varying $z_1$ varies the amount of smile on the face, while varying $z_2$ might give the amount of drooping of eyes. I suggest you check this video from Stanford at ~44:00 to see this actually happens or check this blog. So VAE's might have been useful if your output contained more features which would vary according to variations in latent variables, but a single scalar output can only tell you about the rate of effect on varying a latent variable.

But if your job is only for better regression, Auto-Encoders are the better alternative, since it has an inherent de-noising ability and sufficient training might help in de-noising the input, and thus provide better results if classified on the basis of latent variables.

An approach, which I think is kind of similar (to your thinking) has been proposed by Kingma, et al. for Semi Supervised learning in this paper. The paper has very poor description of the method so I would suggest you check out this blog. They have used an additional classifier for reconstruction of the original input and trained the classifier when labels are present.


What does "shape information" mean in terms of GAN(generative adversarial networks)?

"Shape information" refers to the outlines of objects in an image. It does not include information like color.

You can think of it as the output of an edge detector on a greyscale version of the original image


Why do we regularize the variational autoencoder with a normal distribution?

If you are mathematically inclined, here is an article that discusses the reasoning.

What I get as a take away is that the VAE forces the learned latent space to be Gaussian due to the KL divergence term in the loss function. So, now we have a known distribution to sample from to create input vectors to feed to the decoder, to produce say images of dogs, if the VAE was trained on images of dogs. As you sample from the distribution, you will produce images of different types of dog images.

I assume a different type of distribution could be selected if one uses the proper loss function for that type of distribution, that is, the loss function which would measure the difference in the distribution of the latent space and the desired distribution.

KL divergence is the loss function that forces the latent space distribution to be Gaussian. If you do not "restrict" the latent space, as is the case with a regular autoencoder, you have no idea what kind of vector to select as an input to the decoder to produce a dog image. Without restriction, there are $2^n$ (where $n$ is the dimensions of the latent space) possible vectors you could choose from. Chances of selecting one that produces a dog image would be minuscule.

Well, I hope this helps. I no longer am mathematically proficient (age 75), so I hope my interpretation of the math is correct.

A VAE tends to produce blurry images because there are two terms in the loss function. One term is trying to make the output look like the input while the KL loss term is trying to restrict the latent space distribution. GANs ( generative adversarial networks) don't have this conflict, so they produce much high-quality images.


How do I integrate Tensorflow with EV3DEV?

There is a YouTube video LEGO EV3 Raspberry Pi Tensorflow Sorting Machine by ebswift that should help you although you will need a Raspberry Pi.  From the abstract:

This is a sorting machine based on the EV3 45544 education kit sorting machine.  The colour sorting camera is substituted with a Raspberry Pi with the v2 camera.  The EV3 is controlled over wifi via RPyC and the object recognition work is done via Tensorflow.

A viewer asked: Can you share links about how to train a model on the PC and move the trained model to the Pi?

ebswifft replied:
Glad you like it :).  Training the model on the PC is following the guide for Tensorflow for Poets https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/.  All you do is install Tensorflow on the Raspberry Pi, clone the github repository onto it and copy the model you trained from the codelabs article onto the Raspberry Pi to run the classifier as per step 5 of the codelabs article.  Onboard image capture on the Raspberry Pi is just done using picamera.  I used a tensorflow version that a user compiled for the Raspberry Pi from here: https://github.com/samjabrahams/tensorflow-on-raspberry-pi/issues/92.  I might do up more of a general step-by-step sometime on my site, I'll report back if I can get that up and running.

There is also BrickClassifi3r:

This Lego Mindstorms EV3 robot uses a neural network to recognize a cube, a cylinder or a small cube put on a conveyor belt. See the video how it works. Each object on the conveyor belt is scanned by an IR-sensor every 40ms for about a second. The resulting data are 24 distance values representing one of the three objects. This data is fed into the neural network on the robot to classify the object within 180ms. In a test with 300 objects it reaches 95,6% accuracy. The neural network has been trained before by a machine learning algorithm with TensorFlow on a PC using a set of 375 training examples - 125 examples for each object.

The ev3-myo project uses LEGO Mindstorms EV3, TRACK3R, a Myo armband and TensorFlow for gesture recognition.


About the definition of "soft label" and "hard label"

According to Galstyan and Cohen (2007), a hard label is a label assigned to a member of a class where membership is binary: either the element in question is a member of the class (has the label), or it is not.

A soft label is one which has a score (probability or likelihood) attached to it. So the element is a member of the class in question with probability/likelihood score of eg 0.7; this implies that an element can be a member of multiple classes (presumably with different membership scores), which is usually not possible with hard labels.

One use of soft labels in semi-supervised learning could be that the training set consists of hard labels; a classifier is trained on that using supervised learning. The classifier is then run on unlabelled data, and adds soft labels to the elements. This enlarged data set is then used for further training, where the algorithm can treat hard and soft labels differently.


What are the fundamental differences between VAE and GAN for image generation?

GANs generally produce better photo-realistic images but can be difficult to work with.  Conversely, VAEs are easier to train but don’t usually give the best results.
I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.
There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism.  There is also VAE-GAN and VQ-VAE-GAN.
As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.


What does the argmax of the expectation of the log likelihood mean?

This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts):

Some more explanation from same page:

  In statistics, an expectation–maximization (EM) algorithm is an iterative method to find maximum likelihood or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. The EM iteration alternates between performing an expectation (E) step, which creates a function for the expectation of the log-likelihood evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log-likelihood found on the E step. These parameter-estimates are then used to determine the distribution of the latent variables in the next E step.

Mathematically, E in your equation stands for Expectation Value,  x|theta is conditional probability and x~data and model are sub-titles of source of probability in either case. The arg max theta is argument theta that maximizes the equation.


What does the notation $\nabla_\theta \mathcal{L}$ mean?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element of the parameter vector $\theta$. Therefore, it has the same dimensionality as $\theta$. Each element of the parameter vector is then updated with the respective term in the vector of partial derivatives, which are generally not the same.


Deepmind Spriteworld run_demo.py not found

I posted the question to the project's issue page, where the recommendation was to manually download and run run_demo.py. That worked for me.


What are the specific requirements of the Turing test?

The "Turing Test" is generally taken to mean an updated version of the Imitation Game Alan Turing proposed in his 1951 paper of the same name. An early version had a human (male or female) and a computer, and a judge had to decide which is which, and what gender they were if human. If they were correct less than 50% then the computer was considered "intelligent."

The current generally accepted version requires only one contestant, and a judge to decide whether it is human or machine. So yes, sometimes this will be a placebo, effectively, if we consider a human to be a placebo.

Your first and fourth questions are related - and there are no strict guidelines. If the computer can fool a greater number of judges then it will of course be considered a better AI.

The University of Toronto has a validity section in this paper on Turing, which includes a link to Jason Hutchens' commentary on why the Turing test may not be relevant (humans may also fail it) and the Loebner Prize, a formal instantiation of a Turing Test .


Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?

The autoencoder proposed in the link is similar to a Denoising autoencoder (DAE), in sense that both starting from a noisy image try to reconstruct the original image. The difference is that the noisy pixel are ignored during the backpropagation. The DAE takes in input an image, where before the forward phase noise is applied on the image. On output the DAE have to reconstruct the image without noise.
So, given an image $I$ and $\epsilon \sim \mathcal{N}(0,\,\sigma^{2})$ the noise is applied to the image obtaining the image $\hat{I}$. Now, $\hat{I}$ is the DAE input and the autoencoder is trained to minimize the function $L(I,g(f(\hat{I})))$. Where $f$ is the encoder and $g$ is the decoder.

The image on the left is an image of the MNIST dataset where $\epsilon \sim \mathcal{N}(0,\,0.3)$ and on the right there is the reconstruction of a deep DAE. You can change the type of noise in the image, the results should be the same. So the answer to your question is no, autoencoder are suited also for other type of noise.
If interested you can read the paper "What Regularized Auto-Encoders Learn from the Data-Generating Distribution" Guillaume Alain, Yoshua Bengio where is shown that the DAE is learning an approximation of the gradient of the data distribution.


is your data stored in raw ASCII text, like a CSV file?

Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.

There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.

Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.

In some cases, you may need to resort to a big data platform.
That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.


How to generate a dataset for question answering from books (like Facebook's bAbI)?

If you are talking about "generating" in the sense of generative models , it is pretty tough. since we are still far beyond understanding the actual structure of question-answering.

And even state of the art methods for question answering are also not able to score well on datasets like babi , mostly 16 out of 20 tasks can be solved.


How do I apply the value iteration algorithm when there are two goal states?

What you could do is to trigger environment termination when rat either:

steps into the trap
picks both cheese pieces

The problem with such setup is that, when the rat picks a single piece, it would move one step to the side, and then it would come back to the same cheese spot so it would keep exploiting the same spot indefinitely.

The solution to that would be to simply remove the cheese piece once the rat picks it up, so that it can't exploit it indefinitely.

Sadly, another problem would arise which is partial observability: Markov property wouldn't be fulfilled because the current action wouldn't depend on the current state solely, it would be important whether cheese piece was picked before or not.

The solution to that would be to make environment fully observable. You could accomplish that by expanding the amount of information about your current state. Before, only your position on the grid was important, but now you would also add state features that tell you whether cheese piece at specific position was picked or not. You would basically add a flag for each cheese piece that has value of 1 if piece was picked, or value of 0 if it wasn't. That way you could remove cheese piece it rat picks it, and you would still have full information.

I believe this setup would work.


What does "statistical efficiency" mean in this context?

Statistical efficiency in this context essentially means that a CNN would require fewer training examples than a fully connected network to learn. Intuitively this seems reasonable: more parameters to learn should mean more samples needed. Of course it is always desirable to minimise the number of training samples needed, so that's a definite advantage of CNNs.
There is a paper on the efficiency of CNNs which attempts to make that statement more precise. They examine the case of a convolutional network using a linear activation function.


Concatenate convolution layers with different strides in tensorflow.

Not 100% sure, but the problem is that when you work with different strides, the size of the convolved image change, so you should ensure, that all the convolved images have the same shape before concatenating the output. You can fill with 0s, or considering that the image is periodical in time, so filling with reflections of the image


What does the agent in reinforcement learning exactly do?

The agent in RL is the component that makes the decision of what action to take.

In order to make that decision, the agent is allowed to use any observation from the environment, and any internal rules that it has. Those internal rules can be anything, but typically in RL, it expects the current state to be provided by the environment, for that state to have the Markov property, and then it processes that state using a policy function $\pi(a|s)$ that decides what action to take.

In addition, in RL we usually care about handling a reward signal (received from the environment) and optimising the agent towards maximising the expected reward in future. To do this, the agent will maintain some data which is influenced by the rewards it received in the past, and use that to construct a better policy.

One interesting thing about the definition of an agent, is that the agent/environment boundary is usually considered to be very close to the abstract decision making unit. For instance, for a robot, the agent is typically not the whole robot, but the specific program running on the robot's CPU that makes the decision on the action. All the relays/motors and other parts of the physical body of the robot are parts of the environment in RL terms. Although often loose language is used here, as the distinction might not matter in most descriptions - we would say that "the robot moves its arm to achieve the goal" when in stricter RL terms we should say that "the agent running on the robot CPU instructs the arm motors to move to achieve the goal".

  I think it is not the Neural Net behind?

That is correct, the agent is more than the neural network. One or more neural networks might be part of an agent, and take the role of estimating the value of a state, or state/action pair, or even directly driving the policy function.


Can I train a neural network incrementally given new daily data?

Yes, this is possible. Continuously extending your training data is known as incremental learning.

You might also want to take a look at transfer learning, in which you reuse a trained model for a different purpose. This is very useful if you have a smaller dataset.

In your particular case, you could train a NN once using your data from 2010 to 2019 and use it as a base model. Every time you get new data, you can use transfer learning to slightly re-train this model. Based on parameters such as the number of epochs and the learning rate, you can determine how much of an impact this new data will have.


What happens when you select actions using softmax instead of epsilon greedy in DQN?

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action.

This is a very common choice, because it is simple to implement and quite robust. However, it is not a requirement of DQN. You can use other action choice mechanisms, provided all choices are covered with a non-zero probability of being selected.

What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?

It might in some circumstances. A key benefit is that it will tend to focus on action choices that are close to its current best guess at optimal. One problem is that if there is a large enough error in Q value estimates, it can get stuck as the exploration could heavily favour a current best value estimate. For instance, if one estimate is accurate and relatively high, but another estimate is much lower but in reality would be a good action choice, then the softmax probabilities to resample the bad estimate will be very low and it could take a very long time to fix.
A more major problem is that the Q values are not independent logits that define preferences (whilst they would be in a Policy Gradient approach). The Q values have an inherent meaning and scale based on summed rewards. Which means that differences between optimal and non-optimal Q value estimates could be at any scale, maybe just 0.1 difference in value, or maybe 100 or more. This makes plain softmax a poor choice - it might suggest a near random exploration policy in one problem, and a near determinitsic policy in another, irrespective of what exploration might be useful at the current stage of learning.
A fix for this is to use Gibbs/Boltzmann action selection, which modifies softmax by adding a scaling factor - often called temperature and noted as $T$ - to adjust the relative scale between action choices:
$$\pi(a|s) = \frac{e^{q(s,a)/T}}{\sum_{x \in \mathcal{A}} e^{q(s,x)/T}}$$
This can work nicely to focus later exploration towards refining differences between actions that are likely to be good whilst only rarely making obvious mistakes. However it comes at a cost - you have to decide starting $T$, the rate to decay $T$ and an end value of $T$. A rough idea of min/max action value that the agent is likely to estimate can help.


What's the difference between a static AI and a dynamic AI?

My sense is they're probably talking about the difference between symbolic AI (aka GOFAI)  and learning algorithms.

GOFAI typically uses heuristics, which are static—essentially fixed rules governing decision making.

Statistical AI, aka Machine Learning, is dynamic and can analyze and evaluate the environment to form its own decision rules.

Sometimes a trained NN that is no longer learning is said to have created it's own heuristics.


What are the advantages of complex-valued neural networks?

According to this paper, complex-valued ANNs (C-ANNs) can solve problems such as XOR and symmetry detection with a smaller number of layers than real ANNs (for both of these a 2 layer C-ANN suffices, whereas a 3-layer R-ANN is required).

I believe that it is still an open question as to how useful this result is in practice (e.g. whether it actually makes finding the right topology easier), so at present the key practical advantage of C-ANNs is when they are a closer model for the problem domain.

Application areas are then where complex values arise naturally, e.g. in optics, signal processing/FFT or electrical engineering.


What does the statement with the max do in the recursive best-first search algorithm?

This is probably more easily understood as the collapse/restore macro. The idea is that the previously explored state was collapsed and only the minimum f-cost from the sub-tree was stored. This represents the best unexpanded state in the subtree that was collapsed.

When restoring the portion of the collapsed tree, the f-cost of the restored node could either be the original f-cost (g+h), or it could be the stored f-cost if it is larger. By taking the max, the code ensures that states that are restored maintain at least the cost of the previously best unexpanded state. (If the g+h cost is larger, then we know the state wasn't previously expanded and it wasn't previously the state on the fringe with the minimum edge cost.)

The linked paper gives several examples where similar ideas are used during search.


Is it possible to train a neural network as new classes are given?

I'd like to add to what's been said already that your question touches upon an important notion in machine learning called transfer learning. In practice, very few people train an entire convolutional network from scratch (with random initialization), because it is time consuming and relatively rare to have a dataset of sufficient size.

Modern ConvNets take 2-3 weeks to train across multiple GPUs on ImageNet. So it is common to see people release their final ConvNet checkpoints for the benefit of others who can use the networks for fine-tuning. For example, the Caffe library has a Model Zoo where people share their network weights.

When you need a ConvNet for image recognition, no matter what your application domain is, you should consider taking an existing network, for example VGGNet is a common choice.

There are a few things to keep in mind when performing transfer learning:

Constraints from pretrained models. Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can’t arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides “fit”).
Learning rates. It’s common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don’t wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization).

Additional reference if you are interested in this topic: How transferable are features in deep neural networks?


Graph neural networks, of which GCNs are a specific type, are able to handle arbitrary graphs as input. GNNs operate first over "neighborhoods" of nodes to compute individual node representations and then optionally apply a pooling function to reduce these to a single graph-level representation that can be used in classification. This means that GNNs work "locally" and do not contain implicit assumptions about the graph topology.
For example, the update equation for a given node's representation in a GCN can be written as
$$h_v^{t+1} = \sigma\left({\bf W}^{t+1} \sum_{u \in \mathcal{N}_v} {\bf L}_{uv}~ h_u^t \right) $$
where $h_v^l$ is the representation of node $v$ at update $t$, $\sigma$ is an activation function, ${\bf W^{t}}$ is a weight matrix, ${\bf L_{uv}}$ is the value of the graph Laplacian (which is a matrix) at nodes $u$ and $v$, and finally $\mathcal{N}_v$ is the neighborhood of $v$. Looking at this expression it becomes clear that the value of the summation is always of the same dimension as $h$, no matter how you define the neighborhood $\mathcal{N}_v$. So you are correct that as long as the node representation size is static, the network can take arbitrary graphs as input.
I highly recommend this tutorial paper on GNNs, which I used when first learning about them. Section 2.3 specifically answers your question and discusses how things like cycles and non-positional graphs are handled.


An interesting way to check for redundancy not only in the first layer but in any layer is to see the weights of you neuron. Let's consider the input layer: The nodes that are connected with the weights that have high values are the most important for your NN performance and in other words your input to these nodes in what mostly makes sense for your task.

If you are in Keras, you can use the get_weights method(). You may have a look here.


Once self-driving cars are perfected, will they replace manually driven cars?

It likely to be happen, because it's more convenient that way. In general people, organizations and government are always keen to make things more efficient by standarizing things (computers, technology, law, science, etc.) in order to make it manageable and predictable to reduce the time and minimalize the risk of the same mistakes.

The whole world now moves into technological advancement where automation of everything is where we are going, so we can manage complexities in more reliable way, so we can focus on much bigger picture. This includes technology such as mobiles, computers, UAV (delivery drones), robots and now self-driving cars.

The pros of that change would be:

To have safer streets by introducing autonomous cars on the road.
To have fewer drunk, tired, drugged or crazy drivers.
To avoid poor weather conditions.
To reduce braking distances by dropping driver's reaction time and predicting dangerous situations much earlier.

Source: Cyber Physics
Reducing car deaths and costs of GNP.

  An estimated 1.3 million people die on the world's roads every year with around 50 million injured or disabled by accidents, with accidents costing countries up to four per cent of their Gross National Product (GNP) yearly. - UN News Centre

To have central point of safety improvements, you cannot change people, but you can fix the known safety issue on global scale.
To introduce global standards from the central point (e.g. new law to which manufactures needs to apply).
To increase car safety in general on larger scale.
And so on.

Why we need the 'only self-driving cars'?

The Secretary-General said the UN would work hard to prevent further deaths on the roads:

  Many tragedies can be avoided through a set of proven, simple measures that benefit not only individuals and families but society at large.

Here are my points:

To achieve 'a set of proven measures' - do not allow people to drive - simple.
People tend to break the rules, always, so do not allow them to drive without permission.
Reduce stealing cars and other crime.
Law enforcement dream is to able to stop any car on demand.
Do not allow drunk people to drive a car.
Disallow terrorist attacks, like in Nice where truck killed over 80 people.
Avoid bank robberies and similar which are possible by escaping fast cars.

Is it possible? I believe it depends on specific countries and unions and how quickly we're able to advance and be ready for such change.

To support above points and summarize the 'only self-driving cars' point, please see below references which shows that this is already happening:

2014: Uber will eventually replace all its drivers with self-driving cars
2016: Google's self-driving car system has been officially recognised as a driver in the US.

  The move is seen as a first step towards changing the law for cars that have "no need for a human driver".

2016: Beverly Hills to replace public transport with self-driving cars
2016: San Francisco pitches $149 million plan to replace cars with self-driving vehicles

  San Francisco’s future is autonomous and shared vehicles – and that future may be only a decade away.

2016: Otto Self-Driving Truck Company Wants to Replace Teamsters


During neural network training, can gradients leak sensitive information in case training data fed is encrypted (homomorphic)?

It will recover the encrypted inputs.
The algorithm starts with dummy data and dummy labels, and then iteratively optimizes the dummy gradients to be close as to the original. This makes the dummy data close to the real training data:
$$\mathbf{x}^{\prime *}, \mathbf{y}^{\prime *}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\nabla W^{\prime}-\nabla W\right\|^{2}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\frac{\partial \ell\left(F\left(\mathbf{x}^{\prime}, W\right), \mathbf{y}^{\prime}\right)}{\partial W}-\nabla W\right\|^{2}$$
As the distance is minimized, the algorithm restores the original training data; in the case of encrypted training data - you should get an encrypted input (up to failures when the resulted 'input' isn't close to the original input).


How to do machine translation with no labeled data?

In this paper: Unsupervised Machine Translation Using Monolingual Corpora Only
 the authors proposed a novel method.

Intuitively it is an autoencoder, but the Start Of Sentence token is set to be the language type.

One other advanced method is to use the pre-training model. In this paper: Cross-lingual Language Model Pretraining researchers proposed an algorithm that utilized the pre-trained multi-lingual trained BERT(with labeled data but we don't need to have a labeled dataset for our task) and the autoencoder mentioned previously.


Is understanding value for different features next step for object recognition?

From your question I can assume you are a beginner in the field of AI. Welcome to this exciting field.

To answer your question, we have not yet been able to create a truly artificially intelligent program. They are all apparently intelligent but are just a set of simple/complex rules. An artificially intelligent agent must have at-least 2 aspects in the inside of its head/program. The ability to logically derive conclusions, and a capability for learning (these combined with ability to to take inputs and respond).

Now, logical reasoning part is the field of AI. Lots of simple codes performing complex task already exists.

Your question is based on the learning part, which is handled by Machine Learning programs. They learn iteratively. Object recognition is only one part of ML. They can also predict. Anyways, object recognition is done on the basis of maximizing reward/minimizing penalty (Cost function). Now, this minimizing of penalty is done by giving different weights or valuing different attributes of the objects differently. So this has already been accomplished. Depending upon the task at hand, we supply the attributes which are related to the final task ,otherwise the attribute is of no use. Like economics has no influence on weather.

We also have this thought process by selecting things depending on the goal we need to accomplish, sometimes we select things consciously sometimes we are hardwired by genes to do so (Touching fire). So you see, ML methods are modeled on how we do things and we do things based on weight-age to different factors. If we don't give proper weight-age we either learn by the penalty/punishment imposed upon us or our genes get wiped out from existence and thus the same behavior of not giving proper weight-age is not passed on.

So in short, a ML learning algorithm recognizes objects or does some final task (especially in games like chess) based on the utility or influence of the given features of current state on the final result.


Audio enhancing using AI? (removing background noise during lecture recording)

Yes, it is possible. Usually the noise reduction is done using regular signal processing methods, such as spectral subtraction due to demand for low latency. But of course, modern methods of deep learning is applicable to this problem. For example variational autoencoder is the first that come to my mind, you can check this  project. As another example you can look at this project , which is based on WaveNet.


How to implement or avoid masking for transformer?

If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),
id_to_mask=vocab['<PAD>'])

Here are some resources for building Transformers in Trax:

Named Entity Recognition with Reformer (Masking Implemented) and the associated blog post (contains an in-depth explanation of NER and Transformers)

Deep N-Gram Models (without masking but I recommend you go through these first to get a hang of the library)


Is there a bias-variance equivalent in unsupervised learning?

Yes, the concept applies but it is not really formalized.  Consider unsupervised learning as a form of density estimation or a type of statistical estimate of the density.
Variance: You will train on a finite sample of data selected from this probability distribution and get a model, but if you select a different random sample from this distribution you will get a slightly different unsupervised model.  This variation caused by the selection process of a particular data sample is the variance.
Bias: This is a little more fuzzy depending on the error metric used in the supervised learning.  An unsupervised learning algorithm has parameters that control the flexibility of the model to 'fit' the data.  For example,  k means clustering you control the number of clusters.  Simple example is k means clustering with k=1.  You could imagine a distribution where there are two 'clumps' of data far apart.  The mean would land in the middle where there is no data.  This model is biased to assuming a certain distribution.  For a higher k value, you can imagine other distributions with k+1 clumps that cause the cluster centers to fall in low density areas.  For a low value of parameters, you would also expect to get the same model, even for very different density distributions. This is also a form of bias. This unsupervised model is biased to better 'fit' certain distributions and also can not distinguish between certain distributions.
You can see that because unsupervised models usually don't have a goal directly specified by an error metric, the concept is not as formalized and more conceptual.


Which AGI systems have already been implemented and tested?

As far as I know, no "true" (i.e. as intellectual and physically capable as a human) artificial general intelligent system (AGI) has been implemented or is practically useful (this is confirmed by Ben Goertzel, who is one of the leading researchers in AGI [1, 2]).
The closest to a practical AGI might be Sophia (or similar robots), which may look like an AGI, but it lacks several capabilities that we humans have and its ability to adapt to new circumstances is limited. Sophia uses OpenCog [1], which is supposed to be a software framework to develop AGI. Sophia is used for the loving AI project.
There are also theoretical frameworks for AGI, such as AIXI, which, in any case, have several flaws, such as incomputability (in the case of AIXI). There are approximations of AIXI, but these approximations can only be used to solve toy problems (such as tic-tac-toe), so they aren't really useful to solve complex real-world problems. However, it is possible that better approximations to AIXI and more theoretical frameworks for AGI will be developed that can deal with more complex problems.
It's also important to note that, despite their success, AlphaGo and AlphaStar are narrow AI systems, as they can only solve one specific problem (although the same approach can be adapted to solve very similar problems too, e.g. AlphaZero).
If you want to know more about AGI, the Scholarpedia's article Artificial General Intelligence, curated by Ben Goertzel, provides a good overview of the AGI field, including but not restricted to definitions of AGI and approaches to the development of AGI systems, such as

universal (AIXI was created based on this approach),

symbolic (Soar was created based on this approach), which is based on the physical symbol system hypothesis,

emergentist (or sub-symbolic), which is based on the idea that general intelligence is expected to emerge from sub-symbolic dynamics (where a sub-symbolic system refers e.g. to an artificial neural network),

hybrid (e.g. CLARION), which is a combination of the universal, symbolic or sub-symbolic approaches.

You probably also want to take a look at this question, which has a few answers that mention a paper and book that you can read to know more about the AGI field. You may also be interested in this related question too.


Recommended Modelling Technique for Influencer Marketing Scenario

It depends very much on the structure of the data.

I would think about feature extraction first, which could be certain words occurring in the bio, and a class of user name ('real' name, numerical id, etc). Once you have a set of features for each data item, turn them into a list of feature vectors.

Then run them through a number of machine learning algorithms. This is where the shape of the data matters, as some algorithms will work better than others. I would try eg decision trees (ID3), which are very efficient once trained (but they don't give you a confidence interval). But any other ML algorithm might work. They will all have trade-offs with speed of training, memory requirements, and speed of classification; some will give you a class-label probability, others will just give you one label.

The best way would be to use a sample, and identify which algorithm works well and fits your specific requirements. Then use that for the full data set.

Alternatively you could just use, for example, the Stanford ML classifier. That will give you a confidence interval, and will probably work reasonably well.


How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves.

For exploration it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you will always retain an agent which occasionally makes illegal moves.

To me it makes more sense to just set the probabilities of all illegal moves to zero and renormalise the output vector before you choose your move.


What's the intuition behind contrastive learning?

Contrastive learning is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a dictionary look-up problem.
If I conceptually compare the loss mechanisms for:

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations @ https://arxiv.org/abs/2002.05709
MoCo: Momentum Contrast for Unsupervised Visual Representation Learning @ https://arxiv.org/abs/1911.05722

Both MoCo and SimCLR use varients of a contrastive loss function, like InfoNCE from the paper Representation Learning with Contrastive Predictive Coding
\begin{eqnarray*}
\mathcal{L}_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits_{k^-}exp(q\cdot k^-/\tau)}
\end{eqnarray*}
Here q is a query representation, $k^+$ is a representation of the positive (similar) key sample, and ${k^−}$ are representations of the negative (dissimilar) key samples. $\tau$ is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.

The contrastive loss can be minimized by various mechanisms that differ in how the keys are maintained.
In an end-to-end mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. SimCLR, is based on this mechanism and requires a large batch to provide a large set of negatives.
In the MoCo mechanism i.e. Momentum Contrast (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.


If the only feature you're classifying on is the number of users making a given report, then this isn't really much to do with AI/ML. Just pick a number based on your subjective judgment and go with it.

OTOH, if you can include details of the report itself (as well as the number of reporters), I think you might be able to build a bayesian classifier that would be useful.  If you could consider location, weather, time of day, number of reporters, etc., it seems like you might be able to get something useful put together.


Just try returning a function that takes the state as an input and returns the probabilities for each action in the form of a numpy array of length of the action space (set of possible actions). Here, is one attempt:
def EpsilonGreedyPolicy(Q, epsilon, no_of_actions):

    def policy(state):
        probabilities = np.ones(no_of_actions, dtype = float) *
                    epsilon / num_actions
        best_action = np.argmax(Q[state])
        probabilities[best_action] += (1.0 - epsilon)
        return probabilities

    return policy


The bandit problem has one state, in which you are allowed to choose one lever among $n$ levers to pull.

  Why is there just one state in the formulation of this bandit problem?

There is one state because the state does not change over time. Two notable consequences are that (i) pulling a lever does not change the internals of any slot machine (e.g. the distribution of rewards) and (ii) you are allowed to choose any lever without restrictions.  More generally, there is no sequential aspect of the state in this problem, as future states are unaffected by past states, actions, and rewards.

  It is not a bandit problem if $n$ is large and the share price is affected?

Correct! If the share price is affected, then future states would be influenced by past actions. This is because the price per share is affected, which is one aspect of the state. Thus, you would need to plan a sequential strategy for your purchases.

  It is a bandit problem if $n$ is small and the share price is not affected?

It all depends on the problem: as long as the state before buying shares remains completely the same after you purchase some shares, then yes. Share price being unaffected is only one of the requirements; another example requirement is that the maximum number of shares purchased is fixed at each time step, independent of the shares purchased previously.

  Does it make sense to have a mix of a bandit and non-bandit states for a given RL problem? If so, then the approach to solving should be to consider the issue in its entirety as not being a bandit problem?

It makes sense to allow the share price to either be affected or unaffected based on $n$ in the same problem. Since some actions (large $n$) change the state, then there are multiple states, and actions sequentially affect the next state. Hence it is not a bandit problem as a whole, as you correctly stated.

  The agent potentially can utilise the slot results to determine a state-action value?

Correct! I suggest reading Chapter 2 of Sutton and Barto to learn some fundamental algorithms of developing such strategies.

Nice work on analyzing this problem! To help solidify your understanding and formalize the arguments above, I suggest that you rewrite the variants of this problem as MDPs and determine which variants have multiple states (non-bandit) and which variants have a single state (bandit).


Are there any pathfinding algorithms that take customized rules into account when determining the shortest path?

Yes, this is easily solved using the A* algorithm. Once your agent has visited a particular node, increase the cost of that node to infinity and recalculate the path.


It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent.

So, this is the setting of a conventional MAB problem: at each time step, the agent selects an action to either perform an exploration or exploitation move. It then records the reward of the taken action and updates its estimation/expectation of the usefulness of the action. Then, repeats the procedure (selection, observing, updating).

To chose between exploration and exploitation moves, MAB agents adopt a strategy. The simplest one would probably be $\epsilon$-greedy which agent chooses the most rewarding actions most of the time (1-$\epsilon$ probability) or randomly selects an action ($\epsilon$ probability).


Game AI - Fast python OCR or cropped image input

Option 1 would be a very interesting one from a research perspective. I cannot imagine that CNNs have to capacity at the moment, to learn the concept of numbers and apply them in a useful way. If it was an analog speedometer, things would be different. But it would be really interesting to try it and see what you can achieve with this approach. I haven't read any research papers yet, where such a challenge would have been mastered.

If you are less concerned with research but rather get this project to work, I would propose an approach similar to option 2. The OCR tools you mentioned were designed to identify symbols, even if distorted or otherwise hard to read. In your case, the numbers will always look the same and are most likely always in fixed positions. Therefore, using fancy OCR algorithms or neural nets are overkill for the problem at hand. You can write a simple algorithm that crops the speedometer, searches for the 10 possible patterns (0 to 9) with 10 specific kernels and calculate the speed from the result. This can be implemented efficiently without the need to train a CNN or some other complex algorithm.

Method for implementation:

To answer the first question in the comments. The algorithm for this is really easy. Your image will be represented by a 2D array (or maybe a 3D array if you are working with colors). All you need to do is find out, what your numbers look like and store the appropriate array. You should end up with 10 different arrays for 10 different numbers.

To find out if a number is in your current picture, you only have to check if the array for this number is a subset of the array of your current picture. The position, where the arrays are matching, will also indicate the position of the number. You do this for all 10 numbers and can calculate the actual value displayed on the speedometer afterwards from the results.

This is very easy if the numbers always look exactly identical. In this case you can simply look for an exact match. If there are slight variations, you can look for parts of the image array that is similar to your template arrays. Just sum the absolute values of the diffs of each pixel value and if it is below a certain threshold, you have your match. That's all there is to it.

In python I like working with PIL for this kind of image manipulation, but there are other frameworks out there as well.


Intuition behind $\gamma$-discounted state frequency

Update: I rewrote the first part due to major mistake in the first version

Notice: The notation $P^k$ from Eq.$(20)$ and $(21)$ in the paper does not mean the kth power of some $P$. Instead, $P^k$ should be thought as the $k$ step transition probability of a non-homogeneous Markov chain.

According to the CPO paper, the discounted future state distribution is defined as
$$
d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb(s_k=s|\pi,s_0)\mu(s_0)\tag{1}
$$
Consider function form. Let $Prob_{\pi,k}$ denote the $k$ step probability transition operator induced by $\pi$; here $\pi$ can be a hierarchical policy, $k$ can be larger than $c$.
$$
d_\pi=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb_{\pi,k}\mu\tag{2}
$$
Now apply the similar definition as Eq.$(20)$ and $(21)$ in the paper, let $P^k_\pi$ denote the $k$ step transition probability of the non-homogeneous Markov chain induced by the low level policy, with $k$ smaller or equal to $c$.
\begin{align}
d_\pi&=(1-\gamma)\sum_{m=0}^\infty\gamma^{mc}\sum_{k=0}^{c-1}\gamma^kP^k_\pi(P^c_\pi)^m\mu\\
&=(1-\gamma)\sum_{k=0}^{c-1}\gamma^kP^k_\pi(\sum_{m=0}^\infty\gamma^{mc}(P^c_\pi)^m)\mu\\
&=(1-\gamma)A_\pi(I-\gamma^cP^c)^{-1}\mu\tag{3}
\end{align}
which is exactly the form of Eq.$(22)$ and $(23)$ in the paper, with $A_\pi$ defined similar as Eq.$(24)$ and $(25)$.
The "every-$c$-step discounted state frequency" builds on $(3)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition operator is $P_\pi^c$. Starting with $(3)$, replace $c$ with 1, we get the "every one step future state distribution"
$$
d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{4}
$$
Then replace $\gamma$ and $P_\pi$ in $(4)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution"
$$
d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{5}
$$
By the way, I read your blogpost on this paper. It's very helpful for me, thank you!


Necessity of GPUs and hardware minimum specs for Deep Learning?

For a simulation of SNES or N64, the resolution is probably not that high. You can use either online credits or buy new hardware. For online credits, it is recommended if you do teh simulation/training for only several dozens hours, as eache hour costs around 5-10 dollars. AWS is a good choice as it have a large variety of choice for hardware. For higher performance, choose multi GPU setup.

However, if the simulation requires a long time, or you wish to own the hardware, you should buy your own hardware. Multiple GPU will scale quite nicely, so you may go for multi GPU if performance is needed. For single GPU, a RTX 2000 series is recommended as it have tensor cores to accelerate training of CNN. For a budget option, consider an RTX 2060 SUPER. It have 8 GB of VRAM with a relatively low price and yet it have tensor core functionality. For more powerful GPU, choose the RTX 2080 TI. It have 11GB of VRAM which is enough in most use case.
For multiple GPU, you might want several RTX 2060 SUPER as they are cheap but the performance is quite good. The 8GB VRAM helps as well. For the highest performance, choose 2 or more RTX 2080 TI. It is enough for nearly most task. If this is not enough, renting an online instance is a better idea. Hope it helps and have a nice day.