What are the criteria for choosing Turing award winners?

The ACM A.M. Turing Award is an annual prize given by the Association
  for Computing Machinery (ACM) to "an individual selected for
  contributions of a technical nature made to the computing community".
  It is stipulated that the contributions "should be of lasting and
  major technical importance to the computer field". The Turing Award is
  generally recognized as the highest distinction in computer science
  and the "Nobel Prize of computing".


AI techniques and competencies necessary for creating a Virtual Assistant

First of all, making app for mobile is not normally written in Python, but if you want it, that is possible [1].

Other non ai question is that the counting of speech recognition is so heavy, that at least Alexa etc make it on external server. Wake word recognition is done in machine because of real time needs [3]. Ofcourse your app can trigger from mobile touch event too so speech recognition implementation on mobile can be avoided.

Using server needs knowledge about communication and web apis as well as server maintenance in cloud.

Speech recognition can be made with open source tools, but the dictionaries with enough samples may not yet exist or they're coming [2]. Also Google offers speech api, that needs mainly programming knowledge to attach speech recognition.

For questions you also need to generate answers. To form that you may reverse the flow in speech generation [4]. All the logic to find useful answers needs some more effort depending how intelligent your assistant will be. That maybe is the most challenging part where direct answer do not exist.

Of course, if you could attach to ready assistant's public api, it would be magnificent. Actually, Alexa api at least is open [5]!! Playing with it directly from mobile makes app soon ready but if you want to learn more algorihms, some pointers are on my above other points.

Sources:

[1] https://www.quora.com/Can-I-make-an-Android-app-with-Python

[2] https://research.mozilla.org/machine-learning/

[3] https://iot.stackexchange.com/questions/736/why-do-most-smart-assistants-offer-little-if-any-customisation-of-the-wake-wor

[4] How could you generate sentences from lists of facts

[5] https://github.com/alexa/alexa-avs-sample-app


How do I know if my backpropagation is implemented correctly?

For future reference, I will merely point you to a technique you can implement to test the correctness or lack thereof, of your backpropagation implementation.

Ps: don't feel too bad for having gotten it slightly wrong, "backpropagation is notoriously difficult to implement" - source :).

In fact, there is a technique called "Gradient checking" meant specifically for this purpose (for more information, see Andrew Ng's lecture video on gradient checking and, this notes). I would argue that even gradient checking is a little tricky to implement.

How does Gradient checking work:

Backpropagation computes the gradients (∂J/∂θ)(∂J/∂θ), where θ denotes the parameters of the model. J is computed using forward propagation and your loss function.
But because forward propagation is fairly straightforwards to implement, most people are usually confident that you got its implementation correct. So the trick is to use the value of J to verify your code for computing (∂J/∂θ)(∂J/∂θ).
We know that by definition, the gradient or derivative is given by:

(∂J/∂θ) = lim ε→0 J(θ + ε) − J(θ − ε) / 2ε

now, since we trust our calculation of (∂J/∂θ), we can easily compute the value of J(θ+ε)J(θ+ε) and J(θ−ε)J(θ−ε).

For more information, see here and, over here).


Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e

layer_output = Weights.layer_input + bias

Consider a 2 layer neural network, the outputs from layer one will be:
    x2 = W1*x1 + b1
Now we pass the same input to the second layer, which will be

x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b

Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.

So what is the solution?

We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:

ReLU
Sigmoid
tanh
Softmax

and there are a lot more of them.

Yay! Our network is no more linear!

We have a lot of different non linear functions, and each of them serve a different purpose.

For example, ReLU is simple and computationally cheap.
    ReLU(x) = max(0, x)
Sigmoid outputs are between 0 and 1.
tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
Softmax is usually used if you want to represent any vector as a discrete probability distribution.

Hope you are having a great day!


When to use NLP, NLG and NLU in conversation agents?

They're all important. NLP is an umbrella term that includes the other two; NLG is only concerned with generating language, ie transforming some internal data structure into human language. NLU is about processing information contained in language, and putting it into relation with a knowledge base etc.

If you don't know anything about any of these fields, then I suggest your aim is far too optimistic. I work for a company that provides a conversational AI platform for businesses to develop their own agents, and it is a complex area.

If you want to have a quick go and pick up some experience, I suggest you start with ELIZA. Even though this is 'ancient', many modern chatbots still work on the same principles. There are many implementations in a number of programming languages, so you should be able to find one that suits you and you can tinker around with it.


Should I use leave-one-out cross-validation for testing?

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves.

The important aspect when choosing $k$ is a bias-variance tradeoff. Note that in LOOCV you train each model using almost as many samples as there are available ($n-1$), so the validation step should give you an unbiased estimate of the real test error. However, each model in LOOCV is trained each time on almost exactly same dataset!. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, LOOCV will suffer from huge variance.

On the other hand, in 2-fold CV the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).

What to do in this scenario? Choose something in the middle. Usually $k=5$ and $k=10$ should be a good choice.


CPU preferences and specifications for a multi GPU deep-learning setup

Lets answer the question assuming ideal case.

Say you are training a deep learning algorithm in which p proportion of the algorithm is parllelizable and (1-p) proportion as sequential part . Lets assume you can perfectly divide the program into their parallelizable and sequential parts. I don't exactly know the specs of aforementioned CPU's, but lets assume for now they are symmetrical processors (which is the most common) i.e. all the cores or CPU's perform the task equally and there is  no master CPU coordinating the functioning between the different cores.

Now as per a law called Amdahl's Law which states:

Where p holds the same meaning, while N can be approximated as the number of cores sharing the task and S is the reciprocal of the rate of speedup you will get (that is time required will be T * S(N). So, for each particular GPU's with N number of CPU's you can calculate the times of speedup you will get.

Specifically, calculate the number of calculations your NN will make while training. Here is a quite good calculation of the same. Generally a CPU will take 6-7 clock cycles for multiplication and 60-70 for division. Calculate the total number of clock cycles taken = a by the learning algorithm. Then calculate the time taken to complete the calculation by a single CPU by the formula a/clock_speed. Then using Amdahl's law calculate the speedup where N will be the number of CPU's in the GPU.

So you see, it clearly depends on the type of program you are running. Cache memory also increases the speed of computation. How? Cache memory is basically a temporary storage memory. When the program accesses your data/observations, it first checks if the data is available in cache memory since it can access data faster from cache memory, if not available then it checks the RAM then your hard-disk and loads it first unto RAM then to cache. This is  an oversimplified version of cache working.

Also check the number of actual physical CPU's, virtual CPU's are useless in parallelization. For example i7 generally consists of 4 physical CPU's.

The part I skipped, if there exists a master CPU just replace N with N-1 since it only performs the task of coordination.

So now you just have to find the general class of programs you will  be running and check its sequential and parallelizable parts.

Hope this helps!

EDIT : Recently I learnt that not all GPU's support the most popular deep learning framework tensorflow-gpu (i.e. the package to utilise multiple GPU's). It is specifically supported by NVIDIA GPU's as the CUDA framework required for tensorflow-gpu is specifically made for NVIDIA. Also some newer other brand GPU's are supporting tensorflow-gpu. Although tensorflow CPU only version can be run on any system. (Anyone can clarify this matter if they have more knowledge about this).


How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it.

If you are using an update rule like loss = -log(probabilities) * reward, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.

Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want.

However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, loss = -log(1-probabilities) * reward might be more appropriate when the reward is negative.


What are daily life examples of SAT problems?

SAT problems are decision problems that can be categorised as NPC. This informally means although there has not been any solution that can solve these problem in the polynomial order, the solutions of such problems can be satisfied in $O(n^c)$.

About your question, first, you should see your problem has exponential space and cannot be solved in polynomial order; after that, you have to investigate whether its solutions can be satisfied in polynomial time or not. An easy way to model this is to consider the boolean table and an expression to satisfy. Your problem can be considered as the boolean equation and different possibilities of variables can be possible solutions that you may want to verify.


Dealing with "blank" inputs in prediction of a neural network?

I think the answer to your question would be "yes". Though inference would always be best if you provide representative training data. For example you can train your net with (c, b) pairs, (blank, b) pairs, and (c, blank) pairs. That would make the net more robust and likely to support your use case. Training separate nets for each case would be more efficient and accurate - but I'm not sure what your goals and constraints are.

The point is a "blank" input can be trained for as well in the same net. In your case it's a new kind of body_height or color_of_clothing. I also suspect you might have issues with how you encode blank because e.g. rgb(0, 0, 0) would be black, and perhaps you intended to model blank as zero which would mean a blank is black.


Is the agent aware of a possible different set of actions for each state?

This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions.
If you explicitly represent the entire state space, as you might chose to do with simple benchmark problems that you solve by directly solving an MDP with something like value iteration, then you can also easily explicitly represent exactly the set of actions that the agent can perform in each state, and the agent can learn the expected value of just taking those actions.
If your state space is very large, you may not be able to represent it explicitly, and your agent is more likely to use some approximation of the value function or its policy, as is commonly done in Q-Learning. Here, it is often preferable to define your model of the environment so that taking an invalid action in a state causes some well-defined outcome, or causes the agent to randomly re-select its actions until it ends up picking a valid one. The agent will eventually learn that selecting an invalid action leads to bad outcomes, without "realizing" that the action is invalid.


What exactly are genetic algorithms and what sort of problems are they good for?

Evolutionary algorithms are a family of optimization algorithms based on the principle of Darwinian natural selection. As part of natural selection, a given environment has a population of individuals that compete for survival and reproduction. The ability of each individual to achieve these goals determines their chance to have children, in other words, to pass on their genes to the next generation of individuals, who, for genetic reasons, will have an increased chance of doing well, even better, in realizing these two objectives.
This principle of continuous improvement over the generations is taken by evolutionary algorithms to optimize solutions to a problem. In the initial generation, a population composed of different individuals is generated randomly or by other methods. An individual is a solution to the problem, more or less good: the quality of the individual in regards to the problem is called fitness, which reflects the adequacy of the solution to the problem to be solved. The higher the fitness of an individual, the higher it is likely to pass some or all of its genotype to the individuals of the next generation.
An individual is coded as a genotype, which can have any shape, such as a bit vector (genetic algorithms) or a vector of real (evolution strategies). Each genotype is transformed into a phenotype when assessing the individual, i.e. when its fitness is calculated. In some cases, the phenotype is identical to the genotype: it is called direct coding. Otherwise, the coding is called indirect. For example, suppose you want to optimize the size of a rectangular parallelepiped defined by its length, height and width. To simplify the example, assume that these three quantities are integers between 0 and 15. We can then describe each of them using a 4-bit binary number. An example of a potential solution may be to genotype 0001 0111 1010. The corresponding phenotype is a parallelepiped of length 1, height 7 and width 10.
During the transition from the old to the new generation, the variation operators, whose purpose is to manipulate individuals, are applied. There are two distinct types of variation operators:

the mutation operators, which are used to introduce variations within the same individual, as genetic mutations;
the crossover operators, which are used to cross at least two different genotypes, as genetic crosses from breeding.

Evolutionary algorithms have proven themselves in various fields such as operations research, robotics, biology, nuance, or cryptography. In addition, they can optimize multiple objectives simultaneously and can be used as black boxes because they do not assume any properties in the mathematical model to optimize. Their only real limitation is the computational complexity.


What are applications of object/human tracking in autonomous cars?

There are two important parts:

When you track, you have a trajectory you can expect. So it becomes easier to "look into the future" is you understand the past
When you loose an object you had a trajectory of before, you might be able to come up with an explanation that can influence your behavior


Can the normal equation be used to optimise the RNN's weights?

Unfortunately, this is not possible. The normal equation can only directly optimise a single layer that connects input and output. There is no equivalent for multiple layers such as those in any neural network architecture.


Fitness function in genetic algorithm based on an interval

Genetic algorithms work best when given a scalar fitness value that increases smoothly, so that you can compare two population members regardless of whether they failed or succeeded at the task.

That usually requires you to analyse the problem, and come up with a measure that would improve as an individual gets closer to solving a task. It generally helps if you score better for an individual if they solve a problem "better". You want to avoid simple boolean success/fail metrics.

A simple measure of how well an individual has done is to use the absolute distance it was away from a "perfect" shot (where the middle of the ball hits the middle of the hole). The only issue with this is that a perfect shot scores 0, whilst a miss scores 2+, and you want the best result to have the highest fitness. This can be fixed simply, take the negative of the absolute distance:

$$F = -|D_{hole} - D_{ball}|$$

where $D_{hole}$ - and $D_{ball}$ are horizontal distances from origin to centre of each object.

There is no requirement for fitness score to be positive. This will score $F \lt -2$ for a miss, $-2 \lt F \lt 0$ for hit, and $0$ for a perfect shot.

For GAs you don't have to care as much about differentiability or gradient of the fitness function, provided it gives reasonably good ranks between quality of individuals. So there is no point using e.g. a squared error metric here, although you could if you wished.


Is reinforcement learning suited for real-time systems?

Short answer: Yes, it is.

Explanation

Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs.  As with any online learning algorithm, there are a few things to consider.

The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others techniques to overcome it.

Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.


How can I train a neural network to detect subliminal messages?

Setting aside the dubious nature of subliminal messages. You have to clearly define what subliminal is. In all your cases presented, a manual algorithmic approach would be better. An ANN would tend to average out the subliminal messages as noise, just as your brain would. In the case of #2, #3, it's a matter of contrast, in which case computer vision methods of bringing out low contrast details to the forefront combined with a manual algorithm (perhaps an OCR pass for content not expected to have text) would be the way to go. ANN are not magic bullets, and still need to be used appropriately.


It would be better to run experiments on this. But, here's answer analytically.
The models will be different.
With augmentation, the network starts to learn to combat the noise too.
With late onset augmentation, the network will start to deviate from its original solution to combat noise.
Comparing this to the ball rolling down the hill, in case of augmentation, the ball will roll but will face a bit of friction (the noise due to augmentation). On the other hand, with late onset augmentation, it would be like the ball reached the bottom of hill, and now it has to face an uphill battle to model the noise. The ball will need to go a bit up the hill against gravity and friction due to noise.
I don't think it would be a good thing to do. But, running experiments is the only way we can confirm.


Why is no activation function used at the final layer of super-resolution models?

I am not into the field of super-resolution, but I think this question applies to general neural network construction.
Usually, you try to solve a classification problem or a regression problem with your neural network.

For classification, you try to predict probabilities that a specific output corresponds to a specific class. Therefore, every output value should be a probability and therefore have a range between 0 and 1. To achieve this, you usually use a softmax or sigmoid function as your last layer to squash the output between 0 and 1. In addition to this (which is wanted in classification tasks), these functions raise the probability output of likely classes while decreasing the probability of all other unlikely classes (therefore enforcing the network to choose for one specific class over the others).

For the regression task, you are not looking for probability values as your output values but instead for real-valued numbers. In such a case, no activation function is wanted, since you want to be able to approximate any possible real value and not probabilities.

So, in the case of super-resolution, I think the generated output is a map where each value corresponds to a pixel value of the super-resolution image. In that case, your pixels are real number values and no probabilities. So, you are solving a regression problem.
But you could also go with a classification approach, where you have 256 output maps that give probability to each possible pixel value between $0$ and $255$.


How exactly does nested cross-validation work?

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.
Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the mean cross-validated score of the best estimator (the model with the best hyperparameters). However now you used the same data for training and reporting the performance, although you used CV. Keep in mind that the folds are not independent. Therefore the hyperparameters might be too data specific, i.e. your generalization error estimates are too optimistic. Therefore we like to evaluate our final model performance outside / independent of the hyperparameter selection loop / process (the call cross_val_score()).
In the provided plot, you can clearly see that the reported performance by GridSearchCV() is most of the time better than the performance reported by cross_val_score().


Class imbalance and "all zeros" one-hot encoding?

I understand your questions as follows:

Is it valid to have an all-zeros output for a certain input?

Yes its possible in some cases like:

In the tutorial "jigsaw-toxic-comment-classification-challenge", the data is taken from wikipedia comments, so generally there is no toxic behavior in people's comment, it is very rare that a person posts "bad" comments in an informative source like wikipedia, this might lead to all the labels being zero for a particular example.
In a single label classification like predicting whether a person has a rare disease,the dataset would contain labels that are mostly "zero"-(no disease)-skewed towards zero

This happens in datasets where the positive output to be predicted is a very rare case.

I guess your second question is -whether "accuracy" is a good way of evaluating a model for this type of a problem

You are right, in such a case even a simple program/model that outputs zeros for all the inputs would achieve an accuracy of more than 90%, so here accuracy is not a good metric to evaluate the model on.

You should go through the metrics f1_score,recall, and precision which are ideal for this type of problems.

Basically here we are interested in "out of those which are predicted positive, how many are really positive -precision" and "out of those which should have been predicted positive, how many are really predicted positive"

If my definitions seem confusing to understand, please got through the link below:

f1_score/recall/precision

Hope I was helpful


How is $v_*(s) = \max_{\pi} v_\pi(s)$ also applicable in the case of stochastic policies?

The value function is defined as $v_\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ where $G_t$ are the (discounted) returns from time step $t$. The expectation is taken with respect to the policy $\pi$ and the transition dynamics of the MDP.
Now, as you pointed out the optimal value function is defined as $v_*(s) = \max_\pi v_\pi(s)\; ; \;\forall s \in \mathcal{S}$. All we are doing here is choosing a policy $\pi$ that maximises the value function; this can be a deterministic or a stochastic policy, though intuitively it is likely to be deterministic unless for some states that are two (or more) actions with the same expected value, in which case you can take any of said actions with equal probability, thus making the policy stochastic.
For a finite MDP (which is what I assumed above too), we know that an optimal value function exists (this is mentioned in the book) so taking the maximum is fine here.


What's the difference between estimation and approximation error?

Section 5.2 Error Decomposition of the book Understanding Machine Learning: From Theory to Algorithms (2014) gives a description of the approximation error and estimation error in the context of empirical risk minimization (ERM), so in the context of learning theory. I will just summarise their definition. If you want to know more about these topics, I suggest that you read that section.  If you are looking for the definitions of these topics in other contexts (e.g. reinforcement learning), you should ask a new question and provide more context (i.e. a paper or book that uses those terms in RL), because I also don't intend to extend this answer, although it is possible that these definitions below apply to other contexts too.
Approximation error
The approximation error (AE), aka inductive bias, denote as $\epsilon_{\mathrm{app}}$, is the error due to the specific choice of hypothesis class (or set) $\mathcal{H}$.
Here are some properties.

The larger $\mathcal{H}$ is, the smaller this error is (this should be intuitive!)

This error does not depend on the training data

Estimation error
The estimation error (EE) is the difference between the approximation error $\epsilon_{\mathrm{app}}$ and the training error $L_{\mathcal{D}}\left(h_{S}\right)$, i.e.
$$\epsilon_{\mathrm{est}}=L_{\mathcal{D}}\left(h_{S}\right)-\epsilon_{\mathrm{app}}$$
Here are some properties.

The EE arises because the empirical risk is just a proxy for the expected risk (i.e. the true risk, which is not computable because we don't know the underlying probability distribution that generated the labeled data).

The EE depends on the

training data (given that it is defined as a function of the training error, which depends on the training data), and

the choice of the hypothesis class (given that it is defined as a function of $\epsilon_{\mathrm{app}}$).

Bias-complexity tradeof
If we increase the size and complexity of the hypothesis class, the approximation error decreases, but the estimation error may increase (i.e. we may have over-fitting). On the other hand, if we decrease the size and complexity of the hypothesis class, the estimation error may decrease, but the bias may increase (i.e. we may have under-fitting). So, we have a bias-complexity trade-off (where the bias refers to the approximation error or inductive bias) and the complexity refers to the complexity of the hypothesis class.


Dropout and Max-pooling are performed for different reasons.

Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.

Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).

As for your last question, yes. dropout can be used after a pooling layer.


Is robotic process automation related to AI?

In the context of IT systems, "Robotic Process Automation" (RPA) is a term often used to describe a technique where software systems are integrated or work processes are automated through the existing user interface of the applications rather than writing new software to provide integration points.

In that context, RPA has nothing to do with AI or machine learning. In most cases, it does not even require OCR.

For an example of a common use case, let's say you have an old mainframe IT systems for tracking your subscriptions and a new website to let people order subscriptions from their phone.

In this case you might create an "RPA" job that opens the list of new subscription requests from the website and for each of them  opens the old application, clicks of the "new subscription" button, click on the "Customer Name" field, pastes the name, clicks "Customer address" and pastes in the address etc.

In some cases, the RPA job will be exposed as a service with an API that can be called by the new application, so it can dump data directly into the old application. The benefit is that it can do this without any changes to the old application.

It is attractive because defining the steps of copy this, click that can often be defined in visual tools by non-programmers very quickly and at much lower cost than setting up a systems integration project to connect the two systems and because integration through the existing user interface does not require any changes to the application.

In this way, it is similar in spirit to how Excel allows non-programmers to automate calculations by writing formulae and thus automating their spreadsheets.

You will often see RPA proponents putting some AI buzzwords into their presentations but from what I have seen in industry RPA is mostly just a visual scripting technique that is easy to learn and easy to apply.


Why is it a problem if the outputs of an activation function are not zero-centered?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the weight vector needs more update to be trained properly and the number of epochs needed for the network to get trained also increases. This is why the zero centered property is important, though it is NOT necessary.
Zero-centered activation functions ensure that the mean activation value is around zero. This property is important in deep learning because it has been empirically shown that models operating on normalized data––whether it be inputs or latent activations––enjoy faster convergence.
Unfortunately, zero-centered activation functions like tanh saturate at their asymptotes –– the gradients within this region get vanishingly smaller over time, leading to a weak training signal.
ReLU avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether sigmoid or ReLU can be difficult for gradient-based optimization. So, To solve this problem deep learning practitioners have invented a myriad of Normalization layers (batch norm, layer norm, weight norm, etc.). we can normalize the data in advance to be zero-centered as in batch/layer normalization.
Reference:
A Survey on Activation Functions and their relation with Xavier and He Normal Initialization


Identify "actionable" intents in email messages

This is a categorization problem, not unlike a spam filter. Instead of flagging an email as spam/not-spam, you are flagging whether it has one of the action categories that you have described.

You'll need to start by assembling a training corpus of example email messages and labeling each example to identify which (maybe multiple) of your categories, if any, are actually present in that email.

Next, pre-process that data to extract features for each message. Examples of typical features include word (or n-gram) counts/frequencies (bag of words). As a shortcut, you might to include as a feature a boolean indicating the presence or absence of a particular word or phrase that you suspect will be predictive of one or more categories. Techniques such as stemming can help reduce the number of words/n-grams being used (often increasing accuracy).

Once you have a dataset that consists of features and labels for each training email (possibly breaking this set up into subsets for training, cross-validation, and testing), you'll want to apply a supervised classification algorithm. You might start with linear classifiers such as logistic regression or SVMs, and if you're unsatisfied with the resulting accuracy then you could advance to neural techniques.


Why does the discrepancy measure involve a supremum over the hypothesis space?

The formula $G=\mathbb{E}\left[ f(Z_{T+1}) \mid \mathbf{Z}_1^T\right] - \sum_{t=1}^Tq_t \mathbb{E}\left[ f(Z_t) \mid \mathbf{Z}_1^{t-1} \right]$ actually represents a set, for all possible values of $f$. Therefore, $\text{disc}(\mathbf{q}) = \operatorname{sup}_{f \in \mathcal{F}} \left( \mathbb{E}\left[ f(Z_{T+1}) \mid \mathbf{Z}_1^T\right] - \sum_{t=1}^Tq_t \mathbb{E}\left[ f(Z_t) \mid \mathbf{Z}_1^{t-1} \right] \right)$ is the element not necessarily in that set $G$ that is greater than all elements in that set $G$, but is the smaller than any other element that is greater than any element in that set $G$. In other words, $\text{disc}(\mathbf{q})$ represents an upper bound on the discrepancy $G$, but it is the smallest possible upper bound. See also this answer for more details about the supremum and the relationship between the supremum and upper bounds.

So, unless I am wrong, this is not necessarily a worst-case analysis (but I am not even sure how one would define the worst-case analysis in this context), but we are looking for the smallest upper bound. In the context of algorithms, the worst-case analysis is with respect to the input (i.e. the worst-case input), but you can have upper or lower bounds for worst and best-case scenarios. See, for example, this answer that illustrates this. Why do we want an upper bound? Because we can be certain that we won't have a generalization error worse than it. You can't do this with e.g. a lower bound.


Is there any artificially intelligent system that really mimics human intelligence?

As far as emulating an intelligent being, no. There are a few different potential architectures for possible AGI. Many of these are extremely infantile, as the bulk of AI research is in narrow AI, which focuses on creating algorithms that are highly specialized for a specific task.

With that being said, here is one supervised learning approach to this problem by Andrej Karpathy: Karpathy Method

OpenAI, also has a team who works on the AGI problem.


Do we need non-linear activation function in neural networks whose task isn't classification?

Yes, definitely.
In the simplest example, predicting an output value for a time series is classification. You take in the previous time steps and classify what is the most likely next value. You could do this with a RNN (Recurrent Neural Network) for example.
If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.
As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a ReLU (Rectified Linear Unit) you can quite easily approximate this function. See this implementation on desmos.
If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.


What research has been done in the domain of “general game playing”?

If you haven't already come across DeepMind's advances in developing general game playing AI, you can take a look at it's DQN research. The paper describes how their deep reinforcement learning system is able to beat human levels in nearly Atari 2600 games with raw pixels and scores as input.

Also here's an interesting website - General Video Game AI Competition 2017 that also hosts links to the latest advances and research papers in this field.


Do off-policy policy gradient methods exist?

Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions.

The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio ($0 \leq \rho \leq 1$) which can scale the update to the target policy by the probability of the target policy taking that action. A higher $\rho$, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A $\rho$ of $0$, and the update is ignored.


Do text compression tests qualify winRar or 7zip as intelligent?

The paper suggests an alternative test to the famous Turing test, which tests a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human.
In this test, if winRar or 7zip will compress a file similarly to how a human would compress a file (how does a human compress a file?!), then, yes, those programs will pass the test and will be considered intelligent.

... thus compression ratio on a standard benchmark corpus could be used as an objective and quantitative alternative test for AI (Mahoney, 1999).


What's the role of bounding boxes in object detection?

A bounding box is a rectangle superimposed over an image within which all important features of a particular object is expected to reside. It's purpose is to reduce the range of search for those object features and thereby conserve computing resources: Allocation of memory, processors, cores, processing time, some other resource, or a combination of them. For instance, when a convolution kernel is used, the bounding box can significantly limit the range of the travel for the kernel in relation to the input frame.

When an object is in the forefront of a scene and a surface of that object is faced front with respect to the camera, edge detection leads directly to that surface's outlines, which lead to object extent in the optical focal plane. When edges of object surfaces are partly obscured, the potential visual recognition value of modelling the object, depth of field, stereoscopy, or extrapolation of spin and trajectory increases to make up for the obscurity.

  A classifier can only classify one object per image

A collection of objects is an object, and the objects in the collection or statistics about them can be characterized mathematically as attributes of the collection object. A classifier dealing with such a case can produce a multidimensional classification of that collection object, the dimensions of which can correspond to the objects in the collection. Because of that case, the statement is false.

  1) If I don't need to know 'where' an object is (or objects are) and just interested in the existence of them and how many there are, is it possible to just get rid of the boundary boxes?

If you have sufficient resources or patience to process portions of the frame that don't contain the objects, yes.

Questions (2) and (3) are already addressed above, but let's look at them in that context.

  2.a) If not, how does bounding boxes help detect objects?

It helps by fulfilling its purpose, to reduce the range of the search. If by thrifty method a bounding shape of any type can be created, then the narrowing of focus can be used to reduce the computing burden on the less thrifty method by eliminating pixels that are not necessary to peruse with more resource-consuming-per-pixel methods. These less thrifty methods may be necessary to recognize surfaces, motion, and obscured edges and reflections so that the detection of object trajectory can be obtained with reliability.

That these thrifty mechanisms to find the region of focus and these less thrifty mechanisms to use that information and then determine activity at higher levels of abstraction are artificial networks of this type or that or use algorithms of this type or that is not relevant yet. First understand the need to reduce computing cost in AI, which is a pervasive concept for anything more complex than tic-tac-toe, and then consider how bounding boxes help the AI engineer and the stakeholders of the engineering project to procure technology that is viable in the market.

  2.b) From what I have figured is that a network (if using neural network architectures) predicts the coordinates of the bounding boxes if there is something in the feature map. Doesn't this mean that the detector already knows where the object is (at least briefly)?

  2.c) So continuing from question 1, if I'm not interested in the exact location, would training for bounding boxes be irrelevant?

Cognition is something AI seeks to simulate, and many hope to have robots like in the movies that can help out and be invaluable friends, like TARS in the Nolan brothers 2014 film Interstellar. We're not there. The network knows nothing. It can train a complex connection between an input signal through a series of attenuation matrices and activation functions to produce an output signal statistically consistent with its loss function, value function, or some other criteria.

The inner layers of an artificial net may, if not directed to do so, produce something equivalent to a bounding region only if velocity of convergence is present as a factor in its loss or value function. Otherwise there is nothing in the Jacobian leading convergence to reduce its own time to completion. Therefore, the process may complete, but not as well as if cognition steps in and decides that the bounding region will be found first and then used to reduce the total burden of mechanical (arithmetic) operations to find the desired output signal as a function of input signal.

  3) Finally, in architectures like YOLO, it seems that they predict the probability of each class on each grid (e.g. 7 x 7 for YOLO v1). What would be the purpose of bounding boxes in this architecture other than that it shows exactly where the object is? Obviously, the class has already been predicted so I'm guessing that it doesn't help classify better.

Reading the section in A Real-Time Chinese Traffic Sign Detection Algorithm Based on Modified YOLOv2, J Zhang, M Huang, X Jin, X Li, 2017, may help further comprehension of these principles and their almost universal role in AI, especially the text around their statement, "The Network Architecture of YOLO v2 YOLO employs a single neural network to predict bounding boxes and class probabilities directly from full images in one inference. It divides the input image into S × S grids." This way you can see the use of these principles in the achievement of specific research goals.

Other such applications can be found by simply reading the article full text available on the right side of an academic search for yolo algorithm and using ctrl-f to find the word bound.


Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Is it just about final states? So for $s \in S$ when S is not final?

You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.

As this is a formal exercise from the book, I don't want to write out an answer that could be cut&paste for all students.

Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton & Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.


To complete the first answer that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds.

Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper Geometric deep learning: going beyond Euclidean data (this paper is excellent at clarifying both the intuition and the mathematics of what I'm writing).

1. In case you don't know what a manifold is

As the previously cited paper puts it:

  Roughly, a manifold is a space that is locally Euclidean. One of the
  simplest examples is a spherical surface modeling our planet: around a
  point, it seems to be planar, which has led generations of people to
  believe in the flatness of the Earth. Formally speaking, a
  (differentiable) d-dimensional manifold X is a topological space where
  each point x has a neighborhood that is topologically equivalent
  (homeomorphic) to a d-dimensional Euclidean space, called the tangent
  space.

Good other not-so-technical explanation on stats.stackexchange

Other Wikipedia examples to develop not too abstract understanding

Very shortly put, it's an interesting mathematical set on which to work (different kinds exist, see papers at the end of this answer for DL related manifolds uses). By work, you can typically understand that you constrain the neural net parameters to the manifold you chose (e.g. training with parameters constrained on a hypersphere, among the geomstats paper examples).

Your data can also be represented thanks to a practical manifold. For example, you can choose to work on images and videos by representing the samples using Symmetric Positive Definite (SPD) matrices (see this paper), the space of SPD matrices being a manifold itself.

2. Why bother learning on manifolds ?

Defining a clearer/better adapted set (understand that it's a sort of constraint!) on which to learn parameters and features can make it simpler to formally understand what your model is doing, and can lead to better results. I see it as a part of the effort of deep learning formalization. One could say you're looking for the best information geometry for your task, the one that best captures the desirable data distribution properties.To develop this intuition consider the solar system analogy for manifold learning of this Kaggle kernel:

  Perhaps a good analogy here is that of a solar system: the surface of our planets are the manifolds we're interested in, one for each digit. Now say you're on the surface of the earth which is a 2-manifold and you start moving in a random direction (let's assume gravity doesn't exist and you can go through solid objects). If you don't understand the structure of earth you'll quickly find yourself in space or inside the earth. But if you instead move within the local earth (say spherical) coordinates you will stay on the surface and get to see all the cool stuff.

This analogy reminds us of the spherical surface planet model from Bronstein's paper already quoted above. This paper also describes a typical case for which manifolds are interesting: where graphs (the other example of GDL/DL on non euclidean data) are better at handling data from social or sensor networks, manifolds are good at modeling 3D objects endowed with properties like color texture in computer vision.

3. Regarding deep neural networks on manifolds

I would advise reading the geomstats associated paper, which does a great job at showing what it is and how it can be used, along with example codes (e.g. MNIST on hyperspheres manifold example code here). This library implements manifolds and associated metrics on Keras. The choice of metrics is essential to understand the point of working on manifolds: it's because you need to work on an adapted mathematical set (ie with the right properties) with an adapted distance definition (so that the measure actually means something when considering the problem you're trying to solve) that you switch to working on manifolds.

If you want to dive in the details and examples of deep learning on manifolds here are some papers:

A Riemannian Network for SPD Matrix Learning: new backpropagation to learn SPD matrices on Riemannian manifolds
Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds: using SPD matrices to aggregate convolutional features

4. Why Riemannian manifolds ?

TL;DR: you need a metric to do machine learning (otherwise, how could you evaluate how much you actually learned !)

Still based on Bronstein's paper:

  On each tangent space, we define an inner product [...]. This inner
  product is called a Riemannian metric in differential geometry and
  allows performing local measurements of angles, distances, and
  volumes. A manifold equipped with a metric is called a Riemannian
  manifold.

5. What's the relation between a Riemannian manifold and a Euclidean space ?

Still based on Bronstein's paper:

  a Riemannian manifold can be realized as a subset of a Euclidean space
  (in which case it is said to be embedded in that space) by using the
  structure of the Euclidean space to induce a Riemannian metric.

I leave the details to the paper, otherwise this answer will never end.

6. Answers to questions in comments

Will only answer once I think I've found a relatively well-argued answer, so won't answer everything at once.

Isn't manifold learning just a way of dimensionality reduction?

I don't think so, it isn't just that. I haven't seen any dimensional reduction constraint (yet ?) in the papers I've read (cf. geomstats again).

In the hypersphere/MNIST geomstats code example, you can see the chosen manifold dimension hypersphere_dimension = 17. Since we're working with MNIST data I guess this would mean a dimension reduction in this particular case. I admit I would need to check exactly what that dimension implies on the neural net architecture, I haven't discussed my understanding of this yet.

Disclaimer

I'm still developing a more rigorous mathematical understanding of manifolds, and shall update this post to make additional necessary clarifications: exactly what can be considered as a manifold in a traditional deep learning context, why do we use the word manifold when speaking about the hidden state of auto-encoders (see the previously cited Kaggle kernel that quotes Goodfellow's book on this). All of this if the perfectly clear answer doesn't show up here before !


What regulations are already in place regarding artificial general intelligences?

I don't know that it has yielded any actual reports or regulations yet, but in the USA, the White House has been running a series of interagency workshops / working groups dedicated to Preparing for the Future of Artificial Intelligence.

Some of those sessions have been dedicated to legal / governance issues.


Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:

the odd numbers
the primes
numbers expressed in digits, but listed in alphabetical order of their name in German
numbers listed in the lexical order of the reverse of their representation in base 3
the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber


Can we identify only the objects in specific parts of an image with computer vision?

Giving this small rectangle to a model will probably not provide him enough information to identify what is inside this rectangle. An approach that could work would be to do image segmentation, having the identity of everything in the image, and then extract the label of the pixels in the rectangle.


Generation of 'new log probabilities' in continuous action space PPO

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.
This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first repeat, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these epoch update steps, whereas your new probabilities will keep changing.


In UCB, is the actual upper bound an upper bound of an one-sided or two-sided confidence interval?

The upper bound used here is derived from Hoeffding's inequality, which provides a symmetric, two-sided confidence interval. A good pair of blog posts on how this bound used in UCB for bandits is derived can be found here:

First steps: Explore-then-Commit
The Upper Confidence Bound Algorithm

Indeed, in practice when using this UCB for bandits, we do not actually care about the lower bound. We only need the upper found for the exploration mechanism. But the lower bound does still exist, even if we don't use it.


Can we implement GAN (Generative adversarial neural networks) for classication problem like Fraud detecion?

Generative networks was the first popular class of topologies that had compound feedback, corrective signaling at more than one layer.  In popular network designs such as MLPs (multilayer perceptrons), CNNs (convolutional neural networks), and LSTM (long short term memory) networks, the backpropagation is a single layer mechanism.  The mechanism distributes corrective a signal creating a closed loop that ideally converges to an optimal network behavior.

The speed, accuracy, and reliability of convergence of networks in an AI system depend on a number of engineering choices at a lower level of design and development.

Selection of algorithm variants and extensions
Selection of data set
Drawing of the training samples
Data normalization
Initialization
Assignment of learning rate or learning rate policy
Other hyper-parameters

Generative networks are designed to achieve a higher level balance between two networks, each of which have their own backpropagation, and thus have a compound feedback topology.

Considering the original GAN approach in the context of fraud detection, a discriminative network $D$, in a way, detects fraudulence originating from the generative network $G$, and, in a way, $G$ learns authenticity from the corrective signal provided by $D$, even if the authenticity learned is superficial.  However, in the case of credit card fraud detection from a numerical set of data, the fraudulence is not internal to the system to be designed as in the case of GAN.  Without both the $G$ and $D$ networks, the design is not a generative network.

Since the comment stated, "I have to explore GAN for this problem," the key question is this.

  If this system is to benefit from a GAN, what would the GAN generate that could contribute to the solution?

If the answer is nothing, then a GAN is not the correct approach.  A discriminative network without a generative network is not a GAN.  If this question is from an assignment to train a network using a static and already existing data set, a MLP may suffice, depending on the expectations of the teaching staff.  If that is the case, the goal will be to analyze the data set using statistical tools to determine if it is already normalized well enough to lead to excellent convergence accuracy, speed, and reliability.  If normalization is indicated, find ways to present the data to the input layer of the MLP in such a way that each input is relevant and has a well distributed range of values across the data type.

Ensure there is no data loss in this process, and remove redundancy where it can easily be removed.  By redundancy removal is meant, if one of the dimensions of data have only four possible values of between 10 and 1,000 bytes, encode the four possible values in two bits because to do so removes unnecessary burden from the training.

Similarly, the output layer must be chosen to produce an encoded set of possible output values and the decoding must be written to produce usable output.  In this case, it appears that the labels are, Fraudulent and Authentic.  Which means that the last layer would be a single binary threshold cell.

For this simple MLP approach and for many of the approaches mentioned next, there are examples available in Python, Java, C++ and other languages from which implementation can be initiated.  If the design must stand up to a more realistic fraud detection scenario, either for credit cards, junk mail, banking, log-in, or communications those experienced with fraud detection know that the above approach is entirely insufficient except as a starting point for learning.  In such a case, these considerations further complicate the design of the solution.

Fraudulence and authenticity is not necessarily sufficient information for investigators.
    The likelihood of fraudulence and the reason why fraudulence is suspected is also pertinent for end users of production output,
    so that security holes can be blocked and investigation may be directed toward prosecution or some other set of remedies.
Fraudulence is usually a security breach phenomena that involves continuous improvement in the attacks until the attackers
    are located and their work is interrupted by arrest.
    What initially works as fraud detection may not work a week, a day, an hour, or a minute later.
Fraud detection, if it to be useful in a real financial context, must be part of a fraud prevention system,
    and training on a static data set may be meaningless in that larger usage environment.

In AI systems that work in real production environments, it may be worth investigating using an LSTM network in combination with a reinforcement learner.  Such may be over a beginner's level of design and implementation experience, but using either an LSTM network or a basic Q-learning approach instead of both of them together may be a reasonable starting point for dedicated students.  If the answer to the question about what a GAN could generate that would be of use must be yes, the following ideas may be helpful.

A generative network could generate transaction data with a distribution that resembles authentic transactions, and such has been successful in the past.
A generative network could generate transaction data with a distribution that resembles attacks on an account by a fraudulent user. That has been done too.
A generative network could theoretically generate hacker agents, although this may not have yet been done.
A generative network could be used to generate patterns of authentic transactions based on some authenticity measure other than the data set mentioned, and the trained discriminative network could then be employed in some way in a fraud detection system in production, although it is unclear what advantage would be gained.

The reason a GAN was suggested would need to be clarified before this answer could provide a more exact direction for AI system topology and project approach.


How are "lags" and "exogenous factors" accounted for in reinforcement learning?

One of Reinforcement Learning's core features is the ability to deal with delayed rewards/punishments - i.e. rewards that may occur as a consequence of a decision that occurred multiple time steps ago. That is because the value that it optimises is defined as a long-term sum of immediate rewards. This value is often called the return or utility and although it will take short term rewards into account, they do not have to be the dominant factors, and often are not.

All RL solvers are designed to solve this problem, it is core to the MDP formalism used in RL theory. You can decide to focus on immediate rewards or longer-term ones by adjusting using a discount factor (often represented as $\gamma$ in RL equations). Low discount factors will cause an agent to prefer short-term rewards, high discount factors will cause the agent to prefer for longer-term sums of reward. In continuous problems you must use $\gamma \lt 1$ or instead use average reward as the objective to be maximised.

One classic example of this kind of delay problem is the Mountain Car environment. Here, reward is only granted for reaching a certain point, and it is not possible to reach it by just taking the obvious action of moving directly towards the objective. Most RL algorithms can solve this problem, and get close to optimal solutions, purely from experiencing the final reward and associating it progressively with earlier and earlier decisions.

  In practical systems how is this "lag" accounted for?

As above, this is inherent to RL, as opposed to simplified versions such as Contextual Bandits which don't have time steps and evolving state.

In practice, it may take some experimentation to find the best RL algorithm and solve a problem efficiently. Environments with very large delays and lots of noise are harder to solve.

It is also really important to set the reward mechanism for your true goals, and not some intermediate ones. If your goal is a stable state, you need to reward that, and not necessarily maximum throughput. If both throughput and stability are important to the end result, you may need to reward both - e.g. reward throughput, but penalise short-term fluctuations.

  How are the un-settled (false) results ignored?

In general handling short-term vs long term effects of actions is known as the assignment problem.

This is most often solved statistically, over long term experience, as the agent associates not just immediate reward with an action, but the long term return after making each decision. Different RL solvers have different mechanisms for effectively moving this association back through time.

The key thing here is that an RL solver learns to maximise expected return or utility from any given state. Working with the expected value allows for smoothing out of variations, provided the agent gains enough experience to approach a statistical mean. Extremely rare and large variations can cause problems with this, and might need to be explicitly coded for as opposed to learned.

  How is this noise distinguished from exogenous factors (un-controlled system inputs e.g. an open window exposed to wind)?

This is trickier, and depends more critically on the problem.

A RL-based system that is exposed to these variations whilst learning, and some way to represent their effects in the state variables should learn to pick the best action when these external factors impact it.

RL systems based on maintaining a stable state can sometimes do well when they are moved out of that stable state. A good example here might be pole balancing, where the agent learns enough variations of position and speed that if something knocks the pole out of balance, it will still right itself even though the agent was not trained to react to such events directly.

However, in some cases, if knowledge is not available to the agent, or it has not been trained in that particular scenario, it will fail to generalise and may behave in counter-productive ways.

Either way, if you are concerned about specific scenarios, you need to include them in training, or at least test for them after training.


What are the best programming languages for developing evolutionary algorithms?

Most machine learning applications today are built on tensors, matrices, probabilistic / Bayesian inference, neural networks, etc. But those can all be built with any modern programming language (all the useful ones are Turing complete). And the best performing language for any of those will generally be assembly / machine code.

Python is famous for machine learning, but that may be due to adoption of Python in academia and NumPy, SciPy, etc. Python isn't very performant, but most of the machine libraries leverage native code, so they're fairly performant.

Julia is a new language that is gunning for a lead position in the data science space, which machine learning builds on. It is allegedly very performant over number crunching domains.

Java has a decent developer ecosystem, and is fairly performant, but the highest performing libraries (including those that leverage GPU) tend to call out to native code via JNI. See DeepLearning4J.

I personally like Clojure - a modern Lisp running on the Java JVM. There's a new deep learning project called Cortex built on Clojure and some fast native libraries, including GPU acceleration.

I think Clojure provides a great balance of being able to easily wrap performant libraries with highly expressive, succinct and simple programming idioms.


Can residual neural networks use other activation functions different from ReLU?

The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as saturating activation functions). For example, the sigmoid function has codomain $[0, 1]$, as you can see from the illustration below.

This property/behaviour can lead to the vanishing gradient problem (which was one of the problems that Sepp Hochreiter, the author of the LSTM, was trying to solve in the context of recurrent neural networks, when developing the LSTM, along with his advisor, Schmidhuber).
Empirically, people have noticed that ReLU can avoid this vanishing gradient problem. See e.g. this blog post. The paper Deep Sparse Rectifier Neural Networks provides more details about the advantage of ReLUs (aka rectifiers), so you may want to read it. However, ReLUs can also suffer from another (opposite) problem, i.e. the exploding gradient problem. Nevertheless, there are several ways to combat this issue. See e.g. this blog post.
That being said, I am not an expert on residual networks, but I think that they used the ReLU to further avoid the vanishing gradient problem. This answer (that I gave some time ago) should give you some intuition about why residual networks can avoid the vanishing gradient problem.


Actually it depends on the size of your AE, if you use a small AE with just 500.000 to 1M weigths, the inferencing can be stunningly fast. But even large networks can run very fast, using Tensorflow lite for example, models are compressed and optimized to run faster on Edge-devices (Handys for example, end-user devices). You can find a lot of videos on Youtube, where people test inferencing large networks like Resnet-51 or Resnet-101 on a raspberrypi, or other SOC Chips. Handys are comparable to that, but maybe not that optimized.

For example,I have an Jetson Nano (SOC of Nvidia costs arround 100 euro) and i tried to inference a large Resnet with arround 30 million parameters over my fullHD Webcam. Stable 30 FPS, so speaking in milliseconds its around 33 ms per image.

To answer your question, yes Autoencoders can be fast, also very fast in combination with an optimized model and hardware. Autoencoder structures are quite easy, check out this medium,keras example


Why does the variational auto-encoder use the reconstruction loss?

The VAE uses the ELBO loss, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of your data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term jointly (i.e. the ELBO). Why? Because, as I just said, the ELBO is the Evidence Lower BOund on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read this answer for further details.

  Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.

The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples $z$ (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, $z$ is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss.

In the mathematical formulations, you will see that the likelihood term of the ELBO is $p(x \mid z)$, i.e. the likelihood of the input $x$ given $z$. The $z$ is the input to the decoder, which produces a reconstruction of $x$. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see this PyTorch implementation), which should correspond to this likelihood term $p(x \mid z)$. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, $p(x \mid z)$ does not appear, but the logarithm of $p(x \mid z)$ appears, but, for simplicity, I have used $p(x \mid z)$ rather than $\log p(x \mid z)$ above.)


Is there a way to perform pattern recognition without a labeled training set?

You should look into unsupervised learning, which is machine learning without a human-labeled training set.


Book recommendations on deep learning (convolutional neural networks)

If you are new to neural networks, I recommend studying the free online book Neural Networks and Deep Learning. It teaches you the basic concepts and the underlying math - a great starting point to dig deeper.

Once you understand the basic concepts, I recommend watching the Stanford Lecture CS231n about Convolutional Neural Networks for Visual Recognition. It consists of 16 lectures and teaches you most of the things you'll need to know for your project. The also provide all the slides - check out the links below each video.

Analyzing existing projects and their documentation is a good next step for deeper understanding of what's state of the art in CNN. You could start with this article about Q-Learning based on a CNN architecture, but there are many more out there.

More specific questions will come during your project. You'll find more information in scientific papers and blogs from researchers. Feel free to ask here as well, if you don't find what you are looking for elsewhere.


Is Deep Learning the repeated application of Linear Regression?

A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any  neural network specific techniques such as convolution, residuals, etc., as shown below:
$\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + b_n) \\
= W_n (W_{n-1} \dots (W_1 x + b_1) + \dots + b_{n-1}) + b_n \\
= \left( W_n W_{n-1} \dots W_1 \right) x + \left( W_n W_{n-1} \dots W_2 \right) b_1 + \left( W_n W_{n-1} \dots W_3 \right) b_2 + \dots + W_n b_{n-1} + b_n \\
= W_z x + b_z$
where $W_z = \displaystyle \prod_i W_i$ is a weight matrix and $b_z$ is some vector constant.
This follows the linear regression model form $y = Wx + b$, where $W$ is the weight matrix and $b$ is the vector constant. As a result, you can analytically solve for $W_z$ and $b_z$ using linear regression techniques, and no longer need gradient descent.
Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed (more info). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict (more info*).
*Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.
The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.


Is there a way to calculate the closed-form expression of the function that a neural network computes?

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear.

The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "$\sin(\pi x + \phi) +4$" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?


How do LSTM and GRU avoid to overcome the vanishing gradient problem?

LSTMs solve the problem using a unique additive gradient structure that includes direct access to the forget gate's activations, enabling the network to encourage desired behaviour from the error gradient using frequent gates update on every time step of the learning process.


How to optimize neural network parameters with REINFORCE

The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.
If you're using REINFORCE to estimate policy gradients in RL (this is the common use case), your parameters are parameterizing a policy function. The inputs are states and the outputs are actions. The issue with estimating the gradient of the parameters is as follows. In RL the objective is to maximize expected rewards. This is an expectation with respect to the policy of the cumulative reward of a trajectory. If you take the gradient of this wrt the parameters, since the expectation is wrt the policy which depends on the parameters, you can't move the gradient inside the expectation so you can't estimate the gradient from samples. Using REINFORCE however, you use the "log-derivative" trick to rewrite the objective as an expected gradient, which can be estimated from samples.


AI on Academic vs Industry vs Competition

I cannot comment about the process for AI for academia. I can compare AI for competitions and AI for business. To clarify whatever I say is about ML not any other AI techniques. The process might be different for other techniques. But most of things that I say are general enough that I am assuming should still apply.

The main difference that I saw while doing ML for a competition vs. for a business was that of focus.

When doing it for a competition for Kaggle the focus was mainly creating the model

machine learning metrics are specified for you
some data was given to you
business problem was given to you

When doing it for business what is different

given a business problem finding the parts that can actually benefit from ML. You have to define the ML problem in it and define how it actually benefits the business. This may involve significant discussions with business stakeholders, weighting the pros and cons of doing it versus doing something else, communicate the benefits to the business stakeholders, take them into confidence for the process to start
find the right data for the problem from scratch, ensure it is collected by rest of the system or brought from 3rd parties
define business metrics over and above machine learning metrics. At the end of day nobody really cares about whether ML model recall, accuracy is good or bad. What is important is the relevant business effect.
make the model, deploy it and integrate it with the rest of the system. This is important because if your goal is just to making the model you would not care about the factors associated with actually using it i.e. latency of predictions, cost of machines needed to run it etc.
A/B testing for the models, running multiple models in parallel, dynamically being able to adjust which models to use

Hope this gives some idea about the differences in AI for competitions and AI for business.


How to perform insect classification given two images of the same insect?

There are several ways you can do this.

One is to input both images in input, so it can be a 2 input system or an input with 6 channels.

As you suggested in 1st point, you can make 2 networks, connect them at the end and add another layer for final classification or use outputs from both and train another classifier (like Gradient bosting). You can look up ensemble techniques and ensemble of neural networks for more.

For dual classification and naive approach, you will again have a problem in case there are some insects that can solely be identified from features at a specific angle, if so you can then discard other angles (e.g. ignore moth face if moth back is the distinguishing feature)


How to train a LSTM with multidimensional data

I don't see any special characteristic in the problem you're posing. Any LSTM can handle multidimensional inputs (i.e. multiple features). You just need to prepare your data such as they will have shape [batch_size, time_steps, n_features], which is the format required by all main DL libraries (pytorch, keras and tensorflow).

I linked below 2 tutorials that shows how to implement an LSTM for part of speech tagging in Keras and Pytorch. This task is conceptually identical to what you want to achieve: use 2D inputs (i.e. embeddings) to predict the class (i.e. the pos tags) of each element of a sequence (i.e. every single word). For example:

# Input sample, 4 steps, encoded in 2D embeddings matrix
['we', 'had', 'crispy', 'fries']

# Output predictions, 4 labels
['PRP', 'VBD', 'JJ', 'NNS']

In your case the predicted label would be 0, 1 or 2 and you would have to encode your data in a matrix of shape [n_batch, 600, 5]. The only thing you definetly want to pay attention, since you're using temporal data, is to not shuffle your data at all before the training.

Keras tutorial: Build a POS tagger with an LSTM using Keras

Pytorch tutorial: LSTM’s in Pytorch


Which problems are considered to be the toughest problems in Artificial Intelligence or Machine Learning?

There are many problems with the help of AI we could move to a solution.

An example that I found incredible is in Brazil, where a group of friends decided to create an AI to monitor the transactions of the politicians. They called it the Serenata de Amor and today they show to the whole world various scams and money being used for personal purposes and not how it should be used.

http://serenatadeamor.org/en/

There is a big difference between thefts that replace human work and Artificial Intelligence. An artificial intelligence can rather replace human labor, but its capacity goes much further. Today an AI can discover new ways of working, finding more productive ways to accomplish tasks. Learning from mistakes should be a human task. But an AI takes it very seriously.

In conclusion, anything you think might be better, which could be more productive, which is limited by relying on humans, is a problem that can be solved by an AI.


Is a non-linear activation function needed if we perform max-pooling after the convolution layer?

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function.

A linear function is defined as $y=f(x) = ax + b$, so $y$ linearly increases with $x$. Visually, $f$ corresponds to a straight line (or hyperplane, in the case of 2 or more input variables).

If $z$ does not correspond to such a straight line (or hyperplane), then it cannot be a linear function (by definition).

Let $x_1 = 1$ and let $x_2 \in [0, 2]$. Then $z=\operatorname{max}(x_1, x_2) = x_1$ for all $x_2 \in [0, 1]$. In other words, for the sub-range $x_2 \in [0, 1]$, the maximum between $x_1$ and $x_2$ is a constant function (a horizontal line at $x_1=1$). However, for the sub-range $x_2 \in [1, 2]$, $z$ correspond to $x_2$, that is, $z$ linearly increases with $x_2$. Given that max is not a linear function in a special case, it can't also be a linear function in general.

Here's a plot (computed with Wolfram Alpha) of the maximum between two numbers (so it is clearly a function of two variables, hence the plot is 3D).

Note that, in this plot, both variables, $x$ and $y$, can linearly increase, as opposed to having one of the variables fixed (which I used only to give you a simple and hopefully intuitive example that the maximum is not a linear function).

In the case of convolution networks, although max-pooling is a non-linear operation, it is primarily used to reduce the dimensionality of the input, so that to reduce overfitting and computation. In any case, max-pooling doesn't non-linearly transform the input element-wise.

The average function is a linear function because it linearly increases with the inputs. Here's a plot of the average between two numbers, which is clearly a hyperplane.

In the case of convolution networks, the average pooling is also used to reduce the dimensionality.

To answer your question more directly, the non-linearity is usually applied element-wise, but neither max-pooling nor average pooling can do that (even if you downsample with a $1 \times 1$ window, i.e. you do not downsample at all).

Nevertheless, you don't necessarily need a non-linear activation function after the convolution operation (if you use max-pooling), but the performance will be worse than if you use a non-linear activation, as reported in the paper Systematic evaluation of CNN advances on the
ImageNet (figure 2).


What is wrong with this CNN network, why are there hot pixels?

I've seen this too many times - it's not a problem with your network, it's a problem with matplotlib and how it displays the image. You are probably trying to display a float with range $<0, 255>$. When matplotlib sees float type as input, it assumes a range of $<0, 1>$, and thresholds everything outside of that range, and the results you can see.


What makes neural networks so good at predictions?

Neural networks are good at classifying. In some situations that comes down to prediction, but not necessarily.

The mathematical reason for the neural networks prowess at classifying is the universal approximation theorem. Which states that a neural network can approximate any continuous real-valued function on a compact subset. The quality of the approximation depends on the number of neurons. It has also been shown that adding the neurons in additional layers instead of adding them to existing layers improves the quality of the approximation faster.

Add to that the not well-understood effectiveness of the backpropagation algorithm and you have a setup then can actually learn the function that the UAT promises or something close.


What should I do when the potential value of a state is too high?

I don't think the situation you're sketching should be a problem at all.

If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the state $s$, that you expect to be able to get high future returns from that state.

If you then continuously take actions that keep you in the same state $s$, it is essentially correct to punish these actions (with negative rewards); you were expecting to get high future returns starting from that state, but you just remain stuck in that state. This means that the actions you're taking are not providing the rewards you were expecting, so it should be "punished".

Of course, the exception to the above paragraph is the case where the transition from $s$ to $s$ (staying in the same state) does result in a positive "true reward" (actual/real reward, not shaping-reward). In that case, the true reward will offset the reduction in potential and result in a neutral or positive combined reward if it is sufficiently large.

As for the $\gamma$ parameter, a $\gamma < 1.0$ (such as $\gamma = 0.99$) intuitively means that you prioritize short-term rewards over long-term rewards (if the magnitudes of the undiscounted rewards are similar). This implies that what you wrote in the question is precisely what should happen; if you prioritize short-term rewards over long-term rewards, it's bad to "waste time" by staying in the same state, so actions that don't move you anywhere should be disincentivized through negative rewards. If $\gamma = 1.0$, you have no preference for short-term rewards over long-term rewards, so you'll be fine with "wasting time" by staying in the same state, and therefore no longer have to punish such actions.


Feasibility of generating large images with a convnet

The way you would recognize images (image -> genre) would be very different from the other way around (genre -> image). For the former you are correct on what would be involved.

For the latter, if you want large images then GANs are indeed the way to go. Currently the largest images we can generate are on the order of 220 x 220 pixels, mostly due to memory constraints on the GPU. There is no fundamental problem with having rectangular images, it just so happens that we use squares. You would be able to use identical architectures to train on rectangular data as well.

The reason that some generated images you see (e.g. from DeepDream or NeuralStyle) are larger is that it's not a GAN. Both of these are not generative models in a standard sense, even if they do technically "generate" images. Instead, they merely modify an existing image by running backprop on a specifically designed loss function.

TLDR: your movie genre recognition idea is sound. For generation, have a look at things like this (https://github.com/Newmu/dcgan_code), where Alec generated album covers instead of posters. If you have enough data it will do something half sensible.


How do I design the rewards and penalties for an agent whose goal it is to explore a map

Measure what you want to achieve as directly as possible, and reward that. Later you can add more sophisticated incentives for the type of motion etc, but the key to a good reward signal is that it measures the quality of a solution at a high level, without specifying how to achieve that solution.
If you want your simulated car to explore, you will want to give it a reward signal based on it encountering new unexplored areas. There are lots of reasonable choices here. I suspect a good one will depend on what sensors you can reasonably code for the car, and what you consider to count as exploration - e.g. is it a thorough search of an area, moving far from the original position, experiencing different "views"?
One likely component you will need to give your agent and incorporate into the state representation is a memory. In order to understand whether the agent is exploring, something will need to know whether the agent has experienced something before and how much. A very simple kind of memory would be to add counters to a grid map and allow the agent to know how many time steps it has spent in each position on the map. The reward signal can then be higher when the agent enters a point on the map that it has not been in recently. If you want a non-episodic or repeating tour of exploration you might decay the values over time, so that an area that has not been visited for a long time counts the same as a non-visited one.
A related concept that you might be able to borrow ideas from is curiousity. There have been some interesting attempts to encourage an agent to seek new/interesting states by modifying action selection. Curiosity-driven Exploration by
Self-supervised Prediction is one such attempt, and might be of interest to you. In that case, the authors use an intrinsic model of curiousity that can solve some environments even when there is no external reward signal at all!
Alternatively, if you don't care to get involved in technical solution, you could create a maybe acceptable behaviour for your vehicle by setting a random goal position, then granting a reward and moving it to a new random location each time the car reaches it.


Why does PCA of the vertices of a hexagon result in principal components of equal length?

Assuming that the $6$ vertices of the hexagon are on the unit circle,
>>> from sympy import *
>>> A = Matrix([[ 1, Rational(1,2),-Rational(1,2), -1, -Rational(1,2), Rational(1,2)],
                [ 0,     sqrt(3)/2,     sqrt(3)/2,  0,     -sqrt(3)/2,    -sqrt(3)/2]])
>>> A * A.T
Matrix([[3, 0],
        [0, 3]])

Since ${\rm A} {\rm A}^\top - 3 \, {\rm I}_2 = {\rm O}_2$, any two orthogonal directions could be the principal components.


Can we increase the speed of training a reinforcement learning algorithm?

Can we make the game faster so that model will be training faster?

It depends on how much processing is required to run the simulation, how efficient that is implemented in whichever library you have loaded, and whether there is anything non-necessary for training that you can disable. Some environments for instance deliberately run "real time" so humans can appreciate the video output, and that is not necessary for training purposes (unless you want to experiment with real-time robotics).

For OpenAI Gym, there is one thing you can usually do: Switch off the rendering. The rendering for environments like CartPole slows down each time step considerably, and unless you are learning using computer vision processing on the pixels output, the agent does not need the pictures. You may even notice during training that moving the rendering window so it is not visible will speed up the training process considerably.

What I did for CartPole, LunarLander and a couple of similar environments is turn off rendering for 99 out of 100 episodes, and render just one of them in 100 to help me monitor progress. For Q learning, I also picked that to be a "test" episode where I stopped exploration.

Another option for speeding up training is to run a distributed system with multiple simulations at once. You will need mechanisms to share collected data too, so it is more work, but it is another approach to take if the simulation steps are the bottleneck for training speed.


What's the best architecture for time series prediction with a long dataset?

Yes, LSTM are ideal for this. For even stronger representational capacity, make your LSTM's multi-layered.
Using 1-dimensional convolutions in a CNN is a common way to exctract information from time series too, so there's no harm in trying. Typically, you'll test many models out and take the one that has best validation performance.


Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?

The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of the model.

This is generally not possible with simple neural networks as they simply do not have this property, for this you need a Bayesian Neural Network (BNN). This kind of network learns a distribution of weights instead of scalar or point-wise weights, which then allow to encode model uncertainty, as then the distribution of the output is calibrated and has the properties you want.

This problem is also called out of distribution (OOD) detection, and again it can be done with BNNs, but unfortunately training a full BNN is untractable, so we use approximations.

As a reference, one of these approximations is Deep Ensembles, which train several instances of a model in the same dataset and then average the softmax probabilities, and has good out of distribution detection properties. Check the paper here, in particular section 3.5 which shows results for OOD based on entropy of the ensemble probabilities.


At what point are AWS GPU instances worth it compared to CPU, price-wise?

The answer is it depends.
Are you referring to GPU vs CPU when training, or running inference?
When you say CPU is it an x86 CPU or ARM CPU?
What algorithms are running?
Let's look at image classification. Suppose you select a Convolutional Neural Network(CNN) as your algorithm. CNN's were created in the 1980s, but only really saw use when GPU's were used. In 2004 K. S. Oh and K. Jung showed that a GPU was 20x faster than a CPU. In 2005 Steinkrau et al showed a 3x speed increase.
Dan Ciresan et al showed in 2010 and 2011 that a GPU can be up to 60 times faster than a CPU. They used the MNIST handwritten digits benchmark. A simple google search of performance CPU vs GPU will result in many papers that show how much faster a GPU is compared to a CPU.
In this paper Eric Lind and Avelin Parnigoso compare the performance differences between a CPU and a GPU using TensorFlow. In their study they used a Core i7-6500 2.5 GHz CPU and an Nvidia GTX 1070 GPU. In their study they used three CNN AlexNet, Text Classifiction, and the Mnist Digit Classification.
Eric and Avelin showed that GPU outperforms CPU when training, and that during inference the GPU can be slower. In their study AlexNet for example ran for 4 000 000ms on a CPU and under 100 000ms on a GPU. That's a 40x difference in training performance. Converting this to cost. Eric and Avelin did not mention how much RAM.
I am going to make some assumptions and select the following instance types from AWS for training:

ml.g4dn.xlarge [GPU Accelerated] \$0.736/hour [4 vCPU, 16 GiB RAM]
ml.c5n.xlarge \$0.302/hour [4 vCPU, 10.5 GiB RAM]
ml.c5n.2xlarge \$0.605/hour [8 vCPU, 21GiB RAM]

The c5n's are powered by Intel 3.0 Ghz Xeon CPUs and the g4dn feature Nvidia T4 GPUs.
Based on this GPU Accelerated will run at less than 100 000ms or 100s while non GPU accelerated will run at 4 000 000ms or 4 000s. The result is:

ml.g4dn.xlarge - estimated cost \$1.226
ml.c5n.xlarge - estimated cost \$20.133
ml.c5n.2xlarge - estimated cost \$40.333

Based on the experiments showed by Eric and Avelin the CPU instance must be 40x less in cost to be on par with a GPU instance. As you can see the standard CPU is 20 and 40 times more expensive than the GPU. The thought experiment above is based on the paper times for training, not inference, and naturally to get a better answer should be benchmarked.
However, AWS has released their own ARM based CPU's designed specifically for training, and inference.

AWS Graviton Processor
AWS Inferentia
AWS Trainium

Both the Inferentia and Trainium chips offer 30% higher throughput and 45% lower cost per inference than Amazon EC2 G4.


Counterexamples to the reward hypothesis

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.