Why is Lisp such a good language for AI?

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as a scripting language driving big data or machine learning applications, Guile might be a useful replacement to Python and is also a Lisp dialect. BTW, I do recommend reading SICP, an excellent introduction to programming using Scheme.

Then, Common Lisp (and other dialects of Lisp) is great for symbolic AI. However, many recent machine learning libraries are coded in more mainstream languages, for example TensorFlow is coded in C++ & Python. Deep learning libraries are mostly coded in C++ or Python or C (and sometimes using OpenCL or Cuda for GPU computing parts).

Common Lisp is great for symbolic artificial intelligence because:

it has very good implementations (e.g. SBCL, which compiles to machine code every expression given to the REPL)
it is homoiconic, so it is easy to deal with programs as data, in particular it is easy to generate [sub-]programs, that is use meta-programming techniques.
it has a Read-Eval-Print Loop to ease interactive programming
it provides a very powerful macro machinery (essentially, you define your own domain specific sublanguage for your problem), much more powerful than in other languages like C.
it mandates a garbage collector (even code can be garbage collected)
it provides many container abstract data types, and can easily handle symbols.
you can code both high-level (dynamically typed) and low-level (more or less startically typed) code, thru appropriate annotations.

However most machine learning & neural network libraries are not coded in CL. Notice that neither neural network nor deep learning is in the symbolic artificial intelligence field. See also this question.

Several symbolic AI systems like Eurisko or CyC have been developed in CL (actually, in some DSL built above CL).

Notice that the programming language might not be very important. In the Artificial General Intelligence research topic, some people work on the idea of a AI system which would generate all its own code (so are designing it with a bootstrapping approach). Then, the code which is generated by such a system can even be generated in low level programming languages like C. See J.Pitrat's blog, which has inspired the RefPerSys project.


To deal with infinite loops, should I do a deeper search of the best moves with the same value, in alpha-beta pruning?

I think this issue stems from the fact you aren't taking position into account. I would think this because as the game progresses, the number of moves that will result in a piece being taken becomes less and less, especially when there's only a few pieces left and quite a bit of "chasing" must occur before a piece is taken, likely more chasing then a depth of 14 allows.

To remedy this, you could, towards the end of the game, add to the value of a state the inverse of the total distance each friendly piece has from other pieces, that way the agent will try to move towards other pieces and minimise this distance. If you find the right scale for this heuristic, the agent will prioritise moving towards enemy pieces only when it cant find any moves that result in taking a piece, helping it break out of this loop.


Does Rice's theorem prove safe AI is undecidable?

Every program halts, or continues

Given N steps, enough time and space (*), halting within N steps is provable

3 (from 2). Halting always has proof: run program until halt; count number of steps; verify claim of halting (within number of steps)

(Program is safe) implies (program is proved safe)

(Safety proved) implies (public understands the proof)

(Program is safe) implies (program always halts (safely) or continues (safely)) [from 1, 6] and ((public) understands (safety proof)) [from 7]

(Public not understand claimed safety proof this moment) implies (don't run program this moment) [common sense]

(*) This universe is finite. Some numbers too big to be computed in this universe
Have you seen perfect software?
Have you seen software make mistakes?
Why trust life-death decisions to software?
Why trust government decisions to software?
Why trust business decisions to software?
If scientists may not recognize AI is intelligent, what if you don't recognize something beyond AI in front of you?
(After enough doubt, all you can do is TRUST)


What is happening when a reinforcement learning agent trains itself out of desired behavior?

This is a common problem and does have a name. It is called "catastrophic forgetting" (link is just to a paper I found randomly when searching for the term).

  What could be causing this undesirable behavior?

It happens only when using function approximation for value functions (e.g. a neural network trained to learn Q values), and is caused by the agent's own success. After a while the only samples that you will train with will be near-optimal high return cases, and a neural network will optimise the Q value approximation only for that recent data in order to get the best loss. This will usually mean much poorer predictions on unseen but different data, as the network overfits to states and actions only seen in the optimal policy. Eventually the agent will explore into an area where its predictions are way off. Then, because Q learning also uses its own predictions to bootstrap new Q values, this can start a runaway feedback process where the agent starts to choose a suboptimal path through the environment.

Inside the hidden layers and weights, the neural network may have lost the ability to differentiate well between the states on the old, worse path and the newer better ones. It didn't need to differentiate, because it stopped needing to reduce loss on any data about the old states. So it will also start incorrectly associating the now poor results with the more optimal paths. It will behave at least partially as if the correct policy was set by the overfit Q predictions, but the values need adjusting - so as well as (correctly) reducing its value predictions of the suboptimal paths it has just encountered, it will also (incorrectly) reduce the value predictions of the optimal paths.

Sometimes, the swing back to receiving lower returns during this process is so strong and/or incorrectly associated with the high return states along with the low ones, that the agent never properly recovers. Other times, the agent can re-learn fully and you get random oscillations over time between optimal and non-optimal agent behaviour.

  How can I combat it?

This is still an ongoing area of research. Here are a couple of things that have worked for me:

Low learning rates, and defences against sudden large gradients (e.g. gradient clipping).
Regularisation. Sadly dropout seems not to work in RL, but weight decay is still useful to prevent over-fitting, and it also helps combat catastrophic forgetting because it prevents bootstrap estimates of long-unseen state/action combinations from returning radically different Q values to the rest of the system.
Keep some early experience around from when the agent was still performing badly - this allows the agent to still train with some bad cases and prevents the Q function predicting that "everything is awesome" because it still has examples to learn from where this is not the case.

For simple environments, such as inverted pendulum, just keeping some very early fully random behaviour in the experience replay table is enough. For instance if you have a table with 10000 observations (of $s, a, r, s'$), keep 1000 of the first experiences in that table and don't discard them when the table is full. For more complex environments, this is not so useful, as the early random behaviour is too far removed from what the agent learns.
The DQN "rainbow" paper uses prioritized experience replay to focus on areas where Q value predictions from the NN are not matching the observations.


Training a model to identify certain differences between images?

If your image just slightly changes, all you need is the simple algorithm, just find the keyword "k-Nearest Neighbor" or take a look at this link.
To locate where is the difference, you can find the difference by subtracting two images by this script:
def extract_diff(imageA, imageB):
    '''
    Find the different between two image:
        + Input: two RGB image
        + Output: binary image show different between two image
    Assume the different between two image in each channel will be bigger or equal 30
    '''
    subtract = imageB.astype(np.float32) - imageA.astype(np.float32)
    mask_motion = cv2.inRange(np.abs(subtract),(30,30,30),(255,255,255))
    # mask_motion[mask_motion==255] = 1 # scale to 1 to reduce computation
    return mask_motion

And locate where the position with value 255 in the image. Next time you should this kind of question in the stack overflow community, there is a more active community, they can give you a better solution than mine.


Question on identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.
However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because
$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$
by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.
Regarding your second question, I believe the author's appllied the Duelling architecture on Double Deep Q Networks, which is an improvement over the single DQN used by Minh et al in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.


Is continuous learning possible with a deep convolutional neural network, without changing its topology?

In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?

Your intuition that it is possible to perform incremental (aka continual, continuous or lifelong) learning by changing the NN's topology is correct. However, dynamically adapting the NN's topology is just one approach to continual learning (a specific example of this approach is DEN). So, there are other approaches, such as

regularization approaches (examples are LwF, EWC and SI)

ensemble approaches (examples are Learn++, FEL and PathNet)

rehearsal approaches (an example is iCaRL)

For more details about these and other approaches (and problems related to continual learning and catastrophic forgetting in neural networks), take a look at this very nice review of continual learning approaches in neural networks. You should also check this answer.

Are there ways to implement continuous learning in a deep neural network for image recognition?

Yes. Many of the approaches focus on image recognition and classification, and often the experiments are performed on MNIST or similar datasets (e.g. see this paper).

Does such an implementation make sense if the labels have to be specially prepared in advance?

Yes, you can prepare your dataset in advance, and then later train incrementally (in fact, in the experiments I have seen in some of these papers, they usually do this to simulate the continual learning scenario), but I am not sure about the optimality of this approach. Maybe with batch learning (i.e. the usual offline learning where you train on all data), you would achieve higher performance.


Reinforcement learning with PPO: rewards decreasing

I had the same problem where the reward kept decreasing and started to search for answers in the forum.

I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.

.

The fall continued until around 100k~ steps and did not change a lot for 250k~ steps.
After 350k~ th step, it slowly started to increase. Without knowing your number of steps trained, I would suggest training for more steps.

Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.


You can define different measures in this way:

Maximum distance of the new point with all points of the configuration ($M$)
Minimum distance of the new point with all points of the configuration ($N$)
$\frac{M}{\text{Maximum distance between two points of the configuration(}D)}$: normalize (1) measure
$\frac{N}{D}$:normalize (2) measure

You can get more ideas from distance measures in hierarchical clustering methods. To select a proper one, you need to elaborate on the context of these points.


Image classification with an associated matrix

The topic is multimodal neural networks.

Here some repositories that i hope will help me a lot

https://docs.google.com/presentation/d/1z8-GeTXvSuVbcez8R6HOG1Tw_F3A-WETahQdTV38_uc/edit#slide=id.g1ea5aac985_0_892

https://github.com/prml615/prml

https://github.com/husseinmozannar/multimodal-deep-learning-for-disaster-response

https://github.com/guillaume-be/multimodal-avito

Thanks to kbrose, his suggestion led me to this type of architectures.


I think the answer here lies in that the dictionary definition of randomness you have is not the one used in statistics, ML, or mathematics. We define randomness to mean there exists a distribution with generally greater than 0 uncertainty.

Depending on who you talk to, we live in a random universe (the way we define quantum mechanics depends on a wave function (essentially a probability distribution)

So why if a sequence is drawn from a distribution is it illogical? First, even as humans we can make a strong argument that what we say is random. I mean we speak to convey some form of message or context, but there exists multiple ways to deliver this, but we choose a single one. Our brains inherently model $p(\vec w|c)$ where $\vec w$ is the sequence and $c$ is our context or message we want to convey.

Takeaway: Generating a sequence in an ergodic or uniform manner would be illogical, but that is not what is being modeled or done in practice. Normally its drawn from some complex distribution.

Sidenote: My above claim could make it seem that being uniformly random implicates something illogical, and I want to emphasize that is not the case. It is domain to domain, sometimes that is the most logical solution, just in the case of sentence generation it normally isnt. I would define a logical algorithm as one that given the information at hand acts in a sensible manner towards achieving some goal, and so if something purely random does that, I don't see the problem.


Blocks world Game: A* heuristic solution approach

You may start assigning penalties for undesirable conditions in a state like:

1) Number of blocks outside stack 0.

Supose you penalize with 10 units each block outside stack 0, then the starting state above adds 40 units to the penalty score

2) Number of blocks in the stack 0 in a position different than in the goal state.

Supose you penalize with 50 units each block in stack 0 in the wrong position of the stack 0, then the starting state adds 50 to the penalty score (block D).

And go on
you may add as many penalties as you find helpful for solving the problem.

Then for each step you list all possible block moves, for each one of this moves you calculates the total penalty score that this move produces and choose the one move with the minimal score (if ties choose the first one).
You keep repeating above until you reach the final state (penalty 0).


Does this tutorial use normalization the right way?

You could rank every pixel in terms of brightness before normalization and after to verify that the ranks are preserved. Correct normalization would preserve the  ranks. From the pictures 1 and 2, it seems that the ranks were not preserved (e.g. the top right pixels went from grey to black).
The tutorial's normalization is done incorrectly: x_train is a an array of 2D images, but the normalization was applied along the first axis. What happened is that each column of every image was normalized relatively to itself (compare image 1 and 2 column-wise – the ranks were preserved). The normalization should have been applied along the second axis – that would normalize every pixel relative to the image it's in. In seems that incorrect normalization wasn't enough to sabotage the learning though!
Your proposed normalization should work fine.


What are the most pressing fundamental questions and gaps in AI research?

According to IBM Research organization in the response to the White House as part of preparing for the future of Artificial Intelligence, AI depends upon many long-term advances, not only from AI researchers but from many interdisciplinary teams of experts from many disciplines, including the following challenges:

Machine learning and reasoning.
Currently, AI systems use supervised learning using a huge amount of dataset of labelled data for training. This is very different to how humans learn by creating concepts, relationship, common sense reasoning which gives the ability to learn much without too much data. Therefore machine learning with common-sense reasoning capabilities should be researched further more.

Decision techniques.
Current AI-based systems have very limited ability for making decisions, therefore new techniques must be developed (e.g. modeling systemic risks, analyzing tradeoffs, detecting anomalies in context, analyzing data while preserving privacy).

Domain-specific AI systems.
The current AI-based system is lack of abilities to understand the variety of domains of human expertise (such as medicine, engineering, law and many more). The systems should be able to perform professional-level tasks such as designing problems, experiments, managing contradictions, negotiating, etc.

Data assurance and trust.
The current AI-based systems require huge amounts of data and their behaviour directly depends on the quality of this data which can be biased, incomplete or compromised. This can be expensive and time-consuming especially where it is used for safety-critical systems which potentially can be very dangerous.

Radically efficient computing infrastructure.
The current AI-based systems require unprecedented workloads and computing power which require the development of new computing architectures (such as neuromorphic).

Interpretability and explanations.
For people to follow AI suggestions, they need to trust systems, and this is only when they are capable of knowing users' intents, priorities, reasoning and they can learn from their mistakes. These capabilities are required in many business domains and professionals

Value alignment and ethics.
Humans can share the common knowledge of how the world function, the machine cannot. They can fail by having unintended and unexpected behaviour only because humans did not specify the right goals for them or they omitted essential training details. The systems should be able to correct the specification of the goals and avoid unintended and undesired consequences in the behaviour.

Social AI.
The AI-based systems should be able to work closely with humans in their professional and personal life, therefore they should have significant social capabilities because they can impact our emotions and our decision-making capabilities. Also, sophisticated natural language capabilities will need to be developed to allow a natural interaction and dialogue between humans and machines.

Source: Fundamental questions in AI research, and the most important research gaps (RFI questions 5 and 6). (link no longer available)


Is it okay to calculate the validation loss over batches instead of the whole validation set for speed purposes?

I assume you intended to write compute the evaluation metric over the validation set in batches; you do not compute loss over the validation set!
That is quite a standard practice in many academic implementations (because, when the validation set is large enough, the memory will be a constraint), however, be sure to take the average of the values over all the batches. Using a K-fold setup will increase the confidence in the reported values.


How can I detect datetime patterns in text?

Approaches

There are two main approaches to detecting any human readable representation of a discrete quantity within text.

Detect well known and stable patterns in the input stream and by adjacency determine the output stream.
Windowing through the text in the input stream and directly detect the quantities.

There are other approaches and there are hybrids of these two or one of these and the other approaches, but these two are the theoretically most straightforward and likely to produce both reliability and accuracy.

Re-entrant Learning

Whether the training involves re-entrant learning techniques, such as reinforcement, is a tangential issue that this answer will not address, but know that whether all training is solely a deployment component or whether adaptation and/or convergence occurs in real time is an architectural decision to be made.

Practical Concerns

Practically, the outputs of each recognition are as follows.

Starting index
Ending index
Integer year or null
Integer day of year or null
Integer hour in military time or null
Minute or null
Second or null
Time zone or null
Probability the recognition unit was correctly identified
Probability the recognition produced accurate results

Also practically, the input must either be from within one particular locale's norms in terms of

Calendar,
Time,
Written language,
Character encoding, and
Collation,

... or ...

The learning must occur using training sets that include the locales that will be encountered during system use

... or ...

Much of the locale specific syntax must be normalized to a general date and time language such as this:

जनवरी --> D_01

Enero --> D_01

Janúar --> D_01

so that Filipino and Icelandic names for the first month of the year enter the artificial network as the same binary pattern.

**Date and Time Specifically*

In the case of 1. above, which is semi-heuristic in nature, and assuming that the locale is entirely en-US.utf-8, the CASE INSENSITIVE patterns for a PCRE library or equivalent to use as a search orientation heuristic include the following.

(^|[^0-9a-z])(19|20|21)[0-9][0-9])([^0-9a-z]|$)
(^|[^0-9a-z])(Mon|Monday|Tue|Tues|Tuesday|Wed|Wednesday|Thu|Thur|Thurs|Thursday|Fri|Friday|Sat|Saturday|Sun)([^0-9a-z]|$)
(^|[^0-9a-z])(Jan|January|Feb|February|Mar|March|Apr|April|May|Jun|June|Jul|July|Aug|August|Sep|Sept|September|Oct|October|Nov|November|Dec|December)([^0-9a-z]|$)
(^|[^0-9a-z])(Today|Yesterday|Tomorrow)([^0-9a-z]|$)
(^|[^0-9])[AP]M|[AP][.]M[.]|Noon|Midnight)([^0-9a-z]|$)
(^|[^a-z])(0?[1-9])(:[0-5][0-9]){1,2}([^a-z]|$)

There should be others for time, hyphenated or slash delimited dates, or time zone.

The positions and normalized encoding of these date and time artifacts are then substituted into the artificial network inputs instead of the original text in the stream, reducing redundancy and improving both the speed of training and the resulting accuracy and reliability of recognition.

In the case of 2. above, the entire burden of recognition is left to the artificial network.  The advantage is less reliance on date and time conventions.  The disadvantage is a much larger burden placed on training data variety and training epochs, meaning a much higher burden on computing resources and the pacience of the stake holder for the project.

Windowing

An overlapping windowing strategy is necessary.  Unlike FFT spectral analysis in real time, the windowing must be rectangular, because the size of the window is the width of the input layer of the artificial network.  Experimenting with the normalization of input such that the encoding of text and data and time components entering the input layer could greatly vary the results in terms of training speed, recognition accuracy, reliability, and adaptability to varying statistical distributions of date and time instances and relationships.


Would minimizing influence into the world be a safe directive to a general AI?

I like mindrime's answer as it identifies fundamental, applicable notions.  I'll attempt another short answer from a Game Theory perspective.

Game Theory was founded in the minimax principle.  Specifically, maximizing benefit while minimizing potential downside in a condition of uncertainty.

Minimax works very well in contexts with easily definable parameters, such as combinatorial games and procedural optimization, but it gets more tricky in real-world scenarios where too many parameters may be present, resulting in a combinatorial explosion.

Another issue regarding real-world applications arises from the symbol grounding problem.  In the paper clip scenario, the benefit sought (the goal) is clearly definable, and can be expressed mathematically.  By contrast, the downsides to be avoided are more difficult to define: "Don't destroy the world", "Don't use up resources to a degree that humans suffer", "Don't harm the environment" all rely on language.  This is to say they rely on terms that, at present, constitute symbols that cannot be grounded.  Thus there is room for misinterpretation that may yield unintended consequences.

"Minimizing impact on the system" (minimizing influence in the world) would be the goal, but how do you guarantee an automata has a clear understanding of what this entails in every possible scenario?


What is the purpose of the "gamma" parameter in SVMs?

I've summarized the key ideas of SVMs. So this is how $\gamma$ is used with a gaussian Kernel:

$$K_{\text{Gauss}}(\mathbf{x}_i, \mathbf{x}_j) = e^{\frac{-\gamma\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2 \sigma^2}}$$

The bigger the $\gamma$, the more "linear" the decision boundary will be. The closer to 0, the more support vectors you have/the more non-linear the boundary is (see interactive example)

You can also find it on udacity: SVM Gamma Parameter

In practice, you can use a grid search or random search to get good values.


How are edge features implemented in Geometric Deep Learning?

In the paper Neural Message Passing for Quantum Chemistry (2017), the authors (from Google, Google Brain and Google DeepMind) introduce a framework called message passing neural network (MPNN), which generalizes previously proposed geometric deep learning models. In section 2 of the paper, they describe this MPNN framework and they state that edge features can also be learned, using the MPNN framework, by introducing hidden states for all edges in the graph, $\mathbf{h}_{e_{vw}}^t$, where $t$ is the iteration number and $e_{vw}$ is the edge from node $v$ to node $w$.

See also the paper Machine learning prediction errors better than DFT accuracy, which is cited in the MPNN paper as an example where the authors also learn the edge features. In section 2 of the MPNN paper, they briefly describe this specific instantiation of the MPNN framework, called the Molecular Graph Convolutions.


Does the model learn from the average of all the data points in the mini-batch?

if I input the unshuffled data (as I required to view the predictions) I get a 45% accuracy. How is this possible?

When you build your dataset and then split it into training, test, and validation sets, you have to be sure that the training encompasses aspects of the test set or the validation set and more. That's why shuffling the data is important.
In the case where you didn't shuffle your dataset, your NN was confronted to object it never sees before (i.e. in the training set) and then couldn't classify it properly.

  does the model learn from the average of all the data points in the mini-batch or does it think one mini-batch is one data point and learn in that manner (which would mean order matters of the data)?

When using a mini batch you decide the size (b_size) of the mini batch and the number of mini batches (n_batch). Then, for n_batch times you will draw b_size random indices from X_train and y_train arrays, build with these indices X_batch and y_batch, and tune your model parameters on the 2 latter arrays (and do it n_batch times).

So having shuffled your dataset and takes randomly parts of your dataset to train your model ensure you to get rid of correlations between two points of your datasets and better generalization


Can AlphaFold predict proteins with metals well?

Let me address first some of the things you wrote in your question:

There are certain proteins that contain metal components, known as
metalloproteins.

Natural proteins do not ever contain metal components as far as we know. Natural proteins are composed of natural amino acids which only contain H,C,O,N,S. Selenocysteine contains Se (also a non-metal!) but it's a proteinogenic amino acid, which means it's a precursor to proteins but doesn't typically show up in the protein itself. From the Wikipedia page that you gave us in your question: "Metalloprotein is a generic term for a protein that contains a metal ion cofactor" and "A cofactor is a non-protein chemical compound or metallic ion".
But your question still deserves an answer, because even though it would incorrect to call a co-factor "part" of the protein, they can still affect the folding and the overall shape. But let's first address one last part of your question:

Given that maybe there is not enough structural data about protein
local structure around metal atoms (e.g. Fe/Zn/Mg), then AlphaFold
cannot predict local structure around metals well. Is that right?

The first sentence of the Wikipedia article that you linked in your question, says "For instance, at least 1000 human proteins (out of ~20,000) contain zinc-binding protein domains [3] although there may be up to 3000 human zinc metalloproteins [4]." Therefore, while metalloproteins might not be the majority of all proteins, there's a decent enough number of them that are of relevance to the human body, and therefore constructing training databases that contain enough metalloproteins (or even 100% metalloproteins, if desired) is not difficult.
I mentioned a bit elsewhere that AlphaFold was used to predict protein structures in the CASP competition, for which you can see for yourself that many/most of the proteins for which contestants (such as DeepMind) need to predict the structure, come from studies of proteins of relevance to humans because the CASP structures typically come from X-ray crystallography studies, which are typically done on proteins of relevance to humans.
You can also see for yourself not only the "target list" that I showed above, but also the results of the competition which will show how well AlphaFold performed in CASP13 (2018) and CASP14 (2020) for metalloproteins.
Finally:

I also think that the more complex electron shell of metal also makes
the data less useful, since its bounding pattern is more flexible than
carbon, etc.

It is true that metals are typically harder to model than C,H,O,N,S, and even Se, if doing ab initio calculations on the metals or metail-containing complexes. However, the purpose of machine learning in protein folding studies is to skip ab initio, statistical-dynamical and/or molecular-dynamical calculations of the relevant structures and simply use training data to predict the protein structures. That being said, there needs to be enough training data available (as you correctly pointed out) to learn what happens near the metal co-factors: The answer to this is that there are indeed enough metalloproteins to sufficiently populate a training set, but they won't contain enough of the specific metals involved in every metalloprotein. For example, lots of data will be available for proteins containing Fe since Fe is in hemoglobin (for example) which is essential to the functioning of red blood cells to absorb oxygen, but the protein vanabins contains vanadium which is much rarer and therefore training data involving it will be much less available. You're also correct that metal elements can form more bonds than typical elements found in organic compounds.
So it depends on the metal in the relevant co-factor. Fe-based co-factors will have quite a lot of training data available, as well as Mg-based ones, Zn-based ones, and a lot of other ones which contain the "more common" metals. For proteins like vanabins which contains vanadium, you are quite correct that training data will be limited, but also keep in mind that vanabins is a very rare protein found in sea squirts and we already know more about its structure (through X-ray crystallography, which means we don't need machine learning for it) than we even know about what it even does. The chances of other vanadium-containing co-factors in metalloproteins being very significant is too low to justify working on protein folding algorithms specifically for them.


Probabilistic action selection in pursuit algorithm

If I am understanding your question properly, you could use something like the following:

import numpy as np
p1 = 0.1
if np.random.rand() < p1:
    action = 'greedy'
else:
    action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3'])

return action


How to use MOPSO to align characters vertically?

Your problem appears to be multiple sequence alignment. That problem is well studied, and depending on your application, there are fast algorithms that can perform quite well without recourse to AI or optimization, and which are detailed in the provided link.

If your problem has a lot of local minima, then a PSO might be a good approach. Certainly local search has been widely used for this kind of problem in the past.

Your state should be represented by a list of sequences, each with some blanks inserted. Movement between states can consist of adding or deleting blanks from one or more sequences. The distance between two states can be computed as the sum of the number of operations needed to transform each sequence to its corresponding sequence in the other state, which you can compute in linear time (just sum up the needed additions or deletions between each pair of letters).

Hope that gives you a good starting place!


It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so.

For example:

If an average pooling layer has inputs $z$ and outputs $a$, and each output is average of 4 inputs then $\frac{da}{dz} = 0.25$ (if pooling layers overlap it gets a little more complicated, but you just add things up where they overlap).
A max pooling layer has $\frac{da}{dz} = 1$ for the maximum z, and $\frac{da}{dz} = 0$ for all others.

A pooling layer usually has no learnable parameters, but if you know the gradient of a function at its outputs, you can assign gradient correctly to its inputs using the chain rule. That is essentially all that back propagation is, the chain rule applied to the functions of a neural network.

To answer your question more directly:

  Can non-differentiable layer be used in a neural network, if it's not learned?

No.

There is one exception: If this layer appears directly after the input, then as it has no parameters to learn, and you generally do not care about the gradient of the input data, so you can have a non-differentiable function there. However, this is just the same as transforming your input data in some non-differentiable way, and training the NN with that transformed data instead.

* Technically there are some discontinuities in the gradient of a max function (where any two inputs are equal). However, this is not a problem in practice, as the gradients are well behaved close to these values. When you can safely do this or not is probably the topic of another question.


Formal proof that every purely reactive agent has behaviorally equivalent standard agent

I recommend you to look into the literature about simulation and bisimulation in Automata Theory and its applications to model checking (where you want to make quite regularly proofs of "behavioural equivalence"). One article that discusses this in the context of a technique for model checking known as "Abstraction and Abstraction Refinement" is

Abstraction and Abstraction Refinement
Dennis Dams and Orna Grumberg
In Springer's Handbook of Model Checking, 2018, Chapter 13, pages 385-420

A good (I use it regularly) book that covers behavioural equivalence for a wide variety of automata is

Verification and Control of Hybrid Systems: A Symbolic Approach
Paulo Tabuada
Springer, 2009


What are the jobs that artificial intelligence will create?

A report from PwC states that - AI will displace around 7 million and will be creating around 7.2 million new jobs.

According to MIT researchers –  AI will replace tasks, not jobs.

In my opinion AI revolution is mainly going to effect non-technological jobs. So, the next generation—of employees should focus on creating career in new roles like AI researchers, Data scientist, Robotic Engineer.

(Image Source : theedgesearch)

These are the new jobs that AI is going to create:

AI engineer
Big Data scientist
Real-Time Data Engineer
Machine Linguist
Robot counselor
Telepresence Specialists
Vertical Farming Specialist
Limb Engineer
Sensor Scientist
Virtual Experience Specialist
Robotic Engineer
Virtual Reality Engineer
Genome Specialist
Gaming engineer
IOT Engineer


How to verify classification model trained on classification dataset on a detection dataset for classification purpose?

To verify the accuracy of the classification stage, you will need labeled images with a single car.
To train and verify accuracy of the detection stage and full system, you can:

in the datasets with images with multiple cars, manually, mark the image rectangles that contains one car.
from previous, split the image in one or more ones, each one containing a single car.
pass each one of the previous image with a single car to the classification stage (that means assume classification has 100% accuracy). Record its outputs (labeled cars).
now, from output of steps 1) and 3), you can produce labeled images with multiple cars. Use it to train the detector and verify full system accuracy.


How to compute the target for double Q-learning update step?

$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$
The only difference between the "original" DQN and this one is that you use your $Q_\text{est}$ with the next state to get your action (by choosing the action with the highest Q).
Afterward, you just figure out what the target $Q$ is given that action, by selecting the $Q$ belonging to that action from the target_network (instead of using the argmax a directly on the target Q network).
About the formula

$\theta_{t}^{-}$ above it means frozen weights, so it represents the target Q network.

the other $\theta_{t}$ represents the "learnable weights" so the estimate Q network.


How does MuZero learn to play well for both sides of a two-player game?

Both players are represented by the exact same network with the exact same weights(similar to AplhaGO, AlphaGoZero and AlphaZero). So, they will both behave identical. Because you only have a single network, MuZero can not learn two different policies, but only one.
You can also think of this in the following way: MuZero actually learn to play only with white(or black, but just one of them) without knowing to play with the other color (at least multiple implementations of the previous algorithms like AlphaGo Zero and AlphaZero that are similar to MuZero are doing exactly this). So, in order to trick it to also be able to play with the other color, when the network need to play with black, you just flip the colors on the table so that black becomes white(and white becomes black) and the network knows what to do. After choosing the move, you flip the whole thing back and that is usually how it is done. So, from the perspective of your network, it will always play white, but because you do the flipping of the colors you can actually put them to play against each other without them knowing that.
Even without using this trick of flipping the color of the table, by doing the MCTS simulations, you will have for each state the statistics for the actions, and usually as you do more simulations, this statistics show you which actions are the best in each state. And when you train, you try to imitate this. So, your network will learn in each state which actions are the best, and this is the reason why it learns to take the best possible action in each state.


Why do neural nets and machine learning tend to work well with MCTS, but not with regular Minimax game-playing AI?

It just does.
Take a look at this post explaining how MCTS works.

  In both Alpha Go Lee and Alpha Zero the tree traversal follows the nodes that maximize the following UCT variant:

  \begin{equation}
UCT(v_i,v) = \frac{Q(v_i)}{N(v_i)} + cP(v_i, v)\sqrt{\frac{N(v)}{1+N(v_i)}}
\end{equation}

  where P(vi,v)
  is prior probability of the move (transition from v to vi ), its value comes from the output of deep neural network called Policy Network . Policy Network is a function that consumes game state and produce probability distribution over possible moves.

As you can see the Policy Network(it is actually just one neural network for both the value and the policy) is used to guide the search tree. Not all possible moves are explored. Also during the learning phase the policy network uses the "visit count" of the MCTS Nodes to learn. Moves that were explored more are the better ones. The state-of-the-art chess engine Stockfish evaluates about 1000 times more positions per second than Alpha Zero. It relies on exploring "most of" the possible positions. Calculating the score using heuristics is much faster than using Alpha Zero's 19 layer residual network. If Google were to use minimax then they wouldn't be able to look very far ahead. Alpha Zero explores about 70K moves per second which would be only a couple ply deep. MCTS allows it to explore only the more promising moves and simulate those positions.


How is a ResNet-50 used for deep feature extraction?

The authors use so-called embeddings, it's a form to represent the images in some meaningful vector form.
The procedure to get embedding as follows. First, keep in mind most of the popular convolutional net architectures starts with convolutional layers and then have few fully connected layers. Then do the following.

Train the full network with one-hot encoded labels as usual
Take away the last fully connected layer and use the values on the previous layer as a representation.

In the case of resnet50, you will get a 2048-value float vector. The property of a neural network is that semantically close images usually have close representation on the last layers and you could use euclidian distance to measure the similarity of images in some sense (not really, but it's another long discussion)
I don't know the paper, but I glimpsed through it. You could see in formula 4 the $x$ is embedding representation. Loss at formula 5 is cross-entropy after applying the last linear layer ($Wx + b$) and then softmax.
As a side note, you could skip the first step completely and use pretrained weights as a starting point. I.e. in pytorch you could take pretrained on ImageNet classification weights as follows
import torchvision.models as models
resnet = models.resnet50(pretrained=True)


What kinds of problems can AI solve without using a deep neural network?

I was hoping to see more answers here, but I'll get us started with some examples:

Combinatorial Search Problems: If your problem can be phrased as movement through a combinatorial graph, you don't need a neural network. In particular, your problem should have discrete states, a clear set of actions that are possible in each state, a clear definition of where we start, and a clear definition of what the goal state looks like. The most effective general purpose technique is iterative deepening search. If you have an idea about which moves might be more effective, or better, a function that estimates how far each state is from the goal, you may be able to build a heuristic function and use A* search instead. Common applications for these techniques include pathfinding in video games (or directions in other applications), AI planning, and Automated Theorem Proving.

I'll add some more topics later, but I suspect others have expertise to share here. Let's see some more ideas!


Bayesian Perceptron: Why to marginalize over neuron's output instead of it's weights?

We want a distribution over $w$, don't we?

Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can induce a probability distribution over the possible functions consistent with your data.

Why is $a$ integrated out here and not $w$?

This is just the definition of marginalization.
For simplicity, consider 2 discrete random variables $X$ and $Y$, which can take values $x_i$ and $y_i$, respectively, and for which you can have (at least conceptually) a joint distribution $p_{(X, Y)}(x, y)$. Then the marginal probability that $X$ takes the value $x_i$ independently of any outcome of the r.v. $Y$ (and this is the intuition behind the marginal distribution!) is given by
$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})}, \tag{1}\label{1}$$
So, in the case of equation 14 of the paper, there's the assumption that you have $p(w, a)$. Then $p(w)$ is a marginal distribution over $w$ (the parameters or weights of the neural network), independently of the values of the activations. Of course, $w$ and $a$ can be dependent or correlated, and that's why you also have $p(w, a)$, i.e. the probability distribution that describes how these two r.v.s vary together, but what you want to know is the probability distribution over $w$ independently of $a$, so you "marginalize out" the possible values of $a$ (not $w$).
Remember that $p(x, y) = p(x \mid y) p(y) = p(y \mid x) p(x)$ (this is known as the chain rule of probability for two r.v.s, which leads to the Bayes' theorem), so you could write equation (\ref{1}) as follows
$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})} = \sum _{j}p(x_{i} \mid y_{j}) p(y_j), \tag{2}\label{2}$$
In the case of real-valued r.v.s, you will have an integral rather than a summation, but the idea is roughly the same.

But without marginalization over $w$ there is still uncertainty inside $w$.

What we want to know is $p(w)$, i.e. a distribution over $w$, which models the uncertainty about the possible values that $w$ can take. So, once you know $p(w)$, depending on its shape, we will have more or less uncertainty about the possible values of $w$. For example, let's say $p(w)$ is a Gaussian distribution. Then, depending on the variance, you will have more or less uncertainty about the possible values of $w$.
Your doubt seems to be that you thought that marginalization is a way to not have uncertainty, but that's not true. So, the goal in Bayesian machine/deep learning is not to be certain about the values of the parameters, but to model the uncertainty about the values of the parameters.


Converting age and sex variables to a 64-unit dense layer

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.


What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across "similar" states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.


Is it possible to use an RNN to predict a feature that is not an input feature?

Is it possible to use RNN in this case? Can it also predict features that are not part of the input features?

Yes.

No changes are required to a RNN in order to do this. All you need is correctly labelled data mapping a sequence of $x$ to correct $y$ in order to train, and of course a RNN architecture which has input vectors matching shape of $x$ and output vectors matching shape of $y$. The case where $x$ and $y$ are the same data type is just a special case of RNN design, and not a requirement.

You may need to consider some details:

If the relationship between $x$ and $y$ is complex and non-linear even accounting for accumulated hidden state during the sequence, you may need to add deeper layers. The output of the LSTM can be some vector $h$ and you can add fully-connected layers to help with predicting $y$ from $h$. This, or adding more LSTM layers, is a choice of hyperparameter that you may want to experiment with. Start with a basic LSTM to see how that goes first.
If you wish to predict a sequence of output features that is either not the same length as the input feature sequence, or logically should come after the whole sequence (think language translation) then this may need a slight change in setup to get best results. For a predict-same-kind sequence you can feed your predicted output value into the next input, but if input and output have different data types, this will not work. Instead, you will need to have some dummy input or other setup for creating sequences of $y$.

In your specific case the second point does not seem to apply, as you want to predict a single $y$ immediately after a sequence of $x$.


Which explainable artificial intelligence techniques are there?

Explainable AI and model interpretability are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (often tremendous) success of deep learning models in  various tasks, plus the necessity of algorithmic fairness & accountability.
Here are some state of the art algorithms and approaches, together with implementations and frameworks.

Model-agnostic approaches

LIME: Local Interpretable Model-agnostic Explanations (paper, code, blog post, R port)

SHAP: A Unified Approach to Interpreting Model Predictions (paper, Python package, R package). GPU implementation for tree models by NVIDIA using RAPIDS - GPUTreeShap (paper, code, blog post)

Anchors: High-Precision Model-Agnostic Explanations (paper, authors' Python code, Java implementation)

Diverse Counterfactual Explanations (DiCE) by Microsoft (paper, code, blog post)

Black Box Auditing and Certifying and Removing Disparate Impact (authors' Python code)

FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (blog post, paper, code)

SHAP seems to enjoy high popularity among practitioners; the method has firm theoretical foundations on co-operational game theory (Shapley values), and it has in a great degree integrated the LIME approach under a common framework. Although model-agnostic, specific & efficient implementations are available for neural networks (DeepExplainer) and tree ensembles (TreeExplainer, paper).

Neural network approaches (mostly, but not exclusively, for computer vision models)

The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (2015 paper @ PLoS ONE, 2016 paper @ JMLR, project page, code, TF Slim wrapper)

Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (paper, authors' Torch code, Tensorflow code, PyTorch code, yet another Pytorch implementation, Keras example notebook, Coursera Guided Project)

Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (paper, code)

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (paper, code, Google blog post)

TCAV: Testing with Concept Activation Vectors (ICML 2018 paper, Tensorflow code)

Integrated Gradients (paper, code, Tensorflow tutorial, independent implementations)

Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (project page, Caffe code, PyTorch port)

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (project page, with links to paper & code)

Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (paper, code)

Transparecy-by-Design (TbD) networks (paper, code, demo)

Distilling a Neural Network Into a Soft Decision Tree, a 2017 paper by Geoff Hinton, with various independent PyTorch implementations

Understanding Deep Networks via Extremal Perturbations and Smooth Masks (paper), implemented in TorchRay (see below)

Understanding the Role of Individual Units in a Deep Neural Network (preprint, 2020 paper @ PNAS, code, project page)

GNNExplainer: Generating Explanations for Graph Neural Networks (paper, code)

Benchmarking Deep Learning Interpretability in Time Series Predictions (paper @ NeurIPS 2020, code utilizing Captum)

Concept Whitening for Interpretable Image Recognition (paper, preprint, code)

Libraries & frameworks
As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is a partial list:

The ELI5 Python library (code, documentation)

DALEX - moDel Agnostic Language for Exploration and eXplanation (homepage, code, JMLR paper), part of the DrWhy.AI project

The What-If tool by Google, a feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (project page, code, blog post)

The Language Interpretability Tool (LIT) by Google, a visual, interactive model-understanding tool for NLP models (project page, code, blog post)

Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (code; papers: Feature Visualization, The Building Blocks of Interpretability)

TorchRay by Facebook, a PyTorch package implementing several visualization methods for deep CNNs

iNNvestigate Neural Networks (code, JMLR paper)

tf-explain - interpretability methods as Tensorflow 2.0 callbacks (code, docs, blog post)

InterpretML by Microsoft (homepage, code still in alpha, paper)

Captum by Facebook AI - model interpetability for Pytorch (homepage, code, intro blog post)

Skater, by Oracle (code, docs)

Alibi, by SeldonIO (code, docs)

AI Explainability 360, commenced by IBM and moved to the Linux Foundation (homepage, code, docs, IBM Bluemix, blog post)

Ecco: explaining transformer-based NLP models using interactive visualizations (homepage, code, article).

Recipes for Machine Learning Interpretability in H2O Driverless AI (repo)

Reviews & general papers

A Survey of Methods for Explaining Black Box Models (2018, ACM Computing Surveys)

Definitions, methods, and applications in interpretable machine learning (2019, PNAS)

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019, Nature Machine Intelligence, preprint)

Machine Learning Interpretability: A Survey on Methods and Metrics (2019, Electronics)

Principles and Practice of Explainable Machine Learning (2020, preprint)

Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges (keynote at 2020 ECML XKDD workshop by Christoph Molnar, video & slides)

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020, Information Fusion)

Counterfactual Explanations for Machine Learning: A Review (2020, preprint, critique by Judea Pearl)

Interpretability 2020, an applied research report by Cloudera Fast Forward, updated regularly

Interpreting Predictions of NLP Models (EMNLP 2020 tutorial)

Explainable NLP Datasets (site, preprint, highlights)

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

eBooks (available online)

Interpretable Machine Learning, by Christoph Molnar, with R code available

Explanatory Model Analysis, by DALEX creators Przemyslaw Biecek and Tomasz Burzykowski, with both R & Python code snippets

An Introduction to Machine Learning Interpretability (2nd ed. 2019), by H2O

Online courses & tutorials

Machine Learning Explainability, Kaggle tutorial

Explainable AI: Scene Classification and GradCam Visualization, Coursera guided project

Explainable Machine Learning with LIME and H2O in R, Coursera guided project

Interpretability and Explainability in Machine Learning, Harvard COMPSCI 282BR

Other resources

explained.ai blog

A Twitter thread, linking to several interpretation tools available for R

A whole bunch of resources in the Awesome Machine Learning Interpetability repo


Why can a fully convolutional network accept images of any size?

The reason is that when using a convolutional layer, you select the size of the filter kernels, which are independent of the image/layer input size (provided that images smaller than the kernels are padded appropriately).

When using a dense layer, you specify the size of the layer itself and the resulting weight matrix is a function of both the size of the dense layer and the upstream layer. This is because each neuron in the upstream layer makes a connection to each neuron in the dense layer. So, if you have 50 neurons in the upstream layer and 20 neurons in the dense layer, then the weight matrix has $50 \times 20=1000$ values. Those weights are what get determined during the training phase, and so those layer sizes are fixed.

Now, the output of a CNN layer is a number of images/tensors (specified by the number of filters chosen), whose size is determined by the kernel size and any padding option chosen. If those are fed into a dense layer, then that fixes the the size that those images can be (because of the reason given in the previous paragraph).

On the other hand, if no dense layer is used in the whole network, then the input to the first CNN layer can be any size because the weights are just the individual parameters of the filter kernels, and the filter kernels remain the same size regardless of the input tensor size.


What are "proxy data sets" in machine learning?

In computer science, if you say "A is a proxy for B", then it means that "A replaces B" (temporarily or not), or that "A is used as an intermediary for B".
The term "proxy" usually refers to a server, i.e. there are the so-called proxy servers, which intuitively do the same thing (i.e. they are used as intermediaries). The following picture is worth more than 1000 words.

Let's go back to your context. In the following paragraph

To develop DL techniques for synthesizing high-resolution realistic fundus images serving as proxy data sets for use by retinal specialists and DL machines.

The deep learning techniques will be used to generate data that will be used as a replacement for real data for use by retinal specialists.
In your second paragraph

The advantage of the product state-based view is the focus on the product itself to structure the information and data involved throughout the process. Using the intermediate product states as proxy data for this purpose

The word "intermediate" is even used, so there should not be any need for further explanation, although I admit that I have no idea of what "product states" are in this context, but they will be used as "proxy data" for other data.

Does "proxy data" mean the same thing as "proxy data sets" does, if yes, I will go through that paper.

I guess so, but bear in mind that "data" is more general than "data set".
I have found other slightly different uses of the expression "proxy data", not in the context of computer science, but in the context of paleoclimatology. For example, have a look at this web article What Are Proxy Data?.
Just for completeness, here are some dictionary definitions of proxy.

authority given to a person to act for someone else, such as by voting for them in an election
a person who you choose to do something officially for you
a situation, process, or activity to which another situation, etc. is compared


Should I remove the units of a neural network or increase dropout?

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements.

You are basically suggesting to use dropout to allow the network to learn the same features more than once (creating a redundancies in the model). That is not the purpose of dropout.
Dropout is used to enable the network break unnecessary correlations which occur in the training set. For example if class 1 is the only one that contains in it features A and B, but all the training samples always feature them together. The dropout process can make the model realize that even just one of them is enough to point to class 1.


Is the number of feature maps equal to the number of kernels in the LeNet 5 architecture?

The # of kernels will be the channel length, looking at the image you posted in your comment from post, I do not understand where you see the inconsistency.


Is there a way to understand the type of a sentence?

You can generally identify the mood of a verb by looking at grammatical structures; you don't need any language model for it. The three major moods in English are declarative, interrogative, and imperative. Assuming English is the language you will be working with, here are some questions:

Does he like coffee?
Is this a piece of chocolate?
When did you go there?
How is that possible?
Have you got any cheese?

Apart from the obvious marker '?', these examples all start with either an auxiliary verb or a wh-word, so are fairly easy to recognise with a simple lookup. The one exception I can think of is an imperative (Do be quiet, please), where the do is followed by a further verb, which wouldn't be the case with a question.

Imperatives:

Go to school now!
Eat up your vegetables!
Do shut up.
Have a go at it!

These start with a main verb in the base form, or with an auxiliary followed by a verb/not a pronoun.

Once you identified all interrogative and imperative sentences, all the remaining ones should be declarative.

So, all you would need is a small list of auxiliary verb, pronouns, and wh-words, and with a bit of simple string matching you should get most of the way there. Undoubtedly there will be some exceptions, but there shouldn't be too many of them.

In other languages there will be similar structures; or there might be explicit markers (eg in Hawai'ian an imperative starts with the marker 'e', as in Hele 'oe ma ka hale "you go to the house" vs E hele 'oe ma ka hale "Go to the house!")


Perhaps you can check this dataset out:
http://www.aiskyeye.com/

  The VisDrone2019 dataset is collected by the AISKYEYE team at Lab of
  Machine Learning and Data Mining , Tianjin University, China. The
  benchmark dataset consists of 288 video clips formed by 261,908 frames
  and 10,209 static images, captured by various drone-mounted cameras,
  covering a wide range of aspects including location (taken from 14
  different cities separated by thousands of kilometers in China),
  environment (urban and country), objects (pedestrian, vehicles,
  bicycles, etc.), and density (sparse and crowded scenes). Note that,
  the dataset was collected using various drone platforms (i.e., drones
  with different models), in different scenarios, and under various
  weather and lighting conditions. These frames are manually annotated
  with more than 2.6 million bounding boxes of targets of frequent
  interests, such as pedestrians, cars, bicycles, and tricycles. Some
  important attributes including scene visibility, object class and
  occlusion, are also provided for better data utilization.

It provides many drone view images with bounding boxes. Hope it can help you and have a nice day!


When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.
See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton & Barto)


1 hidden layer with 1000 neurons vs. 10 hidden layers with 100 neurons

Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict:

Big area, higher price
Small amount of bedrooms, lower price

The second layer might conclude:

Big area + small amount of bedrooms = large bedrooms = +- effect

Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.

Check out this answer


Using unsupervised learning for classification problems

You cannot use clustering as an "unsupervised classifier", with the goal of targeting specific known classes that are already defined. For that you need supervised learning and a labelled dataset.

A clustering algorithm would only separate into clusters as you hope if the feature variables you use are already:

All strongly associated with the trait you want the clusters to find
Carefully scaled to match the expected relationship

Now this might happen by chance with an arbitrary dataset. However in practice in a complex dataset with many features (as you might use in cancer classification) you would need to select and scale just relevant features. Thus you would need to already know the relationship between features and class from some other model.

The main task of clustering is to discover naturally occurring groups within your data, that might give you insight into the distributions within a dataset. This might let you compress or summarise information about examples. The summary feature might even be predictive, but it would very rarely overlap with pre-defined classes, even when the full feature vector could be used in an accurate supervised learning predictive model.

With your given example of [age, height] you might discover that the population splits naturally into people who are still growing, fully-grown youths and mature people. If this happens to overlap with predictions of type 1 and type 2 cancer that might be interesting - perhaps a person's cluster is even a good predictive variable for having type 1 or type 2 cancer. However, whatever the features are, it is highly unlikely that it would be so strong a predictive variable that you would use it on its own to make a classification. The only variables that would be that predictive would be the clinical measurements of the cancer growth that clinicians already use to decide the cancer type.

Potentially you could build a Bayesian model around the clusters, where you can ask questions such as:

What is the probability that a random person is in cluster A $p(x \in A)$?
What is the probability that a random person has cancer type 1 $p(x \in C_1)$?
What is the conditional probability that a person with type 1 cancer is found in cluster A $p(x \in A|x \in C_1)$?

Bayes' Theorem would then allow you to calculate the probability of having type 1 cancer based on their clustering:

$$p(x \in C_1 | x \in A) = \frac{p(x \in A|x \in C_1)}{(x \in A)}p(x \in C_1)$$

To do this fully with Bayesian statistics would involve setting some prior beliefs about the probabilities and using the data from your analysis to adjust them, which is beyond the scope of this answer. However, you could as a crude shortcut just take the measured frequencies, and that might be fine as a guide and give you some intuitions about what your clustering is doing towards helping with predictions.


What is the purpose of hidden nodes in neural network?

A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.


Does a differential evolution algorithm mutate its population during a generation?

Quoting the original paper:

For each target vector $x_{i,G}$ ,a mutant vector is generated according to
$$ v_{i,G+1} = x_{r_1,G} + F\left(x_{r_2,G} + x_{r_3,G}\right)$$

And later

To decide whether or not it should become a member of generation $G + 1$, the trial vector $v_{i,G+1}$ is compared to the target vector $x_{i,G}$ using the greedy criterion.

I'd say it is pretty unambiguous that author's intent was to split evolving agents' populations into "generations" indexed by $G$. A new generation $G+1$ is created by applying the DE algorithm to the previous generation $G$. No agents are changed within the generation $G$.
So, it looks like the answer to your question is "no" - the population of the current generation $G$ is not mutated. Your second approach is the correct one.


How can I suppress a CNN’s translation invariant or translation equivariant?

Question 1

To make it simple

You have a transformation $T$ and an operator $C$ acting on a given input $x$
Let's say you do this experiment

compute $y_{1} = C(T(x))$
compute $y_{2} = C(x)$

You can get three different results:

$y_{1} = y_{2}$ then you can say the operator is invariant with respect to the given transformation
$y_{1} = T(y_{2})$ then you can say the operator is equivariant to the given transformation as applying it to the input basically reflects its effect completely on the output
none of the 2

Questions 2

Q2.1

Spatial Pooling is not the only way to perform dimensionality reduction. You can achieve it even simply applying the Conv Kernel with no padding.

For example, let's say your input is a WxHxC tensor and you are applying a kernel which is KxKxC the result will have spatial domain size (W-(K/2))x(H-(K/2)) with K/2 the integer truncation so if K=5 then K/2=2.

Alternatively you can reduce the spatial domain with stride.

Q2.2

It seems to me you are talking about a sort of Non Max Suppression Operator rather an emergent behaviour, but certainly with the proper supervision signal you can train a CNN to do this work (even if practically it does not make sense as you can explicitly define it).


How should we pad an image to be fed in a CNN?

If the computational components of the forward feed through the network have no curvature, which is normally the case in a sum of products, then it can be proven that any constant pixel value is equivalent in terms of effect on convergence results.  We wouldn't expect a proof for that, since it would be too trivial to spend time writing up for publication.  In general, functioning vision systems have feed forward computational components with curvature, so the padding is likely significant.

Even the convolutional layers may have activation functions or something even more complex going forward, as noted in Gauge Equivariant Convolutional Networks and the Icosahedral CNN (Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling, 2019).

If purely stochastic values with value distributions like that of the un-padded coordinates are used, it may be possible to prove that some gain is made, but none appeared in a few academic article searches just made.  Not surprisingly, there are many proofs regarding the properties of various message padding strategies for cryptography.

Short of the inclusion of thermal or quantum noise acquisition devices in VLSI circuitry and exposure of those devices in software, purely stochastic values cannot be generated.  This leaves the risk of a learning approach expected to extract features from frames learning features of the pseudo-random noise generator used to pad.

The answer is that none are universally correct and there appears to be much work to do in proving advantages between different techniques in as many cases as such advantages can be proven.


How to overcome overfitting to single player styles in reinforcement learning?

Both of the solutions you suggest seem to be built around the intuition that it's good to ensure that there is sufficient variety in the experiences that you provide to your RL algorithm.

That intuition is good, but it should not come at (too much of) a cost in playing strength of the opponent. I'm afraid that your first solution may break down because of this; Tic Tac Toe is such a simple game, any agent that doesn't play optimally can really be viewed as a very poor agent... and I think minimax agents with very low limits on search depth may end up playing suboptimally. Your second solution seems better in this regard, that could help.

For this particular case of Tic Tac Toe, I suspect you should be able to train just fine against only optimal minimax agents, as long as you make sure that those optimal minimax agents break ties randomly. In some situations, there may be multiple different actions that are all "equally optimal". Then, you'll want to make sure that your minimax agents break ties randomly, rather than always deterministically picking the same action. This can be done, for example, by making sure to always shuffle your lists of legal moves in minimax after they are generated.


Can entire neural networks be composed of only activation functions?

The Pytorch docs define a fully connected ReLU network as:
torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

Neural networks are called are made of neurons. Activation functions only help determine which of these neurons to fire up, meaning they have no learnable nodes themselves through which we can back-propagate gradients. A network with no learnable parameters is therefore not a neural net. So neural nets can't be composed of activation functions only.

What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable

Yes, what's given here is not a network that can approximate the said function $h(s)$. A two layer ReLU network would resemble:
   x = nn.ReLU(nn.Linear(d_IN, H))
   x = nn.ReLU(nn.Linear(H, H))
   out = nn.Linear(x, d_OUT)

Another way to see it is that a network must have an input and ouput layer, and optional hidden layers. It's not possible to use an activation funtion as an input layer, because then you'll have no way of configuring the number of features to represent your input data. In this context, a ReLU can't represent the features of the observation input s.
To show activation functions have no learnable nodes in them and that this interpretation

h = nn.Sequential([nn.ReLu(), nn.ReLu()])

is not what the authors are driving across, here a script that counts the number of parameters in a network.

import torch.nn as nn
import numpy as np

activation = nn.ReLU

def count_params(module):
    return np.sum([np.prod(x.shape) for x in module.parameters()])

one_linear = nn.Sequential(nn.Linear(32, 10), nn.Linear(10, 1))
linear_act = nn.Sequential(nn.Linear(32, 10), activation(), nn.Linear(10, 1))
act_only = nn.Sequential(activation(), activation())

t_lin = count_params(linear_act)
lin = count_params(one_linear)
act = count_params(act_only)

print(f'Linear only: {lin}, Linear + Activation: {t_lin},' +
        f'Activation only: {act}')

[Out]: Linear only: 341, Linear + Activation: 341, Activation only: 0.0
The activation-function-only module has zero learnable parameters. Likewise, an activation function adds no parameters to the fully connected layers.
Update: Links to implementations
To confirm this answer's interpretation is correct, here are links to GAIL and GAN-GCL example implementations

GAIL : discriminator prediction (Forward call), discriminator architecture (The ReLU net):
GAN-GCL : discriminator prediction, discriminator architecture:


Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0.
If only 25% of your kernel weights are changing that does not imply a vanishing gradient, it might be a factor, but there can be a variety of reasons, such as poor data, loss function used to the optimizer, etc. Kernel's weight not changing only points out that the model is not learning well.
From the histograms, only the conv_2d_2 layer shows any form of vanishing gradients since the numbers are pretty small. But even that seems to be picking after 600 epochs. Usually, a good indicator is to have the mean of the weights in a layer to be closer to 0 and the standard deviation to be closer to 1. So, if this is maintained to a good extent during training, you're good to go.


Why not more TD() in actor-critic algorithms?

Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in High-Dimensional Continuous Control Using Generalized Advantage Estimation, using the $\lambda$-return instead of the MC return can actually help reduce the variance of gradient updates.

The above is similar to my answer in the other question you linked, so let me try to answer your question more specifically. Even though we can use $\lambda$-returns, why are they not too common in practice? I suspect there might be a few reasons:

Empirically, faster credit assignment might be more desirable than lower variance. Sometimes the learning speed of your algorithm is constrained simply by how quickly you can learn about the consequences of certain actions. In this case, it is faster to use the MC return, even if it theoretically has higher variance than the $\lambda$-return.
When proposing a new algorithm, adding $\lambda$-returns to it might give it an "unfair" advantage if the other baseline methods do not use them (the reviewers would not like this), so researchers tend to favor simpler 1-step or MC returns for the sake of consistency. I would guess this is why you don't typically see $\lambda$-returns in papers that propose new actor-critic methods. In some sense, it is generally assumed that you could always add $\lambda$-returns to them later and probably get better performance.$^*$
A decent number of deep RL researchers don't know what $\lambda$-returns are. This is especially true if they come from a pure deep learning background; they may have never read Reinforcement Learning: An Introduction which is where most people are introduced to TD($\lambda$) and $\lambda$-returns.

$^*$Exceptions to this are papers like mine (Reconciling $\lambda$-Returns with Experience Replay) where the contribution is the use of $\lambda$-returns in methods that previously could not use them. But actor-critic methods are pretty straightforward to combine with $\lambda$-returns.


Can I build a CNN for image classification tasks just with OpenCV?

OpenCV does include 2D filter convolution functions for custom separable and non-separable filters. The latter uses DFT for large filters, which may or may not be faster than the conventional method. It also includes (partial?) support for deep nets with various types of layers. Theoretically, you should be able to stitch everything together into a complete CNN. However, I have not used any of those, and I have no idea about the level of maturity of the implementation.

That said, if you are willing to implement a custom CNN from scratch, you will probably get more control over the implementation using a generic (BLAS / OpenCL / CUDA) matrix library.


Can I detect unique people in a video?

Yes. However, for some applications commercial systems are not good enough: German source
The concepts you are looking for are:

Optical Flow: for two frames, define for each point how it moved
Object tracking: given a single object in one frame, find it in the next. Optical flow helps with that.
Object detection: look at YOLO / SSD. Small introduction here.


Training a CNN from scratch over COCO dataset

It's hard to know for sure what's gone wrong, but here are some possibilities:

The problem is difficult. The COCO  paper reports that a typical object covers just 4-6% of the image. A randomly initialized model is therefore likely to do extremely poorly, with an expected precision of between 4 and 6% for detecting the object of interest in a frame. You also have 90 classes in your configuration file. It's not clear whether the model has access to the correct label, but if it's also inferring the class, we'd expect initial accuracy somewhere around 0.06%. That's actually about the precision of your starting model.
You're training on mini-batches of size 32. It's not clear to me from your config whether the lower axis of that graph is iterations or epochs. If it's the former, then your model will have seen only 1.5 million examples during the entire training period. For a problem this hard, that's not nearly enough. Indeed, COCO contains only 328k unique examples, so the model would have seen each of them just 5 times (corresponding to 5 epochs). If you've done 500,000 epochs though, then that ought to be enough.
It's possible that the hyper-parameters are not well set. I'm not an expert at training CNNs, but deep networks are notoriously finicky. I have difficulty reading your config file, but your learning rate looks reasonable to me. It also appears to undergo exponential decay over time though, and it's not clear to me that you want to be doing that, or whether the schedule you're using for the decay makes sense. This might be worth reviewing.


Image prediction model when data-set classes have visual similarity

You could use transfer learning. Find some state of the art neural net (something newer than VGG) already trained for image classification. Then remove the last layer and add a new one, training it while keeping the remaining old weights fixed. A simple softmax at the end would work.This is the simplest approach.

Because you're mostly only training the last layer, you won't need as much data.

I think FaceNet will perform poorly because it uses different features and you would need to retrain all its layers.

If you have too little data for some classes, maybe you should use an autoencoder as a features extractor. Then you could classify an image by comparing its feature vector to those for each breed, for example with cosine similarity.


Can we destroy an artificial general intelligence without its consent?

Firstly, an AGI could conceivably exhibit all of the observable properties of intelligence without being conscious. Although that may seem counter-intuitive, at present we have no physical theory that allows us to detect consciousness (philosophically speaking, a 'Zombie' is indistinguishable from a non-Zombie - see the writing of Daniel Dennett and David Chalmers for more on this). Destroying a non-conscious entity has the same moral cost as destroying a chair.

Also, note that 'destroy' doesn't necessarily mean the same for entities with persistent substrate, i.e. their 'brain state' can be reversibly serialized to some other storage medium and/or multiple copies of them can co-exist. So if by 'destroy' we simply mean 'switch off', then an AGI might conceivably be reassured of a subsequent re-awakening. Douglas Hofstadter gives an interesting description of such an 'episodic consciousness' in "A Conversation with Einstein's Brain"

If by 'destroy', we mean 'irrevocably erase with no chance of re-awakening', then (unless we have a physical test which proves it is not conscious) destroying an entity with a seemingly human-level awareness is clearly morally tantamount to murder. To believe otherwise would be substrate-ist - a moral stance which may one day be seen as antiquated as racism.


Now I want to check if I can predict B directly from A, since, in my understanding, this would mean that info on B is already inside A.

This will help inform you how much redundancy there is between A and B. However, even if you can predict B with 100% accuracy from A, you may still be better off using A+B (instead of A alone) to predict C.

If I get good predictions when testing the model A -> B, is it true then that adding B to A in predicting C is completely useless ?

It is an indicator that adding B probably won't make great improvements to your prediction of C.
The only way to be sure is to make the model that uses A+B and compare its performance against a model that uses only A. If collecting B costs time or other resources, then perform this check by limiting both models to only learn from the subset of data where you have all of A, B, C available.

And furthermore, are there smarter ways to decide if/when a feature is useless ?

Another thing you can do is to try and predict C from B alone. It doesn't need to score well, but may indicate that something useful is in the data if you get better than chance results. Scoring badly unfortunately doesn't rule it out for working well in combination with A.
Generally, if you cannot reason it clearly one way or another from theory, the accepted method is to build variations of your model with and without a feature and test them. You do have to be aware of the chances of spurious correlation though, so usually you need to have some basis or motivation for considering any new feature, from your domain knowledge that applies to the model.


Constraints of embedded Artificial Intelligence for a robot

I think a big problem with intelligent robots is that the world is very dynamic and always changing and our techniques right now are still quite static and not really flexible.

I myself come from a Computer Vision background and in this field I often see some limitations of one of the most promising approaches for AI right now (Deep Learning). For example in Computer Vision we are now able to teach image classification systems that are on the level of human performance (or better) thanks to Deep Learning. But still these classifiers are trained supervised with annotated data and should more be viewed as very smart filters instead of true AI.

In a dynamic world a supervised trained model often fails in specific situations. For example you cannot always train to classify specific objects to specific predefined classes. There are corner cases or no clear boundaries between different classes in some situations that depend on the very specific situation.

An Example:

Consider an autonomous car that classifies between a road and no-road to see were it is able to drive. Most of the time this task is quite straight forward but in some sitations the road border gently transforms into a field of grass or something different that is not driveable. So where would you set the border between both classes i.e. how would you annotate your training data in such a situation? There is no unique solution to this problem.

A supervised trained model would finally probably learn to set the border to somewhere in-between but it doesn't really tell you that there is a corner case or slight transition with a soft border or that it's not really sure about it at all.

So there is problem with the encoding/decoding of information in a neural network. We can apply a full image as an input into the network and that is fine but then we only get a static output of predefined classes or something similar which often is not sufficient information to describe the world to make proper decisions in every situation. I am pretty sure a neural net would be able to learn such rich information but right now there is no real good approach to encode or represent this wanted (dynamic) information in a static predefined network architecture (as far as I know).

And this also brings me to the next problem: there is no real good way to represent uncertainity. There are some approaches like Bayesian Deep Learning which tries to tackle this problem but (as far as I know) this is not really practical or good enough yet. And an intelligent robot should really know its own limits and capabilities to not becoming a serious danger for anyone or itself.

And finally, general learning is usually an ongoing process that means a really intelligent agent should keep learning while it is exploring the world and that best without a supervisor. There are some approaches to this like Deep Reinforcement Learning (where just a reward function is needed) which is also very promising but right now it is very sample inefficient and not easily generally applicable to every task so that it could update its behaviour in an online manner.

Conclusion:

The hard part about it is actually to develop a reliable robot that handles these rare or special cases realliably and/or is able to identify these with its own reliable uncertainity to fail savely. This state is not achieved yet.


Why would the learning rate curve go backwards?

I have not used fastai library but this also happens on tensorboard when you have more than one training being recorded on the same plot.
Looking at the picture, I think this is a very special type of graph because for a single LR value you have 2 loss values associated. Put in other words, you have the same LR value for different loss values. My guess it that some time-dependent issue is messing things here.
Another intuition that may help to solve the issue is representing data differently. If you rotate the graph 90º left-wise you could see how the LR is evolving with different loss values. LR should be decreasing along with loss value, but in this case it is not like that either. So review how you are setting the LR too!
My list of to-check would be:

Check how you update LR according to the loss value
Check the LR recorder instance is only being used by one training at a time
Try plotting the value of LR (y) against your loss values (x)
Try plotting the value of LR (y) against the current epoch number (x)

Hope it helps


Can Q-learning be used to derive a stochastic policy?

No it is not possible to use Q-learning to build a deliberately stochastic policy, as the learning algorithm is designed around choosing solely the maximising value at each step, and this assumption carries forward to the action value update step $Q_{k+1}(S_t,A_t) = Q_k(S_t,A_t) + \alpha(R_{t+1} +\gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_k(S_t,A_t))$ - i.e. the assumption is that the agent will always choose the highest Q value, and that in turn is used to calculate the TD target values. If you use a stochastic policy as the target policy, then the assumption is broken and the Q table (or approximator) would not converge to estimates of action value for the policy*.

The policy produced by Q-learning can only be treated as stochastic when there is more than one maximum action value in a particular state - in which case you can select equivalent maximising values using any distribution.

In theory you could use the Q values to derive various distributions, such as a Boltzmann distribution, or softmax as you suggest (you will want to include some weighting factor to make softmax work in general). These can work nicely for the behaviour policy, for further training, and as an alternative to the more common $\epsilon$-greedy approach. However, they are not optimal policies, and the training algorithm will not adjust the probabilities in any meaningful way related to the problem you want to solve. You can set a value for e.g. $\epsilon$ for $\epsilon$-greedy, or have more sophisticated action choice with more parameters, but no value-based method can provide a way to change those parameters to make action choice optimal.

In cases where a stochastic policy would perform better - e.g. Scissor, Paper, Stone versus an opponent exploiting patterns in the agent's behaviour - then value based methods provide no mechanism to learn a correct distribution, and they typically fail to learn well in such environments. Instead you need to look into policy gradient methods, where the policy function is learned directly and can be stochastic. The most basic policy gradient algorithm is REINFORCE, and variations on Actor-Critic such as A3C are quite popular.

* You could get around this limitation by using an estimator that does work with a stochastic target policy, e.g. SARSA or Expected SARSA. Expected SARSA can even be used off-policy to learn one stochastic policy's Q values whilst behaving differently. However, neither of these provide you with the ability to change the probability distribution towards an optimal one.


How do I get started with multi-agent reinforcement learning?

After checking the Internet, you will probably find several resources such as

https://github.com/mohammadasghari/dqn-multi-agent-rl
https://rlss.inria.fr/files/2019/07/RLSS_Multiagent.pdf
https://arxiv.org/abs/2011.00583
and others.

Try to understand the principles first (see above). After some reasonable amount of coding you can adapt OpenAI gym.
Good luck!


Is there a relation between the size of the neural networks and speed of convergence in deep reinforcement learning?

Speed and size, no. That's because speed is dependent on processor clock periods and the effective parallel nature of the particular deep RL design in the environment, which is also dependent on cores and host clustering. Size is not really quantifiable in a way that is meaningful in this relationship because there is a broader and more complex structure of a network that might be trained to produce a value to use in the RL algorithm chosen.

Width for each layer
Cell type and possibly activation function for each layer
Hyper-parameters

One can say that number of clock cycles times number of effective parallel processors is correlated to all the above, the overhead of the RL algorithm used, and the size of each data tensor.

There are a few inequalities developed in the PAC (probably approximately correct) framework, so it would not be surprising if there were some bounding rules for the relationship between clock cycles, effective parallelism, data widths, activation functions, and RL algorithms for deep RL.

Study of the algorithm, perhaps in conjunction with experimentation, may reveal the primary factors, essentially the processing bottlenecks. Further study of the factors involved in controlling loop iteration counts, which cause the bottleneck, could permit the quantification of computing effort required to maintain a particular maximum allowable response time, but that would be specific to a particular design.

Such might produce a metric that is effectively a count of clock cycles across the system's potentially parallel architecture for a worst case or mean RL action selection. That could then be used to determine the response time for a given system with all of the factors mentioned above fixed, including the priority and scheduling of the processes in each operating system involved.

Here's a guess. Feel free to critique, since such a formulation is a project far beyond the effort that should be put into answering a question online.

$$ t_r \propto e^{k_v} \, n_v \, \eta_v \, t_v \, \mu_v \, \sum c_v + e^{k_d} \, n_d \, \eta_d \, t_d \, \mu_d \, \sum c_d \; \text{,} $$

where $t$ is time, $k$ is tensor complexity, $n$ is number of cores, $\eta$ is the effective efficiency of the parallel processing, $\mu$ is the effective overhead cost of the glue code, $c$ is the cycles require of the significant (and probably repetitive) elements in the algorithms, $t_r$ is the total time to response, and the subscripts $v$ and $d$ designate the variable subscripted as either RL value determination metrics or deep network metrics respectively.


Are all fully observable environments episodic?

No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book:
Fully Observable Environment (section 2.3.2)

If an agent’s sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable. A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action

Episodic Environment (section 2.3.2)

In an episodic task environment, the agent’s experience is divided into atomic episodes. In each episode the agent receives a percept and then performs a single action. Crucially, the next episode does not depend on the actions taken in previous episodes.

Take note of the "crucial" part at the end of the definition of episodic environment. A fully observable environment that is not episodic (and therefore sequential in the book's taxonomy) is chess. Chess is fully observable because the player can view the positions of all active pieces on the chess board, and that is all the information that needs to be known in order to take the optimal action. But chess is not episodic, because the player's current move depends on all previous moves, and the current move will have downstream effects in later turns.
In fact, if you look at Figure 2.6 in the book on pg. 45, they provide three examples of fully observable sequential (i.e. not episodic) environments: crossword puzzles, chess, and backgammon. There are of course many more. Most games are sequential as that is the main appeal of them - how to best sequence my moves now in order to ensure victory over my opponent at a future time?


Are 2 GT 710 graphics cards enough for deep learning?

No, it appears to take large amounts of screenshots to generate the images to be used for training.

Don't waste your money. I would say you could try training on Google's colab if you can upload the training data.