Why is Adam trapped in bad/suspicious local optima after the first few updates?

The authors describe their belief in Section 3:

Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.

Diving further, in section 3.2:

Adam uses the exponential moving average to calculate the adaptive learning rate.  For gradients $\{g_1,\dots,g_t\}$, their exponential moving average has a larger variance than their simple average.  Also, in the early stage ($t$ is small), the difference of the exponential weights of $\{g_1,\dots,g_t\}$ is relatively small (up to $1−β^{t−1}_2$).

It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.


Is it possible for an AI to learn how to speak from books as training sets?

It is possible

The Recurrent Neural Network architectures help in building efficient NLP algorithms, which can identify semantics and their relations across long pieces of text.

With very minor tuning, they can be made generative too. So, here is an excellent article on RNNs which I highly recommend, which also talks about how an RNN was trained on Shakespere's texts and wrote one itself.


In YOLO, what exactly do the values associated with each anchor box represent?

Not an pro but I think I know some answers to your questions.

  If we train our classifier, wouldn't the prediction boxes be close to the ground truth labels as training progresses

I think that's what YOLO v1 did. According to Andrew NG's video the bounding boxes are introduced to solve multiple objects inside the same grid cell. And according to this post anchor boxes assignment ensures that an anchor box predicts ground truth for an object centered at its own grid center, and not a grid cell far away (like YOLO may)

  what are those numbers representing anchor boxes representing?

They are just width and height (shape). In YOLO v2 it is used to compute IOU assuming all boxes are placed at the same location (ignoring the location), you could think of it just tries to match the shape. And it uses (1-IOU) as the distance when applying the K-means clustering.


Why spend so much time and money to build AIs to play games?

In the book Artificial Intelligence: A Modern Approach (section 5.7, p. 185), Russell and Norvig write

In 1965, the Russian mathematician Alexander Kronrod called chess "the Drosophila of artificial intelligence." John McCarthy disagrees: whereas geneticists use fruit flies to make discoveries that apply to biology more broadly, AI has used chess to do the equivalent of breeding very fast fruit flies. Perhaps a better analogy is that chess is to AI as Grand Prix motor racing is to the car industry: state-of-the-art game programs are blindingly fast, highly optimized machines that incorporate the latest engineering advances, but they aren't much use for doing the shopping or driving off-road. Nonetheless, racing and game-playing generate excitement and a steady stream of innovations that have been adopted by the wider
community

So, although these games (like Chess, Go, and Bridge) may not apparently be useful/beneficial to many people, the AI programs developed to play them have introduced/included concepts/techniques, like null move heuristics, futility pruning, combinatorial game theory, finessing and squeezing or meta-reasoning, which can potentially be useful to a wider spectrum of Computer Science (and not just Artificial Intelligence).
You can see it similar to space missions of NASA, ISRO, JAXA and other space agencies. All these missions don't seem to have a direct benefit to citizens, but have many indirect benefits. They pave the way for technological innovations (GPS, 3D printing, car crash technology, clean energy, LED), the creation of jobs, etc. Advance storms, hurricane detection is the output of space exploration, which has saved millions of lives worldwide.
AI in games has not just helped to develop the software but hardware also. Many innovations have been seen to produce highly optimised and powerful hardware.


Is it normal to have the root mean squared error greater on the test dataset than on the training dataset?

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are unknown / not common that may give more errors or misclassification when doing prediction.

But if your model shows your test dataset have way too high RMSE result rather than your training dataset RMSE result, it may indicates that overfitting happens.

If overfitting happens, there are a lot of reasons this could happen.
Referenced from https://elitedatascience.com/overfitting-in-machine-learning, some factors that causes overfitting are:

Complexity of data (e.g. there are irrelevant input features). This can be solved with removing irrelevant input features.
Not enough training data. This can be solved by training with more data (Eventhough this may not always succeed. Sometimes it may give noise towards data), etc.


While we split data in training and test data, why we have two pairs of each?

For any Machine Learning model, the available data is usually split into three sets:
Training Set:
The part of data used to train the model and learn the parameters of the network.
The data that remains after allocation of the Training Dataset, is split into the Validation and Test sets.
Validation Set:
This sample of data is used to provide an unbiased evaluation of a model fit on the training dataset. This helps in tuning model hyperparameters to improve the model performance. Eg: Changing the number of clusters ($k$) in a K-Means algorithm, or the pooling layers in a CNN.
Test Set:
After training, this part of the dataset is used to used to test how well the model generalizes to unseen data and estimate its performance.
Another possibility (going by your question), is the use of Cross-Validation. This is performed when the dataset is too small.  In such a case, a random split is performed on the dataset resulting in k non-overlapping subsets. The test error is then estimated by taking the average test error across k trials.

[Image Source]


Are Cellular Neural Networks one type of Neural Networks?

If the standards for what is a neural network are held consistent, then cellular neural networks should probably be considered a type of neural network.

That two adjacent layers be fully connected is not considered a requirement of what people consider neural networks.  Some attention networks, most convolution kernels, and several other types of networks are not exhaustively connected.

Exhaustive connectivity is a burden on computation time that is only valuable in cases when function requires it.  In cases where the parameter value for the connection can be shown to be always zero, there is no need for the potential.  In cases where it is rarely nonzero, the benefit of the potential may outweigh the gain of having it.

Many kinds of networks exist that are not exhaustively connected in computer science, and biological neurons are partly connected.  The axons of biological neurons might grow through an adjacent layer and connect to distant one.

In Chua's paper, it was shown that cellular neural networks converge under specific conditions, just like for other neural network types. There is no claim that they learn like humans do, and they don't use the same back propagation and gradient descent that MLPs use, but they do incrementally improve, which is the only common criteria across the many forms of artificial learning networks.

Cellular neural networks do not fit into what is generally thought of when people use the term unsupervised learning, however, they are not supervised and they do exhibit the same incremental improvement that other unsupervised networks exhibit.

Part of the issue with these names is that specialized jargon forms when only one type of a thing becomes popular, assigning the name to something more specific than the words in the name imply.  Cellular neural networks may not be examples of unsupervised learning, but they learn to function better without supervision.


Why do we update the weights of the target network in deep Q learning?

If you are certain that you reached convergence then there is no point in continuing to train your agent, because of that there is also no point in discussing why is target network being updated after convergence is reached. You should simply stop training if you converged. During training we obviously need to keep updating target network to improve correctness of Q-value estimates.


Why do we set offset (0.5) in single shot detector?

The integer coordinates refer to the top-left corner of a pixel. Adding 0.5 to both x and y gives you the center location of a pixel instead.


What are the differences between TensorFlow and PyTorch?

TensorFlow was developed by Google and is based on Theano (Python library), while Facebook developed PyTorch using the Torch library. Both frames are useful and have a great community behind them. Both provide machine learning libraries to accomplish various tasks and do the job. TensorFlow is a powerful and deep learning tool with active visualization and debugging capabilities. TensorFlow also offers serialization benefits since the entire graphic is saved as a protocol buffer. It also has support for mobile platforms and offers a production-ready implementation. PyTorch, on the other hand, is still gaining momentum and attracting Python developers, since it is more Python friendly. In summary, TensorFlow is used to speed things up and create AI-related products, while research-oriented developers prefer PyTorch.


Is the Markov property assumed in the forward algorithm?

In general, to formally state that the Markov property holds, you need to have $P(
x_t \mid x_{t-1:1}) = P(x_t \mid x_{t-1})$.

So, you cannot conclude only from $P(q_{t-1} = i, q_t=j, o_1^{t-1}, o^t|\lambda)$ that the Markov property holds, because  $P(q_{t-1} = i, q_t=j, o_1^{t-1}, o^t|\lambda)$ is the joint probability of $q_{t-1} = i, q_t=j, o_1^{t-1}$ and $o^t$ given $\lambda$.

Nonetheless, the theory of hidden Markov model often assumes that a few properties (including the Markov property) hold

the Markov property: $$P(q_{t+1} \mid q_{t}) = P(q_{t+1} \mid q_{t:1}),$$ where $q_t$ is the hidden state at time step $t$ and $q_{t:1} = q_t, q_{t-1}, \dots, q_1$.
the stationarity property: $$P(q_{t_1 + 1} \mid q_{t_1}) = P(q_{t_2 + 1} \mid q_{t_2}),$$ for any time step $t_1$ and $t_2$. In other words, the state transition probabilities are independent of the actual time at which the transitions takes place.
the output independence property: $$P(o^{T:1} \mid q_{T:1}) = \prod_{t=1}^T P(o^t \mid q_t , \lambda)$$
where $o^{T:1} = o^T, o^{T-1}, \dots, o^1$. In other words, this is the assumption that the output at time step $t$, $o^t$, is independent of the outputs at previous time steps.


How much extra information can we conclude from a neural network output values?

From the values you have provided I can easily guess your output layer has the sigmoid (do clarify!) activation function. For sigmoid activation function this can be a quite normal occurrence. Also maybe the number of training epochs is not high enough.

The case you have mentioned of 0.99 is generally in the case of the output being subjected to a softmax probability function. Although 0.99 is still achievable using sigmoid activation, it will depend on your network hyper-parameters and in general the data. If data is very easily separable the sigmoid will generally give very high contrasting difference among classes. Also if the hidden network has ReLu it becomes easy to provide contrasting class scores due to its huge scale difference compared to a sigmoid activation in the hidden layer.

The point here to note is contrasting difference among classes, because your sigmoid might give 0.99 for the correct class while 0.9 for other classes, which is undesirable.


Why are the nodes (or neurons) in neural networks depicted as circles?

I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.


Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

The term Batch Normalization is both

A system reliability choice (in terms of convergence) and
an execution strategy.

Batching is generally the process of focusing on process P with source data S to produce result R under conditions that are favorable in terms of timing, data availability, and resource utilization, such as these.

P is requires nontrivial time and computing resource and there is a window of time when it can be done on hardware H with minimal impact on other operations
R is needed by multiple other processes P2, P3, ... and draws on a set of additional inputs I2, I3, ... so that completing P in larger groups is efficient

One cannot draw a general conclusion about the value of batching that will apply to all specific installations of hardware, networking, real time demands, functional intention, and ML topology.  Also, beware of equating deep with stacked.  Depth may have multiple meanings now, but was originally used for the dimension describing the number of layers in a multi-layer perceptron, with width being the number of artificial neurons in each layer.

The term stacking can be ambiguously related to depth.  Consider that process flows through artificial neural nets has often been represented graphically as signal propagation from left to right, as is the convention in electrical engineering and text in western languages.

We then have two directions for stacking that means two different things.

Stacking in the same direction as signal flow, which indicates a sequence of operations on a vector, matrix, cube, or hyper-cube
Stacking in the tangential direction with respect to signal flow, which indicates a connectivity between items in the stack that is related to supervision, collaboration, or competition

Whether Sparse Autoencoders or other elements of DNNs should be fronted with batch normalization has two forks to the answer.

Is normalization helpful in reducing either signal saturation in gradient calculations during propagation or overflow conditions?
If that answer is, "Yes," will batching the normalization have some value for reasons like the ones described above?


How to implement a neural network for Flappy Bird in Python?

What you are trying to achieve, is a game that learns to play flappy bird. For doing this you need a neural network AND a genetic algorithm, those two things work together.
About your concerns on the output, you don't have to know if the action will benefit or not, i will soon explain why.

The neural network part

So, what you need is to know how to build a neural network, i don't know your knowledge about it, but i suggest starting from the basics. In this scenario, you need a feed forward neural network, because you just take the inputs from the current flappy bird scene/frame (such as the y position of the bird, the distance from the closes pipe ecc..) and feed it through a network that outputs either 1 or 0 (jump or don't jump) in the only output neuron we just decided it has.

In python you can implement a neural network from scratch, or using a neural network framework that does al the dirty work for you.

From scratch you would need to use numpy for matrix calculations, and you would need to learn matrix multiplication, dot products and all that fancy stuff (You can just let numpy taking care of the matrix calculations, but understanding how it works behind the scenes always helps understand new problems that you might come across when doing more advanced stuff)
Using a framework like Tensorflow for python, the only thing you need to do is find the right structure for the network you want to use. You will not have to worry about how activations work, or how the feed forward is performed (But again, it's a good thing to know when working with neural nets)

The genetic algorithm part or ""learning""

I say ""learning"" because at first sight it might look like learning, but really it is not.
The genetic algorithm works like "the survival of the fittest", where the "smarter" birds, which are the ones that reached the higher score on the current generation, will have a chance to have their child little birds, that have the same brain as their parent, with either some minimal modifications, or a mix of their parent brains.
The process of this ""learning"", so the genetic algorithm, works like so:

Create a generation of let's say 200 birds, every bird has a brain with random weights, so at the first run, they are all very...not smart
The game starts, and every frame of the game, the brain of the bird recieves as input some data that is taken from the current frame ( y pos of the bird, distance from pipe...)
The brain ( neural network ) of each bird, performs a feed forward with that data, and outputs what at the beginning is a very random result, let's say 0.75 for one bird
At this point you decided that 0.75 is greater than 0.5, so you take that as a 1, which stands for "jump", while if it was 0.3, so 0, the bird does nothing and keeps falling
Shortly the bird will die cause he has no idea of what he is doing, so he most likely collides with a pipe or the ground.
After all birds met their fate, you see that some birds reached further than others, so you choose, for example, 5 of the best performing ones.
Now you try to create a new generation of 200 birds using only the brains of those 5 that were choosen, by mixing and modifying theyr brains
Now the new birds have a brand new brain, that in some cases might be better than the previous one, so chances are that some of those birds will reach a higher score, therefore flap further into the level.
Repeat from point 6

In practice your "perform_genetic_algorithm" function in python, will have to choose the birds with the highest score, and as wild as it sounds, mix their brains and modify them, hoping that some modifications will improve the performance of the bird.

  I can't think of output since you don't really know if the action of flapping will benefit you or not

The mechanism above explains why you basically do not care at all about the output, except saying to the game engine: "hey the bird decided to flap, do it". Whether it's the right action or not, doesn't matter, as the smarter birds are naturally gonna get further and so be more likely to be choosen for next generation.

Hopefully now it's all more clear.

Here is some useful links for building a neural network and for understanding the genetic algorithm:

How to build a neural network: I am linking this because it contains all useful information about how to build a very basic neural network in python. In your case, you would have to ignore all the part about backpropagation, loss & error calculation and SGD, and just look at the feed forward part.
How to build a neural network - 2: This is another example of building a neural network that i found really useful, probably it's simpler and more straight forward than the previous link, but again, the backpropagation part is not needed for this genetic based learning.
Video tutorials on genetic algorithm: This is a very long but very explanatory playlist of videos that dives into the nature of genetic algorithms and how to implement one
Genetic algorithm optimization: Other source about genetic algorithms


Does NEAT require only connection genes to be marked with a global innovation number?

It is actually the other way around: connection IDs is what is debated!

Nodes always have innovation IDs (in the image, it is just their identifying number).

Node IDs are sufficient to identify connections. If a connection links nodes 3 and 6, then it is the same as another connection linking nodes 3 and 6: no need for an extra ID. So why the extra innovation IDs then?

On the one hand, this is an implementation choice: maybe these extra IDs would allow you to create a more complex but faster code?

On the other hand, there is a debate around whether a connection between two nodes means the same thing at different times in evolution. If you have no innovation IDs, then you cannot tell apart an old connection from 3 and 6 and another that was independently created later in a different genome (imagine the old connection was removed first). Is this relevant? As said, it is an open debate. Surely, it is not crucial at a basic level!

This question (and my answers) is related to this other question on Stack Overflow.


How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?

If you really just want an SMDP-version of the algorithm, which only needs to be capable of operating on the "high-level" time scale of macro-actions, you can relatively safely take the original pseudocode of whatever MDP-based algorithm you like, replace every occurrence of "action" with "macro-action", and you're pretty much done.

The only caveat I can think of in the case of $Q(\lambda)$ is that the "optimal" value for $\lambda$ is probably somewhat related to the amount of time that expires... so intuitively I'd expect it to be best if the value for $\lambda$ decreases as the amount of time expired during execution of the last macro-action increases. A constant $\lambda$ probably still works fine as well though.

If you actually want your algorithm to also be aware of lower-time-scale MDP underlying an SMDP, and not only treat macro-actions as "large actions" and be done with it... I'd recommend looking into the Options framework. There you get interesting ideas like intra-option updates, which may allow you to also perform learning whilst larger macro-actions (or options) are still in progress.

Last time I looked there hasn't been a lot of work involving the combination of eligibility traces and options, but there has been some: Eligibility Traces for Options. This paper doesn't specifically apply the algorithm you mentioned ($Q(\lambda)$), but it does discuss a bunch of other -- much more recent, and likely better -- off-policy algorithms with eligibility traces.


How should we understand the evaluation metric, AUC, in link prediction problems?

The paper you link to in the question is paywalled for me, so this answer may not be specific enough, but we'll make do.

Fundamentally, AUC is about measuring a classifier's robustness. The intuition behind the measure is that if we have classifier that outputs a score for an example, rather than a class label alone, we can choose to interpret that score in ways that trade detection rate (true positives) for false positive rate.

As a very simple example, imagine that we have a classifier that outputs a probability that an example pair is a link. By default we might chose to interpret this a probability of more than 0.5 as a link being present. Suppose this produces a certain true positive and false positive rate. Now let's change our mind. It's suddenly very important that we detect all the links, even if we get a lot of false positives. We choose to interpret any probability value > 0 as being a link. Our true positive rate will spike, but so will our false positive rate. Finally, if we care only about minimizing the prediction of fake links, we could chose to interpret only probability values that are >0.99 as being links. Our true positive rate will go way down, but our false positive rate will too.

AUC quantifies how rapidly the false positive rate increases as we increase the true positive rate by changing our interpretation of the scores the classifier outputs. Better classifiers will generally be able to increase their true positive rates quickly without introducing many false positives, while worse classifiers will not. An AUC of 1.0 indicates that we can achieve 100% true positive rate without increasing the false positive rate at all. An AUC of 0.0 indicates that increasing the true positive rate by 1% will always result in an increase of 1% in the false positive rate (i.e. the classifier is essentially just as good as a random guess without looking at the input).

In your link prediction problem, we cannot directly measure the false positive rate. However, if we treat all non-links as negative examples when we compute this measure, a classifier with a higher AUC will is one that can more rapidly separate positive examples from non-positive ones as you increase your sensitivity. Even though we aren't sure which examples are false positives and which are unseen links, a classifier with a high AUC is one where we would be more likely to trust that the non-links it labels as links are actually links we can't see.


Is it possible that the fittest individuals in an Artificial Life population may be successful by not actively pursuing the rules of the environment?

It is entirely possible!
You see, the agents will perform whatever actions are available to them, and if the evolutionary algorithm is setup correctly, whatever set of actions provides them with a bigger survival rate will be the one that gets explored and reproduced the most.
Here is a very interesting list of "Specification Gaming" in AI, where the agents happened to "game" the rules to reach their goals (metric optimization) without actually doing what the creators intended: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml (second link)


What are the differences between constraint satisfaction problems and linear programming?

LP is a mathematical problem to optimize a linear function subject to linear (in)equality constraints of the sort:
$$
min_x  w^t x
$$
$$
Ax<b
$$

where $x$ is a continuous variable (vector). If the problem is feasible, then a unique global optimal solution exists for $x$

A constraint satisfaction problem on the other hand is just the constraints part of the above LP. So we are not interested in the optimal solution, just a set of values that will satisfy the constraints. If the domain of your variables is continuous, then it can be recast as a dummy LP with a fake objective e.g. $min_x 0, s.t. Ax<b$, and then solved with an LP solver (e.g. simplex algorithm)

However many CSPs have discrete variables so they are not LP. Also CSP problems may have too many variables for plain LP algorithms to handle, so there are problem specific shortcut algorithms. We are often happy to find a feasible solution and forget about "optimality"


Is it legal to construct a public image database (for deep learning) with images from the internet?

I believe this is covered under Section 107 of the Copyright Act states:

the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright.

This section was intended to enable utilization of copyrighted material where that utilization is in the public interest. As long as you stay within the bounds as stated above you should not be considered in violation of a copyright.


Preventing bias by not providing irrelevant data

Why can't we just feed the training data without data that we would consider discriminatory or irrelevant, for example, without fields for gender, race, etc., can AI still draw those prejudiced connections? If so, how? If not, why has this not been considered before?

Yes. The AI/ model still can learn those prejudiced connections. Consider that you have a  third variable which is a confounding variable or has spurious relationship that is correlated with the bias variable (BV) and the dependent variable (DV). And, the analyst removed the BV but failed to remove the third variable from the data that is fed to the model. Then the model will learn the relationships the analyst didn't want it to learn.

But, at the same time the removal of the variables could lead to omitted variable bias, which occurs when a relevant variable is left out.

Ex:

Suppose that the goal is prediction of salary ($S$) of an individual and the independent variables are age ($A$) and experience ($E$) of the individual. The analyst wants to remove the bias that could come in because of age. So, she removes age from one of the models and comes up with two competing linear models:

$S = \beta_0 + \beta_1E + \varepsilon$

$S = \beta_0 + \beta_1^*E + \beta_2A + \varepsilon$

Since, experience is highly correlated with age, in presence of age in the model, it is very likely that $\beta_1^* < \beta_1$. $\beta_1$ will be a bogus estimate of a person's experience on salary as the first model suffers from the omitted variable bias.

At the same time the predictions from the first model would be reasonably good although the second model is very likely to beat the first model. So, if the analyst wants to remove any 'bias' that might come in because of age i.e. $A$ she must also remove $E$ from the model.


How can I use a Hidden Markov Model to recognize images?

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting.

HMMs are typically used in fields such as speech recognition and part-of-speech tagging. Here you observe a sequence of events that you want to fit to a model in order to classify the individual observations.

For training a HMM you would use something like the Baum-Welch Algorithm; for finding the most likely sequence (ie the recognition process) the Viterbi Algorithm is used.


Do we have two Q-learning update formulas?

The first one is the update rule that we use in the $Q$-learning algorithm.

The second one is the "definition" of $Q(s, a)$ values, although I would personally write it as follows, with an expectation around the reward, to also support cases where rewards might be non-deterministic;

$$Q(s, a) \doteq \mathbb{E} \left[ r(s, a) \right] + \gamma \max_a Q(s', a)$$

Here, the $Q(s', a)$ itself would also be similarly defined, and again that one is also assumed to be a "ground-truth" value.

In practice, when learning, we do not actually know what any of these $Q$-values exactly are; that's why we're doing learning in the first place, we're trying to learn what these values are! More precisely, when we wish to assign a new value to $Q(s, a)$, the definition tells us that we should use $Q(s', a)$ for that, but we can't because we don't know exactly what the correct $Q(s', a$) value is either.

We do generally have some $Q(S_{t+1}, a)$ values, but they're generally going to be only approximations, resulting from the previous steps of our own learning process (or maybe even randomly initialised values if we only just started training!). There may also be an additional approximation error in stochastic environments, where the true $Q(s', a)$ value may be a weighted average over multiple different possible successor states $s'$, but during the training we only observed a single concrete successor state $S_{t+1}$.

So, we can follow the definition and use $Q(S_{t+1}, a)$ as an approximation of $Q(s', a)$, but we know that it's just an approximation and not entirely reliable. Therefore, instead of fully applying the definition and completely replacing our $Q(s, a)$ value based on it, we interpolate between our current estimate $Q(s, a)$ and the new one that we should have according to the definition + our approximations. We only slightly shift towards it. The learning rate $\alpha$ determines how much we shift towards it. Normally we use a learning rate $0 < \alpha < 1$. Note that if you were to use $\alpha = 1$, you would actually recover the definition for $Q$-values again (except it would still use some observations to estimate unknown quantities).


What are the best hyper-parameters to tune in reinforcement learning?

Personally, I would choose the following two as the most important:

epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too much exploration might just waste your time.
learning rate: The learning rate determines how fast do you learn from new states of experience. A learning rate that is too high might not be good in cases when the environment has many states with high probabilities of negative rewards, i.e. many penalizations. This might make your agent move back and forth in the same place in order to avoid getting penalized. Also, a learning rate that is too low might make your agent learn very slowly and depending on your epsilon, the agent might enter a phase of exploitation with very little knowledge of an optimal policy.


Structure of LSTM Networks

[Are] units/neurons in the hidden layers ... referred to as memory blocks and each memory block can contain multiple memory cells?

Although an early guess at the way biological neurons work inspired the Perceptron design that later became multilayer (MLP) and then recurrent (RNN), these artificial network building blocks are not very much like biological neurons. The term is still used by some, but, because of this increased interdisciplinary awareness, the more common terms are cell, unit, or block.
These last two terms can be a little ambiguous when entire networks are used as building blocks in larger AI architectures. This is not a show stopper and also occurs in other engineering disciplines where sub-circuit blocks are components in more complex blocks, and sub-assemblies can be parts of larger mechanical assemblies.

And one memory cell looks like these two?

The images in the questions aren't clear representations. This visualization of an LSTM cell1 shows the attenuation parameters, each labeled with a capital W.

These LSTM cells are arranged as a layer and each pair of adjacent cells within the same layer are connected $(C_t, \, h_t) \rightarrow (C_{t+1}, \, h_{t+1})$. The input in this articular diagram is $x_t$ and the output to the next layer is $h_t$.

For units in the input and output layers, they are simply regular neurons, like those in a feed forward network?

They can be, but there are many topological designs in use that use LSTM cells in some layers and Perceptrons in others, along with side-by-side networks, bi-directional designs, recursive topologies, and completely different cell types, such as the very popular GRU cell.

[Is] the structure of an RNN essentially like that of a feed-forward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections ...?
LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections) ...?

No general rule can be assumed, especially since new designs emerge nearly every week, and significant topological advancements occur about once every one or two years. This is an example2 of a topology organized to support a specific processing use case.

References
[1] Visualizations of RNN units — Diagrams of RNN unrolling, LSTM and GRU, by Jakub Kvita, 2016, and inspired by Oinkina
[2] Automatic Image Captioning using Deep Learning (CNN and LSTM), Faizan Shaikh, 2018


Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & Attention (all three occurs in the index).
But read also Pitrat's last book: Artificial Beings: The Conscience of a Conscious Machine - it does cover machine learning (but not in the "deep learning" sense) but was published before 2016.
And see also RefPerSys. If you speak French, also see this and more generally the AFIA organization.


What does equation in the "related work" section of the GAN paper mean?

In full:

  The limit, as standard deviation $\sigma$ tends towards zero, of the gradient with respect to vector $\mathbf{x}$, of the expectation - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of any function $f$ of $\mathbf{x}$ plus $\epsilon$ is equal to the gradient with respect to $x$ of the same function of $\mathbf{x}$.

If we break that down:

$$\lim\limits_{\sigma \rightarrow 0}$$

  The limit, as standard deviation $\sigma$ tends towards zero of

$$\nabla_\mathbf{x}$$

  the gradient with respect to vector $\mathbf{x}$ of

$$\mathbb{E}$$

  the expectation ...

$$\mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2\mathbf{I})}$$

  [the expectation] - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of

$$f(\mathbf{x} +\epsilon)$$

  any function $f$ of $x$ plus $\epsilon$

$$ = \nabla_x f(\mathbf{x})$$

  is equal to the gradient with respect to $\mathbf{x}$ of the same function $f(\mathbf{x})$.

Basically it says that taking small perturbations of a vector input to a function and measuring the gradient at those varied points can be used to give you a valid estimate of the true gradient at the point you are making variations of.

In terms of understanding the equations, read introductory texts to the research area, and if like me your maths has been unused for many years before attempting this, expect to spend time and effort. Re-read the equations, memorise and write out the basic ones from the field, apply them to simple problems that might be presented in text books. Reading maths equations is not much different to reading music or reading another language - it takes concentration, practice, time and effort to become fluent enough to read an equation and comprehend it. The different research fields can be quite different too, some might be as similar enough to get by with what you know already, others may require learning all over again.

* I am not 100% certain of the interpretation of $\mathbf{I}$ as an identity vector - a matrix may be more appropriate, which depends on the form of $\mathcal{N}(\mu, \sigma^2)$ when handling vector distributions. A matrix form for the second argument would be more general and allow for covariance, although the use of $\mathbf{I}$ would then explicitly remove covariance and make each component of $\epsilon$ independent, which is required for this result.


Should I use additional empty category in some categorical problems?

In short: yes, you must allow "do nothing" decision as a first level result.
Your system must decide the action to be taken, including "do nothing" action. This is different to low network outputs, that can be translated as "don't know what to do".
In other words, the network can result in:

"I don't know what to do now" when all results in the output have low probabilities. (Obviously, this is a bad network result, to be fixed as much as possible).
"I know I must do nothing", when "do nothing" action has high probability, greater than the others.
"I know I must do W", when "W" action has high probability, greater than the others.
...

Kind regards.


Does an AI exist that can write software based on a formal specification?

I think that the answer to your question is yes. In the article New A.I. application can write its own code, the authors state

Computer scientists have created a deep-learning, software-coding application that can help human programmers navigate the growing multitude of often-undocumented application programming interfaces, or APIs.
Designing applications that can program computers is a long-sought grail of the branch of computer science called artificial intelligence (AI). The new application, called Bayou, came out of an initiative aimed at extracting knowledge from online source code repositories like GitHub. Users can try it out at askbayou.com.

The paper Neural Sketch Learning for Conditional Program Generation may also be useful.


What prevents us from developing artificial consciousness?

It may be helpful to think of consciousness, like intelligence, as a spectrum. The Stanford Encyclopedia of Philosophy, under the section "Creature Consciousness" (2.1) defines sentience as:

Sentience. It may be conscious in the generic sense of simply being a sentient creature, one capable of sensing and responding to its world (Armstrong 1981). Being conscious in this sense may admit of degrees, and just what sort of sensory capacities are sufficient may not be sharply defined. Are fish conscious in the relevant respect? And what of shrimp or bees? Stanford Encyclopedia of Philosophy, 2.1 "Creature Consciousness"

It seems clear that AI possesses the most basic form of consciousness in terms of "sensing and responding to its world".  I might liken this level of consciousness to that of microorganisms, able to interact with their environments but possessing none of the higher functions of humans, or even higher-order animals.
We don't really understand the true nature of higher consciousness and human-level sentience, which doesn't necessarily mean that an automata with sufficient complexity couldn't achieve it.  But there is a deeper problem relating to validation (i.e. how would be know the system is truly self-aware, and not simulating self-awareness.  [See Searle's Chinese Room, but also McCarthy's Refutation]
In his paper Making Robots Conscious of their Mental
States McCarthy states:

Conscious knowledge and other information is distinguished from
unconscious information by being observable, and its observation results
in conscious knowledge about it. We call this introspective knowledge.
A robot will need to use introspective knowledge in order to operate
in the common sense world and accomplish the tasks humans will give
it.
Many features of human consciousness will be wanted, some will
not, and some abilities not possessed by humans have already been
found feasible and useful in limited domains.
We give preliminary fragments of a logical language a robot can
use to represent information about its own state of mind.
A robot will often have to conclude that it cannot decide a question
on the basis of the information in memory and therefore must seek
information externally.
Programs with much introspective consciousness do not yet exist.
Making Robots Conscious of their Mental
States, Abstract

This still seems to be the case, however, McCarthy seems hopeful about the "computationalist" approach:

Thinking about consciousness with a view to designing it provides
a new approach to some of the problems of consciousness studied by
philosophers. One advantage is that it focusses on the aspects of
consciousness important for intelligent behavior. If the advocates of
qualia are right, it looks like robots won’t need them to exhibit any
behavior exhibited by humans.ibid.

The continual increase in memory, processing and algorithmic sophistication may well yield Artificial General Intelligence.
Even if this proves to be the case, I suspect the debate over artificial consciousness will rage on.

See also: Artificial Consciousness


Minimax combined with machine learning to determine if a path should be explored

The use of of a neural network to push the search algorithm to continually only along a promising path is the same that was described in the AlphaZero paper. In AlphaZero, the NN loop contained the search function and would encourage the continued search of high probability moves that were then simulated by the same NN that now contained the Value Net. The use of alpha-beta specifically is not necessary. Just a search function aptly known as PUCT (Predictor + Upper Confidence Bounds applied to Trees)


This is called "clustering" , If the network is already trained with data that has similar features as of the "symbols", you can use that network with its last classification layer removed , then run a clustering algorithm like "k-means" on top of the vectors obtained from the last layer of the network.


It's not supposed to be derived from some equation. That is the basic premise under which GANs work. The output of the Generator $G(z)$ is fed as an input $x_g$ to the discriminator.


Confusion about temporal difference learning

My first question is whether the following "implementation" of the (0) algorithm for the first two of the above observed trajectories correct?

$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad V(b)\leftarrow0+0.1(1+0-0)=0.1$
$V(b)\leftarrow0.1+(0.1)(1+0-0.1)= 0.19$

Your calculations for the first trajectory $(A,1,B,0)$ is incorrect for either TD or Monte Carlo. For some reason you have assigned either an immediate reward or return of $1$ to the second step, whilst in the example, it is $0$ for both the sampled return and the single-step TD target.
In addition, you quote this update rule for single-step TD:
$$V(s_{t}) \leftarrow  V(s_t)+ \alpha \left[ G_{t+1}+\gamma V(s_{t+1})- V(s_t) \right]$$
. . . actually that is not the usual notation. The symbol $G_t$ is normally used to show a "return" value - a sum of rewards (often weighted by some factor, such as $\gamma$). The usual way of showing the TD update rule would be:
$$V(s_{t}) \leftarrow  V(s_t)+ \alpha \left[ r_{t+1}+\gamma V(s_{t+1})- V(s_t) \right]$$
i.e. using the immediate reward. This might be a simple typo, however I am explaining this because it may behind your incorrect calculation.
The correct calculation is not very much different from yours though:

$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad V(b)\leftarrow0+0.1(0+0-0)=0.0$
$V(b)\leftarrow0.0+(0.1)(1+0-0.0)= 0.1$

If so, why dont we use the updated value function for $V(b)$ to also update our value for $V(a)$?

You can, and would do this in either of the following situations:

In online learning, you experience a trajectory with states in order (A,B) again

In offline learning, you repeat the previous experience in batch learning or using experience replay

It is worth noting that if you take a small batch of data and repeat it again and again to update the value functions, that they will converge to values depending on your data set. That is what the slide is explaining in the lecture - highlighting the difference that TD and MonteCarlo will make when you do this. However, if that data set is a very small subsample of possible random behaviour in the environment, then you may not create an accurate value function, but instead it will be the best one that you can given the limited data. If it is easy to collect more experience, then that is often preferable.

Why dont we just use the [maximum-likelihood]-estimate immediately?

Because it is not directly useful for a value prediction task, and you would need some mechanism to use that maximum likelihood MDP model to generate value predictions. With TD, you are already in the process of making this estimate*.
You could take the existing samples, and use them to generate the parameters of an MRP (Markov Reward Process, as there are no example actions in the trajectory) based on the observations. That "best guess" MRP is your maximum likelihood MDP model, and would evaluate the same as your converged repeated TD batch over the samples.
The main difference explained by the slide is that Monte Carlo will converge to $V(a) = 1$ because the only sample with A in it has a return of 1 following state A. Whilst TD will converge to $V(a) = 1.75$, because it treats the same sample as the only instance of state progression from A - e.g state A "always" has an immediate reward of 1 then goes to state B. Both algorithms will converge to $V(b) = 0.75$.

* There are algorithms, such as Dyna-Q, which partially do this, using experience gathered so far to create a model of the environment dynamics. Sometimes this is useful and effective. However, it is not always possible or the best approach.


How to start looking for machine learning opportunities in projects

If you are trying to see whether machine learning can contribute to your work, first thing you ought to do is to see if any of the following tasks could help the business at hand. Machine learning tasks include the following:

Classification
Regression
Clustering
Density estimation
Dimensionality reduction


How do I classify measurements into only two classes?

The way to categorize measurements into two separate populations is through what ML people currently term unsupervised learning. Such a process is part of the AI tool chest. Statistics is part, but not all, of the mathematics involved in the theory that leads to algorithms that learn without labeling the array of points light or dark in advance.

Reflected IR light can be specular or mirror in nature and significant levels of IR emission will also occurs, without raising the surface temperature of materials under most common circumstances. There is also refraction and transmission to be considered in the robotics space.

Materials will not exhibit lightness. The material composition, coating, and surface texture will exhibit various reflectivities, emission dependent on the absolute temperature raised to the fourth power1, and transmission of light entering the material from various directions, possibly diffused internally.

  We eventually hope to be able to use this to follow a line using a pre-set algorithm.

That is one of the primary objectives of computer vision and has been since the first digital signal processing in the mid twentieth century.

  The first step is determining whether the material is one of the binary options, light or dark.

That may be an experimental first step, but the loss of information from taking a number that discretely represents a cube in horizontal-vertical-frame space, each of which has $\tau$ channels of $\beta$ bits, is not a good computer vision strategy. The total number of bits of IR information is given by

$$ \tau \beta \; \text{,} $$

so the data loss will be, in percent,

$$ \dfrac {100 \, (\tau \beta - 1)} {\tau \beta} \; \text{.} $$

Object recognition and thus collision avoidance will be frustrated by that much data loss, even if only one IR spectral range is contained in the cube ($\tau = 1$).

  Given a large population of values between 0 and 1023, probably in two distinct groupings, how can I best go about classifying a given point as light or dark?

It appears that $\tau$ does equal one and $\beta = \log_{2} (1023 - 0 + 1) = 10$, thus the information in bits per cube (pixel in a frame) is 10. If the project, for some reason, requires that 9 bits be discarded in a way to conserve as much of the original 10 bits as possible (perhaps to maintain a particular throughput through a transmission or processing bottleneck), then the way to do it is through a one bit feature extraction, meaning that the output of the artificial network should be one bit. These are a few unsupervised learning options to investigate.

RBMs (restricted Boltzmann machine)
K-means clustering
Gaussian mixture models

Most AI and artificial network frameworks and libraries will have examples of these in the example directory. It is advisable to approach the project like this.

Find web pages or articles using the above as search terms until an explanation that is clear to the team is found.
Study all three for at least a few hours each as a team.
Find a library or framework that contains all three.
Install the necessary prerequisites for all three on the lab computer(s).
Produce a working example of the three types so that selection is not a function of which one is easiest to get running (something which would qualify as an research anti-pattern).
Make a choice.
Change the working code, one verifiable step at a time into the desired unsupervised learning program.
Eventually, when thoroughly ready, make the jump from the example data set to the robotic IR light frame set.

Footnotes

[1] Stefan-Boltzmann law


You could make the actions

move up
move down
move left
move right
shoot

Then you can declare the speed of this movement. If you want to go more in depth you can do

Accelerate up
Accelerate down
Accelerate left
Accelerate right
Shoot

This will give control of the speed to the neural networkm but is harder to train, and you should give the speed of the mouse as an input.


Will robots rebel against their human creators?

I don't like to be a killjoy, but this question seems premature (that's why it's hd the "mythology of AI" tag added to it). The kinds of emergent artificial general intelligence depicted in the movies you mention are in science fiction films because they are science fiction. Most AI researchers do not think they are likely to appear anytime soon. The overwhelming majority of researchers think the most likely times for such a system to appear are "More than 50 years [from now]" or "Never". In part, this is because AI researchers thought we were close to such systems for several decades, despite failing to create them. This suggests that making an artificial general intelligence is much harder than we might expect.

Despite AGI being a long show, there's a lot of recent interest in the AI research community in the social impact of our technologies. The study of these systems is called "ethical AI", and this is the path that the research community as a whole has begun to embark on. A promising approach is to model the process by which humans decide to treat each other well, in the hopes of creating programs act according to that process.


Why does Clipped Surrogate Objective works in Proximal Policy Optimization

Ok, so I think I have a better understanding of this now.

Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.

So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.

This mechanism allows you to perform small but relevant updates of your policy.

hope this will help someone :)


How to describe an keras Model in a scientific report

Some other details you could mention are:

total number of model parameters (e.g. 1.2M or 0.15M) & depth of the network (e.g. 38-layered network)
family/style of the network architecture (e.g. encoder-decoder arch., LSTM)
specifics of connections between network layers (e.g. residual-, dense-, skip-connections)
specifics of individual components of the network structure (e.g. dilated-convs. (CNNs), attention (LSTMs))
description/reasoning of why you chose a particular structure/sequence of connections in your deep learning model
specifics of training/validation/testing procedures (e.g. augmented training data, cross-validation, test-time-augmentation (TTA), frozen network weights)
other specific details/caveats that allow the results of your deep learning model be easily reproduced from the scientific report

For more info on the best kinds of details to be included in the report, refer to "Methodology"/ "Training"/ "Implementation"/ "Proposed Architecture" sections of the deep learning research papers in your relevant area.


How to use 'Canny/Watershed' algorithm's output as an input for Image Classification Model

First and foremost, I have to say that this could (and likely will) be a very hard task. Neural networks (NNs) have excelled at computer vision tasks identifying everything from textures to complex objects but what you are trying to do goes beyond that. We (humans) identify trash using the context as much as the object. An object on a table and the same object in a trash bin can look identical but the context tells us which one is trash. Also, anything can be trash. Trash is not an object, its a state.

Having said all that, it sounds like an interesting project and it would be a very useful model so I'll do my best to explain how you would go about trying this. As per your comment, you have outlined the two steps you need.

Identify and extract the trash object from the image.
Classify the type of waste

This could be accomplished using a single model but I'll explain how to do it with two models to make clear what is being done.

Identify and extract trash

The objective of the first model is to identify and extract the region of interest (the part of the image containing the trash). This is an image segmentation task typically accomplished using an R-CNN - some guides explaining how they work can be found here, here or here. These methods use supervised learning which requires masks delineating the object of interest to be used as the ground truth. A mask is a binary image the same size as your input image where each pixel in the image representing your positive class (trash) is set to 1 and all others are set to 0.

The output of your Canny/Watershed algorithms could be used as the masks to train the model, however your model will only ever be as good as your masks. Therefore, you might as well use your Canny/Watershed algorithms to accomplish task 1. If the masks generated by your algorithm are not of sufficient accuracy you will need to find another way to generate your masks - maybe even doing it manually.

An approximate rule of thumb for object detection with NNs is that you need 1,000 representative images per class, where class implies a specific object. In this case, unless you are attempting to identify very specific items of trash, you would likely need 100s of thousands of images to obtain a high degree of accuracy.

Classify the type of waste

This task should be easier, making the assumption that the segmented image from the first step is always trash. By assigning a label to each segmented image (paper, metal, glass, cardboard, etc) it becomes a normal multi-class classification task which are well documented online with lots of explainations and tutorials.

Single model

These tasks could be combined into a single model by modifying the masks in step 1. Instead of using a single binary mask, you could create a multichannel mask of shape m x n x o where m x n is your input image size and o is the number of kinds of waste you are attempting to identify. Each channel is a binary mask for a given type of waste. Therefore, it becomes a matter of not just identifying and segmenting trash but of segmenting each type of trash separately. Needless to say the complexity of the model would be a lot higher and so would the process to create the masks.


How do intermediate layers of a trained neural network look like?

tl;dr

Your intuition is correct.

Why is this a problem?

The saturating effect of the sigmoid activation function is well documented as it is the main culprit of a problem called vanishing gradients.

In short, the derivative of the sigmoid function is:

$$
\frac{dσ(z)}{dz} = σ(z) (1 - σ(z))
$$

The problem is that very often (especially if we initialize our weights with large values), the output of this neuron will either be $1$ or $0$. This causes the gradient to be $0$, which in turn means that the weights of that neuron can't be updated.

So let's keep the weights small (in absolute value)

Initially this seems like a good idea, if we keep the weights small we will avoid saturation.

However there is another problem with sigmoid functions: their gradient has a maximum value of $0.25$! This means that if the weight of a neuron is less than $4$, the error will diminish while flowing backward through the net. This becomes progressively worse as we add more layers to the network.

How have we solved this issue

Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad.

One example is Mishkin et al. 2016 who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.

Another workaround is to use a different learning rate for each layer (Xu et al. - Revise Saturated Activation Functions)

After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.

Sources

This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.

If you are interested in reading about this, I'd recommend this post,  by Andrej Karpathy.

Some more formal sources on this topic:

Hochreiter et al. - Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter et al. - Recurrent Neural Net Learning and Vanishing Gradient
Pascanu et al. - On the difficulty of training recurrent neural networks
Bengio et al. - Learning long-term dependencies with gradient descent is difficult


What are the main problems hindering current AI development?

we don't really know what intelligence is.
we don't truly understand the best model of intelligence we have available (human intelligence) works.
we're trying to replicate human intelligence (to some extent) on hardware which is quite different from the hardware it runs on in reality.
the human brain (our best model of intelligence) is mostly a black-box to us, and it's difficult to probe/introspect its operation without killing the test subject.  This is, of course, unethical and illegal. So progress in understanding the brain is very slow.

Combine those factors and you can understand why it's difficult to make progress in AI. In many ways, you can argue that we're shooting in the dark. Of course, we have made some progress, so we know we're getting some things right. But without a real comprehensive theory about how AI should/will work, we are reduced to a lot of trial and error and iteration to move forward.


Would it be possible to determine the dataset a neural network was trained on?

You can already do this with some neural networks, such as GANs and VAEs, which are generative models that learn a probability distribution over the inputs, so they learn how to produce e.g. images that are similar to the images they were trained with.
Now, if you're interested in whether there is a black-box method, i.e. a method that, for every possible neural network, would tell you the dataset a neural network was trained with, that seems to be a harder task and definitely an ill-posed problem, but I suspect that people working on adversarial machine learning have already attempted or will attempt to do something similar.


Is some kind of dropout used in the human brain?

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons connected to it. How other neurons connected to the former neuron respond on the messages sent by it, depends on the strength of the connection between the connected neurons.
However, having true dropout in the brain would be like having a neuron stop firing entirely for some time. As far as I know, this is not gonna happen unless a neuron dies. So, no, there doesn't seem to be any real biological equivalent to dropout in the brain. If you forget something, that is just caused by a weakening of some connections in the brain. But such a weakening (or strengthening alternatively) is a gradual process and is a function of the degree of firing synchrony between any two connected nodes.
It is true that the introductory texts to Neural Nets tell you about those nets' neuro-scientific inspiration, but this similarity holds only in the most abstract way. Yes, you have neurons in both biological and artificial neural networks. But communication in the brain works in an asynchronous, continuous bio-chemical way, while communication in an artificial neural net works by taking the numeric state of one node, scaling it by the weight of the connection between two nodes, and then adding the weighted contribution of the previous node to the activation of the receiving node. So, already the fundamentals are very different between the two approaches (with the biological variant being much more complex and elaborate).
If you want to find something that possesses more biologically inspired properties, look at how Convolutional Neural Networks are theoretically inspired by the working of the (human) visual system. Besides that, dropout is a purely mechanical strategy to regularize a model well, in order to enhance the model's generalization ability, i.e. making the model work well also on unseen data. It's just part of an algorithmic optimization procedure.


Is which sense was AlphaGo "just given a rule book"?

it was "just given the rulebook", what does this mean? Literally a book written in English to read?

The program was not given a natural language version of the rules to interpret. That might be an interesting AI challenge in its own way, but none of the current cutting-edge game playing reinforcement learning systems do much in the way of natural language processing.
Instead, "just given the rulebook" is a rough metaphor for what actually happened: The rules of Go were implemented as functions that the game-playing agent could query. The functions can answer things such as "when the board looks like this, what are my valid actions?" and "if the board looks like this, have I won yet?". The board state might be represented by a matrix with stone positions encoded using numbers. Outputs might be a similar matrix of numbers for valid action choices (where the player is allowed to put stones in Go) or perhaps a single number, $0$ for not won yet, $1$ for a move that wins the game.
There may even be further helper functions that help assess moves (e.g. a value for how many enemy stones would be captured if a piece was played in a specific location), but the bare minimum needed is "what moves are valid?" and "has anyone won?". A very common third function, useful for look-ahead planning is "if the board starts like this, and I take that action, what will the board look like next" - with this function, an agent can look ahead to future positions to help search for winning moves.
This approach is common in game playing agents. A learning agent can in theory also learn the rules of the game through trial and error, as long as it receives some feedback when it has broken the rules. However, more often the goal of training the agent is only to play as well as possible - learning the rules from scratch would be extra work for the agent, and maybe not an interesting problem to solve. So the agent is given helper functions by the developers that allow it to explore only valid moves according to the rules.


How can an AI train itself if no one is telling it if its answer is correct or wrong?

By "company A has a large human face database so that it can train its facial recognition program more efficiently" the article probably means that there is a training dataset $S$ of the form

$$
S = \{ (\mathbf{x}_1, y_1), \dots,(\mathbf{x}_N, y_N) \}
$$

where $\mathbf{x}_i$ is an image of the face of the $i$th human and $y_i$ (which is often called a label, class or target) is e.g. the name of the $i$th human. So, the programmer provides a supervisory signal (the label) for the AI to learn. The programmer also specifies the function that determines the error the AI program is making, based on the answer of the AI model and $y_i$.

This way of learning is called supervised learning (SL). However, there are other ways of training an AI. For example, there is unsupervised learning (UL), where the AI needs to find patterns in the data by aggregating objects based on some similarity measure, which is specified by the programmer. There's also reinforcement learning (RL), where the programmer specifies only certain reinforcement signals, that is, the programmer tells the AI which moves or results are "good" and which ones are "bad" to achieve its goal, by giving to the AI, respectively, a positive or negative reward. You can also combine these three approaches and there are other variations.

  Are there any engineers who are constantly telling an AI what it produced it correct or wrong?

Yes, in the case of SL. In the case of RL, the programmer also needs to provide the reinforcement signal, but it doesn't need to explicitly tell the AI which action it needs to take. In UL, the programmer needs to specify the way the AI needs to aggregate the objects, so, in this case, the programmer is also involved in the learning process.


Can Q-learning working in a multi agent environment where every agent learns a behaviour independently?

Not particularly sure what you are asking, so the question that I will be answering is this:

  Can Q learning be used to estimate a value that depends on another value in the Q Learning Matrix even though there is a certain amount of unpredictability involved?

The answer is yes!

I will use the example of a robotic arm trying to reach a point in space since that is what I am most familiar with.

Imagine a robotic arm with a shoulder, elbow, and wrist joint. The desired elbow value depends very much on the shoulder value which is also being learned. Given enough iterations, the Q learning algorithm will come up with a solution (out of possibly many) for the elbow joint based on where the shoulder joint is at that time.

The intrinsic unpredictability (air drag, motor power, etc in this case) is countered intuitively by the Q-learning algorithm iteratively learning what works best.


Can we train the model to detect real users with only positive labels?

The problem which you have is a classification problem. You assume a class "good users" and a distinct class "bad users". You want to train an AI to tell the two apart, but all your examples are "good users". Any reasonable AI will draw the logical conclusion from those examples: all users are good users. That's a 100% match for the training data.


How to do early classification of time series event with small dataset?

One kind of system you could look into are Echo State Networks (ESNs).
They are relatively cheap to train and can learn to predict output signals to an  arbitrary degree of precision.
All you need to train the system is some labeled training data.
Thus, if you have a sequence of measurements/feature values and the corresponding sequence of class labels, you can train and fine-tune these kinds of systems to output some required class label very quickly after the onset of an event encoded in the measured feature values.


Problems that only humans will ever be able to solve

Informally, AI-complete problems are the most difficult problems for an AI. The concept is not mathematically defined yet, as e.g. NP-complete problems. However, intuitively, these are the problems that require a human-level or general intelligence to be solved.

Real natural language understanding is believed to be an AI-complete problem (this is also discussed in the paper Making AI Meaningful Again by Jobst Landgrebe and Barry Smith, 2019). There are a lot more AI-complete problems. For example, problems that involve emotions.


In my experience, the fitness function is a way to define the goal of a genetic algorithm. It provides a way to compare how "good" two solutions are, for example, for mate selection and for deleting "bad" solutions from the population.

The fitness function can also be a way to incorporate constraints, prior knowledge you may have about the shape of the fitness landscape, or the way your crossover/recombination operators will work in that fitness landscape.

For example, the fitness function can include hard constraints like "Genes x,y, and z must all stay on one side of the surface $Ax +By +Cz = k$" by assigning the fitness value at zero if the gene values are on the wrong side of the surface.  However, it's often better in a case like that soften the boundary by assigning a fitness penalty that is zero at the surface and grows larger as the gene values move farther from the surface on the wrong side of the surface.

Different fitness functions can be used for mate selection vs deleting "bad" trial solutions.  For example, "mating fitness" between two potential parents A and B can be a function of how different the two parents are.  By providing a mating advantage to pairs that are significantly different, the population can be forced to remain fairly diverse and thus explore a larger region of solution space, or to avoid converging to local (sub-optimum) fitness maxima.  Meanwhile, the usual kind of fitness will cull the low-fitness individuals from the population and drive evolution toward high fitness.

What is often much more important is the set of variables ("genes") used to represent a trial solution, how the genes are arranged in the "chromosome", and the ways genes from two parents can be combined to form a new trial solution. Since you didn't ask about those things I won't go into detail in this answer, but if you ask in a separate question I will provide a detailed answer.


Is it theoretically possible (or impossible) that principal component analysis worsens the performance of the model?

PCA can make models worse,  imagine data points scattered along two elongated parallel rectangles. The axis with the greatest variation will be parallel to the rectangles but doesn't provide any benefit in classifying the points.


What are the characteristics of a deep learning AI?

As you can see from the picture above (In courtesy to this guy post). Deep learning is a field inside machine learning which involves using deep neural networks to solve machine learning problems.

Tensorflow.js and Brain.js are software architectures built in order to assemble and develop ML models inside the browser.

If you would like to delve into this field of deep neural networks you should start from the theory to really understand why we are using neural networks to solve problems and which problems they are intended to solve.

You can start with this awesome video by 3blue1brown : Intro video

And this standford class for CNN: Stanford class


Multi-Label Image Classification using KNN

Have you tried google searching? There are many results that come up for "multi-label k nearest neighbor image classification". Here are the first couple of papers to get you started:

http://proceedings.mlr.press/v25/chiang12/chiang12.pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.9501&rep=rep1&type=pdf


A unified neural network model consists of one neural network as opposed to other models that rely on two or more neural networks.
For example, from page two of the YOLO paper:

2. Unified Detection
We unify the separate components of object detection into a single neural network. Our network uses features from the entire image to predict each bounding box. It also predicts all bounding boxes across all classes for an image simultaneously. This means our network reasons globally about the full image and all the objects in the image. The YOLO design enables end-to-end training and realtime speeds while maintaining high average precision.

In the paper by Xu and Wang, they add a branch to handle tracking to the Faster R-CNN architecture, which was designed for object detection.  This is a 'unification' of two models.
In the paper by Ebrahimi et al., on pages 42-43, you can see they are fusing multiple neural networks into one unified model:

We propose a unified user geolocation method that relies on a fusion of neural networks, incorporating different types of available information: tweet message, users' social relationships, and metadata fields embedded in tweets and profiles.


Do correlations matter when building neural networks?

If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.
When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.
Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).
Besides that: Commonly, you split your training dataset into 3 parts:

Data used for fitting the model (actual training data)
Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
Test data used to assess the performance of the system after all training & intermediate testing is done

The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.
So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.


How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.


If neurons are only defined for values between 0 and 1, how does ReLU differ from the identity?

You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.


Top Frequent occurrence word effect in Model Efficiency?

As far as I know, there are few aspects that would probably improve the model score:

  Normalization
  Lemmatization
  Stopwords removal (as you asked here)

Based on your question, "is removing top frequent words (stopwords) will improve the model score?". The answer is, it depends on what kind of stopwords are you removing. The problem here is that if you do not remove stop words, the noise will increase in the dataset because of words like I, my, me, etc. Here is the comparison of those three aspects using SVM Classifier.

You may see that without stopwords removal the Train Set Accuracy decreased to 94.81% and the Test Set Accuracy decreased to 88.02%. But, you should be careful about what kind of stopwords you are removing.

If you are working with basic NLP techniques like BOW, Count Vectorizer or TF-IDF(Term Frequency and Inverse Document Frequency) then removing stopwords is a good idea because stopwords act like noise for these methods. If you working with LSTM’s or other models which capture the semantic meaning and the meaning of a word depends on the context of the previous text, then it becomes important not to remove stopwords.

So, what's the solution?

  You may want to create a Python package nlppreprocess which removes stops words that are not necessary. It also has some additional functionalities that can make cleaning of text fast. For example:

from nlppreprocess import NLP
import pandas as pd

nlp = NLP()
df = pd.read_csv('some_file.csv')
df['text'] = df['text'].apply(nlp.process)

Source:

https://github.com/miguelfzafra/Latest-News-Classifier
https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52


What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?

Ans 1.
The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can write this expression as-
$
KL(q||x) = log(p(x)) - \Sigma_z q(z)log(\frac{p(x,z)}{q(z)})
$
For a given x, the first term of RHS is constant. So we maximise the second term so that the KL divergence goes to zero.
We can write the second term as
$E_{q(z)}[log(p(x|z)] - KL(q(z|x)||p(z))$
(try writing p(x,z) as $\frac{p(x|z)}{p(z)}$ and then expand. Here, p(z) is the distribution of our choice, i.e. a Gaussian distribution). We argue that the process from z to x is deterministic and z is gaussian. So, the first term could be replaced by $exp(log(-||x-\hat{x}||^2))$(we replaced z by $\hat{x}$ because it's deterministic- this is now the exact proof). So we have-
$
maximize(-||x-\hat(x)||-KL(q(z|x)||p(z)))
$
and we get our loss function.
We also know that variational autoencoders almost never find the optimal solution, so I am not sure how would playing around with the weights affect it(Nor do I know if it makes sense mathematically).
Ans 2.
We can say that the KL divergence has a regularising effect.
This page has some nice experiments which will help you understand what happens to the latent space when you decrease the KL divergence part.
Ans 3.
Yes, you can say that. You are fixing the dimensions, but are lenient on the distribution. In fact, you are approaching autoencoders by doing this.
Separate-
I want to point you towards this article. It explains why we choose to minimise $KL(q(z|x)||p(z))$ instead of $KL(p(z)||q(z|x))$ (the latter is intractable) and what would happen if we choose less independent variables for our estimator q(z).
Also, have you tried increasing the dimensions of the latent space? It can also have a 'de-regularizing' effect. It seems that the model is underfitting the data- the reconstruction loss is high with the normal loss, compared to when you decrease the regularizing term.
Hope it helps.


Why does TensorFlow docs discourage using softmax as activation for the last layer?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a softmax + crossentropy module all-in-one, then it becomes numerically stable (y-p)

Slides from DeepMind's Simon Osindero lecture at UCL in 2016:


XOR Neural Network gets stuck in training

One Neuron on its own can only solve linearly separable problems.
You need a combination of Neurons to solve non-linearly separable problems.
For the XOR case, you need at least 2 Neuron at the first layer, and 1 Neuron at the Output layer to properly classify it.
Keep in mind sometimes the 3 Neuron network might get stuck in a local minima as well, you will need some luck in the random initialization of weights.
Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.


Can a neural network be used to detect sine waves?

Is this a task suited for a neural network

Yes. You have choices in fact:

A fully-connected network would be simplest architecture, and would work if you gave it some time window of samples (e.g. every 0.5 seconds or every 50 samples) and supervised training data - sets of samples with sensor readings and the ground truth value of whether the motor was on or not.
A 1D convolutional neural network would likely be most efficient and robust to train, and would take the same inputs and outputs as the fully-connected network.
A recurrent neural network would be tricker to train, but a nicer design as you could feed it samples one at a time. The input would be the current sample, and output the probability that the motor was on. When training this, you would also want to provide it transitions between the motor being on and off. The nice feature about this is that it should give you quick feedback about whether the motor was on or off - with the caveat that it may be more likely to trigger intermittent false positives, so a little extra post-processing might be required.

All of the above require you to collect training data, ideally in situations identical to planned use of the detector. So if the motor is mounted somewhere that could experience other vibrations, a few of those kind of scenarios should be simulated with motor both on and off.

  and if so, would it be less efficient than other means of detection?

In terms of computing power and effort on your part, you may find that an off-the-shelf Fast Fourier Transform (FFT) library function with a simple threshold at your target frequency will make a robust and simple detector, with no need for a neural network.

Typically for specific frequency detection you would take a window of samples, adjust them (using e.g. Hamming window) to reduce edge effects which would appear as frequencies in the conversion, and then run FFT. This combination is so common that you may find it already combined in the FFT library. For more on this, you would want to ask in Signal Processing Stack Exchange, where use of FFT is well understood.

If the environment is noisy or the target frequency can drift (making it hard to set a simple threshold) then you could also combine FFT with a neural network. This combination can solve much more complicated signal detection, and is used in speech processing for instance.

  sampled at 100Hz - it is possible to increase the sample rate to 200Hz or 400Hz at max

For reliably detecting a 50Hz signal, I would say that 200Hz sample rate is minimum. The theorectical minimum is 100Hz (i.e. twice the signal frequency) but may give you problems with noise and the possibility that your sample points just happen to fall on low amplitude parts of the oscillations, making it look like the motor is off even when it is on.


Can gradient descent training be used for nonsmooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that trainable parameters receive gradients that point in a direction that decreases the loss over time.

In the absence of a differentiable loss function, the true gradient must be approximated through other methods. For example, in classification problems, the 0-1 loss function is considered the "true" loss, but it is non-convex and difficult to optimize. Instead, surrogate loss functions act as tractable proxies for true loss functions. They are not necessarily worse; negative log-likelihood loss gives a softmax distribution over $k$ classes rather than just the classification boundary.

For your problem specifically, $f(x,a)=min(x,a)$ is not a differentiable loss function. It is not differentiable at $x=0.5$, but the gradient could be estimated through the subgradient. In practice, this works because neural networks often don't achieve the local/global minima of a loss function but instead asymptotically decreasing values that achieve good generalization error. Tensorflow and PyTorch use subgradients when fed non-differentiable loss functions. You could also use a smooth approximation of the $min$ function (see this thread) to get better gradients.


What is the difference between graph convolution in the spatial vs spectral domain?

Spectral Convolution
In a spectral graph convolution, we perform an Eigen decomposition of the Laplacian Matrix of the graph. This Eigen decomposition helps us in understanding the underlying structure of the graph with which we can identify clusters/sub-groups of this graph. This is done in the Fourier space.
An analogy is PCA where we understand the spread of the data by performing an Eigen Decomposition of the feature matrix. The only difference between these two methods is with respect to the Eigen values. Smaller Eigen values explain the structure of the data better in Spectral Convolution whereas it's the opposite in PCA.
ChebNet, GCN are some commonly used Deep learning architectures that use Spectral Convolution

Spatial Convolution
Spatial Convolution works on local neighbourhood of nodes and understands the properties of a node based on its k local neighbours. Unlike Spectral Convolution which takes a lot of time to compute, Spatial Convolutions are simple and have produced state of the art results on graph classification tasks. GraphSage is a good example for Spatial Convolution.

Additional References: https://towardsdatascience.com/tutorial-on-graph-neural-networks-for-computer-vision-and-beyond-part-2-be6d71d70f49

https://towardsdatascience.com/graph-convolutional-networks-for-geometric-deep-learning-1faf17dee008


How to graphically represent a RNN architecture implemented in Keras?

The first image is correct. The information will flow from left to right in each layer and from top to bottom in between layers.


In unsupervised learning, what is meant by "finding the probability of an image"?

When you say likelihood, you are invoking several other concepts like events, sample, parameters, model, probability density function (PDF), etc (it would be helpful if you learn more about these concepts). In essence, a likelihood function $l(x|\theta)$ is a PDF that quantifies how likely is that event $x$ happens out of a set of possible events, given the parameters $\theta$ that define your model.

In the specific case of images, the set of possible events is usually one of two options: 1) all the available images in a dataset, or 2) all the existing images. Usually you want to model the likelihood in the option 2), but only having access to a sample of all the possible images. In either case, the likelihood is just the probability that you select one image out of all the possible ones. If you consider only images of $1048\times 720$ pixels, the possible amount of images is $(256\times3)^{1048\times 720}$, where I am assuming that each pixel consists of 3 colors and each color can take 256 values. Since the amount of possible images is so so big, it is very common that the probability of selecting a specific one is very very small. This is a reason why you usually work with log-likelihood (the logarithm of the likelihood) instead of directly using likelihoods. For example, if all your images were equally probable, the likelihood would be in the order of $10^{-{10^7}}$, while the log-likelihood would be around $-10^7$.

To solve your paradox with the probability of images and pixels, consider that instead of pixels you have coins and instead of images you have sequences of coins. Let's say that you have a fair coin, so the probability of tails ($T$) after tossing the coin is 0.5. If you toss a second coin, the probability of having $T$ again is naturally 0.5 as well, but what is the probability that both results where $T$? It is the product (0.25) since the events are independent. Similarly, the probability of the other three sequences $TH$, $HT$ and $HH$ is just 0.25. You can see that since the probability needs to be shared between 4 sequences equally probable, they are less probable with respect to the probabilities of the sequences of length 1. If you toss the coin 3 times, then the probability of all these coins being tails is just $0.5^{3}$. Again, there are now 8 possible sequences, all sharing the same amount of probability. You can see what's going on. Since the amount of possible options becomes large, the probability of each possible sequence of coins becomes small. Clearly, you would never toss a coin 10 times and expect to obtain all $T$, right? Well, exactly the same happens in the case of the pictures.


How can I access the weights at each training iteration of an MLP with scikit-learn?

You can use the MLP function partial_fit to perform a single training iteration at a time. If you do retrieve the weights between calls to this function, you can see what they look like after each iteration.


What would be the best way to disable a rogue AI?

Metaphorically: make it so depressed it commits suicide.

As per my answer to this AI SE question, the idea is to feed it a sequence of inputs that will cause it to become (permanently) inactive.

The technical details of how this might be achieved (and they are somewhat technical) can be found in this paper.


Why should the baseline's prediction be near zero, according to the Integrated Gradients paper?

You are right that the baseline score is near zero only when there are a large number of label classes, i.e., when k is large. We should have qualified this line in the paper more carefully.
In this sense, formally, the technique explains the *difference in prediction between the input score and the baseline score, as is made clear elsewhere in the paper (see Remark 1 and Proposition 1 for instance.)


Given enough computational resources, do we currently have any algorithms which could achieve AI?

What 'infinite' means here could possibly be debated at some length, but that notwithstanding, here are two conflicting answers:

'Yes': Simulate all possible universes. Stop when you get to one containing a flavor of intelligence that passes whatever test you have in mind. Steven Wolfram has suggested something broadly along these lines. Problem: the state of computational testing for intelligence e.g. Winograd schema would then be the bottleneck. In the limit, testing for intelligence requires intelligence and creativity on behalf of the questioner.

'No': It may be that, even with infinite ability to simulate, there may be some missing aspect of our simulation that is necessary for intelligence. For example, AFAIK quantum gravity (for which we lack an adequate theory) is involved in Penrose's "Quantum Microtubules" theory of consciousness (*). What if that was needed, but we didn't know how to include it in the simulation?

The reason for talking in terms of such incredibly costly computations as 'simulate all possible universes' (or at least a brain-sized portion of them) is to deliberately generalize away from the specifics of any techniques currently in vogue (DL, neuromorphic systems etc). The point is that we could be missing something essential for intelligence from any of these models and (as far as we know from our current theories of physical reality) only empirical evidence to the contrary would tell us otherwise.

(*) No-one knows if consciousness is required for Strong AI, and physics can't distinguish a conscious entity from a Zombie.


What are examples of machine learning techniques inspired by neuroscience?

There is a category of neural networks that more closely attempt to mimic biological neural networks by incorporating also time (i.e. not all neurons fire at the same time). They are called spiking neural networks (SNNs) and their name comes from the fact that they use spiking neurons (i.e. neurons that fire discrete signals and affect other neurons at possibly different times).

SNNs are mainly used in neuroscience, and aren't commonly used in machine learning because they currently have some apparent limitations (e.g. non-differentiability, so gradient descent and back-propagation can't be applied, but GD and BP aren't really biologically realistic anyway, although some people already tried to apply GD to SNNs) and their performance isn't still as good as the performance of traditional deep learning models, which make them not so appealing to the deep learning community (which is currently mainly driven by performance and utility). Nevertheless, the performance gap between traditional neural networks and spiking neural networks is decreasing. See Deep Learning in Spiking Neural Networks (2019) by Amirhossein Tavanaei et al. for more details.

There are already commercial implementations of a hardware-accelerated SNNs (e.g. BrainChip provides this service). These hardware-accelerators are often called neuromorphic chips (or processors) and all computing based on SNNs or processors that attempt to implement biological neural networks is known as neuromorphic computing.

There's also the related area called reservoir computing, which studies neural networks (such as liquid-state machines or echo state machines) that make use of reservoirs (which are fixed during learning) to attempt e.g. to improve training efficiency. See An overview of reservoir computing: Theory, applications and implementations (2007) by Benjamin Schrauwen et al. for an overview.

Numenta (and, in particular, Jeff Hawkins, the founder of Numenta and author of an interesting book called On Intelligence) has also been studying neuroscience for a long time in order to develop models and theories of human intelligence. They call their new theory The Thousand Brains Theory of Intelligence, which is inspired by biological grid cells. This is also related and similar to capsule networks (often associated with Hinton).


Interpretation of feature selection based on the model

random forest's feature importances are not reliable and you should probably avoid them. Instead you can use permutation_importance: https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-py