Should I compute the gradients with respect to the flatten layer in a convolutional neural network?

I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.


What happened after the second AI winter?

My reading of AI development (somewhat simplified here) is that the availability of large data sets, increased computing power, and the introduction of new machine learning algorithms (which require large data sets and massive computing power) contributed to the resurgence of AI.

However, as witnessed on this site, there has been a paradigm shift within the field: while previous approaches to AI were largely symbolic, with a bit of connectionism thrown in, the current AI mainstream is purely based on statistical models and machine learning.

Massive (distributed) computing power on its own is not sufficient, as there was a bottleneck with domain modelling/knowledge acquisition in traditional symbolic approaches. New algorithms (basically further developments of neural networks) on their own would also not be sufficient without massive amounts of training data. Only the combination of all three elements enabled the AI resurgence.


The first step of optimisation is to measure where inside the implementation most time is spent -- you don't actually optimise the algorithm itself, but a specific implementation of it. This step should give you an overview of where you can make improvements. Speculative changes usually don't do much.

There are several aspects of A* which would be candidates. You are keeping a list of paths currently under investigation. If it is very long, processing might take more time. You could try and prune this list (though it might mean that you won't actually find the best path anymore). Or you could investigate efficient ways of storing it.

Each step you need to compute the 'cost' of each path. This could be cached to stop you from repeating calculations. The heuristic function could be slow -- again something to investigate.

Let me stress again that you need to measure the performance first. There is no point in randomly trying to make things faster; you could waste a lot of time for very little gain. And typically with optimisation, readability of the code suffers, and it becomes harder to maintain. You might find that other parts of your system take a lot longer than the actual path finding, so your time might better be spent on those.


How should I get started with artificial intelligence?

Start with Andrew Ng's introduction to Machine Learning course on Coursera.
There are not many prerequisites for that course, but you will learn how to make some useful things. And, more importantly, it will clearly show you which subjects you need to learn next.


Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation.
Mean square error for training data
Given some training data $\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the sum of squares, i.e. they are the minimisers given by
$$ \mathrm{arg\,min}_{a, b} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
This gives the same coefficients as minimising the mean square error
$$ \mathrm{MSE}\left((x_1, y_1), \dots, (x_n, y_n)\right) = \frac{1}{n} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
So, by definition, the coefficients $(a, b)$ minimise the MSE on the training data. Any regularisation will only increase the MSE on the training data.
Generalisation performance
The main point of regularisation is to prevent overfitting on the data and improve the generalisation performance (i.e. on the test set).
With an appropriate parameter for regularisation, you may obtain a smaller MSE on the test set. This depends on your dataset and the parameters you choose: strong regularisation may lead to underfitting, whereas weak regularisation might not make much difference to the coefficients that you fit.


Neural network to get input attributes using only the output value

Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?

It depends:

If you are happy to find any inputs, even non-realistic ones, that get your desired output, then you can use your trained network, with a minor modification. Freeze all the weights, and allow back-propagation to determine the gradient of the input (which should now be a variable to optimise, not source data). Start with a noise input, back-propagate the error to find gradient to make the input better at creating your desired output, then take a gradient step towards it in the input data. This is essentially how Deep Dream works. Like Deep Dream, you will not necessarily get realistic input values, but will get semi-random ones that cause your network to predict a specific class.
If you want the newly generated input to be a best guess at something from the original dataset, then you have to look at one of more advanced models:

Restricted Boltzmann Machines (RBMs).
Variational Autoencoders (VAEs)
Generative Adversarial Networks (GANs). These have become popular recently, and there are many variations on the basic idea, including VAEGANs that combine VAEs with GANs.

These network types are quite advanced, and can be tricky to understand and train successfully. You will want to spend some time researching each type.

To generalise terribly: A GAN will tend to generate realistic "noise" in the generated items, but at the expense of overall structure and cohesion (images tend to look distorted but with realistic textures). A VAE will tend to produce smooth, coherent inputs, but at the expense of lack of fine detail (VAE images tend to look smoothed and/or blurred).

If not sure what to try, probably GAN is a reasonable choice, since there are lots of tutorials available, and recent advances with image generation can look very impressive.


For each epoch, can I use only on a subset of the full training dataset to train the neural network?

Run the algorithm on a chunk of the data for a large number of iterations and then do the same with another chunk and so on?

  (Such approach will not produce the same result as mini-batch gradient descent, as I am not including all data in one iteration, but rather learning from some data and then proceeding to learn on more data, beginning with the updated weights may still converge to a reasonably trained network.)

This might work if each of the chunks of data individually is still large enough and sufficiently representative of the distribution of the complete population, but probably not the best way to go. In fact, I don't expect it to perform much better than simply using only the very last chunk, and training only on that one. This is because of the following reason. Suppose you first train for a while on chunk A, then for a long time on chunk B, then for a long time on chunk C, etc. While learning on chunk B, there is a significant risk that your model will "forget" everything it learned from chunk A. When learning on chunk C afterwards, it can also "forget" everything learned from chunk B again.

In pseudocode, the approach you proposed here looks as follows:

for each chunk:
    for large number of iterations:
        learn on chunk()

An easy way to improve on that would be to swap the loops around:

for large number of iterations:
    for each chunk:
        learn on chunk()

What I just described there is actually how I interpret "mini-batch gradient descent" though, the chunks would be minibatches (and the minibatches / chunks would be randomly re-selected from the complete population in every iteration of the outer loop, you wouldn't always use the same chunks). Note that this wouldn't be effective if your dataset is so large that it doesn't fit inside your RAM all at the same time, because then you'll have to deal with excessive I/O.

  Run the same algorithm (the same model also with only data varying) on different PC's (each PC using a chunk of the data) and then see the performance on a test set and take the final decision as a weighted average of all the different models' outputs with the weight being high for the model which did the best on test sets?

Yes, this can definitely be effective. This kind of idea (training different models on different subsets of data) is generally referred to as "ensemble" methods. You can even vary the models you use (e.g., have an ensemble with some Random Forests, some SVMs, some Neural Networks, etc.).


What are the necessary components to make AI agent self-programming-capable?

At the highest level, all it needs is for the various systems already discussed to incorporate code objects. If it can interpret its source code / model architecture from the formatted text objects underpinning them, can 'understand' them in terms of having a useful ML model, and alter the code with its reaction, then it can self-program.

That is, the basic loop behind a recursively improving intelligence is simple. It examines itself, writes a new version, and then that new version examines itself and writes a new version, and so on.

The difficult component comes at lower levels. We don't need to invent a new concept like 'sensor,' what we need to do is build very, very sophisticated sensors that are equal to the task of understanding code well enough to detect and write improvements.


Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

shouldn't the expected return be calculated for some faraway time in the future (+) instead of current time $t$?

This is partly a notation issue, but $G_t$ is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g_t$* until after $t+n$. However, you can predict the expected value $\mathbb{E}[G_t]$ provided the environment and policy remain consistent.

Your second equation is pretty similar to the first one, which is why I say it is partly a notation issue. However, the point of calculating the forward-looking expectation for the return is to allow for assessment at time $t$. This is to either predict likely outcome for future rewards knowing the system is in state $s_t$, or in control scenarios to choose an action $a_t$.

In addition, in both prediction and control scenarios, the past (from time steps $0$ to $t-1$) has already happened. Measurements of how well a system did in terms of gathering reward, looking backwards, might be useful metrics for e.g. "how good is this agent?". However, they are not in general a guide to the future. In many sparse environments (e.g. a board game scoring +1 for a win), this data is basically useless for predicting the future and all you want to know is summarised by the current state and the acting policy.

Regardless of when you get to finally calculate a return, the start time step - the part of the trajectory that the return is calculated from, is a key parameter. The end time step is a practical concern for implementation, but can often be considered to be at infinity for theoretical purposes (i.e. we are interested in measuring or optimising all future rewards). So if you are only going to display/use one parameter in the notation, the start time $t$ is the one to use.

There are variants of notation, to show how the return is calculated, where the calculation horizon is made explicit, e.g. $G_{t:t+n}$ for a truncated return or $G_{t:t+1}$ when calculating a one step temporal difference target. All the ones I have seen still maintain the forward view that defines a value associated with a current timestep $t$, for the same reason as explained above - it is at timestep $t$ where this value is of most interest as a prediction.

In practice during training you often wait until $t+n$ before you know the correct value of $g_t$* to apply as a training value - which is then used to update value estimates for $\hat{v}(s_t)$ or $\hat{q}(s_t, a_t)$. It is possible to make partial updates before that ending time step using techniques such as eligibility traces.

* Using notation of uppercase $G$ for random variable, and lowercase $g$ for a measured value.


How to calculate multiobjective optimization cost for ordinary problems?

I eventually used the keep-efficient function from this answer to calculate the Pareto front and used the k-means function to calculate the centroid of the front. This gave me the approximate knee-point of the front, which is usually the optimal solution. One of the calculations was to maximise the distance moved in x direction (dx) vs. minimising the energy consumed (e), so since the x axis needed positive maximization and the y axis needed minimization, I inverted the y axis, since min(f(y)) = - max(-f(y)). This helped get the pareto front toward the top right side of the graph and both the x and y axes were maximization objectives. The optimal point calculated was the robot that had the best fitness.


DQN card game - how to represent the actions?

Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.

Why is this better?
You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.


Why isn't the loss of my neural network reduced after 2500 iterations?

You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.
See if the results has changed after adjusting the learning rate.


Can I train a DQN on the same dataset for multiple epochs?

I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.
Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.
Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?
The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:
References:
Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.
Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.


Is the distribution of state-action pairs from sample based planning accurate for small experience sets?

I am aware that as the experience set grows the Central Limit Theorem
  will come into play and the distribution of experience will more
  accurately represent the true environment's state-actions-rewards
  distribution

I believe here you mean the Law of Large Numbers which states that for a large enough sample ($n \rightarrow \infty$) the sample mean will converge to the true mean. The central limit theorem (CLT) states that if you take the sum/mean of a set of independent random variables then the distribution of this new RV will be approximately normal as $n \rightarrow \infty$.

  before this happens is sampled based planning still useful

As you mentioned, if you had only 5 full episodes of experience to choose from, then this likely would not represent true underlying distributions and so the approximations would not be good -- of course, this will depend on how complex your MDP is, if you had a trivially simple one then it may be enough to represent it well. As David Silver says in his lecture, then one of the disadvantages of planning with a model is that you introduce another set of uncertainty, mainly from when you approximate the properties of the model.


Which books or papers clearly explain the relation between Ising models and deep neural networks?

The following articles

Ising models for networks of real neurons (2006) by Gasper Tkacik et al.
Deep neural networks for direct, featureless learning through observation: The case of two-dimensional spin models (2018) by Kyle Mills et al.
Inverse Ising inference by combining Ornstein-Zernike theory with deep learning (2017) by Soma Turi, Alpha A. Lee et al.
Neural-network based general method for statistical mechanics on sparse systems (2019) by Feng Pan, Hai-Jun Zhou, Pan Zhang et al.

may help you understand the "relationship" between Ising models and DNN, assuming you know what the Ising model is and what a DNN is, the similarity should be fairly intuitive to you.

The Ising model is a sort of floating soup of ferromagnetic particles each generating their own small magnetic field either working against or with their neighbor. When many of the particles aline, they create an aligned field we refer to as a dipole moment in magnetism, while in a DNN we refer to the joined effort of a few entities working to cause a larger effect in another entity an 'activation function'. In a fully connected DNN, where the Euclidean distance weights the connections and the nodes are initialized with a certain magnetic polarity in relation to the axis of the magnetic field it generates, the network would be an almost exact representation of the reality of what the Ising model seeks to simplify.


A* is an informed search algorithm. A* informed because it is based on the use of a heuristic function, which estimates the distance of each node to the goal, that is, the heuristic function provides information about the distance from any node to the goal node. The heuristic function can e.g. be the Euclidean distance (in the case this can be defined).

More precisely, at each step, A* needs to select the next node to explore. It chooses the node $n$ with the smallest value of $$f(n) = g(n) + h(n),$$ where $g(n)$ is the actual distance from the starting node to node $n$ and $h(n)$ is the heuristic function that estimates the distance from node $n$ to the goal node.

For example, suppose that we want to find the shortest path from Paris to Madrid. If you are already in Madrid, then you can estimate that the distance from Madrid to Madrid is zero, so $h(\text{Madrid}) = 0$. However, if you're still in Paris, what is the estimate of the distance from Paris to Madrid? We can e.g. pick up a map of the world and choose $h(\text{Paris})$ to be the length of the straight segment that goes from Paris to Madrid. Similarly, we can do this for other intermediate cities. There are other ways of estimating this distance, but this is an understandable one, given that people are usually familiar with the Euclidean distance. (This example is a little bit misleading because the Earth is not flat and so the Euclidean geometry does not really apply, but, for simplicity, you can ignore this).

For info regarding the completeness and optimality of A*, have a look at https://ai.stackexchange.com/a/8907/2444.


Why can't pure KG embedding methods discover multi-hop relations paths?

To put this insert to context, we should take at least this much of text from the paper:

  One line of research focuses on making recommendations using knowledge graph embedding models, such as TransE [2] and node2vec [5]. These approaches align the knowledge graph in a regularized vector space and uncover the similarity between entities by calculating their representation distance [30]. However,pure KG embedding methods lack the ability to discover multi-hop relational paths.

In my understanding the pure KG embedding here refers to TransE and node2vec solutions. To learn more about those, we should read links [1] and [2]. From [1]:

  Usually, we use a triple (head, relation, tail) to represent a knowledge. Here, head and tail are entities. For example, (sky tree, location, Tokyo). We can use the one-hot vector to represent this knowledge.

Later on same source there is an explanation on the end of definition section of TransE solution:

  But this model only can take care with one-to-one relation, not suitable for one-to-many/many-to-one relation, for example, there is two knowledge, (skytree, location, tokyo) and (gundam, location, tokyo). After training, the 'sky tree' entity vector will be very close with 'gundam' entity vector. But they do not have such similarity in real.

On other hand, [3] tells:

  Knowledge graphs as described above represents a static snapshot of our knowledge. It does not reflect the process of it’s how the knowledge built up. In the real world, we learn by observing temporal patterns. While it’s possible to learn the similarity between nodes A and node B, it will be hard to see the similarity between node A and node C as it was 3 years ago.

Solution in paper states "--,each recommended item is associated with around 1.6 reasoning paths." which is supposedly impossible for TransE solution.

So, knowledge graph, purely, is a static snapshot that can identify by great accuracy one to one findings. Actually according what [2] tells how node2vec works, they can describe and combine also more information (node2vec combines different types of similarities at same time), but anyways I think the main point is actually in one word in the citation: discover!

The model suggested on paper adds Reinforcement Learning principles to the KG modeling, so to say, the pure KG embedding always tells one distance based statistical solution, but a RL based solution may learn more aspects behind the scenes as it learns by try and error more complex paths underlying in the behavior.

On the other hand, the paper says, when relating their solution to pure TransE:

  It can be regarded as a single-hop latent matching method, but the post-hoc explanations do not necessarily reflect the true reason of generating a recommendation. In contrast, our methods generate recommendations through an explicit path reasoning process over knowledge graphs, so that the explanations directly reflect how the decisions are generated, which makes the system transparent.

So, whether TransE and those could actually recommend things in the given environment, the recommendation reasoning paths may stay obscure.

Sources:

[1] https://towardsdatascience.com/summary-of-translate-model-for-knowledge-graph-embedding-29042be64273

[2] https://towardsdatascience.com/node2vec-embeddings-for-graph-data-32a866340fef

[3] https://towardsdatascience.com/extracting-knowledge-from-knowledge-graphs-e5521e4861a0


Commonsense knowledge is the collection of premises that everyone, in a certain context (hence common sense knowledge might be a function of the context), takes for granted. There would exist a lot of miscommunication between a human and an AI if the AI did not possess common sense knowledge. Therefore, commonsense knowledge is fundamental to human-AI interaction.
There are also premises that every human takes for granted independently of the country, culture, or, in general, context. For example, every human (almost since its birth) has a mechanism for reasoning about naive physics, such as space, time, and physical interactions. If an AI does not possess this knowledge, then it cannot perform the tasks that require this knowledge.
Any task that requires a machine to have common sense knowledge (of an average human) is believed to be AI-complete, that is, it requires human-level (or general) intelligence. See section D of AI-Complete, AI-Hard, or AI-Easy – Classification of Problems in AI (2012) by Roman V. Yampolskiy.
Of course, the problems that arise while humans communicate because of different assumptions or premises might also arise between humans and AIs (that possess commonsense knowledge).


What is the most common practice to apply batch normalization?

In the literature, it differs. You will see models do it after or before pooling only, and sometimes you see it after every single convolution.

Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) but lately has been tied to the optimization process (How Does Batch Normalization Help Optimization?).

This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given some initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: Fixup Initialization).

So I would recommend 3 things until it is more understood how to utilize it generally:

Play around, get frisky and experiment. Use what works best.
Use block featurizers that are known to work well like residual blocks. Proven in practice and will probably work for you too.
Do the research and investigate it more, if you find the answer, you'll be helping a lot of people :)


Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?

It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this GithubIssue for details.


A way to give more weight to particular data?

Yes, You just weight the entries of loss function like sum( Wi(li-t2)**2) for MSE. Most frameworks provides this ability. For example in keras fit method has a parameter - sample_weight: Optional Numpy array of weights for the training samples.


Is there a way to train an RL agent without any environment?

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple of reasons why an environment is also used whenever possible.

Eventually, your RL agent will be placed in an environment to take its own actions. This is why we train RL agents. I'm assuming that, per your question, learning does not happen during this phase.

Maybe your agent encounters a novel situation
Hopefully, the experience your agent learns from is extensive enough to include every possible state-action pair $(s,a)$ that your agent will ever encounter. If it isn't, your agent won't have learned about these situations, and it will always perform suboptimally in them. This lack of coverage over the state-action space could be caused by stochasticity or nonstationarity in the environment.

Maybe the teacher isn't perfect
If you don't allow your agent to learn from its own experience, it will only ever perform as well as the agent that collected the demonstration data. That's an upper bound on performance that we have no reason to set for ourselves.


How can we calculate the gradient of the Boltzmann policy over reward function?

The main point here is that you can write $Q(s, a|\theta) = R = \theta^\top \phi(s, a)$. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)

For simplicity, I'm setting $\beta=1$, but you can always put it in once you get the idea. Therefore, the expression for $\pi(s,a,\theta)$ (I'm including $\theta$ here to make the dependence of $\pi$ explicit) is now:
$$
\pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B}
$$

Also, it is standard to compute $\nabla_\theta\log \pi(s,a,\theta)$ instead of $\nabla_\theta \pi(s,a,\theta)$ (although you totally can), so I'll go ahead and do that. To differentiate $\log \pi(s,a,\theta)$ with respect to the parameters, $\theta$, let's do some calculus:
$$
\nabla_\theta \log \pi(s,a,\theta) = \nabla_\theta \log \frac{A}{B} = \nabla_\theta \log A - \nabla_\theta \log B \\
= \nabla_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla_\theta \left[ \log \sum_{a'} exp(\theta^\top \phi(s, a')) \right] \\
= \phi(s,a) - \frac{1}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \sum_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\
= \phi(s,a) - \sum_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\
= \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)
$$

Therefore, you can write $\nabla_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$


Is Russia contributing to open research concerning friendly AI?

One instance of interest to you could be the extremely popular OpenCV library.
https://en.wikipedia.org/wiki/OpenCV

Initiated by Intel in 1999. A majority of the contributors were optimization experts from Russia. It is BSD licensed. Development is now done by core team in Russia with corporate sponsors.

The about page of OpenCV currently states

  Along with well-established companies like Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, Toyota that employ the library, there are many startups such as Applied Minds, VideoSurf, and Zeitera, that make extensive use of OpenCV. OpenCV’s deployed uses span the range from stitching streetview images together, detecting intrusions in surveillance video in Israel, monitoring mine equipment in China, helping robots navigate and pick up objects at Willow Garage, detection of swimming pool drowning accidents in Europe, running interactive art in Spain and New York, checking runways for debris in Turkey, inspecting labels on products in factories around the world on to rapid face detection in Japan.

So, OpenCV has had machine learning, and now it has deep learning as well among its modules in its latest release, so I guess it qualifies for the friendly AI label.

Furthermore, there is a lot of technical research generated in Russia that is published in Russian. I've heard of interesting examples of very advanced aerospace research, collecting dust in Russian journals, that has 'inspired' research in the US. I am not aware of the original researchers were properly credited for their work, or indeed the extent of the veracity of these stories. Nevertheless, it would certainly be to everyone's benefit if language barriers did not impede collaboration between developers and researchers in Russia and the rest of the world.

I feel that the only real hurdle here is that someone has to go and create a portal/framework/institution that provides a platform for developers in Russia to share their work in AI and ease the language and any other non-technical issues that might be impeding this.


In a genetic algorithm, crossover (recombination) is the analogy to mating in the real world. For example, you have some genetic information from each parent. In the case of an optimization where you have vectors of features (design variables), you could represent it as vector 1 and 2. Imagine each vector has 10 values. You grab the first 5 from vector 1, and the second 5 from vector 2, and combine those into a new vector which is your "offspring". Note that there are many different ways to do recombination.

When you are deciding on parents to use for recombination, you want to keep some of them the same. Recombination is basically hoping that combining attributes will give you a more optimal set. However, sometimes, you might want to make some of the offspring be copies of the parents in the next generation. So, as this link explains, the crossover probability determines what percentage of offspring experience recombination. It is probabalistic because there is an element of chance involved in whether or not given offspring experience recombination.


How does a single hidden layer affect output?

When we add a single layer with a non-linear activation function, right after the application of the activation function, a new basis function is found (for that neuron), which is some combination of the weights and biases, which acts as a new way to view or analyze the feature sets.

With an increasingly deeper network, we keep finding representations, which are new basis vectors of the combination of the previous layer's weights and biases, that is, higher-level representations.

If they're error-free, they'll give better performances, but if small errors creep in earlier basis vectors, the error increases through depth.

A nice analogy is the Taylor series, where $1$, $x$, $x^2$, and so on, are the basis vectors for estimating the function in 1D.


setting up last layer in tensoflow for class type of label

Since you're using categorical cross-entropy loss, the last layer (output layer) should come with softmax activation instead of identity (as being blank in layers.Dense(4)).
model = tf.keras.Sequential([
    normalize,
    ...,
    layers.Dense(4, activation=tf.keras.activations.softmax)
])

And SparseCategoricalCrossentropy is different from CategoricalCrossentropy, for categorical cross-entropy, the shape of output and label should be the same, for example:
When using categorical cross-entropy (example labels in one-hot):
Output: [[.1,.0,.9,.0],...] <-- Prediction of 90% for the 3rd class, sum=100%
Label:  [[ 0, 0, 1, 0],...] <-- Third class holds 100% classification possibility

When using sparse categorical cross-entry (labels are class indices):
Output: [[.1,.0,.9,.0],...] <-- Prediction of 90% for the 3rd class, sum=100%
Label:  [[2]          ,...] <-- Third class index is 2


Are there tools to help clean a large dataset so that it only contains faces?

I don't know of a tool but you could write a simple script to detect faces and crop it. It's quite simple with the Haar cascade in openCV to detect faces and use inbuilt functions to crop your image based on the size of the detected face.

Hope that helps !


Looking for approach for monetary evaluation of AI potential in 2025

It would be irresponsible to try. Several in the past, dating back centuries led to various trends that some call winters. I don't agree they are winters. They are just people returning to reality. Creating intelligence in computers has been progressing forward steadily since Charles Babbage. It has accelerated due to advances in digital electronics, in discrete mathematics, and in brain imaging.

It is possible to predict, with some unavoidable uncertainty, the impact on the profitability of existing products or services five years into the future, provided the AI efforts in the corporation are planned at a high level first.

The main issue is that what we mean when we say the word intelligence is really a collection of abilities. Tesla died with no assets yet his designs are in our appliances and the grids that power them. A billionaire may not be able to draw a sketch of himself worth a penny, and Michelangelo may not have been able to design a way to transmit power to homes and businesses.

There is no one thing called intelligence. It is many things. That's why computers exceeded humans in arithmetic long ago, but there are no combinations of hardware and software that can even come close to writing a physics textbook or even passing an arbitrary high school physics final exam. There is much more not known about intelligence then there is known about it.

Monetary evaluation may be necessary as part of the solution, but the end goal is monetary projection. That means one must evaluate the state and trajectory of something AI experts don't yet understand well. We can guess accurately that, if the direction of an AI department or AI groups within multiple existing departments is well managed, gains can be made. We can evaluate the use of AI in the past within the same business type and use the trend we see to make projections along the same line based on how technology trends usually progress. We can also factor in the emergence of competitive businesses also progressing with their use of AI.

That purchasing, R&D, manufacturing, and other departments are interested in AI, have done some initial play with examples and maybe built out some proof of concept code but are unwilling to quantify future potential is not surprising. It is not any easy financial modelling task. To begin the task, some information must be collected and scrutinized to ensure it is an accurate starting assessment.

What are the kinds of products and services of the business?
What are the critical corporate functions that support those?
What are the key technological, market, and production characteristics of the business that places it in a group of like businesses?
What currently developed and known-to-be-effective AI technologies most likely produce profit enhancements for this business?
What past technology advancements are most like those current ones?
Quantitatively, what were the financial outcomes of the leveraging of those past technologies over time?
How is competitive leveraging of the same technologies by competitive businesses likely to draw business and divide profit margins?

Once this information is gathered and its accuracy is ensured, a financial model can be constructed and implemented in software and projections of various scenarios can be made. It is a development project in itself. Many businesses are bypassing this project and just assuming that the current AI craze must have a high ROI because large corporations have made the choice down that road. This question exhibits a more prudent course of action.

In addition to the departments listed, there are a few more to consider.

Marketing
Finance
Strategic planning (at the board and senior management level)
Accounts receivable


How GoogleNet actually deal with reducing overfitting?

The main reason of overfitting in any neural network is having too many unrestricted trainable degrees of freedom in the model. Methods similar to dropout reduce the number of neurons at each training run which effectively means having a smaller network. On the other hand in $l_1$  and $l_2$ regularization, a term added to the loss function which put a constraint on the total loss calculated at each run. So what we are trying to minimize with such regularizations is not just $L$, but $L +l_1*f(w)$ (for example).

What I understand from that paper is that the auxiliary outputs do both at the same time: the point is to build smaller version of the same inception network within itself, and use the losses obtained from those as a constraint on the final loss function. The full model described in the paper is essentially 3 separate models: the first one is a network with 3 inception modules, the second one with 6 and the final one has 9. When loss is calculated, results from the auxiliary outputs added to the total loss with a 0.3 weight. Let us write this as follows:

$L = 0.7*L_9+ 0.3*(L_6+L_3).  $

Here $L_3$ and $L_6$ represent the losses calculated at first and second outputs, and $L_9$ is the loss calculated at the final output of the network.

This is the function we wish to minimize during training. But when the evaluation is made, the auxiliary outputs are discarded, and just the final softmax layer is used. Not very dissimilar from the idea of using dropout during training but using the full model for predictions.


Approaches to an algorithm for crossing a road

Any AI algorithm depends on the environment, and available actuators and sensors. In our case, the environment is a road, street, etc. The primary actuator includes wheels (or legs) of the robot. Sensors include a camera, sonar system, etc.
A simple Model-based reflex algorithm can work in your case:

function MODEL-BASED-REFLEX-AGENT(percept) returns an action
    persistent: state, agents' current conception of the world state
                model, description of how hte next state depends on current state and action
                rules, a set of condition-action rules
                action, most recent action, initially none
    state <-- UPDATE-STATE(state, action, percept, model)
    rule <-- RULE-MATCH(state, rules)
    action <-- rule.ACTION
    return action

Most of the terms and functions are self-explanatory and I will try to explain the important points. This implementation helps the robot to keep track of the external environment by maintaining internal state that depends on the percept history. Updating the internal state requires how the environment works without our agent in it. For e.g., Cars stop at red light and start moving on the green signal. Another thing that is required is how the actions of our agent will affect the world. Since your robot is only trying to cross the road there are not many cases here. Simple things include, stopping motors will stop the bot and starting them will move it forward.
The algorithm above shows how the current percept combined with the old internal state result in generating the updated description of the world, based on agent's model of how the world works. Thus, agent's model of how the world works is the most important part. UPDATE-STATE is responsible for creating new internal state description. The actual implementation of this will depend on the environment and technology that is being used.
The above algorithm has been taken from the book, Artificial Intelligence: A Modern Approach, for agents working in the partially observable environment. Implementation of many functions also depends on how complex you want your robot to be. For e.g., we haven't considered the case of looking up while crossing the road. This is possible that something might be falling from the sky, highly improbable, but possible. The list like this never stops in real life environments.


Why is Monte Carlo used as the tree search algorithm for AlphaGo?

The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS

  Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate the value of each state in a search tree. As more simulations are executed, the search tree grows larger and the relevant values become more accurate. The policy used to select actions during search is also improved over time, by selecting children with higher values. Asymptotically, this policy converges to optimal play, and the evaluations converge to the optimal value function. The strongest current Go programs are based on MCTS, enhanced by policies that are trained to predict human expert moves. These policies are used to narrow the search to a beam of high-probability actions, and to sample actions during rollouts. This approach has achieved strong amateur play.


Can Machine Learning be applied to decipher the script of lost ancient languages?

I would guess no, because if the language is unknown (no data available on it), then we would not have training data with which the machine learning algorithm could learn from.

If it is related to some known language, then some statistical analysis can lead to a guess at decipherment (assuming certain similarities among the two languages).

If interested on general language decipherment, see the following where they decipher scripts using available information on the language of interest: http://www.aclweb.org/anthology/W99-0906. They utilize the Expectation Maximization algorithm.

I'm sure more google searching can lead to other examples that use machine learning algorithms, but they would most likely have some known information or a body of assumptions to make our problem scope easier.


That is one of the good example for research. Personally, I prefer to segment out all the desired outputs at once. Then, check the success rate. If you cannot hit the success rate that you desire, you can go for more specific solutions for the specific problem that you face.

However, in general, the localization, segmentation, recognition are implemented in same network and are obtained all-at-once.


Which neuron represents which part of the input?

The determination of what each neuron represents is dictated by the initial weights of the network. As you may know, the common practice is to initialize the weights randomly. This means that given the same input data, the network may switch what each neuron means. It may also function differently, taking longer or shorter times to train. However, if you set the random seed to some constant, you can make the network deterministic.


an epoch takes 3.5 days

First of all, use colab to iterate quickly, it offers unlimited 12 hours of free GPU.

  to retrain it on Imagenet.

That said, we could use complex models without being afraid of overfitting due to large training size.

  will continued training increases the accuracy of the training set at the expense of the validation set

In many cases that's the case.

  The idea was to mimic the massive feedback projections in the brain.

My suggestion is to read the third part of the deeplearning book which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.

  Part III is the most important for a researcher—someone who wants to understand
  the breadth of perspectives that have been brought to the field of deep
  learning, and push the field forward towards true artificial intelligence.


Does Q Learning learn from an opponent playing random moves?

It should be possible to train an agent using some variant of DQN to beat a random agent around 100% of the time within a few thousand games.

It may require one or two more advanced techniques to get the learning time down to a low number of thousands. However, if your agent is winning ~50% of games against a random agent, something has gone wrong, since that is the performance you would expect of another random agent. Even simple policies, such as always play in same column, will beat a random agent a significant fraction of the time.

First thing to consider is that there are too many states in Connect 4 to use tabular Q learning. You have to use some variant of DQN. As a grid-based board game where winning patterns can repeat, some form of convolutional neural network (CNN) for the Q function is probably a good start.

I think for a first step, you should double-check that you have implemented DQN correctly. Check the TD target formula is correct, and that you have implemented experience replay. Ideally you will also have a delayed-update target network for calculating the TD targets.

As a second step, try some variations of hyper-parameters. The learning rate, exploration rate, size of replay table, number of games to play before starting learning etc. A discount factor $\gamma$ slightly below 1 can help (despite this being an episodic problem) - it makes the agent forget more of the initial bias for early time steps.

  Or should this not matter – if the Q Agent is playing enough games, it doesn't matter how good/bad its opponent is?

Up to a point this is true. It is hard to learn against a perfect agent in Connect 4, because it always wins as player one, which means all policies are equally good and there is nothing to learn. Other than that, if there is a way to win, eventually a Q learning agent with exploration should find it.

Against a random agent, you should be seeing some improvement if your agent is correctly set up for the problem, after a few thousand games. As it happens I am currently training Connect 4 agents using variants of DQN for a Kaggle competition, and they consistently beat random agents with 100% measured success rate after 10,000 training games. I have added a few extras to my agents in order to achieve this - there are some discussions of approaches in the forums at https://www.kaggle.com/c/connectx


How to handle proper names or variable names in word2vec?

Word2vec works on the concept of  typical word co-occurrences. This means that it will work well only for words that occur frequently in the dataset. So proper nouns will not play any role in training the model. You can keep the proper nouns as they are or use only the words the occur more frequently than some threshold value based on the size of your dataset.

Once you use the value stored in variable x for something, and then change the stored value, it will not reflect anywhere unless you us use the variable x again somewhere in the program.

# Example
x = "something something"
print(x + "...")

# Result
something something ...

# Changing x
x = "new value"

# This new value of x will not reflect anywhere in the program
# Unless you use the variable x again.


Who was the first person to recognize the distinction between human-like general intelligence and domain-specific intelligence?

Many publications from the middle of the twentieth century prove the questioner's statement that it was a widely held belief during that period that AI would quickly become conscious, self-aware, and smart.

Great Success

Many tasks and forms of expertise once the exclusive domain of human intelligence, after the development of the Von Neumann general purpose computing architecture became, by the end of that century, more or less the exclusive domain of computers.  These are only a few examples.

Scientific and statistical computation
Drafting and manufacturing process automation (CAD and CAM)
Publishing and typesetting
Certain forms of algebraic and calculus reductions (Maxima and its derivatives)
Circuit analysis
Masterful board game playing
Profitable stock speculation
Pattern recognition (OCR, fingerprint, voice recognition, sorting, terrain)
Programming in predicate logic and recursive predicates
Strategy evaluation

Disappointments (thus far)

In contrast to this impressive array of successes, there is an equally long list of failed expectations.

Consumer available bipedal robots
Automated vacuum cleaning (major disappointment for this answer's author)
Autonomous mechanical factory workers
Automated mathematicians (creative hypothesis generation and proof/disproof to extend theory)
Natural language comprehension
Obedience to arbitrary commands
Human-like expression in conversation
Automated technical innovation
Computer morality
Human (or at least mammalian) emotional states
Asimov's three laws operating system
Adaptive strategy development in arbitrary and shifting set of domains

Domain and Domain-free Distinction

When did it become clear that devising programs that master games like chess resulted in software designs that only applied to games like the ones for which they were programmed?

Although the general public may have thought that a cybernetic chess master would also be smarter than people in other ways, those creating those programs were well aware of the distinction between developing software that exhibited excellence in chess play hard coded and developing software that exhibits the ability to learn chess play and develop excellence iteratively from novice.

The end goal had always been high powered general intelligence.  More short term achievable objectives were created to facilitate the demonstration of progress to investors.  It was the only way to maintain a continuous stream of research funding from the military.

The first milestone was to master a single game without machine learning.  Then research turned to the building of domain knowledge so that a class of solutions, adaptations, and forms of planning could be realized in real time during warfare.  As economic domination became more preferable to military domination during the third quarter of the twentieth century, the vision for AI scaled to embrace the domains of economics and natural resource management.

Consider this spectrum of automation maturity.

A program that enumerates current move sequence possibilities at each turn in the play of a chess game, eliminating probable bad moves at each projected move point, and selects the next move most likely to lead to a win
A program that does the above but also skews probability based on pattern recognition of known winning chess strategies
A program that is designed to be a run time optimized rules engine that centralizes and abstracts the redundant operations of the play of an arbitrary game and isolates and aggregates the representation of chess rules, chess strategies, and chess patterns and anti-patterns
A program that, given a set of rules of a game, can generate a next move based on any game state, remembers success and failure results and the sequences that led to those outcomes, and has the ability to assess the probable loss or gain of individual moves and the game patterns in space and time around them based on history, and then leverages these abilities to learn an arbitrary game, reaching the masterful level of play of chess through learning
A program that learns how to learn games such that, after learning several games, it can learn chess faster than an intellectually gifted human can

The first is easy.  The last is extremely challenging.

When the distinctions between these phases of automation maturity became apparent and how clear people became of those distinctions in which research groups is a complex probabilistic function.

Key Contributors

Who was the first person to recognize the distinction between human-like general intelligence and domain specific intelligence?

Norbert Wiener was likely the first to deeply comprehend the distinction between electronic control of relays (investigated theoretically by Claude Shannon) and closed loop control.  In his book, Cybernetics, a primarily mathematical work, he precisely established the foundation for self-correcting and adaptive systems.  John von Neumann had a comprehension of the distinction between programming good game play and the human ability to learn good game play and published much on the topic.

It was Arthur Lee Samuel who actually wrote the first impressive demonstration of the distinction between game playing software and machine learning.  It was he who bridged Wiener's work with the contemporary digital computer and first coined the term Machine Learning.

Distorted Restatements of Authentic Research and Innovation

The categories artificial narrow intelligence (ANI), artificial general intelligence (AGI), and artificial super intelligence (ASI), proposed in The AI Revolution: The Road to Superintelligence by blogger Tim Urban (Huffington Post, THE BLOG, posted 2/10/2015, updated 4/12/2015), is referenced in AI Stack Exchange in multiple places, but the distinctions between these categories are not precisely defined and the ideas contained therein are neither peer review nor validated by other research or statistics.

The work is no less conjecture than mediocre science fiction — entertaining enough to gain some popularity but not rational conclusions drawn from either repeatable experiments or randomized studies.  The trend graphs provided in the article are of invented shape, not graphical representations of actual data.

Some of the material may later be found to have some truth in it, as in the case for many lay interpretations of scientific research or the futuristic thoughts of science fiction authors.  However, much of the material leads to misconception and false assertions.


Do not understand how genetic algorithms are used in stereo matching.

The first paper referenced in the question is Stereo matching using genetic algorithm with adaptive chromosomes, Kyu-Phil Han, Kun-Woen Song, Eui-Yoon Chung, Seok-Je Cho, Yeong-HoHac, 2000. It summarizes an approach this way.

An individual is a disparity set,
A chromosome has a 2D structure for handling image signals efficiently, and
A fitness function is composed of certain constraints which are commonly used in stereo matching.

The paper also states, "Genetic algorithms are efficient search methods based on principles of population genetics, i.e. mating, chromosome crossover, gene mutation, and natural selection." The extension to genetic algorithm purity is contained in this statement in the paper.

  To improve the convergence, an informed generation, that a higher selected possibility is assigned to the chromosome including a smaller intensity difference, is adopted. The informed generation uses several random generations plus a selection. In a random generation, only a value is randomly generated. Yet, the informed generation of the proposed algorithm selects the gene which has the minimum intensity difference among the randomly generated genes.

The question inquires along a few lines.

  Does it mean that an individual is a disparity map with random numbers?

The goal is to produce order from chaos through successive mutations, combinations, and tests, arriving at genetic code that represents distances of objects that correspond to a left region of pixels and a right region of pixels. Those two regions are identified through alignment of brightness categories, with some permissible sprinkling of outliers.

An individual in this context is a simulation of a sample from a population with a particular aggregation of genetic codes. The codes indicate disparity between matching image features in the left and right images, measured in horizontal pixels.

  [Don't] understand how this works and even WHY you should use genetic algorithm on an algorithm that at its simplest form uses 5 for loops.

The five loops referenced in the question, in the source at github.com/davechristian/Simple-SSD-Stereo, are as follows.

for y in range(kernel_half, h - kernel_half):
    for x in range(kernel_half, w - kernel_half):
        for offset in range(max_offset):
            for v in range(-kernel_half, kernel_half):
                for u in range(-kernel_half, kernel_half):

This exhaustive search has gross shortcomings in relation to the genetic algorithm with informed generation.

The simulation of meiosis in genetic algorithms is essentially a search for hybrids where the best genetic sequences of two parents are preserved in the offspring.
Genetic algorithms are easy to scale across computer clusters without developing specialized hardware acceleration.
Components of failed compositions are not eliminated in an exhaustive search, so there is considerable redundancy in trials.

The second paper referenced in the question is Genetic Stereo Matching Algorithm with Fuzzy Fitness, Haythem Ghazouani, 2014. It proposes, "To get around [convergence speed] limitations, we propose a new encoding for individuals, more compact than binary encoding and requiring much less space. This approach and the use of three lightness classes is shared with the methodology of the first paper.

This second paper introduces fuzzy matching and the use of the Sobel gradient norms to penalize (dismiss) pixels which project onto uniform regions, [and are therefore] less significant pixels. This paper also provides some comparison results with Han's, Dong's, and Nguyen's algorithms as define in the second paper's bibliography.


AI that maximizes the storage of rectangular parallelepipeds in a bigger parallelepiped

A simple approach that gives a good baseline for such problems is simulated annealing. The idea is that you do something random. If it improves things, then it is good. If it makes things worse, you still take it with some probability $p$, where $p$ shrinks over time.

The more bad solutions you can rule out beforehand / the smarter you can encode your problem, the better solutions simulated annealing will give.


How to predict Q-values based on stack of frames

You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$ and the output would be 6 Q-values for frame $F_k$.


What are the main differences between a deep Boltzmann machine and a deep belief network?

The graph that represents a deep Boltzmann machine can be any weighted undirected graph.

However, the graph that represents a deep Belief network must be a connection of graphs that represent restricted Boltzmann machines. Those graphs are bipartite, so there are two groups of vertices in those graphs so that every edge connects two vertices from different groups. Those groups are usually the visible and hidden components of the machine.

Learning is hard and impractical in a general deep Boltzmann machine, but easier and practical in a restricted Boltzmann machine, and hence in a  deep Belief network, which is a connection of some of these machines.


What is the difference between learning without forgetting and transfer learning?

Learning without Forgetting (LwF) is an incremental learning (sometimes also called continual or lifelong learning) technique for neural networks, which is a machine learning technique that attempts to avoid catastrophic forgetting. There are several incremental learning approaches. LwF is an incremental learning approach based on the concept of regularization. In section 3.2 of the paper Continual lifelong learning with neural networks: A review (2019), by Parisi et al., other regularisation-based continual learning techniques are described.

LwF could be seen as a combination of distillation networks and fine-tuning, which refers to the re-training with a low learning rate (which is a very rudimentary technique to avoid catastrophically forgetting the previously learned knowledge) an already trained model $\mathcal{M}$ with new and (usually) more specific dataset, $\mathcal{D}_{\text{new}}$, with respect to the dataset, $\mathcal{D}_{\text{old}}$, with which you originally trained the given model $\mathcal{M}$.

LwF, as opposed to other continual learning techniques, only uses the new data, so it assumes that past data (used to pre-train the network) is unavailable. The paper Learning without Forgetting goes into the details of the technique and it also describes the concepts of feature extraction, fine tuning and multitask learning, which are related to incremental learning techniques.

What is the difference between LwF and transfer learning? LwF is a combination of distillation networks and fine-tuning, which is a transfer learning technique, which is a special case of incremental learning, where the old and new tasks are different, while, in general, in incremental learning, the old and new tasks can also be the same (which is called domain adaptation).


How to train a neural network with a data set that in which the target is a mix of 0-1 label and numeric real value label?

Your question are missing some details and i will assume some scenarios.

If you have a classification problem: you can try group the values in intervals that make sense (you should analyze and decide for this setup), if its possible. For example: 0.000-0.250 (0), 0.251-0.500 (1), 0.501-0.750 (2) and so on. Note that neural networks are sensible for distance between values (1 is closer to 0 than 2, so 1 is more similar to 0 than 2 and so on). If that is not your case, you should binarize the values in One Hot Encode manner.
If you have a regression problem, you should be ok without anything else. You can try normalize your outputs and observe the results, but generally it's not necessary for regression problems.
Be sure if your dataset are free of outliers and noisy data as much as possible.
It's important choose activations functions that are adequate for the range of values in your attributes and output. This can depend on how do you treat and setup your dataset, the range of values, normalization etc.

Update after more details in question

Your neural network should have 3 neurons in the output layer, with linear activation. As said before, normalization usually is not necessary in regression problems, but if your values are too diferent (like the range in resource 1 and resource 2) maybe some kind of adjustment (normalization, standardization etc) can be helpful. But you need try and see the results.


How do you manage negative rewards in policy gradients?

You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.

You also seem to be missing $-$ sign from the loss. The objective function for the vanilla policy gradient algorithm (REINFORCE) which we want to maximize is
\begin{equation}
J = \sum_a \pi(a|s) q_{\pi}(s, a)
\end{equation}
It can be shown that the gradient sample for this policy gradient method is
\begin{equation}
\nabla J = G_t \nabla \log (\pi(A_t|S_t))
\end{equation}
so in TensorFlow you should define your loss as
\begin{equation}
J = - G_t \pi(A_t|S_t)
\end{equation}
We need the $-$ because in TensorFlow you use minimizers, but we want to maximize this function so minimizing this loss is same as maximizing the objective function. In conclusion, the code similar to what you wrote, should be
-return * cross_entropy(action_pdf, selected_action_one_hot)

EDIT

As pointed out in the comment we don't actually need $-$ because it is already included in cross_entropy function.


What layers to use in a Neural Network for card game

The game state consists of the location of all the hidden cards, so you probably need a softmax layer, 52*n, where n is the number of locations.

I'm not very sure that a NN is a good match.


Standard deviation of the total input to a neuron

If n different edges with associated weights $w_1, w_2, . . . , w_n$ point to this node, then after selecting weights
  with uniform probability from the interval $[−α, α]$, the expected total input
  to the node is $$\sum_iw_ix_i$$.
  By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution.

Note that the input to a node  is $$w_1x_1 + w_2x_2 + ... + w_nx_n $$

The variance of the total input to a node is:

$$\sigma^2 = \sum^n_{i=1}E[w_i^2]E[x_i^2] = n(E[w_1^2]E[x_1^2])
= n([\frac{(\alpha-(-\alpha))^2}{12}][\frac{1}{3}])=n\alpha^2\frac{1}{9}$$

since inputs and weights are uncorrelated.

I have taken out the summation and replaced it with $n$ times the variance of an input and a weight -- this can be done because both are independent and identically distributed in their own right and are independent of each other too. Variance of $w_1$ is the same as variance of $w_n$.

  Am I missing some implicance of the law of large numbers use for the input to the node?

Yes.

Implication of Law of Large Numbers:

Note that the random variable $E(w_ix_i)$ in  $$E(\sum^n_i(w_ix_i)^2)$$ is written as product of the random variables $E(w_i)$ and $E(x_i)$ $$\sum^n_{i=1}E[w_i^2]E[x_i^2]$$. This can only be written when the two random variables are independent. And the switching of expectation and summation is due to the linearity of expectation property.
How do we get independence here? LLN comes to the rescue.

That comes from the fact that -- according to the law of large numbers -- the random variable (total input) follows a normal distribution which implies the two random variables i.e. the weights and the inputs also follow normal distribution -- and it is also given that the two random variables are uncorrelated and thus they are independent.


When are compiled vs. interpreted languages more optimal in AI?

Interpreted languages allow for a faster development cycle, as they don't require time for compilation, and fragments can often be run without having a complete program. They often also have fewer constraints for variable declaration or typing. That means they can be used to quickly scope out a problem and try different solutions.

The drawback is the slower execution speed. But during development this is not a big factor; it only becomes important in a production environment. So one option would be to use an interpreted language during the R&D phase, and then re-implement the algorithm in a compiled language for performance improvements.

Since ML and NNs have become more prevalent in AI, numerical computing has become more important. This is an area where interpreted languages traditionally don't perform too well, so one would use a (compiled) library for, say neural networks, or genetic algorithms, and use 'glue code' to integrate this into a bigger system. The glue code would transform/prepare data and convert this between different formats required by libraries. This is often done in interpreted scripting languages, as they might have to be changed more frequently and are not performance critical.

Apart from development, the type of computation is also key: as mentioned, numerical computing generally works better with compiled code, but interpreted languages often have advantages in symbolic programming. This is why Lisp and Prolog have become popular AI languages, as opposed to Fortran or C.

In an ideal world you would use an interpreted language for development, and then compile this once you're done. However, due to the way these languages work, compilation is often non-trivial.


How does a batch normalization layer work?

Definition and Explaination

For how Batch Normalization works exactly, I'll suggest you to read the following papers:

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
How Does Batch Normalization Help Optimization?

The recent interpretation on How BN works is that it can reduce the high-order effect as mentioned in Ian Goodfellow's lecture. So it's not really about reducing the internal covariate shift.

Intuition

For how it works intuitively, you can think that we want to normalize the intermediate outputs (zero mean and unit variance) if the normalization won't remove too much useful information.

However, normalization may not be suitable for all intermediate outputs. So $\beta$ and $\gamma$ is introduced to provide additional flexibility, if normalization removes too much useful information then $\beta$ and $\gamma$ will learn to become the original mean and variance, making the BN layer an identity transformation, as if it doesn't exist.

In practice, $\beta$ and $\gamma$ won't become the original mean and variance, since all intermediate outputs can be normalized in some certain way without losing too much useful information. So you can think of it to be a customized normalization for each BN layer.

tl;dr

BN layer normalize the intermediate outputs in default, however, if the neural network find out that these intermediate outputs should not be normalized, then the neural network undos or provide more flexibility to the normalization.


How does the Kullback-Leibler divergence give "knowledge gained"?

You can know it better, if you know the concept of entropy:

Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event ${\displaystyle E}$  is an increasing function of the reciprocal of the ${\displaystyle p(E)}$ of the event, precisely ${\displaystyle I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$. Shannon defined the entropy Η of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ as:
$${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$
Here ${\displaystyle \operatorname {E} }$ is the expected value operator, and $I$ is the information content of ${\displaystyle I(X)}$ is itself a random variable. The entropy can explicitly be written as
$${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$
where $b$ is the base of the logarithm used.

Now, the KL divergence, try to find the cross entropy of two probability distributions.
So, what is the cross entropy:

The cross entropy between two probability distributions ${\displaystyle p}$ and ${\displaystyle q}$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution ${\displaystyle q}$, rather than the true distribution  ${\displaystyle p}$.
The cross entropy for the distributions ${\displaystyle p}$ and  ${\displaystyle q}$ over a given set is defined as follows:
$${\displaystyle H(p,q)=\operatorname {E} _{p}[-\log q]}.$$
The definition may be formulated using the Kullback–Leibler divergence${\displaystyle D_{\mathrm {KL} }(p\|q)}$ of  ${\displaystyle q}$ from ${\displaystyle p}$ (also known as the relative entropy of ${\displaystyle p}$ with respect to ${\displaystyle q}$).
$${\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$
where ${\displaystyle H(p)}$ is the entropy of ${\displaystyle p}$.

Now, as we want to approximate a distribution function $p$ with other distribution function $q$, we want to minimize the cross entropy between these two. The first part $H(p)$ could not be changed as we want to find a distribution $q$ and $p$ is given. Hence, we need to minimize $KL$ divergence of these two, to minimize the cross entropy and better approximation for the $p$ distribution.


That is a wonderfully fundamental question.

  Learning is the use of a system to change another system so that, instead of doing what it did before, which may have been nothing, it does something else.

In the human brain, the system is the way that genetic expression caused the directed mutability of that brain so that human intentions and responses to external stimuli would alter. The direction of alteration is based on incentives and deterrents. In classes of children, there are things a teacher may do or a culture in the school that motivate learning. Misbehavior is deterred through other mechanisms. In that case, the first system is the educational process and the second system is the brain's ability to use what it was taught.

Even without classes, children learn things that create comfort, so comfort is the internal incentive. It is wired into us as a kind of teacher to dislike groin moisture, so we learn to use bathroom receptacles. It is wired into us to like praise from parents and teachers, so we tend to perform to get it. Things like that are complex losses and rewards and primary systems to incentivize learning.

Learning in artificial networks is not nearly as complex as in human brains. In some cases the artificial results are better. In other cases the abilities of the human brain cannot be approached by artificial networks yet.

The functioning of an artificial network often begins arbitrary and completely useless, but it's a parametric function, meaning the function can be changed by changing numbers called parameters. Each time the function is used to process an example drawn from a data set, the result is evaluated and the evaluation is used to modify the parameters. Care is taken to not over-modify the parameters, otherwise confusion can occur. Mathematicians call this type of system confusion chaos.

Repeating this carefully directed and moderated process ideally leads to something called convergence. The learning system is set up so that the result of convergence is a set of parameters that minimize losses, maximize gains, or both.

Sometimes initial learning is not enough and there are other related things to do later.

Adjusting learned behavior to adapt to new conditions or information
Unlearning things that no longer produce benefit so that new learning can replace them
Relearning things that had faded from memory
Developing greater confidence in what was learned so that unlearning requires greater impetus

There are additional technical terms for the above concepts, more details that can be grasped, many categories of learning system types, varieties within those, and the mathematics that was used as a foundation to construct all this and make it work. Because the question was fundamental, those details and technicalities were omitted.


How do I compute log-likelihood for training set in supervised learning?

The log-likelihood function for the training set (in general, not for deep learning in particular) will depend on your choice of loss function.

I'm guessing you're using something like a quadratic loss function for a binary classification problem, since this is a common approach. In that case, the log-likelihood is the sum of logs of the squared differences between the target label and the softmax value produced by your model. If you want to compute the log likelihood under a particular set of parameters (say, those that minimize validation error), then you just use those parameters in the model when generating softmax values.


Oposite type of predictions for unbalanced dataset

A random forest is a collection of classification trees. If more than 50% of these trees predict class A (and not class B), the random forest will predict class A.

What you can do is lower the percentage needed to classify it as class A (in your case, patient has the virus). This way, you can tell your random forest to predict class A if only 20% (or 10%, or 5%, ...) of the decision trees actually predicts class A.

I don't know what code you are using for the random forest algoritm, but in most you should be able to ask the % of certainty the random forest has of each class.


In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.

A standard Mean Squared Error function cannot be negative. The lowest possible value is $0$, when there is no output error from any example input.

  How can our cost function which is mean squared error have a value under 0?

It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for any cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.

Many loss or cost functions are designed with an absolute minimum of $0$ possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.


What are the disadvantages to using a distance metric in character recognition prediction

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks (CNNs). Particularly, it focuses on the cons of the former option.
There are a few potential problems associated with using distance based OCR systems.
First of all, this approach requires an appropriate distance metric to deliver good results. Different types of distance metrics/functions are sensitive to different features in the input images. For example, some functions might penalize absolute differences, while others penalize squared differences, where the latter punishes differences of magnitude $> 1$ stronger than the former, while the former penalizes differences of (absolute) magnitude $< 1$ stronger than the latter. Which type of distance metric works best for a given problem has commonly to be determined empirically.
Also, distance based OCR systems may possibly be more sensitive to, for example, different levels in illumination than CNNs. While both sorts of classifiers could profit from data augmentation (i.e. adding variations of the existing training data to the dataset), OCR based on CNNs has the benefit that CNN training procedures produce classifiers that commonly generalize well to slight variations in the incoming data, while some slight novel tilt (or again variation in illumination) may break the distance based classification procedure, but may have not too detrimental effects on CNN based classifiers due to their rather strong generalization capabilities.
Of course, one can also try to increase the robustness of distance based OCR systems, but this is commonly associated with developing exhaustive preprocessing pipelines to standardize the appearance of incoming character images.
Thus, in terms of system design, it is often easier to set up a CNN-based neural network architecture and train it (possibly drawing upon regularization to boost generalization of the trained system even further) than trying to design complicated preprocessing strategies to shift, rotate, and normalize character images exhaustively to boost performance of a distance based OCR system.
To sum up, after all, there are a lot of potential issues related to using plain distance (metric/function) based OCR systems, most of which, however, can strongly be alleviated when drawing upon clever (pre-)processing pipelines to standardize input images before performing distance based classification.
From all those issues mentioned above, the strongest disadvantage of this technique, however, (which cannot be alleviated so easily) is that it doesn't generalize too easily to novel input data, where CNNs might perform better with appropriate training data (+ augmentation and regularization in general).
You also mentioned the aspect of computational cost associated with distance based classification. First of all, I am not an expert in computational complexity/cost related questions. However, most image related computations can efficiently be executed on graphics cards. So, subtracting images pixel-wise, for example,is not very costly given it can be executed on a graphics card. But of course, when using distance based OCR, the computational cost associated with computing distances and comparing distance values scales with the reference dataset size.
Using CNNs, the time needed to perform a classification is always the same irrespective of the training dataset size.


Can an Markov decision process be dependent on the past?

It's not totally clear from your description, but it sounds like you may be onto something like an Additive Markov Chain.


Image Recognition and Orientation Detection

This is a quite trivial problem and using a NN would be overkill for this. The symbol you used (triangle) is problematic, because it looks exactly the same in 3 different positions (rotated 0°, 120° and 240°).

The easiest implementation would be using 2 distinguishable points with a fixed distance. To distinguish them you could color code them or give them different sizes. Those 2 points should be easy to identify in your picture using a pattern matching algorithm. Once you have the 2 points you calculate the vector between them and can then calculate the orientation of the vector, which basically tells you the orientation of the whole object your symbol is put on.

Edit - Answering the first comment:

The basic idea is the same, no matter what symbol you use. In case of the "L", your 2 points would be the end of both lines. You can easily find the symbol, calculate the vector between both points and use it to calculate the orientation using this vector. This approach can be used for every symbol that doesn't have radial symmetry. You can always define 2 points in this symbol and apply the method described.

The reason why a NN is overkill is the simplicity of the problem. NNs are mostly used for problems that are easy to solve for humans but very hard to implement with a simple algorithm. Tasks like classifying images, identifying hand written or distorted letters and so on. But your problem is easily solvable with simple algorithms, therefore the complexity of a NN is not necessary.

This doesn't mean that your task cannot be implemented with an NN. Actually it might be quite an interesting project to learn more about NNs. I would say your best bet is implementing it as a CNN, since this type of NN is good at detecting features in images like the direction of your symbol. I can recommend this course from Stanford University, if you want to learn more about CNNs.

For a real world application, there is no need to take the complex route using NNs. My suggested algorithm can be implemented with a few dozen lines of code and should be very robust.


I agree that this notation is unclear. I would interpret it as follows:

Given that the expression is supposed to denote the average norm $|p_i|$ is likely the cardinality of the set $\{p_i\}$.

In that case the expression would just be the sum over all norms divided by the number of norms, resulting in the average norm. The authors likely use this notation because they didn't want to introduce the number $n$ of $p_{i < n}$. $|\{p_i\}|$ would be clearer, but maybe uglier to typeset.


Is Mask R-CNN suited to solve a multi-class classification problem where the classes are related?

Mask RCNN can be a very heavy function for a simple class classification. It is designed to handle multiple object in a single image. So I would suggest you could use much simpler models like VGGnet or Resnet which are backbones of the MaskRCNN.
The biggest hurdle you might face is the dataset. If you are trying to capture even small difference between knives and classify them you will need atleast 2000 images per knife type. which might be difficult to get. You may have to do a data augmentation.


Generalising performance of Q-learning agent through self-play in a two-player game (MCTS?)

To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent,

This seems quite a reasonable approach on the surface, but possibly the agents will still lose generalisation if the solutions in each generation are too similar. It also looks like from your experiment that learning progress is too slow.

One simple thing you could do is progress through the generations faster. You don't need to train until agents win 90% of games before upping the generation number. Yuo could set the target as low as 60% or even 55%.

For generalisation, it may also help to train against a mix of previous generations. E.g. if you use ten opponents, have five from previous generation, two from each of two iterations before that, and one even older one.

Although the setup you have created plays an agent you are training against another agent that you have created, it is not quite self-play. In self-play, an agent plays against itself, and learns as both players simultaneously. This requires a single neural network function that can switch its evaluation to score for each player - you can either make it learn to take the current player into account and make the change in viewpoint itself, or in zero-sum games (which Connect 4 is one) it can be more efficient to have it evaluate the end result for player 1 and simply take the negative of that as the score for player 2. This is also equivalent to using $\text{max}_a$ and $\text{argmax}_a$ for player 1's action choices and  $\text{min}_a$ and $\text{argmin}_a$ for player 2's action choices - applying the concept of minimax to Q learning.

You can take minimax further to improve your algorithm's learning rate and performance during play. Essentially what Q learning and self-play does is learn a heuristic for each state (or state/action pair) that can guide search. You can add search algorithms to your training and play in multiple ways. One simple approach during training is to perform some n-step look ahead using negamax with alpha-beta pruning (an efficient variant of minimax in zero-sum games), and if it finds the end of the game:

when training, use the result (win/draw/lose) as your ground truth value instead of the normal Q-learning TD target.
when evaluating/playing vs human, prefer the action choice over anything the Q function returns. In practice, only bother with the Q function if look-ahead search cannot find a result.

In the last few months, Kaggle have been running a "Connect X" challenge (which is effectively only Connect 4 at the moment). The forums and example scripts (called "Kernels") are a good source of information for writing your own agents, and if you choose to compete, then the leaderboard should give you a sense for how well your agent is performing. The top agents are perfect players, as Connect 4 is a solved game. I am taking part in that competition, and have trained my agent using self-play Q-learning plus negamax search as above - it is not perfect, but is close enough that it can often beat a perfect playing opponent when playing as player 1. It was trained on around 100,000 games of self-play as I described above, plus extra training games versus previous agents.

  I know the most successful techniques (e.g. alpha zero) rely on MCTS, but I'm not sure how to integrate this with Q-learning? Neither how MCTS helps to solve the problem of generalisation?

MCTS is a variant of search algorithm, and could be combined with Q-learning similarly to negamax, although in Alpha Zero it is combined with something more like Actor-Critic. The combination would be similar - from each position in play use MCTS to look ahead, and instead of picking the direct action with the best Q value, pick the one with the best MCTS score. Unlike negamax, MCTS is stochastic, but you can still use its evaluations as ground truth for training.

MCTS does not solve generalisation issues for neural networks, but like negamax it will improve the performance of a game-playing agent by looking ahead. Its main advantage over negamax in board games is a capability to scale to large branching factors. MCTS does work well for Connect 4. Some of the best agents in the Kaggle competition are using MCTS. Howver, it is not necessary for creating a "superhuman" Connect 4 agent, Q-learning plus negamax can do just as well.


What is the purpose of the batch size in neural networks?

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details.

A more detailed explanation...

If you have a small enough number of samples, you can let the network see all of the samples before updating its weights; this is called Gradient Descent. The benefit from this is that you guarantee that the weights will be updated in the direction that reduces the training loss for the whole dataset. The downside is that it is computationally expensive and in most cases infeasible for deep neural nets.

What is done in practice is that the network sees only a batch of the training data, instead of the whole dataset, before updating its weights. However, this technique does not guarantee that the network updates its weights in a way that will reduce the dataset's training loss; instead it reduces the batch's training loss, which might not the same thing. This adds noise to the training process, which can in some cases be a good thing, but requires the network to take too many steps to converge (this isn't a problem since each step is much faster).

What you're saying is essentially training the network each time on a single sample. This is formally called Stochastic Gradient Descent, however the term is used more broadly to include any case where the network is trained on a subset of the whole training set. The problem with this approach is that it adds too much noise to the training process, causing it to require a lot more steps to actually converge.


How can blackjack be formulated as a Markov decision process?

Isn't the environment constantly changing in this game?

The current state of the agent and the environment is constantly changing as you play, but not necessarily the transition probabilities. For simplicity, you may assume that the transition probabilities do not change (e.g. if the dealer and the deck are the same every time you play).

  How would the transition probabilities be fixed in such an environment, when both you and the dealer draw cards?

The actions of the dealer will be incorporated into the transition probabilities of the environment. Whenever the player (or RL agent) takes an action, then it will receive a reward, according to the reward function of the environment (the rules of the game), and the agent and the environment will be moved to the next state, according to the transition probabilities, which do not have to change for stochasticity to occur in the environment. In fact, these transition probabilities already incorporate this stochasticity.

Also, if even the environment was changing, you could still model your problem as an MDP, but the MDP would change accordingly.

The example 5.1 of the book (that you mention) actually explains in detail how to formulate this game as a finite MDP.


What to do with duplicate Innovations in a genome?

Because this is more of an answer than a comment:

  In the add connection
  mutation, a single new connection gene with a random weight is added connecting
  two previously unconnected nodes.

So two nodes that have been mutated with a connection, can't remutate because the nodes should be unconnected.

  In the add node mutation, an existing connection is
  split and the new node placed where the old connection used to be. The old connection
  is disabled and two new connections are added to the genome.

The old connection is disabled, so the same mutation can't occur again on this connection. In practice, you can just remove the disabled connection, because it will be formed again with the add connection mutation.


How do you explain Hebbian Learning in an intuitive way?

Hebbian learning is trying to answer "How the strength of the synapse between 2 neurons evolve over period of time based on the activity of the 2 neurons involved.". You can call it learning if you think learning is just strengthening of synapses.
The connection between 2 neurons are called synapse. A synapse is the point where the axons of a neuron meets with the dendrites of another neuron.
The 2 neurons are connected to each other but they are also connected to other neurons as well. So it may happen that both the neurons gets activated by their other connected neurons at the same time.
The elements refer the 2 neurons. The equation is basically saying that the synapse strength between i and j neuron at time (n+1) depends on its strength at time n plus activations of the i and j neurons at time n.


Are there benchmarks for assessing the speed of the forward-pass of neural networks?

https://github.com/jcjohnson/cnn-benchmarks might be a good start. It mostly focuses on GPUs, but there is also one CPU (Dual Xeon E5-2630 v3).


What activation function does the human brain use?

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron.

It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential.

If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.

TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.

  Does the human brain use a specific activation function?

IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it).

But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.

There are various biological neuron models, but the Hodgkin-Huxley Model is the most notable.

Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)

But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.

  It (activation potential) sounds a lot like ReLU...

It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing.

Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.

This is a ReLU function, with the X-axis being input value and Y-axis being output value.

And this is the action potential with the X-axis being time, and Y being output value.