How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
$$
\sigma(\mathbf z)_{i} = \frac{e^{z_i}}{\sum_k e^{z_k}}
$$
I think, one of the most effective ways of not to get confused about all these matrices with different dimensions, is to simply get rid of the matrices and do all calculations componentwise. So, calculating the $ij$ component of the jacobian matrix, we have:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{e^{z_i}}{\sum_k e^{z_k}}\delta_{ij} - \frac{e^{z_i}e^{z_j}}{(\sum_k e^{z_k})^2}
$$
Consider a cost $C$, and we wish to calculate the derivative with respect to the weights of the last layer, that is, we are running the first step of backpropagation. Applying chain rule to the $ij$-component of the weight:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}}
$$
And, with one more chain rule:
$$
\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}} =
\sum_s\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now we can identify exactly where the jacobian matrix is:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{\partial a^{[L]}_{i}}{\partial z^{[L]}_j}
$$
Thus, the back-propagation step is just:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}J_{ks}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now, in component language, it is much simpler to identify where are all the matrix multiplications: The transpose of the gradient with respect to the cost function, multiplied by the jacobian matrix of the softmax activation (or whatever activation), and finally, the last derivative, which will evaluate to something depending on the activation of the previous layer.
=].


Why would neural networks be a particularly good framework for "embodied AI"?

To my mind the essential reason why neural networks and the brain are powerful is that they create a hierarchical model of data or of the world. If you ask why that makes them powerful, well, that's just the structure of the world. If you are stalked by a wolf, it's not like its upper jaw will attack you frontally, while his lower jaw will attack you from behind. If you want to respond to the threat with a feasible computational effort, you'll have to treat the wolf as one entity. Providing these kinds of entities or concepts from the raw bits and bytes of input is what a hierarchical representation does.

Now, this is quite intuitive for sensory information: lashes, iris, eyebrow make up an eye, eyes, nose and mouth make up a face and so on. What is less obvious, is the fact that motor control works exactly the same way! Only in reverse. If you want to lift your arm, you'll just lift it. But for your brain to actually realise this move, the high level command has to be broken down into precise signals for every muscle involved. And this is done by propagating the command down the hierarchy.

In the brain these two functions are strongly intertwined. You use constant sensory feedback to adapt your motor control and in many cases you'd be incapable of integrating your stream of sensory data into a coherent representation if you didn't have the additional information of what your body is doing to change that stream of data. Saccades are a good example for that.

Of course this doesn't mean that our cognitive functions are dependent on the processing of sensorimotor information. I would be surprised if a pure thinking machine wouldn't be possible. There is however a specific version of this "embodied intelligence hypothesis" that sounds plausible to me:

Creating high level cognitive concepts with unsupervised learning is a really difficult problem. Creating high level motor representation might be significantly easier. The reason is that there is more immediate useful feedback. I have been thinking about how to provide a scaffolding for the learning of a hierarchy of cognitive concepts and one thing I could imagine is that high level cognitive concepts basically hitch a ride with the motor concepts. Just think of what a pantomime can express with movement alone.


Why is cross-over a part of genetic algorithms?

Mutation is usually defined to be a global operator, i.e. iterated mutation is (eventually) capable of reaching every point in the vector space defined by the geneome. In that sense, mutation alone is certainly 'enough'.

Regarding the motivation for crossover - from Essentials of Metaheuristics, p42:

  Crossover was originally based on the premise that highly fit individuals often share certain traits, called building blocks, in common.
  For example, in the boolean individual 10110101, perhaps
  ***101*1 might be a building block

(where * means "either 0 or 1")

So the idea is that crossover works by spreading building blocks quickly throughout the population.

  Crossover methods also assume that there is some degree of linkage between genes on the chromosome: that is, settings for certain genes in groups are strongly correlated to fitness improvement. For example, genes A and B might contribute to fitness only when they’re both set to 1: if either is set to 0, then the fact that the other is set to 1 doesn’t do anything.

Also note that crossover is not a global operator. If the only operator is crossover then (also from p42):

  Eventually the population will converge, and often (unfortunately) prematurely
  converge, to copies of the same individual. At this stage there’s no escape: when an individual crosses over with itself, nothing new is generated.

For this reason, crossover is generally used together with some global mutation operator.


Not all search is planning (is A connected to B), but all planning is search (how do I get from this to that).

Here's an example in Prolog with a domain described in terms of actions, when they are possible, and what the result of the actions are. The description is of an uncomputed graph of un-calculated size where each node is a situation and each edge is an action. Then we have A* search algorithm that searches this graph, calculating it as it goes, to find a plan to reach the goal state. Running the final query will produce a plan via search.


How would you build an AI to output the primary concept of a paragraph?

Just to elaborate my brief comment on the question:

Identifying the primary concepts of a paragraph required understanding of the meaning of the text. In natural language processing we are still a long way off even recognising and representing the meaning of text, let alone summarising the meaning of multiple sentences into a single statement.

Note that this is different from simply summarising a text: this can be done without any understanding based on textual features within the text itself, and ways of doing that have been around for a while. But such approaches will generally remove sentences which seem less relevant to the text, thus shortening it. They will not express the content in different words.

Conceivably people might try this with deep learning, where you train a system with paragraphs and the corresponding concepts, but again such a system would not have any understanding of the meaning, and thus results would be more or less accidental.


Which work originally introduced gradient clipping?

Short Answer

Tomas Mikolov's mention of gradient clipping in a  single paragraph of his PhD thesis in 2012 is the first appearance in the literature.

Long Answer

The first source (Mikolov, 2012) in the Deep Learning book is Mikolov's PhD thesis and can be found here. The end of section 3.2.2 is where gradient clipping is discussed, only it's called turncating.

  ... The exploding gradient problem has
  been described in [4].

  A simple solution to the exploding gradient problem is to truncate values of the gradients. In my experiments, I did limit maximum size of gradients of errors that get accumulated in the hidden neurons to be in a range < −15; 15 >. This greatly increases stability
  of the training, and otherwise it would not be possible to train RNN LMs successfully on
  large data sets.
  ...
  [4] Y. Bengio, P. Simard, P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5, 157-166, 1994.

A search of the referenced paper [4] shows that it does describe the problem as Mikolov said, but it does not present gradient clipping as a solution.

So I had a look at the second source Deep Learning mentioned: On the difficulty of training Recurrent Neural Networks. It directly cites Mikolov as having proposed clipping:

  We would make a final note about the approach proposed by Tomas Mikolov in his PhD thesis (Mikolov, 2012) (and implicitly used in the state of the art results on language modelling (Mikolov et al., 2011)). It involves clipping the gradient’s temporal components element-wise (clipping an entry when it exceeds in absolute value a fixed threshold). Clipping has been shown to do well in practice and it forms the backbone of our approach.

I thought about emailing Mikolov to verify that his thesis was the origin of the idea. But then I noticed that he is a co-author of this paper which cites him as proposing it! Though I still wonder if it was commonly used in practice before
even though it had not been published.


What is easier or more efficient to summarize voice or text? [DP/RN]

Summarizing text is always going to be 'easier or more efficient' than voice simply because voice requires the additional step of converting to text.  That doesn't tell you anything about accuracy.

From an article published on June 1, 2017, Google’s speech recognition is now almost as accurate as humans:
"According to Mary Meeker’s annual Internet Trends Report, Google’s machine learning-backed voice recognition — as of May 2017 — has achieved a 95% word accuracy rate for the English language. That current rate also happens to be the threshold for human accuracy."

If you need this kind of accuracy check out Google's Cloud Speech API.  There is even a speech to text feature on the web page.

Given a speech-to-text conversion accuracy of 95%, voice will be 5% less accurate than text if everything else is equal but it usually isn't.  People generally write better text, such as in documents or emails, than when they speak unless of course they are giving a formal lecture, or talking in a formal meeting.  If one is analyzing text messages, Tweets, or threads found in typical informal forums, you will find very poor quality in grammar, spelling, vocabulary, and punctuation.  The answer to your question will depend on the source of your text.

In another article, dated November 13, 2017, Why 100% Accuracy Is Not Available With Speech Recognition Software Alone, the author gives some reasons, albeit for transcription software which has a special purpose, why there will always be some errors due to:

Speech Patterns and Accents - Regional variations exist, for example English speakers in Boston sound different than Kentucky.  How does the software handle slurred speech or when a person blends their words?
Grammar and Punctuation - speech recognition software doesn't know where a period, comma, or semi-colon belongs
Homonyms and unusual words - "Speech processing software can only recognize words and phrases that it has specifically been trained to recognize."
Ambient Noise, Overlapping Speech, and Number of Speakers

To address your last question about where the technology is going...
Four days ago a paper by Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria entitled Recent Trends in Deep Learning Based
Natural Language Processing was published which gives some of the answers.

From the 'Conclusion' section:
With distributed representation, various deep models have
become the new state-of-the-art methods for NLP problems.
Supervised learning is the most popular practice in recent
deep learning research for NLP. In many real-world scenarios,
however, we have unlabeled data which require advanced
unsupervised or semi-supervised approaches. In cases where
there is lack of labeled data for some particular classes or the
appearance of a new class while testing the model, strategies
like zero-shot learning should be employed. These learning
schemes are still in their developing phase but we expect deep
learning based NLP research to be driven in the direction of
making better use of unlabeled data. We expect such trend to
continue with more and better model designs. We expect to
see more NLP applications that employ reinforcement learning
methods, e.g., dialogue systems. We also expect to see more
research on multimodal learning [167] as, in the real world,
language is often grounded on (or correlated with) other
signals.

Finally, we expect to see more deep learning models whose
internal memory (bottom-up knowledge learned from the data)
is enriched with an external memory (top-down knowledge
inherited from a KB). Coupling symbolic and sub-symbolic AI
will be key for stepping forward in the path from NLP to natural
language understanding. Relying on machine learning, in
fact, is good to make a ‘good guess’ based on past experience,
because sub-symbolic methods encode correlation and their
decision-making process is probabilistic.


Is there a way to protect humanity against the impending singularity?

What you are talking about is known as the Control Problem. We have our own tag for this specific topic here, which you can use for this and similar questions.

How to address the control problem is heavily discussed and still considered unsolved. Two of the important approaches are motivation control and capability control.

Motivation control aims at creating suitable reward functions that only reward behavior that's beneficial to the human race. This topic alone is so complex that many different competing theories exist. Minor errors in the definition of an AI's goal might lead to catastrophe.

Capability control aims at limiting the possible scope of action for the AI. The main problem here is, that most restrictions make the AI less useful. And to ensure robust control a lot of restrictions must be put in place. Finding the right tradeoff is a challenge we need to solve.

In addition to those two mechanisms we must think about how we create a superintelligence. One of the major aspects here is our understanding of the underlying technologies. If we create a self-optimizing neural network from scratch, which we do not understand in depth, it is very likely that we will not understand the potential threats before it is to late. Creating a digital copy of a human brain as a starting point on the other hand is a little saver, because we can assume that it will be driven by similar factors as a real human being. If that's desirable is a totally different discussion of course. But the general idea is, that understanding the motivation of a superintelligence is key for steering it in the right direction.

For more details about the topics I scratched in my answer and the control problem in general, I recommend reading Superintelligence by Nick Bostrom. He spends many chapters on those problems and possible solutions.

If you have more in-depth questions concerning specific topics, please open a new question to get a more detailed answer.


Is the Cognitive Approach (SOAR) equivalent to the Chinese Room argument?

Searle's Chinese room is analogical and is intended to present an easy-to-understand picture of the essential elements and processes of the digital computer. In the room the man (CPU) has a book of intructions (program) for responding to Chinese input questions. That is just one program of many possible programs the room could run. Each different program would be a different instruction book. SOAR would be just one of those books.


Are there any studies which attempt to use AI to guess the human emotion based on the brainwaves?

As per this site

  Researchers recorded the complex patterns of electrical activity generated by someone’s brain, as the subject listened to someone talking.
  By feeding those brainwave patterns into a computer, they were able to translate them back into actual words — the same words that the volunteer had been hearing.

  The scientists behind the work believe they can now go further and read the unspoken thoughts of people using electrodes placed against the brain.

  In the experiment, each patient listened to a recording of spoken words for five to ten minutes, while the net of electrodes placed under their skull monitored activity in a part of the brain involved in understanding speech called Wernicke’s area.

  In one experiment, volunteers looked at black-and-white photographs while the scanner monitored activity in part of the brain that handles vision called the primary visual cortex.
  A computer predicted accurately the image that the person was looking at purely from the pattern of brain activity.

So AI might be able to read our emotions as well in near future.

I found that google glasses can detect people's emotion via facial expression, voice tone e.t.c, (just like us), obviously not what they are thinking in their brain.


Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?

Why has this merge proven beneficial?

If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top rather than Separation of Concerns it makes more sense.

The underlying premise is that the shared part of the network (the ResNet) provides a high-level generalization of the input (the game states leading up to the move) that is a good input representation for both the shallow Value and Policy networks.

When that is the case, we can reduce the computational load a lot by training a single shared ResNet and using it for two much simpler networks than training two ResNets for the Value and Policy. In their case, training the two together also improves regularisation and thus creates a more robust, general representation.

Specifically, the Alpha Go Zero paper by Silver et al., Mastering the Game of Go without Human Knowledge, states that:

  Combining policy and value together into a single network slightly reduced the
  move prediction accuracy, but reduced the value error and boosted playing
  performance in AlphaGo by around another 600 Elo. This is partly due to
  improved computational efficiency, but more importantly the dual objective regularises the network to a common representation that supports multiple use cases.

Can this technique be applied in general or only in special cases?

Like common components in software libraries, it only makes sense when the problems you are trying to solve benefit from a shared representation.

You can use it if you are training classifiers for similar tasks, or training a new task with little data where you already have a classifier trained over a larger, similar dataset.

Outside of Go, it is often used in image recognition.
Deep pre-trained networks such as the ones from the ImageNet ILSVRC competitions are commonly used as a starting point. They are classifiers that have been trained (for weeks!) on over a million images.

Then, say you want to create a network to recognize you favourite brand of bicycles, you start with the general image-recognition pipeline trained on ImageNet, chop of the last layers that do the actual classification ("it's a Border Collie") and add a small new classifier to pick out only the bicycles you care about.

Since the pre-trained classifier already provides high-level image concepts that are good building blocks for image recognition (it classifies 200 categories), this saves you a lot of training and makes for a very robust classifier.

Of course there are many cases where the problems do not have useful shared representations and thus no benefit from a combined network. Nevertheless, it is a useful tool in the right situations.

Look up Transfer Learning or Multi-Task Learning to learn more about this.


Why is a constant plane of ones added into the input features of AlphaGo?

I wonder why these features are necessary, because I think a constant plane contains no information and it makes the the network larger and consequently harder to train.

In many implementations of convolutional layers, the filters do not neatly remain inside the features plane when "sliding" along it, but (conceptually) also partially go "outside" the plane (where always at least one "cell" of the filter will still be inside the plane). Intuitively, in the case of a $3\times3$ filter for example, you can imagine that we pad the input features with an extra border of size $1$, and this padding around the "real" input planes is filled with $0$s.
If it's possible for all input features to also have values of $0$, it may in some situations be difficult or impossible for the neural network to distinguish the "real" $0$ inputs from the $0$ entries in the padding around the board, i.e. it may struggle to know where the game board is and where the game board ends. Having a constant input plane that's always filled with $1$s can help in this respect, because that plane can always reliably be used to distinguish "real" cells that actually exist on the game board from positions that are just outside the game board.
As for the plane filled with $0$s... I have no idea why that would ever be useful. Maybe it was useful due to some peculiar implementation detail. In this thread, some people hypothesise that on specific hardware it might make some computations slightly more efficient because of the layout of the data in memory -- it causes the number of channels to be divisible by $8$, which will.. maybe help? I really don't know too much about this, but I do know that on a smaller scale, sometimes adding unused data in classes can indeed increase performance due to better layout in memory. I suppose it's also possible that it was accidentally added by mistake, or "just in case" and that it doesn't really have much of a purpose. The amount of hardware that the AlphaGo team had available to them is quite insane anyway, one channel more or less probably wasn't too big of a concern for them.

What's more, I don't understand the sharp sign here. Does it mean "the number"? But one number is enough to represent "the number of turns since a move was played", why eight?

This is explained in the "Features for policy/value network" paragraph in the paper. Quote from the paper:

"Each integer feature value is split into multiple $19 \times 19$" planes of binary values (one-hot encoding). For example, separate binary feature planes ares used to represent whether an intersection has $1$ liberty, $2$ liberties, $\dots$, $\geq 8$ liberties.

All feature planes used were strictly binary, no feature planes were used that had integer values $> 1$. This is quite common when possible, because neural networks tend to have a much easier time learning with binary features than with integer- or real-valued features.


Can we make Object Detection as human eyes+brain do?

Object detection can conceivably imitate what the human visual system does.  Research along these lines began in the 1980s in multiple laboratories and was termed Computer Vision.

  How do we see ... and detect things and their location so quick[?]  Is the reason that we have huge gigantic network in our brain and we are trained since birth to till now and still training[?]

Neurological visual systems continue to train as long as the eyes and neurons continue to function.  A person may, long after doing well in sports, learn how to catch glass objects that fall off a counter using peripheral vision, using the same neural pathways as were used for visually coordinated movement in sports but.  They are newly trained to the new cognitive intention of saving glassware and clean-up time.

The mapping of signals from the rods and cones to the visual cortex has been studied and some generalities have influenced artificial vision research and development.  The convolution kernels used in CNNs are partly derived from biological research.  In summary, the way mammals and other organisms with the equivalents of retinas see is via a set of transformations and pattern matching circuits.  The sequence of information types through biological visual networks can be summarized.

Light as a function of compound angle and time.
Edges in both space and time
Elements of objects and actions — These are real world features, not in the machine learning sense but in the natural language use of the term features.
Objects and actions

The development of object and action recognition in computers follows this same basic sequence.  The goal stated in the question is this.

  Achieve an AI to detect object in picture in human ways

Something like this has been the goal of many government, academic, and corporate laboratories, however the ability to recognize objects in still pictures is not as valuable as detecting the trajectories and other expressions of objects (such as changes in facial expressions) in real time.

Automated aiming, piloting, walking, and driving have been computer vision objectives since the first control systems were developed for anti-aircraft defense systems.  Considerable effort is being invested into how steering, breaking, and signaling can be automated in road vehicles.  A large array of household and industrial devices are being developed that rely on mapping the environment and navigating a robotic device through the map.

Stationary object recognition work is largely surrounding either categorizing images or drawing vector graphics from raster images, but these functions have limited application.  As mobile devices develop further and images on the web give way to videos on the web, a very strong and uninterrupted trend, the temporal dimension will continue to gain importance in the field computer vision.

None of these objectives are novel.  They are all based on animal capabilities, only some of which are uniquely human, and have been the subject of research since before the advent of digital computers.  However the feasibility of many designs has improved, largely due to a significant reduction in costs for fixed computing resources.  The vision system hardware of today costs roughly one thousandth of its 1990 cost.


Validation results will almost never be as good as training results; that's just natural. As long as they are not too different, you should be fine. What "too different" means depends on the particular data set and model you're using.
If you plot the curves for varying parameter values, when the training error keeps going down but the validation error starts going up again, that's when you know there is overfitting. In your second graph, after 14 epochs, we might see the start of overfitting. If you continue this until 20 epochs or so, it should be even more clear. I would guess that 12 is probably a good value for the number of epochs for that problem.
In the first graph, we don't see that happening yet. The model might not be well-suited (the gap between training and validation results is a bit larger)  but that can also be because of too little data, or other factors. Perhaps that's just the best you can do; there might be noise in the data or something.


Why do language models place less importance on punctuation?

Language models almost always map every word to an embedding. There are many embedding algorithms with most of them having interpolation properties i.e. If $E(word)$ represents embedding of a word then $E(king)-E(male)+E(female) \sim E(queen)$. The smoother the interpolation properties the better the model understands the word, these properties don't exactly make much sense when it comes to delimiters.

Yet, there are instances where a delimiter embedding is learned ( always has an embedding ). While using these first all punctuation in the text is converted to one specific word, say 'dlmt', the embedding algorithm learns an embedding for this word treating it as if it would have any word. This maintains interpolation properties where the delimiter is understood to be a word that is used to split context.

I have observed that delimiters such as question mark or exclamation marks at the end of sentence are also understood to be breaks in context, in these cases the model learns if the statement is a question or so just by the context given by the words and stops in the sentence


Why is a softmax used rather than dividing each activation by the sum?

There are probably multiple different explanations and reasonings, but I can offer you one. If your output vector contains negative values, to get something that's related to probabilities (all components positive, summing to $1$) you cannot do what you suggested because you can possibly get a negative probability which doesn't make sense. Good property of exponential function used in softmax, in this case, is that it cannot give negative values, so regradless of your output vector, you will never get negative probability.

You could suggest adding some positive offset vector $\mathbf d$ to your output vector to get rid of negative values but there are couple of problems. First, you cannot know in advance your range of negative output values so that you can know what offset vector to use to cover all possible cases. Second, with such strategy in some cases you could get unrealistic results. For example let's assume output vector is $[-0.1, 0.2, 0.3]^T$ and offset vector is $[0.1, 0.1, 0.1]^T$. If you add those 2 you get $[0, 0.3, 0.4]^T$. Probability of first class would be $0$ since the numerator is $0$. That is very overconfident result and we probably wouldn't want to get the result of $0$ for this class. The result would change depending on the offset vector. Let's say that offset vector now is $[0.3, 0.3, 0.3]^T$. Addition of offset vector and output vector gives $[0.2, 0.5, 0.6]$. In this case probability of first class is now $0.2/(0.2 + 0.5 + 0.6) = 0.15$. We see that changing the offset vector changes the values of probabilities and as components of offset vector $\rightarrow \infty$ probabilities for all classes $\rightarrow 0.33$. We would likely want to get the same result regardless of the scale of the values of the vector, we only care about its relative relationships. Another good property of softmax is that it is shift invariant
\begin{align}
p_i &= \frac{e^{x_i + d}}{\sum_{j=1}^n e^{x_j + d}}\\
&= \frac{e^d \cdot e^{x_i}  }{ e^d \cdot \sum_{j=1}^n e^{x_j} }\\
&= \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}
\end{align}
so we see that the probability of $i$-th component is independent of offset vector. Apparently softmax doesn't care about the scale of values, it manages to capture the relative relationships of the components.


Is intelligence limited to thoughts it is able to associate words with?

First thing we were taught in AI:
Humans didn't create airplanes by studying and mimicing birds, they studied aerodynamics and physics.
Although difficult to articulate I've had very similar thought patterns to yours when I was thinking about AI and further I watched very closely the development of my child from being a newborn to where he is now, a year and a bit old.  The human brain is far more complex than what we'll understand, it would take a multi-discipline team effort to even scratch the surface.  So to create a human brain-like thought process and try to put it into a formal language box would be impossible.
I think the breakthrough in AI isn't going to be someone putting together multiple complex algorithms or even a fancy NN running on a supercomputer.  It's going to be something we haven't figured out yet.
Still, fun to think about...  I'd say it's more of a philosophical question than an AI one though...
Just my humble opinion.


Can rule induction be considered a way to "hybridize" probabilistic / statistical approaches and symbolic approaches?

Sure! This is a somewhat hot area right now.

There are lots of ways to do it.

Probably the main line of research is with Bayesian Networks (1980's) and Casual Networks (1990's). These are basically rule-based systems for reasoning probabilistically. They rely on a user-designed model, which corresponds well to rules (e.g. when blood pressure is high, then heart attack rates are elevated), but provide a robust way to reason about uncertainties in the presence of these rules. Contrast this with a pure learning approach, like a decision tree or a neural network, which tends to rely less on rules. Modern research in this area focuses on learning the structure of the network (which corresponds to the learning probabilistic rules) from data.

While it's possible to learn rules from data and then do symbolic reasoning atop them using other techniques (e.g. rule induction), this approach runs into the same problems that plague the learning of the structure of Bayesian networks: when is correlation causation? Causal Diagrams are the only good tool for answering this question, but my impression is that inferring their structure automatically is still an open question.


XML, HTML and less formal languages all respond quite nicely to being transformed or interrogated within a graph framework. XML and HTML are particularly useful in that they conform strictly to a tree-structure. That means that any good data components can be measured in terms of tree-distance to any other "good" data components.

If you extract your regex-friendly terms and keep track of where within their tree they are found, you may be able to cast those values into a general document-space vector (it might only need to be one-dimensional), allowing you to identify clusters of "good" vs anomalous sections of "bad" data based on a simple distance metric, or anomaly detection algorithm - say an isolation-forest that runs on information density for example.

This depends on your data, and how much of it you can find, ideally already tagged up containing good vs bad.

If you're looking to scrape reliable address-contents, then yes, you're likely to score hits on names, addresses, postcodes and phone-numbers all appearing as tightly connected clustered groups, all within one or two nodes-distance from one another.

Meanwhile, an annotation containing a phone number lodged somewhere else is less likely to be a match.

Different documents will have different threshold densities, and differing anomaly to conformant ratios, so you'd have a task on your hands to figure out some way to automatically tune your parameters on any given document set.

In the past, I've tried doing this against html by flattening all the content into a single string of text and a similar approach yielded half-decent results, but if you're looking at XML, it's fair to expect the structure to yield more information.


How is the cost of the path to each node computed in A*?

The evaluation function in A* is $f(n) = g(n) + h(n)$, where $g(n)$ is the cost of the path from the starting node to $n$ and $h(n)$ is an estimate of the distance from $n$ to the goal node. To compute $g(n)$, you simply do $g(n) = g(p) + c(p, n)$, where $p$ is the parent node, and $c(p, n)$ is the $c$ost of the edge between $p$ and $n$. So, yes, to compute $g(n)$ you use $g(p)$.


How can a system like Jarvis understand the commands and take actions?

Jarvis was built using the suite of tools that facebook developers are constantly updating. The answer to this question is that there's no simple answer; it has a lot of moving parts.
Take for example natural language processing. There are a number of sub-topics that are each considered "big" problems, such as part-of-speech recognition, coreference resolution, sentiment analysis, relationship extraction, and many more. Tools have been built to tackle these various topics, but, to my knowledge, none of them really understand language, but rather statistically approximate it.
In the case of Jarvis, since it is a home-automation system, it's probably built with the user-given commands in mind from the beginning, so it's not trying to understand the whole human language, it's built to do some tricks.
Just keep in mind on your journey into this space that true AI like what we imagine will have some defining features like hierarchical representations of tasks and goal-orientation. If you really get into it I'd start with reinforcement learning, or try reading through the Society of Minds.


What are the differences between artificial neural networks and other function approximators?

First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule).

The first difference between neural networks and other function approximators is conceptual. In neural networks, you typically imagine that there are one or more computational units (often called neurons) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of flexibility and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks.

The other difference is that neural networks were originally inspired by the biological counterparts. See A logical calculus of the ideas immanent in nervous activity (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.

There are other technical differences. For example, the Taylor expansion of a function is typically done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the Fourier series can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (at least, this is the setup in the famous paper that proved the universality of neural networks) in many different ways (for example, weighted sums followed by sigmoids).

To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).


Why does Stephen Hawking say "Artificial Intelligence will kill us all"?

It's not just Hawking, you hear variations on this refrain from a lot of people.  And given that they're mostly very smart, well educated, well informed people (Elon Musk is another, for example), it probably shouldn't be dismissed out of hand.

Anyway, the basic idea seems to be this: If we create "real" artificial intelligence, at some point, it will be able to improve itself, which improves it's ability to improve itself, which means it can improve it's ability to improve itself even more, and so on... a runaway cascade leading to "superhuman intelligence".  That is to say, leading to something that more intelligent than we area.

So what happens if there is an entity on this planet which is literally more intelligent than us (humans)? Would it be a threat to us?  Well, it certainly seems reasonable to speculate that it could be so.   OTOH, we have no particular reason, right now, to think that it will be so.

So it seems that Hawking, Musk, etc. are just coming down on the more cautious / fearful side of things.  Since we don't know if a superhuman AI will be dangerous or not, and given that it could be unstoppable if it were to become malicious (remember, it's smarter than we are!), it's a reasonable thing to take under consideration.

Eliezer Yudkowsky has also written quite a bit on this subject, including come up with the famous "AI Box" experiment.  I think anybody interested in this topic should read some of his material.

http://www.yudkowsky.net/singularity/aibox/


What do echo state networks give us over a generic RNN resevoir?

The echo state condition states that differences in the input sequence results in separate trajectories of reservoir states.
Which means that when you make the reservoir large enough any difference in the input sequence result in a linearly separable difference in the state space.
This is quite similar to the Kerneltrick in e.g. Support Vector machines where the data becomes linearly separable in the feature space. In reservoir computing the reservoir is in a way a random feature space.
To sum up, the echo state condition ensures that the signal becomes linearly separable.


What is the concept of the technological singularity?

The technological singularity is a theoretical point in time at which a self-improving artificial general intelligence becomes able to understand and manipulate concepts outside of the human brain's range, that is, the moment when it can understand things humans, by biological design, can't.
The fuzziness about the singularity comes from the fact that, from the singularity onwards, history is effectively unpredictable. Humankind would be unable to predict any future events, or explain any present events, as science itself becomes incapable of describing machine-triggered events. Essentially, machines would think of us the same way we think of ants. Thus, we can make no predictions past the singularity. Furthermore, as a logical consequence, we'd be unable to define the point at which the singularity may occur at all, or even recognize it when it happens.
However, in order for the singularity to take place, AGI needs to be developed, and whether that is possible is quite a hot debate right now. Moreover, an algorithm that creates superhuman intelligence (or superintelligence) out of bits and bytes would have to be designed. By definition, a human programmer wouldn't be able to do such a thing, as his/her brain would need to be able to comprehend concepts beyond its range. There is also the argument that an intelligence explosion (the mechanism by which a technological singularity would theoretically be formed) would be impossible due to the difficulty of the design challenge of making itself more intelligent, getting larger proportionally to its intelligence, and that the difficulty of the design itself may overtake the intelligence required to solve the said challenge.
Also, there are related theories involving machines taking over humankind and all of that sci-fi narrative. However, that's unlikely to happen, if Asimov's laws are followed appropriately. Even if Asimov's laws were not enough, a series of constraints would still be necessary in order to avoid the misuse of AGI by misintentioned individuals, and Asimov's laws are the nearest we have to that.


What is different in each head of a multi-head attention mechanism?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention head is the same. For attention head $i$:
$Q_i(x) = x W_i^Q \\
K_i(x) = x W_i^K \\
V_i(x) = x W_i^V \\
attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\sqrt{d_k}} \right) V_i(x)$.
Notice that the input to each head is $x$ (either the semantic + positional embedding of the decoder input for the first decoder layer, or the output of the previous decoder layer). More info
The question as to why gradient descent learns each set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ to be different across each attention head is very similar to "Is there anything that ensures that convolutional filters end up the same?", so maybe you might find the answer there helpful for you:

No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has n features, but 2 of them are the same, then the model effectively has n−1 features, which is a less expressive model than that of n features, and therefore usually has a larger loss function.


How to define or represent evil in logic

I think you're going to have to be reconciled to the subjective nature of reality.  Objectivity is only possible in very special cases such as a Q.E.D. in mathematics, or a solved gamed.  Rationality is bounded, and any intractable problem results in a state of subjectivity/indeterminacy.  Additionally, pure values do not carry moral implications, despite popular associations, although it would be possible to create a game where certain values have negative effects, and the harm they result in could be understood as evil.  (i.e. 666/616 has numerological associations, and numerology can be understood as a proto form of number theory.)

A simple way to define evil would be through behavioral models in Game Theory.
In Game Theory, there is a concept knowns as the superrational strategy.
Superrationality may be understood as the logical/mathematical expression of the Golden Rule: "do unto others as you would have them do unto you."
The Golden Rule forms the basis for most religions, in the context of empathy/compassion and altruism.
If the agent is evil, it will always betray, even when the competitor has shown a willingness to cooperate.

Thus evil is defined as the opposite of the Golden Rule.  (Possibly we would call this the Brimstone Rule;)


Where can I discuss with deep learning beginners?

I highly recommend to get a local peer group. Meetups might be a good way to get to know some people. As a student at university you also have a lot of other options like going to lectures, creating a student group, organizing a local AI event / hackathon, ...

If this is not possible for you, here is what you can do online:

Reddit: r/deeplearning seems to have a couple of discussions. r/MachineLearning is cool if you want to discuss super recent stuff, but I would not say it is beginner level.
Google+
Coursera: if you take one of their courses, you will see a forum for discussions


How can I build an AI with NLP that reads and understands documents?

When automating tasks involving text, NLP techniques are definitely the way to go! Let me be frank, though, starting from scratch and "read and understand" will be harder with today's tools than directly reading the papers.

Some typical required skills are below, and the bar height will depend on your mileage:

Knowledge of NLP (e.g. terminology like embeddings, bag of words, even perhaps tf-idf).
Knowledge of NLP libraries and implementation frameworks (e.g. Stanford's or TensorFlow).
Knowledge of a programming language (e.g. Python or Java).

In short, the goal you set will probably require many more papers than expected. But if you succeed only the automated reading / summary part, you have a bright future ahead (let aside understanding, a way more contentious problem).

Just for the sake of putting an example, companies like Iris.ai and others try to solve related problems---great potential, but definitely challenging.


Which artificial neural network can mimic biological neurons the most?

Only a small portion of the habituation, sensitization, and classical conditioning behavior of neurons has been primitively simulated in ANN systems. Simulation of actin cytoskeletal machinery1 and other agents of neural plasticity, central to learning new domains, is in its beginnings2.  As of this writing, the complexity of neuron activation dwarfs the models being used in working commercial ANN systems, but the research continues along multiple fronts.

The neuroscience of learning3,
Parallel hardware approaches that better support ANN simulation accuracy4, 5, and
Dynamic frameworks6

This list and the examples referenced in the superscripts, with links below, represent a tiny sample of the information available and the work in progress.

References

[1] Molecular Cell Biology. 4th edition.; Lodish H, Berk A, Zipursky SL, et al.; New York: W. H. Freeman; 2000.; Section 18.1 The Actin Cytoskeleton

[2] NEURON Software; Yale U

[3] Molecular Cell Biology. 4th edition.; Lodish H, Berk A, Zipursky SL, et al.; New York: W. H. Freeman; 2000.; Section 21.7 Learning and Memory

[4] Artificial Neural Networks on Massively Parallel Computer Hardware; Udo Seiffert; University of Magdeburg, Germany

[5] NeuroGrid; Stanford U

[6] Explanation of Dynamic Computational Graph frameworks


How to architect a network to find bounding boxes in simple images?

There are some special architectures of CNNs which are designed exactly for the task you mention. The Detector library includes a collection of these architectures, this paper describes the Mask R-CNN network in detail, which is designed for image segmentation tasks.


What is the time complexity for training a neural network using back-propagation?

I haven't seen an answer from a trusted source, but I'll try to answer this myself, with a simple example (with my current knowledge).

In general, note that training a MLP using back-propagation is usually implemented with matrices.

Time complexity of matrix multiplication

The time complexity of matrix multiplication for $M_{ij} * M_{jk}$ is simply $\mathcal{O}(i*j*k)$.

Notice that we are assuming simplest multiplication algorithm here: there exists some other algorithms with somewhat better time complexity.

Feedforward pass algorithm

Feedforward propagation algorithm is as follows.

First, to go from layer $i$ to $j$, you do

$$S_j = W_{ji}*Z_i$$

Then you apply the activation function

$$Z_j = f(S_j)$$

If we have $N$ layers (including input and output layer), this will run $N-1$ times.

Example

As an example, let's compute the time complexity for the forward pass algorithm for a MLP with $4$ layers, where $i$ denotes the number of nodes of the input layer, $j$ the number of nodes in the second layer, $k$ the number of nodes in the third layer and $l$ the number of nodes in the output layer.

Since there are $4$ layers, you need $3$ matrices to represent weights between these layers. Let's denote them by $W_{ji}$, $W_{kj}$ and $W_{lk}$, where $W_{ji}$ is a matrix with $j$ rows and $i$ columns ($W_{ji}$ thus contains the weights going from layer $i$ to layer $j$).

Assume you have $t$ training examples. For propagating from layer $i$ to $j$, we have first

$$S_{jt} = W_{ji} * Z_{it}$$

and this operation (i.e. matrix multiplcation) has $\mathcal{O}(j*i*t)$ time complexity. Then we apply the activation function

$$
Z_{jt} = f(S_{jt})
$$

and this has $\mathcal{O}(j*t)$ time complexity, because it is an element-wise operation.

So, in total, we have

$$\mathcal{O}(j*i*t + j*t) = \mathcal{O}(j*t*(t + 1)) = \mathcal{O}(j*i*t)$$

Using same logic, for going $j \to k$, we have $\mathcal{O}(k*j*t)$, and, for $k \to l$, we have $\mathcal{O}(l*k*t)$.

In total, the time complexity for feedforward propagation will be

$$\mathcal{O}(j*i*t + k*j*t + l*k*t) = \mathcal{O}(t*(ij + jk + kl))$$

I'm not sure if this can be simplified further or not. Maybe it's just $\mathcal{O}(t*i*j*k*l)$, but I'm  not sure.

Back-propagation algorithm

The back-propagation algorithm proceeds as follows. Starting from the output layer $l \to k$, we compute the error signal, $E_{lt}$, a matrix containing the error signals for nodes at layer $l$

$$
E_{lt} = f'(S_{lt}) \odot {(Z_{lt} - O_{lt})}
$$

where $\odot$ means element-wise multiplication. Note that $E_{lt}$ has $l$ rows and $t$ columns: it simply means each column is the error signal for training example $t$.

We then compute the "delta weights", $D_{lk} \in \mathbb{R}^{l \times k}$ (between layer $l$ and layer $k$)

$$
D_{lk} = E_{lt} * Z_{tk}
$$

where $Z_{tk}$ is the transpose of $Z_{kt}$.

We then adjust the weights

$$
W_{lk} = W_{lk} - D_{lk}
$$

For $l \to k$, we thus have the time complexity $\mathcal{O}(lt + lt + ltk + lk) = \mathcal{O}(l*t*k)$.

Now, going back from $k \to j$. We first have

$$
E_{kt} = f'(S_{kt}) \odot (W_{kl} * E_{lt})
$$

Then

$$
D_{kj} = E_{kt} * Z_{tj}
$$

And then

$$W_{kj} = W_{kj} - D_{kj}$$

where $W_{kl}$ is the transpose of $W_{lk}$. For $k \to j$, we have the time complexity $\mathcal{O}(kt + klt + ktj + kj) = \mathcal{O}(k*t(l+j))$.

And finally, for $j \to i$, we have $\mathcal{O}(j*t(k+i))$. In total, we have

$$\mathcal{O}(ltk + tk(l + j) + tj (k + i)) = \mathcal{O}(t*(lk + kj + ji))$$

which is same as feedforward pass algorithm. Since they are same, the total time complexity for one epoch will be $$O(t*(ij + jk + kl)).$$

This time complexity is then multiplied by number of iterations (epochs). So, we have $$O(n*t*(ij + jk + kl)),$$ where $n$ is number of iterations.

Notes

Note that these matrix operations can greatly be paralelized by GPUs.

Conclusion

We tried to find the time complexity for training a neural network that has 4 layers with respectively $i$, $j$, $k$ and $l$ nodes, with $t$ training examples and $n$ epochs. The result was $\mathcal{O}(nt*(ij + jk + kl))$.

We assumed the simplest form of matrix multiplication that has cubic time complexity. We used batch gradient descent algorithm. The results for stochastic and mini-batch gradient descent should be same. (Let me know if you think the otherwise: note that batch gradient descent is the general form, with little modification, it becomes stochastic or mini-batch)

Also, if you use momentum optimization, you will have same time complexity, because the extra matrix operations required are all element-wise operations, hence they will not affect the time complexity of the algorithm.

I'm not sure what the results would be using other optimizers such as RMSprop.

Sources

The following article http://briandolhansky.com/blog/2014/10/30/artificial-neural-networks-matrix-form-part-5 describes an implementation using matrices. Although this implementation is using "row major", the time complexity is not affected by this.

If you're not familiar with back-propagation, check this article:

http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4


Training an artificial neural network using PSO

Particle Swarm Optimization can be used to optimize a neural network with more than one hidden layer. Instead of optimizing a single weight matrix, and two bias vectors, you are just optimizing more of them. However, PSO is not often used for larger neural networks, because, particle swarm optimization is not all that efficient at working with a large amount of data, which you need to train larger neural networks. Particle Swarm Optimization involves taking many particles, and evaluating the fitness in all of them. If you have let’s say 1,000,000 training examples, you are doing a lot more error calculations, even if you used a small batch size, than if you used another technique lake Backproppogation, or a related technique. Backproppogation and related techniques are used much more in larger neural networks because they are more efficient at working with larger sets of data. As for why the code that you have been using breaks with networks with more than one hidden layer, I cannot explain, but it is fairly easy to write your own basic implementation of PSO to train multi layer neural networks.


Is it appropriate to use a softmax activation with a categorical crossentropy loss?

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE).
Here's the BCE (equation 4.90 from this book)
$$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$
where

$t_{n} \in\{0,1\}$ is the target
$y_n \in [0, 1]$ is the prediction (as produced by the sigmoid), so $1 - y_n$ is the probability that $n$ belongs to the other class

Here's the CCE (equation 4.108)
$$
-\sum_{n=1}^{N} \sum_{k=1}^{K} t_{n k} \ln y_{n k}\label{2}\tag{2},
$$
where

$t_{n k} = \{0, 1\}$ is the target of input $n$ for class $k$, i.e. it's $1$ when $n$ is labelled as $k$ and $0$ otherwise (so it's $0$ for all $K$ except for one of them)
$y_{n k}$ is the probability that $n$ belongs to the class $k$, as produced by the softmax function

Let $K=2$. Then equation \ref{2} becomes
$$
-\sum_{n=1}^{N} \sum_{k=1}^{2} t_{n k} \ln y_{n k} =
-\sum_{n=1}^{N} \left( t_{n 1} \ln y_{n 1} + t_{n 2} \ln y_{n 2} \right)
\label{3}\tag{3}
$$
So, if $[y_{n 1},  y_{n 2}]$ is a probability vector (which is the case if you use the softmax as the activation function of the last layer), then, in theory, the BCE and CCE are equivalent in the case of binary classification. In practice, if you are using TensorFlow, to choose the most suitable loss function for your problem, you could take a look at this answer.


How to alpha-beta pruning to expectiminimax

alpha-beta-prunning algorithm is using for improve performance and reject the options, which not consist the condition - or it's possible to set some factors of probability to take into consideration or not.
This algorithm work on tree structure - and if there are a lot of levels (10-20) - It allows you to eliminate paths - which will logically not be used - saving memory and computing resources.
In this particular case - for finding the minimum value it works like this:
First branch:

Go to B
Go to D - and there is 2 and 3 - so return the min 2
Go From B to E - and choose 5 - the minimal value in B points is actually 3 - so there isn't need for checking the next - cause everything below E, will be higher than D (3)

Second branch:

Go to
Go to F - and check 0 and 1
If C have 1 - than not necessary to go to G - cause 1 is the smaller.

The given sequence is a simplification - and in the case of this algorithm there are also layers - min & max

Source:
https://www.javatpoint.com/ai-alpha-beta-pruning#:~:text=Key%20points%20about%20alpha%2Dbeta,values%20to%20the%20child%20nodes.

Implementation, however, is a more complex matter - unless we do copy and paste :-)
https://gist.github.com/exallium/1446104/5109388cfc21578f555dcac0ba54da680326af7b


Can GANs be used to generate something other than images?

They can indeed. Although generally they are kept to images because at the moment, they are the best at that, but not the best in other areas that you might consider.
GANs can be used for audio generation, with many examples such as GANsynth and GAN voice generation. But each of these tasks are outperformed by other methods. With music generation, WaveNet is the best (last I checked, and it also performs very well at speech synthesis), and a more powerful model for voice generation is achieved through the use of a VAE.
This is only looking at one area that you could use GANs for, because in reality you could use them for any kind of generation if you wanted to, but at the moment the vast majority of the research into GANs is into image generation, and as such other areas do not compete with the current SOTA techniques, unless there's some big paper I've missed within the last few months.


Can AI research lead to new findings in general cognitive science?

This is about hard AI and soft AI: proponents of hard AI work on systems that simulate the way human cognition works, with the eventual (hypothetical) goal of replicating it. This presupposes that you know how cognition works, and presumably you will learn about it as you attempt to replicate it.

Soft AI, on the other hand, tries to emulate the outcomes only. For example, Weizenbaum's ELIZA is clearly on this side, as it uses simple pattern matching, and does not 'understand' anything about the conversations it is having.

Obviously, we don't even know fully what it means to 'understand' something, and building working systems is not really possible with a hard approach. Hence, soft AI is more common, as researchers are usually measured by their outcomes rather than their ideas. As far as I am aware, the hard AI approach has been all but abandoned long ago.

As current AI seems to be dominated by statistical approaches, I doubt that we can find out many useful things about cognition this way.

One interesting side-note: it seems to me that the capabilities of modern AI systems have developed away from human capabilities. A three-year-old can do some things that a sophisticated AI system cannot do, but in some areas (chess, translation, ...) the capabilities of AI systems surpass what humans are capable of. Maybe imitation is indeed not the right way to approach AI.


Should I model my problem as a semi-MDP?

To model the problem through RL,

  it is possible to discretize time horizon into very short time interval (for example 5 minutes as a stage) such that in each time interval, just a single customer enter to our system.
  On the other hand, it is possible that stages are defined as the time when a customer enters our system.

  1) Is the second approach an SMDP and if I want to solve it with RL, I should use Hierarchical RL?

The second approach as you described it does not sound like an SMDP to me... it sounds to me like a regular MDP where the discrete time steps in the MDP do not have any meaningful connection anymore to real-world time in the real-world problem being modelled. This is the approach I would recommend taking if long periods of "inactivity" in between customers are irrelevant and/or if you do not expect your actions to have any influence on the duration of those periods of inactivity.

If you do expect your actions to have meaningful influence on the duration of inactivity, I would not recommend this approach. For example, your actions may make a customer angry and reduce their likelihood of quickly returning. Or your actions may indirectly affect the likelihood of other customers visiting due to interaction between your population of customers, they may or may not recommend your bank to their friends based on your actions.

  2) In the first approach, if a customer enters in a time interval, it is easy to update $Q$-values. However, what should we do, if we are in state $S$ and take action $A$ but, no customer enter our system so we do not receive any reward for the pair of $(S, A)$ there would be no difference if we would take action $A_1$, $A_2$ and so on. This can happen for several consecutive time interval. I think it is more challenging when we consider eligibility trace.

I would actually argue that your first approach can be treated as an SMDP, and that it may be a good idea to do so (personally I prefer the Options framework, it's a bit more general than SMDPs, a bit more flexible, and a bit closer to "standard" RL terminology; there are plenty of free versions of that paper available on goole too).

Whenever you encounter a customer, you could select an option (or "macro-action") in the SMDP, which executes the one "real" action for your current customer once, and is subsequently forced to automatically select no-op actions (which do nothing) until the next customer arrives; that is the point in time where your previous option ends, and a new option can be selected.

An RL algorithm that operates on the SMDP / hierarchical level, i.e. one that only learns to optimize the selection of options / macro-actions and completely skips over all the forced no-op actions, would suffice. This is basically the approach described in Concurrent Reinforcement Learning from Customer Interactions.


What is a good way to create an artificial self-recognition?

Interesting question.  I don't think anybody knows a definite answer, but some rough-sketch ideas seem apparent.  Think about what it means to you to be "self aware".  You'll probably cite the way you "hear" your own thoughts in your head when you think about something.  One can speculate that inside the brain, the various centers that are responsible for hearing, vision, logic, etc. are connected so that as you form a thought, it's being "heard" by the hearing regions, even though it's purely internal instead of actual sound received at the ear.

So in AI terms, it seems likely that self-awareness will somehow involve taking the "thoughts" formed within the AI, and feeding them back into the AI so that it "hears" (or, more broadly, "senses") itself think.

There's this weirdly recursive aspect to all of this, which - interestingly enough - is something Douglas Hofstadter talked about a lot in some of his book, especially GEB.  He was probably onto something.


Heavy loss and inaccurate answer in pytorch

There are a few things you could do to improve this NN, but are probably worth covering in different questions.

Your main problem though is that you forgot to reset the gradient after each training batch. You need to call optim.zero_grad() in order to do this, at the start of each training loop. Otherwise, using PyTorch, the gradient values keep accumulating inside the model's training data (sometimes you want this effect if you are adding gradients from multiple sources, that's why PyTorch is not clearing them automatically for you).

In addition, a learning rate of 0.5 is very high for the Adam optimiser - it is very common to leave it at the default value because Adam is an adaptive optimiser that will adjust step sizes depending on gradients seen so far.

Here is a working version of your code:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):

        super(Net,self).__init__();

        self.fc1 = nn.Linear(in_features=2,out_features=3)
        self.fc2 = nn.Linear(in_features=3,out_features=6)
        self.out = nn.Linear(in_features=6,out_features=1)

    def forward(self,x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = self.fc2(x)
        x = torch.relu(x)
        x = self.out(x)
        return x

data = torch.tensor([[2,4],[3,6],[3,3],[4,4],[100,5]],dtype=torch.float)
values = torch.tensor([[8],[18],[9],[16],[500]],dtype=torch.float)

lossfun = torch.nn.MSELoss()
model=Net()
optim = torch.optim.Adam(model.parameters(),lr=0.001)

for epoch in range(20000):
    optim.zero_grad()
    y_pred=model.forward(data)
    loss = lossfun(y_pred,values)
    if (epoch % 1000 == 0):
        print(loss.item())
    loss.backward()
    optim.step()

This version can be tweaked to quite easily reach 0 loss for your data set.

This has not in really learned how to multiply two values. The approximation to multiplying will be very weak as there is very little data. However, playing with some very basic data and a simple NN is a first step towards understanding details like this . . .


Self-driving control logic based on semantic segmentation

This is a great question. Connecting the AI knowledge to the application is a difficult task and its not easy to test and perfect.

There are a few points regarding autonomous driving algorithms that one should keep in mind.

A lot of data is required before actually applying a model to the real world.
The control algorithms are designed for specific tasks, there is no master algorithm that solves the issue.
Time is very important, decisions have to be made in real time.

The image segmentation you have mentioned in the question is rarely used in current autonomous technologies. The main issue is that

The model is slow and requires massive computing resources. Due to this is cannot help in real time decision making.
A very important parameter depth is missing from the information. This is of utmost importance in autonomous driving technology. A 2D image cannot provide this parameter.

So the current technology use

A range of sensors (LIDAR, Ultrasonic, Cameras etc.) that provide readings 100+ times a second.
A lot of the computer vision algorithms and control algorithms are hard coded. There are set parameters within which decisions are made.
The control algorithms are designed to work with each set of sensors and the mechanical components of the car.

To get a better look at the way these algorithms are used, you can have a look at

Stanley vision system winner of the DARPA grand challenge (Stanford)
How a driverless car sees the road TEDx


Are there local search algorithms that make use of memory to give better solutions?

You could parallelize the search by dividing the global space in distinct regions/subsets. Then apply in each region a local search. This way you can search the global space systematically, more exhaustively and perhaps in different ways (e.g by applying a different local search method to each region). Finally you can compare the results and choose the best one.


Reinforcement learning with industrial continuous process

RL is a generic technique that can be applied to any MDP system. From the looks of it you have data to produce a state-space model of your system (system identification, excluding your existing control loops) and then you can use that to drive a simulated exploration of your process and discover a control policy. As this is a continuous process, some discretization will be required

so yes, your quest is feasible, but not trivial!

I experimented sometime ago with the cartpole balancing system, which is a toy, but I found that a good old PID was much better than AI :)


Why did a Tesla car mistake a truck with a bright sky?

Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies.

Warning: I cannot find again the source critical to the next paragraph, and reading again pages over pages, I cannot find similar argument in other reports. I still remember vividly the point below, but please keep in mind it may be incorrect. The rest of the answer is weakly related, so I leave it all, with this warning.

An independent report (link needed, I can't find it...) explained that the assistive system was unable to detect the truck due to an exceptionally low contrast (bright sky perceived as white---colour of the truck). The report also said that a human driver would have been unable to make the difference either. In other words, it is possible that car sensors (presumably camera) and the human eye could not have detected an obstacle, and could not have triggered any safety measure. This short graphical explanation sums up the car sensors: Camera, radar, GPS, etc.

The assistive sub-system is based on proprietary AI technologies. We can only speculate under some hypothesis. _This is not very useful, honestly, except for illustration purpose. Assuming the assistive system relies on ML technologies to learn about obstacles from a video stream (such systems do exist):

It may be that the learning data was not "good enough" to cover the truck scenario.
It may be the technology was not powerful enough yet (lack generalization power, or simply too slow).
It may be a hardware problem, notably from the sensors: If the "car's eyes" are defective, the "car's brain" (the assistive system) is unable to react properly.

Why those technologies did not work in that case will remain a secret. We can say however that any system---whether built with AI technology or not---has limits. Beyond these limits, the system reaction is unpredictable: It could stop, reset, shutdown. The difficulty here is to define what a "default behaviour" is. A machine will basically do whatever it is designed to do, so an AI-based system too.

We could speculate even more on what would happen if the assistive system was really autonomous, the elusive AGI, but that is really not the case here.


Why is the sample size of stochastic gradient descent a power of 2?

You can take any size you want since most memories are now larger than that, if you were to map the indices to a register, 64 values will take 6 bits whereas we have registers upto 64 bits, so it wouldn't matter much even if you choose a  different sample size.

As far as resolution is concerned, layers like maxpool which reduce the size by half (2x2 kernel is generally used), the resolutions of outputs of layers are multiplied/divided by powers of 2, so keeping that in mind, resolutions as power of 2 makes sense, but then again, we have padding and effects of convolution on borders.


What is the relation between multi-agent learning and reinforcement learning?

I think there is an intersection. There are problems that are in reinforcement learning and in learning in multi-agent systems. There are problems in reinforcement learning, but not exactly in multi-agent systems. And there is learning in multi-agent systems that is not through reinforcement learning. For sort you can say: multi-agent reinforcement learning. I recommend take a look at these references:

Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.
Busoniu, Lucian, Robert Babuska, and Bart De Schutter. "Multi-agent reinforcement learning: A survey." 2006 9th International Conference on Control, Automation, Robotics and Vision. IEEE, 2006.


What is the difference between a language model and a word embedding?

Simplified: Word Embeddings does not consider context, Language Models does.
For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.
Think of the following two sentences:

The fish ate the cat.

and

The cat ate the fish.

If you averaged their word embeddings, they would have the same vector, but, in reality, their meaning (semantic) is very different.
Then the concept of contextualized word embeddings arose with language models that do consider the context, and give different embeddings depending on the context.
Both word embeddings (e.g Word2Vec) and language models (e.g BERT) are ways of representing text, where language models capture more information and are considered state-of-the-art for representing natural language in a vectorized format.
BLEU score is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Which is not directly related to the difference between traditional word embeddings and contextualized word embeddings (aka language models).


How does the crossover operator work when my output contains only 2 states?

How do you calculate fitness for your organism if there are only two states? I would think it would be a solution to work on combinations of cells:

If you have blocks of, say, 4x4 cells, then each organism would be encoded using 16 bits. A crossover then is like overlaying two 4x4 blocks, picking the new block from either the first or the second parent. Effectively you are doing this:

0011                   1111                 xx11
0001  combined with    0000  would become   000x
1001                   0000                 x00x
1111                   0000                 xxxx

(this would be a bottom-right corner combined with a top wall segment)

or, as a single bit-string:

0011000110011111 +
1111000000000000 =
xx11000xx00xxxxx

where 'x' can be either 0 or 1, as this is where the two organisms are different.


Can an artificial intelligence eventually think like a human?

You need to fully understand the human mind before you can say that a  machine can eventually think like a human. However, the human brain is nowhere near fully understood, nor is human cognition.


How exactly is Monte Carlo counterfactual regret minimization with external sampling implemented?

External sampling and outcome sampling are two ways of defining the sets $Q_1, \dots, Q_n$. I think your mistake is that you think of the $Q_i$ as fixed and taken as input in these shampling schemes. It is not the case.
In external sampling, there is as many sets $Q_{\tau}$ as there are pure strategies for the opponent and the chance player (a pure strategy is a deterministic policy). Think of it as "the set of terminal nodes that I can reach if my opponent and chance play in this fixed way".
Sampling a set $Q_{\tau}$ thus means sampling a pure strategy for the opponent and chance node. An alternative is to sample on the fly the opponent's policy $\sigma^{-i}$ (and chance) when needed. It gives identical probabilities for q(z), while does not correspond to a full definition of a deterministic policy $\tau$ (which would need a definition on all the information nodes).
There are several implementations on Github, one can be found here: https://github.com/bakanaouji/cpp-cfr/blob/master/RegretMinimization/Trainer/Trainer.cpp


What does the figure "Blackjack Value Function..." from Sutton represent?

The left hand graphs are showing you the estimated value function from using Monte Carlo evaluation, after 10,000 episodes. They give a sense of what your value table will look like before convergence.

In the case of upper "usable ace" chart, the estimates are still showing a lot of inaccuracy due to variance in the data. This is for two main reasons:

The probability of getting a usable ace at the start is a fraction of all (around 15%), so the number of samples used to build the chart is lower.
There are more variations in play when there is a usable ace, due to the extra flexibility allowed by it, so the end result also varies more, requiring more samples to converge.

In addition, if you are looking at the bottom edge of the chart, this represents the player starting with two aces. If you pick on one of the high points (dealer shows 4), then that also reduces the probability of seeing that particular state. So you are looking at a sample size of typically 4-5, but maybe in this case just one sample or maybe two, which the player then happened to go on to win, even though the odds made it unlikely. There is always some chance of winning and dealer showing 4 is a bad start for the dealer, who has a good chance of going bust provided the player does not.

If it hadn't happened this time for the "two aces + dealer showing 4" state, it may have happened for "two aces + dealer showing 5" state. That's due to  the nature of random sampling - if you have hundreds of states to sample, then a few of them are going to behave like outliers purely by chance until you have taken enough samples.

In short, 10,000 randomly sampled games are nowhere near enough to reduce error bounds on the value estimates to reasonable numbers for special cases such as starting with two aces. However, you can see in the 10,000 samples charts the beginnings of convergence, especially elsewhere in the chart.

  From the graph, for 10000 episodes what i see is that when we don't have a usable ace we always lose the game except if the sum is 20 or 21.

Actually you don't see that, the expected result is not quite -1.0, but a little higher. So that means there is still a chance to win. Under this player policy, the worst chance is no usable ace and score 19, because the policy will be to "hit" and need an Ace or 2 card just to stay in the game. Even then , the value is not quite as low as -1.0, but more like -0.9


What is the difference between an agent function and an agent program?

It looks as if 'function' is being used here in the mathematical (or functional programming) sense of 'pure function', i.e. it is without state or side-effects. Hence the function cannot store previous percepts anywhere, so the entire historical percept sequence is considered to be passed to the function each time.

In contrast, the notion of 'program' appears to allow state/side-effects, so it is assumed that earlier percepts are memoized as needed (or that they otherwise updated the variables used within the program).

The 'function' notion is the conceptually cleaner one, in that the 'program' version can always be abstracted to the functional one. Which aspects of percept history happen to be cached by the 'program' version is merely an implementation detail.


Is it feasible to use GAN for high-quality image synthesis other than human faces?

Generative Adversarial Networks, basically boil down to a combination of a generic Generator and a Discriminator trying to beat each other, so that the generator tries to generate much better images (usually from noise) and discriminator becomes much better at classification. So, no it is not just suited for only synthesis high quality human face synthesis but any image type.

In fact, not only can it be used for any high quality image synthesis, it can work on non-image data types as well (such as text, etc). It all depends on the type of neural network you are using for the discriminator and generator at the end of the day.

http://openaccess.thecvf.com/content_iccv_2017/html/Osokin_GANs_for_Biological_ICCV_2017_paper.html

The above refers to a paper synthesizing cell image through GANs, as I haven't personally used GANs on a practical level.

General blog explaining GANs:
https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/

Human faces are much more frequently tackled, for many reasons, usually that human faces are highly symmetric and have a lot of different features, usually more than other types of images, with the added difficulty of being that we as actual humans are usually good at recognizing faces - making a neural net to fool ourselves, makes it a challenging area of research.

Hope it helped! Do let me know if I'm wrong somewhere.


How to classify when the label is seldom known

Domain knowledge and cluster interpretation
When you have no (or very limited) gold standard labels for your dataset, traditional performance metrics that require knowing how often you're correct (like accuracy, sensitivity, or specificity) simply won't work. If this is the case, you'll need to examine your outliers to see if they make sense in the context of your particular field. You should look at what features are responsible for a sample being classified as an anomaly, for example, you might see that agents classified as spam send a huge number of emails per hour, or that a suspected money laundering bank account interacts frequently with many offshore accounts. In most cases, you will never know with absolute certainty that your classification is correct, but if you can justify the classification in terms of what is known about your particular domain, it can lend credence to a model.
Another approach is to perform unsupervised clustering on your dataset. Hopefully, you should see that your method identifies anomalies which are somehow different from other datapoints, appearing as outliers or a distinct group of samples unlike the others. You will need to do more investigation to be confident that the clusters represent normal people and fraudsters, but clustering can help you identify that there are indeed some samples that appear markedly different from the others. If you can show that certain samples show similar hallmarks of illegal activity, it also helps to gain confidence in what your model is doing.


“Outside-in” versus “Inside-out” machine learning

It sounds like you are interested in the ideas of intrinsic motivation and attention in the context of machine learning. These are big topics, and the subject of much active research.

Intrinsic motivation says that the key to identifying interesting patterns and skills that are worth learning is to give the agent some intrinsic reason to learn to do new things. This is not dissimilar from what humans have: learning new things, and improving or exercising our capabilities to the fullest is what Aristotle identified as the good life. There are thus good reasons to think that intrinsic motivation for AI might solve the problem you identify. Current research in this domain is exploring different mathematical ways to represent intrinsic motivation.

Attention was the subject of a large burst of research in deep neural networks during the last few years. Here's a recent talk from AWS at ICML that provides a good overview. The idea here is that an agent can learn both a reasonable mapping from inputs to outputs for some problem, and a separate mapping that describes how different future inputs should "activate" certain parts of the input/output mapping that the agent has learned. Essentially attention-driven models include a second component that learns which features of the input to pay attention to when engaging in certain kinds of tasks.


Should the actor or actor-target model be used to make predictions after training is complete (DDPG)?

It does not matter, target should converge to the main network after long enough training.


This study from 2012 uses 3D convolutional neural networks (CNN) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning approach based on 3D CNN is demonstrated in the LIRIS and Orange Labs study from 2011.

This Oxford study from 2014 also uses a similar approach, but with two-stream CNN which incorporates spatial and temporal networks which can achieve good performance despite having limited training data. It recognises action from motion in the form of dense optical flow. For example:

Another study from 2007 demonstrates a method by detecting human falls based on a combination of motion history and human shape variation by analysing the video frames. It uses Motion History Image (MHI) to quantify the motion of the person.

Source: harishrithish7/Fall-Detection at GitHub

An alternative general approach could be action detection based on the posture using DNN. See: How to achieve recognition of postures and gestures?


How can a single sample represent the expectation in gradient temporal difference learning?

In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.


Output volume proof for convolutional neural network

You can think about the problem in the following way (without padding, as the padding case is a simple extension of base case with $\tilde{W}:=W + 2P$).
You want to know how many windows are necessary to cover an image of size $W$, given a window of size $K$ and stride $S$. So your image is a vector with indices $1, 2\dots, W$; as you put the first window on the image, the window will cover the indices from $1$ to $K$. As we apply stride (meaning that we translate the window), we will  get a sequence considering the last covered index. The first element of this sequence is $i_1=K$, where the $1$ indexing $i$ is not the number of times we apply stride but the number of covering windows we have. So in this first case applying no stride and we have 1 covering window. Applying stride once to our window, the window will cover the indices form $S$ to $K+S$, so $i_2=K+(2 -1 )S$. In general you get $i_n=K+(n-1)S$.
Now, if we can exactly cover $W$ then there is a number $N$ such that $i_N=W$. This means $$K+(N-1)S=W,$$ which rearranging gives $$N=\frac{W-K}{S} + 1.$$


It is explained in this CrossValidated post.

Top1 accuracy means the best guess (class with highest probability) is the correct result 58.9% of the time, while top5 accuracy means the correct result is in the top 5 best guesses (5 classes with highest probabilities) 87.7% of the time.


What are examples of techniques to prevent bias in artificial intelligence systems?

It's important to note that ultimately, the statistical methods we currently use in ML research are just that: statistical methods. So when they show some "bad behaviour" it's not because of problems with the statistical methods, but with the data we give it. But if the data we give it are as "genuine and unfiltered" as it gets, then it probably shows something about us.

From a cognitive science perspective, its probably the case that the same heuristics and biases that creates stereotypes are also the ones that make us powerful agents(note the similarity between categories and stereotypes), so at least at this moment its unclear how we can segregate desired from undesired behaviour.

To combine the points mode above, it seems we can only either: 1. Remove "bad content" by curating the data by hand or by some metric that we don't know of yet 2. Accept that our methods will produce AI as "bad as we are" cause that's what we are, and let it operate under the knowledge that it might produce undesired behavior sometimes.

Unless we have some crazy new theory of mind that we can begin to analyze this in a more rigour manner, it seems like there is no clear cut solution.


Temporal Depth is a third parameter of a time series data. For example if you have a video clip of length 25 frames and on training a model you are giving first five frames with respect to time. Your temporal depth will be 5.


What does 'democratizing AI' exactly mean?

In this particular context, "Democratize" means to make more accessible to people.

Thus, "Democratizing AI" means to make AI softwares and AI programming available, accessible and easy to use for the vast majority of people.


How to apply Q-learning when rewards is only available at the last state?

Having only a non-zero reward at the very end is not uncommon. When rewards are sparse, it becomes a bit harder to learn compared to having lots of different rewards along the way, but for your problem, the goal state is always reached, so that should not be a problem. (The real problem with sparse rewards is that, if an agent can do a lot of exploration without every finding the goal, it essentially receives no feedback and will behave randomly, until it happens to stumble upon the very rare reward state.)
What concerns me more about your problem is that the final reward depends not just on the last state visited, but also on the chain of actions taken so far. That means that, to make this a proper MDP, you need to keep the chain of action in the state. So, your state would be something of the type $(s_k, [a_1, a_2, \ldots, a_{k-1}])$.
This kind of combinatorial problem is not what RL is really great at. RL is really good when the state and action together give a lot of information about the next state. Here it seems that, in your formulation, the next state is independent of the previous action.
Instead of seeing this as a RL problem, you might want to express this as sequences of actions with an associated reward, and look at it as a combinatorial optimization problem.


What is the difference between Sutton's and Levine's REINFORCE algorithm?

About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$.

About the second question, the Policy Gradient theorem only tells you what is the gradient up to a constant, so basically any constant is irrelevant. Now, even if you do know the constant, you are going to multiply the gradients by an arbitrary learning rate $\alpha$. So, you can think that the factor $\frac{1}{N}$ is actually already considered "inside" $\alpha$.


Here a convolutional neural network (cNN) based approach is presented:
Image Crowd Counting Using Convolutional Neural Network and Markov Random Field

This blog post has more of a tutorial character and also present an approach based on CNNs
Counting Crowds and Lines


Monte Carlo method is an approach where you generate a large number of random values or simulations and form some sort of conlusions based on the general patterns, such as the means and variances.

As an example, you could use it for weather forecasts. Predicting long-term weather is quite difficult, because it is a chaotic system where small changes can lead to very different results. Using Monte Carlo methods, you could run a large number of simulations, each with slightly different atmospheric changes. Then you can analyze the results and for example calculate the probability of rain on a given day based on how many simulations ended up with rain.

As for the use of Monte Carlo in Alpha Go, they seem to be using the so-called Monte Carlo Tree Search. In this approach, you make a tree of possible moves, a few turns into the future, and try to find the best sequence. However, since the number of possible moves in the game of go is very large, you won't be able to explore very far ahead. This means that some of the moves which look good now might turn out to be bad later.

So, in the Monte Carlo Tree Search, you pick a promising sequence of moves and run one or more simulations of how the game might proceed from that point. Then you can use the results of that simulation to get a better idea of how good that specific sequence of moves really is and you update the tree accordingly. Repeat as needed until you find a good move.

If you want more information or to look at some illustrations, I found an interesting paper on the topic: C. Browne et al., A Survey of Monte Carlo Tree Search Methods (open repository / permanent link (paywalled))


Are CRUD webapps today the modern version of the "expert system"?

The key feature of an expert system is that the knowledge base is structured to be traversed by the inference engine.  Web sites like Stack Exchange don't really use an inference engine; they do full-text searches on minimally-structured data.  A real inference engine would be able to answer novel queries by putting together answers to existing questions; Stack Exchange sites can't even tell if a question is duplicate without human confirmation.


Which functions can be activation functions?

The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.
For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a tanh (hyperbolic tangent) or sigmoid function in the last layer
For classification, you want to obtain probabilities: use a softmax function in the last layer.
For the hidden layers some functions are better than others :

The gradient should be fast to compute (from the perspective of your computer).

If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.


Variations of the squared error function

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h_{\theta}(x^{(i)}) \right)^2,$$

where:

$x^{(i)}$ denotes the $i$th training example
$h_{\theta}(x^{(i)})$ denotes the model's output for that instance/example
$y^{(i)}$ denotes the ground truth / target / label for that instance
$m$ denotes the number of training examples

Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( h_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$

Now it already looks quite a lot like your second variation.

The second variation does still have a $\frac{1}{m}$ terms outside the sum. That is because your second variation computes the mean squared error over all the training examples, rather than the total error computed by the first variation.

Either error can be used for training. I'd personally lean towards using the mean error rather than the total error, mainly because the scale of the mean error is independent of the batch size $m$, whereas the scale of the total error is proportional to the batch size used for training. Either option is valid, but they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale.

With that $\frac{1}{m}$ term explained, the only remaining difference is the $\frac{1}{2}$ term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given in the page you linked to for the first variation:

  The $\frac{1}{2}$ is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.


What is the difference between reinforcement learning and optimal control?

As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and answers here (as well as the OP), the line between RL and optimal control can be quite blurry.
Consider the Linear-Quadratic-Gaussian (LQG) algorithm, which is generally considered to be an optimal control method. Here a controller is computed given a stochastic model of the environment and a cost function.
Now, consider AlphaZero, which is obviously thought of as an RL algorithm. AlphaZero learns a value function (and thus a policy/controller) in a perfect information setting with a known deterministic model.
So, it's not the stochasticity that separates RL from optimal control, as some people believe. It's also not the presence of a known model. I argue that the difference between RL and optimal control comes from the generality of the algorithms.
For instance, generally, when applying LQG and other optimal control algorithms, you have a specific environment in mind and the big challenge is modeling the environment and the reward function to achieve the desired behavior. In RL, on the other hand, the environment is generally thought of as a sort of black box. While in the case of AlphaZero the model of the environment is known, the reward function itself was not designed specifically for the game of chess (for instance, it's +1 for a win and -1 for a loss, regardless of chess, go, etc.). Furthermore, the neat thing with AlphaZero is that we can use it to train agents in virtually any perfect information game without changing the algorithm at all. Another difference with RL here is that the agent iteratively improves itself, while optimal control algorithms learn controllers offline and then stay fixed.