In-between your input and desired output, there's obviously a huge space to search. The more relevant domain information you include as features, the higher chance that the Deep Learning (DL) algorithm can find the desired mapping.

At this early stage in DL research, there aren't so many rules of thumb to tell you what features to explicitly encode - not least because it depends on the size of your training corpus. My suggestion would be: obtain (or generate) a large corpus of C code, train on that with the most naive feature representation that you think might work, then repeatedly gather data and add more feature preprocessing as necessary.

This following paper describes a DL approach to what is almost the 'reverse problem' to yours - generating the source code for a program described in natural language.

I found the strength of the results reported in this paper surprising, but it does give me some hope that what you are asking might be possible.


Using convnet to classify language of text contained in images

This sounds like a fairly straightforward task, with low risks. I think the proper term is that you're you are trying to detect the script, which would be either Latin ( "English") or Hangul ("Korean"). The chance is that you end up learning fonts, though.


Who stands a chance against Google in the AI race?

Pretty much any "kid in a garage" so to speak. Nobody knows how to build AI yet and a major breakthrough could come from anywhere. That could knock Google off their perch easily enough.


What are the challenges for recognising the handwritten characters?

Handwritten digit recognition is a standard benchmark in Machine Learning in the form of the MNIST dataset. For example, scikit-learn, a python package for Machine Learning uses it as a tutorial example.

The paper you cite uses this standard task as a proof of concept, to show that their system works.


How do the achievements met in the gaming field (ex. AlphaGo Zero) impact other fields of application?

Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time Difference loss function was that it was essentially training on itself, that is data produced by the same method was used as part of regression target. That was producing the problem of "extrapolation error" - solution would blow up, or oscillate wildly. There was attempts to mitigate that problem (n-steps algorithms), but they weren't helping much. Alpha Zero combined Deep Network with tree search instead (Monte Carlo Tree Search). Tree search algorithm produced wide and long fields of high precision data (value function), with network influence on value much diminished. That way network was training mostly not on itself, but on data produced by tree, and tree search itself was accelerated greatly by network (using it as heuristics). The whole happens to be much more then sum of parts.

This approach is not limited to broad game or RL theory. It may work for any problem for which high-precision simulator could be built. Essentially if problem allow Monte Carlo Tree Search, or other tree search which could be augmented with heuristic, Alpha Zero approach would probably work on it. Of cause Alpha Zero approach is computationally expensive, so it's not always efficient to apply it.


How to manage high numbers of input layer data points

Training with images makes use of pictures that are very large, like 7360×4912 (36 mega pixels). For image recognition at any angle and size, an image is rotated (about 60 times) and resized (depends on the original size, possibly 20 times) many times. This means you end up with a ton of data. In my example here, you would get some 60 × 36 × 2 = 4Gb of data for that one image. Although you really only save the data of interest (noise in the light signal) so in the end the AI does not use as much, but it would still be a quite large data set... especially when you know that a well trained image recognition system uses a good 2,000 images.

So 10 to 20 thousands data points isn't much at all. I guess it depends how many of those you have (one per second would be 86,400 a day...)

I would consider using a system like Cassandra because the database can grow as big as you'd like, if that is one of the problem you're facing. Also you can save the data in binary and the number of columns can vary on each row. However, Cassandra expects a key for each row. This is an important point as it makes use of that key to distribute the data over multiple computers (for faster access later and distribution of the data set over all computers.) It also includes replication "for free" so if a computer fails, you can just replace it (no need to restore it).

What is not a very good idea with Cassandra is creating many indexes. Those use rows (i.e. one index = one row) and thus the few computers handling that one row will work extra-hard on queries on that index... But I would imagine that for AI you would not really need such.

A really large CSV will be difficult to work with because you'll be responsible for accessing the rows (unless you have to access all the rows each time anyway?) Also the file format is Text and the file size will be limited by your OS (which should be really large, but be careful as many I/O calls in higher level libraries miss the fact that the size could be 64 bits.)

If you are looking at a longer/more permanent solution, I would definitely look into Cassandra, especially if you're using Java as a language (also I use it with C++ and it works just fine with C++ too.) If you're just writing a throw away app. anyway, then CSV is probably much easier than handling a whole Cassandra cluster! That's a learning curve...


Is it possible to create a named entity recognition system without using POS tagging in the corpus?

Yes, both of them are different tasks. POS tagging helps NER systems but it is not necessary. You can get features (say BERT/ELMo embedding) for each word in the sentence and train a CRF NER model. This looks like simple example https://www.pragnakalp.com/bert-named-entity-recognition-ner-tutorial-demo/


Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).

  The Q function, however, measures the the value
  of choosing a particular action when in this state. The advantage
  function subtracts the value of the state from the Q
  function to obtain a relative measure of the importance of
  each action.

  Dueling Network Architectures for Deep Reinforcement Learning

To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:

Use a larger batch size and play several steps before updating
Play and train in a separate process
Use asyncronous cuda transfers

RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers.
It includes:

DQN
Double DQN
Dueling DQN
Prioritized Experience Replay
Noise Networks for Exploration
Distributional RL
Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
Distributional RL with Quantile Regression
Hierarchical Deep RL


Do Le et al. (2012) train all three autoencoder layers at a time, or just one?

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.

This also agrees with the fact that only the autoencoder layer has tunable parameters.  The other two layers use uniform weights.

I do not have a good answer for the follow-on question yet... would love to hear what others think!


Answering my own question here.

Looked at the Open Image Dataset by Google @ https://storage.googleapis.com/openimages/web/index.html

They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships.


Are there existing examples of using neural networks for static code analysis?

There's certainly literature on a related topic: code smell detection.

A "code smell" is a sign that code has a maintenance problem, and hints at the presence of technical debt. It is reasonable to suppose that code with a lot of smells is lower quality. Code smells include things like giant classes, high cyclomatic complexity, and more.

Fontana et al. have a good 2016 survey comparing different ML methods for detecting code smells. A reverse citation search on that paper uncovers many other papers that seem relevant, including:

Rasool et al. 2015, A review of code smell mining techniques
da Silva Maldonado et al. 2017 Using Natural Language Processing to Automatically Detect Self-Admitted Technical Debt
Maneerat & Muenchaisri, 2011, Bad-smell prediction from software design model using machine learning techniques
Mansoor et al. 2017, Multi-objective code-smells detection using good and bad design examples

This seems like a pretty well-studied area. I wasn't able to find a cross-language model though, but I suspect one may well exist.


Why is informed search more efficient than uninformed search?

There are several informed and uninformed search algorithms. They do not all have the same time and space complexity (which also depends on the specific implementation). I could come up with an informed search algorithm that is highly inefficient in terms of time or space complexity. So, in general, informed search algorithm are not more efficient than uninformed ones, in terms of space and time complexity.

However, given that informed search algorithms use "domain knowledge" (that is, an heuristic function that estimates the distance to the goal nodes), in practice, they tend to find the goal node more rapidly, given a more informed "heuristic" (which needs to be admissible in order to find the optimal solution). For example, in theory, A* has exponential time and space complexities (with respect to the branching factor and the depth of the tree), but, in practice, it tends to perform decently well. It tends to have an effective branching factor (that is, the branching factor for specific problem instances) quite "small" (for several problems).

What is a more informed heuristic? Intuitively, it is an heuristic that more rapidly focuses on the promising parts of the search space. Let's denoted by $h$ the heuristic function. If $h(n)=0$, for all nodes $n$, then this is an admissible heuristic, because it always underestimates the distance to the goal (that is, it always returns $0$). However, it is a quite uninformed heuristic: either if you are at the start or goal nodes, the estimation is always the same (so you cannot distinguish the start and goal nodes, in terms of estimates). Given two admissible heuristics $h_1$ and $h_2$, $h_2$ is more informed than $h_1$ if $h_1(n) \leq h_2(n)$, for all nodes $n$.

An uninformed search algorithm performs an exhaustive search. There are several ways of performing such exhaustive search (e.g. breadth-first or depth-first), which are more efficient than others (depending on the search space or problem). Given that they perform an exhaustive search, they tend to explore "uninteresting" parts of the search space. Hence, in practice, they might be more inefficient than informed search algorithms (that is, they might require more time to find the solution).


How to correctly label images for multi-label classification?

You can't label things you don't know.  The goal of labeling is to label the things you want the classifier to learn so that when you run it in inference mode you can discover what is in your data (new data that you didn't use for training, validating, or testing).

It is not a good idea to label small objects like the 'blue water' unless it is important to you to discover these fine details in inference mode.


How do neural networks manage to do regression?

I came across the article A visual proof that neural nets can compute any function that explains how a neural network can approximate any function, which pretty much answers my question.

I suppose connections between neurons do not generally arrange themselves in the way described, though, so it would still be interesting to get insight into whether there are patterns of how the inter-neuron connections arrange themselves in neural networks trained to solve actual problems.


Deep Q-Learning: why don't we use mini-batches during experience reply?

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and accuracy of the gradients.

In your first link, and the code excerpt, the sample list is literally called minibatch. However, the developer then goes on to make a series of individual SGD steps per item within that sample.

  Why can't they use mini batches instead?

They can, and it works just fine. In my DQN explorations I tend to use a single mini-batch per timestep, and the Adam optimiser to accelerate learning. I also use a mini-batch to calculate TD target values.

  I'm new to RL, but in deep learning people tends to use mini-batches as they would result in a more stable gradient. Doesn't the same principle apply to RL problems?

It does, but I don't know if this is the most sample efficient that it could be. I prefer to use mini-batches and more complex optimisers because this runs faster (for a given number of replay training samples per time step), and I can run more time steps of the environment in less time.

It is possible for instance that a replay sample size of 32 per time step, with 4 mini-batch updates of 8 each would be the most sample efficient approach for a specific problem (because it makes more updates from the given samples), but it might not be the most CPU efficient.

  Is the randomness/noise introduced actually beneficial to the learning process?

I don't think so, but there are plenty of other tuning issues to consider in RL.

  Am I missing something, or are these sources all wrong?

They are not necessarily wrong, but may be CPU inefficient.

More likely in my opinion, your sources are explaining algorithms in principle, and did not want to add too many layers of optimisation, or too many components that may need tuning to work. There is a lot going on in a typical DQN agent, and having many performance optimisations can hide the details that the person is trying to teach.


What's a good resource for getting familiar with reinforcement learning?

To the good answers here, I would add

A brief overview of RL: Most essential concepts in one place.
Another brief overview, in presentation format.
Ben Recht's An outsider's tour of RL is pretty comprehensive and accessible.
The Bellman equations: central to the whole RL theory.
Policy gradients explained by Andrej Karpathy (mentioned in other answers as "pong from pixels", this is the link).

These barely scratch the surface of RL, but they should get you started.


The analogy is to a federal system of government. In a federation, smaller pieces follow the direction of a higher piece. In federated machine learning, you give your data for processing to the higher machine. The federation in this analogy is a collection of smaller computers. The central computer breaks up your data and gives portions of it to each smaller computer. When those computers are done they return the results and the central computer reassembles them into a single model.
The main idea is distributed processing. Some benefits are:

Cost: it may be cheaper to operate multiple inexpensive computers rather than fewer more expensive computers.
Privacy: if this is sensitive data like healthcare records then perhaps you don't want all of one person's data in a single place where the wrong person can grab it.


Why not go another layer deeper with Auto-AutoML?

Logically it is possible, but you will just end up complicating the entire task.

The aim of AutoML is to provide a drop in solution to the customers. To do this, a trained network decides and generates the model architecture. This is done so that anyone with basic experience is able to integrate the solution into their systems.

Currently, the complicated architectures and networks require experienced data scientist to build, train and deploy. To overcome this bottleneck and make ML accessible to all, AutoML is being developed.

So adding another grand-parent network to optimize the autoML network will just complicate the task in terms to computation time and hyper parameter optimization.

In case we decide to add another network, now the researchers must look at this network and tune it in regards to the inner network and the model both. This means more work and no direct way to understand how the hyper parameters are affecting the final results.


Framework for Joining Multiple Modular Artificial Neural Networks

In recent years the focus has been on layers rather than the more biologically inspired individual nodes.  As stated in the comment by thecomplexitytheorist you could use a computational graph, although then you have issues with distribution and you're limited to one framework.

I created something in my PhD about the same time as the thesis you reference that was a simulation of biological structure and connected in a way not dissimilar to figure b above using ML to determine the weights of all the connections.  The control module of the network was distributed across several neurons, but allowed asymmetric connections between the inputs and outputs.  This was a large effort, and similar results could now be achieved more quickly using the modern frameworks.  This is, I believe, the main reason that biologically inspired neural networks are not as actively researched.  I still believe that there are tricks in biology that we've not encapsulated and would love the head space to create a framework to test this hypothesis!

Multiple Neural Networks are the simplest way to go and I've created several of these over the years (sadly not open source).  These can be distributed and use different frameworks - just wrap up each of your networks in an API that has common inputs and outputs and you can daisy chain them without worrying about the internals.  This would give you the power and flexibility you need and allow you to swap in and out multiple models.  I know I've made that sound simple but at this stage it's more a software engineering problem.  You do need to be wary of compound errors between your networks if you're trickling down as it's basically a decision tree of NNs at that point.


Will computers be able to understand user emotions?

The term Understand is multidimensional, so characterizing the degree of understanding — emotional or otherwise — is a slippery task.  Nonetheless, some forms of AI emotional understanding are possible.

An Interesting Simple Case

Even the embedded programs of velocity sensitive musical keyboards are emotionally aware at a primitive level.  Using the key depression velocity to drive the note parameters of attack, timbre, and volume allows the accomplished user's emotions to pass through the software into the emotional content of the music, which can be experienced by listeners just as with instruments that are not digital in nature.

More Advanced Emotional Capabilities

At the other extreme is a wide array of human emotional capabilities.  One example is the set of complex recognition functions underneath the typical listening skill set of a counselling professional.

The term affective computing was mentioned in the question.  A counselor may note the affect of the counselled, reading tone of voice, facial expression sequences, and body language.  From this, a clearer picture of the internal emotional state of the counselled can be understood.  These recognition and analysis abilities can determine the subject's internal emotional state more accurately and comprehensively than a masterful analysis of a functional MRI or the detection of neuro-chemical metabolites in blood.

There seems to be no theoretical principle that limits computers from mastering the front end of a counselor's skill set, the recognition of emotion to produce a coded sequence of affects.  The degree to which software can interpret the sequence of affects (in combination with concurrent natural language expressions) and determine the emotional condition (and potential market-oriented decision patterns) of the human is yet to be determined.

There are other emotional capabilities beyond the skills of emotional recognition and interpretation.  Compassion is largely the logical integration of ethics in the absence of self-centered motive with the results of emotional recognition and interpretation.  The addition of emotional imitation yields empathy.

The development of such capabilities in automata is more important than the shallow and therefore myopic motives of e-Commerce.  There may be applications for toys, education, entertainment, and remedy for the growing coldness of a technological society.  (Such is discussed further below.)

Functions of recognition, interpretation, the application of rules or meta-rules in the ethical domain to these interpretations, and imitation may well be within the capabilities of digital systems.

Isaac Asimov introduced the possibility of telepathy to the feature set of both the human mind and automata.  It may seem like telepathy is confined to the domain of fiction, however an emoticon can be the tele-form of a component of affect, such as a smile.  In this way, consensual technology assisted telepathy has actually become commonplace.

With the erosion of privacy in a culture with increasing interaction with digital systems by an increasing segment of the public, less consensual telepathic techniques may be developed and mastered.

Deeper Aspects of the Question

The doubt lies in the existence of a soul or some other autonomous and non-deterministic aspect of the human person.  The existence of these things and whether the simulation of them can be accomplished with a Turing machine and practically deployed to the Von Neumann architecture or a collection of them have neither been proven or dis-proven formally.  If autonomous and non-deterministic elements exist in people, then, even if they can be simulated, we cannot infur they can be realized as independently autonomous.

Challenging the implications of the MIT coined term Meat Machine, the notion that humans are capable of intention by fiat is not disproven, yet a pure Turing machine clearly cannot intend in that way.

This is important in the context of the question in that a computer may only be able to simulate frustration when an intention is thwarted by the conditions of life.  Actual frustration may not be possible.

Current Publicly Experienced Progress

One interesting entry point of computer science into the domain of human emotions is the story based memory and reasoning model purported by Roger Schank of Yale U in the 1990s.  Although the machinery used by Amazon underneath the hold of its ever improving ability to recommend movies in a buyer specific way is Company Confidential to Amazon, one must wonder.

Can e-Commerce develop user profiling regarding books and movies to comprehend the story plot or the arc of protagonist's emotional development? Can it do so sufficiently to match user viewing and purchase with other products?  Is that what Amazon is beginning to do?  Short of an unethical Amazon employee, we can only guess.  Certainly, what is transpiring in leading edge e-Commerce has moved beyond word or phrase matching or tracking interest in particular authors, screenwriters, directors, producers, and stars.

If Roger Schank's story based memory and reasoning has penetrated into e-Commerse, emotional analysis of individuals of within public is well underway, since story plot and protagonist developmental arcs are bound in climaxes and setting up climaxes, all related to emotional states.  If not, emotional analysis of groups within the public has been underway since public relations functions entered into the roles of enterprise IT systems.  (I know for a fact that such occurred decades ago.)

Even if the likes of Facebook, Google, and Amazon are attempting to approximate in various ways the emotional states of individuals in real time, the matching of movie plots of protagonist arcs and such pattern matching and naive categorization is far from what a good counselor or a self-actualized friend or family member might do.

Computer systems to which the public is exposed do not yet appear to accurately developing comprehension of the buyer's emotion.  That is one area where even the most progressive interfaces are still shy.  Furthermore, the computer interface itself is still as dry as it was in the late 20th century.

Short of breaches of confidentiality, what capabilities to recognize, compare, analyze, or simulate emotions may exist in proprietary labs is solely a mater of conjecture.

Warm Interfaces

Returning to the question of the general coldness of cybernetic interfaces in a technological society, there is much for the computer science community to learn.  For one thing, the soft skills of actual technical people often leaves much to be desired, so a cultural shift from cold development teams to culturally progressive and warm teams might be necessary.

Furthermore, one who has engaged with various PaaS (personality as a service) devices such as Siri, Google Assistant, Alexa, Cortana, and Microsoft's Office Assistant, Clippy, might find the paperclip personality of Clippy to be the warmer from among the list.  Perhaps the Clippy didn't see a higher level of acceptance because of the disparity between the coldness of a grid of cells in a spreadsheet and the attempted warmth of the animated character was too much, and the little guy tends to get in the way of some portion of the interface, which can be annoying.  (Integration of concepts was poor.)

Nonetheless, the animated style of Clippy is intended to create a mental association with cartoon characters, which is brilliant.  Cartoon character design and storytelling through comic and cartoon media is a creative area of technology that had mastered computer generated warmth long before intelligent assistants began to appear.

One can't help thinking, upon general survey of intelligent assistant history, that the more the system tries to be human, the less human it seems.  Human response to emotional content in a cybernetic interface is like that of color matching or synthesized musical instruments.  If one is attempting to synthesize, the synthesis has to be good.  If it can't be fully convincing, then there is more authenticity in inventing something that is not pretending to be something real.  A caricature of real life objects (like Sponge Bob or Road Runner) avoids exposing an unacceptable margin of error in the imitation of real life. [1]

Our sensitivity in this area is so acute that if a real person, working in a customer service department, acts too mechanically such that we have even the slightest doubt as to whether we are talking to a human being, the impression is negative and we want to hang up and look for the function we wish to initiate on the company's web site.

It is unclear whether the trend for people to become more like cogs in a machine or whether usage patterns will drive companies to design their user interfaces and the mechanisms behind them to exude something like hospitality or congeniality.  Although Jaques Ellul seems to be correct in his proposition that technology has long since been driving humanity rather than the other way around, the DNA nature of the human mind will perhaps force humanity to remain remarkably human.  Thus the evolution of cybernetic interface professionals may continue to seek the meaning of and simulation of warmth.

The problematic, controvertible, and somewhat bizarre pursuit of Turing Testing will probably endure.

Concerns About Emotional Modelling Uses

Whether an emotional recognition, modelling, analysis, and comparison suite, obviously a powerful tool-set, is used for the betterment of humanity or its detriment is a question like that of any other powerful tool-set.  Mass psychology, nuclear science, statistics, and genetics have in common the potential for great good or great evil.

In the context of this question, broadcast or more targeted information dissemination can be characterized as propaganda, which may not necessarily be perilous.  Propaganda was used for women's suffrage and along the path of U.S. independence from Europe.  Elements of broadcast and targeted control have created much geopolitical trouble, a topic outside of the scope of this social network, but pertinent and somewhat obvious, such that further mention is unnecessary anyway.

Individual profiling can be of great good too.  Such can be like a combination of the old library subject indexed Dewey Decimal cards in combination with a knowledgeable librarian when looking for information or items of entertainment or personal expression for borrowing, rental, or purchase.

Of course, profiling in combination with statistics can lead to the ability to obtain covert control over a mass of people, not just for the profit maximization of a company.  An appropriately self-deluded megalomaniac or a collection of them in some kind of secret society could use emotional modelling and comprehensive user profiling for a large sample of national or global population for the injection of emotionally powered systems of belief.  Such a person or group could theoretically succeed in covert and systematic sublimation of liberty to a kind of virtual authoritarian dictatorship.

How technology is used is always a concern.  Transparency and accountability are required features of a society.

Just as in the other areas of science and technology mentioned above, it is up to we who are skilled enough to realize these systems to be intelligent and active elements within our society's accountability mechanisms.  Whether a corporation or governmental department is leading humanity into perdition is an item that can be judged during a job hunt and during the initial work period.

Notes

[1]

This is perhaps the strongest critique of the Turing's Imitation Game philosophy.  Cartoons, comics, and sci-fi creatures demonstrate that people don't seem to care if intelligent beings can be distinguished from humans, and there is no proof that human intelligence is the only or the best example of intelligence.

Along these same lines, heroes are representations of intelligent behavior that epitomize some aspect of humanity that actually doesn't naturally occur in pure form as they do with these protagonists.  Yet actors are branded and marketed as stars because of this idyllic expression of character.  It may be that humans don't really like human intelligence and constantly try to transcend it through fiction, but results in only some public awareness that is never more than an anthropological fantasy.

Even AI is not an imitation game.  It is a transcendence game.  For this reason, the Turing Test is of limited value.


Are there any disadvantages to using a variable population size in genetic algorithms?

Population size is a tricky issue even in pure biological models. Biological population sizes obviously vary. The two great protagonists of the argument were Ronald Fisher and Sewell Wright, with argument being between Fisher favouring few large populations against Wright’s many small interconnected populations. There is evidence that evolution occurs more rapidly in Wright’s model but the evidence is inconclusive. The theory concentrates on the probability that a mutation will occur and then become dominant in a population. In a small population a beneficial mutation is more likely to be selected for reproduction, but premature convergence is a serious danger.  While in a larger population a mutant is less likely to be removed from the population during reproduction. I would strongly recommend a read of Games of life by Karl Sigmund.


What are the real applications of hierarchical temporal memory?

The following figure [1] is inspiring:

It means HTM can be used as a memory unit in different neural networks such as RNN. In the same reference, we can find the following advantages and applications of HTM:

  Sequence learning:
  Being able to model temporally correlated patterns represents a key property of intelligence, as it gives both biological and artificial systems the essential ability to predict the future. It answers the basic question “what will happen next?” based on what it has seen before. Every machine learning algorithm should be able to provide valuable predictions not just based on static spatial information but also grounding it in time.
  High-order predictions:
  Real-world sequences contain contextual dependencies that span multiple time steps, hence the ability to make high-order predictions becomes fundamental. The term “order” refers to Markov's order, specifically the minimum number of previous time steps the algorithm needs to consider in order to make accurate predictions. An ideal algorithm should learn the order automatically and efficiently.
  Multiple simultaneous predictions:
  For a given temporal context, there could be multiple possible future outcomes. With real-world data, it is often insufficient to only consider the single best prediction when information is ambiguous. A good sequence learning algorithm should be able to make multiple predictions simultaneously and evaluate the likelihood of each prediction online. This requires the algorithm to output a distribution of possible future outcomes.
  Continual Learning:
  Continuous data streams often have changing statistics. As a result, the algorithm needs to continuously learn from the data streams and rapidly adapt to changes. This property is important for processing continuous real-time perceptual streams, but has not been well studied in machine learning, especially without storing and reprocessing previously encountered data.
  Online learning:
  For real-time data streams, it is much more valuable if the algorithm can predict and learn new patterns on-the-fly without the need to store entire sequences or batch several sequences together as it normally happens when training gradient-based recurrent neural networks. The ideal sequence learning algorithm should be able to learn from one pattern at a time to improve efficiency and response time as the natural stream unfolds.
  Noise robustness and fault tolerance:
  Real world sequence learning deals with noisy data sources where sensor noise, data transmission errors and inherent device limitations frequently result in inaccurate or missing data. A good sequence learning algorithm should exhibit robustness to noise in the inputs.
  No hyperparameter tuning:
  Learning in the cortex is extremely robust for a wide range of problems. In contrast, most machine-learning algorithms require optimizing a set of hyperparameters for each task. It typically involves searching through a manually specified subset of the hyperparameter space, guided by performance metrics on a cross-validation dataset. Hyperparameter tuning presents a major challenge for applications that require a high degree of automation, like data stream mining. An ideal algorithm should have acceptable performance on a wide range of problems without any task-specific hyperparameter tuning.

One of the applications of HTM is in Anomaly detection. See, for example, this paper Unsupervised real-time anomaly detection for streaming data (2017) which used from a network of HTMs to solve this problem in streaming data.

The implementation of HTM (NuPIC) can help to see more applications of HTM in ML.


How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely

$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$?

Yes this is 100% correct.
There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $v_{\pi}$ and $q_{\pi}$ in that direction.
Note that for a deterministic policy, with $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ then the relationship is
$$v_\pi(s) = q_\pi(s, \pi(s))$$
The related exercise in the book - expressing $q_{\pi}$ in terms of $v_{\pi}$ and the MDP characteristics - is more complex because it involves a time step.


How to use pretrained checkpoints of BERT model on semantic text similarity task?

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d

You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one


Can neural networks modify their own weights without back-propagation and gradient descent?

Artificial neural networks (ANNs) do not modify their own weights! Humans create algorithms that modify the weights or architecture of ANNs.

Having said that, you can change the weights of ANNs using other methods other than gradient descent combined with back-propagation, such as Hebbian learning, evolutionary algorithms or reinforcement learning.


What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub.

In particular, 0 means no action, 1 means fire, which is why they don't have an effect on the racket.

Here's a better way:

env.unwrapped.get_action_meanings()


What effect does batch norm have on the gradient?

"Naturally, this will affect the gradient through the network." this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.

As the title of the paper suggest, the aim of batch normalisation is to decrease training time by reducing covariance shift. What is covariance shift? We can conceive it as the variation that can happen between between the values of two layers of a network. We are all familiar with the concept that if we have input features with different unite scale, like kilos and euros, most likely lot of values will have different magnitude orders, like thousands for the weight could appear often with hundred of thousands for the money. When applying the activation function to different order of magnitude this discrepancy will remain, causing the values in the first layer to assume a really broad range of values. This is not good, since high fluctuation means more time to converge to stable values. This is why values fed to neural nets are always standardised.

The authors applied the same logic to the hidden layers, arguing that a deep neural net can be conceived as a repetition of itself (every hidden layer is an input layer that send features to another hidden layer) ergo the features should be normalised at every layer. How can we do it? Normalising every batch is the most natural way to procede, but in this way there is the risk to end up transforming the internal representation of a layer, because normalising is not a linear transformation. This is way the authors propose a clever way to perform hidden layers normalisation, which consists in a classic normalisation followed by a linear scaling performed with two trainable parameters $\beta$ and $\gamma$ (which appear in the last step of the batch norm scratch code below).

A really important thing to notice is that the mean and variance used to perform the classic normalisation are mean and variance calculated on the mini batch. I will explain why this is important in a sec, first I want to stress out that the $\beta$ parameter can actually bring to increase overfitting when batch norm is randomly stucked on top of other layer. The reason why is that, as we see in the scale and shift step, $\beta$ is nothing but a biased term, added to perform the shift of the mean of the batch hidden values. So, to avoid having an extra bias that lead to overfit, the bays on the previous layer should be removed, leaving only the classic matrix of weights parameters.

Back to the gradient problem, we can see that in itself doesn't necessarily lead to increased performances, but it does provide an advantage in terms of hidden layer values convergence. The x axis on the two right sub plots of the figure below represent the variation of the hidden values of net trained with and without batch norm. When trained with batch norm the hidden values reach stable ranges after few iteration. This help the net to reach high accuracy in less iterations (first subplot on the left) but we can see that even without batch norm the net reach eventually high accuracy.

The only help provided by batch norm to the gradient is the fact that, as noticed before, the normalisation is firstly performed by calculating the mean and variance on individual batches. This is important because this partial estimation of mean and variance introduces noice. Similarly to drop out, which has a regularisation effect due to the noice generated by randomly deactivating part of the weights, batch norm can introduce regularisation by adding noice due to the larger or smaller mean and variance estimated on individual batches. But still, batch norm was not introduced as a regularisation technique, and the equation you put on the question simply prove that it is possible to calculate the derivatives of the equations applied to perform the batch norm.


Neural Network with varying inputs (for a game ai)

The most generic approach is to input all pixels as you have suggested. A CNN would be the best architecture for that. To provide information like speed or velocity, you can feed more than one frame to the CNN (e.g. the last 5 frames or whatever provides enough information). The CNN can learn movement information by comparing those images.

If you want to store additional information (like an inventory item), an input neuron for each value would be an option. You can also look up LSTM (long-short term memory) models, but for your specific situation a hardcoded neuron would be the easier solution.


What is the difference between genetic algorithms and evolutionary game theory algorithms?

Philip's answer is good, but I'll add to it.

In a GA, a population of individuals (typically represented by bit strings) is evaluated for its fitness on a particular task. Each individual is evaluated separately by a fitness function than can determine its quality. In the Traveling Salesman Problem, the bit string might represent a sequence of numbers, for instance, corresponding to an order in which cities are visited during the tour. The fitness function would inspect a single individual, compute the total cost of the tour, and assign that individual a fitness based on that value. Low scoring individuals are removed, high scoring individuals generate variants on themselves, and the process repeats.

In Evolutionary Game Theory, a population of individuals is also evaluated on some task, but usually the task involves interaction between the individuals. For example, you could use an EGT simulation to study what happens in a game like Iterated Prisoner's Dilemma. Here, an individual's fitness doesn't just depend on the rules of the task, but on the behaviors and strategies of the other players in the population. A strategy that is highly effective at first (like always cooperate) will quickly die out once defectors appear. Defectors are effective as long as there are some cooperators to prey upon, but are quickly defeated by strategies like Tit-for-Tat. Usually researchers are not interested in the specific strategies that emerge so much as in the population dynamics over the course of the simulation, and in what kinds of population equilibria can emerge. Check out some of Dan Ashlock's papers on Game Theory for more.


Is there a theoretical maximum for intelligence?

Note that the statement says nothing directly about the limit of intelligence, nor even about the limit of computational intelligence - but about the limit of computing power.

Perhaps the sentence "the speed of light puts a strict upper bound on how much computing can be done" needs a better explanation:

The authors are probably referring to Bremermann's limit, which defines an upper bound in bits per second per kilogram. It is an upper bound on the processing power per unit of time of a computer with a given weight.

There is also Margolus–Levitin theorem which defines an upper limit in operations per second per joule. It is an upper bound on the processing power per unit of energy.

These principles do not define a theoretical limit on computing power, but a practical one. If you'll limit your computer and your energy source to the size and capacity of the earth (or to the those of the universe) - you'll get a very practical limit.

Check the reference section in Wikipedia article Limits to computation


Why is my Keras prediction always close to 100% for one image class?

Traditional neural networks can be over-confident (i.e. give a probability close to $0$ or $1$) even when they are wrong, so you should not interpret the probability that it produces as a measure of uncertainty (i.e. as a measure of how much it is confident that the associated predicted class is the correct one), as that it is essentially wrong. See this and this answers for more details about this.
Given that this overconfidence is not desirable in many scenarios (such as healthcare, where doctors also want to know how confident the model is about its predictions, in order to decide whether to give a certain medication to the patient or not), the ML community has been trying to incorporate uncertainty quantification/estimation in neural networks. If you are interested in this topic, you could read the paper Weight Uncertainty in Neural Network (2015) by Blundell et al., which proposes a specific type of Bayesian neural network, i.e. a neural network that models the uncertainty over the actual values of the weights, from which we may also quantify/estimate the uncertainty about the inputs. This paper should not be too difficult to read if you are already familiar with the details of variational-autoencoders.
So, the answer to your question is: yes, it's possible that the output probability is close to $1$ because neural networks can be over-confident. (I am assuming that the values returned by tf.keras's predict method are probabilities: I don't remember anymore, so I assumed that you did not make any mistake).
A similar question was already asked in the past here. The accepted answer should provide more details about different types of uncertainty and solutions.


Is analogy necessary to artificial general intelligence?

I don't think I can give you a true answer to the actual question as posed, as I don't have a strict definition of "general intelligence". Nor do I have a solid definition of "critical" in context.
However, if we lean on our naive/intuitive understanding of what "general intelligence" and what it means to be critical, you might translate your question as

Would a general intelligence system need analogy-making in order to do certain things that it couldn't otherwise do?

Or, to put it another way

Are there useful behaviors that are enabled by analogical reasoning that can't be replicated any other way?

In the strictest sense, I don't have an answer to either of those questions either, but there is at least evidence to suggest that the answer may be "yes".  See, for reference, the Copycat paper by Hofstadter and Mitchell.
From what I've seen, some of the kinds of problems Copycat solves are different from anything I've seen solved by other approaches.  Now maybe it's just a coincidence that nobody has tried solving those problems with, I don't know, let's say "deep learning" or "rule induction" or "genetic algorithms".  Or maybe they have and I just haven't stumbled across that corpus of research.
Anyway, I'll also add that there is still ongoing research into using analogy for AI/ML.  See, for example, the paper Analogical Inference for Multi-relational Embeddings (2017), where the authors talk about using analogy, but define their approach as "analogical inference" (which they claim is different from "analogical reasoning" as defined during the earlier "GOFAI period").  There is also the paper Evaluating vector-space models of analogy (2017), where another set of authors deal with a form of analogical reasoning.
I don't think there's a consensus as to whether or not some form of analogical reasoning is "critical", but it's definitely a subject that is still being researched.
And to go off on a little bit of a tangent - an interesting related question would be to ask whether or not "analogy making" would be an emergent property of a sufficiently deep/wide ANN, or would such a facility need to be designed and coded up explicitly.


How to design a classifier while the patterns of positive data are changing rapidly?

The phenomenon where the prediction targets (in your case, behaviour) change over time is referred to as "concept drift".

If you search for that term, you'll find that there have been many publications attempting to tackle that over multiple decades, way too many papers to all summarize here in a single answer. It's still a difficult problem though, by no means a "solved" problem.

Two different, broad directions for ideas are:

Frequently re-training (offline) static models on the most recent training data
Using online learning approaches that can continuously be updated from a data stream, online as new labelled data becomes available.

This github page contains a large list of papers on credit card fraud detection, where the problem you describe occurs because fraudsters change their behaviour in an attempt to evade detection. Most of those papers discuss variants of the first approach. Basically, many of those papers use an ensemble of multiple Random Forests. Every day, new labelled data becomes available. They often then remove the oldest of multiple Random Forests, and add a new Random Forest trained on the most recent data made available that day.

There are also some variants where they don't always train new models at a fixed schedule (e.g., every day), but try to detect when the statistical properties of the data have changed using statistical tests, and only train new models when it is "necessary" (due to such changes).

For the second idea, you'll often be thinking of approaches that use Stochastic Gradient Descent-like approaches for learning; with a non-decreasing learning rate / step size, such techniques will naturally, slowly "forget" what they have learned from old data, and focus more on the latter data.

If you have some method to obtain accurate labels for certain instances relatively quickly, you could consider an approach like the one proposed in this paper (disclaimer: I'm an author on this paper). For example, in that paper the assumption is that human experts can relatively quickly investigate and obtain accurate labels for a small selection of transactions, and this can be exploited to quickly learn in an online manner.


How to distinguish AI modeling from implementation?

One good way of differentiating modelling and implementation is to consider that models occupy a much higher level of abstraction.

To continue with the mathematical example: even though experimental mathematics might be dependent on computation, the program can be considered as one possible realization of the necessary conditions of a more abstract existence proof.

Over the last 25 years, software engineering methodologies have become quite good at separating models and implementations, e.g. by using interfaces/typeclasses/abstract base classes to define constraints on behavior that is concretely realized by the implementation of derived classes.

AI has always been a battle between the 'neats and the scruffies'. Neats tend to prefer working 'top down' from clean abstractions, 'scruffies' like to work 'bottom up', and 'bang the bits' of the implementation together, to see what happens.

Of course, in practice, interplay between both styles is necessary, but AI as a science progresses when we abstract mechanisms away from specific implementations into their most general (and hence re-useable) form.


What dataset might Elon Musk's Dall-E have used?

DALL·E is a 12-billion parameter version of GPT-3 trained to generate images from text descriptions

should be the same data they used to train the GPT-3


How can I determine the mathematical relation between the input and output variables?

There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a simplest function (according to whatever criteria you choose) to produce those values.

One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.

Several different algorithms can be used to do the search efficiently.  My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.


Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of just the first degree?"  There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer.  This is not to say that the theorem is wrong, but consider the applied aspects.  We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.
However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons.  There is active research into how to generalize this.
Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to "learning."


You can use append function:

final = df1.append(df2, ignore_index=True)

To set the last column as labels, you set them as so by:

labels = np.array(final["will_buy"])

So, when calling the fit method on the model you build, you set labels = labels.


Automatic prediction of whether a customer will come into the shop or not

This should be possible, but it's not completely clear what you are trying to do.

If you're trying to predict customer age and gender from a video, then you've got a computer vision problem. Deep learning methods are the state of the art for this, and probably some sort of convolutional deep neural network is your friend.

If you're trying to predict whether the customer will enter the shop from their age and gender, then @DuttA suggestion to start with simple classification techniques is probably the best bet. Try out logistic regression as a starting point.

If you really want to do both of these things at once, you can again try out a deep convolutional neural network.

All that said, it's not completely clear that there is signal in the videos you want to collect: it might be that predicting whether they'll enter the shop is not really possible on the basis of a short video. The only way to know for sure is to give it a try however.


Google's Principles of Artificial Intelligence

Artificial Intelligence at Google — Our Principles

Objectives for AI Applications

Be socially beneficial.
Avoid creating or reinforcing unfair bias.
Be built and tested for safety.
Be accountable to people.
Incorporate privacy design principles.
Uphold high standards of scientific excellence.
Be made available for uses that accord with these principles.

AI Applications We Will Not Pursue

Technologies that cause or are likely to cause overall harm. Where there is a material risk of harm, we will proceed only where we believe that the benefits substantially outweigh the risks, and will incorporate appropriate safety constraints.
Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
Technologies that gather or use information for surveillance violating internationally accepted norms.
Technologies whose purpose contravenes widely accepted principles of international law and human rights.

Questions

Are these guidelines sufficient?

They are not sufficiently defined to be as sufficient as a policy, not even in the paragraphs following each, but it is presented as a vision comprised of principles, which should not be expected to be as defined as policy.  Here is at least one caveat for each objective.

Half the world has would have been too poor to communicate globally had Linux Torvalds not somewhat antisocially insist on certain things.
Distributing money equally among homeless heroin addicts and entrepreneurs is considered fair by many, yet it is merely equal and perhaps monumentally unfair.
National defense may be considered for safety, but someone who survived Hiroshima or a Syrian refuge may find the definition tragic.
Google cannot be accountable to people without revealing its company confidential ranking, at which point, it would be exploited and they'd have to change it.
Incorporating privacy design principles is far from not storing data so that it cannot be subpoenaed.
The concepts of the singularity, solar panels sustainability, the climate crisis, the ample availability of fossil fuels within U.S borders, and cell communications technology are all considered scientific excellence by the general public.  That these lack the primary signs of rigor in mathematics, statistics, economics, and engineering doesn't occur to anyone, so upon what basis will Google judge?  Will they seek corroboration between theoretical models and previous ones along with empirical validation for every fact?
Being made available, if they are to be a soluble corporation, will depend on the financial condition of the customer, which is in conflict with fairness (#2) and no I, Robot precedence rules are established.

That segues into the next question, skipping the non-pursuit items, which could be similarly treated.

Are there any "I, Robot" conflicts?

The Three Laws of Asimov worked well with the robotic character R. Daneel Olivaw in the Foundation sci fi novel series.  Daneel simulated human emotion, transcended human selfishness, and maintained the three laws.  In the screenplay adaptation of his I, Robot, the three laws didn't work out as well.  VIKI decided that human freedom was in conflict with law two.

Because the eleven dos and don'ts are not codified in precise policies, there are 10! (3,628,800) potential conflicts.  There is at least one that is already clear, mentioned above.

How much does this matter if other corporations and state agencies don't hew to similar guidelines?

In a world where information may have become more powerful than money, perhaps it matters quite a bit when an information giant takes a position.

It's Wikipedia that is most questionable in that through their web service language evolution may have been democratized to a self-defeating extreme.  One can create a word definition without peer review that gains more public authority than the definition in rigorously prepared and cross-correlated encyclopedias and dictionaries.  Even in this artificial intelligence forum, where the education level is high, people define tags with Wikipedia links.  I am tempted to endeavor to remove them all from here. :)


Is there such a thing like the machine learning paradox?

Basically Machine Learning is used to solve problems for which giving exact solutions is not (practically or theoretically) possible or feasible. If you have a problem that can be solved with exact and proven methods, you don't need Machine Learning. So Machine Learning is rather about finding solutions by approximations with some error margins. In this sense such systems will never perform perfectly and there always will be room for improvements.
(Many real world problems can not be solved perfectly because they are not simply complicated, but chaotic.)


What is the meaning of the terms in this evaluation function for chess?

The function you are showing (from this website) only calculates the score of a position based on how many pieces are on the board. It does not take into account where the pieces are located. So wK, bK, wQ, bQ, etc. are simply the number of the specific pieces that are on the board. The weights rank the pieces according to their importance.
So for example if white has 8 pawns and black has 6 pawns, the last term of the score would be pawnWt * (wP-bP) = 1 * (8-6) = 2. (assuming the value of a pawn to be 1) That leads to a relative advantage for white only based on pawns, but according to how many of the other pieces are on the board, black might still have an advantage.


How can we estimate the transition model and reward function?

Given that model-based RL algorithms do not necessarily estimate or compute the transition model or reward function, in the case these are unknown, how can they be computed or estimated (so that they can be used by the model-based algorithms)?

A generally reliable approach to creating learned models from interacting with the environment, then using those models internally for planning or explicitly model-based learning, is still something of a holy grail in RL. An agent that can do this across multiple domains might be considered a significant step in autonomous AI. Sutton & Barto write in Reinforcement Learning: An Introduction (Chapter 17.5):

More work is needed before planning with learned models can be effective. For example,the learning of the model needs to be selective because the scope of a model strongly affects planning efficiency. If a model focuses on the key consequences of the most important options, then planning can be efficient and rapid, but if a model includes details of unimportant consequences of options that are unlikely to be selected, then planning may be almost useless. Environment models should be constructed judiciously with regard to both their states and dynamics with the goal of optimizing the planning process. The various parts of the model should be continually monitored as to the degree to which they contribute to, or detract from, planning efficiency. The field has not
yet addressed this complex of issues or designed model-learning methods that take into account their implications.

[Emphasis mine]
This was written in 2019, so as far as I know still stands as a summary of state-of-the-art. There is ongoing research into this - for instance the paper Model-Based Reinforcement Learning via Meta-Policy Optimization considers using multiple learned models to assess reliability. I have seen a similar recent paper which also assesses reliability of the learned model and chooses how much it should trust it over a simpler model-free prediction, but cannot recall the name or find it currently.
One very simple form of learned model is to memorise transitions that have been experienced already. This is functionally very similar to the experience replay table used in DQN. The classic RL algorithm for this kind of model is Dyna-Q, where the data stored about known transitions is used to perform background planning. In it's simplest form the algorithm is almost indistinguishable from experience replay in DQN. However, this memorised set of transition records is a learned model, and is used as such in Dyna-Q.
The basic Dyna-Q approach creates a tabular model. It does not generalise to predicting outcomes from previously unseen state, action pairs. However, this is relatively easy to fix - simply feed experience so far as training data into a function approximator and you can create a learned model of the environment that attempts to generalise to new states. This idea has been around a long time. Unfortunately, it has problems - planning accuracy is strongly influenced by the accuracy of the model. This applies for both background planning and looking forward from current state. Approximate models like this to date typically perform worse than simple replay-based approaches.
This general approach - learn the model statistically from observations - can be refined and may work well if there is any decent prior knowledge that restricts the model. For example if you want to model a physical system that is influenced by current air pressure and local gravity, you could have free parameters for those unknowns starting with some standardised guesses, and then refine the model of dynamics when observations are made, with strong constraints about the form it will take.
Similarly in games of chance with hidden state, you may be able to model the unknowns within a broader well-understood model, and use e.g. Bayesian inference to add constraints and best guesses. This is typically what you would do for a POMDP with a "belief state".
Both of the domain-specific approaches in the last two paragraphs can be made to work better than model-free algorithms alone, but they require deep understanding/analysis of the problem being solved by the researcher to set up a parametric model that is both flexible enough to match the environment being learned, but also constrained enough that it cannot become too inaccurate.


How is the Markovian property consistent in reinforcement learning based scheduling?

Lot of real tasks are in reality not markovian, but it doesn't mean you can't try to train an agent on these tasks. It's like saying "we assume variable x to be normally distributed", you just assume that you can condition the probability distributions on the present state of the environment hoping that the agent will learn a good policy. In fact for most applications the challenge is to frame a problem in order to make it as most plausibly markovian as possible (by compressing some important past information into a present state of the environment).

It is pretty common to just give for granted the Markov property, for example in NLP hidden Markov Models are used a lot for sequential tasks like entities detection, this of course leads to well known issues like high error rate on long sentences (the more you look into the future the more the error rate).

Note also that a Markov model can be of first order (probabilities conditioned only on the present state):

$P(W_{t+1} = w | W_{t}, W_{t-1},W_{t-2} ..) = P(W_{t+1} = w | W_{t})$

but they can also be of higher orders (for example second order if conditioning on the present state plus one step in the past):

$P(W_{t+1} = w | W_{t}, W_{t-1},W_{t-n} ..) = P(W_{t+1} = w | W_{t},W_{t-1})$

Of course the more steps in the past you condition the more quickly the problem becomes intractable, that's why almost always first order models are used.

EDIT

I add some comments on the scheduling paper as suggested by nbro.

So, here I would say that the most stryking aspect that makes the process look impossible to describe as an MDP is the presence of dependencies between jobs. Since I might need the result of certain jobs1 before to process  another certain job2, there is surely no way that a time step t could be independent on time step t-1 (I need to know which job I processed in order to know which processes I can process or not).

Here, the trick they use is to learn these dependencies between jobs thanks to a graph network, represented by the DNN in the deep reinforcement learning framework. So, what the agent has to learn is to select a tuple of two actions: "which output (i) a stage designated to be scheduled next, and (ii) an upper limit on the number of executors to use for that stage’s job". The information used to make this selection is the deep representation computed by the graph network on the decency graph of the jobs to schedule. In this sense, since the network is able to represent the 'temporal' relationships between jobs in the present state, this allows to assume that the choice of the next action tuple do not depend on previous states. Hope this is useful and make some sense to you.


Have machine learning techniques been used to play outdoor games, like cricket or badminton?

Yes it is possible.

A group of Chinese college students and teachers made a robot that plays badminton.  I am sure someone will make a robot that can play cricket and other outside games.

Although not an outdoor game, Omron made a robot named Forpheus that plays ping pong.  There is also a robot that plays the sport of curling.

There is an annual event called the RoboCup where robot teams compete in indoor soccer on a scaled down level.  They don't look like they will be beating humans in the next couple of years but it is interesting to watch.  Their web site is: http://www.robocup.org/

There are a few challenges that remain to be resolved for robots to play humans in sports.  The biggest one is self-contained power that will last long enough to play a game.  It takes a lot of power to move a human scaled robot and run its electronics, sensors, and computers.  Other challenges are dexterity and agility.  There have been some advances in these areas as this video shows.


What are the most common methods to enable neural networks to adapt to changing environments?

For the vast majority of cases where you have a dynamic(and assumed non-linear) relationship between your input and output, you would not use modified architecture. You would simply retrain on the new data.

In some cases, based on domain knowledge or intuition, one might put a "weight" on the new data to increase or decrease its importance relative to previous data.

There are some attempts(mostly by those studying one-shot learning) to create NNs that quickly fit to new data effectively with only a few samples. However, most of these are not ready for anything resembling real-world problems(particularly on tabular data).


Why isn't my model learning satisfactorily?

Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.


In novelty search, are the novel structures or behaviour of the neural network rewarded?

In the paper Exploiting Open-Endedness to Solve Problems Through the Search for Novelty (2008), by Joel Lehman and Kenneth O. Stanley, which introduced the novelty search approach, it is written

  Thus this paper introduces the novelty search algorithm, which searches with no objective other than continually finding novel behaviors in the search space.

and

  instead of searching for a final objective, the learning method is rewarded for finding any instance whose functionality is significantly different from what has been discovered before

and

  The novelty of a newly generated individual is computed with respect to the behaviors (i.e. not the genotypes) of an archive of past individuals whose behaviors were highly novel when they originated

Therefore, the goal of a novelty search is to search for novel behavior and not necessarily novel chromosomes (or genotypes).

In the experiments reported in the novelty search paper, the authors use neural networks to represent the policy that controls a robot that needs to navigate a maze, while evolving these neural networks with NEAT (a neuroevolution method) and a novelty metric (rather than a fitness metric, which is used in the original NEAT). In the same experiments section, Lehman and Stanley write

  Thus, for the maze domain, the behavior of a navigator is defined as its ending position. The novelty metric is then the Euclidean distance between the ending positions of two individuals. For example, two robots stuck in the same corner appear similar, while one robot that simply sits at the start position looks very different from one that reaches the goal, though they are both equally viable to the novelty metric.

Therefore, the evolution of the neural networks, which represent the controllers, is not necessarily guided by the novelty of (the architecture of) the neural networks but by the novelty of the behavior generated by the neural networks, even though novel neural networks might correspond or lead to novel behaviors.


How can I detect diagram region and extract (crop) it from a research paper

I will consider that you need to extract(crop) the digram from the pdf research paper. You can use PyPDF2 or PyMuPDF to extract the images from the PDF file and then you can apply machine learning to do recognition and classification of the images. There are different types of machine learning solutions for image classification and you can start with Convolutional Neural Network and you can start here.

For more information for Recognition and classification of figures  see these  and to extract of figures from scholarly documents read this


Are there any references of NLP/text mining techniques for identifying the theme of news headlines?

You could formulate the problem as a topic classification task, hence you need labeled data.
From an unsupervised point of view, you could represent sentences with some fixed feature vector (latent representation).

Generating Sentences from a Continuous Space.
Paragraph2Vec.

BRIEF-Dynasil Corporation Of America Reports Q2 EPS Of $0.08
China's night-owl retail investors leverage up to dominate oil futures trade

Self-attention models would be very useful to this kind of problems since you don't need to encode all the context in the last hidden cell of some RNN model to classify to which theme the headline belongs.


Is there any research work on the usefulness of the concept of "personality" for an AGI?

'Personality' is something of a 'suitcase word' (Minsky) for quite a large collection of (presumably reasonably consistent) observable traits.

It seems clear that there is a certain collective advantage in having a consistent personality - specifically that it affords observers some learning gradient in an otherwise uncertain environment. This is of particular importance because those consistencies might have been arrived at using different learning mechanisms than the ones a given observer has.

Hence, in any non-trivial coevolutionary system, other organisms will inevitably make use of any such consistencies. Consider a simple robot, called Alice, say, that has the trait of 'quickly flashing red when it sees a blue robot'. It makes sense for all observers to exploit everything that they perceive as correlating with Alice's behavior, in particular, the prediction that a blue robot is likely to be present.

The best reference I can recommend on this (which shows that we tend to ascribe 'personality' to even very simple mechanisms) is 'Vehicles' by Valentino Braitenberg.


Is there any programming practice website for beginners in Reinforcement Learning

Firstly, since you are a beginner, I strongly recommend you start reading Sutton's book. It is a really great book.

Then, some tutorials:

udemy rl

udemy deep rl

rl-with-tensorflow

learndatasci

stackabuse


Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of samples. In the second case you have multiple classes and in the current form it's only for a single sample.
In the first case there is only one output, if you had two outputs it would have been
\begin{equation}
-\frac{1}{N} \sum_{n=1}^N \sum_{j=1}^2 y_{n,j} \log(\hat y_{n,j})
\end{equation}
where $n$ iterates over batch samples, and $j$ over two classes. The reason why it was written like that is that if you have two classes you can have only one output because you can immediately conclude about probability of the second class if you have the probability of the first class, it would simply be $p_1 = 1-p_0$.
In the second case, with batch samples included, it would be
\begin{equation}
 -\frac{1}{N} \sum_{n=1}^N \sum_{j=1}^c y_{n,j} \log(\hat y_{n,j})
\end{equation}
wheren $n$ iterates over batch samples, and $j$ over $c$ output classes.


Game AI design for a multiplayer random board game?

Many successful game-playing engines use some form of search to look ahead and plan the next move. This is possible even in stochastic environments by building a tree with probabilities assigned, or by simulating the randomness during planning and relying on lots of samples to get reasonable estimates of value.

There are many approaches possible. However, as Ticket to Ride is a relatively simple game that you could automate to play to the end very quickly (1000s of times per second), then you could look into Monte Carlo Tree Search (MCTS) as a mechanism for the game-playing agents to look ahead and adjust their play to current odds.

The basics of Monte Carlo Tree Search:

On its turn, the agent builds up an evaluation tree local to its current state - each node of the tree is a game state, augmented with statistics from MCTS (how often node was chosen, how many times agent managed to win eventually from that node) and each link is a choice that a player makes. This is built up slowly, typically just adding one node per full simulated game.
For look-ahead search "inside" the tree, the agent explores options according to their relative promise, selecting moves to simulate according to its best estimates of value at the time. It needs to explore alternatives to make sure it doesn't miss anything, and there are a few variations of doing this. One successful approach to action selection inside the tree is Upper Confidence Bound applied to Trees (or UCT for short).
When play gets to a leaf node of the tree, the tree might be expanded slightly, and then afterwards the agent simulates plays of the game either to the end or to a robust evaluation point. This later play might be entirely random - or might still be guided probabilistically by some heuristic - and is called a rollout. The important thing here is to play quickly to get samples just to roughly assess the position at the leaf of the tree.
The evaluation resulting from rollout is fed back into the tree, which builds up sampled statistics of the most promising variations of play.
After a certain number of repeats of the whole algorithm (local tree search, expanding the tree, rollout and update), the agent picks the most promising next step from the tree. At this point it may just discard the whole tree - in a game with lots of randomness, this is probably what you would do now, as things you were considering as just probabilities before will have become historical fact, and the chances are that re-running the tree build from the next start position will be more accurate.

There are a few variants possible, including combinations with neural networks as used in AlphaZero.

The basic principle is to sample from many choices, focusing over time on what looks "best" to each player based on statistics of those choices.

As Ticket To Ride is a card game where cards can be drawn from an unseen deck, and it would be considered cheating to simulate and rollout with the actual deck in play, you would need to have realistic shuffles of the remaining unknown cards used during rollout. It may still work quite well without the realistic part and for speed just assume a random unlimited deck of cards (because re-shuffling on each imagined rollout would be costly).


Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?

It is an error. But it is also not the in final version of the paper (arxiv.) The final version of the paper can be found here where they replace "absolute value of the largest eigenvalue" with "largest singular value".

  We first prove that it is sufficient for $\lambda_1 < \frac{1}{\gamma}$, where $\lambda_1$ is the largest singular value of $W_{rec}$, for the vanishing gradient problem to occur.


Can machine learning be used to improve the average case complexity of an algorithm?

To the best of my knowledge, there haven't yet been many academic publications in this area, which could be broadly said to fall within Search-Based Software Engineering. Here are the ones I know of.

Jerry Swan and Nathan Burles. Templar - A Framework for Template-Method Hyper-Heuristics. In: Genetic Programming - 18th European Conference, EuroGP 2015, Copenhagen, Denmark, April 8-10, 2015, Proceedings. 2015, pp. 205–216. DOI: 10.1007/978-3-319-16501-1_17.

This paper describes 'Hyper-quicksort', a quicksort variant that uses Machine Learning (ML) to generate a pivot function:

A. E. I. Brownlee, N. Burles, and J. Swan. Search-Based Energy Optimization of Some Ubiquitous Algorithms. In: IEEE Transactions on Emerging Topics in Computational Intelligence 1.3 (2017), pp. 188–201.

This paper uses ML to generate energy-efficient variants of some widely-used algorithms

David R. White, Leonid Joffe, Edward Bowles, and Jerry Swan. Deep Parameter Tuning of Concurrent Divide and Conquer Algorithms in Akka. ISBN: 978-3-319-55792-2.

This paper uses ML to optimise the FFT, matrix multiplication and quicksort for concurrency

Nathan Burles, Edward Bowles, Alexander E. I. Brownlee, Zoltan A. Kocsis, Jerry Swan, and Nadarajen Veerapen. Object-Oriented Genetic Improvement for Improved Energy Consumption in Google Guava. DOI: 10.1007/978-3-319-22183-0_20.

This one optimises Google Guava for energy consumption

Zoltan A. Kocsis, Geoff Neumann, Jerry Swan, Michael G. Epitropakis, Alexander E. I. Brownlee, Saemundur O. Haraldsson, and Edward Bowles. Repairing and Optimizing Hadoop hashCode Implementations. In: Search-Based Software Engineering: 6th International Symposium, SSBSE 2014, DOI: 10.1007/978-3-319-09940-8_22.

This one fixes broken hashCodes by using ML to generate new ones

There was also one (from Microsoft Research, I think) entitled something like "The case for self-adjusting data structures". I'll add an edit if I can find it.


What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?

The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write two explicit expectations here because $a$ is a random variable with distribution $\mu_x$ but $X$ is also a random variable with distribution $V$. I believe you are right that $V$ would be analogous to a set of features. So we are saying that $X$ is a random variable over the distribution over the features, or individuals in this context.
As for the $\Delta(A)$, I have never seen this notation used before -- I am not sure if it is standard notation. I Googled to see if it was something that I just hadn't seen before, but there was no such answer. However, from the context of the paper they define $M: V \rightarrow \Delta (A)$ to be mappings from individuals to probability distributions over $A$, so I guess that $\Delta (A) $ is probability distributions over $A$.


What is the purpose of argmax in the PPO algorithm?

In this case yes, $J$ is the big $\min$ expression and you apply Adam on that. But be careful because they say they do ascent, but automatic differentiation software usually minimizes given function so your $J$ would be $−\min(⋅)$.


What's the likelihood in Bayesian Neural Networks?

The likelihood depends on the task that you are solving, so this is similar to traditional neural networks (in fact, even these neural networks have a probabilistic/Bayesian interpretation!).

For binary classification, you should probably use a Bernoulli, which, in practice, corresponds to using a sigmoid with a binary cross-entropy (you can show that the minimization of the cross-entropy is equivalent to the maximization of Bernoulli p.m.f.)

If it's a multi-class classification problem, you should use a categorical distribution, which corresponds to a softmax with a categorical cross-entropy; see e.g. this implementation

If it's a regression problem, the likelihood could be a Gaussian (which is equivalent to using the MSE: you can also show this).

In the case of (mean-field) variational BNNs (VBNNs) (note that not all neural networks denoted as BNNs are VBNNs, but here I will only focus on VBNNs), rather than directly performing Bayesian inference (i.e. applying the Bayes theorem directly), you will perform approximate inference and, more specifically, you cast the inference problem as an equivalent optimization problem, where you typically optimize the loss function known as the evidence lower bound (ELBO), which is composed of two terms

the (log-)likelihood of the parameters (so the Bernoulli/categorical/Gaussian)
the KL divergence (the regularization part)

Theoretically, given that I am not a statistician, I cannot tell you right now why we can have a Bernoulli likelihood and a Gaussian prior (which is typically the case and, more specifically, in the case of this paper, they assume the weights to be independent, which is kind of a strong assumption but simplifies the computations, so this leads to a factorized multi-variate Gaussian over the weights), but this is what I had seen in practice.


What is the surrogate loss function in imitation learning, and how is it different from the true cost?

A surrogate loss is a loss than you use "instead of", "in place of", "as a proxy for" or "as a substitute for" another loss, which is typically the "true" loss.
Surrogate losses are actually common in machine learning (although almost nobody realizes that they are surrogate losses). For example, the empirical risk (which the mean squared error is an instance of) is a surrogate for the expected risk, which is incomputable in almost all cases, given that you do not know the underlying probability distribution. See An Overview of Statistical Learning Theory by V. N. Vapnik for more details. In fact, discussions on generalization arise because of this issue, i.e. you use surrogate losses rather than true losses.
The term "surrogate" is also used in conjunction with the term "model", i.e. "surrogate model", for example, in the context of Bayesian optimization, where a Gaussian process is the surrogate model for the unknown model/function that you want to know about, i.e. you use the Gaussian process to approximate the unknown function/model.
Regarding the excerpt you are quoting and your specific concerns, although I didn't read the paper and I am not an expert in imitation learning, let me try to explain what I understand from this excerpt. Essentially, in imitation learning, you use the expert's policy $\pi^*$ to train the agent, rather than letting him just explore and exploit the environment. So, what you know is $\pi^*$ and you can calculate the "loss" between $\pi^*$ and $\pi$ (the current agent's policy), denoted by $l$. However, this loss $l$ that you calculate is not necessarily the "true" loss (i.e. it is a surrogate loss), given that our goal is not really to imitate the "expert" but to learn an optimal policy to behave in the environment. If the goal was to just imitate the "expert", then $C$ and $l$ would coincide, because, in that case, $l$ would represent the "discrepancy" or "loss" between $\pi$ and the expert's policy $\pi^*$.


How to parse conjunctions in natural language processing?

This seems not easy for NLP. I doubt that state-of-the-art NLP tools can reliably determine the correct hierarchical structure of independent clauses. Examples below.

The Berkeley parser gets it basically right in the sense that it can put its late and I'm tired on parallel, and they together on parallel with the weekend. But still not perfect (the weekend should be in the same subtree with It's rather than if its late and I'm tired)

The Stanford parser, which is available in Python (NLTK), incorrectly parsed I\'m tired to the same level of I drink orange juice.


How can I train a neural network for image classification when the dataset is small?

Use Fine Tuning

You can simply use a pre-trained model on ImageNet, as this data set has multiple snakes classes.

Then you can fine tune the model with your own small data set and outputs.
See this for further understanding :
Fine Tuning in Keras

(if you don't use Keras, there are other tutorials on the internet using other Machine Learning framework)

The idea is just removing the last layer (1000 outputs if you use a model pre-trained with ImageNet) and adding a layer of your choice with random weights and a custom number of outputs (number of your classes).

Then your retrain your network, in general we retrained only the last layers (as first layers have more general features).


To quote the relevant Wikipedia article: "The original creators of the database keep a list of some of the methods tested on it.[5] In their original paper, they use a support vector machine to get an error rate of 0.8 percent"

Feel free to look up that original paper, but to me the quote strongly suggests that the first record holder was a support vector machine.

Edit: As liori points out the quote is misleading: In the original paper Yann LeCun et al. actually tried a slew of methods and one version of ConvNet scored best (0.7).

But according to the MNIST-webpage, after that initial paper, DeCoste and Scholkopf, MLJ 2002, reached an error of 0.56 with a SVM. So if that webpage is complete, the claim is still false.


The best approach would be starting with smaller projects involving neural networks and genetic algorithms to gain experience in order to speedup the coding of the project you have proposed; playing around with TensorFlow and Unreal Engine it is not a bad idea.

Hint: when implementing your idea of artificial life, you should consider that each cell/organism have to have some kind of sensors in order to capture informations from the environment; such informations i.e. the position and the distance of the nearest meal and/or predators, the temperature, the pressure and depth of water, should be passed through the neural network to determine the response of the cell. Also, in your environment you should promote the spreading of organisms which responses are euristically better i.e. cells that don't get caught by predators or don't die by starvation. How? Simply by evolving their brain/brains/sensors through a genetic algorithm that favors individuals/species with good parameters. I recommend you a nature-inspired AI method, it is called NEAT model. It explains how to implement a neural networks that can be evolved. The paper can be found here: Evolving Neural Networks through Augmenting Topologies.

A different approach to NEAT would be Deep Reinforcement Learning; in the link you can find a demo artifical organism that learns how to find meals.

There are a ton of parameters and implementations you can consider, the only limit is your creativity.


Advantages of Kullback-Leibler over L1/L2?

Considering your question was asking why KL-Divergence would be preferred over MSE, RMSE, or L1/L2 Norm between ground truth and predicted, as loss function for predicting a distribution -

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.

L2-norm/MSE/RMSE doesn’t do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly effected by any power operations (Square or Root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.

L1 norm on the other hand, does not have any power operations, making it relatively acceptable.

Loss functions such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence are preffered for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or in other words, assuming $Q$ is the ground truth distribution, KL-Divergence is a measure of how much $P$ deviates from $Q$. Also considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.

More clarity on the motivation behind Kullback-Leibler can be read here. Hope your query has been answered!


Is it feasible to use minimax to solve a board game with a large number of moves?

-The player can choose as many pieces to move as he likes. For example none, all of them, or some number inbetween. (Whereas in chess you can only move one)

That quote specifically is the part that really causes the size of your legal action set to blow up. You have a combinatorial action space here. If each of your pieces has 8 legal moves, then that is:

8 legal moves for the first piece (or 9 if that count didn't already include the "do nothing" option)
for each of those, there are again 8 or 9 different choices for the second piece (leading to e.g. $8 \times 8 = 64$ possible combinations for just the first two pieces)
for each of those, again 8 choices for the third piece (leading to $64 \times 8 = 512$ possible combinations for just the first three pieces).
etc.

This blows up way too quickly, and there's really no hope of ever getting a decent player for this using any MiniMax-based algorithm (including things like alpha-beta pruning, principal variation search etc.).

In the kinds of games that you describe, you'll want to use algorithms that can exploit the "structure" of your action space. A raw enumeration of all possible combinations blows up quickly, but many algorithms can do reasonably well by re-phrasing the problem in such a way that you have more "depth" rather than "breadth". For example, instead of viewing a full combination of choices for all pieces as a single "action", you can treat the choices per piece as a separate "action".

Rather than making a single choice out of $8 \times 8 \times 8 \times \dots$ possibilities every turn, you want to have a search tree where your player makes one choice out of $8$ (for the first piece), followed immediately by another choice out of $8$ (for the second piece), etc. The opposing player only gets to make a choice after the current player has made choices for all pieces. With such a strategy, the breadth of your search tree will no longer be a problem, but the depth will become a problem. To address this, you'll additionally want to make sure that your methods can generalize across different depth levels.

A good place to look would be combinatorial versions of Monte-Carlo Tree Search, such as those described in:

https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf
https://www.jair.org/index.php/jair/article/view/11053
(probably a few other publications by the author of that second link)

These algorithms are quite a bit more complicated than MiniMax though, MiniMax is a very basic algorithm in comparison.


What is the most suitable AI technique to use for path planning?

If I had to implement a path exploration/finding algorithm on a robot, I would follow these steps:

Make sure you can detect your position. You need to be able to record your position otherwise you have no reference for the exploration. You don't need a global positioning system (like GPS), a local one is more than enough in your case. This means that the robot must know that it has been switched on in position (0,0). If you go straight for 5 meters, you'll update it to (5,0) or something like that.
Knowing how much you moved and towards where is the difficult thing.
Once you can know where you are, it is time to record it. As you want to explore the environment, you might want to create a tree with states on the nodes. The node can be open if you can still explore around it, closed if the exploration has been done.
To get the path from a node to another, A* works more than enough.
To know what is around you, you can use the sensors to explore the surroundings and know the position of the obstacles.

This is the general idea:

Tree tree = new Tree()
Node first = new Node(Here)
tree.Add(first)

Node current = first

while(true) {
    ExploreAround(current)
    var nodes = CreateSurroundingNodes()
    tree.Add(nodes)
    current.State = closed
    var next = PickNearestOpenNode()
    MoveTo(next)
    WaitUntilRobotIsOn(next)
    current = next
}

In my bachelor thesis I was implementing a very similar algorithm on a drone on Unity3D. You can find the package here. You can get an idea out of it.

Here is a video of how it worked: https://youtu.be/Xrh9-4Bfcew