Can the agent wait until the end of the episode to determine the reward in SARSA?

There are a couple of things to break down here.

The first thing is to correct this:

  For example, the reward for the game tic-tac-toe is decided at the end of the episode, when the player wins, loses, or draws the match. The reward is not available at each step $t$.

In a Markov Decision Process (MDP), there is always an immediate reward for each time $t$ from $t=1$ to $t=T$ (terminal state). This is the reward distribution that the algorithms refer to as $R_t$.

It does not matter if nearly all the rewards are $0$. That is still a reward value, even if it is not an interesting or informative one. Some environments will have non-zero rewards on nearly all time steps. Some environments might have zero rewards for every transition apart from one or two important exceptions that define the goal of the agent.

So in tic-tac-toe, if the game has not ended, the reward will be $0$ because neither player has won or lost, and obtaining a win is a learning objective. If you use $0$ reward value for all incomplete time steps in tic-tac-toe, and SARSA algorithm as written from Sutton & Barto, then it will work as expected. You don't need to wait until the end of the episode, and the algorithm will still learn to predict values of moves despite experiencing many $r_t = 0$ during start and middle of the game.

You show the MC prediction algorithm, which does wait until the end of each episode until calculating estimated values, using the experienced return.

I think one of the things that you might want to review is the difference between immediate reward following a timestep (noted $R_{t+1}$) and return (noted $G_t$). There is a relationship between the two $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$. All the value based methods, such as MC prediction, or SARSA, are ways to estimate the expected return $\mathbb{E}_{\pi}[G_t]$ given some context, such as the state or state and action at time $t$.

Given that, this should answer your other questions.

  Is determining the reward for each action a requirement of SARSA?

It is a general requirement for MDPs, and is expected by all solvers. However, a default of $0$ is fine, if that's how the environment works. The reinforcement learning (RL) algorithms can all cope with delays to important reward values (such as those from winning or losing a game). Bigger delays - e.g. thousands of $0$ reward steps between more meaningful values - are often associated with harder problems to solve, so the simpler algorithms might not be practical. But they all cope in theory given enough experience.

  Can the agent wait until the end of the episode to determine the reward?

No, you should always be able calulate an immediate reward value $r_{t+1}$ after taking each action $a_t$ from state $s_t$.

However, it is OK in general to wait until you have summed up more of those rewards (e.g. $r_{t+1}$ to $r_{t+n}$) before making an estimate of the return $g_t$ to update a value estimate. Single step SARSA (sometimes called SARSA(0)) does not wait - it makes update steps using a single immediate reward value. However, other agent types including Monte Carlo (MC), and some variants of SARSA, can and do wait - sometimes just for some steps, sometimes until the end of each episode.

It is a choice you can make when designing agents, there are consequences. Waiting until end of the episode (as in MC) means you have unbiased estimates of value, but takes longer, and leads to higher variability in updates. Using reward values after each step (as in SARSA(0)) means you get to make more frequent updates which may converge fatser, but you start with biased incorrect values.


Feature extraction timeseries, model compatibility

there's a lot to un-pack in this question.

Why do they only pick 500 rows?

my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features.  note that when they evaluated their model, they took the last 500 samples.

What's the point of re-arranging the rows/columns?

answer: the data frame format that tsfresh requires in order to calculate features is that format. it is a bit of a pain...especially when you need to keep track of an id-column for other data.

When they build their decicion tree learner with the train data, they build it upon extracted features.

answer: yes

But how can we then use this tree for new sensor data?
Should we extract the features of the new data and pass it as input for the tree?

answer: yes

But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

answer: yes it will. I don't know where your copy-pasted error message came from. When you generate a set of features using tsfresh, you can do it in a couple of different ways. you can generate all of them or you can generate a subset of them---they generated a subset...but then subsetted it once again(using their importance stuff...whether you use the importance methods or other methods to select features you will end up with a bunch of features you need to calculate). If you follow the procedure for generating features based on a pre-determined subset (relevant_features in their case), you will end up with the same # of features.  This needs to be stressed...don't generate tsfeatures that you don't need! as it will take foooorever.


Will it ever be possible to construct a neural network that could have the features of human brain?

Maybe in the distant future they could build a computer powerful enough to simulate the individual neurons of an entire human brain. Then they could carefully copy/paste the connectivity of a sample brain into the computer simulation.

Given that this extreme seems physically possible, it stands to reason that much simpler/smaller alternatives could be engineered in the future assuming continuous advancements in technology.


How does Monte Carlo Tree Search UCT exploitation value change based on perspective?

In UCT, the value of Q(vi) / N(vi) is bounded between 0 and 1. Normally when applying MCTS to 2-player games, what happens is the following:
N(vi) corresponds to the total number of games simulated in node vi.
Q(vi) corresponds to the total number of games simulated and won in node vi.
So in each simulation Q(vi) will add +1 to the winning player and +0 to the losing player.

In a tree representation of a 2-player game, each level will represent player A and player B alternately. So you will add 1 to every other node. I don't think it makes sense to add negative values to Qi.

So to answer your question:
You only need to store N and Q for each node, where N is the total number of times that node was in the path of a simulated node and Q is the number of times it won, out of those simulations.

I can give you the example of the way I implemented that part, in a board game I saved in the state class a variable which said who the next player was ( 1 or 2). So I would know that in that state (each node is a state) the player who made the last move was the other one (by the formula: player = 3 - next_player), and I would add 1 to the Qi of those nodes.


What should I do with the flatten layer during back-propagation?

The Flatten layer has no learnable parameters in itself (the operation it performs is fully defined by construction); still, it has to propagate the gradient to the previous layers.
In general, the Flatten operation is well-posed, as whatever is the input shape you know what the output shape is.
When you backpropagate, you are supposed to do an "Unflatten", which maps a flattened tensor into a tensor of a given shape, and you know what that specific shape is from the forward pass, so it is also a well-posed operation.
More formally
Say you have Img1 in input of your Flatten layer
$$
\begin{pmatrix}
f_{1,1}(x; w_{1,1}) & f_{1,2}(x; w_{1,2}) \\
f_{2,1}(x; w_{2,1}) & f_{2,2}(x; w_{2,2})
\end{pmatrix}
$$
So, in the output you have
$$
\begin{pmatrix}
f_{1,1}(x; w_{1,1}) & f_{1,2}(x; w_{1,2}) & f_{2,1}(x; w_{2,1}) & f_{2,2}(x; w_{2,2})
\end{pmatrix}
$$
When you compute the gradient you have
$$
\frac{df_{i,j}(x; w_{i,j})}{dw_{i,j}}
$$
and everything in the same position as in the forward pass, so the unflatten maps from the (1, 4) tensor to the (2, 2) tensor.


Goal oriented Action Planning with multiple Agents

This evening I got inspired by this paper: http://www.dphrygian.com/bin/David-Pittman-GOAP-Masters-Thesis.doc

(GOAP paired with the Command and Control Pattern)

What do you think about this solution?

Each goal has a relevance (that depends on the agent needs)

When agent1 working on the "AskAgent2ToTalk" Action, it only sends a goal recommendation to agent2. Explicit means this: The agent only sends a list of modifiers for each goal relevance (in this case it could be a bonus for the "SocialInteraction" Goal. The value depends on the relationship between the agents) and the recommend goal.

if agent2's desired goal is the recommend goal, the agent replans.


It depends on the complexity on your sentences. If you have a limited range, you could do simple pattern matching on part-of-speech tags. Put your sentence through a tagger (there are plenty of them around) and look for the first noun following a verb:

I       want an         apple
Pronoun verb determiner noun

(I assume you mean the object, as the subject in that sentence would be I). Scan through the list until you hit a verb, then pick the next noun. This should work for most English sentences.

It gets a bit more complicated if your nouns have additional words around them, for example red apple (where red is an adjective) or bottle of beer (where you have the pattern noun OF noun). So if you want to capture those fully, you might need a few more complex matching rules.

Overall it should still be a lot easier than implementing a full-blown syntactic parser which creates a comprehensive structural analysis of your sentence, most of which you wouldn't need in the first place.


What are the approaches to teach AI to how to render html page based on its source code?

The rendering process for browsers is very well defined, and has a very rigid definite ruleset where (virtually) every accountability is noted and handled. This is not optimal for Machine Learning, which works when we have a large pool of examples, and we don't know the ruleset; it will figure it out. Even if you were to train an Neural Network to process that input, there are several things you must account for:

1. Variance in data.

Not all webpages are equal in length or complexity, and making a neural network to generate output from HTML would produce garbage most of the time.

2. Training time.

The time it would take for a neural network to understand HTML tags, attributes, the DOM Tree, and each and every element, including new ones being added every few years, and how each one renders and behaves, would take an extremely long time, most likely several years on a fast computer, if it even were possible

3. Interactivity.

Web pages aren't just static, they change according HTML, CSS and JavaScript. Not only would you have to design your system to account for the rendering step, you would also make it have to understand the Turing Complete scripting language JavaScript, as well as the less complicated, but inherently intertwined with HTML, CSS stylesheet language. If you thought the rendering process was easy, try training a neural network to handle complicated scripting patterns.

4. New Standards

Not all HTML is equal, because of different standards. WHATWG began working on HTML5 in 2004, and browsers started to implement not long after. In 2004, there were very few examples of HTML5 sites to train your network to begin with. Sure, now it's standardized and every website uses it, but what about HTML6? When the first specification is released (probably 2017-2025), virtually no websites will use it, because no one will support it. Only when it finally becomes standard, probably in the late 2020s or early 2030s, will you have enough data to train your monstrous system of neural networks

As for AI in general, one could argue that browsers already use A.I. in their rendering process. They intelligently decide what to render (taking CSS into account), when in order to get the most efficient render time, they selectively use different JavaScript parsers on different sections of the code to optimize the speed, the whole system has been optimized on another ruleset to make rendering and interacting with a webpage as seamless and easy-to-use as possible. Your system will never be as good as what hundreds of humans have optimized over 20 years.

Trying to solve HTML rendering with Neural Networks is akin to trying to nail a nail with a screwdriver. It's just not going to work

Hope this was helpful!


Deep learning model training and processing requirement for Traffic data

Adjustments

There are a few considerations that may help with problem analysis:

Traffic data alone will produce limited predictive results.  For instance, the onset of precipitation is critical input for any production-ready predictive model.
Traffic congestion is highly chaotic in that events cascade, thus what is colloquially called the butterfly effect, is a dominant challenge.  What you can realistically achieve are what chaoticians call attractors.  They are morphological features in phase spaces and auto-correlations that are predictable under specific conditions.
Modelling traffic requires further segmentation beyond stop locations to include turn options and merge points, both of which are critical factors in free flow and congestion.
Feedback from the field will DRASTICALLY improve prediction.  Expecting the system to predict start to finish scenarios longer than a few minutes of potential congestion dynamics is unrealistic.  Inputs will need to be continuous and the closer to real time the better.
The physical relationships between position, velocity, and acceleration must be built into the parameterized model in which parameters are continuously updated.

We can assume that the system design predicts congestion for some purpose, which was not stated in the question.  It is reasonable to assume, because of the economics of transportation, that the objective is to reduce the area under the probability distribution curve for undesirable events resulting from congestion.

Lost time
Accident occurrence
Fatalities

The mechanism of control through which such objectives could be achieved can be any of these.

Notifying drivers of traffic conditions via cell towers or WiFi
Notifying drivers via dynamic sign display
Adjusting remotely configurable traffic signaling devices
Controlling gates

These controls can be to human drivers or pilots or to automated driving or piloting systems.  The context of the system is key to designing it properly, since the use cases are entirely driven by this context.

The Questions

Is this approach correct, [and] how can I improve it? — suggestions above
I am not sure if the number of inputs can be increased for better results. — absolutely yes
Which activation method will be best to utilize to get a range of conditions/event types rather than binary 1 or 0? — A ReLU or one of its derivatives may produce the best results for off line training, however it may be advisable to look into real time approaches such as Q-learning.  In either case, it is advisable to finish requirements definition, decide upon approach, and produce a general architectural diagram of the system before attending to such minutia as activation functions.
[Given the likely data set size of 200GB] can I use my professional grad laptop to process this data or if I should consider going to Big Data Processing power? — Using VLSI hardware acceleration from vendors such as Intel or NVidia is advisable.  Over the course of time, the initial expense and learning curve may be absorbed by not having to pay for additional bandwidth for PasS or SaaS services.

Regarding computing resources, it is inadvisable to begin with some of the most congested places.

Cities Definitely Too Congested

It is likely that these cities would require a super-computing platform of significant size and a multinational corporation sized budget.

Beijing
Dubai
Tokyo
Los Angeles
Chicago
London
Hong Kong
Shanghai
Paris
Amsterdam
Dallas

Commuter locations such as Melbourne, Australia; Palm Beach, Florida; or New Haven, Connecticut are examples of good choices to start.


How to implement a contextual reinforcement learning model?

In the case in which some of the states -but not all- are fully independent of the actions -but still obviously determine the optimal actions-, how could we take these state variables into account?

I think the key thing here is the caveat but not all. What you have is a fully-featured MDP (states, actions, rewards, timesteps where next reward and next state depend on current state and action). The fact that next state is only marginally affected by the current action does not prevent it being an MDP.

It would be a problem if the current state did not adequately describe this limitation e.g. if some other data, outside of observable state, decided whether action had any effect. Assuming that is not the case, then you still have a full reinforcement learning problem, but one with some unusual qualities.

You can mitigate against problems caused by algorithms that will over-estimate likely reward (caused by algorithm associating lucky state trajectories with action choice), by using "double learning" - one estimate of return is used to select maximising action, and another used to estimate the actual return from the next state. You probably will also prefer single-step learning over learning based on trajectories as most of the time your state trajectories will not contain learnable data. So Double Deep Q Networks might be a good starting algorithm to try.

If you know absolutely from inspecting a current state, that the next state is independent of the current action, and the next state could in theory be almost anything (from either the whole state space or some large subset) then you may be able to adapt the algorithm to allow for that knowledge. You would do this by altering the TD target and replacing the terms for bootstrap estimate of next state with a rolling mean over all reachable next states. In concept this is similar to Expected SARSA - and in practice if it is possible, it would go a long way to reducing variance during learning process, and may speed up learning significantly. If you know the distribution of next states, you could maybe use that, but basing it purely on samples seen should also be fine, provided you can allocate the means to correct group of states (your question implies you have some understanding of how the states will group). Note that transitions from states with no action effect on next state to states which do have an effect from the action will need careful handling - they should not be assigned to the same "mean group", but should instead affect the TD target normally when they occur.

If it is not possible for the agent to know whether it is in a state where its action affects the next state, except through experience, then you really have to use a standard RL solver.

Finally, if you have a situation where the current action does not affect next state, but the current state definitely does in all cases -  the state "evolves", maybe stochastically, independently of the action taken in most cases, and can only change to a relatively small subset of next states from any given current state - then this is best solved with a normal RL solver - again double Q Learning might be a reasonable starting algorithm in that case.


Are Q-learning and SARSA the same when action selection is greedy?

If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ in the episode we do the following:

Take action $A_t$ and observe $R_{t+1}$, $S_{t+1}$
Choose $A_{t+1}$ using $S_{t+1}$ in our policy
$Q(S_t, A_t) = Q(S_t, A_t) + \alpha [R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t, A_t)]$

Now, in Q-learning we replace $Q(S_{t+1},A_{t+1})$ in line 3 with $\max_aQ(S_{t+1},a)$. Recall that in SARSA we chose our $A_{t+1}$ using our policy $\pi$ - if our policy is greedy with respect to the action value function then this simply means the policy is $\pi(a|s) = \text{argmax}_aQ(s,a)$ which is exactly how we choose our weight update in Q-learning.
To answer the question - no, they are not always the same algorithm.
Consider where we transition from $s$ to $s'$ where $s'=s$. I will outline the updates for SARSA and Q-learning indexing the $Q$ functions with $t$ to demonstrate the difference.
For each case, I will assume we are at the start of the episode, as this is the easiest way to illustrate the difference. Note that actions denoted by $A_i$ are for actions taken explicitly in the environment -- in the Q-Learning update the $\max$ action that is chosen for the update is not executed in the environment, the action taken in the environment is chosen by the policy after the update has happened.
SARSA

We initialise $S_0 = s$ and choose $A_0 = \text{argmax}_a Q_0(s,a)$
Take action $A_0$ and observe $R_{1}$ and $S_{1} = s' = s$.
Choose action $A_{1} = \text{argmax}_aQ_{0}(s,a)$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma Q_0(s,A_1) - Q_0(S_0,A_0)]$

Q-Learning

Initialise $S_0 = s$
Choose action $A_0 = \text{argmax}_aQ_0(s,a)$, observe $R_{1}, S_{1} = s' = s$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma \max_aQ_0(s,a) - Q_0(S_0,A_0)]$
Choose action $A_1 = \text{argmax}_aQ_1(s,a)$

As you can see the next action for the updates in SARSA (line 4) and Q-learning (line 3) are taken with respect to the same $Q$ function, but the key difference is that the actual next action taken in $Q$-learning is taken with respect to the updated $Q$-function.
The key for understanding this edge case is that when we transition into the same state, the Q-Learning update will update the Q-function before choosing $A_1$. I have indexed actions and Q-functions by the episode step - hopefully, it makes sense why I have done this for the Q-functions as, usually, this would not make sense, but, because we have two successive states that are the same, it is okay.


Can we ever achieve hypercomputation using recurrent neural networks?

I presume the proof the OP is referring to can be found in this monograph by Hava Siegelmann?

In his article 'The Myth of Hypercomputation', the eminent computer scientist Martin Davis explains (p8-9) that there is nothing 'super Turing' about this formulation.

EDIT: It's looking like the claim about rational weights being super-Turing is made in this  more recent paper by Siegelmann, which introduces an additional assumption of plasticity, i.e. that weights can be dynamically updated.


Why my test error is lower then train error

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher.

That's normal behavior if you work with dropout.


How does publishing in the deep learning world work, with respect to journals and arXiv?

Let me answer your questions one by one.

Submit it to a conference

Let's start with the optimistic case. Say your paper gets accepted! You can upload your preprint on arXiv with the "arXiv.org perpetual, non-exclusive license to distribute this article (Minimal rights required by arXiv.org)". It is a non-Creative Common License that does not provide any exclusive rights to arXiv as per. This is the default permission of arXiv (which does not interfere with the license of the conferences). You can easily find papers from ICML, NeurIPS, and AAAI on their websites as well as arXiv. For, e.g., NeurIPS 2017 began on 4th Dec 2017 and this paper has the latest submission on 6th Dec 2017.

If you say your work was rejected or you are about to submit it, you need to check for the restrictions on uploading the manuscript online before submitting it to the conference. Some conference allows papers to be submitted only if they are not uploaded to sites like arXiv within say 30 days prior to the paper submission deadline.

AAAI does not have restrictions on arXiv as per their policy. ACM also allows non-profit organizations like arXiv as per their policy.
Among all the info on the major publications I could find, I found a common phrase that the paper should not be uploaded to a for-profit digital library. ArXiv is certainly not among them. The reason why editors do not oppose arXiv papers could be that there are a lot of papers that are uploaded to arXiv in order to claim to be first, so banning them would be a big loss for the editors.

Say you comply with the restrictions of the conference and have uploaded your preprint then it brings the risk that reviewers can know who the authors are. They might be biased once they know who the authors are. This works in the favor of renowned/superstars scientists of the field. The reviewer might be a Ph.D. who admires this superstar and hence would try to get it accepted. Or the author might be a known colleague of the author and would hence be biased into accepting it. On the other hand, if the reviewer has some personal bias towards the author then that may be reflected in their scores.

Upload it on arXiv directly

Only uploading to arXiv is of not much academic value because it is not peer-reviewed as you pointed out. Anyone can write an absurd paper with false results and get it published on arXiv. That is why peer-reviewed papers are important.

One reason to upload to arXiv first is to make your work safe from other people coming up with similar approaches. It is not rare at all for people coming up with similar ideas within a duration of 3 months. Deep Learning is a fast-paced domain as it is a hot research field. So getting your paper out first in the name of work in progress saves you from potential rejects from the conference pointing to a similar work done available online before you.

Submit it to a peer-reviewed journal

Same thing as a conference. You can upload your preprint after acceptance of your manuscript with the default license of arXiv. For submitting it before on arXiv needs cross-checking with the rules of the journal. For, e.g., IEEE allows the work to be submitted to arXiv as per.

How do big stars publish?

Coming to your question of how big stars publish. They either publish directly in a conference and then upload to arXiv or they first upload on arXiv, get publicity and citations and then submit to a top conference. Having the preprint available online can also bias the reviewer into accepting their research (as mentioned before). This way they reduce their chance of rejection.

Fun fact

Interestingly, if the preprints become very famous then the reviewer might get confused that your work is a derivative of a popular work which in reality is the preprint of your submitted work! In such cases, you would need to imply this diplomatically. For example, Music Transformer by the Google Brain team (which consisted of authors of the famous Transformer paper "Attention is all you need") was available on arXiv in 2018 and was already being cited by others. When it was submitted to ICLR 2019, a reviewer mistakenly took it as a derivative work of the 2018 arXiv paper and suggested a rejection of the paper. However, after further inspection, he/she realized the confusion which blew their mind! Immediately the paper was given a suitable high score. Source: https://openreview.net/forum?id=rJe4ShAcF7

Where to publish?

Unlike in core electronics and other branches where a journal is much more important than conference papers, in machine learning, conference papers are at par or better than any journal out there. Mostly the quality depends on the Impact Factor of the journal or conference. For example, NeurIPS is a top venue to publish and it's a conference and not a journal.

One reason for conferences to be so important is that they are more popular among researchers. Conferences are fast with their review as compared to the lengthy journal review. This allows researchers to meet and discuss the progress of the field with like-minded people much faster. This is crucial for a rapidly evolving field like artificial intelligence. This makes conference popular which attracts the best researchers to publish their cutting edge which in turn makes the conference more lucrative for other researchers to publish in the same venue as the big stars.

Conferences also allow people from the industry to meet the researchers and hire them. The quick meeting opportunity provided by the conference is beneficial for both the academic researchers as well as the industry for attracting talent.


Using AI to interpret the nature a specific input and use the correct model

Because of all the inputs you could give are obviously a list of number. It will only require some CNNs and fully-conncted, that classify what kind of data is it and then pass the data to the right stack of layers. The only tiny problem is that all the the things you pass into the network must always be a matrix(to handle also images) with a certain size. So even if you  pass only a tiny chunk of data you must transform it into a  fixed-size matrix and fill all the empty places with 0s, and that could bring to an accuracy lack.

But to solve that you could also try to slide over your data a tiny CNN-RNN to handle different sizes .


Is accuracy of a machine learning algorithm independent of the features?

This is a result of overfitting, I think. Your model is allowed to explain the phenomenon using any combination of variables, whether or not such a  combination constitutes a true signal.

I am not familiar with the details of a NBC, but I suspect that the model has some bias/offset/constant variable, that can be combined with your input features.

Thusly, what you see is just an expression that  the classifier has been able to combine each set of features and the constant variables in such a way that they perfectly describe the phenomenon. These combinations will be non-linear, so the logic as to how dependent the model is on any parameter is not a simple "it follows"-reasoning.

However, if you evaluate OUT of sample, you will see a much better reflection of how useful individual features are, and how bad the performance becomes when you remove the first three features.

For example, do a K-folds test (say, K=10) of the six different models, preserving 20% of the dataset as test-data.

Here you would likely see the following rankings:

(4, 5, 6, 1 ~ equal) > 2 > 3


Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.

# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.


Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?

I feel the general answer is that we want to be as efficient as possible in learning from experience.

Policy improvement here always produces an equivalent or better policy, so delaying the improvement step to gather more episodes will only slow down learning.

I would note too that often a different kind of Monte Carlo learning is used. Instead the speed of the update is typically controlled with a new hyper parameter $\alpha$, instead of keeping track of the visit counts. The Q estimate is then updated something like:

$$
Q \leftarrow Q + \alpha \left (G - Q \right)
$$

The value of $\alpha$ then lets you tune how much evaluation vs improvement happens. This is called constant alpha Monte Carlo. Often this is used as a stepping stone to introduce TD methods, e.g., in 6.1 of the Sutton and Barto book.


An ontology at its most abstract is a model of the world. It describes concepts that exist in the world and how those concepts are related.
Ontologies are similar to taxonomies. A taxonomy is a tree-like hierarchy that organizes concepts in increasing levels of specificity. What an ontology adds is a second type of link between those concepts that explains how they are connected. So a taxonomy could say $isA(DOG, ANIMAL)$ ("a dog is an animal"). An ontology may also describe something like: $chase(DOG, CAT)$ ("dogs chase cats").
Why use an ontology? It can be used to reason about the world. I subscribe to a very particular kind of language-centric ontology that isn't worth addressing here. Instead, the Semantic Web project (Tim Berners-Lee) is probably closer to what would interest you. Semantic Web uses a type of description logic, which is outside my realm of expertise. But there are tools for processing these kinds of DL and gaining "understanding" from them. To work with this kind of ontology you'll want to be familiar with the idea of Resource Description Framework triples.


Are there architectures to generate pictures from four labels? (VAEs, GANs)

This is quite a difficult task and is still an open area of research. The idea behind the GANs is to map latent features (e.g. rotation, age) to an output image, but the problem is that the source data comes from an unknown probability distribution. GANs aim to approximate it by sampling this latent feature vector from a simple known distribution (e.g. normal).
However, such a distribution usually won't fit the target distribution and hence the latent features become entangled as they will reproduce statistics of the real data (e.g. gender and beard)
There are several attempts to solve this problem. One of them is Conditional GAN. For example, with a conditional GAN it can be possible to translate a segmentation mask into an image (Pix2Pix). Another approach has been proposed in StyleGAN. The authors introduce an intermediate feature vector that the model learns to disentangle in an unsupervised manner. Each value in this intermediate represents a meaningful attribute of the resulting image, but it is still not possible to control the latent features in a supervised manner. That is, after training you should manually test each of them to identify its relations.


Can residual connections be beneficial when we have a small training dataset?

The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
It is very hard to tell, a priori, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.
In this case, you are wondering whether residual networks using skip connections might help when you have a relatively low amount of training data.

On the pro side, effects of skip connections that help correct vanishing gradients, and treat each block as learning the difference from an identity function, will still work for your problem. That means you will have some freedom to explore adding layers without worrying about the negative impacts of doing so.

On the con side, it is unlikley that you will benefit from very deep networks as there will not be enough examples to learn truly complex functions from.

You may find that adding depth, but reducing "width", i.e. the number of artificial neurons in each layer, will work.
If you have a low amount of training data and a difficult problem to solve, then residual networks are not a magic fix. The best you could hope for is a relatively stable statistical model that works on the simpler differences between your training examples. However, it may be possible that tuning a neural network by searching through different architectures will be a worthwhile exercise.
I would also suggest that for one of the good networks that you vary the number of examples used to train the network in multiple training runs, and plot a learning graph showing number of examples versus accuracy (or other metric that you may be interested in). This graph will help you decide whether collecting more training data would be worthwhile because you will have a rough estimate of the gradient for how much new training examples could improve your results.


Benchmarks for reinforcement learning in discrete MDPs

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms.

MDP vs POMDP

In Markovian Decision Processes (MDPs) the whole state space is known, this means you know all the information for your problem; therefore, you can use them to find solutions for perfect information problems or games.
Many of these games could use an MDP, some examples: 2048 and chess. Note that you must take into mind that the computational complexity grows with the number of states.
Although I could not find any benchmarks for MDPs, games with perfect information can be used to compare MDP solvers.

When a problem or game has imperfect information, you should use a  Partially Observable Markovian Decision Processes (POMDPs); in which you do not need to know the current state, but you keep track of the probabilities of being in any of the (discrete) states.

POMDP Benchmarks

Since I worked with POMDPs, I will comment some of the benchmarks researches used for discrete POMDPs (Pineau et al. (2003), Spaan and Vlassis (2004), Kurniawati et al. (2008), Ong et al. (2010), ArayaLopez et al. (2010)):

Tag: a robot and target move in a grid environment and can move one step at a time, moving has a cost, and a reward is gained if the robot is at the same position as the target (i.e. tagged it).
Two-Robot Tag: two robots attempt to catch a target, thereby sharing their observations and actions; the target tries to get away from them.
Mazes (Littman et al. (1995), Kaelbling et al. (1998), Spaan and Vlassis (2004)):

Hallway and Hallway2 are robot navigation tasks in a hallway, where the robot has only local noisy sensor information. The difficulty of hallways is it being long areas which look alike, which causes ambiguity in the localization.
Tiger-grid a two world state with a tiger being either behind the left or right door. The actions are listen, open the right or left door and there is a positive reward when opening the door without the tiger, otherwise a large negative reward.

Rock Sample: a rover explores a grid area, it knows its own position and the position of the rocks, however, it does not know which rocks are valuable. The rover can sense how valuable they are, but this sensor is less reliable when it is used farther away.

The tag game: the robot (blue) and target on a map with 29 positions and 870 states (29 for the robot, 29 + 1 (tagged) for the target).

These problems tend to be of the same size (number of states and actions) such that the results of different algorithms can be compared easily.

References:

Araya-Lopez, M., Thomas, V., Buffet, O., and Charpillet, F. (2010). A closer look at MOMDPs. In 2010 22nd IEEE International Conference on Tools with Artificial Intelligence, volume 2, pages 197–204.
Kaelbling, L.P., Littman, M.L., Cassandra, A.R. (1998).
Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2): 99-134
Kurniawati, H., Hsu, D., and Lee, W. (2008). SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Proceedings of Robotics: Science and Systems IV, Zurich, Switzerland.
Littman, M.L., Cassandra, A.R. and Kaelbling, L.P. (1995). Learning policies for partially observable environments: Scaling up. in Proc. 12th Int. Conf. on Machine Learning, San Francisco, CA.
Ong, S. C. W., Png, S. W., Hsu, D., and Lee, W. S. (2010). Planning under Uncertainty for Robotic Tasks with Mixed Observability. The International Journal of Robotics Research, 29(8):1053–1068.
Pineau, J., Gordon, G., and Thrun, S. (2003). Point-based value iteration: An anytime algorithm for POMDPs. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI),
pages 477–484.
Spaan, M. T. J. and Vlassis, N. (2004). A point-based POMDP algorithm for robot planning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2399–2404, New Orleans, Louisiana.


How can I compress the states of a reinforcement learning agent?

Compression will be lossy, some detailed features in the state will be off calculation.
A common technique might be using max-pool function or layer (before feeding to policy network if RL here is deep RL).
Max-pooling is very lossy, use some other classic compression algos such as Zip, Rar but using these classic no-loss compression is weird and extremely slow in the model pipeline.
Possible solution if allowing lossy data, commonly: Use max-pool (giving out high contrast data), average-pool (giving out blurred data).
To keep data intact, TensorFlow can compress tensors: "only sacrificing a tiny fraction of model performance. It can compress any floating point tensor to a much smaller sequence of bits."
See: https://github.com/tensorflow/compression


Why tf object detection api needs so few pictures?

I guess that they need so little data because their models are already trained on huge datasets, and they are just transferring the learning (using those pre-trained models as starting point).


Can A3C update the policy / critic on a local machine without needing to copy?

You might want to check out this paper relating to Phizaz's comment: Asynchronous Methods for Deep Reinforcement Learning (specifically search for Hogwild).


Is existence and uniqueness of state-value function at $\gamma < 1$ theoretical?

In essence, your question is about convergence of infinite series. The mathematical discipline that studies such series is hundreds (if not thousands) years old an has nothing to do with "hardware architecture".
A basic example of an infinite series is the geometric series:
$$ S = 1 + \gamma +  \gamma^2 + \gamma^3 + \dots$$
Note that the series is infinite - no one says that $\gamma^k$ "becomes zero at sufficiently large $k$". To compute the infinite sum we represent it as a limit of partial sums:
$$ S = \lim_{k\to\infty} ( 1 + \gamma +  \gamma^2 + \gamma^3 + \dots + \gamma^k)$$
The partial sum is called the geometric progression and has a well-known expression for it:
$$ 1 + \gamma +  \gamma^2 + \gamma^3 + \dots + \gamma^k = \frac{1-\gamma^{k-1}}{1-\gamma}$$
As a result, the sum of our geometric series is:
$$ S = \lim_{k\to\infty} \frac{1-\gamma^{k-1}}{1-\gamma} = \frac{1}{1-\gamma}\quad\text{if}\;|\gamma| < 1$$
In Reinforcement Learning we are dealing with similar infinite series. In partcular, when we are talking about returns:
$$R_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \dots = \sum_{k=0}^\infty\gamma^kr_{t+k+1}$$
The convergence of this is acutally discussed in the  Sutton and Barto's book:

If $\gamma < 1$, the infinite sum has a finite value as long as the reward sequence $\{r_k\}$ is bounded.

I guess, the simplest way to prove that would be to use the direct comparison test. Having $\{r_k\}$ bounded means that there exist such number $C$ so that $|r_k| \leq C$ for all $k$. So we've got the dominating series for the return :
$$ \left|\gamma^kr_{t+k+1}\right| \leq C |\gamma^k|$$
And, since the $\gamma^k$ series absolutely converges for $0 \leq \gamma < 1$, so does $\gamma^kr_{t+k+1}$ series.


Will there be some promising techniques that can make AI greener and affordable in the future?

As far as I know, green AI and red AI are very recent terms and/or research areas, but their importance will or should definitely be more highlighted in the coming years (for obvious reasons).
I don't know how many people are actively and directly researching this topic, but, in the past, I've already read a research paper on the topic, so a few people are already trying to at least raise awareness about these issues. Moreover, they also suggest that researchers report the number of floating-point operations (FLOPs) performed during the experiments, in addition to the usual performance metrics (such as the accuracy). I don't remember all the details of this paper, but I really recommend that you read it. This another possibly useful paper that people interested in this topic might want to read.
Although I'm currently not doing research on this topic, from an algorithmic perspective, improvements in the following areas will definitely contribute to greener AI

zero-, one- or few-shot learning
transfer learning
sample efficiency (in reinforcement learning)

Why? Because learning from fewer samples typically means fewer computations, so fewer emissions.
So, I think that making AI greener is definitely possible, as it's possible to use an electric car rather than a petrol/diesel car, although more research on the topics mentioned above (and other topics) needs to be done to make it practically useful.


The analysis of the dynamic behaviour of neural networks involving the application of feedback

Although you have not given the entire context but if I were to speculate I would suggest the author is simply trying to tell that Neural Networks are quite difficult to analyse mathematically due to their non-linear nature (due to the use of non-linear activation). There are many questions on this Stack about the mathematical basis of NN.

Do scientists know what is happening inside artificial neural networks?

Also forward propagation is difficult enough, and now we that we have included feedback it becomes exponentially tough. You can easily understand the difficulty posed in such analysis if you take the case of Feedback Amplifiers in electronics. In normal sound amplifiers to avoid distortion we use a negative feedback. This roughly has an inverted waveform compared to i/p signal. It adds up with the input signal and smoothens out the distortions (as opposite wavefroms of not entirely same magnitude adds up).

|W| > 1 should be the case of Positive feedback, which can cause massive variations in output due to small changes due to its self reinforcing property. The wave-forms are of the same shape and they add up, this results in an even more larger waveform in the next cycle and so on. So probably the author is speaking something along these lines.


Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?

This is mostly an implementation architecture problem, and the thing is that basically you can implement anything in the traditional setting. To do so instead of having Env<->Agent1<->Agent2, you should have Agent1<->SuperEnv<->Agent2 where SuperEnv contains Env, and simply uses the reward given to SuperEnv by Agent1 and passes it to Agent2.
I know this might seem a little counter-intuitive when comparing the implementation to the real-world problem setting, but the consistency of the RL structure (i.e. Environment that interacts with all the agent) is very important for your solutions to be easily understandable by others.


can I use a different architecture for the decoder, or will this
  introduce weird artifacts?

If you are using U-net -like architecture with skip connection from corresponding encoding to decoding layer outputs of corresponding layers should have the same spatial resolution. There is no other commonly recognized limitations on decoder architecture for convolutional networks.


It actually depends on a couple of things here -

How many output classes do you have? If you have only 2 or 3 classes, it is a very easy task for the classifier that you have built. So, it is highly possible that convergence has occurred.
As @Djib2011 mentioned already, if your input training set is not balanced and is heavier with one of the output classes (95%), then this accuracy you see makes sense but note that your model won't do well in production.
Do not try to evaluate your model on the basis of your training accuracy. Test it on data your model has never seen before and then evaluate the classification accuracy making sure that your training/testing data is not heavy with one of the classes.


What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

Combining ReLU, the hyper-parameterized1 leaky variant, and variant with dynamic parametrization during learning confuses two distinct things:

The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation — Saturation is thee loss of signal to either zero gradient2 or the dominance of chaotic noise arising from digital rounding3.
The comparison between training-dynamic activation (called parametric in the literature) and training-static activation must be based on whether the non-linear or non-smooth characteristics of activation have any value related to the rate of convergence4.

The reason ReLU is never parametric is that to make it so would be redundant.  In the negative domain, it is the constant zero.  In the non-negative domain, its derivative is constant.  Since the activation input vector is already attenuated with a vector-matrix product (where the matrix, cube, or hyper-cube contains the attenuation parameters) there is no useful purpose in adding a parameter to vary the constant derivative for the non-negative domain.

When there is curvature in the activation, it is no longer true that all the coefficients of activation are redundant as parameters.  Their values may considerably alter the training process and thus the speed and reliability of convergence.

For substantially deep networks, the redundancy reemerges, and there is evidence of this, both in theory and practice in the literature.

In algebraic terms, the disparity between ReLU and parametrically dynamic activations derived from it approaches zero as the depth (in number of layers) approaches infinity.
In descriptive terms, ReLU can accurately approximate functions with curvature5 if given a sufficient number of layers to do so.

That is why the ELU variety, which is advantageous for averting the saturation issues mentioned above for shallower networks is not used for deeper ones.

So one must decided two things.

Whether parametric activation is helpful is often based on experimentation with several samples from a statistical population.  But there is no need to experiment at all with it if the layer depth is high.
Whether the leaky variant is of value has much to do with the numerical ranges encountered during back propagation.  If the gradient becomes vanishingly small during back propagation at any point during training, a constant portion of the activation curve may be problematic.  In such a scase one of the smooth functions or leaky RelU with it's two non-zero slopes may provide adequate solution.

In summary, the choice is never a choice of convenience.

Footnotes

[1] Hyper-parameters are parameters that affect the signaling through the layer that are not part of the attenuation of inputs for that layer.  The attenuation weights are parameters.  Any other parametrization is in the set of hyper-parameters.  This may include learning rate, dampening of high frequencies in the back propagation, and a wide variety of other learning controls that are set for the entire layer, if not the entire network.

[2] If the gradient is zero, then there cannot be any intelligent adjustment of the parameters because the direction of the adjustment is unknown, and its magnitude must be zero.  Learning stops.

[3] If chaotic noise, which can arise as the CPU rounds extremely small values to their closest digital representation, dominates the correction signal that is intended to propagate back to the layers, then the correction becomes nonsense and learning stops.

[4] Rate of convergence is a measure of the speed (either relative to microseconds or relative to the iteration index of the algorithm) in which the result of learning (system behavior) approaches what is considered good enough.  That's usually some specified proximity to some formal acceptance criteria for the convergence (learning).

[5] Functions with curvature are ones that are not visualized as straight or flat.  A parabola has curvature.  A straight line does not.  The surface of an egg has curvature.  A perfect flat plane does not.  Mathematically, if any of the elements of the Hessian of the function is non-zero, the function has curvature.


The reason for robot like speech may be because tacotron uses griffin lim for vocoder, which cannot reproduce sound with perfection, often introducing robot like sound artifects.

A vocoder is a network that transforms a transform a spectrogram image back to speech waveform. Tacotron and many other speech generation neural network uses CNN to generate spectrogram instead of raw waveforms as output. Spectrogram is a lossy representation of raw audio waveform, so a perfect reconstruction of audio waveform is not possible. Griffin-Lim is a vocoder that uses algorithmic way to transform spectrogram to audio waveform, but often introduces a robot-like quality to generated waveforms. A neural network based vocoder can solve the problem. The wavenet vocoder is often used in speech generation as it can transform the spectrogram to audio with little artifects. Many new speech generation models use the wavenet vocoder as the deafult vocoder of the generation model. For a public implementation, this is a good github repository: https://github.com/r9y9/wavenet_vocoder

You can also use the newer tacotron 2 which uses the wavenet vocoder as the default vocoder. You can check it out here: https://github.com/Rayhane-mamah/Tacotron-2


In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE, ...). However, in general, you don't have to follow this alternation strictly in order to converge (in the limit) to the optimal policy. For example, value iteration (VI) is an example of a truncated policy iteration that still converges to the optimal policy.

The term generalized policy iteration (GPI) refers to all algorithms based on policy iteration, such as value iteration, that alternate in some order PI and PE, and that are guaranteed to converge to the optimal policy, provided PE and PI are executed enough times.


Is there a theory behind which model is good for a classification task for the convolutional neural network?

Ensembles aren't very popular in the field of computer vision. The main reason why this is, is that models are already so large parameter-wise that it is hard to fit multiple models in-memory for classification. Since there are effective ways of training very large models, people would rather create a larger networks if they had the capacity than averaging the results from multiple ones.

That being said, there is no reason why ensembling wouldn't have beneficial results for your task.

One way would be, as you do, to average the results of the models. This is usually used to reduce the bias of weaker models. Another way would be to use meta-modelling, i.e. create a fourth model (even as simple as a linear classifier) that will be trained with the outputs of the three CNNs as its input features. The idea is that the meta-model will learn the best way to weight the outputs of the CNNs so that, instead of them all having an equal vote (as is the case when you average them), the meta-model will learn the best way to weigh them.


Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it.

When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment.
In general I would say that there are two issues related to sample efficiency:
1 the size of the 'action'+'environment states' space 2 the exploration strategy used.

Regarding the first point, in reinforcement learning is really easy to encounter situations in which the number of combinations of possible actions and possible environment states explode, becoming intractable. Lets for example consider the Atari game from the Rainbow paper you linked: the environment in which the agent operate in this case is composed of rgb images of size (210, 160, 3). This means that the agent 'see' a vector of size 100800. The actions that an agent can take are simply modifications of this vector, e.g. I can move to the left a character, slightly changing the whole picture. Despite the fact that in lot of games the number of possible actions is rather small, we must keep in mind that there are also other objects in the environment which change position as well. What other object/enemies do obviously influence the choice of the best action to perform in the next time step. To a high number of possible combinations between actions and environment states is associated a high number of observations/interaction required to learn a good model of the environment. Up to now, what people usually do is to compress the information of the environment (for example by resizing and converting the pictures to grayscale), to reduce the total number of possible states to observe. DQL itself is based on the idea of using neural networks to compress the information gathered from the environment in a dense representation of fixed size.

For what concern the exploration strategy, we can again divide the issue in subcategories: 1 how do we explore the environment 2 how much information do we get from each exploration. Exploration is usually tuned  through the greedy hyper-parameter. Once in a while we let the agent perform a random action, to avoid to get stuck in suboptimal policies (like not moving at all to avoid to fall into a trap, eventually thanks to a greedy action the agent will try to jump and learn that it gives a higher reward). Exploration comes with the cost of more simulations to perform, so people quickly realise that we can't rely only on more exploration to train better policies. One way to boost performance is to leverage not only the present iteration but also past interactions as well, this approach is called experience replay. The underline idea is to update the q-values depending also on weighted past rewards, stored in a memory buffer. Other approaches point to computation efficiency rather than decreasing the amount of simulations. An old proposed technique that follow this direction is prioritised sweeping Moore et al. 1993, in which big changes in q values are prioritised, i.e. q-values that are stable over iterations are basically ignored (this is a really crude way to put it, I have to admit that I still have to grasp this concept properly). Both this techniques were actually applied in the Rainbow paper.

On a more personal level (just pure opinions of mine from here) I would say that the problem between RL agents and humans is the fact that we (humans) have lot of common sense knowledge we can leverage, and somehow we are able, through cognitive heuristics and shortcuts, to pay attention to what is relevant without even being aware of it. RL agents learn to interact with an environment without any prior knowledge, they just learn some probability distribution through trial and errors, and if something completely new happens they have no ability at all to pick up an action based on external knowledge.
One interesting future direction in my opinion is reward modelling, described in this video: https://youtu.be/PYylPRX6z4Q

I particularly like the emphasis on the fact that the only true thing that human are good at is judging. We don't know how to design proper reward functions, because again, most of the actions we perform in real life are driven by reward of which we are not aware, but we are able in a glimpse to say if an agent is performing a task in a proper way or not. Combining this 'judging power' into RL exploration seems to be a really powerful way to increase sample efficiency in RL.


Does TD(0) prediction require Robbins-Monro conditions to converge to the value function?

The paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo) shows (theorem 1) that Q-learning, a TD(0) algorithm, converges with probability 1 to the optimal Q-function as long as the Robbins-Monro conditions, for all combinations of states and actions, are satisfied. In other words, the Robbins-Monro conditions are sufficient for Q-learning to converge to the optimal Q-function in the case of a finite MDP. The proof of theorem 1 uses another theorem from stochastic approximation (theorem 2).

You are interested in the prediction problem, that is, the problem of predicting the expected return (i.e. a value function) from a fixed policy. However, Q-learning is also a control algorithm, given that it can find the optimal policy from the corresponding learned Q-function.

See also the question Why doesn't Q-learning converge when using function approximation?.


What approaches are there to apply AI to global economic processes?

It's currently just too complex
The different sources of information are too varied, in economics this is often referred to as a local knowledge problem, which hampers many large scale plans. Humans can react to slight differences like respecting local traditions, landscapes, history but an artificial intelligence would (currently at least) struggle not to generalise over such a large scale as a whole country's economy.
The real work in this case (and actually most 'AI' tasks) would be collecting all the necessary data. Here that part job is currently insurmountable.
Currently a human lead planned economy would do a better job.


What algorithms does stackoverflow use for classifying duplicate questions?

I do not believe that StackExchange has published precisely what algorithm(s) they use for that, so we can't tell for sure.

However, in this meta.stackexchange question, you can follow some of the efforts that were undertaken in collecting training data for training such a classifier. The post also links to the "CQADupStack: Gold or Silver?" paper which describes the analysis of such a dataset that comes directly from StackOverflow. You might be able to find interesting literature by browsing google scholars for papers that cite this paper.

There is also another meta.stackoverflow discussion on this topic, where answers link to various community-developed projects / bots for this purpose. Again, not necessarily what is actually used by the StackExchange sites, but likely similar.

Finally, there is definitely a lot of research on performing such classifications (one example found by a quick google search is "Duplicate Question Detection in Stack Overflow: A Reproducibility Study", where many other relevant publications can be found in the list of References). This again does not necessarily lead to be precisely the algorithm that StackExchange happens to use, but many relevant ones, one or more of which they might be using.


What are good alternatives to the expression "Artificial Intelligence"?

Artificial is said to derive from the Latin word "artificium" which connotes ideas such as crafting.  Thus, artificial is a correct usage, and algorithms can be regarded as "artifacts" in the context of information as opposed to physical manifestation of information (i.e. matter).

However, I agree that the use of artificial is problematic in that, should strong Artificial General Intelligence ever be achieved, there is a stigma to "artificiality" that could have implications regarding personhood.

My personal feeling is that we should be using:

Algorithmic Intelligence

which this is functional definition, and therefore more meaningful than "artificial".  Additionally, "algorithmic" is a neutral term, and provides a very accurate description of what these systems are.

In terms of what is considered "intelligent", you may want to look at the concept of Bounded Rationality.  There is no hard definition of "intelligence", just degrees of optimality in regard to decision making in a condition of uncertainty.

Because this is subjective for any problem that is not solved, modifiers are utilized, and thus we refer to AI as "strong" or "weak". These terms are also used to describe the degree to which certain types of problems (for instance a non-chance, perfect information game like Checkers) has been solved. Complexity theory will shed more light on this concept.

For more insight on "artificial", you might find this question on the philosophical origin of the Turing Test interesting, because it partly involves the meaning of a "thing". (There were multiple words for this in Ancient Greek.)


Is consciousness necessary for any AI task?

No-one knows.

Why: because it's not possible to formally determine even whether your fellow human beings are actually conscious (they may instead be what is philosophically termed a 'Zombie'). No test known to modern physics suffices to decide. Hence it's possible that you are the only sentient being, and everyone else is a robot.

Consequently, we cannot determine which tasks require sentience.

Note that the ontological status of Zombies is controversial: some philosophers of AI (e.g. Daniel Dennett) claim that Zombies are logically impossible while others such as David Chalmers would claim that a Zombie would be compelled to assert that they experience qualia (i.e. are sentient) even though they do not. Here is a very readable paper by Flanagan and Polger that also explains why a stronger neurological version of the Turing test is insufficient to detect a Zombie.

EDIT: In response to the comment about whether an objective test for distinguishing sentience from non-sentience exists:

No-one knows. What we do believe is that this would require something in addition to what modern physics can currently tell us. David Chalmers has speculated that qualia should be introduced as a new form of physical unit, orthogonal to the others in the same way that electrical charge is orthogonal to distance.

In the absence of an objective test, we have to rely on Turing test variants, which no more guarantee consciousness in the subject than they do intelligence.


More effective way to improve the heuristics of an AI... evolution or testing between thousands of pre-determined sets of heuristics?

Hmmm, I see some issues that are actually present in both of the approaches you propose.

It is important to note that the depth level that your Minimax search process manages to reach, and therefore also the speed with which it can traverse the tree, is extremely important for the algorithm's performance. Therefore, when evaluating how good or bad a particular heuristic function for move ordering is, it is not only important to look at how well it ordered moves; it is also important to take into account the runtime overhead of the heuristic function call. If your heuristic functions manages to sort well, but is so computationally expensive that you can't search as deep in the tree, it's often not really worth it. Neither of the solutions you propose are able to take this into account.

Another issue is that it's not trivial to measure what ordering is the "best". A heuristic that has the highest accuracy for the position of the best move only is not necessarily the best heuristic. For example, a heuristic that always places the best move in the second position ($0\%$ accuracy because it's in the wrong position, should be first position) might be better than a heuristic that places the best move in the first position $50\%$ of the time ($50\%$ accuracy), and places the best move last in the other $50\%$ of cases.

I would be more inclined to evaluate the performance of different heuristic functions by setting up tournaments where different versions of your AI (same search algorithm, same processing time constraints per turn, different heuristic function) play against each other, and measuring the win percentage.

This set up can also be done with two variants analogous to what you proposed; you can exhaustively put all the heuristic functions you can come up with against each other in tournaments, or you can let let an evolutionary algorithm sequentially generate populations of hypothesis-heuristic-functions, and run a tournament with each population. Generally, I would lean towards the evolutionary approach, since we expect it to search the same search space of hypotheses (heuristic functions), but we expect it to do so in a more clever / efficient manner than an exhaustive search. Of course, if you happen to have a ridiculous amount of hardware available (e.g., if you're Google), you might be able to perform the complete exhaustive search at once in parallel.

Note that there are also ways to do fairly decent move ordering without heuristic functions like the ones you suggested.

For example, you likely should be using iterative deepening; this is a variant of your search algorithm where you first only perform a search with a depth limit $d = 1$, then repeat the complete search process with a depth limit $d = 2$, then again with a limit $d = 3$, etc., until processing time runs out.

Once you have completed such a search process for a depth limit $d$, and move on to the subsequent search process with a limit of $d + 1$, you can order order the moves in the root node according to your evaluations from the previous search process (with depth limit $d$). Yes, here you would only have move ordering in the root node, and nowhere else, but this is by far the most influential / important place in the tree to do move ordering. Move ordering becomes less and less important as you move further away from the root.

If you're using a transposition table (TT), it is also common to store the "best move" found for every state in your TT. If, later on, you run into a state that already exists in your TT (which will be very often if you're using iterative deepening), and if you cannot directly take the stored value but have to actually do a search (for instance, because your depth limit increased due to iterative deepening), you can search the "best move" stored in the TT first. This is very light move ordering in that you only put one move at the front and don't order the rest, but it can still be effective.


"Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.".

Source: "Attacking Machine Learning with Adversarial Examples"

You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.

Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.


Could curiosity improve artificial intelligence?

Does this addition of curosity changes clarifai into a true AI?

As per my answer to this question, we don't know what the ingredients for a 'true AI' are. Via the Turing Test and its variants, the best we can do is "know one when we see one".

Curiosity certainly appears necessary for intelligence, though it doesn't seem sufficient - a lemming-like creature curious to see what's at the bottom of a steep cliff might not survive long enough to learn caution, even if it had the learning mechanisms to do so.

Here is some work by Schmidhuber on Artificial Curiousity.

Pierre-Yves Oudeyer has also done quite a lot of work on this and Active Learning/Intrinsic motivation.


Which humanoid robots have open-source software (or hardware)?

There are quite a few ROS (Robot Operating System) based humanoid robots. Not all of them will be open-source down to the firmware level but most will at least be to the control level. I'm not aware of any that have open-source hardware as well.

Some examples include:

Talos
Reem-C
Nao
OP3
Thormang3


Would an artificial general intelligence have to be Turing complete?

My answer is yes, but in a trivial way. The least you would expect from an intelligent agent is that it is able to execute a given Turing machine on a given input. This requires actually no intelligence, just following rules. If however, you are referring to the capability of predicting if the Turing machine will terminate on the given input, that is another matter. I don't think it is reasonable to expect such "undecidable" computational power for an agent to be generally intelligent.


Why is On-Policy MC/TD Algorithm guaranteed to converge to optimal policy?

Your reasoning is fine. GLIE - Greedy in the Limit with Infinite Exploration assumes that our agent does not act greedily all the time. As the number of samples approaches infinity all state-action pairs will be explored -> hence the policy will converge on a greedy policy. The emphasis is on "number of samples approaches infinity". Also for GLIE Monte-Carlo initial values of Q does not matter, since they are replaced after the first update.


How can I model any structure for a neural network?

You seem to be seeking an implementation of a Residual Neural Network (https://en.m.wikipedia.org/wiki/Residual_neural_network), or ResNET for short. If you want some premade networks, the module tf.keras.applications.resnet from tensorflow (do check TF's documentation) might help you.


What is the best machine learning algorithm to select best 3 variable combinations?

What you have is an instance of a local search or black-box optimization problem.
There are many AI algorithms to do this, though as commenters have noted, with only 10 binary variables, you're happier to just look at all $2^{10}$ of them.
If you had, say, 60, then looking at them all is no longer a good idea ($2^{60}$ is very large).
The simplest BBO technique is called hill climbing. It looks like this:

Pick a random subset of the variables to be true.
For each variable, determine the effect of flipping that variable from true to false in the current assignment.
Select the 'flip' that leads to the largest value of your function.
If the 'flip' has a larger value of your function than the current assignment, GOTO 2.
Otherwise, return the current assignment.

This amounts to: 'Pick an assignment at random, then keep making small changes that improve it until you can't anymore. Claim that this is the best solution.'
Clearly, this algorithm will not recover the globally best solution with certainty, but you could run it many times, and eventually, it will find it with high probability.
A complete algorithm, that will always uncover the global optimum, is simulated annealing. This algorithm has a temperature parameter, usually denoted T, which changes according to a temperature schedule. If you start T large 'enough', and decrease it slowly 'enough', then it will find the global optimum. Unfortunately, 'enough' depends a lot on the shape of the function you are optimizing, which you generally cannot observe. There are some common guesses for T, like multiplying it by, e.g. 0.9999 at each step in the search, that seem to work okay in practice.
Both of these algorithms are widely available in packages.


I don't think the AI has gotten to that point yet. Here are some of the interesting papers on the subject:

A paper was recently written that attempted to generate jokes using unsupervised learning. The jokes are formulaic: they're all of the form "I like my X like I like my Y: Z" where X and Y are nouns, and Z is an adjective that can describe both X and Y. Here are some of the jokes generated in this paper:

I like my relationships like I like my source, open
I like my coffee like I like my war, cold
I like my boys like I like my sectors, bad

How funny these jokes are is a matter of personal taste I guess.
Another paper by Dario Bertero and Pascale Fung makes use of an LSTM to predict humor from a dataset of the Big Bang theory shows. This is not generating jokes but finding out where the jokes are said in this dataset (so theoretically, the resulting labelled dataset can hopefully be used to train a model to create jokes).
Yet another paper is that by He Ren, Quan Yang. Unlike the first paper mentioned above which was unsupervised, this is a supervised learning model. Their neural network model, generates jokes such as:

Apple is teaming up with Playboy Magazine in the self driving office.
One of the top economy in China , Lady Gaga says today that Obama is legal.
Google Plus has introduced the remains that lowers the age of coffee.
According to a new study , the governor of film welcome the leading actor of Los Angeles area , Donald Trump .

My two cents:

As of this writing, it appears that Multi-layer Recurrent Neural Networks (LSTM, GRU, RNN) for character-level language models are by far the most promising way to go about it. Maybe if you find some really cool data you can come up with some funny jokes, similar to how Janelle Shane was able to generate what I find to be really funny pickup lines such as:

Are you a 4loce? Because you’re so hot!
I want to get my heart with you.
You are so beautiful that you know what I mean.
I have a cenver? Because I just stowe must your worms.
Hey baby, I’m swirked to gave ever to say it for drive.
If I were to ask you out?
You must be a tringle? Cause you’re the only thing here.
I’m not on your wears, but I want to see your start.
You are so beautiful that you make me feel better to see you.
Hey baby, you’re to be a key? Because I can bear your toot?
I don’t know you.
I have to give you a book, because you’re the only thing in your eyes.
Are you a candle? Because you’re so hot of the looks with you.
I want to see you to my heart.
If I had a rose for every time I thought of you, I have a price tighting.
I have a really falling for you.
Your beauty have a fine to me.
Are you a camera? Because I want to see the most beautiful than you.
I had a come to got your heart.
You’re so beautiful that you say a bat on me and baby.
You look like a thing and I love you.
Hello.


Is it possible to run SSD or YOLO object detection on Raspberry Pi 3 for live object detection?

No.
YOLO and SSD are based on Nvidia's proprietary CUDA technology which is not available on Raspberry simply because of the GPU vendor is not Nvidia.
Even more, there seems to be no implementation of even OpenCL for the Raspberry's GPU.
What you can do is to try port YOLO's of SSD's CNN core from CUDA to Raspberry GPU's assembler, in the way described in, for example,  https://rpiplayground.wordpress.com/2014/05/03/hacking-the-gpu-for-fun-and-profit-pt-1/


What is the difference between an observation and a state in reinforcement learning?

Sometimes observation and state overlap completely, which is convenient. However, there is no reason to expect it in all cases, and that's where interesting problems occur.

Reinforcement learning theory is based on Markov Decision Processes. This leads to a formal definition of state. Most importantly, the state must have the Markov property. Which means that for RL to work according to theory, that knowing the state means that you know everything knowable that could determine the response of the environment to a specific action. Everything that remains must be purely stochastic and unknowable in principle until after the action is resolved.

Systems like deterministic or probability-driven games, and computer-controlled simulations can be designed to have easily observable states that have this property. Games with this trait are often called "games of perfect information", although you may have unknown information, provided it is revealed in a purely stochastic manner.

In practice, real world interactions contain far too much detail for any observation to be a true state with the Markov property. For instance, consider the inverted pendulum environment, a classic RL toy problem. A real inverted pendulum would behave differently depending on its temperature, which could vary along its length. The joint and actuators might be sticky. Rotations and movement will alter temperature and friction, etc. However, a RL agent will typically only consider current motion and position of the trolley and pendulum. In this case, the observation of 4 traits is usually good enough, and a state based on this almost has the Markov property.

There are also problems where observations are not enough to make usable state data for a RL system. The Deep Mind Atari DQN paper had examples of a couple of these. The first example is that a single frame lost data about motion. This could be addressed by taking four consecutive frames and combining them to make a single state. It could be argued that each frame is an observation, and that four observations had to be combined in order to construct a more useful state (although this could be put aside as just semantics).

The second example in Atari DQN is that the pixel observations did not include data that the game was tracking but that was not visible on screen. Games with large scrolling maps are a weakness of the Atari-playing DQN, because its state has no memory of screens other than the four used for movement. An example of such a game, where Deep Mind's player did much worse than a human player is Montezuma's Revenge, where to progress it is necessary to remember some off-screen locations.

There are ways to address knowledge that there is unobserved but relevant state in a problem. The general framework for describing the problem is Partially Observable Markov Decision Processes (POMDPs). Workable solutions include adding explicit memory or "belief state" to the state representation, or using a system such as RNN in order to internalise the learning of a state representation driven by a sequence of observations.


How do I normalise/un-normalise data when loading a model?

How do I convert this to a number between 0-1 without having access to the original car data?

You save the normalisation parameters (typically an offset and a multiplier for each column), and consider that part of the model. Typically you do this when you originally scale training data.

When you want to re-use the model, as well as loading the neural network architecture and weights, you need to load the normalisation parameters in order to re-use those too and scale your inputs.

When tutorials present a self-contained neural network that loads training data, builds a model, then tests that model, all in the same process, then often this step is not shown. However, saving the normalisation data is important, and basically should be considered part of the model, even though it is not directly part of the neural network parameters or hyper-parameters.


Can strong AI now be achieved with Prolog?

The application of Prolog, the declarative programming language may or may not be closely related to AI strength.  Prolog is a language that was designed to produce a Turing Complete backward chaining general programming language.  The strength or weakness of Prolog programs, just as for programs in Scheme, Common LISP, Java or Scala with DRools rules engine, or neuromorphic libraries in Python or C++ depends on more than the language and libraries chosen.

The strength of an artificially intelligent system, from a mathematical perspective, is as ambiguous in meaning as IQ test results.  Did human kind within one decade cross some critical level where they were once stupid and became intelligent or their weak intelligence became strong?There is no evidence for such discrete transition.  Evidence indicates that DNA advancements, many of them, and cultural changes to increase intelligence capacities occurred over thousands or tens of thousands of years.

The same is true of general and narrow AI.  All intelligent systems have strengths, weakness, generalizations, and narrows. This is true whether the intelligence arises out of billions of organic neurons and the brain's complex systems of chemistry, organelle structures, and plasticity capabilities or whether the intelligence arises from designs involving ones and zeros on a collection of interconnected silicon wafers.

Classifying systems into a four quadrant system is counter productive.

Weak narrow
Strong narrow — Extremely effective systems applicable to a narrow set of usage contexts
Weak general — Systems that show adaptive and proactive capabilities over a wide range of usage contexts but demonstrating only limited effectiveness in any given context compared to a typical human
Strong general

The binary system of weak = narrow and strong = general is even more limiting because of the existence of the middle two bullet points above in human populations.

Judgments about intelligence are ALWAYS

Relative to some standard,
Time dependent, and
Context dependent.

More importantly, there is more than one or two dimensions to intelligence — possibly hundreds.

Ken Thompson and Linux Torvalds, clearly the most successful operating system writers in terms of influence on currently deployed von Neuman computers may have great difficulty with writing a song that sustains sales over decades.  Bono and Paul McCartney may have great difficulty with writing an operating system that is downloaded every minute for decades.  Both features and breadths of intelligence have value in current culture and economics.

The appreciation of the Prolog programming language by those like the author of the page referenced in the question is not without reason.  The design of the language provides an advantage for R&D over other languages in rules based contexts.

There may be value in reconsidering the value of the language in light of the significant improvements in computing platforms since earlier efforts were made to use inference to produce learning systems.  The rule type used to learn other rules were called meta-rules, and meta-rules have produced some valuable behaviors in adaptive systems based on Java components such as DRools.


Why would someone use NEAT over other machine learning algorithms?

The main difference leading to strengths and weaknesses of NEAT algorithm, is that it does not use any gradient calculations. That means for NEAT, neither the cost function, nor the activation functions of the neurons are required to be differentiable. In some circumstances - e.g. where agents directly compete, and you can select them for next generation if they win versus one or more opponents - you may not even need a cost function to optimise. Quite often the cost function can be a simple number generated from a complex evaluation of the network.

Therefore, NEAT can be used in situations where the formula for a cost function, in terms of single feed-forward runs of the network, is not very clear. It can also be used to explore activation functions such as step functions or stochastic firing neurons, where gradient based methods are difficult to impossible to apply.

NEAT can perform well for simple control scenarios, as a policy network that outputs actions given some sensor inputs. For example, it is popular to use it to create agents for racing games or simulated robotics controllers. When used as a policy-based controller, NEAT is in competition with Reinforcement Learning (RL), and has basic similarities with policy gradient methods - the nature of the controller is similar and often you could use the same reward/fitness function.

As NEAT is already an evolution-inspired algorithm, it also fits well with a-life code as the "brains" of simulated creatures.

The main disadvantage of NEAT is slow convergence to optimal results, especially in complex or challenging environments. Gradient methods can be much faster, and recent advances in Deep RL Policy Gradients (algorithms like A3C and DDPG) means that RL can tackle much more complex environments than NEAT.

I would suggest to use NEAT when:

The problem is easy to assess - either via a measurement of performance or via competition between agents - but might be hard to specify as a loss function
A relatively small neural network should be able to approximate the target function
There is a goal to assess a non-differentiable activation function

If you are looking at a sequential control problem, and could use a standard feed-forward neural network to approximate a policy, it is difficult to say in advance whether NEAT or some form of Deep RL would be better.


Which neural networks can be used only for storing and retrieving information?

Neural networks are usually trained to produce a certain output given a certain input. Often the output is a classification of the input or some other form of input description. Sometimes it is an action in a game and sometimes it is indeed stored data, a memory if you will.

In that case the input is often a part of the stored data, so the NN actually completes the given input. This setup is called autoassociative memory, Hopfield networks are an example for this.

In your example you might give this NN the first frame of the movie Avatar and it would output the complete movie. Unfortunately this would probably be absolutely crazy inefficient.


Difficult to build learning curve for a simple linear regression

Look a the CV description I just posted at SE CV.  In the first sentence there is a link to Kohavi, which explains bootstrap bias, or estimating error as a function of increasing sample size -- which is what you want.  There's also a link to my blog "Help my Neural Network Does Not Work."


Methods to tell if a question can be answered from a paragraph

This problem is very challenging since you need to evaluate the quality of the candidate answers.

In question answering, there are common steps that you need to follow. To summarise you need first to find the sentence that can answer the question and then compose the final answer.

In the first step you can measure the semantical similarity between Q and A,   this a first filter where you can use several deep learning methods. Also, you can define a threshold to validate if the pair QA are enough related.

In the second step, you must extract the answer, if the answer is a fact you can use KB extraction, or if it is a summary, list you can use other DL methods. There is also the possibility that you must infer the answer, also suitable for DL methods.

I suggest to check this paper:

Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks


Good question!

AlphaZero, though a major milestone, is most definitely not an AGI :)

AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a single problem or type of problem (such as Go and other non-chance, perfect information games.)

AGI, at minimum, must be about as strong as humans in all problems worked on or solved by humans.

AGI is often associated with superintelligence, defined as intelligence that surpasses human levels.

AGI does not necessarily imply super-intelligence, in the sense that we'd consider an android that can perform all human activities with the same capability as humans as an Artificial General Intelligence.

But technically, AlphaGo is a narrow superintelligence in that it exceeds all human performance in a single problem.


Is binary classification using CNN possible if the training data only consists of one class?

It probably won't work because of in the training of the artificial intelligence, it'll set the weights to always answer class 1, and your data will say you're right, and it'll continue forever.


How to simplify policy gradient theoram to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

When the authors write go from
$$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$
to
$$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t;\theta)\right]\;$$
they are simply taking an expectation where the only random variable is the state $S_t$. This is because, as they say in the book, the Policy Gradient Theorem is a sum over a state distribution, therefore it can be written as an expectation. The expectation is over the states wrt the state distribution; the $\pi$ subscript here does not mean we are taking expectation with respect to the action policy $\pi$, they use the notation $\pi$ to emphasise that the state distribution is induced by the current policy $\pi$ -- this is what makes REINFORCE an on-policy algorithm because the state distribution with which we take the expectation must come from the current policy $\pi$, not some arbitrary distribution.
Now, as you don't seem to understand how they go from this to the Stochastic Gradient Ascent (SGA) (NOT descent as you keep referring to it as) update rule in the book, I will explain further. As you rightly say we can go from my previous equation to the following
$$\nabla_{\theta}J = E_{\pi}\left[G_t \nabla_{\theta}\log\pi(A_t|S_t;\theta)\right]\;$$
by doing some re-arranging and noting that $G_t$ is an unbiased estimate of $q_\pi(s, a)$ when taking expectation over the actions and states. Now, this expectation we have is with respect to the actions and states; your first mistake is that you have made it an expectation over the actions without signifying that the action is now a random variable (maybe you could look up expectations and random variables to understand really what this means).
Your second mistake is by not seeing how you can use this gradient to optimise the policy parameters. As we want to maximise this objective we want to perform SGA to it; that is we want to perform
$$\theta = \theta + \alpha \nabla_\theta J\;;$$
but $\nabla_\theta J$ is an expectation. It would be costly to evaluate this at every single state-action pair that we see in an episode, thus we only perform SGA for one state-action tuple at a time. Now, you might think that this is not correct, but it is in fact an unbiased estimate as we will sample lots of state-action pairs; this kind of estimation is known as Monte Carlo Sampling. Therefore, we can write our REINFORCE update rule as
$$\theta = \theta + \alpha G_t \nabla_\theta \log \pi(A_t | S_t; \theta)\;;$$
which is exactly the update rule in the book.
Note that at the start of the chapter in Sutton and Barto they also make it clear that we wish to perform
$$\theta = \theta + \alpha \hat{\nabla J(\theta)} \;;$$
where $\hat{\nabla J(\theta)}$ is a stochastic estimate whose expectation approximates the true gradient; this is exactly what we are doing, $G_t \nabla_\theta \log \pi(A_t | S_t; \theta)$ is our stochastic estimate.


Separating spam and ham emails using the ID3 algorithm

Before thinking about implementation details such as what data structures to use, consider what you need as input for the ID3 algorithm.  You need a training set which comprises a list of categorised objects (your email, categorised as 'spam' or 'ham'), where each object is represented as a list of features (preferably binary, but not necessarily).

What features are you going to use? This could be the existence of certain words, but maybe phrases might be more useful. Does the email contain more than 5% wrongly spelled words? Are there funny non-ASCII characters in the text? Anything you think might be useful in separating the spam from the ham could be a feature.

The proper term for this representation is as a feature vector, so once you come to the implementation stage, you might want an array/list/vector to represent the feature values for each email. This set of feature vectors is then the input to your ID£ algorithm.

So, to answer your question: start by thinking what features to use, how to extract them from the data, and how to represent them effectively. The better you plan this, the easier the implementation will be.


What is difference between edge computing and federated learning?

Federated systems or Fog models, basically push computation from the active system side to server or network side processes. This is commonly used when computationally expensive services are required on limited systems (such as running some AI or augmentation occlusion processing from on phone), allowing for distributed data processing. Here is a good paper on the matter-
https://arxiv.org/abs/1602.05629
What you might be caught up on is edge computing processes and Fog both operate the same. It's just the model differentiating where the services run.


How to fight with unstability in self play?

The AlphaZero paper mentions an "evaluation" step that seems to deal with the  the problem similar to yours:

... we evaluate each new neural network checkpoint against the current best network $f_{\theta_*}$ before using it for data generation ... Each evaluation consists of 400 games ... If the new player wins by a margin of > 55% (to avoid selecting on noise alone) then it becomes the best player $\alpha_{\theta_*}$ , and is subsequently used for self-play generation, and also becomes the baseline for subsequent comparisons

In the AlphaStar they've use a whole league of agents that was constantly played against each other.


How to avoid over-fitting using early stopping when using R cross validation package caret

You are handling a very small dataset. The only way to prevent overfitting then is to choose a very restrictive model search space. The simpler the better, and you should prefer models involving some regularization. Even tuning hyperparameters will be hard, so avoid families with many hyperparameters. Neural networks are definitely a no-go, IMHO.
Cross-validation is important in this case, a.o. for tuning the regularization hyperparameter. You were right to perform the extra leave-one-out cross-validation. But remember checking the confidence intervals, which will be very broad.
I would suggest to use some prior business knowledge to further reduce the number of candidate predictor variables to 2-4. Then you can use a linear regression or a ridge regression (i.e. linear regression with L2 regularization, amount to be tuned as hyperparameter) using only these variables. If you have reasons to believe that the relation is very non-linear, you'll need to find another family.
Another possibility is to perform a data-driven feature selection (see https://topepo.github.io/caret/feature-selection-overview.html), but it is harder to properly implement and cross-validate. I wouldn't recommend it to a beginner. Remember that this step can also introduce overfit or instability.