Why most imperfect information games usually use non machine learning AI?

A heuristic search using MCTS + minimax + alphabeta pruning is a highly efficient AI planning process. What the AI techniques of reinforcement learning (RL) plus neural networks (NNs) typically add to this is a way to establish better heuristics.

  My intuition tells me that this is way harder and far more complex.

It's not actually that much more complex in concept. Replace the hand-coded heuristic with a learning engine, e.g. DQN or AC3. Train the learning engine from human expert play examples and/or from self play.

It is harder though, because there are many things that can go wrong with an NN-based estimator in a RL context. You will need to make many experiments with different hyper-parameters of the learning engine. For complex games, you may have to invest many 100s of hours of training, which you might want to compare against the end result of a similar amount of time spent refining expert heuristic systems.

For imperfect information games, you may also want to use something that can learn an internal state representation. That could be some kind of explicit belief state that you maintain like an expert system, or something that attempts to learn a good representation, such as an RNN (e.g. LSTM). This may not be necessary for a first try at an agent though, since the MCTS search will make up for some inadequacies of low accuracy heuristics.

  Is that the only reason the NN method is not used?

Up until quite recently, approaches using RL and NN were far harder to find examples of outside of academic machine learning research, and there were not any pre-written frameworks for LSTM or e.g. AC3. In the last few years, RL and NN frameworks have started to appear making an AI self-learning approach far more approachable.

I would expect that many hobby-level coders considering game-playing AI nowadays would seriously take a look at RL and NNs in order to learn robust heuristics for their game projects. However, the "traditional" search-based methods still work in conjunction with these for a completed agent.

  Has there been any research to prove how much efficiency edge would be between those 2 approaches?

For card games, I am not aware of any specific research, although I am just a hobbyist, yet to write any specific game engine more complex than tic-tac-toe.

For perfect information board games, the chess playing variant of AlphaZero demonstrates applicability of RL+NN self-play approach versus "traditional" heuristics plus search (represented by Stockfish). However, the framing of the tournament has been criticised as unfair to Stockfish, so it is not necessarily an open-and-shut case that RL is strictly better.


How does t-SNE preserves embedding orders?

You just to look at the method of T-SNE. It "computes probabilities $p_{ij}$ that are proportional to the similarity of objects ${x} _{i}$ and ${x} _{j}$". This means the same as the sentence that you have already quoted:

t-SNE (t-distributed Stochastic Neighbor Embedding) preserves embedding orders via probability distributions, whereas triplet loss works directly on embedded distances.


Doubt regarding the proof of convergence of $\epsilon$ soft policies without exploring starts

Let's first clarify a couple of details:

The policy $\pi$ we're talking about is an $\epsilon$-soft policy (defined to mean that $\pi(a \vert s) \geq \frac{\epsilon}{\vert \mathcal{A}(s) \vert}$ for all states and all actions).
We're not trying to prove equality of $v_{\pi}$ and $v_*$, but of $v_{\pi}$ and $\tilde{v}_*$, where $\tilde{v}_*$ denotes the optimal value function in this "new environment" that we're constructing.

So, "environment" is basically the "world" that our agent "lives" and acts in. You can think of it as the "rules" that we "play" by. So, you could think of the definitions of our complete state and action spaces as part of the environment, and the function that tells us which successor state $s'$ we end up in whenever we pick an action $a$ in a state $s$ (i.e. the state transition dynamics) are a part of the environment. And the function that tells us what Rewards we'll obtain in what situations is also a part of the environment. The policy $\pi$ is not a part of the environment; this is the "brain" of the agent itself.

Now, recall that here we're not interested in proving that $v_{\pi}$ moves towards the true optimal value function $v_*$ of the "real" environment. We know for a fact that we won't ever become completely equal to that, because we're forcing our policies to have exploratory behaviour by requiring them to be $\epsilon$-soft, so it would be hopeless to prove such a thing. Instead, we're interested in proving that $v_{\pi}$ will move towards whatever value function is the best one that we could possibly achieve under the restriction that we must have an $\epsilon$-soft policy.

What we do in the book here is that we slightly "transform" our environment into a new environment (i.e. we change the rules that we play by just a little bit). This is done in a clever way, such that the thing I just described that we want to prove becomes mathematically equivalent to just proving that $v_{\pi}$ moves towards (or becomes equal to) $\tilde{v}_*$. Now, if we can prove this for the new environment, we'll automatically have proven the thing that we actually wanted to prove for the "real" environment.


How agent's reasoning skills can improve its reinforcement learning?

It sounds like you are describing a synthesis of two competing ways to solve the MDP problem.

In reinforcement learning, we solve the MDP problem by having the agent move around its environment, observe rewards and transitions in response to the actions it takes, and build a model of the relationship between actions and rewards that allows it to maximize rewards.

An older approach is to give the agent facts about the world that can be encoded as logical rules. The agent then uses unification to reason about the consequences of actions within that framework of rules. The agent then takes actions that maximize the rewards it can expect, given the rules and the information at hand. A problem with this approach was that it did not work well in problem domains with probabilistic rules (i.e. X usually happens when action Y is taken).

A hybrid approach, somewhere between these two is the use of Value Iteration or Policy Iteration methods. These are so-called "model-based" reinforcement learning algorithms (although, I would tend to say that makes them something other than true reinforcement learning...). Like the older logic-based approaches, they start by writing down a series of rules that fully describe how things happen in the world, and then derive the logical consequences of those rules to compute the best actions the agent could take. Like reinforcement learning however, they are able to account for probabilistic rules and probabilistic rewards. If you had an exact description of a game of chance, you could write it down as an MDP, and then solve it exactly using these techniques.

Importantly, value & policy iteration methods are not feasible if the state and action spaces are very large (as they often are), and are really not feasible if the MDP is not known exactly (i.e. if you don't know all the rules of the game in advance). That's where reinforcement learning shines.


Why does the training time of SVMs dramatically decrease after applying dimensionality reduction to the features?

SVM complexity is $O(\max(n,d)\min(n,d)^2)$ according to Chapelle, Olivier. "Training a support vector machine in the primal." Neural Computation 19.5 (2007): 1155-1178.
$n$ is the number of instances and $d$ is the number of dimensions. I'm assuming that you have more instances than dimensions giving a complexity of $O(nd^2)$. Hopefully this explains fully why reducing the number of dimensions will reduce the training time.


Why are deep neural networks and deep learning insufficient to achieve general intelligence?

Everyone dealing with neural networks misses an important point when comparing systems with human like intelligence. A human takes many months to do anything intelligible, let alone being able to solve problems where adult humans can barely manage. That and the size of human brain is enormous compared to our neural networks. Direction might be right, but the scale is way off. Number of neurons in human brain can be matched memory-wise but the amount of parallelism to simulate it real-time cannot yet be achieved (at least for a random researcher). While a little old this might give you an idea of how much we lack the processing power.


In the multi-linear regression, how is the value of weight $b_2$ calculated?

It is calculated the same way $b_1$ is calculated.

Nearly following your notation, say your multiple linear regression function is

$H(X_i) =  b_0 + b_1x_{1,i} + ...+ b_nx_{n, i}$

for data instance $X_i=x_{1,i},...,x_{n, i}$
and weights $b_0,...,b_n$.

And say your error function is $E(X,Y) = \sum_i(H(X_i)-Y_i)^2$

where $X$ is the collection of all data points $X_i, Y_i$.

From your error function $E$, for whatever weights you have (with a gradient based method), calculate the partial derivative
$\partial E /\partial b_i$ and use this to update all of your weights at once in each iteration of your optimization routine.


Does the weight vector form imply feature space curvature?

No, it does not take into account the curvature. But, if curvature is important for you, then, it would be a good idea to look at Ricci flow and its applications in neural networks.


At a very high level, regarding evolutionary game theory and genetic algorithms, it is absolutely possible that AI could develop a state that is analogous with suffering, although, as you astutely point out, it would involve conditions that a computer cares about.  (For instance, it might develop a feeling analogous to "being aggrieved" over non-optimality in the algorithmic sense, or "frustration" at equations don't add up, or "dissatisfaction" over goals that have not been achieved.)
The robot tormented by small children at the mall can certainly be said to be "suffering" in that the children block the performance of the robot's function, but the robot is not conscious and suffering might be said to require awareness.  However, even without consciousness, this very simple robot can learn new behaviors through which it mitigates or avoids the "suffering" brought on by not being able to fulfill its function.
You definitely want to look into the concept of suffering in a philosophical context and Epicurus would be a very useful place to start.
Epicurus is directly relevant in an algorithmic sense because he uses the term "ataraxia" meaning calm, and is derived from the verb "tarasso" which means to agitate or disturb.
Ataraxia can be mathematically expressed as an equilibrium.  Tarasso can be mathematically expressed as disequilibrium.
This relates directly to Game Theory in that disequilibrium can be said to be the primary requirement of games, and to AI in that Game Theory can be said to be the root of all AI.
Ataraxia is also understood in the sense of "freedom from fear", which is temporal in that fear is a function of uncertainty as it relates to the future in a predictive sense, and involves current condition vs. possible, less optimal future conditions.
Thus fear, which is a form of suffering, is rooted in computational intractability, even where the "computer" is is a human brain.
Early philosophers such as Democritus are especially useful because they were exploring critical, fundamental concepts, many of which can now be expressed with modern mathematics.
To wit: you can't arrive at suffering until you first define "the Good" and "the Bad", which is a binary relationship in which neither term can be said to have meaning without the opposite.  (Mathematically, it can be expressed in its simplest form as a finite, one-dimensional graph.)  This understanding is quite ancient.

It is worth noting that the continuing value of the early philosophers is partly a factor of wisdom not being dependent on the volume of knowledge, demonstrated by Socrates in the idea that wisdom may be as simple as knowing you don't know something.
The ancient sages didn't have the benefit of powerful measurement tools, advanced mathematics, or scientific method, but they were very smart, and even more importantly, wise.


What is a working configuration of a neuronal network (number of layers, lerning rate and so on) for a specific dataset?

I ran a lot of randomly created networks to solve this problem, but none of the structures where able to reliably "solve" this problem.

Of course, some of them where able to solve it one time, some of them even twice but there where only one which solved it 3 times:

LearningRate: 0,510141694690167
Momentum: 0,962972165068133;
Layer/Neuron-Count: 2 (14, 9)
SigmundAlphaValue: 2;


How are artificial neural networks different from normal computer programs?

Well, a computer consists of the hardware (CPU, GPU, memory, etc.), the operational system (OS) and software/programs (OS is also a software, but it is a very special software).

A neural network is simply a program that runs on a computer. Like a calculator app. In fact, the most simple NNs are indeed just some sort of a calculator that performs matrix multiplications.


How could I use machine learning to detect text and non-text regions in scanned documents?

TextDetector, Tesseract and other open source packages implement text detection (object detection for text). There's also a pretrained Tensorflow model that does text detection. A text detector will give you the bounding boxes in your image for any text that it recognizes. In the case of Tesseract, it will also output the text (OCR is built in). So you can read the code in these packages to get ideas for your own machine learning pipeline. Basically you need both a regressor (for the bounding boxes) and a classifier (to detect whether the box contains text or not).


What are the state-of-the-art results on the generalization ability of deep learning methods?

Introduction

The paper Generalization in Deep Learning provides a good overview (in section 2) of several results regarding the concept of generalisation in deep learning. I will try to describe one of the results (which is based on concepts from computational or statistical learning theory, so you should expect a technical answer), but I will first introduce and describe the general machine learning problem and I will give a definition of the generalisation gap problem. To keep this answer relatively short, I will assume the reader is familiar with certain basic machine learning and mathematical concepts, such as expected risk minimization, but, nonetheless, I will refer the reader to more detailed explanations of the concepts (at least the first time they are mentioned). If you are familiar with the basic concepts of computational learning theory (e.g. hypotheses), you will be advantaged.

Machine Learning Problem

In the following description, unless stated otherwise, I do not make any assumption about the nature of the variables. However, I will occasionally provide examples of concrete values for these variables.

Let $x \in \mathcal{X}$ be an input and let $y \in \mathcal{Y}$ be a target. Let $\mathcal{L}$ be a loss function (e.g. MSE).

Then the expected risk of a function (or hypothesis) $f$ is defined as

\begin{align}
R[f]
&= \mathbb{E}_{x, y \sim \mathbb{P}(X, Y)} \left[ \mathcal{L}(f(x), y) \right] \\
&= \int \mathcal{L}(f(x), y) d\mathbb{P}(X=x, Y=y),
\end{align}

where $\mathbb{P}(X, Y)$ is the true joint probability distribution of the inputs and targets. In other words, each $(x, y)$ is drawn from the joint distribution $\mathbb{P}(X, Y)$, which contains or represents all the information required to understand the relationship between the inputs and the targets.

Let $A$ be a learning algorithm or learner (e.g. gradient descent), which is the algorithm responsible for choosing a hypothesis $f$ (which can e.g. be represented by a neural network with parameters $\theta$). Let

$$S_m = \{(x_i, y_i) \}_{i=1}^m$$

be the training dataset. Let

$$f_{A(S_m)} : \mathcal{X} \rightarrow \mathcal{Y}$$

be the hypothesis (or model) chosen by the learning algorithm $A$ using the training dataset $S_m$.

The empirical risk can then be defined as

$$
R_{S_m}[f] = \frac{1}{m} \sum_{i=1}^m \mathcal{L} (f(x_i), y_i)
$$

where $m$ is the total number of training examples.

Let $F$ be the hypothesis space (for example, the space of all neural networks).

Let

$$
\mathcal{L_F} = \{ g : f \in F , g(x, y) = \mathcal{L}(f(x), y)\}
$$ be a family of loss functions associated with the hypothesis space $F$.

Expected Risk Minimization

In machine learning, the goal can be framed as the minimization of the expected risk

\begin{align}
f^*_{A(S_m)}
&= \operatorname{argmin}_{f_{A(S_m)}} R[f_{A(S_m)}] \\
&= \operatorname{argmin}_{f_{A(S_m)}} \mathbb{E}_{x, y \sim \mathbb{P}(X, Y)} \left[ \mathcal{L}(f_{A(S_m)}(x), y) \right] \tag{1}\label{1}
\end{align}

However, the expected risk $R[f_{A(S_m)}]$ is incomputable, because it is defined as an expectation over $x, y \sim \mathbb{P}(X, Y)$ (which  is defined as an integral), but the true joint probability distribution $\mathbb{P}(X, Y)$ is unknown.

Empirical Risk Minimization

Therefore, we solve the approximate problem, which is called the empirical risk minimization problem

\begin{align}
f^*_{A(S_m)} &= \operatorname{argmin}_{f_{A(S_m)} \in F} R_S[f_{A(S_m)}] \\
&=
\operatorname{argmin}_{f_{A(S_m)} \in F} \frac{1}{m} \sum_{i=1}^m \mathcal{L} (f_{A(S_m)}(x_i), y_i)
\end{align}

Generalization

In order to understand the generalization ability of $f_{A(S_m)}$, the hypothesis chosen by the learner $A$ with training dataset $S_m$, we need to understand when the empirical risk minimization problem is a good proxy for the expected risk minimization problem. In other words, we want to study the following problem

\begin{align}
R[f_{A(S_m)}] -  R_S[f_{A(S_m)}] \tag{2}\label{2}
\end{align}

which can be called the generalization gap problem. So, in generalization theory, one goal is to study the gap between the expected and empirical risks.

Clearly, we would like the expected risk to be equal to the empirical risk $$R_S[f_{A(S_m)}] = R[f_{A(S_m)}]$$ because this would allow us to measure the performance of the hypothesis (or model) $f_{A(S_m)}$ with the empirical risk, which can be computed. So, if $R_S[f_{A(S_m)}] = R[f_{A(S_m)}]$, the generalization ability of $f_{A(S_m)}$ roughly corresponds to $R_S[f_{A(S_m)}]$.

Therefore, in generalization theory, one goal is to provide bounds for the generalisation gap $R[f_{A(S_m)}] -  R_S[f_{A(S_m)}]$.

Dependency on $S$

The hypothesis $f_{A(S_m)}$ is explicitly dependent on the training dataset $S$. How does this dependency affect $f_{A(S_m)}$? Can we avoid this dependency? Several approaches have been proposed to deal with this dependency.

In the following sub-section, I will describe one approach to deal with the generalization gap problem, but you can find a description of the stability, robustness and flat minima approaches in Generalization in Deep Learning.

Hypothesis-space Complexity

In this approach, we try to avoid the dependency of the hypothesis $f_{A(S_m)}$ by considering the worst-case generalization problem in the hypothesis space $F$

$$
R[f_{A(S_m)}] -  R_S[f_{A(S_m)}] \leq \sup_{f \in F} \left( R[f] -  R_S[f] \right)
$$
where $\sup_{f \in F} \left( R[f] -  R_S[f] \right)$ is the supremum of a more general generalization gap problem, which is greater or equal to \ref{2}. In other words, we solve a more general problem to decouple the hypothesis (or model) from the training dataset $S$.

Bound 1

If you assume the loss function $\mathcal{L}$ to take values in the range $[0, 1]$, then, for any $\delta > 0$, with probability $1 - \delta$ (or more), the following bound holds

\begin{align}
\sup_{f \in F} \left( R[f] -  R_S[f] \right) \leq 2 \mathcal{R}_m \left( \mathcal{L}_F \right) + \sqrt{\frac{\log{\frac{1}{\delta}} }{2m}} \tag{3} \label{3}
\end{align}
where $m$ is the size of the training dataset, $\mathcal{R}_m$ is the Rademacher complexity of $\mathcal{L}_F$, which is the family of loss functions for the hypothesis space $F$ (defined above).

This theorem is proved in Foundations of machine learning (2nd edition, 2018) by Mehryar Mohri et al.

There are other bounds to this bound, but I will not list or describe them here. If you want to know more, have a look at the literature.

I will also not attempt to give you an intuitive explanation of this bound (given that I am also not very familiar with the Rademacher complexity). However, we can already understand how a change in $m$ affects the bound. What happens to the bound if $m$ increases (or decreases)?

Conclusion

There are several approaches to find bounds for the generalisation gap problem \ref{2}

Hypothesis-space complexity
Stability
Robustness
Flat minima

In section 2 of the paper Generalization in Deep Learning, bounds for problem \ref{2} are given based on the stability and robustness approaches.

To conclude, the study of the generalization ability of deep learning models is based on computational or statistical learning theory. There are many more results related to this topic. You can find some of them in Generalization in Deep Learning. The studies and results are highly technical, so, if you want to understand something, good knowledge of mathematics, proofs, and computational learning theory is required.


What are the minimum requirements to call something AI?

It's true that the term has become a buzzword, and is now widely used to a point of confusion - however if you look at the definition provided by Stuart Russell and Peter Norvig, they write it as follows:

  We define AI as the study of agents that receive percepts from the environment and perform actions. Each such agent implements a function that maps percept sequences to actions, and we cover different ways to represent these functions, such as reactive agents, real-time planners, and decision-theoretic systems. We explain the role of learning as extending the reach of the designer into unknown environments, and we show how that role constrains agent design, favoring explicit knowledge representation and reasoning.

Artificial Intelligence: A Modern Approach - Stuart Russell and Peter Norvig

So the example you cite, "autopilot for cars/planes", is actually a (famous) form of AI as it has to use a form of knowledge representation to deal with unknown environments and circumstances. Ultimately, these systems also collect data so that the knowledge representation can be updated to deal with the new inputs that they have found. They do this with autopilot for cars all the time

So, directly to your question, for something to be considered as "having AI", it needs to be able to deal with unknown environments/circumstances in order to achieve its objective/goal, and render knowledge in a manner that provides for new learning/information to be added easily. There are many different types of well defined knowledge representation methods, ranging from the popular neural net, through to probabilistic models like bayesian networks (belief networks) - but fundamentally actions by the system must be derived from whichever representation of knowledge you choose for it to be considered as AI.


Which machine learning algorithm can be used to identify patterns in a dataset of the cache performance of a CPU?

You're basically looking for is unsupervised learning (UL).  There are a lot of UL techniques around, but I'm not sure you'll find one that does exactly what you want with no user input at all. Still, if you skim the literature on these approaches, you may well find something useful.

One option is DBSCAN, a very popular clustering algorithm that does not require the user to input an initial target number of clusters (something that most clustering algorithms do require).  But even then, you still have to give the algorithm values for epsilon (a distance used in calculating the clusters) and minPts (the minimum number of points required to constitute a "dense" region).

You might also look at self-organizing maps, an approach to unsupervised learning for neural networks.

Some other search terms that might lead you in a useful direction include "data mining" and "knowledge discovery in databases" (KDD).


In a typical situation, for the emulation of physical environments, you need to define all physical rules and forces. In the "learned emulators", they use some machine learning techniques to learn those rules by supervising and interacting (instead of formalizing all of them). In this case, they do not need any exact formulation of the physical environment to emulate it.
An instance of this simulation can be this article "Realistic Atomistic Structure of Amorphous Silicon from Machine-Learning-Driven Molecular Dynamics", i.e., a machine-learning-driven simulation instead of exact formulation of the environment.


As an AI researcher, what subjects do you find yourself referring to most often?

I think AI researcher mostly google some new papers because now there's just a crazy amount of them published. Sometimes people just forget the new concept which was introduced in a paper they read several months ago and they google that concept. Sometimes I forget some loss functions (and intuition behind them) used in a specific area like computer vision, natural language processing or audio processing. Like dice loss or contrastive losses. So these are more advanced things than pros and cons of L1 and L2 losses and how to code backprop. Usually I find the answers on https://distill.pub/ or https://towardsdatascience.com/. Have a look at those, I think they represent correctly current interests and topics for refreshment in AI research community. From my experience I can say not much statistics is used in contemporary AI research (unless you're doing research in statistical learning theory). Sometimes I google some statistical tests to prove results of my experiments are statistically significant and I think that's it.


How to train a policy model incrementally to solve a problem similar to the vehicle routing problem?

I found out a concept called 'Experience Replay', which trains a single step every time a new data sample is added instead of training to max epochs.
That is, instead of this training loop:
for i in range(max_paces):
    find action for max reward;
    add to trajectory to make inp;
    train(batch_size=len(inp), epochs=max_epochs)

Do training this way (for a single-episode ML problem, no incremental):
for i in range(max_epochs):
    reset environment;

    for j in range(max_paces):
        find action for max reward;
        add to trajectory to make inp;
        train(batch_size=len(inp), epochs=1)

Do training this way (for a multi-episode ML problem, incremental data):
for i in range(max_epochs):
    reset environment;
    inc_trajectory = get_random_past_data();

    for j in range(max_paces):
        find action for max reward;
        add to inc_trajectory to make inp;
        train(batch_size=len(inp), epochs=1)

For multiple-episode problems especially those problems with unlimited episodes, the training loop needs to forget (ie. exclude from training) old episodes which are very distant in the past, or select a random number of old episodes (consider them a batch, random batch) to be in every round of experience replay. Anyway, without eliminating some old data, the amount of training data are too much since it's unlimited number of episodes.


Alternative Loss Functions to Classification Exercise

Bucketing is usually used in industry because we are not sure what we are going to get. Some amount of observation on the distribution will give you a good idea of bucketing. But, there is caveat, if there isn't a lot of variance then bucketing won't help much.
Also, in terms of losses, since different buckets will have different number of examples, class imbalance will arise. So, focal loss is the better loss here.
I hope you get results you can write home about.


Is there some known pattern for selecting a batch of candidates for the next generation?

The most general descriptive frameworks covering what you are trying to do are:

Sequential decision making (article is a stub, but the term a good launching point to discover different wys of modelling and solving these kind of problems)

Optimal control

These put some context around your problem, and might give you some pointers. For instance, reinforcement learning is an alternative approach to the evolutionary system you are trying to build.
The specific AI system you appear to be building is a genetic algorithm, and more specific still you are attempting to find a neural network that is optimal at a task by searching for the best network using a system of population generation, selection and mutation which repeats.
There are lots of ways to set up a system like this, so your approach is not necessarily wrong. However, I think there are two key things that would improve what you have built so far:

Use a fitness function for selection. Score each car, perhaps by how far it got before crashing when the episode ends. To reduce luck factor on random courses, you could make this score the mean result from e.g. 3 different courses (it is not necessary, but may address your concern that selection is too random in your case). Select some fraction of top scoring cars, or look into other selection approaches - e.g. weighted selection based on fitness score or ranking.

Add "sex", more properly known as genome crossover between selected population members. Mutating individuals is limiting because it silos improvements to a single line of ancestry - if there are two good mutations found at random you rely on that single line finding both of them. Whilst crossover allows sharing of good mutations between lines, making it much more likely that two good mutations will end up in the same individual.

There is a framework called NEAT which covers the issues above plus has other features useful for evolving neural networks. It often does well at control scenarios like the one you are considering. You may want to look into it, if your focus is mainly on solving the control problem. However, it is relatively advanced from where you are, so if your current focus to learn by building from scratch you may get more initially from implementing fitness functions and crossover yourself.


How to deal with a huge action space, where, at every step, there is a variable number of legal actions?

One way I can think of is to redefine "actions" in a game to make them more fragmented, in such a way that a player has multiple actions per turn. In chess, for example, we can define an action as choosing a tile from which to move, or choosing the motion from the chosen tile, as 2 separate actions.

As an example a turn might consist of the following two actions:

Choose E4
Move forward 2 spaces

That way there are 64 + 73, rather than 64 * 73 possible actions. The transition model would indicate that it's still the same player's turn after a "tile selection" action is done.

Of course, this would require increasing the state space in such a way that you can determine which action is legal. So there's a difference between a board state where nothing is "selected" and the same board state where one tile is selected by one player. In the chess example, this would require 2 more boolean CNN layers, one for each player, indicating which tile (if any) is "selected".

I never tried this myself, and I imagine that this might make the learning slower and more difficult, since it requires a deeper tree in the MCTS for the same set of actions.


How do I know if my dataset is ready for a machine learning model?

Before jumping to modeling, there are a few tasks a data scientist (or ML/AI practitioner) must do:

Ideation (or hypothesizing): Before applying any modeling approach, we need to ask the right questions. We must clearly mention our assumptions and declare how we want to measure the effectiveness of the pipeline. Note that, some tools/algorithms might not fit to the made assumptions or may not lead to the best values in the defined metrics. So, the pipeline must be designed in a way it serves the purpose of answering the defined questions.
Data Cleansing: Real-world data sets are usually not clean. They have all sorts of data issues such as missing value, duplicates, outliers, wrong measurements, fragments, inconsistency, etc. Most of the ML techniques are sensitive (to different extents, of course) to such issues. Therefore, the data should be cleaned before any modeling can be done.
Data Wrangling (or Feature Engineering): In many cases, the gathered data (even cleaned) is not immediately suitable for any modeling/analysis. For example, we may need to convert the documents of a text corpus to vectors of numbers (via TF-IDF or Embedding techniques) before being able to apply a text classifier simply because our classifier only takes numeric data. Converting measurements to other units, breaking addresses to their components, converting times and dates to different formats or timezones are just a few examples of data wrangling tasks (in a broader context, feature engineering may also refer to dimension reduction or feature selection/projection).
Exploratory Data Analysis (EDA): To do the cleansing, understanding the data set features, and ideation, we often need to explore the given data set using visual (e.g., dashboards and diagrams) or summary statistics tools.

Disclaimer: I have no business interest with Udemy. The links are just shared because @pedro-de-sá mentioned they take some courses from Udemy.


Extracting Descriptors and feature points for 3d mesh

The work was done through the following:

Extract feature points using a Detector
Extract the descriptor for these feature points
Make matching using similarity measure between two descriptors from two meshes.


Is there any way, in which we can monitor what's going on the screen in android to generate a model from it?

The software that you require to provide your AI agent remote access is called Virtual Network Computing (VNC). A VNC is a desktop sharing system that transmits keyboard and mouse movements, over a network connection, from the server (the AI) to the client (training environment).

This allows your AI agent to use a computer like a human does, which is by looking at the screen pixels and operating a virtual keyboard and mouse. Infact VNC's are used conventionally for remote technical support and accessing files on one's computer remotely.

An excellent use case example is Open AI's Universe platform. Alongside Gym, which is Open AI's toolkit for developing RL algorithms. Universe launches programs behind a VNC remote desktop, which enables the training of AI agents by only showing them the screen pixels and allowing them to operate a virtual keyboard or mouse.

Since you want to train your AI agent on an Android based training environment, my suggestion is that you consider installing either Real VNC (android client) or Droid VNC on your Android device. You can then install the VNC server software on your training machine (i.e. an nvidia docker instance). Your AI agent will now have full access to the training environment screen for reinforcement learning.

I also recommend that you look at this similar question on the network.

https://askubuntu.com/questions/414189/how-to-remotely-control-my-android


Appropriateness of 3D Convolutional Neural Network for segmentation of medical image data

In CNN, different height/width are not a problem. You can choose one specific size for your model and pads zero around the slices to make them of equal size.

No, your assumption is not correct. You are doing end to end training, so it does not matter, which sequence are you following.


It should be mentioned that RELU is the current activation function standard. But to answer your question:
The importance here is that it is very common to have normalized your data (e.g. using batch normalization), then the data is centered around 0.

As @DuttaA commented, look at this answer from Cross-Validated:

  Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].

And

  The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.


No, here is why. No approach can simulate the mind with 100% accuracy. a major notion that AI theorist refuse to note is that you cant take an orange and by virtue of technology turn it into an apple lets apply the same logic here.

neurons are temporary things in our brains, daily we are trimming our brains and growing our brains, in order to "Engineer" a human Brain a probability whatever or not that neuron would die or not would have to be present. But that probability is flawed as well because, Nature is not math. In this case we are trying to apply something to Math that is unpredictable. Math is something we have used to try to rationalize the way nature works but it out right fails. If nature is completely math then predicting a hurricane would be easy.

So to answer your question, no because a Neuron is more then a light switch, it is not mathematical in nature. their is no universality in everyone. plus their is more variables that you would have to account for that are way beyond a simple decision and a simple mathematical formula.


What is the difference between the heuristic function and the evaluation function in A*?

The evaluation function, often denoted as $f$, is the function that you use to choose which node to expand during one iteration of A* (i.e. decide which node to take from the frontier, determine the next possible actions and which next nodes those actions lead to, and add those nodes to the frontier). Typically, you expand the node $n$ such that $f(n)$ is the smallest, i.e. $n^* = \operatorname{argmin}f(n)$.
In the case of informed search algorithms (such as A*), the heuristic function is a component of $f$, which can be written as $f(n) = g(n) + h(n)$, where $h(n)$ is the heuristic function. The heuristic function estimates the cost of the cheapest path from $n$ to the goal. Just for completeness, $g(n)$ is the actual cost from the start node to $n$ (which can be computed exactly during the search). In the case of uninformed search algorithms, you can actually view the evaluation function as just $f(n) = g(n)$, i.e. the heuristic function is always zero.
So, you're right.
For more details, you can take a look at section 3.5 (p. 92) of the book Artificial Intelligence: A Modern Approach (3rd edition), by Norvig and Russell (you can find the pdf online).


How to train the NN of simple agents given a reward system?

You might be able to glean what you want from Chapter 13 or Sutton & Barto's Reinforcement Learning: An Introduction, which deals with policy gradient algorithms, and includes pseudocode for a variety of agents based on linear approximation using softmax regression. From your description, you appear to be using - or should consider - softmax regression for the policy function.
Probably a good bet using your setup will be REINFORCE with baseline, which does not add requirements to track anything more than your current weights, sum of rewards seen after each state & action pair, and mean rewards (or other baseline). It does require an episodic environment though, such as a game that ends.
Essentially REINFORCE and its variants require you to figure out the return (sum of rewards received after each action to the end of the episode) and perform a gradient step update multiplied by that return (minus a baseline which can help with stability).
With REINFORCE and a discrete distribution, the gradient due to action selection is the same as supervised learning for multi-classification evaluated as if whichever action the agent took was the correct one. When you multiply this gradient by the return as well, it will make steps in the direction of better action choices larger. Regardless of the fact that you may train towards selecting many different actions, the better actions will win the race in the end due to this weighting, and have the highest probability of being selected after a large number of training episodes.
A multi-agent problem may bring you some problems, such as instability due to dependency of performance on what the other agents are doing. It is not possible to say how much of an issue this will be. However, I would advise not jumping straight into your main problem, and try some reinforcement learning methods with a single agent on toy problems first. If you build up to solving the problem you are currently facing, then you will be much more confident in how you have implemented the learning. There are plenty of stepping-stone projects and pre-built environments to solve if you search for them. A good starting point is OpenAI's Gym, which presents a standardised set of toy - and not so toy - problems that you can test any learning algorithm against.


Steps for final Logistic Regression Modal

In general these steps are correct, but are few clarifications would be good. For supervised learning, you can:

pick model to use
split data into training and test set
train model, or determine weights by optimization of our error function (inputting in training data)
use trained model to predict on test data
utilize metrics of interest to evaluate our model (e.g. MSE for continuous outcomes, AUC for binary)

This however is a very elementary pipeline to train a supervised machine learning model. There are many other steps taken to ensure our model is as robust as possible (like cross validation, feature selection, hyperparameter optimization, model selection...). I would strongly recommend learning about these other steps and seeing how they integrate with the above pipeline.

For unsupervised learning, the process depends a lot on the task, but in general it involves optimizing some objective function using the data provided. This is usually done using all the data in one go (not split into training and test) unlike supervised learning. This can be seen more clearly with examples, take methods like PCA and k clustering; the steps taken to apply these methods very much depend on the method used.

Going further, I would recommend Andrew Ng's coursera course on machine learning. It is very accessible and a great introduction to this field, covering both unsupervised/supervised learning.


Why does C++ seem less widely used than Python in AI?

You don't need a powerful language for programming AI. Most of the developers are using libraries like Keras, Torch, Caffe, Watson, TensorFlow, etc. Those low level libraries are highly optimized and handle all the tough work. They are built with high-performance languages, like C, C++. Python is just there for high level task like describing the neural network layers, load data, launch the processing, and display results.

Using C++ for high level task instead of Python would give barely any performance improvement, but it would be harder for non-developers as it requires to care for memory management. Also, several AI people may not have a very solid programming or computer science background.

Another similar example would be game development, where the engine is coded in C/C++, and, often, all the game logic scripted in a higher-level language.


What topologies support recognition of action sequences?

This is an old area of AI called "Plan Recognition", which has about 3.5 million results in Google Scholar.

A lot of the modern work is done with classical search techniques coupled with expert domain knowledge, or related reasoning concepts like Hierarchical Task Networks.

I'm not aware of or able to find recent research using deep neural networks for this problem, but I think there are some data-drive approaches to the related work in video-game player modeling.


How can we compare, in terms of similarity, two pieces of text?

There are more than 1 way of doing this:

You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link.
You can convert them into 2 vectors using doc2vec and find the similarity between the vectors using cosine similarity.
Siamese networks are something similar to what you are asking. They are neural nets that use distance metric for learning rather than a loss metric.

I don't understand why do you want to use neural network for comparing two text pieces. Generally comparisons are done by some of the distance metrics not by neural network.


How does LSTM in deep reinforcement learning differ from experience replay?

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques?

Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but a history of observations might give better information. Another way is to learn a Hidden Markov model. Both of these approaches build an internal representation that is effectively considered part of the state when making a decision by the agent. They are a way to approach solving POMDPs.

You can consider using individual frame images from Atari games as state as a POMDP, because each individual frame does not contain information about velocity. Velocity of objects in play is an important concept in many video games. By formulating the problem as a POMDP with individual image inputs, this challenges the agent to find some representation of velocity (or something similar conceptually) from a sequence of images. Technically a NN may also do this using fixed inputs of 4 frames at a time (as per the original DQN Atari paper), but in that case the designers have deliberately "solved" the partially observable part of the problem for the agent in advance, by selecting a better state representation from the start.

Experience replay solves some different problems:

Efficient use of experience, by learning repeatedly from observed transitions. This is important when the agent needs to use a low learning rate, as it does when the environment has stochastic elements or when the agent includes a complex non-linear function approximator like a neural network.
De-correlating samples to avoid problems with function approximators that work best with i.i.d. data. If you didn't effectively shuffle the dataset, the correlations between each time step could cause significant issues with a feed-forward neural network.

These two issues are important to learning stability for neural networks in DQN. Without experience replay, often Q-learning with neural networks will fail to converge at all.

  Can they work together?

Sort of, but not quite directly, because LSTM requires input of multiple related time steps at once, as opposed to randomly sampled individual time steps. However, you could keep a history of longer trajectories, and sample sections from it for the history in order to train a LSTM. This would still achieve the goal of using experience efficiently. Depending on the LSTM architecture, you may need to sample quite long trajectories or even complete episodes in order to do this.

From comments by Muppet, it seems that is even possible to sample more randomly with individual steps by saving LSTM state. For instance, there is a paper "Deep reinforcement learning for time series: playing idealized trading games" where the authors get a working system doing this. I have no experience of this approach myself, and there are theoretical reasons why this may not work in all cases, but it is an option.

  If they can work together, does it mean that the state is no longer a single state but a set of contiguous states?

Not really, the state at any time step is still a single state representation, is separate conceptually from an observation, and is separate conceptually from a trajectory or sequence of states used to train a RNN (other RL approaches such as TD($\lambda$) also require longer trajectories). Using an LSTM implies you have hidden state on each time step (compared to what you are able to observe), and that you hope the LSTM will discover a way to represent it.

One way to think of this is that the state is the current observation, plus a summary of observation history. The original Atari DQN paper simply used the previous three observations hard-coded as this "summary", which appeared to capture enough information to make predicting value functions reliable.

The LSTM approach is partly of interest, because it does not rely on human input to decide how to construct state from the observations, but discovers this by itself. One key goal of deep learning is designs and architectures that are much less dependent on human interpretations of the problem (typically these use feature engineering to assist in learning process). An agent that can work directly from raw observations has solved more of the problem by itself without injection of knowledge by the engineers that built it.


When is content-based more appropriate than collaborative filtering?

Some of the cases content-based filtering is useful is:

Cold-start problem: it happens when no previous information about user history is available to build collaborative filtering, so in this case, we offer to the user some items then recommend based on the similarity between these items and other items in the dataset alternate of recommending any items that maybe not with the user taste.
Transparency: collaborative method gives you the recommendation because some unknown users have the same taste as you that cause a problem if your data is biased towards one taste makes new user haven't enough similar users have the same taste, but the content-based method can tell you they recommend the items based on what features, that helps you to determine which factors affect in the recommendation.


Why is the derivative 0 if the policy is deterministic?

First, a small clarification; they do not say that "the gradient of a deterministic policy is 0" (I'm not sure what, if anything, "the gradient of a policy") would mean. They are talking about how "this gradient" (referring to the gradient on the slide visible in the video) is 0 in the case where the policy ($\pi$) is deterministic.

Now, here is the gradient that they are discussing in the video:

$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right)$$

In this equation, $\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t})$ denotes the probability of our policy $\pi_{\theta}$ selecting the actions $\mathbf{a}_{i, t}$ that it actually ended up selecting in practice, given the states $\mathbf{s}_{i, t}$ that it encountered during the episode that we're looking at.

In the case of a deterministic policy $\pi_{\theta}$, we know for sure that the probability of it selecting the actions that it did select must be $1$ (and the probability of it selecting any other actions would be $0$, but such a term does not show up in the equation). So, we have $\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) = 1$ for every instance of that term in the above equation. Because $\log 1 = 0$, this leads to:

\begin{aligned}
\nabla_{\theta} J(\theta) &\approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
%
&= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log 1 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
%
&= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} 0 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
%
&= \frac{1}{N} \sum_{i=1}^N 0 \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
%
&= 0 \\
\end{aligned}

(i.e. you end up with a sum of terms that are all multiplied by $0$).


There is much discussion in philosophy about inner language and the ability to perceive pain (see Pain in philosophy article). Your question is in the area of philosophy and not science. If you define emotion as some state then you can construct simple automata with two states (emotion vs no-emotion). It can be a very complicated state with degrees of truth (percentage of emotion).

Basically, to mimic human emotion you need to make a living human-like organism, and still with todays understanding and technology you will not be able to recognize emotion in it. The only thing you can do is trust when it says "I'm sad". Now we are in the area of the Turing test, which is again philosophy, and not science.


The problem might not be caused by your loss function. Deep learning models tend to be computationally demanding. Mobile devices are not usually prepared for handling models with high throughput. That being said, you might try to:

Prepare smaller model - less layers, less computation
Mobile models optimization - Google provides some materials on optimizing Tensorflow models for mobile inference:

https://www.tensorflow.org/mobile/prepare_models
https://www.tensorflow.org/mobile/optimizing


What is the need for so many filters in a CNN?

Then how do each filter differ by? Is it in hovering over the input matrix? Or is it in the values contained by filter itself? Or differs in both hovering and content?

The filters (aka kernels) are the learnable parameters of the CNN, in the same way that the weights of the connections between the neurons (or nodes) are the learnable parameters of a multi-layer perceptron (or feed-forward neural network).
So, the value of these filters is not fixed or pre-determined, but will depend on how you train the CNN, i.e. the learning algorithm, the objective function and the data. If you use gradient descent as the learning algorithm, you will be minimizing a loss (aka cost or error) function (e.g. the cross-entropy, in the case of classification problems). To do that, you need to find the gradient of the loss function with respect to the filters. You then apply a step of gradient descent (i.e. you add a scaled version of the gradient of the loss function with respect to the parameters to the parameters), so that this loss decreases.
To answer your question more directly, the only thing that usually changes is just the value of the filters. The convolution (or cross-correlation) operation is the same for all filters.
Why do you use more than one filter? The usual explanation is that each filter, when convolved with the input, will extract different features from it, and the specific features that they will extract will depend on the specific values of the filters, which, in turn, depend on the data, so we can say that CNNs are data-driven feature extractors. If you are familiar with image processing techniques, then you know that different filters, when convolved with the same image, can have different effects (e.g. blurring or de-noising).


Recent work on “Oracular” systems such as Eliza?

Tariq's comment hints at this, but this is still in some sense a very mainstream idea.

Check out the transcripts for this year's Loebner Prize. The winner (called Tutor) is again making use of Eliza-like deflections. Some of the other candidates try to use seduction (Columbia), or speaking in a flighty or insistent way (like 2017's Rose).

In all cases, the bots are relying on humans to read meaning into ambiguous or vague statements, which we seem to be hardwired to do. Kind of neat!


What are acting as weights in a convolution neural network?

There are many resources that answer your question, but, given that you're apparently new to machine learning (ML), deep learning (DL), and neural networks (NN), let me provide a simple answer that should clarify your doubts.
The term weight in the context of ML, DL, and NN is a synonym for parameter (sometimes, in some contexts, such as linear regression, it is also known as coefficient), which can be constant (i.e. do not change during, for example, learning/training) or learnable (i.e. change during the training/learning process). In a feedforward neural network (FFNN), with and without recurrent connections, the weights are the numbers on the connections between neurons in different or same layers. They are called weights to emphasize that their role is to weigh the effect of one neuron on the other.
In a CNN, the weights are the kernels/filters of the CNN, i.e. the matrices that you use to perform the convolution (or cross-correlation) operation in a convolutional layer. So, given that CNNs perform an operation that seems to be different than the linear combination followed by the non-linear activation function in FFNNs, you could think that the weights in a CNN are not very similar to the weights in FFNNs. However, this is not fully true, as CNNs can be viewed as FFNNs (with some specific structure): maybe this answer will provide you more info about this topic. So, your notes are right!
In my opinion, the best way to understand the role of the kernels (i.e. weights) of a CNN and all the details behind the convolution (or cross-correlation) operation is to first study some traditional computer vision and image processing techniques, so maybe you should pick a book that covers them.
In this answer, I briefly try to describe what a CNN is and what it can be used for. There are other answers on this site, such as this, this, and this, which you may want to read, if not now later, to understand even more all the details of CNNs. A  decent reference book that covers CNNs (and other DL topics) is Deep Learning by Ian Goodfellow et al.


Do I need to denormalise results in linear regression?

Okay, I figured it out by myself, simply by trial and error:

When you normalize your training and test data, you also need to normalize the input you want to have a prediction for. You will also need to denormalize the result, to get a reasonable prediction.

Specifically, in my case, this means:

First I calculated the mean and the standard deviation, and then subtracted the mean and divided by standard deviation, to normalize my training and test data.
So to normalize the input I want to have a prediction for, I also subtract the mean and divide by the standard deviation.
Finally, when I get a result, I multiple by the standard deviation, and add the mean.

This way, I get reasonable results


How come that the addition of features can decrease the performance of a neural network?

Additional features can also cause overfitting if they have low or misleading information.

Consider the following problem:

$X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.

Suppose that the real dataset was generated from the relationship:

$Y = X$, with a probability of 0.2 of adding or subtracting 1.

A reasonable model estimate is $Y = X$. Note that no model can fit this data perfectly, because the two 3 inputs map to different outputs.

Now, suppose we add a new feature: a random number between $0$ and $10$:
$W = [1 ,5, 2, 6, 3]$

It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:

$g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.

$g(W) = 0$ otherwise.

and define a new prediction: $Y = X - g(W)$.

This happens to produce a perfect fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.


What fast loss convergence indicates on a CNN?

Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph.

Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.


Where can I find short videos of examples of RL being used?

You can take some ideas from this YouTube video .
In addition, you should consider that page which is about Deep Reinforcement Learning used in a game (Pong from Pixels) .


Can we create AI to not only recognize itself, but to recognize other AI systems as well?

In the abstract, mechanisms for self-recognition (I personally prefer the phrase 'metacognition', since it carries fewer spurious associations) and recognition of intelligence in other systems can be considered to be pretty much equivalent.

In fact, both can be characterized in the standard percept/action framework: the task in both cases involves (however coarsely) classifying/predicting the behaviour of a black box system.

Such tasks can be universally characterized in terms of frameworks such as Solomonoff Induction or (more recently) AIXI.


How does the optimization process in hindsight experience replay exactly work?

Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments


How do I label images for deep learning classification?

If each photo is intended to show a photo of weed or crops you should give one label. If your task is different where you also try to localize weed or crops in the image, then you need to label accordingly. My understanding is you are trying to do the first case, therefore, there should be one label for each image.


Autoencoder for color images in Keras backed by MXNet

It was not about the model, but about ImageDataGenerator. I found the answer here: https://stackoverflow.com/a/51673998/493080


What is the difference between ConvLSTM and CNN LSTM?

After doing a bit of research I found that the LSTM whose gates perform convolutions is called ConvLSTM.

The term CNN LSTM is loose and may mean stacking up LSTM on top of CNN for tasks like video classification

Reddit thread discussing this


Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?

No, there is no computational advantage of the second method over the first, if you neglect the computational requirements for the calculation of $\sigma$ and $\mu$.

We generally use the first method for better results. This is because if you separate your dataset into train and test data, then you may normalise the train data perfectly between $0$ and $1$ by taking $max$ and $min$ from the train dataset. Your algorithm will only see data between $0-1$, but there is no way to ensure normalising the test data on the same $max$ and $min$ will result in the data being between $0$ and $1$..

So a better way is to use the Gaussian Distribution method, as the only assumption it makes is the data distribution is Gaussian which is just about true for any phenomenon (after some functional transformations which might be handled by the ML algorithm). I answered a similar question here.


What is the meaning of Model(s, a) in the prioritized sweeping algorithm?

I think pseudocode was made for tabular case with an assumption of deterministic environment. $Model(s, a)$ would then be a table with information of the next state and reward after taking action $a$ from state $s$. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.


Does "transition model" alone in an MDP imply it's non-deterministic?

Yes, in which case, it will be more like a search problem, if it is not POMDP and have finite number of states. Or you can use the same framework (used for POMDP) with constrained (deterministic) transition matrix modeling for model-based systems.
If you think about it after you train any model/agent with an MDP modeling, during test time the optimal strategy is generally deterministic, i.e., given a feature/state you will take a particular action even if it has more than one non-zero element in each row of the transition matrix.


Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping

So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter?

My guess would be that your intuition is correct, and that it's mostly introduced just to clarify exactly which "version" of our value function approximator is going to be used in any particular equation. In previous chapters, which discuss single-step update rules, I guess the authors assumed there was less potential for confusion, and therefore no need to clarify this. Without the clarification, some people might for instance wonder if we should use $V_t$ for our value estimates of an $n$-step return $G_{t+n}$, regardless of how large $n$ is.

  However, I think this introduced a technical error, because suppose we just learned from an 10-step episode ($T=10$), using $n = 3$. Then the latest estimate of $v_\pi$ is $V_{T-1} = V_{10 - 1} = V_{9}$. Then at the next episode, the first time $V_{t + n}$ is used to inform a target update, it will be for $\tau = 0$ (from the pseudo-code), which implies $t - n + 1 = 0$, so $t = n - 1$, that is, $V_{t+n}=V_{n-1+n}=V_{2n-1}=V_5$, which is not the most up-to-date estimate $V_9$ of $v_\pi$.

Once we start considering a situation with more than a single episode, the $V_t$ notation becomes quite confusing. You should read $V_t$ as "the value function approximator that we have available at time $t$ of the current episode". So, if we were to use the symbol $V_0$ within the context of a second episode, that would be identical to what was referred to as $V_T$ in the context of the first episode. The $V_t$ notation can be convenient if we're thinking about our equations with our minds in "math-mode", but becomes highly confusing once we start thinking about practical implementations involving multiple episodes -- this is probably why they did not include it in the pseudocode.

If you really wanted to use the subscript-notation in the pseudocode, you'd have to add an extra term in the subscript that adds up all the durations of all previous episodes. If we then try to work out your example situation, we'd run into another problem though... we'd want to use $V_{t+n+T} = V_{2n-1+T} = V_{15}$ at the first iteration where $\tau = 0$ in the second episode. But, across the two episodes, only $13$ steps have passed, so this does not yet exist! You run into the same issue if you try to work out what happened when $\tau = 0$ in the first episode: applying exactly the same reasoning as in your quote, we would have wanted to use $V_5$ after only $3$ time steps passed in the first episode.

The problem here is that you're trying to use the variable named $t$ in the pseudocode as the subscript for $V$. To get a better idea of what's going on here, let's loop back to the previous page and examine the definition of the $n$-step return:

$$G_{t:t+n} \doteq R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1} R_{t+n} + \gamma^n V_{t+n-1} (S_{t+n}).$$

Ok, we've got that. Now, let's take another look at the update rule in which we use this quantity:

$$V_{t+n}(S_t) \doteq V_{t+n-1}(S_t) + \alpha \left[ G_{t:t+n} - V_{t+n-1} (S_t) \right].$$

Ok. So $V_{t+n-1}$ appears three times in the update rule. Two times explicitly, estimating the value of $S_t$, and once more "hidden" in the definition of $G_{t:t+n}$, where it is used to estimate the value of $S_{t+n}$. Note very carefully what it is that this update rule is doing; it's updating the state estimate of $S_t$. If you now look at the pseudocode again, you'll see a comment on the line where $\tau$ is computed: ($\tau$ is the time whose state's estimate is being updated)!

What this means, is that in the pseudocode, you should be using $\tau$ as the subscript for $V$! If you do that, it'll at least be correct for the first episode. In the pseudocode, the update rule looks like:

$$V(S_{\tau}) \gets V(S_{\tau}) + \dots$$

Plugging in the subscripts from the mathematical definition leads to:

$$V_{\tau + n}(S_{\tau}) \gets V_{\tau + n - 1}(S_{\tau}) + \dots$$

Since the pseudocode defines $\tau = t - n + 1$, we can substitute above:

$$
\begin{aligned}
V_{t - n + 1 + n}(S_{t - n + 1}) &\gets V_{t - n + 1 + n - 1}(S_{t - n + 1}) + \dots\\
V_{t + 1}(S_{t - n + 1}) &\gets V_{t}(S_{t - n + 1}) + \dots
\end{aligned}$$

and now it should make sense again from a practical point of view. At every time step $t$, where $t$ measures number of steps of experience that we have simulated, we simply use the latest value function $V_t$ we have available at that time for bootstrapping in the update rule. When $t + 1 < n$, $S_{t - n + 1}$ is undefined. In these cases, the above update rule doesn't work, which makes sense intuitively because we have not yet progressed far enough into the episode to be capable of compute $n$-step returns.


If $h_i$ are consistent and admissible, are their sum, maximum, minimum and average also consistent and admissible?

The issue is that you must include assumptions about hopping into your heuristic. In particular, if you are considering individual cars then you must assume that they might be able to hop all of the way to the goal. Thus, your heuristic for each car should be Manhattan distance divided by 2. That's guaranteed to be admissible when you take the max.

If you consider all possible cars you can do better, but you'll need to reason out all the cases. (In general every car either waits or moves, and for every waiting car one car can hop. So, by looking at the minimum distance for any car to reach the goal you can start to reduce your heuristic.) But, that is a different question.


Are there any load balancing approaches for employee cases based on complexity score?

It sounds like you're looking at the Partition Problem. https://en.wikipedia.org/wiki/Partition_problem

The task of slicing one set into N sets so that each set is equal or as close to equal as possible.

Obtaining an exact solution is NP-hard (you can't do much better than trying all combinations), however you can get an approximate answer in polynomial time.

Greedy approach:

Create N sets, each initially empty
Sort all items in descending order (by their complexity score)
for each item add it to the set with the lowest sum

If N is large, you may want to put the sets into a min heap/priority queue.


How could AI be used to augment human biological intelligence?

'Direct augmentation' of human intelligence, of the sort that you would see in science fiction, looks to be very hard. Most of our promising approaches deal with avoiding damage rather than adding capabilities--there's no drug that you can take now that will make you smarter to the degree that missing a night of sleep can make you dumber.

The most informative area of current practice is probably game-playing, where 'centaurs,' or humans working with computers, outcompete human players or computer players.

But a centaur player doesn't have a wire jutting out of their skull to jack into the computer; they're looking at a laptop screen. One of the reasons to be pessimistic about cyborg augmentation is because current I/O technology is already so good. Why install a new wire to put information into your visual cortex, when you come already equipped with two?

If you could think code directly onto the screen, how much better would that be than typing code through a keyboard? Probably some, but I find it difficult to imagine that it'll be more than twice as good. So most human-computer intelligence augmentation will look like people using software, and software using human inputs, rather than humans and computers evolving together.

Transcranial Direct Current Stimulation (TDCS) and similar approaches cause temporary changes in mental abilities by raising or lowering the activation potentials of neurons in particular regions of the brain. (I've done it myself, a few years ago, and what weak effects I noticed were probably negative. Not too much surprise for a DIY setup!)

It looks like it has a number of useful implications. One article about TDCS that I found particularly striking was the journalist who tried it gushing about how their anxiety disappeared for a few days, presumably because the part of their brain behind the anxiety was dampened. One could imagine it being useful for the treatment of many different mental disorders.

That said, I'm pessimistic that it will translate into superior peak performance, and I think that's the sort of thing that's more relevant for discussions of augmentation. (Is there TDCS that we could do that would make Terrence Tao better at doing mathematics?)

Where improved AI methods will come into play is by improving our models of the brain, allowing us to better target interventions, much in the way that AI methods are improving our treatment of cancer (through superior diagnosis and targeting of radiotherapy, as two easy examples). These effects will all be indirect--for example, AI empowering an app or gadget that helps you sleep better won't directly augment your intelligence, but will cause population-level increases in effective intelligence through reducing sleep deprivation.

I haven't talked yet about nootropics, chemicals that increase intelligence, but it's reasonable to expect that AI will improve drug discovery there like it improves drug discovery for anything else. But the same caveats apply--the effect of nootropics seem to be negatively correlated with intelligence (that is, the smarter someone already is, the harder it is to increase their intelligence further).


What is the best neural network architecture for this problem?

I'm not completely sure I understand your simulation. What I think you are doing is:

Generate 5 angles specified in radians (Are these always normalized to within (0, $2*\pi$)?).
Interpret each angle as a unit vector in a 2d space.
Add the unit vectors together, yielding a vector that lies somewhere inside a circle with radius 5.
Ask whether the summed vector is more or less than a distance $y$ from the origin of the circle.

If you're doing that, your problem looks like trying to learn a separation of two concentric rings, which is a well known benchmarking problem for classification.

I am reasonably certain you can learn this pattern with several layers of ReLU neurons. I'm not certain that convolutional layers will help you much here. The main patterns I'd expect the network to learn are:

perhaps 2-3 layers to learn whether the point lies far away from the origin in each of several different directions.
1 layer to learn where the decision boundary is in each of 4 directions away from the origin.
1 layer to inclusive-OR the 4 decision boundaries together.

My guess is that this is fairly easy to learn with 4 layers of 8 ReLU neurons, or something like it.


After my initial comment (where I suggest that it might not be enough info) I believe I actually came up with an idea.
Start with the full set of pokemon. For every possible type, identify the count of pokemon that are strong against that type. For this, you'll end up with a List<(pokemonId, types, List<weakAgainst>)>.
Minimize List<weakAgainst>.Count() and from the possible set of pokemonIds, select one at random. Without knowing anything else besides type, this pokemon is as good as any other with the same weakness count (this is the point of my original comment).
From the list of weaknesses that this selected pokemon has, select a pokemon from your list that is strong against the weakness, minimizing the amount of weaknesses again. Likely more than one will match this criteria, again, select one at random.
Keep repeating this pattern until you obtain the 6 in your team. This is, statistically speaking, one of the best teams that you can gather.
For all the combinations that you might find here, some teams will have less weaknesses, since we're "randomly" walking down a tree of possibilities. This very much sounds like a minimax-prunning algorithm, where each pokemon selection (minimizing your weaknesses) can be met with potential opponents that will maximize your weak points.
Simplified, put together:
input: allPokemon: list<pokemonId, weakAgainst, strongAgainst>

var: teamWeakAgainst: []
var: teamStrongAgainst: []
var: selectedTeam: []

while (size(selectedTeam) < 6)
  goodMatches <- allPokemon.max(p -> size(p.strongAgainst.intersect(teamWeakAgainst)))
  goodMatches <- goodMatches.min(p -> size(p.weakAgainst))
  goodMatches <- goodMatches.max(p -> size(p.strongAgainst))

  selectedPokemon <- goodMatches.random()

  teamWeakAgainst -= selectedPokemon.strongAgainst
  teamWeakAgainst += selectedPokemon.weakAgainst # not counting previously selected pokemon because the current one adds another "weakness", even if it was already accounted for

  selectedTeam += selectedPokemon

output: selectedTeam

From this algorithm is it not obvious where the "max" portion is. We're minimizing our losses (weaknesses) but we're considering all possible opponent teams equally, so there is no real maximization of the opponent choices. For a set of ideas, check below.
Note that this algorithm will give you a set of "teams" that are equally good in the sense that they'll have the same amount of minimized weaknesses and maximized strengths against other possible teams. But even if pokemon are different, the numbers will be the same, just different types.
For a more complex approach, you might want to consider how prevalent some pokemon are (you might not need to optimize against a super rare mythical type, but rather the very common types available in the game), how likely is it that certain pokemon can have better / faster attacks, what is the probability of battle IVs, how frequent can a trainer switch pokemon in battle, etc. Again, I know this is not what you asked for, but for the sake of the example, this will become so complex that instead of a search algorithm, a simulation (Monte Carlo?) approach might be simpler to build teams out of statistical testing.


Are connections genes in a genome ever deleted or just disabled?

The original paper never goes further than disabling the link, but I have seen implementations on github that did have a probability of deleting the link.

python-neat is such an eaxample.


How to detect patterns in a data set of given IP addresses using a neural network?

It seems that you want to detect ranges of IP addresses that are vulnerable/dangerous/etc, right? Such ranges are essentially numeric intervals, and so my suggestion is to look at decision tree learning instead of neural networks, because you are essentially doing a classification task where you want to test both categorical data and splits over numerical attributes.

The result will be a tree-like function (nested conditionals) of the form

IF ...> address > ...
   THEN [vulnerable]
   ELSE IF port=...
              THEN [not vulnerable]
              ELSE [vulnerable]

where a huge benefit is that it is also more human-readable than a neural net.

The most prominent algorithms for decision trees are ID 3 and its successor C4.5.


If you are training a neural network, it should learn correct weights to use the most predictive feature without any interference. For a strongly predictive feature matched with weaker ones, most NNs will learn the stronger association very quickly. I'm not sure whether there is an easy way to add this as a prior. You could do things such as inject the most influential feature in one or more hidden layers, but that does seem ugly, and only really helps if the predictive feature has certain types of relationship to output (it would probably help most if there is a strong linear relationship).

Training problems are usually in the opposite direction - how to extract the small amount of information from more noisy and less influential features that may still add up and sometimes contradict the main predictor.

The only thing that springs to mind for NNs that may help in this situation (mostly about fixing the problem of using the less reliable features) is the architecture of residual neural networks. The use of skip connections builds the neural network in such a way that a default "do nothing" identity function is encouraged as layers are added, and this allows for combining weak but complex relationships with stronger simple ones. A residual NN can do this with less compromise than changing network hyper parameters to find best fit on other NNs.

However, residual neural networks are good for managing different degrees of complexity that manifest within a problem, which is not necessarily the same as different degrees of predictive power between features.

  Would just applying a large scalar coefficient to it as preprocessing work?

No, this would usually be counter-productive for training a neural network. You should instead be normalising all inputs - a common and effective technique for neural networks is to ensure that each feature is scaled and offset so it has mean 0, standard deviation 1.


What are some examples of statistical AI applied to real-world problems?

There are several examples. For example, one instance of using Statistical AI from my workplace is:

Analyzing the behavior of the customer and their food-ordering trends, and then trying to upsell by recommending them the dishes which they might like to order/eat. This can be done through the apriori and FP-growth algorithms. We then, automated the algorithm, and then the algorithm improves itself through an Ordered/Not-Ordered metric.
Self-driving cars. They use reinforcement and supervised learning algorithms for learning the route and the gradient/texture of the road.


How can computers beat humans at image recognition?

When researchers claim "better than human accuracy", they are demonstrating that a computer can beat an individual human on a test. And that is because the ground truth labels are actually higher accuracy than a single human could label the images individually.

There are at least two major ways that ground truth labels can beat an individual human on image tasks.

Additional information is available from the same source as the image. For instance many pictures of pets in the ImageNet database are labeled with a specific breed of animal, due to how they are sourced. Most people
who are not experts at pet breeds will score quite badly on a test to identify dog breeds at the fine grained level that ImageNet presents.
Ground truth based on expert opinion can be sourced from multiple experts and their opinions combined. This approach can independently be shown to be more reliable than the opinion of a single person.

So in short yes computers can beat humans when they have had access to better original ground truth, and that is possible, even if that ground truth is generated by humans.

However, in general your concern stands. Ground truth data is a limiting factor. It might be possible in theory for a computer model to have an even better accuracy at the "real" task than the ground truth for a supervised learning task. However, this is next to impossible to prove, and other concerns, such as changes to distribution of real data as opposed to training data, are usually more important at that level of accuracy.


How do I know how changes in the weights are changing the reward in Reinforcement Learning

You have the concept slightly wrong.

This part is mostly correct:

  In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward r.

Technically neural networks are not required in RL, and it is really worth studying some simple systems that don't need them. It will make everything much clearer.

A reward $r$ can be received on every time step. However, some environments will only have a single reward at the end for success or failure for a whole episode - e.g. an instance of a game like chess where a player wins or loses.

This part is where things go a bit off track:

  Furthermore a q-Value gets defined by the [developer]. This reward times the q-Value q to the power of the position n of the action will be [fed] back using BP.

Q values are one type of data that can be calculated for an agent acting in a Markov Decision Process. They are also called "action values" and they are not usually defined by a developer. The q value, if correct should return the expected future sum of rewards from following a current policy. One way of writing this is:

$$q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}| S_t=s, A_t=a]$$

In natural language, the q value for state s and action a is the expected value (when following the given policy) of the discounted sum of rewards, starting from the given state and action. The discount factor, $\gamma$ can take any value from $0$ up to $1$, but only strictly episodic problems (which always terminate) should use the value $1$

A developer does not get to define that (except they might get to choose reward system and value of $\gamma$). Instead, they need to implement something that estimates the value of $q(s,a)$ based on what the agent has experienced. There are a few different algorithms that can do this. A popular one is called Q learning.

Regarding "[fed] back using BP", this is correct if you are using a neural network. Typically in DQN (Q learning with neural networks), this just consists of creating a small sample of training data from recent experience and training the neural network almost identically to supervised learning.

  So how do I know how slight chances in $\vec{w}$ are changing $rq^n$?

Definitely don't use $rq^n$ - there is no purpose to that quantity in RL. Instead for value-based RL, you are mostly interested in your estimate for Q value. This might be written $\hat{q}(s,a,\vec{w}) \approx q(s,a)$.

However, in general your question stands. If you have implemented a neural network to learn q values, how do you know if it is working?

There are actually two parts to this problem:

How do you know whether the agent is getting better at it's task?
How do you know whether the q values are getting more accurate?

What you need to do is measure, and maybe plot some relevant quantities.

For the first question, you would typically plot the total reward that the agent gets each episode. This will be noisy, so it is a good idea to smooth it out by taking some kind of moving average (e.g. average total reward over last 100 episodes).

For the second question, it is normal to plot some loss function of the network, just like supervised learning. Typically this is Mean Squared Error loss, as the network is learning a regression to predict q values given $s$ and $a$.  You can compare observed sums of discounted reward (aka "return" or "utility") with the earlier predicted ones, and take the error function. You need to get some measure of a "true" value of q - usually a noisy sample taken during training or testing, and measure loss. For MSE that might be

$$J(\vec{w}) = \frac{1}{2|D|}\sum_{(s,a) \in D}(\hat{q}(s,a, \vec{w}) - q(s,a))^2$$

Where $D$ is some dataset you have put together of $s,a$ and $q(s,a)$ measurements to test with. If this looks familiar to you from supervised learning MSE loss, then that's correct - it is essentially the same thing, just different how you go about collecting the data.

You may expect the loss function for $\hat{q}$ in Q learning to be somewhat unstable as the agent learns. That's because in Q learning, the policy is updating at the same time as the estimates are improving. Which makes the estimates out-of-date. However, it should still be possible to see a reduction in error as learning progresses. If it becomes stable at a relatively low value compared to initially, then the agent has probably learned all that it can - although sometimes new discoveries by the agent can open up more improvements, even late in training, and throw the error function out again.

Note that a low value of the error function does not mean you have an optimal agent. It means that the value function estimate is good for how the agent is currently behaving. In turn that means the agent cannot make further improvements without new and different experience.


The general research area is known as grammar induction.

It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding parse tree.
The training set often consists of both positive and negative examples.

There is no single best method for achieving this, but some of the techniques that have been used to date include:

Bayesian approaches
Genetic Algorithms
Genetic Programming
Blackboard Architectures
The UpWrite Predictor


Can someone help me to understand the alpha-beta pruning algorithm?

Suppose that you have already search a part of the complete search tree, for example the complete left half. This may not yet give you the true game-theoretic value for the root node, but it can already give you some bounds on the game-theoretic value that the player to play in the root node (let's say, the max player) can guarantee by moving into that part of the search tree. Those bounds / guarantees are:

$\alpha$: the minimum score that the maximizing player already knows it can guarantee if we move into the part of the search tree searched so far. Maybe it can still do better (get higher values) by moving into the unsearched part, but it can already definitely get this value.
$\beta$: the maximum score that the minimizing player already knows it can guarantee if we moves into the part of the search tree searched so far. Maybe it can still do better (get lower values) by moving into the unsearched part, but it can already definitely get this value.

The intuitive idea behind alpha-beta pruning is to prune chunks of the search tree that become uninteresting for either player because they already know they can guarantee better based on the $\alpha$ or $\beta$ bounds.

For a simple example, suppose $\alpha = 1$, which means that the maximizing player already has explored a part of the search tree such that it can guarantee at least a value of $1$ by playing inside that part (the minimizing player has no options inside that entire tree to reduce the value below $1$, if the maximizing player plays optimally in that part).

Suppose that, in the current search process, we have arrived at a node where the minimizing player is to play, and it has a long list of child nodes. We evaluate the first of those children, and find a value of $0$. This means that, under the assumption that we reach this node, the minimizing player can already guarantee a value of $0$ (and possibly get even lower, we didn't evaluate the other children yet). But this is worse (for the maximizing player) than the $\alpha = 1$ bound we already had. Without evaluating any of the other children, we can already tell that this part of the search tree is uninteresting, that the maximizing player would make sure that we never end up here, so we can prune the remaining children (which could each have large subtrees below them).


How to handle infeasibility caused due to crossover and mutation in genetic algorithm for optimization?

You have two broad categories of options, prevention and repair.

Prevention means defining a crossover and mutation operator that try to be more intelligent about respecting the constraints. Suppose you have an encoding where each individual is a list of integers, and the constraint is that there can't be duplicates. You might define a crossover operator that did something like the following. For each position in the offspring, choose randomly from one parent such that, if possible, you choose a value that hasn't already appeared in the offspring. If both parents have values at that position that are already used in the offspring, choose a random value.

That operator would avoid violating the constraint in the first place, while attempting as much as possible to have the offspring inherit information from the parents.

Another option is to just let the operator violate constraints and then deal with the ramifications afterward. You don't go into details about what your constraints actually are, just that c1=[0.1, 0.6, 0.7] violates them. Let's say the constraint is that the third position should not be more than 4x greater than the first one. OK, so then let's take this offspring and adjust either the first or third item. Maybe we make the new individual into c1=[0.2, 0.6, 0.7].

Often, you want some element of randomness in either option so that you don't strongly bias the production of new search points. In my example, don't always make the first element larger. Sometimes, make the third element smaller or produce some random combination of both to repair the constraint violation.

Finally, both options typically strongly benefit from domain knowledge. Design an operator that understands your problem and tries to intelligently solve the problem.


Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is often not feasible due to time and power constraints. They can both be used, and there is not one better than the other. It really depends on the specific case, the size of the dataset and the time and hardware available. Note that overfitting can be (partially) remedied by things such as dropout.

To be fair: it is fine to have a discussion about this with your colleagues, but as so often there is no one correct answer. If you really want proof, you can test it out and compare them. But performance-wise (i.e. the model's predictive power), the difference will be small.


Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.

It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not be worth it for most environments, and I have not seen an implementation which does that.

  Am I missing something?

It is to do with stability of the Q-learning algorithm when using function approximation (i.e. the neural network). Using a separate target network, updated every so many steps with a copy of the latest learned parameters, helps keep runaway bias from bootstrapping from dominating the system numerically, causing the estimated Q values to diverge.

Imagine one of the data points (at $s, a, r, s'$) causes a currently poor over-estimate for $q(s', a')$ to get worse. Maybe $s', a'$ has not even been visited yet, or the values of $r$ seen so far is higher than average, just by chance. If a sample of $(s, a)$ cropped up multiple times in experience replay, it would get worse again each time, because the update to $q(s,a)$ is based on the TD target $r + \text{max}_{a'} q(s',a')$. Fixing the target network limits the damage that such over-estimates can do, giving the learning network time to converge and lose more of its initial bias.

In this respect, using a separate target network has a very similar purpose to experience replay. It stabilises an algorithm that otherwise has problems converging.

It is also possible to have DQN with "double learning" to address a separate issue: Maximisation bias. In that case you may see DQN implementations with 4 neural networks.


Has an AI ever solved a detective mystery?

I might be wrong, but I do not believe that something of the scope you describe would be possible with the current state of technology. It would require a lot of things which are still in relatively early stages of research.

For one, just extracting relevant information from text is a huge task by itself. Doubly so with a novel which contains a large amount of unimportant details.

It might perhaps be easier if the input was presented in the form of some sort of list of important facts. But it would still be rather difficult for the AI to connect them and find a solution.

As an example, let's say that we have these two facts:

Alice died of a snake bite.
Bob was seen buying a couple of mice recently.

To a human, it seems obvious that the mice were bought to feed a venomous snake. However, it would probably require a tremendous effort to teach an AI to make such connections.

Disclaimer: I don't do text processing myself, so I'm not quite up-to-date on the current state-of-the-art. It's possible that some of these things have already been done in some form. If anyone knows more about the subject, please correct me if I'm wrong.


What is the difference between artificial intelligence and machine learning?

Machine learning has been defined by many people in multiple (often similar) ways [1, 2]. One definition says that machine learning (ML) is the field of study that gives computers the ability to learn without being explicitly programmed.
Given the above definition, we might say that machine learning is geared towards problems for which we have (lots of) data (experience), from which a program can learn and can get better at a task.
Artificial intelligence has many more aspects, where machines may not get better at tasks by learning from data, but may exhibit intelligence through rules (e.g. expert systems like Mycin), logic or algorithms, e.g. path-finding).
The book Artificial Intelligence: A Modern Approach shows more research fields of AI, like Constraint Satisfaction Problems, Probabilistic Reasoning or Philosophical Foundations.