The meaning of invertible here is the standard definition of invertibility for a mathematical function $f \colon X \to Y$. Invertible simply means "the function has an inverse map $f^{-1} \colon Y \to X$". Equivalently the function $f$ is bijective, which means the following two conditions hold:

$f$ is injective: for any two distinct $x_1, x_2 \in X$, $f(x_1) \ne f(x_2)$.

$f$ is surjective: for any $y \in Y$, there exists an $x \in X$ such that $f(x) = y$.

If this is unfamiliar, you should be able to find some helpful references on Google using these terms.
Most obvious neural network architectures cannot possibly be invertible. Consider for example a classifier which takes an image or some other high-dimensional input, and outputs a classification label. This network could only be invertible if there was only one possible input1 which corresponds to each label, which is not the goal of the network.

1 I mean this rather literally: if one pixel is even slightly different, this would be a different input to the network and would have to have a different output.


How is $\Delta$ updated in true online TD($\lambda$)?

Let us denote the state we are in at time $t$ by $S_t$. Then at iteration $t$ we create a placeholder $V_{old} = V(S_{t+1})$ for the state we will transition into. We then update the value function $V(s) \; \forall s \in \mathcal{S}$ - i.e. we update the value function for all states in our state space. Let us denote this updated value function by $V'(S)$.

At iteration $t+1$ we calculate $\Delta = V'(S_{t+1}) - V_{old} = V'(S_{t+1}) - V(S_{t+1})$, which does not necessarily equal 0 because the placeholder $V_{old}$ was created using the value function before the last update.


What is the advantage of using Google's Coral over Nvidia's Xavier?

Quantization is a technique used to make deep learning models smaller and faster to run.

Deep learning models are essentially collections of real-valued numbers. Because there are infinitely many real numbers, computers represent them using a format call 'floating point' numbers, which are not completely accurate. For example, a 32-bit floating point number can only represent at most $2^{32}$ distinct values. In contrast, a 64-bit floating point number can represent $2^{64}$ distinct values.

Most CPUs and GPUs cannot operate directly on large floating point numbers. This means that to do something like multiply two floating point numbers, the CPU might have to work on half of each number at a time, and do some tricky work to combine the results. Some CPUs and GPUs can operate directly on large floating point numbers, but only by using more than one core to work on a single number.

To get around this, you might chose to take a model you have that was trained with high-precision floating point weights, and reduce it to lower precision. The weights won't be exactly the same, but they'll be very close. Doing this will make the models run much faster, but you might lose some accuracy.

So the advantage of using a tool that only supports Quantized models is that models will run faster, but might be slightly less accurate.


The Flatten layer is used for collapsing an ND tensor into a 1D tensor. In your case, the inputs appear to be $28\times28$ images, so Flatten will convert that into a tensor with shape $1\times768$. Note that no information is lost. Flatten layers are usually used where you have a convolutional layer with dimensions $N\times M \times C$ (where $N$,$M$ are the feature map sizes and $C$ is the number of channels) and want to fully connect with a Dense layer or another layer that only accepts 1D inputs. Flatten can also be used when the network is meant to output a feature vector from a final convolutional layer for image classification purposes using a different technique.


How can AI be used to more reliably analyze and plan around the tie between climate and emissions?

Can AI provide a more reliable analysis of the gross effects of carbon emissions on extinctions of species ice-cap melting, and other effects?
Yes. The work of Judea Pearl and others over the last 20 years began out of a desire to address uncertainty within AI. Eventually, this led Pearl to become fascinated by the need to quantifiably determine when one event has caused another, the problem at the root of "correlation is not causation". He substantively succeeded with the combination of causal modeling and the do-calculus. The do-calculus allows you to formulate queries of the form "To what degree did X cause Y?", and to automatically determine what measurements are needed to determine the answer in a statistically meaningful way. The algorithms used under the hood are closely related to those used in AI and robotics systems to reason about uncertainty (i.e. Bayesian Inference). Causal modeling is still relatively new, and is not yet as widely used as it could be. An open problem is how to specify what the model of the world looks like from data (rather than just reasoning over a model that is given). If this problem is solved, we could see major improvements in the ability to provide analyses like those you ask about.
Can AI provide a better understanding about whether the runaway scenario described by some climatologists is realistic?
Probably not. Predicting future events without any observable precedents is not something that anyone can be sure about. The questions raised in those kinds of scenarios aren't about calculation, but about modeling. For example, if you believe that there is a large amount of methane trapped in the arctic permafrost, and you believe that temperatures above a certain range release this much faster than in the past, and you believe that methane warms the climate rapidly, then you would tend to believe that the runaway scenarios are plausible. AI can't tell us whether the methane is there (we have to go measure it, though maybe AI can make the measurement more accurate), and can't tell us how temperature will affect the release rate (again, we have models of this, but they rely on different assumptions, and we have to go measure to find out which are right).
Can AI provide better simulations of the impacts of interventions on both climate and the economy, to inform decision making?
Maybe. Agent-based modeling can help with this to some degree, and is arguably part of AI. In fact, it already has been to some degree, by Beeger & Troost in J. Agriculture Economics, in 2014 and again in 2017, though interest in this kind of modeling looks like a pretty new development in this area. Although ABM can give us reasonable models and help simulate the impact of interventions, ultimately they are just one modeling tool among many. Their potency may improve if more realistic agent models are used, but it is not clear that AI is going to provide advances in this area in the near future.
Automation of climate planning so that the harmful effects of the irresponsibility of one geopolitical entity wishing to industrialize without constraint on other geopolitical entities can be mitigated
Probably not. Although AI techniques have made some kinds of economic planning problems a lot easier, the main barrier to the effects you describe is a social/political one: countries are sovereign, and the world operates as a de facto anarchy (i.e. the UN is impotent). Your AI model can tell, say, India not to industrialize, but Indians want to enjoy the same kind of lifestyle improvements that Americans do, and would rather enjoy them sooner than later. India would collectively rather that Americans put an enormous tax on their carbon emissions, drive less, eat much, much less beef, and stop flying everywhere, than that Indians continue living on the equivalent of $7,000 each per year. In contrast, Americans would rather that Indians just wait a few decades while the developed world decarbonizes, and only industrialize once we have enough solar panels to replace all our current needs within current industrial economies.
Basically this is a resource allocation problem within an anarchy: we can only burn X carbon within Y time, everyone wants to burn some, and the only way to enforce contracts is with the threat of massive violence (or massive economic sanctions, which in turn, are imposed through the threat of violence against other actors if they trade with the sanctioned country). AI can help us answer questions like "How much should the USA pay India in exchange for India not industrializing this year?", see, e.g. work in Auction Theory, but AI can't actually make nations do those things if they can't reach a diplomatic compromise.


Which matrix represents the similarity between words when using SVD?

You can find some material here and here but the idea (at least in this case) is the following: consider the full SVD decomposition of the symmetric matrix $A = W \Delta W^T$. We want to calculate the cosine similarity between the $i$-th column (aka word) $a_i$ and the $j$-th column $a_j$ of $A$. Then $a_k = A e_k$, where $e_k$ is the $k$-th vector of the canonical basis of $\mathbb{R}$. Let's call $\cos(a_i,a_j)$ the cosine between $a_i,a_j$. Then
$$\cos(a_i,a_j) = \cos(Ae_i,Ae_j) = \cos(W \Delta W^T e_i,W \Delta W^T e_j) = \cos(\Delta W^T e_i,\Delta W^T e_j)$$
where the last equality holds because $W$ is an orthogonal matrix (and so $W$ is conformal, i.e. it preserves angles). So you can calculate the cosine similarity between the columns of $\Delta W^T$. A $k$-truncated SVD gives a well-enough approximation. In general, columns of $W \Delta$ and rows of $W$ have different meanings!


Why coupling coefficients in capsule neural networks can't be learned by back-propagation?

Very interesting paper.

We can see that they have effectively modeled probabilistically the fuzzy existence of an object with various geometric attributes as objects in a scene with a hook to the region of pixels that correspond to its possible existence.  I agree with the authors that this is a much more robust approach than CNN and LSTM and may compete well with the emerging attention based approaches.

Back propagation does seem to be involved, although that is not the focus of the paper.

Notice, "there is top-down feedback," and, "an appropriate parent in the layer above," which are hints to a hierarchical approach, and we can see this approach is not absent of its context in the overall feeding forward of input signalling to output, all of which must be trained as a whole.  But just as with much older control systems, the end-to-end convergence is facilitated by (not replaced by) more local convergence of subsystems with their own control objectives.

The overview is shown in section 4 on page 4.

The entire thing could conceivably be done without capsules at all, as with the aforementioned designs, but this group is apparently showing improved results when stopping short of full convergence on the MNIST.  No comparison is shown with LSTM, but even if LSTM was showing greater results, this direction of research is an excellent one because of the probabilitic way that it approaches an object.

Consider the classic case of waving back at someone who, as it turns out, was waving at someone behind you.  The existence of an object or an action is necessarily probabilitic, and to construct the depth of network required to model all that complexity with ReLU is probably unrealistic as the expectations increase in AI system requirements.


Implementing Gradient Descent Algorithm in Python, bit confused regarding equations

There seems to be a mismatch between the weights you provide and your network diagram. Since w[0] (the yellow connections) is meant to transform $ x \in \mathbb{R}^2 $ into the layer 0 activations which are $ \mathbb{R}^2 $, w[0] should be a matrix $ \in \mathbb{R}^{2 \times 2} $, not a vector in $\mathbb{R}^2 $ as you have. Likewise, your w[1] (the red connections) should be a vector $ \in \mathbb{R^2} $ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.
In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.
$ x \in \mathbb{R}^{2} $, an input example
$ W^0 \in \mathbb{R}^{2 \times 2} $, the yellow connections
$ W^1 \in \mathbb{R}^2 $, the red connections
$ z^0 = xW^0 \in \mathbb{R}^{2} $, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ z^1 = a^0 W^1 \in \mathbb{R} $, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.
$ a^1 = \sigma(z^1) \in \mathbb{R} $, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.
Weight Updates
As you say before your edit, $\delta^1$, as the product of two scalars $\nabla_a C$ and $\sigma'(z^1)$, is also a scalar. Since $a^0$ is a vector in $\mathbb{R}^2$, then
$\delta^1(a^0)^T$ is also a vector in $\mathbb{R}^2$. This matches what we expect, as it should match the dimensions of $W^1$ to allow the element-wise subtraction in the weight update equation.
NB. It is not the case, as you say in your edit, that the shape of $\delta^l$ should match the shape of $W^l$. It should instead match the number of nodes, and it is the shape of $\delta^l(a^{l-1})^T$ that should match the shape of $W^l$. You had this right in your original post.
Bias Updates
This brings us to the bias updates.
There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $\mathbb{R}^2$ for layer 0 and $\mathbb{R}$ for layer 1). Now, we saw above that the shape of $\delta^l$ also matches the number of nodes in layer $l$, so again the element-wise subtraction in your original bias update equation works.
I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.
A final piece of advice that helped me: drop the $x$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $\mathbb{R}^{1 \times 1}$, $X$ is a matrix in $\mathbb{R}^{N \times D}$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.


Why does CNN forward pass take longer compared to MLP forward pass?

From the NumPy Linear algebra documentation:

The NumPy linear algebra functions rely on BLAS and LAPACK to provide
efficient low level implementations of standard linear algebra
algorithms. Those libraries may be provided by NumPy itself using C
versions of a subset of their reference implementations but, when
possible, highly optimized libraries that take advantage of
specialized processor functionality are preferred.

In your case, np.dot() is a matrix-vector multiplication which internally calls  such a highly optimized BLAS routine. Those libraries are implemented in low-level languages like Fortan or C, they exist for many years now and are still unbeaten in terms of speed.
Your naive implementation of the convolution operation in python on the other hand is not optimized for execution speed and will be executed by the python interpreter (which is much slower than the compiled C functions).
You could try to replace your manual convolution by the SciPy's equivalent scipy.signal.convolve2d(), to make advantage of optimized libraries as well and get a speed up.


What is the best approach for writing a program to identify objects in a picture then crop them a specific way?

Depending on kind and amount of data you posess, there are few approaches that you might consider.

Marking target objects on dataset and training CNN that returns coordinates of target object. In this case, remember that it is usually faster when training data ROIs have their coordinates relative to image size.
Use some kind of focus mechanism, like spatial transformer network:

https://arxiv.org/abs/1506.02025

This kind of network component is able to learn image transformation (including crop) that maximazes target metric for main classifier. This tutorial on pytorch:

https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html

shows some nice visualizations of STN results. Good thing about this kind of network is that, given enough data, it might learn proper transformation from image classification data (photo -> class). One does not need to explicitly mark target objects on image!
Object detection networks, like YOLO, Faster-RCNN. There are many tutorials on that matter, eg:

https://www.datacamp.com/community/tutorials/object-detection-guide

Saliency extraction. Simple idea is to generate heatmap showing what parts of input image activates classifier the most. I guess you could try calculate bounding box basing on such heatmap. Example research paper:

https://arxiv.org/abs/1805.08249

Points 1 and 2 are probably easies to implement, so I would start with them.


Why is the log probability replaced with the importance sampling in the loss function?

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:

In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:

  While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates

This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an off-policy situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.

With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage $\hat{A}_t$ corresponding to actions $a_t$ that have become more likely in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become less likely in the new policy relative to the old policy.

If an action $a_t$ in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have $\pi_{\theta} (a_t \vert s_t) < \pi_{\theta_{\text{old}}} (a_t \vert s_t)$, which means that $\frac{\pi_{\theta} (a_t \vert s_t)}{\pi_{\theta_{\text{old}}} (a_t \vert s_t)}$ becomes close to $0$, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.

The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.


Why a single trajectory can be used to update the policy network $\theta$ in A3C?

I guess the gradient of the expectation of the Utility function, $\nabla_{\theta}J(\theta)$ in policy gradient methods where $\nabla_{\theta}J(\theta) = E_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla_{\theta}log p(\tau;\theta)]$ can be approximated using a single sample trajectory as shown in a deep reinforcement learning lecture by stanford where $J(\theta) \approx \sum_{t > 0}r(\tau)\nabla_{\theta}log\pi_{\theta}(a_t |s_t)$ and an average over sampled trajectories is not needed to compute the gradient for $\theta$ update in the direction of gradient.


Is there a mathematical example for Conditional Random Fields?

CRF is for HMM in sequences classification as Logistic Regression is for Naive Bayes for simple classification. For a in depth difference between them I strongly suggest you to read the classical Sutton article "An Introduction to Conditional
Random Fields". For a more practical example explaining how to apply the math I suggest you to read this article.


OpenAI Gym: Multiple actions in one step

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.
I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.
action = {}
for agent_id, agent_obs in obs.items():
    # get the policy of the current agent
    policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
    # use the policy and the agent's observations to compute its next action
    action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)

My environment's step(self, action) function expects action to be such a dict and knows how to handle it.
It applies all actions in the dict before calculating each agent's reward and progressing time in the environment.
So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.
For the multi-agent RL approach, I used PPO and the Ray RLlib framework.


How to manually collect rectangular training data samples from images?

Maybe LabelImg is what you are looking for?

LabelImg is a graphical image annotation tool and label object
bounding boxes in images.

If not, maybe you can find other options for your problem on this summary of computer vision tools.


Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model.

That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of many individual neural networks (with 'results' understood to mean activations at every layer, rather than just the output layer).


Classification with different approaches

As far as I understand your question, you are trying to match the learning algorithm with their results.

Firstly, I don't think the answer mapping you have provided is correct.

First, we see the options and find out which one linearly seperates the data:
We have 3 candidates for the job:

A. Linear SVM

C. Perceptron

D. Logistic Regression

In the datasets only 1, 2 and 3 are linearly seperated. Since, we know linear SVM will search for decision boundary with maximum separation between 2 linearly separable data sets hence A goes with 3. Pereceptron is basically the linear version of logistic regression. Now, LR is an exponential algorithm hence the error does not appear as large as it should. Say that for a particular input vector x the y = 1.Say the hypothesis (x * transpose(theta)) your algorithm generated is 5. For perceptron error will be 5 - 1 = 4 whereas for LR it will be 1 / (1 + e^(-5)) - 1 = 0.9932. If hypothesis generates 6 error will be 5 for perceptron and about 0.9974 for LR. So we see even for very great increase in errors the cost for LR will not increase significantly (inherent property of exponentials) whereas it increases linearly for perceptron.

Hence, 1 goes with C since it tries to minimize the error, whereas 2 will go with D since even though there is a greater cost it gets scaled due to exponential function.

And now I think 5 will go with F since the curve has a smoothness associated with it, whereas due to Re-Lu's the curve in 6 will go to E (due to an associated sharpness) and finally B goes to 4 as it has a kernel of order 2 and x^2/a^2 + y^2/b^2 +xy = c is the equation of a hyperbola of order 2.

Hence final mapping:

A - 3
B - 4
C - 1
D - 2
E - 6
F - 5

I may have missed few other mathematical subtleties, and I would be grateful if someone points them out.


Why is the access to the dynamics model unrealistic in Q-Learning?

It is an unreasonable assumption for many scenarios to know the full transition model of an environment.

In the deterministic toy scenarios used for explaining RL, such as a grid world, it seems like a trivial requirement. However, it starts to get harder very quickly as problems and state features get more complex.

Even in relatively simple environments that are possible to model in this way, the calculations start to become difficult enough that there is an argument to not bother. For instance, a common toy example is the game of Blackjack - in order to calculate the probability of different rewards after sticking you would need to resolve the remaining game tree of typically 0,1,2, but up to more, for the dealer, accounting for their starting card, and perhaps accounting for which cards have been drawn so far (if you are simulating this more realistically, without replacement).

However, that could be resolved if we wanted. To make things harder, you need to consider agents that are set to solve problems where we don't have any meaningful probabilistic model of how the state will progress.

A good example would be any real world agent that used a vision system as part of its state. Consider a flying drone set with the task of locating and tracking migrating whales over some area of the ocean. It is rewarded for capturing footage of whales. Its state is the current image plus GPS and telemetrics data.

Knowing $P(s' \mid s,a)$ in the whale-tracking example implies that you would know the probability distribution of the next video frame, somehow accounting for the chaotic movement of waves, the likely behaviour of any creatures in shot, and effects of wind turbulence on drone position etc. Even the most sophisticated physics simulations with ray-tracing rendering engines cannot rise to that challenge with much accuracy, and the computation required even for a rough render of expected image(s) based on a physics engine would be far beyond anything you could run in real time on the drone. However, it is still possible to build an agent which learns how to solve this kind of environment through gathering statistics on its task using images as state data.

Scaling that challenge back a bit, you can view many of the Atari game-playing challenges like this. The input state is a collection of still frames from the game. Knowing $P(s' \mid s,a)$ implies knowing the probability distribution of next set of frame images, given the current inputs. This is a near impossible challenge, even if you know the code for the game, providing an analytical result for $P(s' \mid s,a)$ is unrealistic - the best way to do it would be to have a memory clone of the game that you could use to look ahead then reset. Whilst this is theoretically possible in a computer game, it is not something you can consider when interacting with the real world.


Is there anything novel about Zuckerberg's Jarvis?

No, there is nothing novel about this system. The main hurdle he had to pass was problems that you will face when your system has a lot of integration points across various APIs provided by different vendors with messy and often outdated documentation.

As far as attention is concerned we live in a world where so called celebrities get attention for anything that they do. Remember that media have only one goal - More and more eyeballs aka more money and nothing else.


What is the definition of artificial intelligence?

AI is a broad term referring to more than one concept, each with its own definition.
Different Types of AI

The lowest levels are extremely simple and common, such are an artificial chess opponent. They work well because the programs internal model of reality is a 8x8 grid with only a few rules. The program chooses a preferred action by running simulations in it's internal model of reality.

What is often meant by AI is a "General Intelligence" that can come up with a creative solution to any directive, based on it's internal comprehension of the world. There is no existing example of this as of yet. The problem is that its internal model of reality needs to provide for every possible action. And it's possible actions and reactions may not be limited to a finite number of discretely distinct moves as in a game of chess. (At least if it works on the basis of conventional programming) Even ignoring the computational power needed to run such a broad and inefficient program, it would also require some stupendous amount of labour to program this internal model of reality in the first place.

I think in Sci-Fi, when people say AI, they mean a computer program that has a kind of awareness of itself and the world around it and can come up with creative and unexpected courses of actions in order to achieve its objective. Often the AI is NOT sentient, which is why it does not understand that its actions are morally wrong, or that its solution defeats the underlying intention of its assigned directive. The Horror lay in the concept that an amoral entity has more processing power than human kind.


Is there a graph neural network algorithm that can deal with a different number of input and output nodes?

I suggest you look into link prediction. I have had good luck with the StellarGraph library. They have several algorithms implemented, including GCN.
Link prediction is a binary classification problem. Given two nodes, $v_i$ and $v_j$, does there exist a link between them? Using a library like StellarGraph will also produce node embeddings while performing link prediction.
For you scenario I'm picturing a three step process:

Link prediction and node embeddings on $G$.
Link prediction and node embeddings on $G'$.
Link prediction reusing existing embeddings where each link is between the two graphs. So each link is a tuple of the form: $(v_i, v_j')$ where $v_i \in V$ and $v_j' \in V'$. If there were no links predicted from $v_i$ to $v_j'$ then that might suggest to remove $v_j'$.

In the link prediction tasks that I've outlined, you can use GCN with StellarGraph. So there should be no problems in terms of the number of nodes.


Normalization of possibly not fully representative data

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.
You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
2. It's not a fact that this will break the model.
I think everything is clear here, but just in case:
Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:
X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).
3. You can check if this breaks the model now.
Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ...
You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation.
Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value


Categorizing text into dynamic amount of categories

Transfer Learning allows you to add new categories to be predicted to the output layer without needing to re-train the entire model every time a new category needs to be classified. Rather, the weights of all initial layers up to the last few layers of the network can be frozen and only the last one or two layers can be made trainable for fine-tuning the classification rather than training from scratch. This approach would be relatively fast than other methods since the dataset in your case is small.

In order for model to flag the need of a new output class by inferring on a test document, you could include a class "Unknown" in addition to existing N classes (hence, the output layer now contains N+1 classes). If the model predicts "Unknown" with highest probability, you could add a new class to the output layer after examining the test data.


Your initial idea seems about right. Before creating your own classifier you might want to try transfer learning, using some pretrained network like VGG16 that is incorporated in most of machine learning frameworks.

As to inference on mobile devices, TensorFlow offers some tutorials in this subject:
https://www.tensorflow.org/mobile/tflite/


This problem is typically called parameter estimation or inverse modelling, and there are a variety of techniques to solve it.

If your free parameters are all continuous (i.e. none are discrete, such as integers), and your model function is differentiable, then you can turn the model into a computation graph in e.g. TensorFlow and use gradient descent methods with MSE loss, learning parameters on the data in much the same way as a neural networks. Most of the toolkits built for neural networks can do this, you just have to ignore the pre-packaged layer models and the utilities to manage them. You will need to take care with parameter initialisation, ideally you already have some rough starting values.

You can use other approaches, such as search methods too. Even when you have some discrete-valued parameters then you can use genetic algorithms for instance.

The best method to use will depend on details of your model, it is not possible to point out a generic "best". If you can use a gradient-based method directly, as you suggest in the question, then that might be the most efficient in terms of computation, provided you have some way to set reasonable initial parameters.


What are some possible projects that can be finished in a 4-week time-frame?

Welcome to AI.SE @Kate_Catelena!

I teach AI courses at the undergraduate level, and so have seen a lot of semester projects over the years. Here are some templates that often lead to exciting outcomes:

Pick a new board or card game, and write a program to play it. Your course has probably covered Adversarial Search, and may also have covered Monte Carlo Tree Search, or self-play reinforcement learning approaches. These projects are often fun to mark and creative because they are easy enough to be well done, and yet there are always new, exciting, domains to apply these algorithms to. Some examples of past projects that I thought were neat were an AI to play the boardgame Tac (mostly A* Search), and an AI to play the card game Love Letter (mostly Counter-factual minimax regret, the algorithm used to solve poker).
Pick a question that you would like to know the answer to, that could be addressed with machine learning. Then implement your own ML algorithm (decision tree learners are fairly easy), gather your own data, and show a result. Examples of interesting projects I've seen in the past are using ML to find out which of a number of factors most strongly influenced a students' subjective quality of sleep; and which items are most commonly purchased along with camping supplies (using association rule mining).
Anything involving reinforcement learning. RL projects are always neat to see if accompanied by a visualization showing the learner's behavior at different stages. A strong past project involved a student simply replicating Sutton & Barto's Acrobot experiments with their own implementation of the SARSA-Lambda algorithm.  Other things that might be neat include making a trainable "pet" that the user can influence, or solving games using self-play.
Theoretical results might seem intimidating but are often more accessible than one might think, especially if your discrete math skills are strong. I have had many student projects where the student went away to look at theory papers in ML or Multiagent Systems, found a suggestion in the future work sections that wasn't a big result, but that was fairly easy to prove, and proved it. Sometimes these are even publishable.
Replications. Go find an interesting AI paper (use scholar.google.com), and then see if you can do exactly what the authors suggest and if you get the same result or not. Then, if you have time, see if you can improve on the results. These are often most interesting when you find a paper written in a different field that uses AI. Often the authors of such papers know less about AI than you do, and so it can be fairly easy to improve on their results. I have had several students do projects like this to great effect.

Those categories are a bit vague, but remember: AI touches almost anything. Pick your favorite hobby, and see whether you can relate it to AI using one of the approaches above. Nothing makes a project stand out like one that applies AI to solve some real issues in an exciting domain. Good luck!


Any library recommended for recognizing squares in an image?

You might want to look into building convolutional neural network (CNN) for object detection using Keras. With plain white squares, it should work pretty good.


How exactly does MICE imputation combine multiple datasets into one?

MICE is a Multiple imputation method, it contains three phases:

Imputation phase: The missing data are estimated M times from a specific model to obtain M complete and potentially different data sets.
Separate analysis phase: The selected analysis is performed separately on each of m = 1, ..., M sets of imputed data to obtain estimates (central tendency and variance).
Combined analysis phase: The results obtained  are combined according to the rules established by Rubin to obtain a single final estimate.


When to use RMSE as opposed to MSE and vice versa?

Answers in the comments are decent, particularly DuttA.
DuttA gives these, approximately

ease of derivative
Don't have to worry about ~0 in denominator causing huge gradient
But to me the most important is mathematical convenience, someone
might easily make the mistake of RMSE is just equal the difference
y−y′ instead of root of mean square of y−y′. The answer this might
start depending on conventions.
In maths (Don't know the reason and might be inaccurate) we mainly
work with variances instead of standard deviation.

Here are my reasons for using the MSE instead of RMSE:

Doesn't have the sqrt operations, so it computes faster
the square root isn't easy, its Newtons method, so it could be a
dozens steps per iteration
MSE has all the information of RMSE, there is 1-to-1 mapping, so no loss
the storage of the square root typically doesn't save any memory in
IEEE-784 and compute vs. memory is big thing in complexity
tools like gradient boosted machine can "recycle" the squared error
computation for speedup and working on O(n) complexity
there is hidden scaling and regularization because many gpu hardware
elements are fundamentally 8-bit, so if you can make your code more
8-bit in its guts then you don't have as much in the back-conversion
and it runs a lot faster


Are POS tagging, Chunking, Disambiguation, etc. subtasks of annotation?

The procedures you mention don't need to involve writing code. There are now many ready-made tools available which implement various algorithms.

POS-Tagging, Chunking, and Semantic Tagging/Disambiguation are knowledge-based procedures which can all be seen as classification/clustering tasks: POS-Tagging and Disambiguation are classifications, in that they propose a label (a POS- or semantic tag) that is assigned to a token. Chunking is a clustering algorithm, as it finds coherent groups in sequences of tokens. You could also view it as segmentation, as it splits a sentence into parts.

In principle they don't have anything to do with machine learning; the algorithm you are using (ML or not) is an implementation detail. Early Taggers were mostly rule-based, but with increasing availability of annotated data it became more feasible to use ML algorithms. This, however, has got nothing to do with how you classify the procedures.

In general I would call the activity you describe as 'annotation', as you enrich the source text by adding descriptive categories to tokens and token sequences.

So, no, they do not have to involve programming or implementing algorithms, and you don't need to be technical to perform them. Though some knowledge of linguistic concepts would help.


Isn't a simulation a great model for model-based reinforcement learning?

I will give one perspective on this from the domain of robotics. You are right that most RL agents are trained in simulation particularly for research papers, because it allows researchers to in theory benchmark their approaches in a common environment. Many of the environments exist strictly as a test bed for new algorithms and are not even physically realizable, e.g. HalfCheetah. You could in theory have a separate simulator say running in another process that you use as your planning model, and the "real" simulator is then your environment. But really that's just a mocked setup for what you really want in the end, which is having a real-world agent in a real-world environment.
What you describe could be very useful, with one important caveat: the simulator needs to in fact be a good model of the real environment. For robotics and many other interesting domains, this is a tall order. Getting a physics simulator that faithfully replicates the real-world environment can be tricky, as one may need accurate friction coefficients, mass and center of mass, restitution coefficients, material properties, contact models, and so on. Oftentimes the simulator is too crude an approximation of the real-world environment to be useful as a planner.
That doesn't mean we're completely hosed though. This paper uses highly parallelized simulators to search for simulation parameters that approximate the real-world well. What's interesting is it's not even necessarily finding the correct real-world values for e.g. friction coefficients and such, but it finds values for parameters that, taken together, produces simulations that match the real-world experience. The better the simulation gets at approximating what's going on in the real world, the more viable it is to use the simulator for task planning. I think with the advent of GPU-optimized physics simulators we will see simulators be a more useful tool even for real-world agents, as you can try many different things in parallel to get a sense of what is the likely outcome of a planned action sequence.


Identifying if an image contains an object with very small (five image) training data set

I would do the following:

Add some images without the item present to your training data. Ideally some other item taken against similar backgrounds, lighting etc. And maybe some of just the background.
Obtain a pre-trained neural network CNN model, such as AlexNet, VGG-19 or Inception v5.
Load the neural network model, remove the very last layer (typically a 1000-way softmax), and replace with a brand new layer for classifying whether your object is present or not. Maybe use a 2-way softmax for "my object" vs "unknown object".
Freeze all the layers other than your own (most NN libraries will have a way to do this, or you could just run the old network truncated to extract a feature vector from penultimate layer.
Train using your 5 images. This kind of limited training is often called "fine tuning".
Find some way to test your classifier before relying on it. You may have to do a leave-one-out test where you train your network 5 times, each with 4 of the images, and see whether it correctly identifies the 5th one vs a few images without the item in it. Even with that approach, with such a small sample, you will have only the vaguest idea of how reliable your new classifier is.

You'll do a lot better if you can obtain more images to train and test with. If you get enough images (both positive and negative), you can consider replacing more layers before fine tuning. Otherwise you are heavily relying on your object being similar enough to something from the ImageNet collection that a classifier trained on that will extract useful high-level features.


Calculation of FPS on object detection task

You can use the dataset test set as "frames" of video. Test the images with your model and calculate the images per second of the result and that is the same as frames per second. However you should set the batch size to 1 as in the real world scenario. You should also display each image with teh corresponding boxes after inference and remove the accuracy calculation as to imitate the real world situation.


Input optimization on a supervised learning system

You can use a neural network for this (sort of), and you've got the first step right - train the network to predict $y$ given $x$ (i.e. train it to approximate the function $f$ such that $y = f(x)$). Because the entire neural network is differentiable (presumably), you can take gradients of the input (some $x$) with respect to the predicted output. Then you can use this to update $x$ the same way that you update the weights during training of the network. This let you find a local optimum from $x$. I don't know of a way to try to find a global optimum with a neural network like this except to find the local optimum near many $x$'s and then take the best of those.

If you want a concrete example, take a look at the tutorial on Neural Style Transfer with PyTorch: here you have a noise image as the input and you optimize it to minimize the "style distance" to a reference style image and the "content distance" to a reference content image (i.e. starting with noise, make it look like the content of one image but in the style of another). There's full code there, but here's a short PyTorch snippet that shows the main idea:

# I'm just optimizing one input; clone so that you don't modify the
# original tensor; let it know that we want gradients computed
inputs = train_inputs[0:1].clone().requires_grad_()
input_optimizer = torch.optim.Adam([inputs])

def optimize_input():
    input_optimizer.zero_grad()
    # where `model` is your trained neural network
    output = model(inp)
    (-output).backward()
    return output

output = []
for _ in range(10000):
    output.append(input_optimizer.step(optimize_input).detach().numpy())

output is a list at the end of what the predicted output was after each step of optimizing the input. You can use this to see if you've converged to a local optimum yet (and take more steps/fewer next time as necessary). inputs will be the optimized input at the end. Note that if there are constraints that your input should satisfy, you'll need to enforce those yourself (e.g. in neural style transfer, they have to enforce that the values are valid for an RGB image).

Note also that how well this works really depends on how well your network approximates $f$; you may well get some unreasonable $x$ for which the network predicts a very large $y$ because there was no training data similar to that $x$ so the network isn't constrained in that region. In general, you should probably be cautious of an $x$ you generate this way that doesn't seem similar to your training data (e.g. has larger/smaller value than any $x$ the network has seen before; interpolation is much easier than extrapolation).

I'm assuming that you have a single data set; if you're able to query for the $y$ values of given $x$'s, then you might want to take a look at Bayesian optimization - essentially the field of trying to find $x$ that maximizes $y = f(x)$ when $f$ is expensive to evaluate and you don't have gradients of it. Bayesian optimization seeks a global optimum.


Is a policy in reinforcement learning analogous to a field such as APF?

Since most policies depend solely on actions and states/observations, then if you model the space of this field as the Cartesian Product of your state and action spaces, then the policy learns a surface over this combined space, similar to the way a field is parameterized.
The policy an agent learns could exhibit the same behavior as the field you describe above (obstacles form an repulsive field, and goal(s) form an attractive field).  However, unlike the field described above, it is not guaranteed that the learned policy will capture this behavior - the policy learned depends on:

The learning algorithm used
The approximators (e.g. neural networks) used for learning, and their respective hyperparameters
The formulation of the reward function
The number of episodes/total steps the policy/agent is trained over.

To sum this answer up, I believe you could train the policy in such a way (using the mechanisms above) such that it resembles the field you describe.


This is a start-up company, not a software. There are many clients cannot find right freelancers in a short time, sometimes they take more than a month to find the right freelancer.
They use artificial intelligence to help businesses find the ones they expect. Tara AI's software helps businesses automatically scope out and manage projects with freelancers, monitoring the pace and quality of their work, and making sure contractors are paid on-time for their code commits, for example.
There are now a lot of companies that use AI technology for the same purpose, for example, Amazon, Facebook, Getdone or Google.


What are the state-of-the-art results in OpenAI's gym environments?

There is the leaderboard page at the gym GitHub repository that contains links to specific implementations that "solve" the different gym environments, where "to solve" means "to reach a certain level of performance", which, given a fixed reward function, is typically measured as the average (episodic) return/reward. For example, in the case of the CartPole environment, you solve it when you get an average reward of $195.0$ over $100$ consecutive trials.


How to determine if Q-learning has converged in practice？

A typical and practical way to measure the convergence to some solution (so not necessarily the optimal one!) of any numerical iterative algorithm (such as RL algorithms) is to check if the current solution has not changed (much) with respect to the previous one. In your case, the solutions are value functions, so you could check if your algorithm has converged to some value function e.g. as follows
$$
c(q_t, q_{t-1}, \epsilon) =
\begin{cases}
1,  &\text{if } |q_t(s, a) - q_{t-1}(s, a)| < \epsilon, \forall s \in S, a \in A \\
0, & \text{otherwise}
\end{cases},
\tag{1}\label{1}
$$
where

$c$ is the "convergence" function (aka termination condition) that returns $1$ (true) if your RL algorithm has converged to some small enough neighbourhood of value functions (where those value functions are "indistinguishable"), and $0$ otherwise
$q_t$ is the value function at iteration $t$
$\epsilon$ is a threshold (aka precision or tolerance) value, which is a hyper-parameter that you can set depending on your "tolerance" (hence the name); this value is typically something like $10^{-6}$

Of course, this requires that you keep track of two value functions.
You can also define your "convergence" function $c$ in \ref{1} differently. For example, rather than using the absolute value, you could use the relative error, i.e. $\left|\frac{q_t(s, a) - q_{t-1}(s, a)}{q_t(s, a)} \right|$. Moreover, given that RL algorithms are exploratory (i.e. stochastic) algorithms, the value function may not change (much) from one iteration to the other, but, in the next one, it could significantly change because of your exploratory/behavioural actions, so you may also want to take into account more iterations, i.e. after e.g. $N > 1$ iterations, if the value function does not change much, then you could say (maybe probabilistically) that your RL algorithm has converged to some small neighbourhood of value functions in the space of value functions.
Note that these approaches do not guarantee that your RL algorithm has converged to the global optimal value function, but to some locally optimal value function (or, more precisely, small neighborhood of value functions). Q-learning is guaranteed to converge to the optimal value function in the tabular setting (your setting), but this is in the limit; in practice, it is more difficult to know if Q-learning has converged to an optimal or near-optimal value function.
Maybe you can also have a look at episodic returns of the policy derived from your final value function, but without upper and lower bounds on the optimal returns, you don't know much about the global optimality of your policy/value function.
Yes, you can check if the policy makes the agent reach the goal, but many policies could do that job, i.e. that does not say that the policy is the best (or optimal) one, i.e. that's a necessary (provided the goal is reachable and the reward function models your actual goal) but not sufficient condition (for optimality). The optimality here is usually a function of the return (given that is what you are usually trying to optimize).


Is there any board game where a human can still beat an AI?

Not all games (or even board games) are computationally algorithmic. Even the least skilled player is likely to trounce the hottest pattern-matching algorithm in a game of Pictionary (for example).

If you want to say that the movement of pieces upon successful completion of a task is only ancelary to the object of the game, than your answer will be largely self-selecting. A sufficiently sophisticated algorithm will brute force a computational problem better than human intuition… eventually.


Are there any AI that have passed the MIST test so far?

Yes, although how useful this AI can be is another question entirely.

mpgac is a "minimally intelligent AGI" trained on the GAC-80K corpus of MIST questions. As a result, it should be able to "minimally" pass this test. However, being trained on the GAC-80K corpus obviously make it lacking for any practical purposes. From the README:

  Obviously this should only be capable of producing a minimally intelligent signal when ordinary commonsense questions are asked, of the kind depicted above, using questions which would have made sense to an average human between the years 2000 and 2005. On expert knowledge or current affairs related questions it should perform no better than chance.

The point of mpgac is to compare it to other AIs that could be built to pass this test. Or as the writer wrote in the README:

  When scanning the skies how can we tell whether the radio signals detected are from an intelligent source, or are merely just background or sensor noise?

Ideally, you would want to build a program that is "better" than mpgac. In much the same way as ELIZA can be seen as a baseline for the Turing Test, mpgac is the baseline for the MIST test.

The GitHub repo of mpgac (as well as the GAC-80K corpus) is available here.


Type of artificial neural network suitable for learning and then predicting forest growth

My suggestion is not to use an ANN, but instead to use a simpler regression algorithm. The main reason for this is that ANNs take a long time to train, and work better when a lot of data is used. They also require a lot of expertise in parameter tuning to apply well. Since you say you don't have a lot of data, and also don't have a lot of experience using them, I think you will be better off applying something else first. If the other techniques don't work at all, then you might think about using ANNs, but again, they tend to want a lot of data.

If you have tried ordinary least squares regression, and found it does not work well, my next choice would be a Classification And Regression Tree. These models can make good decisions with small amounts of data, and do not require a lot of time to train. They can handle real-valued outputs like the height and width of a tree. Weka's REPTree might be a good starting place.

If Trees don't work out, my next suggestion would be to try regression using a Support Vector Machine. SciKitLearn's SVR is a good choice for this. SVRs can sometimes be very effective when data is limited, because they make assumptions about how to handle data-poor regions that seem to be generally applicable. An SVM can also report low confidence when estimating in those regions. They also train fairly fast when using small amounts of data, and can learn non-linear functions from the data.

If you really want to use an ANN, I would start with a simple Multi-layer perceptron. This model has few parameters to play with, and can probably fit well to your regression. It may make strange decisions in regions with less data however.

Hope this helps!


What does "we wrap the individual and reuse the codons" mean in the paper "Grammatical Evolution" by Neill and Ryan?

In GE, the genotype is a linear sequence of codons. By "wrapping" it, you make it a circular sequence that never ends. It allows you to build a bigger tree, while having only a few codons. Still, it is possible to find such a combination of a genotype and a grammar that defines an infinitely deep expansion — such combinations are hardly suited for practical purposes.
I learned about GE recently. I implemented a GP/GE system to solve the Santa Fe Trail problem. I chose not to perform wrapping to make genotype-to-phenotype mapping more predictable (I always generate enough codons to complete a grammar expansion, and prune unused tails). I also went with a subtree crossover, and a subtree-local mutation, which effectively make the system more a classical GP (with a fancy genotype-to-phenotype mapping) than GE. So there are some options.


What is the right way to train a generator in a GAN?

Why I have to set to real these fake images and what fake images are these?

You set them to "real" label for the discriminator when training the generator, because that is the goal of the generator, to produce an output of 1 (probability of being a real image) when tested.
Usually you will generate a new batch of generated images for this step in training. You just used the last generated mini-batch to train the discriminator, so you expect them to score worse. Sending the exact same images again will cause correlation between the two minibatches that you want to avoid. It would not be a disaster, but training GANs can be quite difficult and sensitive to details like this, so it is better to keep generating new images and not re-use the previous ones.

The one generated in the first round from the generator itself?

No. New images generated just for training the generator.

Or only the one classified as faked by the discriminator? (then they could be both real images classified wrongly or fake images classified in the right way).

No. New images generated just for training the generator.
Out of interest though, if the discriminator classifies a fake image as 100% real (with a probability close to 1), then the generator will not learn anything from that. The gradients would all be zero.

Finally what the generator does to these faked images?

Nothing is done to the images themselves - unless perhaps you are keeping some copies to render and monitor training progress etc. The images occur within the combined generator/discriminator network, effectively as a hidden layer. The images are represented as artificial neuron output, so they are involved in backpropagation calculations for that layer (with no difference to any other hidden layer in a CNN), but are not changed directly.
The generator uses the gradients calculated from the combined discriminator/generator network to update its weights using gradient descent. Importantly in this phase of the updates, the discriminator weights are not changed.
In terms of training the generator/discriminator combined network to update the generator:

The input to the combined network is some new random input vectors (typically a vector with independent truncated normal distribution for each element).

The "ground truth" label is 1 for every item.

The discriminator parameters must be "frozen", somehow excluded from being updated.

Run the minibatch forward to get loss and backpropagate to get gradients for the whole network including the generator.

Apply a gradient step (usually via some optimiser, such as Adam).


Can current AI techniques distinguish a fake old paper from a real one?

Is current AI sufficient to distinguish a fake old paper from a real one?

It depends. Specifically, it depends on how much information you (or any automated system that feeds data to the AI), gathers about the paper you want to test, and how good that information is at distinguishing between real and fake old paper.

A model that distinguishes between cases in an input domain (which can be any consistent data, such as images of a subject) is called a classifier. The current state of the art classifiers working with sensory data, if fed enough training data of sufficient quality, can out-perform humans at certain tasks. This is a success of "narrow AI" over humans. This is not some magical ability of machine learning that always applies though, there are some strong caveats:

The task must be feasible given the data collected. Low-resolution photos of samples of the paper might not contain enough information for your example task at all.
When comparing with a human performing the same task, they must be given identical data. If a human would gain more data by interacting with the paper in your example, it can give the human an advantage that a narrow AI cannot easily replicate.
The quality of training data "ground truth" must be higher than achievable by a single human. In your example, this could be achieved simply by knowing the provenance of all the training data.
There often needs to be a large amount of training data to achieve better-than-human ability, although the actual amount required depends heavily on the task.

Expanding on the first point, image classifiers can sometimes be surprisingly good, they have different strengths and weaknesses compared to human vision, so it is possible for them to sometimes be good at making classifications that humans are bad at. However, there is no guarantee of getting that kind of result for any specific task.

Expanding on the second point, if interacting with a subject is allowed in order to classify it, this moves from the domain of "narrow AI" classifiers into active rational agents.

If you gave a piece of paper to a human expert on old paper, and ask them to classify it, they might first look at the paper, then maybe sniff it, touch it . . . still undecided, so get out a magnifying glass to check a couple of features that might be good telltale signs . . . etc.

To build an equivalent artificial agent is harder than a single classifier: It might be possible to purpose-build a collection of narrow AI classifiers, that output a class and confidence levels that decided whether certain tests should be carried out. It might even be possible to give a more general AI the same toolkit plus a very long time to experiment and it could learn for itself which tests to do in which order and how much to rely on the separate results. Or some hybrid of the two approaches. However, any such sophisticated system would still be bound by the limits of individual classifiers.


There are probably many different approaches to doing what you are talking about, but the most common would probably be using long short term memory recurrent neural networks to operate on the data through time. You would be able to train your neural network to do the things that you want to do such as classify the speaker, and classify noises into different categories. However, to train neural networks like these, it requires a lot of training data as does any other deep neural network. So, if you want to apply these kinds of neural networks to practical problems you will need a lot of training data. Or, you could just use a smaller simpler neural network.


Is the cube root function suitable as a n activation function?

There are a few traits that you want the activation function to have, and cube roots rate as OK-ish:

Nonlinear – check.
Continuously differentiable – no. There is a problem at $x=0$. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach $x=0$, because $\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$
Range considerations – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.
Monotonic – check.
Monotonic derivative - no.
Approximates identity near the origin – no, the approximation is bad near the origin.

If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.

I would worry about the high gradients near $x=0$, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than $1$ (or less than $-1$) then treat the gradient as if it were $1$ (or $-1$) for the rest of backpropagation.

The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.


How do we reach at the formula for UCB action-selection in multi-armed bandit problem?

Here is an intuitive description/explanation.
$c$ is there for a trade-off between exploration and exploitation. If $c=0$ then you only consider $Q_t(a)$ (no exploration). If $c \rightarrow \infty$ then you only consider exploration term.
$\frac{\ln t}{N_t(a)}$ is there to balance out exploration term. If you consider a simple case where you only have one action (then it wouldn't make sense to explore you could always pick that action but let's pretend there is) then as $t \rightarrow \infty$, because $\ln t$ has sublinear growth,
\begin{equation}
\frac{\ln t}{N_t} \rightarrow 0
\end{equation}
So, after you picked an action infinitely many times, the exploration term will completely diminish, i.e. you already know a lot about what that action does. If you picked a numerator that doesn't have sublinear growth, then as $t \rightarrow \infty$ exploration term would not diminish, so you would always have a chance to explore and exploration term can "overpower" action value term if $Q_t$ is very small, even after you picked an action infinitely many times, which is not desired.
A similar thing is with multiple actions, $\ln t$ will make sure the exploration term $\rightarrow 0$ if you picked it many times, but it's still better than constant term $K/N_t(a)$, where $K$ is some constant, because it can diminish too fast.
With $\ln t$ you will also not stop exploring completely if you haven't picked some action in a long time, because $\ln t$ will keep growing and $N_t(a)$ will remain the same, so their fraction will increase with time, which is useful in non-stationary environments.
The square root is also there probably to balance out the magnitude of exploration term.
You can also see this answer. It has couple of links to some papers for a more mathematical description.


What should I do when the new generated state has bigger distance to the goal than the parent state?

If you don't go back to your previous state, the best you will do is get stuck in a local maxima. So, to answer your question, yes, backtrack if your hill climbing has failed. The number of times you want to do this depends on you and your implementation. Hill climbing does not guarantee to find the global maxima and, in many cases, you won't know if it is a local maxima or a possible global maxima, unless you try the hill climbing a few times.

If you haven't already seen its Wikipedia article, it has a cool graph showing what I mean.


Which AI methods are most appropriate for login face recognition?

One of the methods which is quite fast and easy to implement.

You can do Principal Component Analysis (PCA) based face recognition. You can go through this paper for the theory behind it. For an example implementation you can see this blog post.

The process, roughly, is as following:

If you have a grayscale image of size $(20,20)$, then this image can be flattened to a vector of size $400$. If you have $5$ images of size $(20,20)$ then your data matrix will be of size $(5,400)$.
We wish to find the principal components of the distribution of faces, or the eigenvectors of the covariance matrix of the data matrix, which will be of size $(5,5)$. Each eigenvector (out of total $5$), of size $(5,1)$, accounts for some amount of variation among the images, and they are ordered by the amount (the eigenvalues corresponding to the eigenvectors) they account for. These eigenvectors can be thought of as a set of features that characterize the variation between the images. Recognition is done by projecting a new image (flattened to size $(400,1)$)into the subspace spanned by the eigenvectors of the covariance matrix and then classifying the face by comparing its coordinates in the face space with the coordinates of the known individuals.

You can see an implementation in this git repo.


What happens if I train a network for more epochs, without using early stopping?

Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.


What is the difference between search and learning?

In the context of AI:

Search refers to Simon & Newell's General Problem Solver, and it's many (many) descendant algorithms. These algorithms take the form:

a. Represent a current state of some part of the world as a vertex in a graph.

b. Represent, connected to the current state by edges, all states of the world that could be reached from the current state by changing the world with a single action, and represent all subsequent states in the same manner.

c. Algorithmically find a sequence of actions that leads from a current state to some more desired goal state, by walking around on this graph.

An example of an application that uses search is Google Maps. Another is Google Flights.

Learning refers to any algorithm that refines a belief about the world through the exposure to experiences or to examples of others' experiences. Learning algorithms do not have a clear parent, as they were developed separately in many different subfields or disciplines. A reasonable taxonomy is the 5 tribes model. Some learning algorithms actually use search within themselves to figure out how to change their beliefs in response to new experiences!

An example of a learning algorithm used today is Q-learning, which is part of the more general family of reinforcement learning algorithms. Q-learning works like this:

a. The learning program (usually called the agent) is given a representation of the current state of the world, and a list of actions that it could choose to perform.

b. If the agent has not seen this state of the world before, it assigns a random number to the reward it expects to get for performing each action. It stores this number as $Q(s,a)$, its guess at the quality of performing action $a$ in-state $s$.

c. The agent looks at $Q(s,a)$ for each action it could perform. It picks the best action with some probability $\epsilon$ and otherwise acts randomly.

d. The action of the agent causes the world to change and may result in the agent receiving a reward from the environment. The agent makes a note of whether it got a reward (and how much the reward was), and what the new state of the world is like. It then adjusts its belief about the quality of performing the action it performed in the state it used to be in, so that its belief about the quality of that action is closer to the reality of the reward it got, and the quality of where it ended up.

e. The agent repeats steps b-d forever. Over time, its beliefs about the quality of different state/action pairs will converge to match reality more and more closely.

An example of an application that uses learning is AI.SEs recommendations, which are made by a program that likely analyzes the relationships between different combinations of words in pairs of posts, and the likelihood that someone will click on them. Every time someone clicks on them, it learns something about whether listing a post as related is a good idea or not. Facebook's feed is another everyday example.


How can I use a 2-dimensional feature matrix as the input to a neural network?

The most obvious way to do this would be to simply "unroll" your matrix into a vector. Your example input matrix would get turned into the following input vector:

$$\left( \begin{array}{} a_1 & a_2 & \dots & a_t & b_1 & b_2 & \dots & b_t & c_1 & c_2 & \dots & c_t \end{array} \right)$$

I don't think there are any other clear ways to use an "input matrix" really. The only benefit I could see in using an input matrix rather than an unrolled vector (if it were possible to do so in whatever way) would be if doing so would somehow enable the learning algorithm to exploit the "domain knowledge" that certain input features are related to each other in special ways (i.e. features in the same row belong to the same unit, and features in the same column are the same "type" of feature, or other way around). Intuitively, I suspect something like this could be accomplished by restricting the number of connections you make to the next layer. For example, you could make a part of the next layer only be connected to all the $a_i$ features, a different part connected only to all the $b_i$ features, etc. Similarly, you could have a part that is connected only to the $a_1, b_1, c_1, \dots$ features, a different part only connected to the $a_2, b_2, c_2, \dots$ features, etc. I don't know for sure how well this would work though... just think that it could.


How to represent and work with the feature matrix for graph convolutional network (GCN) if the number of features for each node is different?

The simplest way I could come with is to pad with 0 each feature which is not present. You said that you're going to add too much noise to the network, but I don't see the problem (please correct me if I'm wrong). For example we have two nodes, the first one has only 2 features with the 3rd one missing and the second node has all features X=[[1,2,0], [3,4,5]]. Now we can project the nodes to a hidden representation (pretty common). I'm going to use a weight matrix of W=[[1], [1], [1]]. The output of XW will be [[3], [12]]. Now let's add a new feature to the second node X=[[1,2,0, 0], [3,4,5,6]] and apply the same transformation W=[[1], [1], [1], [1]] the output will be [[3], [18]] you can see that the first node is not affected by the number of missing features.

Another way you could achieve this if you don't want to use the projection could be using a mask. For example give the same example above we could create a mask M=[[1,1,0], [1,1,1]] where each entry represents if a specific feature is present in a specific node. Now usually a GCN layer is defined as H=f(AHW) where A is the adjacency matrix. We could change the propagation rule to H=f(AH*MW) where * is the pointwise multiplication. Like this if a node is missing a feature it can not "access" information from others that are having that feature.


Is one big network faster than several small ones?

you need to understand about layerwise interpretability of neural networks before this . each layer is activated by some patterns "activations" that are much more complex than its previous layers (in terms of texture etc).
so deeper the network , more complex functions it can compute . therefore you can't replace a deeper network with a several shallow ones. i suggest you to skim through https://arxiv.org/abs/1310.6343 , In this paper it is proven that no t/2 layered network can capture the distribution that of a t-layered one.


This policy is usually referred to as the Behaviour Policy when talking about off-policy learning. It is the policy that actually takes actions in the world. The policy that one wants to learn, such as a greedy policy in Q learning, is know as the target policy. In this way the behaviour policy “logs” experience that the target policy (or policies) may learn from.


What if the more fit parent has fewer nodes compared to the other, will the disjoint and excess genes be discarded?

When crossover happens and one parent is fitter than the other, the nodes from the more fit parent are carried over to the child. This is the case as disjoint and excess genes are only carried over from the fittest parent.
Here's an example:

// Node Crossover
Parent 1 Nodes: {[0][1][2]} // more fit parent
Parent 2 Nodes: {[0][1][2][3]}

Child Nodes:    {[0][1][2]} // after crossover

Gene information is also passed to the child during crossover. Matching genes (those that have the same innovation number) are chosen at random and passed to the child. The disjoint and excess genes are choose from the more fit parent.

// Gene Crossover
Parent 1 Genes: [1][2][3]      [6]   [8][9][10] // more fit parent
Parent 2 Genes: [1][2][3][4][5]   [7]

Child Genes: [1][2][3][6][8][9][10] // after crossover

As you can see, the gene innovation numbers in the child match up with the innovation numbers of the fittest parent. However the gene information from matching genes (in the example genes 1, 2 and 3 match) have an equal chance of being carried over to the child. In the example, the child's first three genes could have come from either parent.


What's the definition of "singularity" in the context of neural networks?

Firstly, I invite you to take this answer with a grain of salt (and possibly suggest edits) as I am not specifically familiar with the issue of singularities myself. Also, I appologise for all the links being at the bottom of the answer, I am unable to reuse links throughout my answer due to my reputation on this particular exchange.

This appears to refer to singular points in Algebraic Geometry. At singular points, no singlar tagent can be defined using the usual methods. This is as opposed to regular (i.e non-singular points), where one (and only one) tangent can be regularly defined.

The Easy Answer

In simpler terms, and particularly with respect to neural networks, it refers to the points on a hypersurface (e.g. a line, plane, surface, etc) at which the gradient/slope is ambigious.

I've taken example of this from the wikipedia page on singular points.

In the above image, the hyperspace is a line. On this line, there is a singularity at (0,0), where the line crosses itself. At this point, there are two slopes (one going up, and the other going down). There is no way to choose between the slopes without applying arbatrary rules, so the gradient is ambigous.

These singularities are problematic for some of the algorithms used in deep learning, such as Gradient Descent, which uses the the derivative of objective functions (e.g. loss functions) to search for the weights/parameters that yield the lowest error.

Gradient Descent In This Example

For anyone unfamiliar with gradient descent, the problem with singularaties can be understood with the following example:

Say that the above line represents a path. Starting at the top right, you want to find the lowest point on the path. At any time, all you can know is the slope at the current point. You can turn left or right freely, but will never move upwards, so you will stop as soon as you reach the bottom of a valley.

Initially moving down and to the left, you will eventually reach the singularity and need to choose to go either left or right. Because you have no information about the path beyond your current location, you have no way of knowing which way leads to the bottom of the path.

If you choose at random, you have a 50% chance of getting stuck at in the valley at the bottom of the loop on the left of the path.

You could impose a rule like "always follow the gradient most different to the current gradient", then you'd go right, and find the true lowest point. However, if the path was slightly different (e.g. if it curved up at (0.25, 0.25) this would be bad, as the correct path would then be on the left.

In practice, there is no way of knowing which way is better without a deep understanding of the hypersurface of your objective functions (and if you knew that much about the objective function, you likely wouldn't need to use machine learning in the first place).

Research/Resources

I discovered this in order to answer this questions so, as justification, here is the process and resources I used to come to this conclusion:

I dug through the references in the paper you linked. Of particular relevance was Amari et al. (2006), which mentions Hironaka’s
theorem of singularity resolution.

In the introductory paragraph, the wikipedia page mentions the concept of Non-Singularity Variety, and links to this page on singular points in Algebraic Geometry.

The example on singular points article is analogous to a 1d hypersurface version of many of the figures in Amari et al. (e.g. figures 1,2,4,5, and 6), and the discussion therein generally centers around the point at which the shapes converge.

My answer was composed based on this discovery process, as well my prior-knowledge of Gradient-Descent


How to efficiently interpret phrases in a Python AI?

To compare the strings you can use Fuzzy string matching. FuzzyWuzzy is a python package that does this using the Levenshtein distance, which calculates the difference between two strings by counting the number of single character edits (insert, delete, substitute) that have to be done to make them equal.

To install the package you can use pip for example:

pip install fuzzywuzzy

Then you can use it to check the similarity between two strings (100 is the highest score; higher score means more similar):

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio("How are you?", "How are you?")
100
>>> fuzz.ratio("How are you?", "How are you doing?")
80

Or, which might be most interesting to you, find the most similar string from a list of strings:

from fuzzywuzzy import process
choices = ['HI', 'HELLO', 'HEY', 'GOOD MORNING','TELL ME A JOKE', 'JOKE', 'FUNNY','GIVE ME A TIP', 'TIP', 'LESSON','WHAT IS YOUR NAME', 'WHO ARE YOU', 'HOW ARE YOU?']

You can get a list of the phrases with the highest score (parameter limit, 2 in this example):

>>> process.extract("hello there", choices, limit=2)
[('HELLO', 90), ('HEY', 60)]
>>> process.extract("please make a joke", choices, limit=2)
[('JOKE', 90), ('TELL ME A JOKE', 69)]

Or just get the one with the highest score:

>>> process.extractOne("whoe are you?", choices)
('WHO ARE YOU', 96)
>>> process.extractOne("I want a lesson", choices)
('LESSON', 90)

Note:

There are cases where another phrase might give a relatively high score also:

>>> fuzz.ratio("How are you?", "Who are you?")
83

But as long as you also have "Who are you" and "How are you" in the list, they will be detected correctly:

>>> process.extract("Who are you?", choices, limit=2)
[('WHO ARE YOU', 100), ('HOW ARE YOU?', 91)]
>>> process.extract("How are you?", choices, limit=2)
[('HOW ARE YOU?', 100), ('WHO ARE YOU', 91)]

Also see: FuzzyWuzzy: Fuzzy String Matching in Python


What are good learning strategies for Deep Q-Network with opponents?

In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play.

  How should an opponent be implemented to get good and fast improvements?

You don't need to think in terms of agent vs opponent, instead think in terms of coding both the players' goals into a single Q function. Score +1 if player A wins, -1 if player B wins, and zero for a draw. It is then player A's goal to maximise the score and player B's goal to minimise the score.

You can then implement and learn both player strategies in the same self-play learning session and same Q function, using minimax. In practice that means where in Q learning you generally pick the maximising action on next state to bootstrap Q values, in a minimax variant, you pick maximising or minimising action depending on whose turn it is. Otherwise the Q learning algorithm is the same as normal. I have implemented this, but not for DQN, just for tabular Q-learning - feel free to learn from, copy and/or re-use any part of that code.

  Is it reasonable to play against a random player, a perfect player or should the opponent be a DQN player as well?

The Q learner will learn to optimise against whichever player you make it play against. Against a random player, it will not necessarily learn to play well, just well enough to defeat the randomness. It may even make deliberate mistakes - such as not blocking a winning line - knowing it has a better chance to win due to random opponent.

Against a perfect player is possible with tic tac toe (because you can construct such a player), although there might be gaps in the trained Q values - game states never seen - which mean that an imperfect opponent could actually defeat the trained agent! In practice this does not scale to more complex unsolved games, because no perfect players exist.

Another DQN player should work fine. You would end up with two agents, each specialising in playing one player's turns. This is less efficient than a single minimax-based player, but no expected problems. It may be a preferred choice for some games, especially if they are not zero-sum.


How can I determine whether a car in a video is moving or not?

CNNs are translation invariant.

You are over complicating the problem.  The easiest thing you can do is define a region of interest (ROI) of the hood.  In the first case where the car is moving and the reflections are dynamic.  In the second case they are static.  Just do frame-to-frame image subtraction of the hood.  If the vehicle is moving you will have lots of 'edge energy'.  If it is not moving it will be just noise.

You can apply the same method to the whole image too.  In the static case the image subtraction method may become messy as the clouds are moving along with vehicles and pedestrians.  For this case use the image subtractions as input to your method.

Another approach is to run an image stabilization algorithm.  OpenCV has one.  Look at the transformation outputs (translation, rotation, scale, rigid, similarity, affine, etc.).  If you can't make a simple filter on them to determine the two cases, train a classifier.


Is language understanding possible without qualia?

I'm going to be controversial here; so please don't vote this answer down if you just disagree with it.

Your question presupposes that machines do not or cannot possess qualia, which are required for true understanding. Given that we don't really know what it means to 'understand' something, and that even the meaning of 'meaning' itself is by no means a resolved issue, it might be overly specific.

In one strand of linguistics, the meaning of a word is defined by its use, and by the context of surrounding words. We could hazard a guess that children acquire the meaning of words through exposure to language, and the correlation of experiences with the corresponding sounds. How that works in detail is AFAIK not fully understood. But there would be nothing 'inherent' in a new-born human that would enable it to 'understand' anything.

If that is the case, then we could train a machine to do the same. Obviously, it would be a long and tedious process, and there is probably a reason why it takes us years to become proficient in our use of language. But if we correlate sensory input with linguistic utterances, a sufficiently sophisticated learning algorithm might be able to acquire some meaning for such utterances from the way they are used.

There are, of course, rather a lot of unknowns here. That is because the topic straddles various fields, from child language acquisition, corpus linguistics, the psychology of learning, and many more. And to my knowledge, none of these fields is sufficiently advanced to shed any light on this issue yet. There is the whole question of abstract words and concepts. How do we segment the continuous stream of sounds into discrete units (phonemes) without knowing what they are? With all that complexity I begin to appreciate why Chomsky opted for his Language Acquisition Device to avoid getting frustrated... :)

So, to answer your question: yes, it should be possible. A properly set up machine, which would be able to simulate human learning, would pick up its own mapping of linguistic structures to its experience from the world outside. And if we call this mapping the 'meaning' of those structures, then a machine can learn this, and presumably 'understand' language. If we ever get to that stage with AI is another question.


LIDARs, especially cheap LIDARs, have problems with reflective surfaces (like metallic paint on cars), strong lights like car headlights, weather(rain, snow, hail, fog), and have a considerably shorter range than comparable in price radars. Of course, they have much better precision, so some hardware stacks for cars are using both.


The use of classification algorithms nowdays

Machine learning algorithms lay the foundation for deep learning architectures so they pretty much play a role in which ever deep learning application one can make.

The improvement of these machine learning algorithms coupled with deep learning techniques have lead to the establishment of "strong-narrow AI"


How do mutation and crossover work with real-valued chromosomes?

As @Thomas W said, you can be pretty immaginative when you're developing mutation and crossover methods. Each problem has its own caracteristics and, therefore, requires a different strategy.

BUT, from my experience, I'd say that 90% of crossovers and mutation on real numbers genotypes are solved using the BLX-α algorithm.

Crossover:

This algorithm is really simple. Given the parents X and Y and an α value (inside the range [0,1], generally around 0.1/0.15, but it depends by the problem), For each gene of your genotype:

extract the genes xi and yi
find the minimum and the maximum values
the new gene will be a random number in the interval [min - range * α, max + range * α]

A variation of this algorithm is the BLX-αβ, in which we take into account which parent performed better and use two constants (α > β) to increase the probabilities that the new value will be closer to the one of the fittest parent

Mutation:

With the mutation the situation is similar: we need to get a random value that is related to our problem domain (we do not want the mutations to be destructive! They have the function of exploring the space).
In these cases it is useful to determine a range for the mutation and use that range to find the new value of the gene using BLX-α.

A more sophisticated mutation algorithm can be achieved using BLX-α on boundaries that depend on the actual value of the gene and the fitness function of the individual.
Let's imagine that our individual performs in a very bad way; in that case the mutation operator will be used to shift the individual to a distant point in the search space, where it will probably perform better.
On the other hand, if the individual is already fit we may not want to introduce some dramatic changes using the mutation. In that case the mutation range would be more contained and would have the function of tuning the genotype instead of exploring for better alternatives.


When will we have computer programs that can compose mathematical proofs?

Arguably we have had this since 1957, with the General Problem Solver. However, it is a thorny problem, and like so much in AI, it works fine in toy domains (like the Towers of Hanoi problem), but fails in real life, as real life is too complex for it to cope.

There is a list of solvers (the category of programs dealing with this issue) on Wikipedia. As of today, it comprises about 40 systems (split into two sub-categories). These work with mathematical equations and presumably also more general problems, assuming they are not too complex.


Which kind of data does sigmoid kernel performance well?

The sigmoid kernel is better suited for binary classifications. As the IRIS dataset is for multi-class classification, its performance was not as good as other kernels. You can train with only 2 types of flowers to see if the sigmoid kernel can perform well.


In Q-learning, wouldn't it be better to simply iterate through all possible states?

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future reward).
In this case, the difference is that you would pass as much time to explore each possible couple of (state, action) while Q-learning would pass more time on the pair which seem more promising and, as you've said this wouldn't be efficient for a problem with a huge number of pair (state, action).
If the algorithm is executed only once, then, even for a problem with a few pairs (state, action), you need to assume that an action effected on a state will always bear the same result for your method to work.
In most cases, it isn't true either because there is some sort of randomness in the reward system or in the action (your agent can fail to make an action) or because the state of your agent is limited to its knowledge and so doesn't represent perfectly the world (and so the consequence of its action can vary just like if the reward had some randomness).
Finally, your algorithm doesn't look at the expected future reward, so it would be equivalent to having $\gamma = 0$. This could be fixed by adding a new loop updating the table after your current loops if you execute your algorithm only one time or by adding the expected future reward directly to your Q-table if there is an outer loop.
So, in conclusion, without the outer loop, your idea would work for a system with few pairs of (state, action), where your agent has a perfect and complete knowledge of its world, the reward doesn't vary, and where an agent can't fail to accomplish an action.
While these kinds of systems indeed exist, I don't think that it's an environment where one should use Q-learning (or another form of reinforcement learning), except if it's for educational purposes.
With an outer loop, your idea would work if you are willing to pass more time training to have a more precise Q-table on the least promising pair of (state, action).


Neural network to control movement and "home in" on a target

I would go with a NEAT algorithm for this one. Basically, you "breed" the right network that does the best every generation. Then Rinse and repeat. Doesn't require a training datbase. Bots compete and evolve as they go.

For example, this demonstration shows how Neataptic.js teaches a bunch of bots to follow a target:
https://wagenaartje.github.io/neataptic/articles/targetseeking/

Here is more complex problem - a game of Agar.io:
https://wagenaartje.github.io/neataptic/articles/agario/


How can both agents know the terminal reward in self-play reinforcement learning?

When one agent makes a move, that move should be perceived as part of the "state transition" executed "by the environment" from the perspective of the other agent.

For example, suppose that, as a "neutral third party" we view the game as follows, as a sequence of states, actions and a terminal reward. I use A to denote actions selected by the first player, and B to denote actions selected by the second player:

S1 -> A1 -> S2 -> B1 -> S3 -> A2 -> S4 -> B2 -> S5 -> A3 -> Terminal Reward

Then, the first player should only get the following observations:

S1 -> A1 -> S3 -> A2 -> S5 -> A3 -> Terminal Reward

note how states S2 and S4 are skipped entirely, they are not really states from the perspective of the first player, they're just halfway through the transition caused by the first player's action and are not interesting for the first player.

Similarly, the second player should only get the following observations:

S2 -> B1 -> S4 -> B2 -> Terminal Reward


In terms of transfer learning, semantic gap means different meanings and purposes behind the same syntax between two or more domains. For example, suppose that we have a deep learning application to detect and label a sequence of actions/words $a_1, a_2, \ldots, a_n$ in a video/text as a "greeting" in a society A. However, this knowledge in Society A cannot be transferred to another society B that the same sequence of actions in that society means "criticizing"! Although the example is very abstract, it shows the semantic gap between the two domains. You can see the different meanings behind the same syntax or sequence of actions in two domains: Societies A and B. This phenomenon is called the "semantic gap".