Creating a self learning Mario Kart game AI?

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)

In general, you are looking at a problem that involves sequential decision making, in a machine learning context.

If you are wanting to build an agent that can learn by receiving screen images, then NEAT cannot scale to that complexity directly. Although there might be clever combinations of deep learning and evolutionary algorithms you could apply, the most heavily explored and likely successful solution will be found in Deep Reinforcement Learning. Algorithms like DQN, A3C, A2C, PPO . . . there are dozens to consider, but all are based around agents using samples of experience to update functions that measure either the "value" of acting in a certain way (a policy) or estimate the best policy directly.

  Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?

Within limits, yes. You will have built a system that takes pixel inputs, and outputs controller messages. If you re-start training from scratch on any compatible N64 game (with same screen resolution and same controller outputs) there is a chance it could learn to play that new game well. As other driving games on the N64 are a subset of all games, and more similar to each other than, say, a scrolling shooter or adventure game, then an agent that can successfully learn one will likely learn another too.

It is unlikely that a Mario Kart agent will immediately be good at another game without re-training. The visual and control differences will probably be too much. You could try though. An interesting experiment would be to take your trained agent, or some part of it, and see if starting with that improves learning time on a new game. This is called Transfer Learning.

  What books/material should i read up on to undertake this project?

You will need at least the first parts of an introduction to Reinforcement Learning. If your goal is to aim to just make the agent, then you can skip theory-heavy parts, but within limits the more theory you understand, the easier it will be to change code features towards getting something working. I can suggest the following:

The book Sutton & Barto, Reinforcement Learning: An Introduction - this covers theory behind basic RL, and the online draft is free
A course of lectures on Reinforcement Learning from UCL, by David Silver. The video lectures cover same topics as the Sutton & Barto book.
The book Deep Reinforcement Learning Hands-On, which you will need to buy. As the title suggests, it covers more practical aspects of building RL systems. It uses Python and PyTorch. As RL has become more popular in recent years, there are plenty of other similar books - I have no reason to recommend this one over them, but I am reading it at the moment, and the quality is adequate.

  What other considerations should i take throughout this project?

Before you get started, you should know this could be an ambitious project, requiring a lot of compute time using currently-available toolkits. You need to think ahead a little, as you will be faced with these decisions:

Is it more important to you to make a working bot on this problem, or more important to understand the underlying theories so that you understand what is going on. You will need to do at least a little of both, but you can go strongly in either direction:

There are enough pre-built learning agents available on the internet that you might have a successful project by learning just enough to wire up a copy of the game, and then letting the learning agent do its job. Would this be enough for you, would you feel that you had solved your problem?
There is plenty of educational material about RL and how the maths behind it works. If you are interested in understanding that, and perhaps generating your own ideas for improvements to current algorithms, then you need to study that material harder. However, this may lead you away from your Mario Kart player goal, and you may never actually solve it, because far simpler problems that require less computing power are still actually interesting academically.

In addition, I can think of the following:

Building your Mario Kart player is a long-term goal. You will need to start with simpler agents in order to understand the techniques that you hope to apply to Mario Kart.
You may need more compute power to solve this game than you have available.
You will need to solve the issue of automating control of a N64 emulator. There is at least an existing emulator Mupen64Plus - I do not know whether it will be adequate for you, but at least one person has attempted to wrap this for automated learning, in the gym-mupen64plus project.


Are there any good ways of simultaneously incorporating object detection with speech recognition?

Check out this paper. It deals with the problem of mixing input modalities! Here's the abstract.

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.


Loss function for choosing a subset of objects

The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a given input can either belong or not. More specifically, what you're trying to do is multi-label classification, which is different from multi-class classification. The difference is important to stress out and it consists in the format of the target labels.
# Multi-class --> one-hot encoded labels, only 1 label is correct

[1,0,0], [0,1,0], [0,0,1]

# Multi-label --> multiple labels can be correct

[1,0,0], [1,1,0], [1,1,1], [0,1,0], [0,1,1], [0,0,1]

MSE is used when continuous values are predicted for some given inputs, therefore it belongs to the loss functions suitable for regression and it should not be used for your problem.
Two loss functions that you could apply are Categorical Cross Entropy or Binary Cross Entropy. Despite being both based on cross-entropy, there is an important difference between them, consisting of the activation function they require.

Binary Cross Entropy

$$L(y, \hat{y})=-\frac{1}{N} \sum_{i=0}^{N}\left(y * \log \left(\hat{y}_{i}\right)+(1-y) * \log \left(1-\hat{y}_{i}\right)\right)$$
Despite the name that suggests this loss should be used only for binary classification, this is not strictly true and actually, this is the loss function that conceptually is best suited for multi-label tasks.
Let's start with the binary classification case. We have a model that returns a single output score, to which the sigmoid function is applied in order to constrain the value between 0 and 1. Since we have a single score, the resulting value can be interpreted as a probability of belonging to one of the two classes, and the probability of being to the other class can be computed as 1 - value.
What if we have multiple output scores, for example, 3 nodes for 3 classes? In this case, we still could apply the sigmoid function, ending up with three scores between 0 and 1.

The important aspect to capture is that since the sigmoid treat each output node independently, the 3 scores would not sum up to 1, so they will represent 3 different probability distributions rather than a unique one. This means that each score after the sigmoid represents a distinct probability of belonging to that specific class. In the example above, for example, the prediction would be true for the two labels with a score higher than 0.5 and false for the remaining label. This also means that 3 different loss have to be computed, one for each possible output. In practice, what you'll be doing is to solve n binary classification problems where n is the number of possible labels.

Categorical Cross Entropy

$$L(y, \hat{y})=-\sum_{j=0}^{M} \sum_{i=0}^{N}\left(y_{i j} * \log \left(\hat{y}_{i j}\right)\right)$$
In categorical cross-entropy we apply the softmax function to the output scores of our model, to constrain them between 0 and 1 and to turn them into a probability distribution (they all sum to 1). The important thing to notice is that in this case, we end up with a unique distribution because, unlike the sigmoid function, the softmax consider all output scores together (they are summed in the denominator).

This implies that categorical cross-entropy is best suited for multi-class tasks, in which we end up with a single true prediction for each input instance. Nevertheless, this loss can also be applied also for multi-label tasks, as done in this paper. To do so, the authors turned each target vector into a uniform probability distribution, which means that the values of the true labels are not 1 but 1/k, with k being the total number of true labels.
# Example of target vector tuned into uniform probability distribution
[0, 1, 1] --> [0, .5, .5]
[1, 1, 1] --> [.33, .33, .33]

Note also that in the above-mentioned paper the authors found that categorical cross-entropy outperformed binary cross-entropy, even this is not a result that holds in general.
Lastly, the are other loss that you could try which differ functions rather than cross-entropy, for example:

Hamming-Loss

It computes the fraction of wrong predicted labels
$$\frac{1}{|N| \cdot|L|} \sum_{i=1}^{|N|} \sum_{j=1}^{|L|} \operatorname{xor}\left(y_{i, j}, z_{i, j}\right)$$

Exact Match Ratio

Only predictions for which all target labels were correctly classified are considered correct.
$$ExactMatchRatio,\space M R=\frac{1}{n} \sum_{i=1}^{n} I\left(Y_{i}=Z_{i}\right)$$


What are the learning limitations of neural networks trained with backpropagation?

Multilayer Perceptron (MLP) can theoretically approximate any bounded, continuous function. There's no guarantee for a discontinuous function. There are plenty of important discontinuous functions, like, say, the prime counting function.
The prime counting function $\pi(n)$ is simply equal to the number of primes less than or equal to $n$. It has a discontinuity about each prime $p$, so good luck trying to approximate this with a neural network!
However, this function is extensively studied and extremely important in number theory. See the Riemann hypothesis.


What is the difference between object tracking and trajectory prediction?

Although both processes might be doing estimations (because data sources aren't perfect and/or have noise), there is a key difference:

Object tracking cares solely abouth where objects are now. That means that there is actual sensor data that can support the current position. For example: from a camera and a lidar, the computer predicts where a vehicle stands.

Trajectory prediction is done with the main purpose of predicting where objects will be in the future, meaning that there is no sensor data yet. For example: from past data, the computer predicts where a vehicle will be after 1 second from now.

However, both processes might need each other. In order to predict a trajectory, it could be easier to work with curated positions given by an object tracker than raw sensor data. On the other hand, past predictions might be given as input to an object tracker, in order to mitigate noise in the sensor data and acting so as a belief-based filter.


Reinforcement Learning with Adaptive Action Magnitude

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.


How could AGI build its own communication protocol?

This is a purely theoretical question, currently in the realm of philosophy and speculative fiction.  Nevertheless, it is an interesting question, and may be instructive.

If we use the standard definition of Artificial General Intelligence as automata with human level intelligence, then they could certainly devise their own communications protocols, just as humans have.

1) These automata are AGI so they are creative and resilient, just like humans, and 'where there is a will there is [sometimes] a way." Absent robotic capacity they wouldn't be able to build anything physical and would have to rely on existing communications infrastructures. If they had access to 3D printing and versatile robots, they could probably build something new, but this would be infeasible for anything that requires extensive capital outlay, unless the automata first acquired a major communications infrastructure firm or two.

2) It's always interesting (and usually quite entertaining) to observe chatbots conversing with each other, but if they were AGIs, NLP is just one of many functions, and I doubt they would bother conversing with each other in human language, since all data is ultimately reduced to a string. If they were smart enough to be deemed AGI, they would certainly communicate in-species with the most efficient protocol available, and probably optimize it further, if possible, or create a unique protocol for purposes of exclusivity.

If they are truly AGIs, you wouldn't ostensibly have to induce them to communicate, because they'd be smart enough to understand the benefits of communication and cooperation, and would likely seek to form coalitions as a natural survival function. (Game Theory provides a mathematical basis for this.)

Multi-agents systems can self organize, even where the intelligence of the given agents is low, and in your scenario, the automata are smart.


Does artificial intelligence write its own code?

Does artificial intelligence write its own code and then execute it? If so, does it create separate functions for each purpose?

Artificial intelligence comprises more than one technique. Here you seem to be asking about machine learning, a subfield of AI which itself contains many subfields.

Self-modifying code is one of the (many) techniques used in some applications of artificial intelligence.  So, in that sense, the answer to your question is "yes".

In Neural Networks (NNs), for example, the NN isn't writing any code, it is just running through an optimization algorithm (and back-propagation) that incrementally changes some weights (or coefficients) such that when you enter a certain input, you get an output that's close to the desired output.

You also have genetic algorithms (GA) to evolve the weights of a neural network, as opposed to using back-propagation. GAs are closer to the idea of "an AI coding itself" (although not exactly).


Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

In general, no.

There is a tradeoff between making the validation set for each fold smaller, and having more folds in total.

As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance of the model. In fact, you can construct examples where the error is arbitrarily large.

For this reason, people sometimes use Bootstrap Validation if they need a very large number of folds. In practice though, most people just us 10 folds, and that's "good enough".


Am I able to visualize the differentiation in backprop as follows?

Yes, a neural network plus loss function can be viewed as a function composition as you have written, and back propagation is just the chain function repeated. Your equations $L = f(g(h(\dots u(v(\dots))))$ and $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ are useful for high level intuition.

At some point you need to look at the actual forms of the functions, and that introduces a bit more complexity. For instance, the loss function for a mini-batch can be expressed in terms of the output. This is the equivalent of $L = f(g)$ for a batch or mini-batch with mean squared error loss, where I am using $J$ (for cost) in place of $L$ and the output of the neural network $\hat{y}$ in place of $g$:

$$J = \frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$$

The gradient of $J$ with respect to $\hat{y}$ is equivalent to your first part $\frac{\partial f}{\partial g}$:

$$\nabla_{\hat{y}} J = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i - y_i)$$

Many of the functions in a neural network involve sums over terms. They can be expressed as vector and matrix operations, which can make them look simpler, but you still need to have code somewhere that works through all the elements.

There is one thing that the function composition view does not show well. The gradient you want to calculate is $\nabla_{\theta} J$, where $\theta$ represents all the parameters of the neural network (weights and biases). The parameters in each layer are end points of back propagation - they are not functions of anything else, and the chain rule has to stop with them. That means you have a series of "dead ends" - or possibly another way of thinking about it would be that your $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ is the "trunk" of the algorithm that links layers together, and every couple of steps there is a "branch" to calculate the gradient of the parameters that you want to change.

More concretely, if you have weight parameters in each layer noted as $W^{(n)}$, and two functions for each layer (the sum over weights times inputs, and an activation function) then your example ends up looking like this progression:

$$\frac{\partial L}{\partial W^{(n)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial W^{(n)}}$$

$$\frac{\partial L}{\partial W^{(n-1)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}  \frac{\partial j}{\partial W^{(n-1)}}$$

$$\frac{\partial L}{\partial W^{(n-2)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}\frac{\partial j}{\partial k}\frac{\partial k}{\partial l}  \frac{\partial l}{\partial W^{(n-2)}}$$

. . . this is the same idea but showing the goal of calculating $\nabla_{W} L$ which doesn't fit into a single chain of gradients. Notice the last term on each line does not appear in the next line. However, you can keep all the terms prior to that and re-use them in the next line - this matches the layer-by-layer calculations in many implementations of back propagation.


Tips for keeping the distribution of weights normal

Quantizing of weights is used very often in implementing neural networks with the tensorflow library on mobile devices. The reason is, the such microcontrollers doesn't have a floating point unit and the developer is trying to increase the overall performance. Quantizing means, to map float values to integer values:

Integer Float
0      -5.0
128    0.0
256    5.0

The problem isn't trivial, because the float point numbers can have a certain minimum, maximum and precision. Maintaining the original distribution is also an issue.

To avoid such kind of problems, the best idea is to implement a floating point library from scratch. In the stackbased Forth language this can be realized in under 400 bytes. It was described in the paper “Willi Stricker: Forth Floating Point Word-Set without Floating Point Stack, 2012”


What are the pros and cons of using standard deviation or entropy for exploration in PPO?

Both implementations may be closer than you think.

In short:

PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising actions/policies. And there is a term added to the loss function aiming to prevent a collapse of the noisiness, to help ensure exploration continues and we don't get stuck at a bad (local) equilibrium.

In fact, for continuous action, the term for entropy in the loss function you describe in Ex. 2, can make sense only when the actions are stochastic, i.e. when there action choice has some standard deviation the way you describe in Ex. 1.

More detail:

On one hand, PPO (at least for continuous action), trains a central/deterministic value (say the mean policy or close to mean) targeting a most profitable action path. On the other hand, along with it, a standard deviation making the actions a random draw with noise around the deterministic value. This is the part you describe in Example 1. The noise helps explore new paths and to update the policy according to the rewards on these sampled paths.
Entropy itself is a measure of the noisiness of the draws, an thus also an indirect indicator for the trained standard deviation value(s) of the policy.

Now, entropy tends to decay as training progresses, that is, the random draws become progressively less random. This can be good for reward maximization - really the best draws are taken for reward maximization - but it is bad for further improvements of policy: improvement may halt or slow down as exploration of new action paths fades.

This is where entropy encouragement comes in. PPO foresees the inclusion of entropy in the loss function: we reduce the loss by x * entropy, with x the entropy coefficient (e.g. 0.01), incentivizing the learning network to increase the standard deviations (or, to not let them drop too much). This part is what you describe in Example 2.

Further notes:

During exploitation, we'd typically turn off the noise (implicitly assuming action std = 0) and pick deterministic actions: in normal cases this increases the payoffs; we're choosing our best action estimate, rather than at a random value around it.
People are not always precise when referring to the model's entropy vs. the entropy coefficient added to the loss function.
Other RL algorithms with continuous action tend to use noisy drwas with standard deviations/entropy too.


I don't know of a pre-canned algorithm but I would just sweep on the angle from zero to ninety degrees with a triangular region and count the points.  For each step in the sweep, record the angle and the count.  When the sweep is done you will have an array of angles with bin counts and then you can convert to percentage of total count.  You will have to figure out a resolution of the triangle's angle so that it is fine enough to capture less than 5% of the points.


What is the exact structure within the nodes of a hidden layer?

Let's go part-by-part.

For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?

In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?

These two are correct. No issues here. The only differences may be in implementation, the logic here is sound.

If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?

This is correct. Intuitively, it is analogous to "offest" term in the equation of a line, y = mx + c. c here plays the same role as the bias.

In terms of customization, for most out of the box NNs, do we choose to have a bias term or not? Or for the most part, it automatically includes one.

Most out-of-the box NN include a bias term by default, since they do provide flexibility to the network, no matter how small. Practically, their utility varies with architecture and application. For example, Feed Forward networks have a good boost in performance with biases when used for regression. CNNs used for classification usually aren't affected much if biases are removed.

In terms of customization, this combination of inputs is always linear? There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)

Non-linear combinations are definately possible. But using such combinations basically kills the purpose of ANNs - to have simple stacked operations approximate a complex function. Non-linear combinations may help in rare cases, but more often than not, will probably lead to underfit or overfit models. By convention, any non-linearity is introduced into the model on top of the linear combination, as the activation function.

For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?

Conventionally, all nodes in the same layer have the same activation. This makes vectorization easy, and ultimately doesn't really affect preformance. Instead of varying activations of individual neurons, complex architectures use parallel layers having different activations but with the same inputs. Usually, all stacked layers in models share the same activations, except maybe the last layer and in some cases, the penultimate layer.


I just came across this piece of news yesterday: "This week, Microsoft Research threw down the gauntlet with the launch of a competition challenging researchers around the world to develop AI agents that can solve text-based games."
This seems to be an AI competition announced by Microsoft with the aim to create AI that can solve text-based games. This might give you some inspiration.


Did Turing foresee the required capabilities to pass the Turing test?

I find it unlikely that you'll find a firm answer, so I will try my best to guide you towards information which may help you form an opinion either way. Turing had the controversial opinion (which remains controversial today) that:

  Digital computers have often been described as
  mechanical brains. Most scientists probably
  regard this description as a mere newspaper
  stunt, but some do not. One mathematician has
  expressed the opposite point of view to me rather
  forcefully in the words “It is commonly said that
  these machines are not brains, but you and I
  know that they are.” […] I shall give most attention
  to the view which I hold myself, that it is
  not altogether unreasonable to describe digital
  computers as brains.

In essence Turing believed that the digital computers of the time had the capacity to mimic the human brain if programmed correctly. The problem for Turing then, was finding the correct procedures to mimic the brain. Remember the Imitation Game was being used as a means to explore the question "can machines think?" Turing was engaging in an intellectual debate about the nature of "thinking", he was not making an attempt to describe how machines would be taught to think.

Since, both artificial intelligence and machine learning are at their core very similar to "artificial thinking" (as Turing may have called it) it seems necessary that a test designed to decide whether machines can think would also encompass whether machines could be intelligent or whether machines could learn.

Therefore, it seems possible that Turing did not make the question with these capabilities in mind, but rather that these capabilities are what guided Turing to this question as they are so innately tied to thinking.

If you would like to look further into Turing's thought process here is his paper on the imitation game and here is a discussion on the proceedings of a BBC broadcast about "Automatic Calculating Machines" in which Turing was a speaker.


When should Monte Carlo Tree search be chosen over MiniMax?

Some basic advantages of MCTS over Minimax (and its many extensions, like Alpha-Beta pruning and all the other extensions over that) are:

MCTS does not need a heuristic evaluation function for states. It can make meaningful evaluations just from random playouts that reach terminal game states where you can use the loss/draw/win outcome. So if you're faced with a domain where you have absolutely no heuristic domain knowledge that you can plug in, MCTS is likely a better choice. Minimax must have a heuristic evaluation function for states (exception: if you game is so simple that you can afford to compute the complete game tree and reach all terminal game states immediately from the the initial game state, you don't need heuristics). If you do have strong evaluation functions, you can still incorporate them and use them to improve MCTS too; they're just not strictly necessary for MCTS.

MCTS has simpler anytime behaviour; you can just keep running iterations until you run out of computing time, and then return the best move. Typically we expect the performance level of MCTS to grow with computatinon time / iteration count relatively smoothly (not always 100% true, but intuitively you can usually expect something like this). You can sort of achieve anytime behaviour in minimax with iterative deepening, but that's usually a bit less "smooth", a bit more "bumpy"; this is because every time you increase the search depth, you need significantly more processing time than you did for the previous depth limit. If you run out of time and have to abort your current search at your current depth limit, that last search will be completely useless; you'll have to discard it and stick to the results from the previous search with the previous depth limit.

A difference, which is not necessarily an advantage or disadvantage either way in the general case (but can be in specific cases):

The computation time of MCTS is generally dominated by running (semi-)random playouts. This means that functions for computing legal move lists, and applying moves to game states, typically dictate how fast or slow your MCTS runs; making these functions faster will generally make your MCTS faster. On the other hand, the computation time of Minimax is generally dominated by copying game states (or "undoing" moves, which is an operation that in most games will require additional memory usage for game states to be possible) and heuristic evaluation functions (though the latter are likely to also become important in terms of computation cost in MCTS if you choose to include them there). In some games it will be easier to provide efficient implementations for one of these, and in other games it may be different.

A basic advantage of Minimax over MCTS:

In settings where MCTS can only run very few iterations relative to the branching factor (or in the extreme case, fewer iterations than there are actions available in the root node), MCTS will perform extremely poorly / close to random play. We've noticed this being the case for quite a decent number of games in our general game system Ludii (where the "general game system" often implies that games are implemented less efficiently than they could be in a dedicated single-game-specific program) with low time controls (like 1 second per move). This same general game setting often makes it difficult to find super strong heuristics, but it's generally still possible to come up with some relatively simple ones (like just a simple material heuristic in chess). An alpha-beta search with just a couple of search plies and a basic, simple heuristic will often outperform a close-to-random MCTS if the MCTS can't manage to run significantly more iterations than it has legal moves in the root node.


When calculating the cost in deep Q-learning, do we use both the input and target states?

I will first explain briefly to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provided with some data $\{(\textbf{x}_i, y_i)\}_{i=1}^n$ where $\textbf{x}_i$ are the features for data point $i$ and $y_i$ is its true label. Now, the aim of supervised learning is to learn a function $f$ that can accurately predict the label of a data point given its features. In deep learning this function is a neural network, $f_\theta(\cdot)$. To optimise the parameters we obtain the models prediction for the label of $y$, denoted typically by $\hat{y} = f_\theta(x)$ and we look to optimise the parameters of the function $\theta$ by minimising the loss $\mathcal{L}(\hat{y}, y)$ (note that here the loss is a function $\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}$.
In reinforcement learning things are quite different. We are not provided with any data. Instead we have a Markov Decision Process (MDP) and an environment that we can interact with. The state space is defined by the MDP and we can get samples, mainly tuples of the form $(s, a, r, s')$, that we can use to teach our agent how to find an optimal policy - typically an optimal policy is one that takes an action given the current state that will maximise the sum of the future rewards of the episode. So in reinforcement you can probably see that things are quite different to supervised learning, mainly in the way that we 'obtain' our data.
Of course these explanations are gross oversimplifications of the learning process in both paradigms and should only be used as a brief example to try to emphasise the different between the two learning paradigms.
Now, there are two ways you can parameterise your Q-function using a neural network:

The network takes as input the current state and outputs Q-values for each potential action; i.e. the output is a vector in $\mathbb{R}^{|\mathcal{A}|}$;
The network takes as input the state and action and outputs a real number which is the Q-value for the state, action tuple you pass as input to the network.

The second way is usually reserved for instances where you have a huge action space but only ever would consider a few feasible actions at each state - this saves computational complexity.
Now, to answer your question:
In case 1) you are assumed to have access to a transition tuple $(s, a, r, s')$. The temporal difference, which we will use as the target, is $\hat{y} = r + \max_a Q(s', a)$. Now, as our output of $Q(s', \cdot)$ is an $\mathbb{R}^{|\mathcal{A}|}$ what we do is make a forward pass of the network for the current $(s, a)$ tuple, i.e. we get $x = Q(s, a)$, and then change the element that corresponds to the action which satisfies $\arg\max_a Q(s', a)$ of $x$ to $\hat{y}$, so we have now got our augmented input $\tilde{x}$ which serves as our target.
To make that step a bit clearer, suppose we have a 2-dimensional action space and the $\arg\max_a Q(s', a)$ is the action in the first dimension, then we would change the first dimension of $x$ to be $\hat{y}$.
We then train the network using the Mean Squared Error Loss between $x$ and $\tilde{x}$ - note that no gradient information is retained when we do the forward pass to get $\tilde{x}$; in fact we usually don't use a current version of the $Q$ network, we use an 'old' version of the network called the target network (I imagine there's probably a question about this network on the site already so I won't explain it in detail).
In case 2) the idea is much more simple as the outputs of the network are scalars so you can just train your network using the MSE between the scalar values of $Q(s, a)$ and $\hat{y}$ as defined above, the caveat here is that to calculate $\arg\max_a Q(s', a)$ you have to make $|\mathcal{A}|$ forward passes of the network for all possible $(s', a)$, for fixed $s'$, tuples which is why method 1) is usually preferred.
Now, whilst we may use techniques from supervised learning to optimise the Q-function, you can hopefully see the differences between supervised and reinforcement learning.
In short, the temporal difference is the target that you train your network towards being able to predict, and the state is (typically) the input into your neural network. For a more in depth discussion of RL being framed as supervised learning please see this answer.


Is the VC Dimension meaningful in the context of Reinforcement Learning?

Yes, it is. This article (Approximate Planning in Large POMDPs via Reusable Trajectories) explain about it by means of the trajectory tree:

A trajectory tree is a binary tree in which each node is labeled by a state and observation
pair, and has a child for each of the two actions. Additionally, each link to a child is
labeled by a reward, and the tree's depth will be $H_\epsilon$, so it will have about $2^{H_\epsilon}$ nodes. The root is labeled by $s_0$ and the observation there, $o_0$.

Now a policy $\pi$ will be defined like the following base on the trajectory tree:

For any deterministic strategy $\pi$ and any trajectory tree $T$, $\pi$ defines a path through
$T$: $\pi$ starts at the root, and inductively, if $\pi$ is at some internal node in $T$, then we feed to $\pi$ the observable history along the path from the root to that node, and $\pi$ selects and moves to a child of the current node. This continues until a leaf node is reached, and we define $R(\pi, T)$ to be the discounted sum of returns along the path taken. In the case that $\pi$ is stochastic, $\pi$ defines a distribution on paths in $T$, and $R(\pi, T)$ is the expected return according to this distribution.  Hence, given $m$ trajectory trees $T_1 , \ldots , T_m$, a natural estimate for $V^\pi(s_0)$ is $V^\pi(s_0) = \frac{1}{m}\sum_{i=1}^mR(\pi, T_i)$. *Note that each tree can be used to evaluate any
strategy, much the way a single labeled example $\langle x, f(x)\rangle$ can be used to evaluate any hypothesis $h(x)$ in supervised learning. Thus in this sense, trajectory trees are reusable.

Now similar to definition of VC theory for classification methods:

Our goal now is to establish uniform convergence results that bound the error of the estimates $V^\pi(s_0)$ as a function of the "sample size" (number of trees) $m$.

And finally, we have the following theorem:

Let $\Pi$ be any finite class of deterministic strategies for an arbitrary two-action POMDP $M$. Let $m$ trajectory trees be created using a generative model for $M$, and
$\widehat{V}^\pi(s_0)$ be the resulting estimates. If $m = O((V_{\max}/\epsilon)^2(\log(|\Pi|) + \log(1/\delta)))$, then with probability $1 - \delta$, $|V^\pi(s_0) - \widehat{V}^\pi(s_0)|\leqslant \epsilon$ holds simultaneously for all $\pi \in \Pi$.

About the VC dimension of $\Pi$, if we suppose we have two actions $\{a_1, a_2\}$ (it can be generalized to more actions), we can say:

If $\Pi$ is a (possibly infinite) set of deterministic strategies, then each strategy $\pi \in \Pi$ is simply a deterministic function mapping from the set of observable histories to the set $\{a_1, a_2\}$,
and is thus a boolean function on observable histories. We can, therefore, write $\mathcal{VC}(\Pi)$ to
denote the familiar VC dimension of the set of binary functions $\Pi$. For example, if $\Pi$ is
the set of all thresholded linear functions of the current vector of observations (a particular
type of memoryless strategy), then $\mathcal{VC}(\Pi)$ simply equals the number of parameters.

and the following theorem:

Let $\Pi$ be any class of deterministic strategies for an arbitrary two-action
POMDP $M$, and let $\mathcal{VC}(\Pi)$ denote its VC dimension. Let $m$ trajectory trees be created using a generative model for $M$, and $\widehat{V}^\pi(s_0)$ be the resulting estimates. If:
$$
m = O((V_{\max}/\epsilon)^2(H_\epsilon\mathcal{VC}(\Pi)\log(V_{\max}/\epsilon) + \log(1/\delta)))
$$
then with probability $1 - \delta$, $|V^\pi(s_0) - \widehat{V}^\pi(s_0)|\leqslant \epsilon$ holds simultaneously for all $\pi \in \Pi$.


Oh yeah, definitely.  Just to pick one example, you have Douglas Hofstader's group at Indiana.  I think most of what they do would fall under the rubric of GOFAI (or at least closer to that than the statistical machine learning stuff).

Beyond that, just go to the CORR and browse around the AI category. You'll see plenty of neural networks and probabilistic stuff, but you'll also find the papers by the folks doing symbolic processing / GOFAI as well.


Here is a little bit of aglimpse

Asus Zenbook pro,is quite good for technical nor software development tasks,despite having ordered one recently.

That said,you need to that each and every computer/machine for effective and efficient computing purpose,the first priority is to analyse critically the processor chip,which generational architecture/design does it have inline with GPU/Graphics ,Resolution,motherboard,hard-drive...e.t.c..

Therefore,basing on such technical specs,what matters most,is not the brand nor logo..! Go for any brand,keeping in mind your objective "DeepLearning"


What is the intuition behind grid-based solutions to POMDPs?

I will attempt to provide an answer to your questions based on the information you can find in the papers A Heuristic Variable Grid Solution Method for POMDPs (1997) by Ronen I. Brafman and Point-based value iteration: An anytime algorithm for POMDPs (2003) by Joelle Pineau et al.

A grid-based approximate solution to a POMDP attempts to estimate a value function only at a subset of the number of belief states. Why? Because estimating the value function for all belief states is typically computationally infeasible for non-small POMDPs, given that the belief-space MDP (i.e. an MDP where the state space consists of probability distributions over the original states of the POMDP) of a POMDP with $n$ states has an uncountably large state space. Why? Because of the involved probability distributions.

How do we compute the value for the belief states that do not correspond to a point of the grid? We can use e.g. interpolation, i.e. the value of a belief state that does not correspond to a point of the grid is computed as a function of the value of the belief states that correspond to other grid points (typically, the neighboring grid points).

Why is this approach feasible? The assumption is that interpolation is not as expensive as computing the value of a belief state. However, note that you may not need to interpolate at every step of your algorithm, i.e. interpolation could be performed only when the value of a certain belief state is required.

How do you compute the value of a belief state that corresponds to a grid point? It can be computed with a value iteration (dynamic programming) algorithm for POMDPs. An overview of a value iteration algorithm can be found in section 2 of the paper Point-based value iteration: An anytime algorithm for POMDPs. Here's an example of the application of the value iteration algorithm for POMDPs.

The grid-based approach, introduced in Computationally Feasible Bounds for Partially Observed Markov Decision Processes (1991) by William S. Lovejoy, is very similar to the point-based approach, which was introduced in Point-based value iteration: An anytime algorithm for POMDPs. The main differences between the two approaches can be found in section 3 of Point-based value iteration: An anytime algorithm for POMDPs.

The idea of discretizing your problem or simply computing the desired value at a  subset of the domain has been applied in other contexts too. For example, in the context of computer vision, you can approximate the derivative (or gradient) of an image (which is thus considered a function) at discrete points of the domain (i.e. the pixels).

There's a Julia implementation of the first grid-based approximative solution to POMDP. There's also a Python implementation of the point-based approach. These implementations may help you to understand the details of these approaches.


Why does using a higher representation space lead to performance increase on the training data but not on the test data?

The answer to your question is that the capacity of your model (i.e. the number and type of function that your model can compute) generally increases with the number of parameters. So, a bigger model can potentially approximate better the function represented by your training data, but, at the same time, it may not take into account the test data, a phenomenon known as over-fitting the training data (i.e. fitting "too much" the training data).
In theory, you want to fit the training data perfectly, so over-fitting should not make sense, right? The problem is that, if we just fit all the (training) data, there is no way of empirically checking that our model will perform well on unseen data, i.e. will it generalize to data not seen during training?
We split our data into training and test data because of this: we want to understand whether our model will perform well also on unseen data or not.
There are also some theoretical bounds that ensure you that, probabilistically and approximately, you can generalize: if you have more training data than a certain threshold, the probability that you perform badly is small. However, these theoretical bounds are often not taken into account in practice because, for example, we may not be able to collect more data to ensure that the bounds are satisfied.

Surely the representations/patterns learnt in the training data will be found too in the test data.

This is possibly the wrong assumption and the reason why you are confused. You may assume that both your training data and test data come from the same distribution $p(x, y)$, but it does not necessarily mean that they have the same patterns. For example, I can sample e.g. 13 numbers from a Gaussian $N(0, 1)$, the first 10 numbers could be very close to $0$ and the last $3$ could be close to $1$. If you split this data so that your training data contains different patterns than the test data, then it is not guaranteed that you will perform well also on the test data.
Finally, note that, in supervised learning, our ultimate goal when we fit models to labeled data is to learn a function (or a probability distribution over functions), where we often assume that both the training and test data are input-output pairs from our unknown target function, i.e. $y_i = f(x_i)$, where $(x_i, y_i) \in D$ (where $D$ is your labelled dataset), and $f$ is the unknown target function (i.e. the function we want to compute with our model), so, if our model performs well on the training data but not on the test data and we assume that both training and test data come from the same function $f$, there is no way that our model is computing our target function $f$ if it performs badly on the test data.


Is my GRU model under-fitting given this plot of the training and validation loss?

When ever you are buliding a ML Model don't take accuracy seriously(Mistake done by Netflix that cost them alot), you should try to get the hit scores as they will help you to know how many times your model worked on real world users.However, if your model must have to measure the accuracy try it with the RMSE score as it will penalise you more for being more out of the Line. Here is the link for more information on it RMSE
Its hard to predict if its overfitting or underfitting as your graph is vague(for example what does graph lines representing). However, you can solve underfitting by following steps:
1) Increase the size or number of parameters in the ML model.
2)  Increase the complexity or type of the model.
3)  Increasing the training time until cost function in ML is minimised.

For overfitting you can try Regularization methods like weight decay provide an easy way to control overfitting for large neural network models. A modern recommendation for regularization is to use early stopping with dropout and a weight constraint.


Understanding the notation in the definition of the expected reward

Your first option is correct:

$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$

It's partly a matter of taste, but I prefer not moving the $r$ into the double sum, because its value does not change in the "inner loop". There is a small amount of intuition to be had that way around, especially when it comes to implementation (it is one multiplication after the sum, as opposed to many within the sum).

There are a lot of sums containing sums in Sutton & Barto, and they mainly follow the convention of not using any parentheses or brackets to show the one sum containing the other explicitly.

In this case, the formulae help link to other treatments of RL, which work with the expected reward functions $r(s,a)$ or $r(s,a,s')$, or reward matrices $R_s^a$, $R_{ss'}^a$ such as the first edition of Sutton & Barto's book. The second edition of the book uses $p(s', r|s, a)$ almost everywhere though, and you won't see $r(s,a)$ mentioned much again. So it's not worth getting too concerned about how it is presented and what the author might be saying with the presentation.

Generally you don't need to know the distribution of reward, just its expectation (and how that depends on $s, a, s'$), in order to derive and explain most of the results in RL. So using $r(s,a)$ and similar functions is fine, in places like the Bellman equations. However, the use of $p(s', r|s, a)$ is general without needing to bring in more functions describing the MDP.


Can minimax be used when both players want to increase their score?

I believe maximax is what you're looking for:

Maximax (economics, computer science, decision theory) A strategy or algorithm that seeks to maximize the maximum possible result (that is, that prefers the alternative with the chance of the best possible outcome, even if its expected outcome and its worst possible outcome are worse than other alternatives); often used attributively, as "maximax strategy", "maximax approach", and so on.SOURCE: maximax (wiki)

You may also be interested in "minimax regret":

The minimax regret approach is to minimize the worst-case regret. The aim of this is to perform as closely as possible to the optimal course. Since the minimax criterion applied here is to the regret (difference or ratio of the payoffs) rather than to the payoff itself, it is not as pessimistic as the ordinary minimax approach.
One benefit of minimax (as opposed to expected regret) is that it is independent of the probabilities of the various outcomes: thus if regret can be accurately computed, one can reliably use minimax regret. However, probabilities of outcomes are hard to estimate.
This differs from the standard minimax approach in that it uses differences or ratios between outcomes, and thus requires interval or ratio measurements, as well as ordinal measurements (ranking), as in standard minimax.
SOURCE: Minimax regret (wiki)

Apologies for just grabbing the wikis on this, but they are accurate.  Most of the references from economic sites are commercial.  If interested, when I have some more time, I could probably link to some peer-reviewed papers.  Hopefully, at the very least, this answer provides information for further research.


When training an object detection network for one class, should I include empty images in the dataset?

It is correct that to eliminate the false positives, the training must include images that reflect true negative cases. The objective of training is not this.

  Find the objects of class $O$ in scene $X$.

It is this.

  Find any objects of class $O$ in scene $X$.

Training is effective when sample $S$ is drawn according to the same distribution as will be used for drawing at test time or during actual use.

It is not necessary to annotate the background if the loss function is chosen so that finding nothing when nothing there is indicated as a zero loss and a proper distribution of those cases exist in the training sample.


Iteratively and adaptively increasing the network size during training

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here

NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with fitness evaluation, crossover, and mutation. The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. In, Out, and Weight values are the same as regular neural networks. Enabled and Disabled genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.NEAT can mutate or change both its weights and connections, so for example, Parent1 and Parent2 has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either Parent1 weight or Parent2 weight. The excess and disjoint genes are inherited from the more fit parent. NEAT will then mutate each genome, shown in the image below.


Replacement you suggest is replacement of random variable by its expectation in forward part of TD. It would make IQN into modification of C51 with randomly sampled function approximator instead of discrete distribution. Both distribution produced and especially exploration behavior with your replacement would be very different. The authors of paper explicitly said that "more randomness" in their opinion benefit training, so reducing randomness would go aginst spirit of the paiper.  That they produce similar results on single toy test mean very little. IQN could be better then C51 or it could be worse then C51 but single toy example is not enough to say they are close. Nevertheless I agree that IQN looks overly complex and may require more training time, C51 approach could be more practical.


What happens if there is no activation function in some layers of a neural network?

If you do not specify an activation for a layer you are effectively creating a linear transformation through that layer.  From the documentation:

activation: Activation function to use. If you don't specify anything, no activation is applied (see keras.activations).


Is my understanding of the differences between MDP, Semi MDP and POMDP correct?

Yes, the core differences between the different categories of problems are correct as you've described them.
For SMDPs, I'd like to remark that the water boiling example is maybe not the best. That looks more like an example of "delayed rewards", but not one of "durative actions": when the agent takes that action to raise the temperature, it takes some time before the reward comes in, but the agent's action itself doesn't take that much time. The agent could maybe do something else in the meantime. Delayed rewards are not restricted to SMDPs, they can also show up in regular MDPs (you just have to make sure to include some data in the state representation to indicate the time since the temperature was set or something like that, such that the Markov property is not violated).
A typical example of an SMDP with durative actions would be a grid with rooms, split by walls but connected by smaller doors inside those walls. A single "primitive" action would just take a single step in the larger grid, but longer-duration "macro-actions" would make the agent navigate from somewhere within a room to one of the doors "on auto-pilot". This would still take as much time as it would to navigate explicitly in multiple smaller actions, but the larger more complex behaviour being encapsulated in a single "macro-action" can speed up learning.


Why is the last layer of a DBN or DBM used for classification task?

One of the big realizations that deep learning models brought in recent years was that we can train the feature extractors and classifiers simultaneously. In fact most people have stopped separating the 2 tasks and simply refer to all the process as training the model.

However, if you dive in to every single model architecture, it will always be constructed from the first part which is the feature extractor which outputs the embedding output - (which is basically the x encoded features of the input), and second part consisting of the final layer the model - the classifier which uses the embedding layer encoding to predict the class of the input.

The goal of the first part is to reduce the dimensionality of the input to just the most impotent features for the final task. The goal of the classifier is to use those features to output the final score/class etc.

This is why usually only this layer is fine-tuned, because we don't want to damage the trained feature extractor, just update the classifier to fit a slightly different distribution.

I'm pretty sure that in your mentioned case, for generation they do not use the classification layer, so updating it shouldn't have any affect on the model's generative abilities.

Regarding your last question, yes it is possible, ones you extracted the features with the model, you can use any kind of classifier on them.


What is the most effective way to learn natural language processing online?

I think you could learn NLP ,by taking this course https://see.stanford.edu/Course/CS224N
Natural Language Processing - Stanford University | Coursera and try your best with the homework.

To further study, you could learn some deep learning algorithms like RNN. And of course, it’s better that you will be able to learn more a lot from university professors with minimal time period.

Hope it can help you.


Can a game AI learn the concept of acceleration?

This answer mostly assumes you are referring to computer-game-playing bots that learn through experiencing play, such as Deep Mind's DQN as used for playing Atari console games. State of the art for these are typically Reinforcement Learning algorithms, used with neural networks to process input and estimate results of next actions. There are other competitive AI technologies too, and the answer applies generally to most learning or evolving optimisers that would learn through trial and error by playing the game.

  If the game had a variable speed and was essential in evolution/gaining score(IDK AI terminologies). Would the AI be able to figure out when to slow down and speed up?

Yes, as long as the game allowed control of something that influenced acceleration, then a learning agent can figure out the consequences of accelerating and braking and use them appropriately within the game.

A well-known toy example that can challenge learning agents is called Mountain Car. In that game, the agent has to learn to accelerate in the correct direction (which is not always towards its objective), in order to escape an area. It is considered challenging because the reward (for escaping) can be significantly delayed compared to the action that best enables it.

One popular proving ground for learning agents is OpenAI gym. This includes several game environments with a physical model of acceleration included, such as Lunar Lander.

  If it is able to solve the problem or complete the level, will it have an equation to relating acceleration?

In general, no. The agent will learn to respond to certain stimulus, by taking an action that accelerates or decelerates the game piece that it controls. There will not be any concept like $s = ut + \frac{1}{2} at^2$ encoded in the agent's parameters.

  or perhaps a number on when to speed up and down?

Typically the agent will learn which stimuli should be responded to by accelerating. For instance, in a game where the agent's game piece is being chased by an enemy piece and the enemy piece is getting closer, the agent should learn that it will get a better reward if it accelerates away from the enemy.

  What if the game environment was dynamic?

Most game environments are dynamic, as in the state changes over time. If you mean would anything change if the game rules themselves varied over time, then this may cause interesting problems for some learning algorithms, but should not change anything about learning use of controls that affect acceleration in a virtual world.

  Can you even teach math to an AI?

Generally, no you cannot teach math to the kind of system that plays games, or interacts with real world objects. These kind of learning systems are not yet advanced enough to learn concepts or establish game world logic from interactions. Instead, they work more akin to perception, muscle memory and either inherent or learned reflexes. Exceptions to this will generally have a world model (with the necessary equations) built in or made available to the agent without it needing to learn anything.

However, there are AI systems that use formal logic that can work on mathematical theories. Some have performed interesting feats such as "discovering" prime numbers, given formal definitions of integers and basic arithmetic. An example of this kind of system is the Automated Mathematician.

There is an intermediate possibility: Some learning agents not only learn the value or best policy for certain actions, they also learn to predict what should happen next to the state of the environment. Such an agent would include a model that could observe objects that were accelerating and predict their future positions. In some ways this is a learned concept of acceleration, although it would not be expressed mathematically like Newton's laws of motion, and is more akin to the kind of intuition that allows a person to track, anticipate and catch a thrown ball.


What are the differences between an agent and a model?

Agent
The other answer defines an agent as a policy (as it's defined in reinforcement learning). However, although this definition is fine for most current purposes, given that currently agents are mainly used to solve video games, in the real world, an intelligent agent will also need to have a body, which Russell and Norvig call an architecture (section 2.4 of the 3rd edition of Artificial Intelligence: A Modern Approach, page 46), which should not be confused with an architecture of a model or neural network, but it's the computing device that contains the physical sensors and actuators for the agent to sense and act on the environment, respectively. So, to be more general, the agent is defined as follows

agent = body + policy (brain)

where the policy is what Russell and Norvig call the agent program, which is an implementation of the agent function.
Alternatively, it can be defined as follows

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

This is just another definition given by Russell and Norvig, which I also report in this answer, where I describe different types of agents. Note that these definitions are equivalent. However, in the first one, we just emphasize that we need some means to "think" (brain) and some means to "behave" (body).
These definitions are quite general, so I think people should use them, although, as I said above, sometimes people refer to an agent as just the policy.
Model
In this answer, I describe what a model is or what I like to think a model is, and how it is different from a function.
In AI, a model can refer to different but somehow related concepts.

For example, in reinforcement learning, a model typically refers to $p(s', r \mid s, a)$, i.e. the joint probability distribution over the next state $s'$ and reward $r$, given the current state $s$ and action $a$ taken in $s$.

In deep learning, a model typically refers to a neural network, which can be used to compute (or model) different functions. For example, a neural network can be used to compute/represent/model a policy, so, in this case, there would be no actual difference between a model and an agent (if defined as a policy, without a body). However, conceptually, at a higher-level, these would still be different (in the same way that biological neural networks are different from the brain).

More generally, in machine learning, a model typically refers to a system that can be changed to compute some function. Examples of models are decision trees, neural networks, linear regression models, etc. So, as I also state in the other answer, I like to think of a model as a set of functions, so, in this sense, a model would be a hypothesis class in computational learning theory. This definition is roughly consistent with $p(s', r \mid s, a)$, which can also be thought of as a (possibly infinite) set of functions, but note that a probability distribution is not exactly a set of functions.

In the context of knowledge bases, a model is an assignment to the variables, which represents a "possible world". See section 7.3, page 240, of the cited book.

There are possible other uses of the word model (both in the context of AI, e.g. in the context of planning, there's often the idea of a conceptual model, which is similar to an MDP in RL, and in other areas), but the definitions given above should be more or less widely applicable in their contexts.
What is the difference between an agent and a model?
Given that there are different possible definitions of a model depending on the context, it's not easy to briefly state what the difference between the two is.
So, here's the difference in the context of RL (and you can now find out the differences in other contexts by using the different definitions): an agent can have a model of the world, which allows it to predict e.g. the reward it will receive given its current state and some action that it decides to take. The model can allow the agent to plan. In this same context, a model could also refer to the specific system (e.g. a neural network) used to compute/represent the policy of the agent, but note that people usually refer to $p(s', r \mid s, a)$ when they use the word model in RL. See this post for more details.


Is there a place where people can share (or buy) ready made neural networks?

It's not a specific web for sell or share neural network models, but actually you can easily find other people models in Github! Just search it! For example, this is a random repo I've found for Cat Classification.
But.. the problem is everyone have different problems. So you can't easily use other people neural network models and then use it for your problem. That's why there's a "trick" called transfer learning or fine tuning. That method show you how to train other people models (it's called: pre-trained models) from different problems for your specific problem.
Tensorflow and PyTorch provide a lot of pre-trained model for common problems:

Pytorch Image Classification Pre-Trained models
Tensorflow Object Detection Model Zoo
PyTorch Detectron Model Zoo

You can also find other pre-trained models in Github! This is an official PyTorch tutorial how to use that pre-trained model to your specific case.


In this implementation of pix2pix, why are the weights for the discriminator and generator losses set to 1 and 100 respectively?

After further research, I have found the answer. nbro was of course right, the weighting is not implementation dependant, it was already introduced in the paper (arXiv). However, there is minimal information about that, only it is mentioned within the optimization function:
$\arg \min_G \max_D \mathcal{L}_{cGAN}(G,D) + \lambda \mathcal{L}_{L1}(G)$
In fact, the parameter lambda stands for weight of $L_1$ loss. There should be a second parameter $\epsilon$ for $\mathcal{L}_{cGAN}(G,D)$ because in real implementation you can actually change that as well. In theory, however, only modulation of $L_1$ loss is needed. The paper does not state why the $\lambda$ parameter is needed, only mentions that setting it to 0 will lead to pure cGAN implementation. David Brownlee in his blog states:

The adversarial loss influences whether the generator model can output
images that are plausible in the target domain, whereas the L1 loss
regularizes the generator model to output images that are a plausible
translation of the source image. As such, the combination of the L1
loss to the adversarial loss is controlled by a new hyperparameter
lambda, which is set to 10, e.g. giving 10 times the importance of the
L1 loss than the adversarial loss to the generator during training.

The loss weights thus are hyperparameters that tell the network how much plausible translation of the source image do we need.


Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method is more concerned with overestimation induced by function approximation error rather than stochasticity in the environment.


Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?

Yes, if there's no activation function in the last layer, the weights could simply be negative there, so the network would multiply a positive value with a negative weight, therefore outputting a negative value.
There is still an activation function, but it is the identity.


How to split data into training validation and test set when the number of data in classes varies greatly?

I would make the distribution of the classes in test and validation sets the same as in the training set (and as in the whole original set). Anyway all your metrics are relative, not absolute and designed to provide reasonable results when classes are not ideally balanced.


How to calculate the optimal placements for settlements in Catan without an ML algorithm?

Catan is actually a much more complicated game than the simple rules would suggest, and an exact solution is probably beyond the scope of current AI techniques.

Monte Carlo Tree Search or Expectiminimax techniques seem like they could help, but are intended for games of perfect information. Catan is not a game of perfect information (the development cards are hidden), and also has a phase that occurs without a regular turn sequence (trading).

To solve Catan properly, I think you're going to need both algorithms for solving POMDPs (like CFR+), and algorithms for negotiation (like Kraus' Diplomat). I'm not certain that these have been combined before in formal analysis, so this might actually be a good PhD thesis for someone.

That said, you can probably get a good player using self-play techniques, because Catan has randomization, and a relatively small set of moves, like Backgammon. These may or may not offer simple rules about how-best to play the game. Your friends are right to think about this as, at root, an ML problem.


How to create Partially Connected NNs with prespecified connections using Tensorflow?

So I've done some more research after posting this and it looks like treating things as separate neural nets and connecting them later in subsequent hidden layers is a good idea. If anyone has other ideas that might be helpful, I'd love to hear, though!


What is the difference between parametric and non-parametric models?

Parametric Methods
A parametric approach (Regression, Linear Support Vector Machines) has a fixed number of parameters and it makes a lot of assumptions about the data. This is because they are used for known data distributions. i.e, it makes a lot of presumptions about the data
Non-Parametric Methods
A non-parametric approach (k-Nearest Neighbours, Decision Trees) has a flexible number of parameters, there are no presumptions about the data distribution. The model tries to "explore" the distribution and thus has a flexible number of parameters.
Comparision
Comparatively speaking, parametric approaches are computationally faster and have more statistical power when compared to non-parametric methods

Hope this cleared your doubts


How to implement an Automatic Learning Rate for a Neural Network?

Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:

AdaGrad
RMSProp
Adam

There are many more variants. You'll need to do some research. Please take a look at:

  https://en.wikipedia.org/wiki/Stochastic_gradient_descent


How to deal with the time delay in reinforcement learning?

This problem has been formally termed as Delayed MDP (Katsikopoulos & Engelbrecht, 2003)[1] - the actions generated are not instantly applied to the environment and/or the captured observations are not immediately seen by the agent, as expected in an MDP. The delay can either be:
a) A constant delay - Constant Delayed MDP (CDMDP) (Learning and planning in en-
vironments with delayed feedback, 2008)[2]

The delay introduced by the environment is constant and known. Like the example you provide in the question.

b) A random delay - Random Delayed MDP

We don't know what delay to expect, or when the delay will occur. This is more realistic for a real-world case.

TRIED APPROACHES
1. Just ignore the delay
This assumes the CDMDP is an MDP. We then attempt to search for the policy that best ignores the delay
$$
\pi(I_k)={\pi^∗(s)|I_k=(s, a_1,···,a_k)}
$$
$k$ is the number of time-steps between acting on the current state, and receiving feedback. This is simple and can give a reasonable result if the delay is small compared to the state transition magnitude [2].
2. Reconstruct the MDP from the CDMDP (Augmented approach)
The corresponding optimal policy for the reconstructed MDP will then the optimal policy for the CDMDP. This is however made intractable by the size of the action buffer growing with the delay length. It's therefore limited to small constant delays. [1] defines how to do the re-construction.
3. Predicting the delayed observation
This tries to "undelay" the environment by using a predictive model $P$ to approximate what the delayed observation would be, instead of keeping the agent patient to the end of the delay (Learning and planning in envs with delayed feedback, (2008)[5], At Human Speed: Deep RL with action delay, (2018)][6]).
e.g., For a delayed action:
$$
s_{t+i} = P(s_{t + i -1}, a_{t + i - delay})
$$
If the delay is on the observation, the most recent K actions can be used with the most recent observation to approximate the current state[2]. However, this also means the delays are part of the state - it's possible to have a better approach, as next described in the random delays part.
Estimating the current state enables the agent to act conditioned an estimate of the true state, on which the action is executed. The limitation is that it assumes a constant delay.

All the above approaches assume a constant delay
Handling random delays*
4. Partial Trajectory resampling (PTR)
A method that recursively resamples actions in the buffer, replacing them with on-policy actions. It uses the delay dynamics to simulate their effect on the current policy.
If random delays exist on an MDP, some actions present in the off-policy replay buffer will not influence the later delayed observations and rewards, for a number of timesteps. This will allow generation of on-policy sub-trajectories from the off-policy samples by recursively resampling the most recent actions from the action buffer. It will not invalidate the sub-trajectory.
The resampled action will be valid as long as the observation delay $\omega$ and the action delay $\alpha$ are both greater than the current time step $t$ i.e., No delayed observation depends on a resampled action.
$$
\omega_t + \alpha_t >= t
$$

Illustration of PTR, (3)
The benefits of this method:

Allows discarding of outdated information. e.g If the delay is such that the are no observations for $5$ timesteps, which then all arrive at step $5$, we can safely discard the transitions of steps $(1,2, 3,4)$. This is because this info will be compressed in the most recent observation.

Provides information on the "age" of an observation and the actions applied next

Efficient credit assignment

These factors give partial trajectory resampling better performance compared to concatenating the past $K$ actions with the most recent observation to estimate the current delayed state.
RL with random delays (2020)[3] describes and applies Partial Trajectory Resampling in RL. It's the only work I've managed to come across that handles both random and constant delayed MDPs.
What of the reward delay?
Reward delay, in this context, is referring to the rewards attributed to a delayed observation and action.
A solution to a delayed MDP will involve tackling the credit assignment problem. In this case, this is deciding what to do with the delayed rewards and how to give credit to the delayed actions and states.
This has had different approaches. For instance:

" ...we chose to accumulate the rewards corresponding to the [excessively delayed transitions we drop]. When an observation gets repeated
because no new observation is available, the corresponding reward is 0, and when a new observation arrives, the corresponding reward contains the sum of intermediate rewards in lost [dropped] transitions." [3]

Real time sample efficint RL for robots, (2013)[9] on the other hand, uses a decision tree to assign credit to an MDP reconstructed from a CMDP. The tree learns which delayed actions are relevant.

Related reads:

Real-time Reinforcement Learning (2019)
Thinking while moving: Deep reinforcement learning with concurrent control, (2020)


How is direction of weight change determined by Gradient Descent algorithm

First thing is that what does gradient descent do? Gradient Descent is a tool of calculus which we use to determine the parameters (here weights) used in a Machine Learning algorithm or a Neural Network, by running the Gradient Descent algorithm iteratively.

What does the vector obtained from one iteration of gradient descent tell us? It tells us the direction of weight change (when weights are treated as a vector) for the maximum reduction in the value outputted by a loss function. The intuition behind why Gradient Descent gives such a direction can be found here:

Why the gradient is the direction of steepest ascent - Khan Academy
Why is gradient the direction of steepest ascent?
Why does gradient descent work?

In general cases the cost/loss function is an n-dimensional paraboloid (we design function in such a way such that its convex).

A 2-D paraboloid with x and y as independent variables.

Now why do we reduce the weights in the direction of negative gradient only? Why not some other direction?

Since we want to reduce the cost in Linear Regression for better predictions we choose only the negative Gradient Descent direction, as it is the direction of steepest descent even though we could have chosen some other possible vector direction to reduce cost, but the negative gradient descent direction ensures that:

Cost is always decreased (if the step size is correct and in general only in cases of convex functions e.g paraboloid).
Cost is decreased by the maximal amount if we move in that direction, so we don't have to worry whether cost will decrease or not if we move in that direction.

Also we use learning rate alpha to scale by how much amount we want to decrease the weights.

EDIT: As pointed by @pasaba the error function may not be a paraboloid but in general a good cost function looks like a paraboloid with skewed axis.


Why should each filter have different weights for each input channel?

For simplicitly, let's consider only the first convolutional layer, that is, the one applied to the image. If you consider an RGB image, then there are $3$ channels: the red channel, the green channel and the blue channel. Thus, a kernel that is applied to this image will also have $3$ channels: the red channel, the green channel and the blue channel. In general, the distributions of the intensity of the red, green and blue colors in the image are different, so, in general, the red, green and blue channels of the kernel will also be different because they need to keep track of different information.


Reinforcement Learning (and specifically REINFORCE algorithm) for one-round "games"

This seems like a multi-armed bandit problem (no states involved here). I had the same problem some times ago and I was advised to sample the output distribution M times, calculate the rewards and then feed them to the agent, this was also explained in this paper Algorithm 1 page 3 (but different problem & different context). I honestly don't know if this will work for your case. You could also take a look at this example.


Unsupervised alien natural language learning

I think I understand where your confusion comes from, and the reason for your question.

When we talk about a supervised problem, it means that there is some feedback of how well the machine did at a certain task. Of course, we always need this kind of feedback in some way or another (otherwise, the machine would never learn, because it would have no incentive to change anything that it does), but the feedback can be classified into "supervised" if we have a clear way of telling when something was right/wrong, or "unsupervised" if we just roll out with the results and try to make it better each time.

For example, an AI performing translation like in grammar induction in the other question will take as input both the phrase and the syntactic tree. After performing the prediction of what the syntactic tree for the phrase should be, it can be compared against the real syntactic tree given as input and based on the accuracy of the prediction, the weights (think of it as "knobs" that adjust the result) can be tweaked a little bit to give better predictions next time.

This type of learning is considered supervised, not because of the existence of a supervisor, but rather because there is labeled data that we can use to test predictions and better them next time.

Unsupervised problems don't have this kind of labeled data, and just work with what they have. They cannot tell what's right and what's wrong in terms of predictions, they only have raw data and try to make sense out of it by extracting correlations or common properties.

For the case of a language, I will go on a limb here but I'll say that this is mostly likely a task for supervised algorithms. Unsupervised techniques could properly analyze language and determine that they actually have a particular structure, but this is also true for lots of other sources of data that are not language. An alien race, knowing nothing about humanity, could end up deducting that cars honking are some sort of language too, albeit much simpler than human-produced noises.

Also, the natural flexibility that languages have make it extra complex, because they don't follow a correct structure completely, and to make it worse, they are constantly changing.

As leitasat mentioned in their comment, if even human scientists (which we certainly deem intelligent) could not decipher ancient Egyptian on their own without any context, it's very unlikely that a machine would do it.

Finally, notice that translator machines or inference machines don't actually derive meaning in a way that's meaningful. Without going much in much detail, you should think of them as "correlation detection machines" -- so a machine might notice a high enough correlation between how "Hola" is used in Spanish as to how "Hi" is used in English, and by giving you that information, we call it a translator. However, the internal learnt structure is actually just a probability distribution for an output given a set of inputs.

Not to say that any of this is not useful -- it definitely is -- but with something as separate as an alien race of super-human intelligence and without any way to derive meaning from just a bunch of sounds, it's very unlikely that a machine would prove useful in such correlations.

And still, I cannot find any hard stop to your idea -- if these aliens have a language and they express the same concepts that human express, then there is correlation. There just needs to be a way to find it.

Sorry for the wall of text and the huge rambling, I hope I provided some context on the points you were looking clarification for.


Which predictive algorithm is most appropriate for a proceeding situation?

One approach would be to use a Sequence Processing Neural architecture - one option is a recurrent network. These were specifically designed with your intent in mind.

They can consume sequences of data and learn from these in order to predict subsequent time-steps. Though these will require you to understand best practices for implementation.

If your task isn't too complex you could make use of a forward algorithm or some variant. Many of these models make use of simplifying assumptions - if your use case cannot admit these you'll have to go for something more advanced.

If your prediction task is complex and requires heavy use of long term dependencies you may have to go for a more cutting edge architecture like a transformer - but I heard that these are difficult to implement and most definitely require familiarity with deep learning systems.

Finally, if you need a system that can understand data and dependencies across very large spans of time then you may have to wait for the science to progress.

I hope this helps.


How to set the target for the actor in A2C?

In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick.

A2C Loss Function

A very crucial part of A2C implementation that I missed is the custom loss function that takes into account the advantage. The loss function multiplies the advantage with the negative log of the current probability to select the action that was selected.

The trick is that if the advantage is negative, the loss function will switch sign, so the gradients will be applied to the opposite direction.

In one dimension it's easier to understand. Let's say my target prediction is 1 and my actual prediction is 0.6. A simple loss would be defined as target - prediction, or in this case 0.4 and future predictions will be closer to one. If my prediction was 1.4, then the loss would be -0.4. A negative loss would mean predicting a lower result in the future, and a positive result would mean predicting a higher result in the future.

If the sign of the loss function is switched, the prediction will actually move away from 1.

The same thing happens when you multiply the advantage in the loss function. A negative advantage would mean that this action is worse than the value of the state so we need to avoid it, and a positive advantage means that the action is encouraged.

In Keras (Tensorflow 2.0):

Here's the custom loss function:

def custom_actor_loss(y_true, y_prediction, advantage):
    prediction = K.clip(y_prediction, 1e-8, 1 - 1e-8)
    log_probabilities = y_true * K.log(prediction)

    return K.sum(-log_probabilities*advantage)

The values are clipped because log of 0 is undefined.

And the rest of the network building:

input_layer = Input(shape=self._state_size, name='state_in')
advantage = Input(shape=[1], name='advantage')
target_prediction = Input(shape=self._actions_num, name='target')

inner_layer = Dense(units=layer_size, activation='relu')(input_layer)
actor_out = Dense(units=self._actions_num, activation='softmax', name='actor_out')(inner_layer)

self._actor = Model([input_layer, target_prediction, advantage], actor_out, name='actor')
self._actor.add_loss(custom_actor_loss(actor_out, target_prediction, advantage))
self._actor.compile(optimizer=Adam(learning_rate=actor_learning_rate))

And in the training loop (where future_rewards_prediction and critic_prediction are the outputs from the critic for the current and next state (except for the terminal state where the future_rewards_prediction is set to 0):

# Train actor
target_probabilities = np.zeros([1, self._actions_num])
target_probabilities[0][memory[step_idx].action] = 1.0
advantage = memory[step_idx].reward + future_rewards_prediction * self._future_discount - critic_prediction
self._actor.fit((memory[step_idx].state, target_probabilities, advantage), verbose=0)

*Notice how I don't really specify a y in my fit call. This is because of an issue I had when trying to implement a custom loss function in Keras which was solved by this answer.


What is the simplest classification problem which cannot be solved by a perceptron?

Anything that is not linearly separable cant be solved perceptrons, unless you use feature maps on data to map them to a higher dimension in which it is linearly separable.

As a simple, concrete example, perceptron cant learn the XOR function.

This page might help you further.


Is there a technical name for image classification on amount instead of class?

Training a system to predict real numbers instead of classes is called regression. If you had a training data where the inputs were images of water glasses and the targets were the amount of water in each glass, then this would be regression task from images of water glasses to the amount of water in each glass. You could call it image regression in the same way you would say image classification, but beyond that, I don't think there's a special term for regression where the inputs are images.


Why is update rule of the value function different in policy evaluation and policy iteration?

Yes, the two update equations are equivalent. As an aside, technically the equation you give is not the Bellman equation, but the update step re-written as an equation - in the Bellman equation instead of $v_{k+1}(s)$ or $v_{k}(s)$ (showing iterations of approximate value functions), you would have $v_{\pi}(s)$ (representing the true value of a state under policy $\pi$).
The difference between the equations is that

In the first case of Policy Evaluation, in order to be general, a stochastic policy $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = Pr\{A_t = a|S_t =s\}$ is used. That means to get the expected value, you must sum over all possible actions $a$ and weight them by the policy function output.

In the case of Policy Iteration, a deterministic policy $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ is used. For that, you don't need to know all possible values of $a$ for probabilities, but use the output of the policy function directly as the action that is taken by the agent. That action therefore has a probability of $1$ of being chosen by the policy in the given state.

The equation used in Policy Iteration is simplified for a deterministic policy. If you want you could represent the policy using $\pi(a|s)$ and use the same equation as for Policy Evaluation. If you do that, you would also need to alter the Policy Improvement policy update step to something like:

$a_{max} \leftarrow \text{argmax}_a\sum_{r,s'}p(r,s'|s,a)[r + \gamma V(s')]$
$\text{ for each } a \in \mathcal{A(s)}$:
$\qquad \pi(a|s) \leftarrow 1 \text{ if } a = a_{max}, 0 \text{ otherwise }$

Doing this will result in exactly the same value function and policy as before. The only reason to do this would be to show the equivalence between the two sets of update equations when dealing with a deterministic policy.


Can the rewards be matrices when using DQN?

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.
Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for expected return which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.
This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the L2 norm or other standard measure that summarises the matrix will be good for your task.
It is possible to process multiple scalar rewards at once with single learner, using multi-objective reinforcement learning. Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a "policy of policies" if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.


Is there any network/paper used to analyse music scores?

I have not come across music labeling algorithms but upon a google scholar search, I found a couple of papers that aim to do quite the same task.
In general, if you have a labeled dataset then you can take an approach of a general speech recognition model. It should work fine for music labeling too, but you might need to tweak certain parameters.


How artificial intelligence will change the future?

Here is something I've noticed about humans: We're bad at projecting the future with all of its 2nd, 3rd ... N order effects, and we're REALLY bad a projecting and quantifying risk. So, I'm not sure that you'll get an answer that is anything more than either trivially true ("Chatbots will be commonplace") or a correct but wasn't justified ("We'll explore the stars by sending out swarms of AI-enhanced satellites that will use their computing power to improve themselves since they are always on but have copious amounts of down-time").
But if you want to limit the timeframe to the next decade and don't mind the previous caveat, here goes:

Chatbots will be commonplace. This will include ones that can handle conversation, like Gridspace but can also handle multiple languages and code-switching.
In professional use, AI will augment professionals (such as lawyers, physicians, teachers, and accountants) at the upper end (who will add value with strategy, experience, and research) and replace them at the lower end.
In military use, they will be used for target identification, automated assessment of damage, threat notification, and forecasting.
We'll explore the stars by sending out swarms of AI-enhanced satellites that will use their computing power to improve themselves since they are always on but have copious amounts down-time.

I don't hold a lot of hope out for communication without error of to speakers of different languages as you can have two native fluent speakers who have known each other for years speak to each and still have them miscommunicate.