Can we use Q-learning update for policy evaluation (not control)?

For off-policy learning you must have two policies - a behaviour policy and a target policy. If the two policies are the same, then you end up with SARSA, not Q learning.
You cannot use Q learning directly for evaluating a fixed target policy, because it directly learns optimal value function as the target policy, regardless of the behaviour policy. Instead you must use another variant of off-policy learning that can evaluate an arbitrary target policy.
Your suggested algorithm is:

Have the policy to be evaluated as the behaviour policy.
Update the Q value conventionally (i.e. updating $Q(s,a)$ using the action $a'$ giving highest $Q(s',a')$ value)
The final $Q(s,a)$ values will reflect the values for the policy being evaluated.

This will not work for evaluating the behaviour policy. If the behaviour policy was stochastic and covered all possible state/action choices, then it will still be Q learning and converge on the optimal value function - maybe very slowly if the behaviour policy did not get to important states very often.
The "trick" to off-policy is that the environment interaction part uses the behaviour policy to collect data, and the update step uses the target policy to calculate estimated returns. In general for off-policy updates, there can be corrections required to re-weight the estimated returns. However, one nice thing about single-step TD methods is that there are no such additional corrections needed.
So this gives a way to do off-policy TD learning, using an approach called Expected SARSA. To use Expected SARSA, you will need to know the distribution of action choices i.e. know $\pi(a|s)$ for the target policy.
This is the variant of your description that will work to evaluate your target policy $\pi(a|s)$:

Have any stochastic policy that "covers" the target policy as the behaviour policy.
Update the Q value using Expected SARSA $Q(s,a) = Q(s,a) + \alpha(r + \gamma [\sum_{a'} \pi(a'|s')Q(s',a')]  - Q(s,a))$
The final $Q(s,a)$ values will reflect the values for the policy being evaluated.

Worth noting that Expected SARSA with a target policy of $\pi(s) = \text{argmax}_a Q(s,a)$ is exactly Q learning. Expected SARSA is a strict generalisation of Q learning that allows for learning the value function of any target policy. You may not see it used as much as Q learning, because the goal of learning an optimal value function is more common in practice.


Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, thus returns are defined as the total sum of discounted rewards from the current time step onwards. This allows the agent to make decisions about which actions to take based on how valuable taking said action is in the current state at time $t$ -- clearly the rewards previous to this have no effect upon that.


How to make a distinction between item feature and environment feature?

I am curious to know if I can put the economy feature such as 'national interest rate' or 'unemployment rate' besides each stocks' features.

The variables are macro-econometric and they, in general, seem to have some influence on stocks' prices. This inclusion might as well increase your model's prediction accuracy. You can definitely use them as predictors. As mentioned in comments - Experimentation is the way to go.

  Can I feed the data to my neural network like the table above?

In general, you can have any kind of numeric variables as input to a neural network. Things will work out fine. The important thing is the selection of relevant predictor variables that, potentially, have some relationship with the response variable.


How do I poison an SVM with manifold regularization?

I think, one could generate a data set, with a random variable in each component of the data vector, add this data to the training data set, and then shuffle the combined data set.


Clarification on definition of convolution: Is adding the Frobenius inner products between filter and input part of convolution or a separate step?

Both are incorrect.

using your notation
You do not take a sliding frobenius inner product of a singular channel of $I$ with $F$, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has $k_1$ channels and you have $k_2$ filters.

This means $shape(I) = (N, M, k_1)$ and each of the $k_2$ filters is of shape $(n, m,k_2)$ and your output shape is $(N-n+1, M-m+1, k_2)$ assuming your not using padding.

So i guess trying to put it in the manner you use: The convolution's output's $i^{th}$ channel is taking the sliding Frobenius inner product between all $n \times m$ cross-section of $I$ (including all input channels) with filter $F_i$.


Is there a taxonomy of adversarial attacks?

There are already a couple of papers in the literature that attempt to provide a taxonomy and survey of adversarial attacks. I will just list the two that I think are reliable enough that you can probably use as a reference.

A taxonomy and survey of attacks against machine learning (2019) by Nikolaos Pitropakis et al., published in Computer Science Review

Adversarial Examples: Attacks and Defenses for Deep Learning (2019) by Xiaoyong Yuan et al., published in IEEE Transactions on Neural Networks and Learning Systems.

Needless to say, there are different adversarial attacks, such as the Fast Gradient Signed Method (FGSM), and they can be classified into different categories, such as evasion attacks or poising attacks. You can find a lot more info in these cited papers.


Genetic algorithm: How to crossover 2D permutation?

First, store the two parent chromosomes into a sorted dictionary (in terms of implementation, std::map in C++ might be a good option) where the key is the letter and the value is a pair of ints (the second gene element). When you populate the map, add all the letters from each chromosome as keys and take 0 (or a negative number) to indicate that a gene with the corresponding letter does not exist in one of the parents.

Next, create a second map, this time with the number as a key. Iterate over the first map. If both parent chromosomes contain a gene with the same letter, the value field in the map would contain two positive numbers - pick one at random and insert the {number, letter} combination into the second map only if the number doesn't exist as a key. If the gene exists only in one parent, either pick the gene as is or choose to skip that gene altogether.

This would guarantee that 1) your genomes contain a {letter, number} gene only once and 2) the genes in the offspring's chromosomes would be automatically sorted by letter. If you don't need the genes to be sorted but still need them to be unique, you can use an unordered map (a hash table).

In your example, the map would look like this:

Mapped parent chromosomes
{a: (1,2)}
{b: (0,1)}
{c: (2,4)}
{d: (0,3)}
{e: (3,0)}
{g: (4,0)}

Offspring chromosome (OC)
{}

Where 0 indicates that the corresponding gene does not exist in that parent. Iterating over this table from top to bottom, you might end up doing the following:

Key a: pick 1; 1 is not a key in OC, so insert {1,a}
Key b: pick 1; 1 is a key in OC, so skip letter b
Key c: pick 4; 4 is not a key in OC, so insert {4,c}
Key d: choose to skip
Key e: pick 3; 3 is not a key in OC, so insert {3,e}
Key g: pick 4; 4 is a key in OC, so skip letter g

Final offspring chromosome:

{a,1}{c,4}{e,3}


Are neurons instantly feed forward when input arrives?

It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in turn introduce new computations to the neural network.
This is your network graph if I got it right:

The calculation in a neural network without refractory periods and resting potentials without time, would instantaneously modify the weight of your node4 layer 5 (n4-L5) if there is input in column 3:

Additional inputs on other columns would just add up unless you have some other explicit computation on any layer.
If you wanted to emulate a Neuron, each node would need to have a resting potential, that is: a level above which it will fire ( in the above example zero), and a refractory period:  a time before it would fire again, as well as a clock to keep it in sync, this would be a crude realtime fascimile:
http://codepen.io/k3no/pen/WRQbYV

A common alternative is to use sequential phases or steps.
Reference/source
Artificial Intelligence: A Modern Approach, by S. Russell and P. Norvig. Deals with a general step approach to A.I.
Gateway to Memory, by Mark A. Gluck and Catherine E. Myers Presents a great and readable introduction to modeling neural networks.
I published the little neural network model in a medium article : Memory and the machine, relevant sources are there.


I don't actually understand your question, but if your data is completely arbitrary, there is nothing to predict, it have no patterns to recognize or something like that.
But if you say that you are working with time-series data and it have some patterns, then you could start by trying to implement just the forward propagation of a simple RNN. I am really new as well at AI and the first RNN I code was a Elman RNN, and they were very simple equations to implement. I recommend you trying that, and then implement the backpropagation to that RNN.
Something that really helped me starting was searching simple github scrips for RNNs (around 100 lines of code) so that you can see their architecture.
About the example you give, the one about songs, they are actually really predictable, because you have the rime, rithm and tempo, and also they use to repeat the same secuence of notes every stanza.
Hope it helps :)


How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).

However, the reliance on future states is only there during training, it is no longer required for deployment.

During training, we use the following $Q$-learning update rule:

$$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max_{a'} Q(s', a')} \right),$$

where $s'$ is the state we reach after executing $a$ in $s$. Here, the $\color{red}{\text{red}}$ part is the part where we rely on knowledge of the future $s'$. This is available in training because we can simply pick $a$, execute it in $s$, observe $s'$, and only then trigger our update step.

Outside of training (and actually also during training), we also rely on our $Q(s, \cdot)$ function for the selection of actions. We typically select an action $a$ according to $a = \arg\max_a Q(s, a)$; we select the action $a$ that maximises $Q(s, a)$ in our current state $s$. The important thing to note here is that there is no $s'$ term in this description of how we select actions: we do not require knowledge of our future state.

Note: in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how $Q$-learning can still work in the described setting where actions $a$ have no influence whatsoever on the future state reached.

In practice, I would never recommend actually using $Q$-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.


How can I implement an AI system that learns to recognise a context-free grammar?

The process of automatically learning a grammar from examples of a language is called grammar induction. Since you mention that L can be "augmented and changed", it might be feasible to solve this problem using an adaptive parser.


Is the PyTorch official tutorial really about Q-learning?

TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.

Q learning* and value iteration are very strongly related. When considering action values, both approaches use the same Bellman equation for optimal policy, $q^*(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma \text{max}_{a'} q^*(s', a'))$ as the basis for update steps. The differences are:

Value iteration makes updates using a model of the environment, Q learning works from samples from the environment made by an active agent.

By working from a simulated environment rather than a real one, it may not be clear when an agent is model-free or model-based (or planning rather than acting). However, the way that the simulated environment in the PyTorch example is used is consistent with a model-free method.

Value iteration loops through all possible states and actions for updates independently of any action an agent might take (in fact the agent need not exist). Q learning works with whichever states the agent experienced.

By adding experience replay memory in DQN, Q learning becomes a little bit closer to value iteration, as you can frame the memory as a learned model, plus consider it to be a type of planning (or a "sweep" through states). This is how it is described for instance in DynaQ which is an almost identical algorithm to experience replay as used in DQN when both are used in the simplest versions - see Sutton & Barto chapter 8.

Value iteration value update steps are over an expectation of next states and rewards - it processes the weighted sum $\sum_{r,s'}p(r,s'|s,a)$. Q learning update steps are over sampled next states and rewards - it ends up approximating the same expectation over many separate updates.

Even using large amounts of experience replay memory does not get Q learning the same as value iteration on this issue, samples are not guaranteed perfect. However, in a deterministic environment, this difference is not meaningful. So if you have a deterministic environment, Q learning and value iteration may also be considered a little closer in nature.

* Technically this applies to single-step Q-learning. n-step Q-learning and Q($\lambda$) use different estimates of future expected return, that are related but not the same as the single-step version shown here.


How to deal with very, very small time-series?

Of course depends on your type of data, but Holt-Winter models can have different degree of complexity and use moving average, trend, and seasonality. This is most useful if the data is not hierarchical, meaning that the time-series are independent from each other.
If time-series are relatives of each other then you can also try aggregating them, predict at aggregate level and then disaggregate. The following can be a good resource:
https://otexts.com/fpp2/


How to identify too small network in reinforcement learning?

Check the function loss.
It might be that your environment is impossible to learn. However, most likely the network simply can't handle it. By measuring the loss during the learning stage, if you find it is always very high and does not decrease, it's a strong indication this might be the issue.
Because the network is too simple, when you optimize for some states, you ruin others. There is not formal way to find out if this is the case, but since the same algorithm works elsewhere, it's either a problem of your environment, or of the network.


When is it time to switch to deep neural networks from simple networks in text classification problems?

Source: https://blog.easysol.net/building-ai-applications/

When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can eliminate bias and don't lead to high variance.

Deep learning models can be tweaked(hyperparameters) and regularized(parameters), and it is worth the work.


What happens to the training data after your machine learning model has been trained?

In many cases, a production-ready model has everything it needs to make predictions without retaining training data. For example: a linear model might only need the coefficients, a decision tree just needs rules/splits, and a neural network needs architecture and weights. The training data isn't required as all the information needed to make a prediction is incorporated into the model.

However, some algorithms retain some or all of the training data. A support vector machine stores the points ('support vectors') closest to the separating hyperplane, so that portion of the training data will be stored with the model. Further, k-nearest neighbours must evaluate all points in the dataset every time a prediction is made, and as a result the model incorporates the entire training set.

Having said that, where possible the training data would be retained. If additional data is received, a new model can be trained on the enlarged dataset. If it is decided a different approach is required, or if there are concerns about concept drift, then it's good to have the original data still on hand. In many cases, the training data might comprise personal data or make a company's competitive advantage, so the model and the data should stay separate.

If you'd like to see how this can work, this Keras blog post has some information (note: no training data required to make predictions once a model is re-instantiated).


How to handle rectangular images in convolutional neural networks?

I think the squared image is more a choice for simplicity.

There are two types of convolutional neural networks

Traditional CNNs: CNNs that have fully connected layers at the end, and
fully convolutional networks (FCNs): they are only made of convolutional layers (and subsampling and upsampling layers), so they do not contain fully connected layers

With traditional CNNs, the inputs always need to have the same shape, because you flatten the last convolutional layer, with a fixed size. As the flatten layer has a fixed size, the feature map shape from the layer before has to be the same shape, and so the inputs (images) also have to.

However, in FCN, you don't flatten the last convolutional layer, so you don't need a fixed feature map shape, and so you don't need an input with a fixed size.

In both cases, you don't need a squared image. You just have to be careful in the case you use CNN with a fully connected layer, to have the right shape for the flatten layer.

For instance, if you have an input size $320 \times 160$, and you have 3 pooling layers, so your output in the last convolutional layer is $40 \times 20 \times c$ (with $c$ the number of filters/channels), then you just need the flatten layer to have $40*20*c$ neurons.

If you create a new network, just design it to handle a rectangle image.

If you want to use an already pre-trained one, I think the better choice is to resize the image.

If the information in the cropped parts is important, maybe your prediction can be wrong (it depends if the object of interest is in the parts of the image that is cropped). Actually, in Yolo (an object recognition network), images are resized if they don't fit the input requirements. See figure 1 of the YOLO paper. It's because you don't need a high resolution to detect an object (for example, the CIFAR dataset has images of shape $32 \times 32$, but the network can still predict the correct label). So, I think that resizing your image may not affect the prediction much (unless the new size is very different from the original)


If $h_1(n)$ is admissible, why does A* tree search with $h_2(n) = 3h_1(n)$ return a path that is at most thrice as long as the optimal path?

The sketch of the proof for your first question:
for an open node $n$, if $f_1(n) = g(n) + h_1(n)$, in the same situation in using $h_2$, it will be $f_2(n) = g(n) + 3  h_1(n)$. Hence, all the time for any node $n$, $f_2(n) \leqslant 3f_1(n)$. On the other hand, we know that A* with the admissible husritic function $h_1$ will be admissible (from Theorem 2 of Chapter 3 in "Heuristics Intelligent Search Strategies for Computer Problem Solving" book by Judera Pearl), i.e., for the node $n^*$ with optimal value, $f_1(n^*) \leqslant C^*$ that $C^*$ is the optimal value. Therefore, A* with $h_2$ will return a solution in node $n'$ by the cost of $f_2(n') \leqslant 3 f_1(n^*) \leqslant 3C^*$, as $f_1(n') \leqslant 3 f_1(n^*)$ (see more details of the proof in Theorem 13 of Chapter 13 in the same reference).
You can find more about $h_2$ under the title of $\epsilon$-admissibility as it is $(1 + \epsilon) h_1$ that $h_1$ is an admissible heuristic function. In your case, $\epsilon = 2$.


At this time, Google is aggressively pursuing research into AI of many sorts. All AI at this point is constrained by the size and accuracy of available training data. Getting human curated data is expensive and time consuming. Crowdsourcing through Captcha gets access to that without paying directly.

Why roads specifically? It can hardly be a coincidence that Google's holding company also owns a self driving car company Waymo. They need machines to learn what road signs look like in real life, so that they can respond to them.


Illustration of Von Neumann's Minimax theorem in games?

Here is one of many web pages devoted to the application of Minimax in Tic-Tac-Toe solving: Minimax Algorithm in Game Theory | Set 3 (Tic-Tac-Toe AI – Finding optimal move)

Google search will yield many such results. (I will return to this answer when I have more time, and post more links with more context.)

Here's a high-level explanation on how minimax may be used in the non-chance, perfect information, sequential, partisan, zero-sum set of games [M]:

A non minimax evaluation function would look to place integers on the gameboard in positions that grant maximum value (power & influence)
A minimax evaluation function would look to place integers on the gameboard in positions that maximize value while minimizing the max value of the opponents next placement

This is partly due to the utilization of "resource stealing strategies" in [M] games.

Note: I need to do some more thinking on the sub questions regarding application beyond zero-sum, perfect-information games, but I don't see any issue with it.   Minimax/maximin has wide and fundamental utility in an array of contexts, not strictly related to games except in the sense that every problem can be understood as a type of game or puzzle.


Can training a model on a dataset composed by real images and drawings hurt the training process of a real-world application model?

To my knowledge the deployment model (that you will test on underwater images) as inference will not have a negative effect. Yet drawings may even help differentiate some classes at training and inference. Provided that you won't use drawings in inference, adding them in training phase will not necessarily hurt the accuracy. Note that a drawing of a particular class should not be in the search domains of other classes, namely, a drawing of a particular class should not be the same with the other classes.


What do neural connection weights represent 'conceptually'?

I don't know if my intuition is correct but I will give it a try.

You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing I'm referring to the output of a specific neuron. I don't think that we can say what the output of a neuron represents in the real world unless we directly relate it through an error function or if the function used to compute that particular value have some meaning in the real world.

Edit:

If you want, you could actually build your neural network such that its neurons represent something. It's also very simple. you have only to write down all the equations relative to that particular topic. You could put them in a big system or, and this is better, you could put them in several systems such that the outputs of system 1 are the input of system 2 and so on. You could convert each system into a layer where each neuron represents an equation. Note that in this case, you would have the classical neuron with

z = dot(w.T,x) + b
a = g(x)

but a more complex equation for z (but still based on weights) and a linear activation function for a. In this case, you could name each neuron and say what they represent in the real world.

However, this isn't the purpose of a neural network. A neural network should have neurons with simple equations to be fast thus the linear interpolating function dot(w.T,x) + b is the best choice (the fact that the activation function is almost always non linear and in some cases a non-banal function is due to other thing and could be an interesting question). A neural network should also be as general as possible because usually is build upon a system that you don't know completely.

So I modify slightly my answer: is not simply that you don't know what a neuron represent, excluding the ones of the output layer, you don't want that they have a meaning in the real world.


In effect, the midpoint is contained in cell $2$. Cells $1,3,4$ will be shown at $P_c=0$ according to the YOLO algorithm, which only takes in count the cell that contains the midpoint and calculates the bounding box, as you mentioned, with $b_y b_x, b_y, b_h, b_w$. With the proposition of $Y = [1, 0.9, 0.1, 2, 2]$, I would think that if you take the point $(0,0)$ as the left-upper corner, $Y$ would be more likely to be $Y = [1,0.2,0.57,0.21,0.15]$, being calculated with a grid of $19 \times 19$.


If two objects are too close to each other, would an object detector do a poor job of correctly classifying them?

As you ask, "in general...", I will answer generally, however this changes a lot from model to model and the way they handle close objects.
In general, yes, they would do a poor job detecting very close objects, switch to segmentation models for that (for class or better, instance segmentation).
In general, objects detectors learn to tell an object from other based in 2 criterion:

Intersection over union: for object of the same class
Class probability: for objects of different class

So, if two objects of the same class are very close, the 2 detected bounding boxes will be highly overlapping, then, the Non Maximal Suppression filter will remove one of them. This is where objects detector, in general, perform worse.
Similarly, if two objects belong to different classes the 2 detected bounding boxes will be highly overlapping but the NMS filter won't remove them (again, in general, NMS is set only for same class objects). However when 2 objects are very close, there is a high chance they are partially occluded. Objects detectors, in general, don't handle occlusions very well.
So, in conclusion, objects detectors will perform better detecting far-away objects.


How to choose the activation function in neuroevolution?

When it comes to genetic algorithms (neuroevolution), you can pretty much evolve any kind of parameters. You just need to have a fairly complex system where each parameter changes the way the inputs are affecting the outputs.

So, to answer your question, any activation function should work! It would be surprising that the activation function is your problem. But if you want to test some activation functions, here's a list of activation functions. But tanh should work perfectly fine.

As I already mentioned in another answer, the key to have a great neuroevolution algorithm is the fitness function. Your rewards should reflect the goal you're aiming for. You didn't mentioned what fitness function you used, but here's what I suggest: reward a player ONLY if he progresses in the game. For example, your fitness function could simply be the distance traveled by your birds. This could maybe solve your problem...


What will Q-values look like in self-play tic-tac-toe?

Chapter 1 of Sutton & Barto, doesn't introduce the full version Q learning, and you are probably not expected to explain the full distribution of values at that stage.

Probably what you are expected to notice is that the maximum Q values out of possible next states - after training/convergence - should represent the agent's best choice of move. What the actual optimal values are depends on how the opponent plays. In self-play it is possible to find optimal play for both players in the game, and thus the Q values represent true optimal play. However what "optimal play" means is dependent on the goals you have set the agent implied by the reward values.

Any move which leads to a guarantee that a player can force a win regardless of what the opponent does, would have a Q value of +1. If the agent or opponent can force a draw at best, then it will have the Q value of the draw, and if the opponent can force a win (i.e. the current agent would lose), then the move will have the Q value of a loss. This happens because the learning process copies best case values backwards from the end game states to earlier game states that lead to them.

In a game with two perfect players, and +1, 0, -1 reward system, then each player will on its turn only see the 0 and -1 moves available. That is because there is no way to force a win in tic-tac-toe, and the perfect opponent will always act to block winning moves. The best choice out of 0 or -1 is 0: each player, when acting under its value estimates, will force a draw. There will be states defined that have a value of +1, but they will never appear as a choice to either player.

What happens if you don't make a difference in rewards between drawing and losing? In the extreme case of having win +1, lose or draw 0 against a perfect opponent, then all of the agent's available Q values will always be 0. The agent would then be faced with no way to choose between defensive plays that force a draw and mistake plays that allow the opponent to win. In turn that means some chance that the opponent will win, even when the agent had learned optimal play.

When two agents learn through self-play using the +1, 0, 0 reward scheme it gets more complicated. That is because the opponent's behaviour is part of the Q value system. Some positions will have more opportunities for the opponent to make mistakes, and score more highly. A mistake that allows an opponent to force a win will actually score worse, because the opponent will not make mistakes once it has a sure route to a +1 score. So even though the agent apparently cannot tell the difference between a loss and a draw, it should still at least partially learn to avoid losses. In fact, without running the experiment, I am not sure whether this would be enough to still learn optimal play.

Intuitively, I think it would be possible for the +1, 0, 0 agent to still learn optimal play, although maybe more slowly than the +1, 0, -1 system, because any situation that gave an opponent a chance of winning would allow it to pick the move with best score, reducing the first agents score for a move that arrived there - and this difference will be backed up to earlier positions. However, the learning would become unstable as described above, as against a perfect opponent the difference disappears as all the best options are draws or losses, and the agent will start to make mistakes again.


Why is automated theorem proving so hard?

There are two ways to look at the problem, one in terms of logic and the other in terms of psychology.
To get any -start- on automation of mathematics, you need to formalize the part you want. It has only been since the early part of the 20th c that most day to day math had been formalized with logic and set theory. And even though Gödel's incompleteness theorems say (very loosely) that there is no algorithm to decide theorem-hood for mathematical statements (that include a theory of arithmetic), that still leaves a lot of math that -can- be decided. But that has taken the Reverse Mathematics program (still ongoing) to say specifically what subsets of math are decidable or to what degree (what logical assumptions are necessary) they are undecidable.
So theorems in arithmetic of just '+' (that is, dropping '*') can be decided, Euclidean geometry can be decided, single variable differential calculus can be decided but not single variable integral calculus. These examples show that what we know to be decidable is pretty elementary. And most of the things we care about are very un-elementary (almost by definition).
As to psychology, the theorems and proofs that you learn in mathematics classes are nowhere near like their formalizations. Most mathematicians aren't pushing symbols around in their heads like a computer does. A mathematician is more like an artist, visualizing dreams and connecting metaphors just on their barely conscious images borne out of repetition. That is, machines and mathematicians just work on different representations (despite what non-mathematicians might imagine).

To address your specific question, yes, mathematical theorems and the systems to prove them are very similar in a technical sense. Games (often, not always) can be modeled as trees, And likewise proofs can often be modeled as trees. Without writing you a library of books about games and proofs, let's just say that the mathematical proofs that are like games that are won by Alpha Zero are not for particularly interesting theorems. Winning a game of go is more like proving that a very very large boolean formula. Most mathematical theorems require a lot of ingenuity in introducing steps in their proof trees. It may be mechanical after the fact to check that a proof is correct, but discovering the proof almost needs magic to come up with a step in the game. Sure, some things in math are automatable (as mentioned before, derivatives), but some mathematical systems (such as integration) are provably impossible to find proofs of all true statements.
Another difference between theorem proving and games is that proofs have to be air tight on all paths, whereas with games one side just has to eke out a single win over the other side.

A separate issue entirely that may contribute to the difficulty is that we just may not yet have the tooling available, ie editors, notation, proof assistants that make it easy to do what should be easy. Or it could just be that mathematicians don't have the fluency with theorem proving systems.
Or it could be that if there were automated theorem provers good enough, mathematicians just wouldn't care too much for them because they'd take away the fun of finding the proofs themselves.


Any comparison between transformer and RNN+Attention on the same dataset?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.
Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU value of the transformer superior to the ByteNet (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:

Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).


What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets compounded (there is a nice example in the Sutton and Barto book that illustrates this). The idea behind tabular double $Q$-learning is to have two $Q$-networks, $Q_1,Q_2$, and you choose an action $a$ from them, e.g. from $Q_1 + Q_2$. You then flip a coin to decide which to update. If you choose to update $Q_1$ then the update target becomes $r + \gamma Q_2(s', \arg\max_a Q_1(s',a))$.
The idea is that if you overshoot your estimate on one $Q$ network then having the second will hopefully control this bias when you would take the max.
In Deep Double $Q$-learning the idea is essentially the same but instead of having to maintain and train two $Q$-networks, they use the target network from vanilla DQN to provide the target. To make this more concrete, the update target they use is
$$r + \gamma Q(s', \arg\max_aQ(s',a;\theta);\theta^-)\;,$$
where $Q(s,a;\theta^-)$ denotes the target network whose parameters are only updated to the current networks every $C$ time steps.
As before, the idea is that if we have overestimated our value of being state $s'$ in our current network when taking the max action, using the target network to provide the target will help control for this bias.
Maximisation Bias
I will here explain maximisation bias from the simple example given from the Sutton and Barto book.

The Markov Decision Process in the image is defined as follows: we start in state A and can take the 'right' action which gives us 0 reward and immediately leads to termination. If we choose 'left' we get 0 immediate reward where we then move to state B. From there, we have an arbitrary number of action we can take where they all lead to the terminal state and the reward is drawn from a Normal(-0.1,1) distribution.
Clearly, the optimal action is always to move to the right from state A as this gives 0 expected future returns. Taking the left action will give a $\gamma \times -0.1$ expected future returns (the $\gamma$ is our discount factor).
Now, if we got into state $B$ and took some random action our initial reward could be bigger than 0 -- after all it is drawn from a Normal(-0.1,1) distribution.
Now, consider we are updating our $Q$-function for state A and taking the left action. Our update target will be $0 + \gamma \max_a Q(B,a)$. Because we are taking the max over all possible actions, this will lead to a positive reward and so we are backing up the belief of our expected future rewards from taking action left in state A to be something positive -- clearly this is wrong since we know it should be -0.1. This is what is known as the maximisation bias, because it gives us a kind of 'optimistic' estimate of the action value!
I've attached an image below that shows the %age of time the agent chose the left action, which it shouldn't be choosing). As you can see, it takes normal $Q$-learning along time to even start to correct itself, whereas double $Q$-learning corrects the mistake almost immediately.


What is the difference between Stochastic Hill Climbing and Simulated Annealing?

Russell and Norvig's book (3rd edition) describe these two algorithms (section 4.1.1., p. 122) and this book is the reference that you should generally use when studying search algorithms in artificial intelligence. I am familiar with simulated annealing (SA), given that I implemented it in the past to solve a combinatorial problem, but I am not very familiar with stochastic hill climbing (SHC), so let me quote the parts of the book that describe SHC.

Stochastic hill climbing chooses at random from among the uphill moves; the probability of selection can vary with the steepness of the uphill move. This usually converges more slowly than steepest ascent, but in some state landscapes, it finds better solutions.

So, SHC chooses at random one "uphill move", i.e. a move that improves the objective function (for example, if you're trying to solve the travelling salesman problem, a "uphill move" could be any change to the current Hamiltonian cycle, a solution, so that the new Halmitonian cycle has a shorter cost) among the uphill moves (so among some set of moves that improve the objective).
In simulated annealing, you perform some move. If that move leads to a better solution, you always keep the better solution. If it leads to a worse solution, you accept that worse solution with a certain probability. There are other details, such as how you accept the worse solution (which you can find in Russell and Norvig's book), but this should already clarify that SA is different from SHC: SA can accept worse solutions in order to escape from local minima, while SHC accepts only uphill moves.


Can hidden Markov models be used to model any time series data?

Yes, you can fit any time series (with or without external variables) using HMM, but there are some constraints:

It should follow the Markov property.
There is some variance that other models are not able to capture (in other words, the system is partially observable).

Adding to point 1, for HMM, it should hold true, but the way Baum Welch algorithm works, indirectly it considers the values of more than the previous state for HMM (order-1). The state $t-1$  depends on $t-2$, which in turn depends on $t-3$. The calculation of parameters (transition, emission, starting probabilities) happens over multiple iterations and it finds parameters in such a way that holds Markov property true.

I think that, when they say 'any', they mean even when you don't have all variables needed to forecast future values.


Is REINFORCE the same as 'vanilla policy gradient'?

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization(or whatever is trying to be communicated).

For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla_\theta J(\theta)  \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla_\theta J(\theta)  $ with $\alpha$ being our step size.

This will, of course, vary as notational convention and term definition can change between practitioners.


How does a model based agent learn the model?

If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to 'cover' enough of the state/action space so that your model remains accurate. For instance, when you start training your agent it will likely start to see more of the state space than it did at the start of training (imagine playing Atari, initially your agent will die quickly but as it gets better episodes will get longer) so you would need to make sure you have data for these states that appear late in episodes, otherwise your model will just be overfitting to the start of the episode and will give a poor performance on these other states, thus slowing down or even prohibiting learning of an optimal policy.


Does data skew matter in classification problem?

Yes. Skewed data is one of the biggest problems in AI applications. As you rightly identified, the real world distribution is skewed. Doing a random sampling results in one major issue of an uneven sampling (like in your case). Even worse could happen, all of the samples may fall into a single class and other classes may not even be recognized by your classifier. This is called as the class imbalance problem.

There are many ways to mitigate this problem. Few of them have been mentioned here. I'll summarize them for you:

Sampling based mitigation (when you try to deal with skewed data)
Cost function based mitigation (when you try to improve the classifier, in case skewed data cannot be avoided)

Sampling based mitigation can be done by oversampling from the minority class(es), or undersampling from the majority class(es) or using a combination of both.

Another fairer method to do this is by doing a stratified sampling. If class A has 1/6th, class B has 2/6th and class C has 3/6th of the total population, then, you should take 1/6th of the samples by random sampling from objects in class A, 2/6th from class B and 3/6th from class C. This way all of the sample may represent the population in the right proportion, and none of the classes will be missed.

In case this doesn't help (say, you dataset is small and sampling doesn't make sense) and the data is skewed too much), the cost function can be modified to make a more sensible classifier. Keep different costs for misclassification and/or correct classification. Misclassification could be given a higher penalty. This could help in reaching to a better classifier with a skewed data.


How widely accepted is the definition of intelligence by Marcus Hutter & Shane Legg?

My sense is that everyone is pretending Intelligence doesn't have a grounded definition, from which all other definitions arise:

Intelligence is a measure of utility in an action space μ(υ)

It can be a relative measure, in relation to other rational agents, or absolute in relation to solved games (problems).  An action space is any context, and formalized as problems or sets of problems, typically grouped by complexity class.
Hutter and Legg is an explication of this grounded definition which accounts for unlimited contexts (complexity classes/environments) and for increasing utility of a given agent over time (learning/optimization.)  Intelligence itself does not require learning or general applicability, but Hutter & Legg does not refute this, merely grades static intelligence and narrow intelligence as more limited.
Even this is subject to context, as more limited rationality can be more optimal.

The definition of intelligence is grounded because, while the term "intelligence" is a symbol, intelligence itself is function, the strength of which evaluated by a measurement of some result (utility)

It doesn't require defining the function to understand it as a function:  measurement of a result requires mechanism and decision/action.
You will find this natural language definition applies to even emotional intelligence, which relates to the observational capability of the rational agent in context, and allow that rational agent to make more optimal decisions in context.
This is similar in spirit to truth, only grounded in a formal logic context, where it is a condition and result, not an assertion.  By contrast, the the conditionality of truth is often obscured in a natural language context, where is is routinely applied to unvalidatable informal statements, and can even be conflated with the statement itself à la: "this is the truth!"


What does it mean if classification error is equal between two networks but the MSE is different?

MSE just measures the squared difference between actual and target values. It can still correctly classify the values, but perhaps not with the same confidence - leading to a higher loss (e.g. an output of 0.77 vs 0.98 when the target is 1). In terms of which is better, I wouldn't know without the specifics of your problem. It is possible the higher loss could be more robust since it is less likely to have overfitted on the data, yet achieves the same accuracy.


You can compute "Silhouette Coefficient" for your aim. Its values mean:

1: Means clusters are well apart from each other and clearly distinguished.
0: Means clusters are indifferent, or we can say that the distance between clusters is not significant.
-1: Means clusters are assigned in the wrong way.

Also other measures such as purity and mutual information can be possible by ``computing an external criterion that evaluates how well the clustering matches the gold standard classes.''.


How are the inputs passed to the neural network during training for the XOR classification task?

Hello and welcome to the community. There are multiple ways you can train a neural network: stochastic, mini-batch and batch. What you explained is the stochastic mode, where you input one training example 01 for example, calculate the gradients and update the networks weights before the next training example is fed. You could also select multiple such examples (a mini-batch) and update the weights only after you computed all the outputs (for this particular mini-batch). Finally you can use a batch size which is equal with the total number of examples in your dataset so you will update the weights only after you have all the outputs for all samples. Each of those methods have their own strengths and weaknesses, depending on your dataset you might prefer one over the others.


How can I implement the reward function for an 8-DOF robot arm with TRPO?

The most important part of the RL is a reward function. If we want agent to do something speciﬁc, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour.
Assume, the robot's goal is to reach desired position as fast as possible. You can construct you reward function so, that it will take into account the Euclidean distance to the position. If the arm moves to the position directly, you will reward the agent with a positive value, otherwise you will punish it with a deviation from the direct line. You probably have other parameters of the joints, such as position and velocity. It can be also included in your reward function, in order to find optimal movements.

Check out this video from the free udacity overview course on RL and this paper "Setting up a Reinforcement Learning Task with a Real-World Robot"
Here is also related DeepMind's article and paper
I also have a project on github, where I implemented custom Gazebo environment for OpenAI Gym. This allows you to run the test even on a Jupyter Notebook. Check out my example


How and when should we update the Q-target in deep Q-learning?

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.

It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target network like DQN. The authors of the paper state that:

  The weights of these target networks are then updated by having them slowly track the learned networks: $\theta ' \leftarrow \tau \theta + (1 − \tau )\theta'$ with $\tau << 1$. This means that the target values are constrained to change slowly, greatly
  improving the stability of learning.

This update will be made in each time step as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This does not go against the purpose of fixed target networks (which have been introduced to address the problem of “moving targets”). In fact, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed.

Seeing that this resulted in improvements in DDPG, some DQN implementations/tutorials started to use soft updates for the target network.

This is opposed to the hard update scheme used in the original DQN paper, i.e. the weights are copied every $C$ steps. This means that the target network is kept fixed for $C$ steps (10000 in the paper) and then gets a significant update.


Is this technique image processing or computer vision?

This is a computer vision task, given that it requires a high-level understanding of the image (that is, it contains text, and the text needs to be translated to another language), which can use an image processing technique (for example, to deblur the possibly blurry original photo).


Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.

Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.


What's the difference between a 1d tensor and a 2d tensor with 1 dimension?

The required shape of the tensor $T$ depends on the shape of other tensors that are involved in the same operations of that same tensor $T$ and the required/desired shape of the resulting tensor, in the same way that the number of columns of the matrix $M \in \mathbb{R}^{n \times m}$ needs to match the number of rows of the matrix $M' \in \mathbb{R}^{n' \times m'}$ when you perform the matrix multiplication $M M'$, i.e. in order for $M M'$ to be well-defined, $m = n'$.
So, in your case, although the tensors contain the same elements, it might not be possible to use both in the same operations. Without more context/details, I cannot specifically answer why the first tensor was used in the tutorial you're mentioning.


What is the difference between hypothesis space and representational capacity?

A hypothesis space/class is the set of functions that the learning algorithm considers when picking one function to minimize some risk/loss functional.
The capacity of a hypothesis space is a number or bound that quantifies the size (or richness) of the hypothesis space, i.e. the number (and type) of functions that can be represented by the hypothesis space. So a hypothesis space has a capacity. The two most famous measures of capacity are VC dimension and Rademacher complexity.
In other words, the hypothesis class is the object and the capacity is a property (that can be measured or quantified) of this object, but there is not a big difference between hypothesis class and its capacity, in the sense that a hypothesis class naturally defines a capacity, but two (different) hypothesis classes could have the same capacity.
Note that representational capacity (not capacity, which is common!) is not a standard term in computational learning theory, while hypothesis space/class is commonly used. For example, this famous book on machine learning and learning theory uses the term hypothesis class in many places, but it never uses the term representational capacity.
Your book's definition of representational capacity is bad, in my opinion, if representational capacity is supposed to be a synonym for capacity, given that that definition also coincides with the definition of hypothesis class, so your confusion is understandable.


Since you said detection, you could do this on OpenCV: HAAR Cascade Detection for faces. It's swift, and should fit your use case. I've seen such scripts running on small Tello drones and gets job done relatively well.

Definitely there is hardware built around for this purpose. Look for Nvidia Jetson Nano and Jetson TK1/TX1. It's got sufficient juice to take even 30fps IMO.


Training a neural network to output the conditional probability of an event when the training data output is only binary

To sort the terms a bit, back propagation (repeatedly applying the chain rule of differentiation) is an implementation of stochastic gradient descent, which is an implementation of optimization. It is not necessarily the square of the error that is reduced by optimization - it is up to us to choose the loss function.

The loss function should ideally mirror the utility function (what you are truly want to optimize), but it needs a useful derivative. For example, the 0-1 loss hasn't one.

I think that what is called linear confidence guess in the question is the conditional probability of the event, given the input. Then the binary cross entropy would be a valid choice of a loss function.

From deeplearningbook.org

  The negative log-likelihood allows the model to estimate the conditional probability of the classes, given the input, and if the model can do that well, then it can pick the classes that yield the least classiﬁcation error in expectation.


Can a computer conclude following philosophical concepts from a story?

No. This is currently out of the scope for any language processing system. It requires a general understanding of abstract concepts which is not possible for machines at present.

In order to recognise a self-fulfilling prophecy, you first need to identify that something is a prophecy. So it needs to be something that expresses a possible future state, for which you need to identify what possible future states are; and then you need to see whether it is self-fulfilling. Conceptually this is far too complex to do.

You might get away for some of these with formal criteria (eg use of future tense for something describing a future state/event), but this is far too imprecise.

"Altruism" requires knowledge about typical expected behaviour; you would need to be able to identify motives behind people's actions, and then decide whether it was altruistic or not. This is just too complex for now (and the foreseeable future).


Can an RL algorithm trained in one environment be successful in a different one?

Can an RL algorithm trained in one environment be succesfull on a different one?

Strictly the answer here is "no". You train an agent to solve a single environment. If a second environment is similar enough, you can probably re-train the agent on the new environment with little or no changes to the agent, and to get results just as good in the second agent.

In some cases, you could even start with the trained agent and "fine tune" it to the new environment. That probably would not work well with the labyrinth example though, especially if the start and end points are moved.

  Example: If I train a model to go through one labyrinth, could this model also go through different but similar labyrinth or would it need a new training process?

This could be subtly different. It is down to how you define the environment. It is possible to define an environment not as "this maze" but for example "all possible mazes in a 30x20 grid pattern". To do so, you need to make the maze configuration part of the environment's state.

Expanding from a singular environment example to all similar environments has a cost. In the labyrinth example, this is significant. A single 30x20 maze has 600 states, which is a trivial problem in RL. All possible mazes of the same size has closer to $2^{1200}$ states, which is much larger and requires different kinds of approaches, different RL methods.

You would expect to train on many example mazes (typically a randomly generated new one for each episode) and use some kind of generalisation - e.g. a neural network, maybe a CNN due to the grid design - when handling the value function or policy function. Training time for a single maze will be under a second, and the policy will be perfect. Training time for all possible mazes measured in hours or days and the policy will still get things wrong from time-to-time.


How can I solve the zero subset sum problem with hill climbing?

The hill-climbing algorithm to implement is as follows:

The algorithm should take four inputs: as always, there will be a multiset S and integer
k, which are the Subset and Sum for the Subset Sum problem; in addition, there
will be two integers q and r, with roles defined below.
Do the following q times:

(a) Choose a random subset (multiset) $S_0$ of S as the current subset.

(b) Do the following (hill climbing) r times:

i. Find a random neighbor T (see definition of neighbor below) of the current
subset.

ii. If neighbor T has smaller residue, then make T the current subset.

(c) Keep track of the residue of the final current subset when starting with subset
$S_0$.

Return the smallest residue of the q subsets tested by the algorithm.

Definition: Subset (multiset) B ⊆ S is a neighbor of a subset A of S if you can transform
A into B by moving one or two integers from A to B, or by moving one or two integers from
B to A, or by swapping one integer in A with one integer in B.
An easy way to generate a random neighbor B of a subset A of S is as follows:

Order the elements of S as $x_1, x_2, ..., x_n$.
Initialize B to be a clone of A.
Choose two distinct random indices i and j, where $1 ≤ i; j ≤ n$.
if $x_i$ is in A, remove it from B. Otherwise, add xi to B.
if $x_j$ is in A, then with probability 0.5, remove it from B. If $x_j$ is not in A, then with
probability 0.5, add $x_j$ to B.


It is important to think about what sort of patterns in the data are being represented.

Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something that PCA would work fine for, because each of the dimensions (we can think of each pixel as a different dimension) is perfectly linearly correlated.

Suppose instead that you have a dataset of black and white 128x128px bitmap images of centred circles. As a human brain you'd quickly realise that every element in this dataset can be fully described by a single numeric parameter, which is the radius of the circle. That is a very impressive level of reduction from 16384 binary dimensions, and perhaps more importantly it's a semantically meaningful property of the data. However, PCA probably won't be able to find that pattern.

Your question was "Why can't we simply use PCA, if the purpose is dimensionality reduction?" The simple answer is that PCA is the simplest tool for dimensionality reduction, but it can miss a lot of relationships that more powerful techniques such as autoencoders might find.


What is a "batch" in batch normalization?

The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)

generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.


Should I be trying to create a generic or specific (to particular game) reinforcement learning agent?

If what you mean by a generic reinforcement-learning application is an application that can learn any game (or some games), then you can't do it. Why? Because the goal of each game isn't the same, so you have to adapt the rewards depending on the game. If you just want to make an AI for Connect Four, I suggest you to make a specific RL application for that game.

I want to mention another thing: you shouldn't give a reward based on token's position, because it's hard to know what token's position is the best. Instead, just assign a big reward to the winner. That way, you're generalizing your algorithm, and you're avoiding your AI to focus on the wrong goal.

You have to be careful, you don't know for sure what token's position is the best. Try to give a reward ONLY when a player wins the game, because you can know for sure that winning is the best thing to do. It may sound idiot said like that, but that way, you're ensuring you AI to learn by itself what token's position is the best.


Standard deviation in the # of neurons per layer

Input layers will always have the dimensionality of your input data(for every model I can think of).
See above, the deviation between hidden layers can be significant. For example, 128 in the first hidden and 64 in the rest(or vice versa).
This question in particular will always be problem dependent. It is decided via architecture search or intuition/experience combined with some exploratory search.


Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?

$\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.

The minus is there because we are performing gradient descent. For more information on this, see e.g. the wikipedia page


Does this argument refuting the existence of superintelligence work?

I believe this argument is based on the fact that intelligence is a single dimension when it really isn't. Are machines and humans really on the same level if a machine can solve a complex problem in a millionth of the time a human can?

It also assumes that the Turing machine is still the best computational model for the time period that you are in, which is not necessarily true for the future, it is just true until this point in time.


Has any schema-agnostic database engine been implemented?

A very large one, the world wide web with highly scaled and optimized indexing by Google.com is the most distributed and robust schema-agnostic database known today.  Without the schema-awareness Google brought to the table by applying more rigorous information science to the table, it was almost useless to those that did not know the URL of the target document in advance.

Schema agnosticism is another way of saying that the database cannot

Provide meta information to the services accessing it,
Normalize the structure using simple SQL query-insert combinations
Proactively optimize the keys automatically as is now possible with machine learning, or
Validate insertions

Without first detecting a schema from data patterns.  Moving away from structure is appealing because you can just jam data in like a librarian without a book shelf.   However, the data scientist will point out that adding entropy working alongside thermodynamic devolution into stochasm.

The purpose of storing data is to be able to retrieve it.  Feature extraction is an opportunity to improve structure automatically during the storing structure, rather than store documents chaotically, a trend that will not lead anywhere good for the world of IT.

Consider whether Google is successful because it organizes its data as it crawls or later as we enter key phrases.  Which is the efficient sequence?

One more point, Wikipedia is a blog, and they know this, which is why they want peer review for everything now (after much of the information was added without peer review).  It is a good place to find lists but not verified facts.  The existence of a Wikipedia page is definitely not an indication of the value of the concept on it.


Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?

The discount factor is not something you should be optimising. It is typically part of the problem statement.

For practical purposes, you may set it below 1.0 for continuous problems when in fact you care about best long-term reward. Another option to avoid infinities on continuous problems is to re-formulate the problem as optimising average reward. A high discount factor of e.g. 0.99 or 0.999 should produce a similar policy as one based on average reward.

  Is the author correct here or does it depend on cases?

The author appears to be either completely wrong, or just poor at explaining themselves on this part.

  If it is, then what kind of problems and/or situations should I set the discount factor as low as such figure at?

A low discount factor is for when you care much more about immediate rewards. You set it that low when that is the case. You decide what you care about when you set the learning problem. The value of the discount factor is part of the setup that decides what the optimal policy is. You never set it low "to help with optimising" because changing the value could change the optimal policy.


Does a neural network exist that can learn every possible training data?

The branch of AI research that answers questions like this is called computational learning theory.

For the specific question you have asked, the universal approximation theorem does indeed prove that any function can be modeled by a sufficiently wide neural network. The definition of a function includes the requirement that each input be mapped to exactly one output, so contradictory labels in training data are excluded explicitly.

Here is a rough sketch that can provide an intuition behind why this is true. This is not a proper proof, but it gives you an idea of the power of "finite number of neurons" in a hidden layer:

A single neuron can essentially learn to draw a straight line across the space of input features. It outputs a value arbitrarily close to 1 for things on one side of the line, and arbitrarily close to 0 for things on the other side of the line.
For any given datapoint, it is possible to enclose the hyper-volume containing that datapoint, and no others by drawing a series of lines in the input space, and defining one side of each line as "inside" the hyper-volume, and the other side as "outside". In 2-d space, this corresponds to drawing the 4 sides of a square around a point.
An output neuron learns to draw lines across the outputs of the hidden layer neurons. It can therefore decide to output 1 only when, say, 4 other neurons are all simultaneously active.

It should seem natural then that a sufficiently wide neural network can memorize all of an input pattern. Since memorization is sufficient for "learning" to fit a pattern this should give you an intuition for the ability of neural networks to fit things.


Initialising DQN with weights from imitation learning rather than policy gradient network

My understanding is that you are first training a policy network using imitation learning. Then you are adjusting that trained network in some way to be a value network for DQN. The most obvious change would be to remove softmax activation whilst keeping the network layer sizes identical. This would then present Q estimates for all actions from any given state.
The initial estimates would not be trained Q values though, they would be the "preferences" or the logits for probabilities to support a near optimal action choice. The main thing that will be likely in the new network is that for the one near optimal action choice, the network would predict the highest action value. As you derive the target policy by taking the maximising action, initially this looks good. However, the problem is that the Q values that this network predicts can have little to no relation to the real expected returns experienced by the agent under the target policy.

Initial rewards from running the policy are high but start to decrease (for a while) as the DQN trains and later increases again.

I think what is happening is that initially the greedy policy derived from your Q network is very similar to the policy learned during imitation learning. However, the value estimates are very wrong. This leads to large error values, large corrections needed, and radical changes to network weights throughout in order to change the network from an approximate policy function to an approximate action value function. The loss of performance occurs because there is not a smooth transition between the two very different functions that also maintains correct maximising actions.
I don't think this can be completely fixed. However you might get some insight into potential work-arounds by considering that you are not just doing imitation learning here. Instead you are performing both imitation learning (to copy a near optimal policy) and transfer learning (to re-use network weights on a related task).
Approaches that help with transfer learning may also help here. For instance, you could freeze the layers closer to input features, or reduce the learning rate for those layers. You do either of these things on the assumption that the low-level derived features (in the hidden layers) that the first network has learned are still useful for the new task.


What is the correct way to use the implication in first-order logic?

When we state in English that "All As are Bs", this means that we gain information as soon as we observe an A, we can immediately deduce that it must also be a B. These are the kinds of situations where we use an implication. So, this would be written in formal logic as:

$$\forall X \left( A(X) \rightarrow B(X) \right)$$

When we state in English that "Some As are Bs", we do not gain any new information just from observing that something is an A, we cannot deduce anything about that A. It might happen to be one of the As that simultaneously is a B, but it also might happen not to be one of those examples. So, it would be wrong to use an implication here. The only information that the English sentence gives us is that there is at least one thing somewhere that happens to be an A as well as a B, which is written formally as:

$$\exists X (A(X) \land B(X))$$

Suppose that we would have written the following in logic:

$$\exists X (A(X) \rightarrow B(X))$$

This would be translated to English as follows:

  There exists some $X$ such that, if it is an $A$, it is also a $B$.

That bolded part is very important there. Note that this logical statement is also true as soon as I find one example $X$ that is not an $A$. For example, the following statement is true in the real world:

  There exists some human $X$ such that, if $X$ can fly, $X$ can also shoot fireballs from his or her hands.

(this is true in the real world, because I can come up with many examples of humans who cannot fly)


How to track performance of your model during experimenting?

Maybe you are looking for a combination of a version control system (like git and Github) and a tool like comet.ml. In the past, I used comet.ml to keep track of different experiments performed with different hyper-parameters or different versions of the code. There are other alternatives to comet.ml, such as sacred, but they may also have different features and may not be as visually pleasing as comet.ml or even free. Personally, I liked comet.ml (even though, at the time, it still lacked some features). In any case, a VCS, like git, is widely used in software development (not just in AI projects) to keep track of different versions of the code, etc. You may also be interested in continuous integration (e.g. Travis CI) and code review (e.g. codacy) tools.


Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?

From wikipedia article on  Connect Four:

  Connect Four is a solved game. The first player can always win by
  playing the right moves.

It's pretty reasonable that even without training 1st player win more often by small margin.
With some training  1st player should mostly win.


Generate credit cards dataset for locating number region

I am assuming the question you are asking is how to prevent over-fitting on the maximum accuracy. Your graph does show that your model over-fits.

There is a couple of different methods to prevent over-fitting from happening. You can specify training to stop after a certain amount of epochs. In your case it seems to be 2 or 3 epochs. Take care as a new initialization might require more or less epochs to reach optimal accuracy thus it is required run your model a couple of times to determine the correct amount of epochs.

You can also specify the accuracy you want your model to reach before you stop training, this can be dangerous as a local-minima could be found below your expected accuracy thus resulting in your training to run indefinitely. Also your model could have become more accurate then your expected accuracy but stopped prematurely due to the model reaching your stopping-condition.

You can combine the two. Terminate training once it reach your expected accuracy otherwise terminate it if it reaches a certain amount of epochs. This way you prevent your training from happening indefinitely.

In Tensorflow you would want to build your own custom training seen here

The method you are interested is

epoch_accuracy = tfe.metrics.Accuracy()

In their example they get the accuracy after each epoch, you would want to create your own batches and apply it there.

If you really want to dive deeper, you can implement methods for the model to detect when it starts over-fitting such as looking at the standard deviation of the accuracy across a certain measure (usually epochs but your model starts over-fitting after 5 so its not a good measure in your case, maybe do standard deviation over a certain amount of batches trained) if the accuracy leaves the standard deviation space then it might be good idea to stop training. The problem with this technique is that your model has to over-fit a bit to know it needs to stop. The other problem is that your model might seem it is over-fitting but was just a dip in accuracy and was stopped prematurely. (There is methods around this but can't recall them)

Another technique is to add drop out. This introduce noise and pushes your model out of a local-minima. Also forces nodes that was ignored to be used and optimized. This surprisingly works well but is not a surefire way to prevent over-fitting. This technique is built into Tensorflow and don't need custom training. You can find more reading about Tensorflow dropout here, also they have more techniques on how to overcome over-fitting but their solutions is to change the model and your model seems to be fine.

tf.keras.layers.Dropout(0.2),

These are just some techniques I listed but there is a lot more out there. Unfortunately there is no concrete method to prevent it but steps can be taken to stop well before over-fitting occurs. The first two techniques are mostly used together.


How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

As explained here, I can write $R(s,a) = R(s)\  \forall a$ since the reward of my specific MDP is dependent exclusively to the state $s$.


Is it possible to classify data using a genetic algorithm?

It is possible, but is a pretty terrible idea.

There are a few options. One is to not use the GA as a direct classifier, but instead use a GA to learn the parameters of another classification model like a neural network. The basic idea of a GA is that it (very roughly speaking) forms a black-box method for searching an arbitrary space for solutions that minimize or maximize some function.

Here, you would be searching the space of possible neural network topologies and/or weights to find one that minimizes the misclassification rate.

Another approach is that taken by what are sometimes called Learning Classifier Systems (LCS) or Genetics Based Machine Learning (GBML). This approach is to use evolutionary mechanics to evolve rule sets of the form "if X condition is true, then do/classify Y". That's a more direct method of solving this sort of problem. You define some features on your dataset, and the algorithm tries to learn rules based on those features.

The problem with any of these approaches is just that there are so many better ways to solve the problem. Remember, a GA is basically a black-box that's supposed to work acceptably well for a huge range of unknown problems. But I'm not solving a huge range of unknown problems. I'm trying to separate ham from spam on one dataset. I can come up with methods that simply do that job better and more quickly than a GA has any real hope of doing.


How can I use Generative Adversarial Networks to solve the imbalanced class problem?

In my experience, GANs work really well for the scenario of semi-supervised learning, where you don't necessarily have labels for all your class $B$ data, but you do have a balanced dataset. In my (limited) experience, you do have to have a balanced (in numbers) set of $A$ and $B$ objects, even though you are not sure of the labels.
And yes, GANs can overfit to outliers as well, especially in the absence of a lot of examples, so be cautious.
Currently, the version that works best for me (in terms of GANs) is WGAN-GP or WGAN-LP in combination with Optimistic Mirror Descent Adam (here, Ncritic/Nactor = 1). Take a look at the paper by Adiwardana et al., especially Fig. 7 (astonishing!), for semi-supervised learning with a limited number of class labels.


How does the repetition of features across states at different time steps affect learning?

In RL, neural networks may intuitively be thought of as using the input features as a representation that "identifies" the input state (or input state + action pair). Think back to the "tabular" RL setting that most people first study when they learn about RL. In tabular RL, you have a table of values (state values $V(s)$, or state-action values $Q(s, a)$), with unique entries in the table for every state. Such a table can perfectly identify states or, in other words, perfectly disambiguate different states.

In a non-tabular, function approximation setting, with function approximators such as Neural Networks, you can generally no longer uniquely identify every single state. Instead, you use approximate representations of these states, and the approximation implies that it's possible that you have multiple different states that look identical; they have identical input features. This is the case you're dealing with. Now, you specified explicitly that these multiple states with identical representations / input features follow each other up immediately in a single episode, but I don't think this detail is particularly important. You'd have exactly the same problems if these different states with identical representation showed up at different times within an episode. The only problem that you really have is a disambiguation problem: you don't know how to disambiguate these states, since they look identical to the network.

How significant that problem is depends on your domain. Based on your domain knowledge, do you expect the optimal action, or the optimal values, to be kind of similar in all these states that have identical features? If so, no problem! Your network already thinks they're the same anyway, so it will learn that the same actions / same values are the best in those states. But do you expect the optimal actions / true value functions to be wildly different in these states despite the fact that a network can't disambiguate them? In this case, the problem will be more severe because you can't realistically expect your network to learn the optimal actions / value functions for all these different states. At best, it can learn a weighted average among them (weighted by how commonly they occur in your training episodes).


What kind of data structures are needed to efficiently do back-propagation in a feedforward neural network?

TL;DR: You'll need to store a little bit more to perform backward passes. You'll need to store data from the forward pass. This stored information is used for calculating the gradient.

Overview (warning: not trivial)

  I know the weights can just be stored in an array

You'll need a little more:

To update the weights you need to keep a "cache" of the forward pass intermediate terms. That is, forward propagation can be seen as a series of transformations on your input $X$:
$$X\xrightarrow{\Theta^{[1]}+b^{[1]}} [ Z^{[1]} \xrightarrow{\alpha^{[1]}} A^{[1]}] \xrightarrow{\Theta^{[2]}+b^{[2]}}
\dots
\xrightarrow{\Theta^{[L]}+b^{[L]}} [ Z^{[L]} \xrightarrow{\alpha^{[L]}} A^{[L]}]\xrightarrow{\frac{1}{m}\sum\limits_m\sum\limits_{n_L} loss\{A^{[L]},y\}} J
$$

where:

$Z^{[1]}=\Theta^{[1]}X+b^{[1]}$ (ie the linear part)

$A^{[l]}=\alpha^{[l]}(Z^{[l]})$ (ie element wise activation over linear part)

You need to store the $Z^{[l]}$ & $A^{[l]}$ terms in said "cache." You could store these in an array or some other similar data structure. You need these for calculating the gradient during the backwards pass.

Syntax

$A^{[k]}$ - this means we are indexing by layer (eg $\alpha^{[k]}$ is the activation for k-th layer)

$m$ - is the number of examples in the batch

$n_k$ - denotes the number of neurons in the k-th layer

$L$ - the number of layers (so $n_L$ is the number of neurons in last layer)

$\Theta$ - The set of all weights (notice no superscript)

Backprop

In the case of neural networks the cost is a scalar function of inputs and parameters. To get backprop started calculate the scalar by matrix derivative of the cost with respect to the activations of the last layer call this matrix $dA^{[L]}$. Observe:

$dA^{[L]} = \frac{\partial J(\Theta,X)}{\partial A^{[L]}}$

Next, we calculate scalar-by-matrix derivative of $Z^{[L]}$. Doing this one realizes:

$dZ^{[L]} = \frac{\partial J(\Theta,X)}{\partial Z^{[L]}} = dA\odot\alpha'^{[L]}(Z^{[L]})$

Where $\odot$ denotes element wise (Hadamard) product.

With the above one can make use of the matrix definitions for back propagation:

$\text{(A)}\quad d\Theta^{[l]} = \frac{1}{m}dZ^{[l]}\times (A^{[l-1]})^T$

$\text{(B)}\quad db^{[l]} = \frac{1}{m}\sum_{c=1}^m dZ^{[l](c)}$ (where the new superscript in $dZ^{[l](c)}$ denotes summing along the batch dimension )

$\text{(C)} \quad dZ^{[l]}= dA^{[l]}\odot \alpha^{'[l]}(Z^{[l]})$

$\text{(D)}\quad dA^{[k]} = (\Theta^{[k+1]})^T\times dZ^{[k+1]}$

And of course the wight updates are:

$\Theta^{[L]} \leftarrow \Theta^{[L]} - \frac{\eta}{m}d\Theta $

$b^{[L]} \leftarrow b^{[L]} - \frac{\eta}{m}db $

(where $\eta$ is the learning rate)

Observe, how the forward pass terms are used during the backprop calculations.

A recommendation

Take the A. Ng deep learning specialization. He does a good job explaining the intuition and even has a project to implement this. Though, he does not derive the back propagation equations. You can find a not so easy derivation here.


Overfitted model performs better in test set

Whether a model is over-fitting the data cannot be determined solely from false positive and false negative data on training and test runs.  If the sets are too small, very little can be determined.  If the sample selection method is choosing inequitably with respect to the training model, accuracy information may be misleading.

If the first run was just a matter of luck in selection, that could be determined by making multiple training runs with the first model.

Even if over-fitting is the cause of the drop in testing accuracy for the first model, that does not mean that the over-fitting model isn't none-the-less more fit that the second model.  It doesn't appear to be over-fit from that one training and test run, but may be deficient in other ways.  The second model may under-fit, for instance.


The keyword here is Parameter Sharing or Weight Sharing across various image portions.

If we take a simple example of grayscale binary image of an alphabet 'F', it is a combination of multiple patterns. The patterns here are vertical lines and horizontal lines. These patterns are based on relation between intensities of contiguous cells. This relation between contiguous cells is established using a weight matrix.

Also, for identifying multiple horizontal lines, we dont need multiple node-sets in hidden dense layer trying to identify different horizontal lines in the image. The pattern is same but present in different locations. Hence the sharing of weights comes into picture.

In the 1st hidden layer, encode the pattern horizontal line in a weight matrix(learnt during training and used in testing). Place it over small grid and check for presence. As this matrix is slided and tested across the image, the presence of horizontal lines is marked in various locations. This weight matrix is called a kernel.

Combining the above points, kernel provides a way of handling parameter / weight sharing between contiguous cells to identify patterns. Dense layer instead of kernels would solve it eventually but start in a random manner. Since a efficient way was identified, it is being used.

Next to identify vertical lines, another kernel needed and slide across.

Suppose next we have dense layer as 2nd hidden layer. This layer looks for combination of patterns ('p' horizontal lines and 'q' vertical lines in this case for 'F') present and learns combinations to identify output.

Just to compare with traditional programming, kernels are like regular expressions. dense layers are like loops. Just sharing my thoughts. Any better explanation is welcome.


Training problem with deep neural network model

Without seeing your code and/or data, it's going to be difficult to give a concrete answer.

Possibly reasons might include:

Your model isn't complex enough to fit the patterns in the data, and so it reaches a local minimum in the error function.
The patterns in the data are not self-consistent, and so this is actually the global minimum in the error function.
There's something wrong with your learning rate parameter, or with the gradient in the problem. Try increasing the learning rate, and ensure your model is being initialized with random weights.


Does this prove AI Safety is undecidable?

In my opinion, there are several flaws in your proof and reasonings.

First, note that, in the case of Turing's proof, h will actually loop forever (i.e. not halt) when the oracle says that h halts. In this case, there's an actual contradiction, because h will do the opposite of what the oracle says.

So, to follow Turing's proof, you would need to make h behave unsafely if the oracle says h is safe. But how should we define a safe or unsafe program? There are many unsafe behaviors. For example, in a certain context, an insult could be unsafe, in other contexts, a certain limb movement could be unsafe, and so on. So, an agent is unsafe or behaves unsafely usually with respect to another agent (or itself) or environment. You probably need to keep this in mind if you want to prove anything about the safety of AI agents.

In your second assumption, you are implicitly saying that any machine that produces the output unsafe_output is unsafe, but, of course, this definition is not a realistic definition of an unsafe program.

To help you define safety in a more reasonable and natural way, I think it may be useful to reason first in terms of artificial agents, which are higher-level concepts than Turing machines. Then you could find a way of mapping agents to TMs and attempt to prove your conjectures by using the tools of the theory of computation.


Do deeper residual networks perform better or worse?

Just by having more parameters, the deeper model has a higher capacity than the smaller one. This means that theoretically it can learn to extract more complex features from the data. Additionally, more layers means that the model can extract even higher-level features from the data.
So, generally speaking, deeper models will most of the times outperform shallow ones for more difficult tasks.

The downside is that if you have a small amount of data, a high-capacity model has the ability of memorizing the training set, which would lead to overfitting.
Besides performance, deeper models require better hardware and larger training times. So, there are plenty of reasons for one to prefer a more shallow model to a deeper one.


Why does the number of feature maps increases in the VGG model?

Why are we going from one image of $224 \times 224 \times 3$ to two images of $224 \times 224 \times 64$?

They do a Convolution with a $1 \times 1$ kernel, with $64$ filters. That way, you keep your size as the inputs ($224 \times 224$), but changes the number of filters to 64.

And this is not two image, but two layers !

  Why does this halving continue? What is this meant to represent?

This is the maxpooling operation (the layer is in red colour, see the legend). In general, max pooling is applied with a kernel stride of 2 (rarely 3, but not more, because you lose too much informations). That way your size is cut by 2, and the activation maps are more little, and you have faster computation.

  It continues on to $56 \times 56 \times 256$. Why does this number continue to halve, and the number, at the end, the $256$, continues to double?

Still max-pooling. There are no reason to double the numbers of filters, though. This just a trend, but you can put the numbers of filters that you want.


Can an AI distinguish between good and bad according to people living in a restricted geographical area?

Even if you could do that (which I believe is a long way off), what would be the point?

If I understand you correctly, you want an AI system to learn through observation of human beings what their 'rules of interaction' are. It sees a person killing someone else, and then that person is punished by the community, so the AI learns that killing people is not right. However, that is already codified in laws... So what it would pick up are social behaviours, which hopefully be things like "be nice to other people", "don't do anybody any harm", "be honest and truthful", etc.

The first question then is how an AI could assess events[*] as to whether they are 'good' or 'bad'. If everybody lies and steals, that would be learnt as normal behaviour, and the AI would not be able to pick up that most people would see this as 'bad'[**]. Causal relations are also hard to grasp. Somebody steals something. Then later someone else buys him a drink. So stealing stuff means other people will buy you drinks? These are really hard problems to solve. You need to know about people's motivations, and the multitude of 'threads' of interaction happening at the same time, even in a very limited area.

So, recognising events and causal links, plus a moral evaluation of them is pretty difficult. I don't think we will get there anytime soon. Unsupervised learning of behavioural is also pretty difficult, as you only have unlabelled observations and no real criteria to classify them. Plus, many actions are morally ambiguous. Killing people is generally seen as bad. What about the officers who tried to assassinate Hitler in 1944? Life is complex, and our artificial models are not anywhere near that.

So even if you were able to do all that, what would you end up with? An AI system that has picked up a lot of unwritten rules about human behaviour, and then postulates that as laws? So everyone has to behave the same way? I just don't see the point of that, even as a thought experiment.

[*] leaving aside here the question how you determine what an 'event' is in the first place
[**] Please note that even if people steal things, they steal can view theft as a bad thing.


What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know.

Lets take this simple sequence first,

   X = [a,b,c,d,.......,y,z]
   Y = [b,c,d,e,.......,z,a]

Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details doesn't matter.

We can write this relationship in maths as

  $f(x)\rightarrow y$

where x is an element of X and y is an element of Y and f() is our neural network.

After training, if given input x = a our neural network will give an output b because f() learned the mapping between the sequence X and Y.

Now instead of the above sequence try teach this sequence to the same neural network.

 X = [a,a,b,b,c,c,.........y,z,z]
 Y = [a,b,c,,...z,a,b,c......y,z]

More than likely this neural netwok will  not be able to learn the relationship between X and Y. This is because a simple neural network can't learn and understand the relationship between the previous and current characters.

Now we train the same sequence in RNN, in RNN we take two inputs one for our input and a previous hidden values and two outputs one for output and next hidden values

  $f(x,h)\rightarrow (y,h+t)$

Important: here  h+t represents next hidden value and not the arithematic addition with time step t.

We will execute some sequence of this RNN model, at the start hidden values are considerd as zeros

   x = a and h = 0
   f(x,h) = (a,next_hidden)
   prev_hidden = next_hidden

   x = a and h = prev_hidden
   f(x,h) = (b,next_hidden)
   prev_hidden = next_hidden

   x = b and h = prev_hidden
   f(x,h) = (c,next_hidden)
   prev_hidden = next_hidden

   and so on

If we look at the above process we can see that we are taking previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.


Is it normal that SOM clusters the instances with the "versicolor" class into multiple different BMUs?

This is not a bug, assuming you hve implemented the SOM properly with best decay rules of learning rate and neighbourhood strength (in short best hyperparameters).

Think of the map as finding many local clusters in the Iris-Versicolor class. Since SOM's behave somewhat like k-means clustering, I would say this implies Iris Versicolor has many local clusters which are as closely knit together as the entire classes of Iris setosa and Iris virginica.