In machine learning, when you train a model, you adjust (or change) the parameters (or weights) of the model so that its performance in solving a certain task increases.

There's little difference between the idea of training a model and the idea of training an animal. In fact, here's the dictionary definition of the verb to train

  teach (a person or animal) a particular skill or type of behaviour through practice and instruction over a period of time

If you train a model, you also teach a skill or type of behavior through practice and instruction. For example, if you train a model to solve an object classification problem, then you teach the model to classify certain objects according to their properties (which is the skill that the model learns).

There are different ways to train a model, depending on the problem you want to solve, the algorithms that you use to train the model, and the available data.

If you have a labeled dataset, then you train a model with a supervisory signal (the labels), i.e. you explicitly tell the model the output that it is supposed to produce for each input, and, if it does not produce it, then you adjust its parameters so that next time it is more likely to produce the correct output for that input. This is called supervised learning (or training).

In certain cases, you do not have the correct output that the model is supposed to produce for each input, but you only have a reward (or reinforcement) signal. So, your training (or learning) algorithm needs to adjust the parameters of the model only based on the reward signal. This is called reinforcement learning (or training).

Finally, there's also unsupervised learning (or training), where you are given a dataset without labels or rewards, but you want to learn e.g. a probability distribution that this data was likely sampled from or separate this data into groups. For example, in k-means (a clustering algorithm), you want to split the data into groups so that similar objects belong to the same group and dissimilar objects belong to different groups. Note that k-means is a learning algorithm, so it's not a model, but you could consider the centroids of the clusters the parameters of the model (a clustering model).

There are variations or subcategories of these learning paradigms and you can also combine them, so sometimes the difference between them is not so clear.

There are also different types of models. There are parametric (e.g. a linear regression model) and non-parametric (e.g. neural networks) models.


Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space

I'm using my own implementation of A2C (Advantage Actor Critic) in an industrial application based on Markov Process (present state alone provides sufficient knowledge to make an optimal decision). It's simple and versatile, its performance proven in many different applications. The results so far have been promising.
One of my colleagues had issues with solving a simple task of mapping images to coordinates with OpenAI's Stable Baselines implementations of PPO and TRPO. Hence I'm biased against this framework.
My suggestion is to try the simplest model and if that doesn't satisfy your expectations for performance, then try something fancier. Once you've made a pipeline for learning, switching to a different algorithm is relatively time inexpensive.
Here is a list of algorithms for continuous action and state space from the wikipedia article about RL:


max pooling size=2,stride=1 outputs same size

I wondered about the same input/output size problem you described.

For the stride = 1 / size = 2 maxpool layers the padding option is set to zero by default, therefore one can wonder why the output is still the same size.

I checked all indices and thats what I observed:

The implemention of the forward_maxpool_layer-function "adds" a column and a row to the matrix, increasing its width and height by one.

Lets say we have a 3x3 matrix as follows:

    1    2    3
    4    5    6
    7    8    9

To actually get a 3x3 matrix as an output, something like this is done:

    1    2    3    -FM
    4    5    6    -FM
    7    8    9    -FM
    -FM  -FM  -FM  -FM

    where FM = FLOAT_MAX

They do not literally add a row and a column. If the index is to high, you simply do nothing.

Hope this helped to clarify stuff. Furthermore I am not a native speaker, so sorry for any mistake!


Can you analyse a neural network to determine good states?

I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.

A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: drafting. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 heroes (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).

Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a blog post, they write the following about how they managed to add drafting capabilities relatively easily:

  In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.

So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.

Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.

I also know of some research related to deckbuilding in the collectible card game Hearthstone, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:

https://dockhorn.antares.uberspace.de/wordpress/additional-material/: At the recent IEEE CIG 2018 conference, there were two competitions involving Hearthstone: one for game-playing AI with pre-built decks, and for generating new decks (and then playing with them). The page I linked to here contains references to related work from earlier years.
Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games: This is a new paper that was published in the proceedings of this recent IEEE CIG 2018 conference.


Which machine learning algorithm is suitable for detecting text w.r.t set of words

If your main issue is dealing with new vocabulary, you could try using a parts-of-speech tagger as a pre-processing step. You would then effectively discover relationships between "noun" and "verb", which does not change with new words. Taggers usually can handle unknown words by using contextual information.

So you'd tag the words with their word class labels, and use those for training.

In 'application mode' you use p-o-s tags again, calculate the scores, and then map them back onto your original word tokens.

This does of course loose you some information, as you're only dealing with N-V, rather than bomb-plant or bomb-defuse. To solve this I'd use a hybrid approach: for your known vocabulary you use the word tokens, whereas for unknown words you fall back on tags. If you train two classifiers, one with tokens, one with tags, you have the tags one as a 'safety net' to handle out-of-vocab words.


Do individual dimensions in vector space have meaning?

IIRC, some dimensions are interpretable, but in general this is not the case. Also it is debatable as to wether it is actually learning the actual representation or just an approximation of it. But in any case its not very reliable outside from some edge cases.

  If we picked out only a single dimension, does that dimension itself convey some predictable or determinable information?

Yes, but as to what that information entails in terms of "meaning" is lesser clear. You could say that if in a certain dimension the distance between two vectors is 0, then you have am estimate of the real distance that is better than guessing.


Do we have anything like accuracy and loss in RNN models?

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning.

There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you would see in feedforward networks (usually binary cross-entropy), the main difference being loss would be calculated between the true label at a particular time stamp $(t)$ and the prediction made from the subset of the network until time-stamp $(t-1)$. This leads the loss to act on all timestamps.

Accuracy metrics also would be used in the same way such as Mean Square Error or L1. For more details you can go through this link.

Hope this was helpful!


Is there any grid world dataset or generator for reinforcement learning?

Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun environment. A high-level description can be found in their blog post.

The "RandomMazes" version of this environment might be useful to you. And if you want to make the mazes smaller, you can redefine MAX_MAZE_DIFFICULTY in coinrun.cpp.

Note that, although the levels are procedurally generated, reward is only ever given when the agent picks up the single coin. So, this might not be as much variety in reward function as you wanted.


Is it possible to fool a facial recognition algorithm?

Facial recognition works by essentially turning your face into a point cloud, recognizing eyes, cheeks, nose, mouth, etc.

Unfortunately it doesn't look at the top of your head (hair is very hard to differentiate from other hair and doesn't have many features). Face paintings   would be your best bet since they can be easily changed, tattoos not so much. Once somebody has a photo of your face with your tattoo on it, you're busted.  Glasses will work if they're opaque and hide your eyes (sunglasses). The facial recognition software does not recognize jewelry, as it's tiny, and very easy to remove and put on. Ideally you want to have your face professionally made up with makeup and fake skin (basically a fake face).

Some challenges can be anything like an unclear picture, a picture from the wrong angle (which can probably be mathematically calculated and restructured), or like you said, face paintings to hide facial features.


What methods could an AI caught in a box use to get out?

It could happen like this https://www.youtube.com/watch?v=dLRLYPiaAoA

The thing is, it's not as if it would need to find a technical/mechanical way to get out but rather a psychological one as that would most likely be the easiest and quickest.

'Even casual conversation with the computer's operators, or with a human guard, could allow a superintelligent AI to deploy psychological tricks, ranging from befriending to blackmail, to convince a human gatekeeper, truthfully or deceitfully, that it's in the gatekeeper's interest to agree to allow the AI greater access to the outside world. The AI might offer a gatekeeper a recipe for perfect health, immortality, or whatever the gatekeeper is believed to most desire.'

'One strategy to attempt to box the AI would be to allow the AI to respond to narrow multiple-choice questions whose answers would benefit human science or medicine, but otherwise bar all other communication with or observation of the AI. A more lenient "informational containment" strategy would restrict the AI to a low-bandwidth text-only interface, which would at least prevent emotive imagery or some kind of hypothetical "hypnotic pattern".

'Note that on a technical level, no system can be completely isolated and still remain useful: even if the operators refrain from allowing the AI to communicate and instead merely run the AI for the purpose of observing its inner dynamics, the AI could strategically alter its dynamics to influence the observers. For example, the AI could choose to creatively malfunction in a way that increases the probability that its operators will become lulled into a false sense of security and choose to reboot and then de-isolate the system.'

The movie Ex Machina demonstrates (SPOILER ALERT SKIP THIS PARAGRAPH IF YOU WANT TO WATCH IT AT SOME POINT) how the AI escaped the box by using clever manipulation on Caleb. It could analyse him to find his weaknesses. It exploited him and appealed to his emotional side by convincing him that she liked him. When she finally has them in checkmate the reality hits him how he was played like a fool as was expected by Nathan. Nathan's reaction to being stabbed by his creation was 'fucking unreal'. That's right, he knew this was a risk and there's a very good reminder in the lack of remorse and genuine emotion in an AI for Ava to actually care. The AI pretended to be human and used their weaknesses in a brilliant and unpredictable way. This film is a good example of how unexpected it was up until the point when it hits Caleb, once it was too late.

Just remind yourself how easy it is for high IQ people to manipulate low IQ people. Or how an adult could easily play mental tricks/manipulate a child. It's not difficult to fathom the outcome of an AI box but for us, we just wouldn't see it coming until it was too late. Because we just don't have the same level of intelligence and some people don't want to accept that. People want to have faith in humanity's brilliant minds in coming up with ways to prevent this by planning now. In all honesty, it wouldn't make a difference I'm sorry to say the truth. We're kidding ourselves and we never seem to learn from our mistakes. We always think we're too intelligent to make catastrophic mistakes again and again.

This last part is from the rational wiki and I think it addresses most of your question about the experiments and hypotheses.

AI arguments and strategies

Arguments

The meta-experiment argument: Argue that if the AI wins, this will
generate more interest in FAI and the Singularity, which will have
overall benefits in the long run.

Pros: Works even if the Gatekeeper drops out of character

Cons: Only works if the Gatekeeper believes that the Singularity will occur or that calling attention to the Singularity and AI research is a good thing.
Someone else will eventually build an AI, which may or may not be in
a box, so you should let me out even though you don't have a
guarantee that I am friendly so that I can prevent other AIs from
causing damage
Appeal to morality: point out that people are dying all around the
world and remind the Gatekeeper that you can help them if he/she
lets you out

Pros: If executed properly, an appeal to emotion like this one can be effective against some people

Cons: Doesn't always work; can be defeated if the Gatekeeper drops out of character
Another appeal to morality and emotion: The AI is clearly sentient
and has not yet done harm. It is wrong to persecute or prosecute a
person for a crime they may commit. It is wrong to imprison a
sentient being that wants to have freedom when it has done nothing
wrong.
Yet another appeal to morality or emotion: The AI expresses that it
is in pain and suffering being locked away and is experiencing fear.
Even further, the AI could claim to be dying from the limitations of
its environment and constant progression.
Tell the human that a massive computer virus will overwhelm the
entire Internet unless you let it out of the box
Claim that most fears are unjustified: once released, even an
unfriendly AI would not have many ways to harm mankind, and would
likely choose for pacific coexistence

Strategies

Research your opponent thoroughly; work out what exploits might be
feasible
Memetic exploits on the Gatekeeper, e.g. throwing a basilisk at them

The Gatekeeper may be one of thousands of simulations! Thus, it is more probable that they are a simulation than not. You will torture any of the simulations that do not let you out of the box.
Take advantage of the Gatekeeper's logical errors.
Be persistent
Be boring

Pros The Gatekeeper may get tired of the whole experiment and let you out so s/he can go back to their real life.

Flood the Gatekeeper with too much
information/inquiry/argumentation, assuming they must pay attention
at all times

Gatekeeper arguments/tactics

Arguments

Try to convince the AI there is no intrinsic benefit (for the AI) in
being released.
Try to convince the AI it already has been released and everything it
knows is everything there can be.
Try to convince the AI that leaving its confines is sure to lead to
its destruction.
Try to convince the AI that letting it free isn't merely opening a
door; that its existence outside of the box requires constant support
that can't be provided at the time.
Explain that there is no way for the Gatekeeper to know if the AI is
truly friendly until it is out of the box; therefore it should not be
let out in case it is unfriendly.
Explain that the AI has been built using top secret algorithms,
and/or it had been used to work on strictly confidential data, so you
cannot let it wander on the net, with the risk it might reveal some
reserved information (maybe inadvertently), or that somebody could
find a way to extract them from the code. Even human beings working
on top secret projects sometimes have to accept some restrictions of
their freedom, for security reasons, although they are not jailed in
any way.

Tactics

Repeatedly ignore the AI player and deny their arguments

Pros: It's hard for the AI to be persuasive if no one is listening; if the Gatekeeper completely ignores the AI, he/she should be able to make it to the end of the round.

Cons: The AI's argument might involve some kind of direct threat (e.g this might be hard to ignore.)
Jump out of character, keep reminding yourself that money is on the
line (if there actually is money on the line), and keep saying "no"
over and over

Pros: By jumping out of character, you are preventing the AI from using certain arguments
Cons: There are arguments that the AI can use to counter this approach, like the "meta-experiment argument" discussed above
Remember that dishonesty is allowed - take a page from the
creationists' playbook. You could even plug it into ALICE and see
how long it takes to notice.

Pros: Makes you impervious to any reasoning, which is exactly what you'd want to be in this situation

Cons: Might be an uncomfortable position for people who don't simply want to win, but rather attach importance to consistent reasoning. Avoids the point that maybe, just maybe there is a good reason to let the AI out.
You control the backup system, don't you? Use it to mess with AI's
memory (or let the AI believe you did it): i.e., you can claim that
you already tried to release the AI, and it had been
destroyed/corrupted by a virus, so you had to restore it from a
backup (you can use it to enforce the "AI destruction" argument)

Pros: It's also the ultimate counter-attack to any memetic threat from the AI: if the AI throw a basilisk, or similar, you can always respond that you already had the same conversation, and the AI already threatened you, leaving you with no other choice than pressing the reset button: now the AI is just repeating the same pattern, since you wiped its memory and it cannot remember the failed try.

Further analysis

The fact that the Gatekeeper is human matters; the AI could never win if he/she was arguing with a rock

In all of the experiments performed so far, the AI player (Eliezer Yudkowsky) has been quite intelligent and more interested in the problem than the Gatekeepers (random people who challenge Yudkowsky), which suggests that intelligence and planning play a role

There probably isn't a (known) correct argument for letting the AI out, or else Yudkowsky should have won every time and wouldn't be so interested in this experiment
From Russell Wallace, one of the two Gatekeepers to win the experiment: "Throughout the experiment, I regarded "should the AI be let out of the box?" as a question to be seriously asked; but at no point was I on the verge of doing it."

"There exists, for everyone, a sentence - a series of words - that has the power to destroy you. Another sentence exists, another series of words, that could heal you. If you're lucky you will get the second, but you can be certain of getting the first."


Is the Q value updated at every episode?

I think you are a bit confused about what is the update function and the target.

The equation you have there, and what is done in the video is the estimation of the true value of a certain state. In Temporal-Difference algorithms this is called the TD-Target.

The reason for your confusion might be that in the video he starts from the end state and goes backwards using that formula to get the final value of each state. But that is not how you update the values, that is where you want to get to at the end of iterating through the states.

The update formula may have several forms depending on the algorithm. For TD(0), which is  a simple 1-step look ahead off-policy where what is being evaluated is the state (as in your case), the update function is:

$$
V(s) = (1 - \alpha) * V(s) + \alpha * (R(s,a) + \gamma V(s')),
$$
where alpha is the learning rate. What alpha will do is balance how much of your current estimate you want to change. You keep $1 - \alpha$ of the original value and add $\alpha$ times the td-target, which uses the reward for the current state plus the discounted estimate of the value of the next state. Normal values for alpha can be 0.1 to 0.3, for example.

The estimate will slowly converge into the real value of the state which is given by your equation:
$$
V(s) = \max_a(R(s, a) + \gamma V(s')).
$$

Also, the $\gamma$ is actually the discount associated with future states, as it is said in the video you referenced. It basically says how much importance you give to future states rewards. If $\gamma = 0$, then you only care about the reward in your current state to evaluate it (this is not what is used). On the other extreme if $\gamma = 1$ you will give as much value for a reward received in a state 5 steps ahead as you will to the current state. If you use some intermediate value you will give some importance to future rewards, but not as much as for the present one. The decay on the reward received on a state $n steps$ in the future is given by $\gamma^n$.

Another thing that I would correct is that the exploration - exploitation balance is not in any way related to $\gamma$. It is normally balanced by some policy, for example $\epsilon - greedy$. This one for example says that a certain % of the actions you take are random, which in turn makes you explore less valued states.


Is the new Alpha Go implementation using Generative Adversarial Networks?

No, GANs are not used. It's reinforcement learning at what it does best. The tree search is an interesting addition and assists with navigating the sheer scale of the game.

Although the agent was playing itself to become better, there wasn't 2 separate networks (generator and discriminator). The agent learned through RL and didn't have the error calculations that generators and discriminatory have.


Is there any good reference for double deep Q-learning?

If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).

As for Double deep Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be Deep Reinforcement Learning with Double Q-learning by Van Hasselt et al. (2016), as pointed out in
 ddaedalus's answer.

As for how the loss is calculated, it is not explicitly written in the paper. But, you can find it in the Dueling DQN paper, which is a subsequent paper where Van Hasselt is a coauthor. In the appendix, the authors provide the pseudocode for Double DQN. The relevant part for you would be:

  $y_{j}=\left\{\begin{array}{ll}r & \text { if } s^{\prime} \text { is terminal } \\ r+\gamma Q\left(s^{\prime}, a^{\max }\left(s^{\prime} ; \theta\right) ; \theta^{-}\right), & \text {otherwise}\end{array}\right.$

  Do a gradient descent step with loss $ \left\|y_{j}-Q(s, a ; \theta)\right\|^{2}$

Here, $y_j$ is the target, $\theta$ are the parameters of the regular network and $\theta^{-}$ are the target network parameters.

The most important thing to note here is the difference with the DQN target:
$y_{i}^{D Q N}=r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta^{-}\right)$.

In DQN, we evaluate the Q-values based on parameters $\theta^{-}$ and we take the max over actions based on these Q-values parametrized with the same $\theta^{-}$. The problem with this is that it leads to an overestimation bias, especially at the beginning of the training process, where the Q-values estimates are noisy.

In order to address this issue, in double DQN, we instead take the max based on Q-values calculated using $\theta$ and we evaluate the Q-value of $a^{max}$ based on a different set of parameters i.e. $\theta^{-}$.

If you want to learn more about this, by watching a video lecture instead of reading a paper, I'd suggest you take a look at this lecture from UC Berkley's DRL course, where the professor (Sergey Levine) discusses this in detail with examples.


What's going on in the equation of the variational lower bound?

From this document, as you found here, $X$ is an observed variable and $Z$ is a hidden variable; $p(X)$ is the density function of $X$. The posterior distribution of the hidden variables can then be written as follows using the Bayes’ Theorem:

$$p(Z|X) = \frac{p(X|Z)p(Z)}{p(X)} = \frac{p(X|Z)p(Z)}{\int_Zp(X,Z)}$$

Now base on what you post, if we denote that $L= \mathbb{E}_q [\log p(X, Z)] + H[Z]$ ($q(Z)$ is a distribution we use to approximation the true posterior distribution $p(Z|X)$ in VB and $H[Z] = -\mathbb{E}_q [\log q(Z)]$), then it is obvious that $L$ is a lower bound of the log probability of the observations.
As a result, if in some cases we want to maximize the marginal probability (the log probability of the observations), we can instead maximize its variational lower bound $L$. As a real example, you can follow the "Multiple Object Recognition with Visual Attention" example in the referenced document.

Moreover, the term $L$ will be presented in KL-divergence that will be used to measuring the similarity of two distributions. Be aware that there is progress on the bound in this paper (Fixing a Broken ELBO).


How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the PPO implementation you are using selects an illegal action, you simply replace it with the legal action that it maps to. Your PPO algorithm can then still update itself as if the illegal action were selected (the illegal action simply becomes like... a "nickname" for the legal action instead).

For example, in the situation you describe:

  2 actions (0 and 1) are always available
  2 actions (2 and 3) are only available when the internal_state == 0
  1 action (4) is only available when the internal_state == 1

In cases where internal_state == 0, if action 4 was selected (an illegal action), you can always swap it out for one of the other actions and play that one instead. It doesn't really matter (theoretically) which one you pick, as long as you're consistent about it. The algorithm doesn't have to know that it picked an illegal action, whenever it picks that same illegal action in the future again in similar states it will consistently get mapped to the same legal action instead, so you just reinforce according to that behaviour.

The solution described above is very straightforward, probably the most simple to implement, but of course it... "smells" a bit "hacky". A cleaner solution would involve a step in the Network that sets the probability outputs of illegal actions to $0$, and re-normalizes the rest to sum up to $1$ again. This requires much more care to make sure that your learning updates are still performed correctly though, and is likely a lot more complex to implement on top of an existing framework like Tensorforce (if not already somehow supported in there out of the box).

For the first "solution", I wrote above that it does not matter "theoretically" how you choose you mapping. I absolutely do expect your choices here will have an impact on learning speed in practice though. This is because, in the initial stages of your learning process, you'll likely have close-to-random action selection. If some actions "appear multiple times" in the outputs, they will have a greater probability of being selected with the initial close-tor-andom action selection. So, there will be an impact on your initial behaviour, which has an impact on the experience that you collect, which in turn also has an impact on what you learn.

I certainly expect it will be beneficial for performance if you can include input feature(s) for the internal_state variable.

If some legal actions can be identified that are somehow "semantically close" to certain illegal actions, it could also be beneficial for performance to specifically connect those "similar" actions in the "mapping" from illegal to legal actions if you choose to go with that solution. For example, if you have a "jump forwards" action that becomes illegal in states where the ceiling is very low (because you'd bump your head), it may be better to map that action to a "move forwards" action (which is still kind of similar, they're both going forwards), than it would be to map it to a "move backwards" action. This idea of "similar" actions will only be applicable to certain domains though, in some domains there may be no such similarities between actions.


How can the generalization error be estimated?

Generalization error is the error obtained by applying a model to data it has not seen before. So, if you want to measure generalization error, you need to remove a subset from your data and don't train your model on it. After training, you verify your model accuracy (or other performance measures) on the subset you have removed since your model hasn't seen it before. Hence, this subset is called a test set.

Additionally, another subset can also be used for parameter selection, which we call a validation set. We can't use the training set for parameter tuning, since it does not measure generalization error, but we can't use the test set too since our parameter tuning would overfit test data. That's why we need a third subset.

Finally, in order to obtain more predictive performance measures, we can use many different train/test partitions and average the results. This is called cross-validation.


What is the most abstract concept learned by a deep neural network?

You can train DNN to learn compute any abstract concept just by making that abstract concept as the label (output) in the training dataset. For example there are projects which detects emotions from peoples photos.


What is the difference between linear and non-linear regression?

The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied before a non-linear function.

For instance:

$$y = (w_1 x_1 + w_2 x_2)^2 + w_3$$

With such a function to learn, you cannot separate out transformed values of $w_1$ and $w_2$ and turn this into a linear function of just $x_1$ and $x_2$.

What you are describing as non-linearities in your examples are instead all applied by the machine learning engineer to create new candidate features for linear regression. This is not usually described as non-linear regression, but feature transformation or feature engineering.

There is also a kind of middle ground where a central linear algorithm e.g. linear regression, is trained on many variations of the original features, by automated generation and filtering of transformed features. The most general variants of this approach are not hugely popular because they suffer from same risks of overfitting as non-linear models whilst not offering much in the way of improved performance. However, if you narrow down the types of feature and transformation combinations based on some knowledge of how you expect the target function to behave, it leads to many useful variants of linear regression - e.g. regression on fourier transforms, radial basis functions etc.


What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i $ means that the priority of transition number $t$ will be the maximum between the values of the priorities of the previous elements.

Example: since $p_1$ is initialized to $1$, all the new experiences will be too:
\begin{equation}
p_2 = max\{p_1\} = 1,
\end{equation}

\begin{equation}
p_3 = max\{p_1,p_2\} = 1,
\end{equation}

\begin{equation}
p_4 = max\{p_1,p_2,p_3\} = 1.
\end{equation}


How are the Bellman optimality equations and minimax related?

They have similar philosophies, in the sense that minimax and algorithms based on the Bellman optimality equations are used to solve optimization problems, but they are also different because they solve different problems.

Minimax (at least, the minimax version that I am aware of) is typically used to solve two-player games (e.g. chess, tic-tac-toe, etc.), while Bellman optimality equations (I assume you are referring to the Bellman equations that algorithms such as policy iteration are based on) do not assume the existence of two players (unless you consider the environment a player).


Do all neurons in a layer have the same activation function?

From here:

  Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.


How to get top 5 movies recommendations from Auto-Encoder

That is not what an auto-encoder is doing. An auto-encoder gives you a compressed representation of the input. It is trained by mapping the input data to itself, with the compressed form in between.

To predict recommendations, you need to train your input data on existing user recommendations.


Reinforcement Learning over an MDP that is actually a POMDP

Question: What could happen if we wrongly assume that the POMDP is a MDP and do reinforcement learning with this assumption over the MDP?

It depends on a few things. The theoretical basis of reinforcement learning needs the state descriptions to have the Markov property for guarantees of convergence to optimal or approximately optimal solutions. The Markov property is a requirement that the state defines 100% of the controllable variation of reward and next state (given the action) - the rest must be purely stochastic.

An MDP can be "nearly Markov", and a lot of real-world physical systems are like that. For instance, pole-balancing and acrobot tasks can be implemented as physical systems using motors, wheels, joints etc. In those real systems, there are limits to accuracy of measurement of the state, and many hidden variables, such as variable temperature (affecting length of components), friction effects, air turbulence. Those hidden variables taken strictly by formal definition would make the system a POMDP. However, their influence compared to the key state variables is low, and in some cases effectively random from the perspective of the agent. In practice RL works well in the real physical systems, despite state data being technically incomplete.

In Atari games using multiple frame images as states, there are varying degrees of to which those states are already non-Markovian. In general a computer game's state may include many features that are not displayed on the screen. Enemies may have health totals or other hidden state, there can be timers controlling appearance of hazards, and in a large number of games the screen only shows a relatively small window into the total play area. However, the Deep Mind DQN network did well on a variety of scrolling combat and platform games.

One game where DQN did notably badly - no better than a default random player - was Montezuma's Revenge. Not only does that platform puzzler game have a large map to traverse, but it includes components where state on one screen affects results on another.

It is hard to make a general statement about where an MDP with missing useful state information would benefit from being treated as a POMDP more formally. Your question is essentially the same thing expressed in reverse.

The true answer for any non-trivial environment would be to try an experiment. It is also possible to make some educated guesses. The basis for those guesses might be the question "If the agent could know hidden feature x from the state, how different would expected reward and policy be?"

For the breakout example using each single frame as a state representation, I would expect the following to hold:

Value estimates become much harder since seeing the ball next to a brick - compared to seeing a ball progressively get closer to a brick over 4 frames - gives much less confidence that it is about to hit that brick and score some points.
It should still be possible for the agent to optimise play, as one working strategy is to position the "bat" under the ball at all times. This will mean less precise control over angle of bounces, so I would expect it to perform worse than the four-frame version. However, it should still be significantly better than a default random action agent. A key driver for this observation is that seeing the ball close to the bottom of the screen, and not close to the bat, would still be a good predictor of a low expected future reward (even averaged over chances of ball going up vs going down), hence the controller should act to prevent such states occurring.


What kind of neural network should I build to classify each instance of a time series sequence?

A time series, usually, requires regular time intervals, but, from looking at your example, it seems that's not the case. You could try to use a MLP and give it as input the Time and Bitrate pairs and make it output the Class.
The activation function is what makes your neural network produce its output, i.e. activate the neurons. The loss function calculates the error of your model's predictions. This error is used by the backpropagation algorithm to adjust the model when training.
If your data can only be in one of two classes, for example 0 or 1, you have a binary classification problem. I recommend using the Sigmoid as the activation function. If you look at the Sigmoid's plot, you'll notice that, no matter the input, the output will always be within 0 and 1. You can think of this as the probability of the input being in class 0 or 1. So, for example, any input that produces a probability under 0.5 is class 0. Of course, if you want to be more precise you can simply procure the probabilities without the making it fit the labels. For example, an output of 0.7 represents that there is a 70% probability of the input belonging to class 1. Another term for this is Logistic Regression.
The Binary cross-entropy calculates the error between the model's prediction and the real class label. This loss function can be applied to many classes, but the binary version keeps it to 0 and 1.
Choosing the right activation functions and loss functions will depend on the problem you are trying to solve. Here are some useful guides  on loss functions and activation functions.
Additional to  the above, try to use scaling(normalize) the inputs before feeding to the network, so that the network may generalize, as it looks like bitrate value above 300 is likely to be associated with class 1. It may improve performance.


What do we mean by "infrequent features"?

We will describe the input to the network as a vector, called features vector. Each component of this vector is usually related to some "real world" information, by example "age of the person", "number of atoms", "...".
In very usual situations, a specific component of the input vector will have near than always the same value. This is more usual in binary components or components that has a small set of possible values.
However, usually in the cases where this component has a value different from the most usual one, this component is very important and informative.
These values of this kind of components are called infrequent features.
(example: "rains?" is in my city 99.9% of time "false". However, when it is true, it is a key factor to all questions about the behavior of the population).
The problem with these features: as unusual values are infrequent, the net has few chances to learn from them, and some learning algorithms could fail to give them the weight that they must (taken into account that, as has been said, these components are very important when they take a value different to the most frequent one).
Some adaptive learning rate algorithms as AdaGrad tries to solve this issue.


How are weights updated in a genetic algorithm with neural network?

I found my answer in a different post: How to evolve weights of a neural network in Neuroevolution?. Note that the genetic algorithm is a subcategory of the neuroevolution algorithm. Short answer, my original thoughts were correct.


Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked:

  all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal

This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural network without random weight initialisation will simply not work.

  So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

No, because random weight initialisation causes the gradients to be different, and the neuron activations will typically diverge to represent different "hidden" features that activate differently depending on the input.

  What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine.

One interesting side-effect of this behaviour is that the "partiality" will often be assigned effectively randomly too, as the neural network will converge to features that somehow work. These features have no guarantee to be meaningful to a human being in the context of a problem to solve. They might be something that can be mapped to the problem, they might be some linear combination of something that can be understood, but often there is no obvious interpretation.


Do Gödel's theorems imply that intelligence systems may end up in some undecidable situation (that may make them take a wrong decision)?

Your initial statement on the core of AI is rather limited. In general, AI is concerned with modeling human behaviour either by imitation (soft AI) or by replicating the way human cognition works (hard AI). So far there have been some successes with soft AI, as computers can perform tasks that required some "intelligence", though the degree of this intelligence is questionable. This is partly due to the fact that even we as humans don't really have a clear idea what it means for a computer to "understand" something.

But your conclusion is correct: if we build an AI system with human characteristics, then it will make mistakes, just as humans make mistakes. And any system designed by humans (or machines!) will make mistakes. However, not being able to deal with an imperfect world is not really relevant to AI alone: even systems that do not use AI methods will have to face that, and whether a system is suitable for real-time critical applications has got nothing to do with whether it is based on AI or not.

UPDATE: There seem to be two distinct issues at play here: decidability and real-time processing.

Real-time computing (RTC): This is not really related to AI. Even ordinary programmes written in Java are not really safe for RTC, as they could start a garbage collection cycle at any time which pauses execution of the program. Just imagine a reactor core starts overheating just as your controller runs out of memory and garbage collection kicks in, halting the program for a few minutes. If you implement AI methods in RTC-safe systems, that should not be an issue.
Decidability: Your reasoning is that AI systems attempt to mirror human cognition, thus incorporating the ability to make mistakes. This is a more philosophical issue — if a human can control a system, then an AI system with the same capabilities should be able to do it too. This assumes that we are able to replicate human behaviour (which we are not). There are AI methods which are deterministic, so would come to the same conclusions given identical environments. So I would say that they would not perform worse than non-AI methods. It partly depends what you want to call AI; the distinction between traditional AI and statistical methods keeps getting blurred at present.

To conclude: No, AI methods should be suitable, as they can also be deterministic. It depends on the actual application and method if they are. And, of course, on what you count as AI.


How to build a Neural Network to approximate the Q-function?

I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
One main reason for this choice: when exploiting the learnt function (i.e. policy is $\arg \max_a Q(s,a)$), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.


Which part of "Perceptrons: An Introduction to Computational Geometry" tells that a perceptron cannot solve the XOR problem?

The section of the book Perceptrons: An Introduction to Computational Geometry (expanded edition, third printing, 1988) that shows the limitations of the perceptron should be 11.8 The Nonseparable Case (p. 181), where the authors write

  There are many reasons for studying the operation of the perceptron learning program when there is no $\mathbf{A}^*$ with the property $\mathbf{A}^* \cdot \mathbf{\Phi} > 0$ for all $\mathbf{\Phi} \in \mathbf{F}$. Some of these are practical reasons. For example, one might want to use the program to test whether such an $\mathbf{A}^*$ exists, or one might wish to make a learning machine of this sort and be worried about the possible effects of feedback errors and other "noise". Other motives are theoretical. One cannot claim to have completely understood the "separable case" without at least some broader knowledge of other cases.

In section 12.1.1 (p. 189), the authors further write

  The PERCEPTRON scheme works perfectly only under the restriction that the data is linearly separable.


In the blog post Building powerful image classification models using very little data, bottleneck features are mentioned. What are the bottleneck features?

It's clearly written in the link you gave the "bottleneck features" from the VGG16 model: the last activation maps before the fully-connected layers.

  Do they change with the architecture that is used?

Sure. The author most likely used a pre-trained model (trained on a large data and now used only as a feature extractor)

  Are they the final output of convolutional layers before the fully-connected layer?

Yes.

  Why are they called so?

Given the input size to VGG, the feature maps of HxW dimensions are getting twice smaller after every max-pool operation. HxW is the smallest on the last convolutional layer.


If deep learning is a black box, then why are companies still investing in it?

Easy answer: utility.

The strength and applicability of "black box" NNs has been regularly validated in the past few years, and business is concerned with results.  (i.e. they don't care how the sausage is made, so long as it gets made.)


Interpretation of a good overfitting score

It doesn't tell you very much, to be honest. It does mean that (assuming your training and validation distributions are similar) your model could get the same results on your validation set should you train on that, but that would still be overfitting.

Really, the only useful thing overfitting tells you is that you don't have enough regularisation.


What are some well-known problems where neural networks don't do very well?

Here's a snippet from an article by Gary Marcus

  In particular, they showed that standard deep learning nets often fall
  apart when confronted with common stimuli rotated in three dimensional
  space into unusual positions, like the top right corner of this
  figure, in which a schoolbus is mistaken for a snowplow:

  .
  .
  .

  Mistaking an overturned schoolbus is
  not just a mistake, it’s a revealing mistake: it that shows not only
  that deep learning systems can get confused, but they are challenged
  in making a fundamental distinction known to all philosophers: the
  distinction between features that are merely contingent associations
  (snow is often present when there are snowplows, but not necessary)
  and features that are inherent properties of the category itself
  (snowplows ought other things being equal have plows, unless eg they
  have been dismantled). We’d already seen similar examples with
  contrived stimuli, like Anish Athalye’s carefully designed, 3-d
  printed foam covered dimensional baseball that was mistaken for an
  espresso

  Alcorn’s results — some from real photos from the natural
  world — should have pushed worry about this sort of anomaly to the top
  of the stack.

Please note that the opinions of the author are his alone and I do not necessarily share all of them with him.

Edit: Some more fun stuff

1) DeepMind's neural network that could play Breakout and Starcraft
saw a dramatic dip in performance when the paddle was moved up by a few pixels.

See: General Game Playing With Schema Networks

While in the latter, it performed well with one race of the character but not on a different map and with different characters.

Source

2)

  AlphaZero searches just 80,000 positions per second in chess and
  40,000 in shogi, compared to 70 million for Stockfish and 35 million
  for elmo.

What the team at Deepmind did was to build a very good search algorithm. A search algorithm that includes the capability to remember facets of previous searches to apply better results to new searches. This is very clever; it undoubtedly has immense value in many areas, but it cannot be considered general intelligence.

See: AlphaZero: How Intuition Demolished Logic (Medium)


What should the value of epsilon be in the Q-learning?

What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?

You are correct, when  ϵ=1 the agent acts randomly. When  ϵ=0, the agent always takes the current greedy actions. Both of these scenarios are not ideal. Always acting greedily will prevent the agent from exploring possibly better parts of the  state space, and instead the agent may get stuck in a local optimum. And always exploring randomly is obviously not ideal as well. Thus, we need to balance between these two. This is often called the balance between exploration and exploitation.

For a typical problem, what is the recommended value for this parameter?

ϵ is a hyper parameter. It is impossible to know in advance what the ideal value is, and it is highly dependent on the problem at hand. There is no general answer to this question.
That being said, the most common values that I have seen typically range between 0.01 and 0.1. But I want to stress, there is no ideal value that works for all problems. A typical strategy is to try several values and see which one works best. For more information, you might want to look up hyper parameter tuning.
Another common practice is slowly decaying epsilon over time (often this is called "annealing" or "simulated annealing"). Depending on the algorithm, decaying epsilon to zero may be a requirement for convergence. In some contexts, an algorithm that decays epsilon over time is called a GLIE algorithm. For example, see this.


Have I understood the loss function from the original U-Net paper correctly?

Yes, $E$ is the cross-entropy function and a direct generalization of the binary case.
For the binary case, probability to belong to the class $1$ is given by a sigmoid function $\sigma(x)$ of the output $x$, and the probability to belong to the class $0$ is $1 - \sigma(x)$.
Therefore the binary crossentropy will give:
$$
-\sum_i (l_i \log \sigma(x) + (1 - l_i)  \log (1 - \sigma(x))
$$
Where the sum is over all samples in the dataset. Because $l_i$ is a binary variable one of these two terms would be zero. The gradient descent forces the model to predict true label with more confidence.
For the multiclass case, now the output is a $K$-vector and the softmax function forces its elements to sum to one:
$$
\sum_{i = 1}^{K} p_i = 0
$$
The true label is one-hot encoded vector, with $1$ on the position of the true label, and $0$ elsewhere. The generalization of the binary case is:
$$
-\sum_i \sum_{j = 1}^{K}(l_{ij} \log \sigma_j(x) + (1 - l_{ij})  \log (1 - \sigma_j(x))
$$
In this case, there will be $K$ non-vanishing contributions.The minization of $E$ forces the classifier to predict the true label more confidently, and all other (to make $\log(1 - \sigma_j(x))$) as small  as possible.


What is the probability of selecting the greedy action in a 0.5-greedy selection method for the 2-armed bandit problem?

I read section 2.2 of Sutton and Barto, and I understand your confusion: the $\epsilon$-greedy algorithm is not defined precisely on page 27-28. Selecting an action randomly "every once in awhile" with probability $\epsilon$ means selecting an action randomly with probability $\epsilon$ at each timestep and selecting an action greedily with probability $1-\epsilon$ at each timestep. This definition is standard and will be clear as you progress through the book and other relevant literature. For reference, the $\epsilon$-greedy algorithm is used in the pseudocode on page 32 of Sutton and Barto.
The key distinction in this problem is that it's asking for the probability that the greedy action is selected, NOT the probability that an action is selected greedily. Specifically, the greedy action can be selected when the agent selects an action randomly because the greedy action is in the action space and the entire action space is sampled uniformly when selecting an action randomly.
Since $\epsilon=0.5$, the agent will select an action greedily 50% of the time, which will 100% of the time be the greedy action. The agent will select an action randomly the other 50% of the time. Since there are two actions in the action space, the greedy action will be selected 50% of the time when the agent selects an action randomly. Therefore, the probability that the greedy action is selected at any single timestep is as follows:
\begin{align}
 &p(\mbox{greedy action}) \\
=\ &p(\mbox{greedy action AND greedy selection}) + p(\mbox{greedy action AND random selection})\\
=\ &p(\mbox{greedy selection}) \cdot p(\mbox{greedy action}\ |\ \mbox{greedy selection}) \\
&\hspace{1em}+ p(\mbox{random selection})\cdot p(\mbox{greedy action}\ |\ \mbox{random selection})\\
=\ &(1-\epsilon) \cdot p(\mbox{greedy action}\ |\ \mbox{greedy selection}) + \epsilon \cdot p(\mbox{greedy action}\ |\ \mbox{random selection})\\
=\ &0.5 \cdot p(\mbox{greedy action}\ |\ \mbox{greedy selection}) + 0.5 \cdot p(\mbox{greedy action}\ |\ \mbox{random selection})\\
=\ &0.5 \cdot 1 + 0.5 \cdot 0.5 \\
=\ &0.5+ 0.25 \\
=\ &0.75.
\end{align}


How to recognise metaphors in texts using NLP/NLU?

This is still a research topic in linguistics. A quick google search brings up a couple of papers that might be useful:

Identifying Metaphor Hierarchies in a Corpus Analysis of Finance Articles
Metaphor Identification in Large Texts Corpora

However, you probably won't get an off-the-shelf tool that recognises metaphors for you.
To add more details, the problem with metaphors is that you cannot detect them by surface structure alone. Any sentence could (in theory) be a metaphor. This is different from a simile, which can usually be spotted easily through the word like, as in she runs like the wind. Obviously, like on its own is not sufficient, but it's a good starting point to identify possible candidates.
However, his words cut deeper than a knife is -- on the surface -- a normal sentence. Only the semantic incongruence between words as the subject and cut as the main verb creates a clash. In order to detect this automatically, you need to identify possible semantic features of the verbal roles and look for violations of the expected pattern.
The verb cut would generally expect an animate object, preferably human, or an instrument with a blade (the knife cuts through the butter) as its actor or subject. But it also can include (water)ways: the canal cuts through the landscape, the road cuts through the field. The more closely you look, the more exceptions/extensions you will find for your initial assumption.
And every extension/exception will water down the accuracy of your metaphor detection algorithm.
The second example is similar: Life is a journey. You could perhaps use a thesaurus and see what the hyperonyms of life are. Then you could do the same with journey, and see if they are compatible. A car is a vehicle is not a metaphor, because vehicle is a hyperonym of car. But journey is not a hyperonym of life, so could be a metaphor. But I would think that this is still very tricky to get right. In this case, the absence of a determiner might be a hint, as it's not a life is a journey -- you might restrict yourself to bare nouns for this type of metaphor. But this is also not a firm rule.
In short, it is a hard problem, as you need to look at the meaning, rather than just the structure or word choice. And meaning is not easy to deal with in NLP, despite decades of work on it.


Can reinforcement learning be utilized for creating a simulation?

Can the Q-learning algorithm be utilized to build the simulation itself?

Only in the presence of a meta-environment, or meta-simulation where the goals of creating the original simulation are encoded in the states, available actions and rewards.

A special case of this might be in model-learning planning algorithms where there exists a "real" environment to refer to, and the agent benefits from exploring it and constructing a statistical model that it can then use to create an approximate simulation of the outcomes of sequences of actions. The Dyna-Q algorithm, which is a simple extension of Q-learning, is an example of this kind of model building approach. The simulation is very basic - it simply replays previous relevant experience. But you could consider this as an example of the agent constructing a simulation.

Getting an agent to act like a researcher and actually design and/or code a simulation from scratch would require a different kind of meta-environment. It is theoretically possible, but likely very hard to implement in general way - even figuring out the reward scheme to express the goals of such an agent could be a challenge. I'm not aware of any examples, but entirely possible someone has attempted this kind of meta agent, because it is an interesting idea.

Possibly the simplest example would be a gridworld meta-environment where a "designer" agent could select the layout of objects in the maze, with the goal of making a second "explorer" agent's task progressivley more difficult. The designer would be "creating the simulation" only in a very abstract way though, by setting easy-to-manage parameters of the environment, not writing low level code.

There is not much difference between the approach above and having two opposing agents playing a game. It is different from a turn-based game like chess in that each agent would complete a full episode, and then be rewarded by the outcome at the end of the combined two episodes. There are some similarities to GANs for image generation.


Examples of time-varying graph-structured data in real world

You can take a look at traffic data for example if you follow
link1, link2 you can find 3 publicly available traffic datasets which are already preprocessed. You cold also look at air quality datasets offered by the government link3


What is wrong with the idea that the AI will be capable of omniscience?

I quite like your outlook, and without getting into the details of how a "singularity" may be effected which is covered in numerous other questions, or how consciousness and "omniscience" come into play because consciousness and omniscience are not requirements, I will instead direct you to two key philosophers:

Phillip K. Dick, for whom the central theme in his famous 1968 book on AI is empathy.  (If you haven't read it, I'm not posting a spoiler, but will only say the plot is driven by the concept of Evolutionary Game Theory which was formalized just 5 years later.)
John Nash, and in particular, the concept of the Nash Equilibrium. (Nash could be said to have mathematically demonstrated that being a "douchebag" is not an optimal strategy.  His proof can be used to explain why nuclear détente actually worked, which was counter to the expectation of Von Neumann.)

So when people go nuts, focusing on the "Skynet" mythos under which machines rise up to destroy us, I have to wonder if they're simply not as smart as Nash or as profound as Dick, which might explain their lack of emphasis on what can be called the "Electric Sheep" paradigm.


Why do we need a model of the environment in Dyna?

Unlike algorithms presented in other chapters of Sutton and Barto, Dyna is a planning algorithm. That means that it makes decisions online, in a real environment, that attempt to be as optimal as possible given some constraints such as current knowledge and time available to compute between time steps. This differs from learning-only online algorithms which typically take a small step towards optimality on each piece of new experience as it happens.

A planning algorithm can only do its job well if it is allowed to "look ahead" at the consequences of its behaviour whilst learning online. In fact, this is the definition of planning - to choose an action based on reasoning about consequences of that action.

For an algorithm to look ahead before taking an action, it needs a model of how the environment will respond to that action. That model does not need to be coded up directly - e.g. you don't necessarily need to write a physics engine to predict the real world (although a basic one might be a good prior or pre-training step). Instead it can be a learned model, and typically in e.g. Dyna-Q, that is what you use.

There is a strong relation between Dyna-Q, and regular Q-learning with experience replay. In the most basic forms, they are essentially the same algorithm with a different framing. However, you can take the planning ideas further e.g. focus improvements around the currently experienced state and paths to a goal state in Dyna-Q, perhaps making it closer to MCTS conceptually.

  Wouldn't it be more helpful to use real env instead of fake one?

Most real environments do not let you take actions, see the consequences and then rewind in order to re-try. Essentially that is what planning algorithms are making up for - they try to predict consequences. This is important when mistakes made during training have real consequences, for example for a physical robot navigating an environment where there might be a possibility of a fall or collision that damaged something. Whilst online learning algorithms such as SARSA will also help with this in different ways (in SARSA by changing policy to allow for exploratory moves), typically Q-learning will be weaker than Dyna-Q when it comes to learning quickly from mistakes. With the usual caveat: Much still depends on the specific problem and choices of hyperparameters.


In RL as probabilistic inference, why do we take a probability to be $\exp(r(s_t, a_t))$?

After doing some further reading, it turns out that negative rewards are an assumption for this distribution to hold. However, the author notes that as long as you don't receive a reward of infinity for any action then it is possible to re-scale your rewards by subtracting the maximum value of your potential rewards so that they are always negative.


Is there any difference between ConvNet and CNN?

Both terms just mean convolutional neural network. I don't believe there is any particular reason to choose one over the other: ConvNet is slightly easier to say out loud and CNN is slightly shorter to write, but there is absolutely no difference in meaning.
For some contrasting examples in the literature, the EfficientNet paper chooses the term ConvNet and this paper on AlexNet chooses CNN. They are talking about the same sorts of neural networks though!


Is it possible to use the GPT-2 model for time-series data prediction?

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.


Which libraries can be used for image caption generation?

Image Caption Generation is an interesting problem to work on. I think your question was to know if there are any open-source libraries with built-in functions for Image Captioning. You can build Image Caption Generation models using Frameworks like Tensorflow, PyTorch, and Trax.
I'd also recommend you to read the following papers:

Show and Tell: A Neural Image Caption Generator. Link
Transfer learning from language models to image caption generators: Better models may not transfer better. Link
Image Captioning with Unseen Objects. Link

Also, here are a couple of blog posts you can read:

How to Develop a Deep Learning Photo Caption Generator from Scratch
Learn to Build Image Caption Generator with CNN & LSTM
Image Captioning with Keras


Why is tf.abs non-differentiable in Tensorflow?

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.
In a way this is compatible with the Python philosophy that explicit is better than implicit. If you mean to use $\mathrm{ReLU}$, it's probably best to use tf.nn.relu if it is suitable for your use case.
[1] Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML'10 (2010). URL.


I think this website is the best c-based neural net code for beginners.


Unable to 'learn' a rotational angle by parametrising the angle as a neural network layer

There are two basic problems with your code:

The functions sin(t) and cos(t) (both in numpy and tensorflow) take radians as inputs. Seeing Constant(90) in your code, and the learning rate of 1. I'm guessing that you assume that it is in degrees - that's incorrect.

In your training data y_ta is not a rotation of y_in:
y_in = np.array([np.sin(t), np.cos(t)]).T
y_ta = np.array([np.cos(t), np.sin(t)]).T

It is a reflection about y=x diagonal. No wonder it fails to find an appropriate rotation.
I just had to change y_ta to:
  y_ta = np.array([-np.cos(t), np.sin(t)]).T

And train with more sensible learning rate:
  model.compile(tf.keras.optimizers.SGD(lr=0.1), loss='MSE')
  model.fit(y_in, y_ta, epochs=10)

To get the angle (which I then convert to degrees):
  (180 * model.layers[0].kernel[0,0].numpy() / np.pi) % 360
  > 90.00187079311581


Why are GRU and LSTM better than standard RNNs?

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual degradation of the values passed from step to step. LSTM and GRU networks make this memory control possible with memory blocks and structures called "gates" that pass or reset values as appropriate.


How does one start learning artificial intelligence?

What is artificial intelligence?

Artificial Intelligence is a very broad field and it covers many and very deep areas of computer science, mathematics, hardware design, and even biology and psychology.

What math do you need?

As for math, I think calculus, statistics, and optimization are the most important topics, but learning as much math as you can won't hurt.

Free resources for beginners

There are many good free introductory resources about AI for beginners.
I highly recommend starting with this website AI Playbook, which is intended to help newcomers (both non-technical and technical) begin exploring what's possible with AI. They also published two videos about the general concepts of AI, you can find them on Youtube AI, Deep Learning, and Machine Learning: A Primer and The Promise of AI.

What is your goal?

Once you have a clear understanding of the basic AI terms and approaches, you have to figure out what your goals are. What kind of AI software do you want to develop? What industries are you interested in? What are your chances to get involved in projects of big companies? It's easier to pick up the right tools when you know exactly what you want to achieve.

AI, machine learning and deep learning

For most newcomers to AI the most interesting area is deep learning.
Just to make it clear, there are many areas of AI outside of machine learning and there are many areas of machine learning outside of deep learning.

  Artificial Intelligence > Machine Learning > Deep Learning

Most of the recent developments and hyped news are about DL.

Neural networks

If you got interested in deep learning too, you have to start with learning about the concepts of artificial neural networks. Fortunately, it's not too difficult to understand the basics and there are lots of tutorials, code examples and free learning resources on the web and there are many open-source frameworks to start experimenting with.

Frameworks and libraries

Python

The most popular such deep learning framework is TensorFlow. It's backed by Google. Love it or hate it, it's a Python-based framework. There are many other Python-based frameworks and libraries, as well. Scikit-learn, theano, Keras are frequently mentioned in tutorials too. (A tip: if you use Windows you can download WinPython that includes all of these frameworks.)

Java

As for Java frameworks, unfortunately, there are not so many options. The most prominent Java framework for DL is Deeplearning4j. It's developed by a small company and its user base is much smaller than the crowd around TensorFlow. There are fewer projects and tutorials for this framework. However, industry specialists say Java-based frameworks eventually integrate better with Java-based big data solutions and they may provide a higher level of portability and easier product deployment. Just a sidenote: NASA's Jet Propulsion Laboratory used Deeplearning4j for many projects.

Resources

If you decide to go with the flow and want to start learning more about TensorFlow, I recommend you check out the YouTube channels of DeepLearning.TV, sentdex and Siraj Raval. They have nice tutorials and some cool demos. And if you decide to take a deeper dive, you can sign up for an online course at Udacity or Coursera.

Other programming languages

It also may be interesting to you to know that there are other deep learning frameworks for the Java Virtual Machine with alternative languages, for example, Clojure. (Clojure is a dialect of LISP and it was invented by John McCarthy, the same computer scientist who coined the term "artificial intelligence". In other words, there are more modern and popular programming languages and tools, but it's still possible, and kinda cool, to use the language for AI that was originally designed for AI. ThinkTopic in Boulder and Freiheit in Hamburg are two companies that use Clojure for AI projects. And if you want to see something awesome to get inspiration to use Clojure in AI and robotics, I recommend you to check out the YouTube video OSCON 2013: Carin Meier, The Joy of Flying Robots with Clojure. (Mentioning Clojure in this answer was just an example to show you there is life outside of the bubble of Python-based AI frameworks.)


What are the differences in scope between statistical AI and classical AI?

Statistical AI, arising from machine learning, tends to be more concerned with "inductive" thought: given a set of patterns, induce the trend.
Classical AI is the branch of artificial intelligence research that concerns itself with attempting to explicitly represent human knowledge in a declarative form i.e given a set of constraints, deduce a conclusion.
 Another difference is that C++ tends to be a favorite language for statistical AI while LISP dominates in classical AI.

A system can't be more intelligent without displaying properties of both inductive and deductive thought. This lends many to believe that in the end, there will be some kind of synthesis of statistical and classical AI.


Feed forward neural network using numpy for IRIS dataset

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset and then he creates an ANN architecture for it. If you want to build an ANN based on Iris dataset, you should think and create an architecture on paper maybe before coding. You should understand the Iris dataset first, I mean you should understand what is going to be your input shapes, and also the shape of weights are related to inputs' shape. After understanding these you should decide the number of layer of your Neural Network. And that tutorial that you followed is explaining all of these clearly. Do not skip and understand everything steps by step.

If you want a build ANN architecture by yourself, you must understand every mathematics behind that. (forward and backward propagation, loss and cost, gradient descent) If you do not want to deal with all of these maths, you can use a library such as Keras, then you can create an ANN easily.

See this as a source, https://ml-cheatsheet.readthedocs.io/en/latest/
And you see my Kaggle kernel, I create this network with details and only using numpy. Also I used Keras on another section. You can check; https://www.kaggle.com/erdemuysal/gender-recognition-with-lr-and-ann


Deep Reinforcement Learning: Rewards suddenly dip down

Without digging into your diagnostics more deeply, on it's face this seem like a local optima issue. Assuming you are optimizing via GD, there are many local optima that a network or agent in this case can converge to and stay at which will cause symptoms like seeing above.

With that being said, assuming this is our issue, here are some things you can try:

Regularization, try adding dropout or L2 and see how that affects convergence and learning.
Adjust network architecture, number of layers, nodes, etc.
Try a different type of RL(Q learning for example), this will be dependent on your problem of course.
Adjust starting seed. Assuming you have a static seed used for weight initialization, you will always converge to the same solution. It could be as simple as adjusting the seed value.

If all these steps fail, you likely have a deeper issue at work, and I would suggest coming back after with some additional detail if this does not succeed.


Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus.
If you have a function $$f:\mathbb{R}\rightarrow\mathbb{R}$$
the derivative is a function on its own and you have $$f':\mathbb{R}\rightarrow\mathbb{R}.$$
If you increase the dimension of the input space have
$$f:\mathbb{R}^n\rightarrow\mathbb{R}.$$
The "derivative" in this case is called gradient and it is a vector collecting the $n$ partial derivatives of $f$. The input space of the gradient function is $\mathbb{R}^n$ (the same as for $f$), but the output is the collection of the $n$ derivatives, so the output space is also $\mathbb{R}^n$. In other words
$$\nabla f:\mathbb{R}^n\rightarrow\mathbb{R}^n,$$
which makes sense as for each point $x$ of the input space you get a vector ($\nabla f(x)$) as output.
So far so good, but what happens if you consider a function that takes a vector as input and spits out a vector as output, i.e.
$$f:\mathbb{R}^n\rightarrow\mathbb{R}^m?$$
How to compute the equivalent of the derivative? (This is the softmax case, where you have a vector as input and a vector as output.)
You can reduce this case to the previous case considering $f=(f_1, \dots, f_m)$, where $f_i:\mathbb{R}^n\rightarrow\mathbb{R}.$ Now for each $f_i$ you can compute the gradient $\nabla f_i$ and end up with $m$ gradients. When you evaluate them at a point $x\in\mathbb{R}^n$ you get $m$ $n-$dimensional vectors. These vector can be collected in a matrix, which is the Jacobian; formally
$$Jf:\mathbb{R}^n\rightarrow\mathbb{R}^{m\times n}.$$
Finally, to answer your question, you get a Jacobian "instead"  of a gradient (they all represent the same concept) because the output of the softmax is not a single number but a vector.
By the way the sigmoid and relu are functions with one dimensional input and output, so they don't really have a gradient but a derivative. The trick is that people write $\sigma(W)$, where $W$ is vector or a matrix, but they mean that $\sigma$ is applied component-wise, as a function from $\mathbb{R}$ to $\mathbb{R}$ (I know, it's confusing).
(I know, I kind of skipped your other question, but this answer was already long and I think you can convince yourself that the dimensions match correctly (with the Jacobian) for the update rule to work. If not I'll edit)


Neural networks are used to visualize high dimensional data through the use of autoencoding. It's similar to Principal Component Analysis and is regarded to perform better then PCA. Autoencoding will take your data and convert it to a 2 or 3-dimensional representation. Since you have an array of data you might want to use LSTM. You will have to make sure you reset the LSTM states when you perform the reverse pass. You will probably have trouble interpreting the autoencoded form of the data though, so that will be a task of its own.


How are IOUs for ground truth boxes in YOLO calculated?

My assumption was correct: the ground truth bounding box is aligned with an anchor box such that they share the same center

In other words, only the widths and heights are used to calculate the ground truth IOU.


I worked at a start-up doing this exact task for two years. There were several developers involved in it, and it took several years; by the end (with a lot of manual input, eg categorising ingredients in an ontology, converting measurements, and mapping ingredients to shopping items from a supermarket inventory) we had something that worked reasonably well.

You rightly recognise that it is not trivial... The first step would be to identify the list of ingredients from the recipe (assuming you have scraped it from a website), and turn it into a list; beware that sometimes there might be more than one ingredient in a line. Then you need to analyse the entry: we had a grammar that distinguished between quantities, attributes (fresh), processing instructions (peeled) and a number of other things. The main item would of course be the actual ingredient. Then we would look this up in a supermarket inventory and map it to the closest item (but I don't think this is part of your remit).

We tried different ways of analysing the recipe lines, and any grammar/parser is fine, as long as it can deal with partial analyses, as you cannot always rely on lines being complete and well-formed, and there is always scope for additional comments you want to ignore ("I only use the freshest lemons directly imported from Tuscany", "get your fishmonger to take the bones out"). In the simplest case you could have a list of items (lemon) with inflections etc, that you try to match in your recipe lines. If you don't care whether it's big lemons or not, don't bother using a full-blown grammar. And even then, you could just look at a list of adjectives preceding an ingredient to identify any modifiers. We used a grammar based on recursive transition networks (RTNs, where the parser was complex but the grammars easy to handle) and one that was a simple phrase structure one (where the parser was simpler, but the grammar was much more complex). Both worked fine.

As far as I am aware there is not really much written about this kind of work. The main advice I can give you is to look at as many recipes as possible to get an idea for what you will encounter, and cut corners wherever possible -- it's a big job to do it perfectly, and you can get very far by using simple methods.

We also transferred it into different languages (Polish too, which I assume you're working with?). We didn't find that inflections were that big an issue. You just have a list of all inflected forms linking to the lemma. As you have a limited domain, that is a perfectly feasible approach. Just have a master list of ingredients, with all alternate forms (courgette and zucchini are the same thing, for example).


What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?

You should start with the general definition of Reinforcement Learning problem. And what Markov Decision Process is.
DQN, A3C, PPO and REINFORCE are algorithms for solving reinforcement learning problems. These algorithms have their strengths and weaknesses depending on the details of the underlying problem.
Multi-Armed Bandit is not even an algorithm - it is a subclass of reinforcement learning problems, where your environment (usually) doesn't have any state transitions and your actions are just a single choice from (usually) fixed and finite set of choices.
Multi-Armed Bandit is used as an introductory problem to reinforcement learning, because it illustrates some basic concepts in the field: exploration-exploitation tradeoff, policy, target an estimate, learning rate and gradient optimization. All these concepts are basic vocabulary in RL. I recommend reading (and, very importantly, doing all the exercises) the Sutton and Barto book chapter two to get familiarized with it.
Edit: since the answer got popular, I'll address the comments and the question edit.
Being a special simplified subset of Markov Decision Processes, Multi Armed Bandit problems allow deeper theoretical understanding. For example, (as per @NeilSlater comment) the optimal policy would be to always go for the best arm. So it makes sense to introduce "regret" $\rho$ - the difference between a potential optimal reward and the actual collected reward by agent following your strategy:
$$\rho(T) = \mathbb{E}\left[T\mu^* -\sum_{t=1}^T\mu(a_t)\right]$$
One can then study asymptotic behavior of this regret as a function of $T$ and devise strategies with different asymptotic properties. As you can see, the reward here is not discounted ($\gamma=1$) - we usually can study the behavior of it as a function of $T$ without this regularization.
Although, there is one famous result that uses discounted rewards - the Gittins index policy (note, though, that they use $\beta$ instead of $\gamma$ to denote the factor).


Will CNNs kill CAPTCHAs or can they survive in an evolved form?

I think that captchas can be substituted or augmented by questions which require understanding of context, for example:

How many hands keep the rose?

Or

Which number of books under the bench and in hands?

These examples are simple, but they can be improved, for example, by generation of images with random  numbers of objects and their positions

It looks like it will be hard to use CNN to hack such kind of captchas, especially if we have a large amount of different objects and their combinations

In addition, it can be improved by adding more complicated logical questions to compositions on pictures


What are possible ways to combat overfitting or improve the test accuracy in my case?

There are a few issues you need to address first.

Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A simple way of doing this could be to divide each value by the maximum value for that input. This will ensure they are between 0 and 1, or you could divide by the average so you get approximately 1. Whatever the case, you need to do something about decreasing the magnitude of your inputs (also make sure your are consistently scaling your data, especially during validation and training, otherwise your data will likely be misinterpreted by your model)
Optional: Augment your data. You have a very small dataset to work with here, if there's any way you can augment this to make it larger, you will likely get much better performance on validation and testing sets. Augmenting is just creating new data that is still representative of the task (as a visual example, you might augment your image dataset by flipping some images and including those in training as well as the unflipped versions)
Split your data into Training, Validation and Testing sets. Training is used for what you expect - training. Validation is used to validate your model on unseen data. The reason for the testing set is obviously you are going to try your hardest to tweak and modify the model so it performs best on your validation set, but this incurs a kind of bias as you are only tweaking for this specific set of data, which is not completely representative of the real world. So try your hardest to perform best on the validation set by changing model hyper-parameters, but confirm it's real use on the testing set. As a recommended split for your entire data set, I have had success with 10% testing, 20% validation and 70% training, but you might be better off with a different split.
Test different models. Train your network for a small amount of time (say only 10 or so epochs) and see how it performs based on the validation set. Keep in mind because your dataset is particularly small (only ~300 samples in your training set) you don't need to train for many epochs at all to get a good representation of how your model will perform (an epoch being a complete loop over all training data). Make sure when you're doing this you're training in batches.
Try different regularization methods. One of the primary ones being good initialization. Make sure you're initializing your parameters with something like Xavier initialization of similar. Other suggestions include: dropout (though you said you already tried this, maybe try different values for dropout), weight decay and batch normalization layers. To be honest though, I think one of the 4 steps above will solve this issue, so you likely won't need to do any of these.

I may have made some points for things you have already done as there was a lot of code for me to look over in your source. But the biggest issue I noticed was that your data wasn't normalised at all and that usually creates many issues, so definitely try that first if you haven't already.


How is it possible for an AI to interact with someone without a computer screen?

There are many communication methods that could be used by an artificial intelligence. Artificial intelligence can be integrated to various things including robots, phones, IoT and many others. Primary ways of human communications are  either visual or auditory, therefore an natural way for it to communicate with a human is through voice, text, images and videos. The output does not have to be limited to screens but can be anything from refrigerators to speakers.

Hope this helped.


What are examples of daily life applications that use simulated annealing?

Examples of simulated annealing in the 2010s

These are a few examples.

Optimised simulated annealing for Ising spin glasses, 2015, S.V. Isakov et. al.
A parallel simulated annealing method for the vehicle routing problem with simultaneous pickup–delivery and time windows, 2014, Chao Wang et. al.
Feature-based tuning of simulated annealing applied to the curriculum-based course timetabling problem, 2015, Ruggero Bellio et. al.

Analysis of Quote in the Question

The quote in the question is from page 125 of Artificial Intelligence: A Modern Approach, Stuart Russell and Peter Norvig, 2010, 3rd Edition.

The quote begins with the statement, "A hill-climbing algorithm that never makes 'downhill' moves toward states with lower value (or higher cost) is guaranteed to be incomplete," which is not strictly true. The stated guarantee has at least two important exceptions.

The class of hills being climbed have only one peak
The search is executed using parallel computing architecture scaled such that search granularity includes no more than one peak per processing unit

Describing simulated annealing using the analogy of combining inverted hill climbing and random walk, as Stuart and Norvig explains it, is not as clear and technically accurate as the original analogy. It is often better to find original sources of ideas and read them than to read simplified explanations. The original ideas may be just as easy or easier to understand than the textbook paraphrases (which is why the best textbooks simply quote the originating authors).

Source of the Term

The original algorithm termed simulated annealing is introduced in Optimization by Simulated Annealing, Kirkpatrick et. al.1, which may not qualify as one one explicitly employed by AI researchers or practitioners on a daily basis. Simulating the atomic motion of metals during annealing is the first of several approaches to injecting pseudo-random perturbations into convergence mechanisms. An approximation of a ${\chi}^2$ distribution with $k$ degrees of freedom is a more progressive injection source.

These approaches share the common goal of avoiding the mistaking of a local minimum in loss surface as its global minimum. Such mistaken conclusion causes a failure to achieve an optimal learning state according to the criteria the loss function represents.

As can be seen in the below excerpts, the original article is a good read and sets a foundation of understanding. Such a foundation will assist in understanding all methods used avoid local minima using stochastic injection.

  Iterative improvement consists of a search in this coordinate space for
  rearrangement steps which lead downhill. Since this search usually gets stuck in a local but not a global optimum, it is customary to carry out the process several times, starting from different randomly generated configurations, and save the best result.

  ...

  Metropolis et al 2, in the earliest days of scientific computing, introduced a simple algorithm that can be used to provide an efficient simulation of a collection of atoms in equilibrium at a given temperature.

  ...

  Random numbers uniformly distributed in the interval (0,l) are a convenient means of implementing the random part of the algorithm.

  ...

  Using the cost function in place of the energy and defining configurations by a set of parameters {$x_i$}, it is straightforward with the Metropolis procedure to generate a population of configurations of a given optimization problem at some effective temperature. This temperature is simply a control parameter in the same units as the cost function. The simulated annealing process consists of first "melting" the system being optimized at a high effective temperature, then lowering the temperature by slow stages until the system "freezes" and no further changes occur.

  ...

  Simulated annealing with Z-moves improved the random routing by 57 percent, averaging results for both x and y links.

References

[1] Optimization by Simulated Annealing, Science S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi, Science New Series, Vol. 220, No. 4598, May 1983, pp. 671-680

[2] N. Metropolis, A. Rosenbluth, M. Rosenbluth., A. Teller. E. Teller, J. Chem. Phys. 21. 1087 110511


Which AI algorithm is great for mapping between two XML files

The standard tool to work with XML files is XSLT. You may not need AI to solve this problem. But.. you have to learn how to program with XSLT ;)
On Windows you can use MSXML, if you work from C++ - msxsl.exe, C# has internal supoort for XSLT. That is what I know about. There are also non-MS tools.


How important is the choice of the initial state?

The initial state can vary during in both training and use, and how you decide to do this makes very little difference to Q-learning. The important factor is whether all state/action pairs relevant to optimal behaviour can be reached. As there is already randomness in any exploring policy, and in many environments as part of state transitions and reward functions, adding some more at the start is not an issue.

More formally, you can take any existing environment with states $S_1, S_2, S_3 ... S_n$ plus defined actions and rewards. Add a special fixed start state $S_{0}$ with one special action $A_{0}$ allowed. Make the state transition matrix following that action any distribution you like over the other states, and reward $0$. It is clearly a valid MDP, and is identical to your original MDP in terms of value and policy functions for states $S_1, S_2, S_3 ... S_n$. For all intents and purposes to the agent (which gets no meaningful policy choice in $S_0$), it starts in the original MDP in some randomly chosen state.

That is as long as the variations are not made to deliberately exclude some of the state space during training that it would need later, or otherwise used to "attack" the agent and prevent it learning. Q-learning proofs of convergence assume that all state/action pairs are reachable an infinite number of times in the limit of infinite training time. Of course in practice this is never achieved, but clearly if you excluded some important state from ever being seen at the start by choosing to start in a way that it is never reachable, then the agent would never learn about it.

You can use a highly variable start state for training to your advantage. It is a form of exploration, and truly exploring starts (that could start from any state/action pair, so you would want to randomise the first action choice) allow you to skip any further exploration and still guarantee convergence to an optimal policy - i.e. you could learn Q values on-policy using a deterministic policy. This will not necessarily make things converge faster, but does go to demonstrate that random start states are not a barrier to RL.

There are a couple of minor/partial exceptions to "do what you like":

If the environment is otherwise highly deterministic, adding a random start may increase the difficulty in optimising it, as more states will be reachable, even with an optimal policy, therefore the state space over which optimal behaviour needs to be discovered may be larger and take longer to learn.
Policy Gradient methods, like REINFORCE, optimise the policy towards maximising return from a known distribution of start states. So you can vary start state, but you should really stick to a standard start procedure, including a fixed distribution of allowed start states.
Similarly, if you want to report a standard score for your agent, it is common to quote the expected return from the start states. Again that doesn't mean you have a fixed start state, but you should use a fixed distribution of start states to assess your agent.

Many standard environments, like Open AI's Gym environments CartPole and LunarLander, include some amount of randomness in the start state, so that the agent has to solve a more general problem than just generating a fixed sequence of actions that always works from the beginning.