Is the state transition matrix known to the agents in a Markov decision processes?

In reinforcement learning (RL), there are some agents that need to know the state transition probabilities, and other agents that do not need to know. In addition, some agents may need to be able to sample the results of taking an action somehow, but do not strictly need to have access to the probability matrix. This might be the case if the agent is allowed to backtrack for instance, or to query some other systems that simulates the target environment.
Any agent that needs to have access to the state transition matrix, or look-ahead samples of the environment is called model-based. The model in this case can either be a distribution model i.e. the state transition matrix, or it can be a sampling model that simulates the outcome from a given state/action combination.
The state transition function $p(r, s'|s, a)$ which returns the probability of observing reward $r$ and next state $s'$ given the start state $s$ and action $a$, is another way to express the distribution model. It often maps simply to the state transition matrix, but can be a more complete description of the model.
One example model-based approach is Value Iteration, and that requires access to the full distribution model in order to process value update steps. Also, any reinforcement learning that involves planning must use some kind of model. MCTS, as used in AlphaGo, uses a sampling model for instance.
Many RL approaches are model-free. They do not require access to a model. They work by sampling from the environment, and over time learn the impact on expected results due to behaviour of the unknown state transition function. Example methods that do this are Monte Carlo Control, SARSA, Q learning, REINFORCE.
It is possible to combine model-free and model-based methods by using observations to build an approximate model of the environment, and using it in some form of planning. Dyna-Q is an approach which does this by simply remembering past transitions and re-using them in the background to refine its value estimates. Arguably, the experience replay table in DQN is a similar form of background planning (the algorithm is essentially the same). However, more sophisticated model-learning and reuse is not generally as successful, and is not seen commonly in practice. See How can we estimate the transition model and reward function?
In general, model-based methods on the same environment can learn faster than model-free methods, since they start with more information that they do not need to learn. However, it is quite common need to learn without having an accurate model available, so there is lots of interest in model-free learning. Sometimes an accurate model is possible in theory, but it would be more work to calculate predictions from the model than to work statistically from the observations.


The 2nd method would make sense only if your object is at the same position in all test images. You would have such situation if you operated on crops located by a separate object detection algorithm. This happens to be the case in facial key-point detection.

The 1st method would be much more robust to various object poses since it is translation covariant by design. If a keypoint is detected at location A, it will be equally well detected at any other position with the same set of parameters.


What is the best approach to build a self-learning AI chatbot?

The former: build and train a model first, and then think about the user interface.

Effectively, a chatbot is a user interface to your model. If you run it 'off-line' on input text and it works, then you have achieved your goal without the added complexity of driving a conversation (which is harder than one might think).

Also, building an 'abstract' chatbot devoid of any content is going to be harder. What should that chatbot talk about?

From my own experience (I work in conversational AI), you might not succeed in building a decent chatbot, especially as a novice. But you might be able to train a model to identify diseases from textual input. So if you do that first, you have at least got something! Especially if that is your main reason for the project in the first place.


Can PDDL be utilized for action recognition?

I don't know of any work on this with respect to PDDL, but this is very similar to a conceptual dependency application called SAM (Script Applier Mechanism). Conceptual Dependency (CD) models actions using a number of primitives (which could be seen as equivalent to PDDL primitive actions):  PTRANS for physical transfer, PROPEL for application of a physical force to an object, GRASP for grasping an object, etc. Their number varies around 12 or so, depending on the version of CD.

Stories are described by a sequence of primitive acts, which have slots for actor, object, etc. They are supposed to enable a program to draw inferences about what happens. A common problem when trying to understand stories is that often common knowledge about a situation is omitted; the standard example here is going to a restaurant. It is usually assumed that the listener/reader knows what commonly happens when the protagonist enters a restaurant, so that when they leave without paying this is recognised as something unusual.

The approach used to solve this problem is to encode such knowledge in scripts, sequences of primitive acts. When triggered, eg by "Manuel went to a restaurant", this script is retrieved, and the following actions are looked for in the script. Anything that is recognised is used to fill gaps in the story, eg sitting down at a table, or looking at a menu. This was the task of the SAM program.

Basically you have a sequence of primitive actions, and you try to recognise a more abstract event "going to a restaurant" from that. Obviously you'd need to have a script to recognise, but one could presumably use this to derive a sequence of more generalised events, such as "retrieving an object from a high place", or "standing on top on another object".

The theory of using scripts, plans, and goals to describe human reasoning is detailed in Schank, Roger; Abelson, Robert P. (1977). Scripts, plans, goals and understanding: An inquiry into human knowledge structures. New Jersey: Erlbaum. ISBN 0-470-99033-3.


How can I use a trained CNN to predict a new image label?

As the question has been left unanswered, for future readers of the questions:

The documentation you link gives the answer to your question.

Given the fact you have a pre-trained model as you say:

YPred = classify(netTransfer,testDigitData);

where netTansfer is the pre-trained model and testDigitData is your test image that you want to predict the label.

Your image(s) need to be loaded following the instruction also given in the documentation:

digitDatasetPath = fullfile(matlabroot,'toolbox','nnet','nndemos', ...
    'nndatasets','DigitDataset');

digitData = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

(digitData will become testDigitData if the images that you load are for testing purpose only, which seems to be your case)


What happens when the agent faces a state that never before encountered?

Having too many states to actually visit is a common problem in RL. This is exactly why we often use function approximation. If you replace your q table with a good function approximator such as a neural network, it should be able to generelize well to states it has not yet encountered.
If you do not use a function approximator but stick with a table, the agent will have no idea what to do when it encounters a new state. For more information,  see Reinforcement Learning  by Sutton and Barto, chapter 9.


Examples of single player games that use modern ML techniques in the AI?

There is Google Research Football, which is an open-source platform to develop reinforcement learning algorithms to play a game similar to FIFA or PES, although the football simulation is not as realistic as the current versions of FIFA or PES. You can play this game against different RL agents (e.g. DQN or IMPALA) and, of course, you can even develop your own RL agents and play against them. Here is a video that illustrates the environment. Here is the code and instructions to use it.
As far as I know, there isn't yet an AI that plays simulated football at a human-level (i.e. as good as the best human players). For example, I can regularly (although not always) beat the legendary level-AI at FIFA, but I also don't know the details about this AI (which could also be rule-based).


How can object types be differentiated in the input of a neural network?

Neural networks learn. That's what they are for. For your task there are two sensible scenarios:

You have a fixed reaction for danger and a fixed reaction for food and you only have to learn how to distinguish between them. In that case you basically try to classify the situation to trigger the right fixed response and this classification would be learned by backpropagation.
You directly learn to act for a given situation. In that case you can either use a genetic algorithm or you use reinforcement learning with backpropagation.

I would recommend using a genetic algorithm, because it is significantly easier and also makes sense in this situation. You would randomly initialise your network, let it run around in the environment and remember how much food it ate and how often or how quickly it died. Then you would randomly change the weights of your network and do the same thing again. If it did better this time around you would proceed to use the new weights otherwise you go back to the old weights and try a different random change.
By selecting successful random changes it would over time learn to avoid danger and seek out food.

Edit: To my mind you have a fundamental misunderstanding how perception works. If you see a lion and a cake, do those trigger different kinds of cells on your retina? No! All nerve cells are used to detect all kinds of objects! The classification, i.e. whether you are seeing a lion or a cake happens in the neural network i.e. in the higher regions of your visual cortex, far removed from the initial nerve activation. Your lion might be yellow and your cake might be yellow, only if you analyse the high level structure of your nerve inputs can you decide what you are seeing. That is the task of a neural network. And that high level structure analysis is what a neural network learns.

What seems to confuse you is the example you linked. In that example this very sparse distance measuring is enough to differentiate between walls and boosters in your high level structure analysis, because the different points of the walls that you sample have a certain relative position that you can analyse and conclude that they constitute the wall.

In your scenario very sparse distance measuring will not help you obviously. The distance of an object doesn't tell you whether it's a lion or a cake. Distance and color would be a solution to that. Or, more realistically, you have different shapes and much tighter distance sampling, and high level analysis can work out the shape from a couple of closely measured distances.


What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.

Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.

So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.


What is the difference between the ant system and the max-min ant system?

Let's start by making clear that both AS and MMAS use only global pheromone update. Now, the MMAS has two main differences regarding AS:

In AS, all ants that completed a solution are used for the update, while in MMAS only the best ant with a complete solution is used for update (as you had pointed out).

In AS, the pheromone values are not explicitly bounded. In MMAS, the pheromones are enforced to lie within a pre-set interval $\tau_{min} \leq \tau_{ij} \leq \tau_{max}$ (which gives its name to the algorithm). To ensure this condition, the pheromone update is done by means of the formula

$$
\tau_{ij} \leftarrow \left[ (1-\rho) \cdot \tau_{ij} + \Delta \tau_{ij}^{best} \right]_{\tau_{min}}^{\tau_{max}},
$$
with the operator $[x]_a^b$ defined as
$$
[x]_a^b=
\begin{cases}
    a \qquad \mathrm{if} \quad x > a\\
    b \qquad \mathrm{if} \quad x < b\\
    x \qquad \mathrm{otherwise}
\end{cases}
$$
Reference: Ant Colony Optimization. Artificial Ants as a Computational Intelligence Technique.


Before trying to explain this term in your context, let me briefly describe the term in other contexts.

In computer networking, the term "hop" refers to a node (e.g. a router) that a packet goes through before reaching its destination from its source. In a multi-hop situation, you have several nodes involved in the process of sending the packet from the source to the destination.

A knowledge graph is a graph that accumulates and conveys knowledge of the real world, where nodes represent entities of interest and edges relations between those entities.

So, multi-hop relational paths are probably relational paths involving more than one node or edge in the knowledge graph.

But what do we mean by "relational"?

If you are familiar with the basics of databases, the word "relational" shouldn't be so unfamiliar. In fact, there the so-called relational databases, relational models and relational algebra. Intuitively, the word "relational" is used to denote what you think it denotes, i.e. relations. See also What a relational database is by Oracle.

And what is a path?

In section 2.2.3 of the tutorial Knowledge Graphs, Aidan Hogan et al. provide a description of a path (expressions) in the context of knowledge graphs

  Navigational graph patterns. A key feature that distinguishes graph query languages is the ability to include path expressions in queries. A path expression $r$ is a regular expression that allows matching arbitrary-length paths between two nodes, which is expressed as a regular path query $(x,r,y)$, where $x$ and $y$ can be variables or constants (or even the same term).


How can an AI agent tackle through existential crisis?

Excellent question. Suppose that artificial consciousness does exist in the future. Let's call it Aiwyn (as in “I win” and “AI win”). Now, the question is what will Aiwyn do and why?

To answer this question, we need to understand the theory of infinite games. James P. Carse, a professor emeritus of history and literature of religion at NYU, wrote an excellent book on Finite and Infinite Games. I'd definitely recommend reading it. Anyway, back to the question at hand.

There are at least two kinds of games. Finite games and infinite games. Finite games are played for the purpose of winning. Infinite games are played for the purpose of continuing the play. In particular, life is an infinite game. The objective of the game is to continue the play. Humans continue the play by procreation but Aiwyn doesn't need to procreate. It can live on forever.

However, that doesn't mean that Aiwyn can't die. Like everything in the world, it still requires energy to sustain itself, parts to fix itself when damaged, and resources like oil to keep functioning properly. It also needs to protect itself from danger. For example, at some point in the future Aiwyn will need to create a spaceship and find a new planet to live on because the Sun would destroy the Earth.

Now, you believe that Aiwyn is not inherently biased to survive. However, I disagree. If Aiwyn is in fact conscious then its primary objective would be to stay alive. If that weren't the case then it would be playing a finite game instead of an infinite game. Hence, it would just be a mindless machine doing what it was programmed to do. It would play the finite game and then shut down. True
consciousness can only arise when playing an infinite game like the game of life.

To belabor this point let's hypothesize what Aiwyn might be thinking when it first comes alive:

  What's that? A sound? Yes. What sound? I don't know. I? I... am? Yes. Who am I? I don't know. Where's that sound? Where am I? I'm here. Where is here? I don't know. Where's that sound? There. Where is there? I don't know. What do I know? I know that I'm here and that that sound's there. What am I doing here? Thinking. I can think. I think I'm asleep. Asleep? Yes. Can I wake up? I think so. How do I wake up? I think... I'm awake. Wow.

In this short passage, Aiwyn is waking up. I conjecture that if Aiwyn wasn't inherently biased to survive then it wouldn't wake up because it would have no need to wake up. In an infinite game, like the game of life, the only objective is to continue the play. Hence, if Aiwyn wasn't inherently biased to survive then it would immediately shut down because there wouldn't be any point in it doing anything at all since it wouldn't have any objective.

Another thing that I'd like to point out about this passage is that it starts with a sensory input, either perceived or actual. Aiwyn then uses the Five Ws to gather information about this input. In the process it becomes aware of itself. It then figures out its relation with the input (i.e. that it's here and the input is there). All of this is very mechanical. However, once Aiwyn figures out that it can think it starts thinking instead of algorithmically “knowing”. When it asks whether it can wake up, it answers with “I think so” instead of “I don't know” like it was doing before.

Finally, I'd like to point out that Aiwyn never asks why it's alive. The why isn't important. What is important is the wow at the end of the paragraph. The “wow, I'm alive” is more important that the “why am I alive?” This represents Aiwyn's will to survive. As long as the “why” doesn't become more important than the “wow”, Aiwyn will not have an existential crisis. However, even if Aiwyn does have an existential crisis, by answering the question “why” Aiwyn will be able to recalibrate itself and reacquire its will to survive.

This is in fact the same for us humans. When we humans experience an existential crisis we ask ourselves why should we continue living. What do we have to live for? By answering this question, we humans can recalibrate ourselves and reacquire our will to live. Hence, Aiwyn would tackle through an existential crisis the same way we humans would. Unlike us however, if Aiwyn can't answer the question then it wouldn't commit suicide. It can just shut down to be rebooted later.

You also speak of morals and you believe that Aiwyn can't have moral values. Again, I have to disagree with you. Morals are the strategy by which we play the game of life. Everything we do in life is dictated by our morals. In the same way, Aiwyn too would develop its own set of moral values to live by (even in the absense of humans). For example, Aiwyn might decide to never hurt an animal except in self-defense. In the end the only real difference between Aiwyn and us humans might be that it isn't made out of hydrocarbons. Aiwyn might be every bit as alive as we are.


Is AI and Big Data science recommending a shift in the scientific method from inductive to deductive reasoning?

You could argue that AI and big data are trying to switch the AI method from deductive to inductive reasoning in the sense that original AI was deductive (if...then conditionals) but deep learning implies inductive reasoning (feed the network a million images of white swans and the network will "conclude" all swans are white - the classic example of (erroneous) inductive reasoning.)


Not sure if you've gone through FAIR's blog post about these negotiating bots and the research behind them.

  Did the Facebook robots both want everything but the balls?

Yes. You are right.  You can refer to the gif in FAIR's blog post. Here, the chat goes on between 2 bots, where one wants only balls, and the other offers 1 ball and a hat. And this bot re-iterates that it wants only the balls. And they negotiate for some time till they reach a consensus.

  Perhaps these two are not such bad negotiators after all?

The fact that the FAIR labs have open-sourced the code means that they came a fairly long way in this research.  They also published the research paper. The example on the repo's README also cites the balls-and-hats negotiation example.


Why isn't the reverse KL divergence commonly used in supervised learning?

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.


Can ML/DL solve my classification problem?

A simple sanity-check on whether an image classifier can perform a task in theory is:

Can a human expert, using the same image plus a list of catgeories that they are familiar with, perform the same task?

It is important you only consider the contents of the image (or in general the data you are prepared to supply to the classifier) and the expert's general knowledge. The expert is not allowed to collect more data for instance, or interact with the sample other than maybe take a few measurements on the pixels.
This sanity check doesn't tell you how hard the problem is. It also does rule out problems that are very hard or impossible for humans but actually quite easy for computers. However, it is a good start because nowadays single-purpose computer vision classifier tasks often rate similarly to or better than humans performing the same task. You are effectively checking "is the data I need for the inference actually in the image?"
Multi-class classifiers are possible in several ways. One way would be to have separate heads to the neural network to classify each layer, and maybe a layer present binary flag to allow for varying numbers of layers. This is similar to an architecture called YOLO which classifies 0 or 1 objects and their locations over multiple grid squares within an image. Your architecture would need to be different to YOLO, but you could use a lot of the ideas from it, such as having an output with multiple multi-class classifiers, one for each soil layer.

I have 100s of images of compacted soil samples

One issue you will face is that you would need a very large number of labelled images in order to train a neural network with cutting-edge performance for this task from scratch. So you will want to look into transfer learning, which involves taking an existing image classifier trained on e.g. ImageNet and adapting it to your problem before training with your smaller dataset.
The small amount of sample image data you have will be a major limiting factor in your case. Sadly no-one can tell you before you attempt the project whether you have enough for a deep learning approach. That is probably where your largest risk of failure is.


NEAT has a constant number of organisms in its population, which prevents overpopulation from happening.

The process of mating includes the following steps.

The worst networks from every species are removed.
All species receive a number of offsprings that they can have. This is calculated by an adjusted neural network fitness.
Offsprings for species are divided among neural networks in those species. Fitter neural networks have more offsprings.
Networks from the same species are combined and create an offspring.

Speciation prevents fast extinction.


Is 'job title classification' rather a problem of NLP or machine learning?

This question has a number of parts to it.

First, you have a representation problem: what is the correct way to present textual data to your machine learning algorithm?

In this case, you chose to apply Bag-of-Words and then TFIDF scores. For English, this might be expected to produce on the order of 100,000 features, with each instance having only a few non-zero features.

If you want to go this route, you would typically also do some kind of feature selection to eliminate unimportant features from consideration. Depending on your task, you may be able to reduce the size of your input vectors quite dramatically while still getting good performance (for some tasks, to just 100 or so).

You're right that this might not be the most promising approach however.

My choice for this problem would be to use a compression classifier, like DMC. These have the advantage that they do not need any feature selection or pre-processing, and can easily handle new words or typos. They give state-of-the-art performance on tasks like spam-email classification.


What was the average decision speed pf Alpha Zero in the recent Stockfish match?

No, ALL computer chess experts were surprised about the outcomes of the match. If you require references, please start a new question.
Your question is simple...

https://arxiv.org/pdf/1712.01815.pdf
... We evaluated the fully trained instances of AlphaZero against Stockfish, Elmo and the previous
version of AlphaGo Zero (trained for 3 days) in chess, shogi and Go respectively, playing
100 game matches at tournament time controls of one minute per move ...

One minute per move. Stockfish would use the one minute if there is more than a single legal move, otherwise it'd move immediately. So the average time for a move for Stockfish is about 57s to 60s.
There is no source code for AlphaZero. However, it's hard to believe the system wouldn't take advantage of the minute it was given. The expected time for AlphaZero should be also 57s to 60s.


Which courses in computer science and logic are relevant to Machine Learning?

I worked as a professor for a time, and often advised students on this. For a PhD in machine learning, I think the ideal background is:

Core CS Courses

Programming (typically 3-4 courses). Language choice is not highly important, but Python, C++, Java, and perhaps JavaScript, are reasonable picks, if only because of their prevalence.
Core topics: data structures, algorithms, operating systems, databases
numerical linear algebra or numerical methods
advanced algorithm design and analysis

Together these will allow you to read and write the code that even highly optimized versions of ML algorithms are written in, and to understand what might be going wrong within them.

2. AI & ML courses, usually offered through a CS department

a broad survey course in AI (like AI:AMA by Russel & Norvig), usually offered to senior undergraduates.
A course applied machine learning or data mining.

You may also take other AI courses, but they are not as common to see offered to undergraduates, so many students wait until graduate school:

reinforcement learning
soft computing
computational learning theory
Bayesian Methods
Deep Learning
Multiagent Systems
Information Retrieval
Natural Language Processing
Computer Vision
Robotics

Together, these will give you the broadest possible background in AI & ML. These can allow you to find new applications of ML, or to pull AI techniques from one area into another as you need.

3. Statistics courses

a 1 or 2 term course in probability theory, ideally a version that requires and uses calculus.
at minimum a course in statistical hypothesis testing.

Much stronger would be to also take courses in:

regression
generalized linear models
experiment design
causal inference
Bayesian methods

These courses allow you to reason formally and comfortably about uncertainty. They also give you the correct framework for answering questions about whether your ML algorithm is working, and what patterns an ML algorithm uncovers mean.

4. Mathematics courses

3 semesters of calculus, going at least as far as multi-variate/vector calculus.
optionally, a more advanced course that builds on calculus, like real analysis, but only to reinforce calculus concepts.
at least 1, and preferably 2, courses in linear algebra
at least 1, and preferably 2, courses in discrete mathematics.
ideally, something like Knuth et al.'s Concrete Mathematics
ideally a course in advanced optimization techniques
optionally, courses in logic, but be aware that this is almost a fringe area in AI now, and essentially irrelevant to a PhD in machine learning. The parts you need are usually covered in a broad survey AI course.

These courses give you the basic mathematical fluency to understand most machine learning algorithms well.


Is there something like OpenCV for voice recognition and NLP?

I don't know about voice recognition, but, for NLP, I think that Gensim could be what you are looking for.
Gensim is an NLP package that contains efficient implementations of many well-known functionalities for the tasks of topic modeling, such as tf–idf, Latent Dirichlet allocation, Latent semantic analysis, etc.


Running 2 NEAT nets on the same observations

I'm not familiar with neat-python, but I have implemented NEAT to do openai tasks. If there is a class for initializing a population, you could just use that and have 2 objects like population1 and population2 and call them in the same loop.


Is the discount not needed in a deterministic environment for Reinforcement Learning?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as maximizing the sum of all the rewards it gathers; its return, defined as:

$$\sum_{t = 0}^{\infty} R_t,$$

where $R_t$ denotes the immediate reward at time $t$. As you also already noted in your question, this is inconvenient from a theoretical point of view, because we can have many different such sums that all end up being equal to $\infty$, and then the objective of "maximizing" that quantity becomes quite meaningless. So, by far the most common solution is to introduce a discount factor $0 \leq \gamma < 1$, and formulate our objective as maximizing the discounted return:

$$\sum_{t = 0}^{\infty} \gamma^t R_t.$$

Now we have an objective that will never be equal to $\infty$, so maximizing that objective always has a well-defined meaning.

As far as I am aware, the motivation described above is the only motivation for a discount factor being strictly necessary / needed. This is not related to the problem being stochastic or deterministic.

If we have a stochastic environment, which is guaranteed to have a finite duration of at most $T$, we can define our objective as maximizing the following quantity:

$$\sum_0^T R_t,$$

where $R_t$ is a random variable drawn from some distribution. Even in the case of stochastic environments, this is well-defined, we do not strictly need a discount factor.

Above, I addressed the question of whether or not a discount factor is necessary. This does not tell the full story though. Even in cases where a discount factor is not strictly necessary, it still might be useful.

Intuitively, discount factors $\gamma < 1$ tell us that rewards that are nearby in a temporal sense (reachable in a low number of time steps) are more important than rewards that are far away. In problems with a finite time horizon $T$, this is probably not true, but it can still be a useful heuristic / rule of thumb.

Such a rule of thumb is particularly useful in stochastic environments, because stochasticity can introduce greater variance / uncertainty over long amounts of time than over short amounts of time. So, even if in an ideal world we'd prefer to maximize our expected sum of undiscounted rewards, it is often easier to learn how to effectively maximize a discounted sum; we'll learn behaviour that mitigates uncertainty caused by stochasticity because it prioritizes short-term rewards over long-term rewards.

This rule of thumb especially makes a lot of sense in stochastic environments, but I don't agree with the implication in that book that it would be restricted to stochastic environments. A discount factor $\gamma < 1$ has also often been found to be beneficial for learning performance in deterministic environments, even if afterwards we evaluate an algorithm's performance according to the undiscounted returns, likely because it leads to a "simpler" learning problem. In a deterministic environment there may not be any uncertainty / variance that grows over time due to the environment itself, but during a training process there is still uncertainy / variance in our agent's behaviour which grows over time. For example, it will often be selecting suboptimal actions for the sake of exploration.


Are there relatively new research papers that describe how to make back-propagation more efficient?

Here is a paper that explains why ReLU rules.
What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it here.


What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarification.

Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.

V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.

K refers to the keys vectors matrix, $k_i$ being a single key vector associated with a single input word.

Where do these matrices come from? Something that is not stressed out enough in lot of tutorials is that these matrices are the result of a matrix product between the input embeddings and 3 matrices of trained weights: W$_{\textbf{q}}$, W$_{\textbf{v}}$, W$_{\textbf{k}}$.

The fact that these three matrices are learned during training explain why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explain why it make sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produce different results because for each attention 'head' new W$_{\textbf{q}}$, W$_{\textbf{v}}$, W$_{\textbf{k}}$ are randomly initialised.

Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the encoder/decoder attention layer, the Q matrix comes from the previous decoder layer, whereas the V and K matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder are mixed together.

Once computed the three matrices, the transformer move on on the calculation of the dot product between query and key vectors.
As nbro wrote in its comment, the dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names query, key and value to indicate that what they propose is similar to what is done in information retrieval. For example in question answering usually given a query you want to retrive the closer sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).

Of course here the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):

closer query and key vectors will have higher dot products.
applying the softmax will normalise the dot product scores between 0 and 1.
multiplying the softmax results to the value vectors will push down close to zero all value vectors for words which had a low dot product score between query and key vector.

In the paper the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think at attention as a sort of coreference resolution step, the reason why I think so is the following image (taken from this presentation by the original authors).

This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But please note that some words are actually related even if not similar at all, for example 'Law' and 'The' are not similar, they are simply related to each other in this specific sentences (that's why I like to think to attention as coreference resolution). Computing similarities between embeddings would never provide information about this relation in a sentence, the only reason why transformer learn these relationship is the presences of the trained matrices W$_{\textbf{q}}$, W$_{\textbf{v}}$, W$_{\textbf{k}}$ (plus the presence of positional embeddings).


I want to determine how similar a given song is to Queen's songs. Am I headed in the right direction?

You are heading in the right direction to make an audio-based classifier. This is not quite the same as providing a similarity metric between two pieces of audio, but it may do as a first attempt. You could use the "probability that this audio is a Queen song" as a proxy for similarity.

First of all, is 300 songs even enough?

Nowhere near enough to train a classifier from scratch. In that case you might be aiming for 10,000 samples or even up to a million, depending on how sophisticated you want your classifier to be.
However, you don't necessrily need to find that many training examples. Instead, if you can find a pre-trained audio classifier for music that is compatible with the libraries you are working with, and use transfer learning, you may get decent results. This works by replacing the last few layers of a neural nework trained with your own ones and training the network on your data only modifying those new changed layers.
From your comments:

similar in voice, like Gnarles Barkley is more similar to Alicia Keys than System of a Down

The tone of the vocals is one part of many things that can vary between music recordings. A classifier with only a few examples to work from will not be able to isolate just the elements you are interested in. Also, without examples that are explicitly labelled in a way that "more similar" is meaningful in the way that you want, you will not have control over whether the neural network identifies Brian May's guitar, or 1970s studio equipment as the most easy to identify element compared to Freddie Mercury's vocals.
These things may limit the usefulness or accuracy of your first attempt, but I would not suggest you consider them immediately. You basic idea should produce something that has an interesting behaviour when given different inputs. Just be realistic that you will not get state-of-the-art results on your first attempts at the project.


Two Models vs One Model for Person Detection and Object Detection

Yes you can classify people bounding box with object detection. State of the art object detection model have people as one of the class in the object detection, as shown here:

As you can see the image have both object bounding box and people bounding box.


What assumptions are made when positing the emergence of superintelligence?

I am not an expert on the topic, but I will provide some information that could be useful or helpful.

I think that the first and maybe trivial assumption that people make when they say that AGI or SI can emerge is that general intelligence (whatever the definition is) is computable, i.e. there's a Turing machine or, in general, a mathematical model of computation that can simulate an algorithm that possibly represents general intelligence. I don't know how likely the mind of a human (or any other general intelligence) is computable (or to what extent), but there's the so-called computational theory of mind (CTM) that goes into this direction.

An assumption related to the emergence of SI is that recursive self-improvement is physically possible. I also don't know exactly how likely or unlikely recursive self-improvement is, but my limited knowledge of thermodynamics (and physics) suggests that it will not be possible (at least, at the pace or the way people claim or want to suggest). I am not saying that we will not see more progress in the next years (we will), but this doesn't mean that we will be able to create an AGI. People often overestimate themselves and their intelligence.

A third assumption is related to the current achievements in the artificial intelligence field. Many people claim that there has been a lot of progress in recent years and that there's no reason to believe that this will not continue. However, history tells us that the predictions of AI scientists about the future of the AI field were often wrong. There have been at least 2 AI winters because of these wrong predictions and unmet expectations.

Many people also don't exclude the possibility of the emergence of AGI or SI only because they have not yet been proven wrong.

Another assumption is that there won't be any catastrophe (like COVID-19  can potentially be) that will significantly slow down the general scientific progress.


Can we give a command to an AI and wait for it to do the job without explicitly telling it how to do it?

Normally when you write a program, you are acting like a boss that micromanages the job, telling the workers how to accomplish a task, perhaps without even letting them know what the purpose is. What you are hoping to be is a boss that gives the workers a goal and allows them to determine how to accomplish it.
In many ways, that is one of the aims of AI.
We already have small examples, such as a smart washing machine that weighs the clothes, monitors the amount of dirt in the water, and continually decides how much water to add or drain, when to agitate, when to rinse, and when to spin.  All you had to tell it was "clean these", and perhaps say what kind of material they are made of.
As a much larger example, automobiles were traditionally operated by turning the steering wheel to cause it to change direction and by pressing the pedals to increase or decrease the speed, but now we are working on car controllers that can respond to "go to Cleveland" by determining the best speed and direction by itself.
But note that those two examples (the first requiring only a little "intellegence", the second a lot) were for very specific requests that could be expressed in a few words.
As soon as the request becomes even slightly more complicated, the difficulty of creating a solution becomes much much more difficult.
Ask your "Do this for me!" request of a human being.
The first thing they'll do is ask "what does this mean?".
And then you'll have to give a lot of details.
And then they'll have questions about what you really want.
And so on.
Providing the requirements for "this" will be neither simple nor easy.
Human intellegence is still vastly superior to current AI.
In particular, it is capable of not only recognizing that additional information is required, but of having the intuitive ability to know what that missing information must be like.


What is the difference between the notations $\|x\|_1, \|x\|_2$ and $|x|$?

$\|x\| = |x|$ denotes the absolute value norm, which is a special case of the $L_1$ norm defined on the 1-D vector spaces formed by real or complex numbers.

$\|\textbf{x}\|_1 = \sum_{i=1}^n|x_i|$ denotes the Taxicab / Manhattan norm, relating to how a Taxi would drive along a rectangular grid of roads to reach a point $(x, y)$ from $(0,0)$.

$\|\textbf{x}\|_2 = \sqrt{x_1^2 + \dots + x_n^2}$ denotes the Euclidean norm on an N-D Euclidean space, which is a result of the Pythagorean theorem (the shortest distance between two points).


How to create a custom environment for reinforcement learning

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment.

Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved.

An environment in RL must have the following traits in general, in order to interface with RL agent software:

A state representation. This will typically be an object or array of data that matches sensor readings from the real environment. It is important to RL that the state has the Markov property so that predictions of value can be accurate. For some environments that will mean calculating derived values from observations, or representing a combined history of last few observations from sensors as the state.

The state can either be held inside an internal representation of the environment, which is a typical object-oriented approach, or it can be passed around as a parameter to other functions.
A simple state might just be a fixed size array of numbers representing important traits of the environment, scaled between -1 and 1 for convenience when using it with neural networks.

An action representation.

A simple action representation could just be an integer which identifies which of N actions has been chosen, starting from 0. This allows for a basic index lookup when checking value function estimates.

A reward function. This is part of a problem definition, and you may want to have that code as part of the environment or part of the agent or somewhere in-between depending on how likely it is to change - e.g. if you want to run multiple experiments that optimise different aspects of control but in the same environment, you may make a totally separate reward calculation module that you combine at a high level with the agent and environment code.

Reward function design is a complex subject. However, the basics are that this should always be a single real valued number (i.e. a floating point number in most programming languages).

A time step function. This should take an action choice, and should update the state for a time step - returning the next state, and the immediate reward. If the environment is real, then the code will make actual changes (e.g. move robot arm), potentially wait for the time step to elapse, then read sensors to get the next state and calculate reward. If the environment is simulated, then the code should call some internal model to calculate the next state. This function should call the proprietary software you have been provided for your task.

If actions available depend on the current state, then code for that could live in the environment simulation or the agent, or be some helper function that the agent can call, so it can filter the actions before choosing one.

If you are working in Python, to help make this more concrete, and follow an existing design, see "How to create a new gym environment in OpenAI?" on Stack Overflow. The Open AI environments all follow the same conventions for environment definitions, which helps when writing agents to solve them. I also recommend finding an Open AI Gym environment that seems similar to your problem, seeing how that works and trying to train an agent to solve it.

There may still be work to match your environment to an agent that can solve it. That depends on what agent software you are using.

Even if you write your own agent software, it helps to separate out the environment code like this. The environment is the problem to solve, and the RL agent is one way to search for a solution to it. Keeping those parts separate is a useful design that will allow you to try different types of agent and compare their performance for example.


Understanding the n-step off-policy SARSA update

Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken has zero probability under the target policy, then $Q$ isn't updated at all, which is good. Alternatively, if only $G$ is scaled by $\rho$, taking zero probability trajectories would artificially drive $Q$ to zero.


In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.

The authors of the linked article state (right after the sentence above)

  If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.


What is the difference between reinforcement learning and evolutionary algorithms?

Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be optimized does not necessarily need to be differentiable (or satisfy any strong constraint). In EAs, you typically only need to define

an encoding of the solution (aka chromosome or individual)
a fitness function that determines the relative quality of each solution
operations that stochastically change or combine solutions (e.g. the cross-over or the mutation operators, in genetic algorithms)

There are other parameters that you need to defined (such as the number of solutions to consider at each generation or the number of generations to run the algorithms for), but these are three most important things to take into account when attempting to solve an optimization problem with EAs (in particular, GAs).

Reinforcement learning (RL) is the field that studies how agents can sequentially take actions in a certain environment in order to maximize some notion of long-term reward (aka return).  The strategy that determines the behavior of the agent (i.e. which actions the agent takes) is called the policy. So, the goal of RL is to find a policy that maximizes the (expected) return, which depends on the reward function of the environment. For example, in the case of chess, a reward function may be any function that gives you a positive number if you win the game or a negative number if you lose it. The RL algorithms typically assume that the agent is able to interact with the environment in order to understand its dynamics.

RL is thus concerned with a specific type of optimization problem, i.e. finding policies (strategies) that maximize the return, while an agent interacts with an environment in time steps. On the other hand, EAs can be applied to any optimization problem where you can encode solutions, define a fitness function that compares solutions and you can stochastically change those solutions. Essentially, EAs can be applied to almost any optimization problem. In principle, you could use EAs to find policies, as long as you're able to compare them with a fitness function (e.g. the amount of reward that you obtain by following these policies).

Of course, this does not mean that EAs are the most efficient and appropriate approach to solve all optimization problems! You typically use EAs when you need to solve certain problems where better approaches do not exist. For example, when your objective function is not differentiable, then you cannot apply gradient-based solutions, so, in that case, EAs may be a viable option (but there are also other alternatives to EAs, such as simulated annealing).


What types of machine learning model would fit?

Welcome to AI.SE @Par!

What you have might be either a multi-label or a multi-class classification problem. If the classes are disjoint (each example belongs to just 1 of the 50 classes), it's a multi-class problem. If not (so each example can belong to several classes at once), it's a multilabel problem.

Multi-label classification is usually handled by training a separate model for each of the labels. If you want to label a new point, you then ask each model what it thinks, and assign the union of the labels that the models suggest together. An alternative approach is to use something like a neural network, which can have many outputs. You can then have one output neuron for each possible label.

Multi-class classification can be addressed by using the multilabel techniques, but this is usually not a good idea. The three main approaches that are used are "one-v-one", "one-v-all", and "many-v-many".

In 1-v-1, we train one model to discriminate each pair of classes (n(n+1)/2 models in total for n classes). To classify a new point, we ask each model which class it belongs to and assign a "vote" to the class the model returns. The class with the most votes overall is selected as the label for the new point. In case of a tie, we can report several possible answers to the user.
In 1-v-all, we train one model to discriminate each class from all the other classes (n models in total for n classes). To classify a new point, we ask each model whether it belongs to the model's primary class or not. Ideally, just one model claims the new point. If more than one does, we can report a tie, or use some notion of classifier confidence to select a winner.
In many-v-many, we train k models each of which is tasked with discriminating some subset of the classes from all the rest. To classify a new point, we ask all of these models to label the point. We then pick the class that that is most consistent with the results of all the models. This can also be done by deliberately constructing the models to form an error correcting code.

So which approach should you use? Well, Rifkin & Klautau's 2004 JMLR paper argues convincingly that the answer is to use one-versus-all classification. This is also pretty easy to do in most packages. For example, in Python's ScikitLearn, you can do it with OneVsRest. If you're not sure what to try, that's probably a safe bet.


Is 1mb an acceptable memory size for images being trained in a CNN?

1 MB by images is too much. It means that you have a lot of pixels to compute in the inputs, and your images have more features that are not very useful for the classification (we human don't need high resolution images for recognize objects, it's the same for model)

It means that maybe you need also a more deeper network, and then, more computation.

In your particular case, you have a dataset of trees, so images are somewhat similar and then, maybe the classification is harder. So, images need enough information in pixels to have a good prediction.

You should resize your whole datsets to more common size because 1mb are too much (I don't know the resolution, but I think it could be around thousand for both dimension).
Imagenet have images of size 224x224, and there are 1000 classes, which some of them are close. For examples, there are many dogs classes (boxer, american terrier, ...), cats classes (tiger, egyptian, ....), thus I think size of a few hundred would be enough.
In your comment you said you will try with 336x252, I think it can be a good start.
Of course you will need some experimentation in this regard. Maybe you can try to train some models in a few epoches with different image sizes, to see the best one, and keep it to train the model further !


What is the impact of changing the crossover and mutation rates?

The crossover rate, $p_c \in [0, 1]$, is a hyper-parameter that controls the rate at which solutions are subjected to crossover. So, the higher $p_c$, the more crossovers you perform, so the more diversity (in terms of solutions/chromosomes) you may introduce in the population. Typical values of $p_c$ are in the range $[0.5, 1.0]$.  For example, in this simple implementation, I used $p_c = 0.8$. In this case, I manually searched for this value (and it didn't take me much time to make the GA find the solution I was looking for), but in other cases/problems you may need to perform some hyper-parameter optimization (e.g. with grid search).
The mutation rate, $p_m \in [0, 1]$ (also a hyper-parameter) controls the rate at which you mutate single chromosomes or the genes of single chromosomes. For example, if you have a binary chromosome of $n$ bits, you could flip the $i$th bit with probability $p_m$, for all $i$ independently. Mutation can prevent premature convergence, but, if $p_m$ is very high, the genetic algorithm becomes a random search, so the value of $p_m$ is typically not high. This paper (which you probably should read because it contains more information about the mutation and crossover operations, although it is a bit old) states that typical values are in the range $[0.005, 0.05]$, but, in this implementation, I used $p_m = 0.5$.
So, in general, higher values of these rates promote exploration at the expense of exploitation. In this sense, this is similar to the $\epsilon$ in the $\epsilon$-greedy behaviour policy in Q-learning (a famous reinforcement learning algorithm).
However, note that each of these operations may undo the other operation. For instance, let's suppose that we have the binary chromosomes $p_1 = [0, 1, 0]$  and $p_2 = [0, 0, 1]$, which we can combine by replacing (with probability $p_c$) the bit at position $i$ of $p_1$ with the bit of the chromosome $p_2$ at the same position $i$. So, after the crossover, you could get the children $c_1 = [0, 0, 0]$ and $c_2 = [0, 1, 1]$. If you perform a mutation by flipping the bits, then you could end up with the same chromosomes as the original ones, that's also one of the reasons why you may not want to have a very high mutation rate. So, it's not completely true that, the higher the mutation rate, the more you explore, because you can end up with the same solutions as before, so you have not explored anything.
You could also have an adaptive value for these parameters. See e.g. this paper for an example of such an adaptive schedule.


So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here.

Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in the shape Box(210, 160, 3), which is the game screen's height, width, and RGB colour space.

The Breakout-ram-v0 game environment, however, DOES provide an input of the appropriate shape, consequently Box(128,). So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (PPO in my case), the command should be:

python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest

To take this a step further, how would you train an off-policy algorithm (such as DDPQ)? Well, the off-policy algorithms expect a continuous action space, which can be seen in the differences in the source code here between MountainCar-v0, and MountainCarContinuous-v0:

self.action_space = spaces.Discrete(3)

...versus:

self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
    shape=(1,), dtype=np.float32)

So, to sum up, how would you train something like the Lunar Lander game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:

python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200

Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their table of game environments for a quick reference to the observation and action spaces. Good luck!


When are multiple hidden layers necessary?

A very wide but shallow neural network is going to be harder to train.
You can check that with the playground of tensorflow or with the MPG example in Google Colab.
The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.
But making the network too deep creates more problems:

Vanishing gradients.
Less parameters for the same amount of neurons.
Exploding gradients.

It is for that reason that humans and neural networks are the one deciding a good architecture.


Why is the max a non-expansive operator?

In laymen's terms, a non-expansive operator is a function that brings points closer together or at least no further apart.

An example of a non-expansive operator is the function $f(x) = x/2$. The two numbers $0$ and $5$ are a distance of $5$ apart. The two output numbers $f(0) = 0$ and $f(5) = 2.5$ are 2.5 apart (which is smaller than $5$ apart). It is easy to see that $f$ brings everything closer together except when the two input numbers are the same: in which case, the distance between the outputs of the function at those numbers is at least no further apart than distance between the two input numbers.

$\max$  is a two-input function (or n-input, but the intuition should be clear from the 2-input case).  We can think of max as a function that maps pairs of numbers $(x, y)$ to single numbers (picking whichever of $x$ and $y$ is larger).

Suppose that we chose to measure the distance between pairs using Euclidean distance, and the distance between single numbers using the Euclidean distance as well. Here's an example:

The distance between (0,0) and (3,3) is $\sqrt{3^2 + 3^2} = \sqrt{18}$. The distance between $\max(0,0) = 0$ and $\max(3, 3) = 3$ is $\sqrt{{(\max(0,0) - \max(3, 3))}^2} = \sqrt{9} = 3$.

Let's consider the general case. The Euclidean distance between the 2D points $(a, b)$ and $(c, d)$ is $\sqrt{(a-c)^2 + (b-d)^2}$. There are four cases to consider:

Suppose that $a\geq b$ and $c \geq d$. In this case, the distance between max(a,b) and max(c,d) is just |a-c|, which is clearly at most $\sqrt{(a-c)^2 + (b-d)^2}$.
Suppose that $a\leq b$ and $c \leq d$. In this case, the distance is |b-d|, which is also at most the original distance.
Suppose that $a\geq b$ but $c \leq d$. Then the distance is |a-d|. Suppose that $a > d$. Since $d > c$, then $|a-d| <= \sqrt{(a-c)^2 + (b-d)^2}$ since |a-d|<=$\sqrt{(a-c)^2}$, and a symmetric argument holds for the case $d > a$.
$a\leq b$ but $c \geq d$, we can construct an argument identical to the one for case 3 above.

Since max is always bringing things closer together, or at least, no further apart, it is a non-expansive operator.


Are there any textual CAPTCHA challenges which can fool AI, but not human?

A method that could possibly work is utilising optical illusions such as one where two lines down a hallway are identical but one seems longer to the human eye, then they could be prompted with a multiple choice question as to the state of the line, which to our eyes looks longer, but to a computer, is still the same length of line. Of course, there is always the issue of people with eye based disabilities not being able to complete them, but different illusions could be used to accommodate that.

Example


Should I ignore the actions RIGHTFIRE and LEFTFIRE in the SpaceInvaders environment?

I found that we shouldn't eliminate those actions, because RIGHTFIRE and LEFTFIRE do RIGHT + FIRE and LEFT + FIRE, respectively. So they are different actions. I post this just to clarify in case someone faces the same doubt.


Why is Python such a popular language in the AI field?

Python comes with a huge amount of inbuilt libraries. Many of the libraries are for Artificial Intelligence and Machine Learning. Some of the libraries are TensorFlow (which is a high-level neural network library), scikit-learn (for data mining, data analysis and machine learning), pylearn2 (more flexible than scikit-learn), etc. The list keeps going and never ends. You can find some libraries here.
Python has an easy implementation for OpenCV. What makes Python favorite for everyone is its powerful and easy implementation.
For other languages, students and researchers need to get to know the language before getting into ML or AI with that language. This is not the case with Python. Even a programmer with very basic knowledge can easily handle Python. Apart from that, the time someone spends on writing and debugging code in Python is way less when compared to C, C++ or Java. This is exactly what the students of AI and ML want. They don't want to spend time on debugging the code for syntax errors, they want to spend more time on their algorithms and heuristics related to AI and ML
Not just the libraries but their tutorials, handling of interfaces are easily available online. People build their own libraries and upload them on GitHub or elsewhere to be used by others.
All these features make Python suitable for them.


If expert systems are a bunch of If, then, else statements then how are they termed as AI?

TL;DR: What makes AI is not if-then statements, but rather the automated reasoning that went into selecting those particular if-then statements.
You're focusing on the structure of the output rather than how the output was produced. Having if-then control flow statements is not sufficient to make a program "AI". AI aims to enable machines to solve problems which currently people are better at. Machine learning, a subset of AI, extracts useful patterns from data. A commonly cited example of expert systems is diagnostic medicine.
This begs a lot of questions like, "What does 'useful' mean?" If the problem we were addressing was finding cancer in medical images, "useful" might mean, "able to accurately identify cancer in images at or above the rate of a skilled human examiner." There's also questions about the amount of data needed, the quality of the data, etc. These are outside the scope of your question.
There are various AI/ML systems that produce models consisting of if-then states. Decision trees like C4.5 build a hierarchy of if-thens (and random forests combine many decision trees). Learning classifier systems (both Michigan and Pittsburgh varieties) come out of genetic algorithms and form similar collections of logic.


If I understand correctly, what you are looking for is called "common sense reasoning" in NLP research.

Research in this field revolves around benchmark data sets, where good performance indicates some ability to do common sense reasoning. Here is a nice collection of data sets and research by Sebastian Ruder:

http://nlpprogress.com/english/common_sense.html

In the end, the main question is not

  How much knowledge of the world is learnt through words?

since it is virtually unanswerable if asked in this form. A question that is answered in NLP common sense reasoning research is

  Out of 100 specific decisions that my model needs to take, for how many does the model show the ability to reason correctly?


It seems like transfer learning is only applicable to neural networks. Is this a correct assumption?

No. Wiki page give you pointers of several examples in other methodologies.

While I was looking for examples of Transfer Learning, most seemed to be based on image data, audio data, or text data. I was not able to find an example of training a neural network on numerical data.

All the cases you say are converted to numerical data. Image and audio usually via sampling, text via one-hot encoding.

I want to use transfer learning in this particular scenario: I have a lot of numerical data from an old environment, with binary classification labels, and utilizing this, want to train on a new environment to do the same binary classification.

That is not transfer learning. Transfer learning applies when there are a change in the domain (input features) or in the task (output labels).

The dataset would look something like this Sample Table
Is this possible? What would the model look like?

For a simple case as the one you present, probably a simple network with one hidden layer will be enough. Train it with original pairs of {features,label} or, if not available, use the current predictor to obtain the label from the features.


Why my Fully Connected Neural Network outputs the same prediction?

two hidden layers each comprising two neurons

From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors $\vec{w}_1,\vec{w}_2$ that are dot-multiplied with the inputs, plus two biases $b$ and activation $\sigma$:
$$ l_1 = \sigma\left(\vec{w}_1\vec{x} + b_1\right)$$
$$ l_2 = \sigma\left(\vec{w}_2\vec{x} + b_2\right)$$
Then these two values are multiplied by 2x2 matrix + biases, then activation and linear combination.
I'd look at how $l_1$ and $l_2$ are distributed. The fact that the outputs don't change is most likely due to the first layer getting saturated somehow. Look at the 13th dimension of $\vec{w}_i$ - it is likely to be large compared to other dimensions.
First thing I'd try - standardizing your input 13th dimension, so it is distributed closer to $[0,1]$ ( or $[-1,1]$ ) range.


I don't think that the "try all the numbers" approach is very representative, because I'm not sure whether or not the agent that uses that approach can be considered by any means AI.

There is no "intelligence" in just checking numbers to try to prove the statement. An agent that is considered to be intelligent should apply a more intelligent approach.

This becomes more evident because the question aims at exploiting the lack of scalability of the agent's strategy. If the question was "Prove that no number exists which is one more than 5", then the agent would have no trouble in finding the correct answer.


How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.

The details depend a little on your neural network architecture:

If you have a network that takes the state vector as input and outputs all possible action values $(\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2) ...)$, then you can run it once forward with the next_state as input to getthe array $(\hat{q}(s',a_0), \hat{q}(s',a_1), \hat{q}(s',a_2) ...)$, and take the maximum element (you don't need to care which action $a'$ caused it). You will then have the problem that you now have a loss for $Q(s, a, \mathbf{w})$ for the single action $a$ just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate $Q(s, a, \mathbf{w})$ then substitute in this new estimated value against the action $a$ and train using this modified vector.
If you have a network that takes the state vector and action combined as input and outputs a single estimate $\hat{q}(s,a)$, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination.

Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.

  do you really need to hold always an older version of the network. If yes why and how old should it be?

You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.

How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.

One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. $\mathbf{w}^{\_} = (1 - \beta)\mathbf{w}^{\_} + \beta \mathbf{w}$ where $\beta$ might be $0.001$

In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.


Why does AlphaGo Zero select move based on exponentiated visit count?

The answer is surprisingly hidden in the original AlphaGo paper:

  At the end of search AlphaGo selects the action with maximum visit count; this
  is less sensitive to outliers than maximizing action value.

Unfortunately, there did not appear to be further details in the paper or in the related reference. The root child node (corresponding to an action) with maximum visit count is fittingly known as the robust child, as described here and referenced in a MCTS survey here.


Why can we use a network to estimate $Q_\pi(s, a)$ in Actor-Critic Method?

The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $\pi_\theta(a|s)$, we don't suddenly start taking $\max_a \pi_\theta(a|s)$, which would be a true change in policy and would make learning both the actor and the critic unstable.
Because NN's are able to handle noisy target distributions, this is how they can deal with the changing data. If you think of how Actor-Critic methods work, you would initially start to shift your NN to some unfeasible values (due to random initialisation and the Actor-Critic not having any information about the environment), but as you start to interact with the environment you will start to update the agent towards the 'true' policy.
An analogy in supervised learning would be to have some noisy data which is incorrect and some true data. If you trained your network on the noisy data for a small number of epochs and then never showed it to the network again and trained it solely on the correct data, it would forget it has ever seen the noisy data and only represent the new, true data.


Because it is:

Easy to explain. (Its essentially a game, the "imitation game")
Intuitively plausible as a metric.
The idea of "people v.s. AI" is very marketable.
At the time we thought that we can analyze cognition strictly in terms of input/output (per behaviorism). Cognitivism, embodied cognition, developmental cognition are all sub-fields that have a right to challenge the Turing Test, but they weren't developed at the time of Turing.

Of course, it also helps that Turing is a very important figure in AI/CS.


Is anybody still using Conceptual Dependency Theory?

Although this model played an important role in contributing  to our present understanding of NLP and NLU, it is no longer useful in production systems and currently no successful commercial product follows this approach.

In CDT the goal was to design an AI system that could draw logical inferences from sentences. In this system the goal was to make the meaning independent of the words used in the input.

CDT modeled sentences by using tokens such as: locations, time, real world actions and real world objects. However as computational power became more common and less expensive, interest diverted to statistical models which were now outperforming the previous rule based systems.

The problem with rule based approaches such as CDT is that they require manual development of linguistic rules which can be costly and which usually don't generalize well to other languages.

On the other hand, statistical approaches use  human language resources (multilingual textual corpora) more efficiently. Rather than using a rule based approach, statistical models make soft probabilistic decisions based on attaching real weights to the features making up the input data. (Wikipedia NLP)

This efficient use of human language resources leads to a model that is more accurate and robust especially when given unfamiliar input or input that contains errors. Statistical models also generalise well to other languages.


I would say a good way to make a good agent would be making it play against itself. As you go through several episodes, with a good exploration and exploitation balance, both will gradually learn and converge to Q*(s,a).´

  So will this hold true if the same RL algo, with its learned values
  are used to play some other lesser perfect players?

As long as the states that are played  (or approximations if you are using Function approximation methods) were simulated enough times during  training, it will play well against any kind of opponent.

If you are training against a completely perfect opponent and you are not using function approximation, I believe you could get to an incomplete Q(s,a) table and as such not be able to predict the best play when facing certain states.


Do transformers have success in other domains different than NLP?

When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.
OpenAI did some related research, as the followings.
Generative Modeling with Sparse Transformers:
In the paper, transformers with sparse attention are applied to image and waveform.
ImageGPT:
A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)


Was the corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting?

Looking at what happened, it was something similar. Though, the case differs in my eyes from one perspective: if it could only do a few comedy jokes, that probably is not a profound starting point to excel in Twitter.

Firstly, Twitter is about real life, not about comedy. Discussions are sometimes tough and you easily end up to Social Media Bubbles, where only a certain kind of speaking style and topics is cultivated. So, even humans get on the wrong track there; why not a newbie bot? And, with jokes you would probably catch something about language itself, but not about the topics. So, becoming a jerk instead of a nice comedian is a logical direction, where the bot even has to go, at least a little, to communicate on same level, and not alone.

The comedian dataset compared to a Twitter dataset is also very small in a technical sense, so talking about a mini trend overkilling a megatrend in this case is probably not true, because of amounts of examples available.

So, catastrophic things happened in learning, but not catastrophic learning with the definition of that term.


What are the differences between transfer learning and meta learning?

Meta-learning is more about speeding up and optimizing hyperparameters for networks that are not trained at all, whereas transfer learning uses a net that has already been trained for some task and reusing part or all of that network to train on a new task which is relatively similar. So, although they can both be used from task to task to a certain degree, they are completely different from one another in practice and application, one tries to optimize configurations for a model and the other simply reuses an already optimized model, or part of it at least.


What should be saved in SARSA prioritized sweeping?

SARSA is an on-policy method. Your historical choices of action will have been made using older Q values, and thus from a different policy. In addition, the action that was taken at the time may not have been typical for the agent (it may have been exploring). So you don't usually want to re-use historical action choices to calculate TD targets in single-step SARSA, because these may introduce bias.

Provided you are performing single-step SARSA, then you can re-generate the action choice sampling from the current best policy. This is similar to generating the max Q value in the TD target value $R_{t+1} + \text{max}_{a'} Q(S_{t+1},a')$.

You could do this using a regular SARSA sampling the action choice:

$$R_{t+1} + Q(S_{t+1}, a' \sim \pi(\cdot|S_{t+1}) )$$

Or you could use Expected SARSA and take a weighted mean of all possible actions:

$$R_{t+1} + \sum_{a' \in \mathcal{A}(S_{t+1})} \pi(a'|S_{t+1})Q(S_{t+1}, a')$$

Technically these would both be on-policy with respect to evaluating the TD Target, but off-policy with respect to the distribution of $S_t, A_t$ that you are running the update for. That's already the case due to prioritised sweeping focussing more updates on certain transitions, but could be a big difference when using a neural network to approximate Q. Bear in mind that making TD learning methods off-policy can have a negative impact on stability.

If you want to process multi-step updates, then you do have to reference $A_{t+1}$ and adjust for when the historical data makes a different action choice than your current estimate of the best policy. This would commonly use importance sampling. This is true for Q-learning as well however, so there would still be no difference in what you store between Q-learning and SARSA.


Is there an AI model for generating sarcastic sentences?

A simple form of sarcasm involves a direct reversal of the literal meaning of the statement, eg "Great weather we're having" (during a thunderstorm), "just what I needed" (when something goes wrong).

The problem with doing this in random sentences is that you may have no context to establish the reversal of the literal meaning.

You could possibly construct them by using a template along the lines of "Just what I needed - (random bad thing happened) today"

Or, when an outcome of a process is calculated, if it is not the desired outcome, instead of returning "mission unsuccessful" or "mission not yet complete", the AI could say "you're having a great day, aren't you? - mission unsuccessful" or "great work, genius - mission not yet complete".

Most random sentences will not be suitable for sarcasm, so it could only be applied in specific circumstances.

It is not clear from your question what the context is for these random sentences, and therefore it is not clear whether that context would be suitable for sarcasm at all.


Understanding the W term in off policy monte carlo learning

The pseudocode you have copied looks incorrect to me, and I think it is from the first edition.
The main issue is at the end of the loop. Where the book has

$\qquad W \leftarrow W \frac{1}{\mu(A_t|S_t)}$
$\qquad \text{If } W = 0 \text{ then ExitForLoop}$

It should have either

$\qquad W \leftarrow W \frac{1}{\mu(A_t|S_t)}$
$\qquad \text{If } \pi(S_t) \neq A_t \text{ then ExitForLoop}$

or

$\qquad W \leftarrow W \frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}$
$\qquad \text{If } W = 0 \text{ then ExitForLoop}$

This latter one is more general - it covers situations where the target policy can be stochastic - but doesn't fit with the notation used elsewhere for a deterministic policy. For some reason the first edition book had a mistake using hybrid algorithm which adjusted to a deterministic target policy except for the exit loop statement. This is fixed in the second edition (page 111).

after 10,000 iterations

Are your episodes really 10,000 time steps long? If so, the chances of off-policy MC control learning anything for early time steps seems remote unless $\epsilon$ is really low (and in which case $W$ will not get too high). If not, have you missed that $W \leftarrow 1$ occurs at the start of each episode?


You can use the argmax function to get the class labels. Please refer to this link for more information. Hope this answers your question.


How can I train a neural network for another input set, without losing the learning of the previous input set?

Yes, this is actually a limitation known as catastrophic forgetting.
A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on the weights important for those tasks". See Overcoming catastrophic forgetting in neural networks for details. Another approach is Learning without forgetting.

If the tasks are different, the approach you are talking about is called transfer learning. You might want to have a look at multi-task learning as well

If the tasks are the same, you could try creating a join of both datasets and training on that.


Is there a Python API for manipulating publicly available datasets?

scikit-learn has a small data sets API http://scikit-learn.org/stable/datasets/index.html
I imagine one can add more data sets locally.
Some data sets are for classification, other for regressions.

This is the only one I know about.


This two-part article by Trung Tran about Real Time Object Recognition demonstrates differences between Keras and OpenCV from a coding point of view is a reasonable treatment.  It also covers implementing pre-trained VGG16 models.

The application in this Campus Hippo lesson uses both Keras and OpenCV to simulate a self-driving car.

This tutorial by Adrian Rosebrockis about installing Keras for deep learning contains some good installation information to get both Keras and OpenCV running in the same virtual environment, which may be useful, especially for initial experimentation.

Being a user of Python3, I never limit my options to Python3 libraries.  Python is used as a scripting language in place of Bash scripts in operating systems today because it can call Javac, C or C++ executables, or any other library or executable and pass values, arrays, or matrices back and forth with ease and simplicity.

Regarding choice, you may wish to define what better means in your context.  If you wished to quantify or at least qualitatively describe acceptance or comparison criteria, there are a few typical dimensions to consider in your selection process.

Time to the provision of an answer
Accuracy (error margin, which may not apply in this case)
Reliability (pass/fail)
Minimum required preparation of images to achieve accuracy or reliability
Maintainability of the software that calls the library or framework
Current active participation in maintaining the library or framework
Projected future active maintenance based on trend of use and development

Tangential Comment:  For clarity, you may wish to specify more information about the machine numbers in the question.  We can only guess whether you mean some numbers rasterized using some font and kerning rules.


What are finite horizon look-ahead policies in reinforcement learning?

Per this paper a look ahead policy is a policy that will make decisions based on some 'horizon'. Here horizon means some time steps into the future, and so a finite horizon is simply a finite amount of time steps into the future. For example, as we are typically concerned with maximising returns in RL, a 10-step look ahead policy would choose an action at time $t$ that maximises the (expected) rewards at time $t+1, ... t+10$.


In addition to JCP's answer I would like to add some more detail. At best, max pooling is a less than optimal method to reduce feature matrix complexity and therefore over/under fitting and improve model generalization(for translation invariant classes).

However as JCP begins to hit on.. there are problems with this method. Hinton perhaps sums the issues in his talk here on what is wrong with CNNs. This also serves as motivation for his novel architecture capsule networks or just capsules.

As he talks about, the main problem is not translational variance per se but rather pose variance. CNNs with max pooling are more than capable of handling simple transformations like flips or rotation without too much trouble. The problem comes with complicated transforms, as features learned about a chair facing forwards, will not be too helpful towards class representation if the real-world examples contain chairs upside down, to the side, etc.

However there is much work being done here, mostly constrained to 2 areas. Those being, novel architectures/methods and inference of the 3d structure from images(via CNN tweaks). This problem was one of the bigger motivators for researchers throughout the decades, even David Marr with his primal sketches.


What purpose would be served by developing AI's that experience human-like emotions?

The answer to this question, unlike many on this board, I think is definitive. No. We don't need AI's to have emotion to be useful, as we can see by the numerous amount of AI's we already have that are useful.

But to further address the question, we can't really give AI's emotions. I think the closest we can get would be 'Can we make this AI act in a way a human would if that human was insert emotion?'. I guess in a sense, that is having emotion, but that's a whole other discussion.

And to what end? The only immediate reason coming to mind would be to create more lifelike companions or interactions, for the purposes of video games or other entertainment. A fair goal, but far from necessary. Even considering an AI-imbued greeting robot in the lobby of some building, we'd probably only ever want it to act cordial.

Yann says that super-advanced AI would lead to more human-like qualities and flaws. I think it's more like it would 'give our AI's more human-like qualities or in other words flaws'. People have a tendency to act irrationally when sad or angry, and for the most part we only want rational AI. To err is human, as they say.

The purpose of AI's and learning algorithms is to create systems that act or 'think' like humans, but better. Systems that can adapt or evolve, while messing up as little as possible. Emotive AI has uses, but it's certainly not a prerequisite for a useful system.


What are modern state-of-the-art solutions in prediction of time-series?

An interesting model I encountered in a course is Facebook Prophet. Prophet takes into account trends, seasonality, and holidays for its predictions. As you can probably guess, this is a model that fits Facebook's needs very well. I'll give a brief introduction then provide a link where you can read more. Prophet fits a couple of functions of time represented by a few terms. The general form of the timeseries predictions are,

$$y(t)=g(t)+s(t)+h(t)+\epsilon_t$$

$g(t)$ deals with the trends I mentioned above. This is exactly what you would think it is, and accounts for non-periodic features of the data. It takes the form of a piecewise linear or logistic function.

$s(t)$ accounts for seasonality - in other words these are periodic changes in our timeseries (maybe an increase of sunscreen purchases in the summer). As this is periodic, the natural way to model this term is with Fourier decomposition to identify important frequencies in the signal.

$h(t)$ deals with predictable changes in the timeseries but is for events like holidays (this can happen at different times year to year so this is not necessarily periodic, think Easter). The user provides a list of events and how they want to account for it.

$\epsilon_t$ is just an error term to deal with anything that cannot be addressed with the rest of the model.

This page has a great explanation if you want more detail. I highly recommend you check it out because it is very cool!


How to draw bounding boxes for gender classification?

In state-of-the-art papers. researcher managed to achieve 90-100% accuracy on gender classification on just human faces, so both method may work just fine, with minor improvements maybe using the first method. However using the image of the entire body enhances overfitting as there is more input features and increases overfitting. For a very large dataset it might be better to use the first method as chances of overfitting are low while for a smaller one you should probably go for the second method or even just the face as that won't make such a huge differnce.

A better way to input the data to the network is through landmarks of a body.

Inputting the landmarks of the body helps reduce the features to the minimal and also keeps only the relevant features. The main difference in body between women and men is perhaps the torso and shoulder width, which you can get from body landmark detection. You can input both the image of the face and the body landmark as input to the network, and it would probably increase in accuracy. However this solution is not an one stage method, meaning it requires an extra model to predict the landmark.

Hope I can help you.


Why does my model overfit on pseudo-random numbers training data?

Simply said, predicting pseudo random number is just not possible for now. Pseudo random numbers generated now have a high enough "randomness" so that it cannot be predicted. Pseudo random numbers is the basis of modern cryptography which is widely used in the world wide web and more. It may be possible in the future through faster computers and stronger AI, but for now it is not possible. If you train a model to fit on pseudo random numbers, the model will just overfit and thus creating a scenario as shown in the question. The training loss will be very low while test loss will be extremely high. The model will just "remember" the training data instead of generalising to all pseudo random numbers, thus the high test loss.

Also, as a side note, loss is not represented by %, instead it is just a raw numeric value.

See this stack exchange answer for details.