What are some tools for labelling images and videos?

There are a few.

Sloth
LabelBox
RectLabel
LabelMe

OpenCV also has some facilitation for annotating.

Annotation is the more common name used in software suites for tools that facilitate adding labels to images and frames of movies because.  Simple categorization is of limited use in the real world, especially with moving pictures where action may be important and requires recognizing consistent position and size change.

Annotation is more than adding a number or concept class instance in object recognition. There are five numbers that define the frame, potentially seven if both certainty and pixel depth layers are independently annotated as in the case of heat signatures.

$h_0$: Box horizontal min
$h_1$: Box horizontal max
$v_0$: Box vertical min
$v_1$: Box vertical max
$t$: Time or frame index
$P$: Optional probability that the object in the box qualifies if the annotation involves doubt
$\lambda$: Optional pixel layer depth indicating range of wavelengths, important for discriminating warm blooded people and pets from immobile objects for vehicle automation or the heat signatures to discriminate ballistic missiles from benign aeronautics


Is AI programming useful in everyday programs?

Yes, but probably only to a limited degree in the near term.

Where people draw the boundaries around 'artificial intelligence' is fuzzy, but if one takes the broad view, where it incorporates any sort of coding of explicitly cognitive functions, then many routine economic tasks can benefit from artificial intelligence. Many search engines, for example, can be seen as offering artificial intelligence applications as a service.

For more 'standard' applications, most near-team applications of AI have to deal with fraud detection and prevention. If you track a user's cursor moving across the screen, for example, you can build a model that differentiates between humans and bots, and treat the two separately. See this article for an example.

In the longer term, of course, a program that could write programs could write these sort of applications like any other.


How well can CNN for bounding box detection generalise?

I suggest you to go through the r-cnn paper or go through a tutorial on it . CNNs transform the image into high dimensional vector in their last layer , in case of classification this vector is sent to a "softmax" layer , in case of bounding box regression , four values :length , breadth , location of one of the points of the bounding box , are regressed from this vector , so if you use a cnn with one regression head you end up with one bounding box irrespective of the training set.


Can the hidden state of an RNN be a matrix?

Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are:
$$
h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h)
$$
$$
y_t = \sigma_y(W_y h_t + b_y)
$$
Here $W_h$ is input to hidden state matrix, and $U_h$ is hidden state to hidden state, $W_y$ is hidden state to output. One can simply upgrade these matrix products to more general case of tensor products, such that they could handle multidimensional input data and hidden state.
The contraction of the $m$-dimensional tensor $X$ with $n$-dimensional Y (we assume $m \geq n$) by maximal amount of indices will be $m-n$ dimensional tensor $Z$:
$$
X_{i_1 \ldots i_m} Y_{i_1 \ldots i_n} = Z_{i_{n+1} \ldots i_m}
$$
Namely, imagine, that the $x_t = (x_t)_{i_1 \ldots i_X}$ is an $d_x$-dimensional tensor. Then take $W_h$ to be  $d_{wh}$ dimensional tensor, hidden state $h$ to be $d_h = d_{wh} - d_{x}$ -dimensional, $U_h$ is $d_{uh}$ - dimensional, $W_y$ is $d_{wy}$ -dimensional. The output $y_t$ of the RNN will be $d_{wy} - d_{h}$ dimensional tensor.
Here note, that in order for the sum to be make sense in the first equation, dimensions need to match:
$$
d_{uh} - d_{h} = d_{h}
$$


Can genetic algorithms be used to learn to play multiple games of the same type?

Genetic algorithms and Neural Networks both are "general" methods, in the sense that they are not "domain-specific", they do not rely specifically on any domain knowledge of the game of Mario. So yes, if they can be used to successfully learn how to play Mario, it is likely that they can also be applied with similar success to other Platformers (or even completely different games). Of course, some games may be more complex than others. Learning Tic Tac Toe will likely be easier than Mario, and learning Mario will likely be easier than StarCraft. But in principle the techniques should be similarly applicable.

If you only want to learn in one environment (e.g., Mario), and then immediately play a different game without separately training again, that's much more complicated. For research in that area you'll want to look for Transfer Learning and/or Multi-Task learning. There has definitely been research there, with the latest developments that I'm aware of having been published yesterday (this is Deep Reinforcement Learning though, no GAs I think).

The most "famous" recent work on training Neural Networks to play games using Genetic Algorithms that I'm aware of is this work by Uber (blog post links to multiple papers). I'm not 100% sure if that really is the state of the art anymore, if it's the best work, etc... I didn't follow all the work on GAs in sufficient detail to tell for sure. It'll be relevant at least though.

I know there's also been quite a lot of work on AI in general for Mario / other platformers (for instance in venues such as the IEEE Conference on Computational Intelligence and Games, and the TCIAIG journal).


Advice on creating a new environment using OpenAI Gym

This stackoverflow post provides a good answer.  It recommends creating the environment as a new package.

Note the updated link to the instructions on OpenAI.

So that's what I did and it's honestly not that much work and then you can import the environment into your script like this:

import gym
import gym_foo
env = gym.make('foo-v0')

This blog post also describes the process in a little bit more detail.


How can we connect artificial intelligence with cognitive psychology?

AI is already connected with cognitive psychology - there are dozens of AIs right this minute attempting to predict things like which Facebook posts you will like, and which ads you are most likely to click on. In other words, they are trying to predict how you think.

For more detailed info on this AI/cognitive science connection, there is some suggested reading on AITopics.org, such as Paul Thagard's summary of cognitive science.


How does a robot protect its own existence

Right now, the scenarios where automata protect their own existence is limited.

In the case of autonomous vehicles, primary goals would certainly be collision avoidance, and other hazards (deep water, cliff faces, etc.)

It's possible that arial drones could have certain automated defense mechanisms specific to airborne threats such a missiles.

Bots might be said to protect their existence by copying themselves onto new systems.


If features are always positives, why do we use RELU activation functions?

The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too.

Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input.
Remember that only layer outputs, not their weights, are passed through ReLu, so, weights of a model could contain negative values.


How could facts be distinguished from opinions?

I will be starting my PhD in natural language processing in a few days and this is very similar to my proposed topic. It's an open problem that ties NLP and AI into philosophy of science and epistemology and is, I think, extremely interesting. I say all this to drive home the point that this is not a simple problem.
Two major theoretical concerns come to my mind:

What is a "fact"? Is it a universal truth, if there is such a thing? Or is it a generally accepted theory, and if so how do you measure acceptance? That is, accepted by whom, where, when?

Are there any linguistic markers of opinions vs. facts? Only in rare cases, such as when the speaker prefaces their statement with something like "I believe". In most cases, I think, opinions will be stated linguistically similarly to facts. For example, compare "Cats are felines." (a "fact") with "Cats are aliens." (an opinion some may hold). They have the exact same syntactic structure. The difference here is deeply semantic, and probably relates to the speaker's intention. I'd venture that often people state their opinions with the intention of communicating a "fact".

Some more practical concerns are:

Information extraction (also called relationship extraction, text mining, etc.), which for the most part assumes that the "facts" given in the labeled datasets are correct, is far from a solved problem. E.g. the state of the art model developed for a task released in 2010 has an F1 of only 76! What you propose adds significant uncertainty to these types of tasks.

I suspect that even if you were able to compile a dataset of facts and opinions with corresponding labels you would encounter a number of modeling problems. Given the linguistic similarity between the statements of facts and opinions, I'd guess that your model will simply memorize the dataset, making it generalize poorly to your test set. Either that or it would would pick up on random, hidden correlations in the data to solve the problem (neural nets are really good at this), perhaps generalizing to the test set, but failing to apply to any other data.

Fact vs. opinion is something that is embedded in a cultural milieu, so a model would, I think, need access to some proxy for what is culturally accepted in order to make this distinction, perhaps a via knowledge base. This may be feasible for limited, highly curated domains (e.g. biomedicine), but there is currently nothing suitable for a general-purpose fact finder.

tldr: No, it is not enough to simply create a dataset of facts vs. opinions. This problem poses major theoretical concerns related to epistemology, linguistics, and cognitive science. Additionally, there are more mundane (but non-trivial!) modeling issues to consider. @Sceptre is right that it will be impossible to start this without knowledge of AI/ML/NLP, especially a rather deep knowledge of what current AI systems are really capable of.


Could the deployment of self-driving cars make rides free?

You ask an interesting question.  There have been many discussions by industry on this topic.  A company called Vugo has their business model based on advertising to passengers.  An article about Vugo, "The Quest to Make Ridesharing Free", states:

Flessner is co-founder and CEO of Minneapolis-based rideshare advertising platform Vugo, which uses its patented TripIntent technology to display targeted advertisements to Uber and Lyft passengers on a tablet attached to the back of the vehicle’s headrests.
The company’s founders predict that within the next few years, all ridesharing vehicles in the U.S. will be driverless, and in-vehicle advertising tailored to passengers’ destinations and interests will lead to free transportation. “The idea is to put brands in front of passengers who are en route to make purchases,” says Flessner. For example, retailers will sponsor transportation to their stores so they can preview products to customers who are headed their way.

This article "How free self-driving car rides could change everything" makes these comments:

Car data is so lucrative that Ben Volkow -- CEO of otonomo, an Israeli startup that sells vehicle data -- expects automakers to make more money selling data than vehicles by 2020.
If the money made off self-driving vehicle data outweighs the costs of offering rides, then it becomes reasonable for a business to offer free rides broadly.

A paper entitled "Leveraging Adverts in the Coming
Autonomous Car Eco-system" published by Berkeley, University of California, proposes "a world where rideshares are free, or in some cases, heavily subsidized through the use of advertisements".


Is a Sobel filter for edge detection a type of Cellular Neural Network?

You're right about the basic arrangement of the inputs, but there are a number of differences:

Artificial neural networks typically use exemplar data as inputs for the purpose of training, or adjusting the weights of its internal connections, to accurately classify them within a certain error range. The network is then applied to unknown data to classify them. Edge detection filters are just blind operators that transform input data regardless of how it can be classified. There is no training, so any intelligence exists only in the mind of the filter developer.
A CNN could be trained to be an effective Sobel (edge detection) filter, as described in this paper, but a Sobel filter couldn't be an effective learning algorithm.
Training neural networks is more non-deterministic, with outputs depending on what data they are trained with and potentially even the operational computations that are used for classification. Applying filters is typically deterministic, i.e. they will transform the same data exactly the same way if applied twice.

One succinct way of expressing the biggest difference is: a cellular neural network is looking for a function, while a Sobel filter is a function.

Note that there are types of neural networks called Convolutional Neural Networks, which can use Sobel and other filters in their input layers, as described here. Though, these are neither of the things you are asking about. :)


Can I recognize the faces of people around the world?

I thought it might be a helpful place to start:
https://en.wikipedia.org/wiki/Google_Street_View_privacy_concerns
The wiki is well cited, so should lead to some useful tidbits.  They break it down by continent and country.
Seems like a parallel to what you're doing, although, if you're not making your data public, I doubt you'll be facing the same privacy issues.
Also relevant would be: https://en.wikipedia.org/wiki/Expectation_of_privacy

Examples of places where a person has a reasonable expectation of privacy are a person's residence or hotel room and public places which have been specifically provided by businesses or the public sector in order to ensure privacy, such as public restrooms...
In general, one cannot have a reasonable expectation of privacy in things held out to the public.

My understanding is that there is no expectation of privacy in public places where you'd be capturing the images.


5 years later, are maxout networks dead, and why?

Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout.

If you see the empirical results on Table 5 (of the page 5) of the maxout's original paper, you find that the misclassification rate is only very, very slightly lower than that of dropouts. (2.47 % instead of 2.78%)

That could explain the relatively lower interest in the work.


In MCTS, what to do if I do not want to simulate till the end of the game?

Famous example is AlphaZero. It doesn't do unrolls, but consults the value network for leaf node evaluation. The paper has the details on how the update is performed afterwards:

The leaf $s'$ position is expanded and evaluated only once by the network to gene-rate both prior probabilities and evaluation, $(P(s′ , \cdot),V(s ′ )) = f_\theta(s′ )$. Each edge $(s, a)$ traversed in the simulation is updated to increment its visit count $N(s, a)$, and to update its action value to the mean evaluation over these simulations, $Q(s,a) = \frac{1}{N(s,a)}\sum_{s,a\to s'}V(s')$  , where $s, a\to s′$ indicates that a simulation eventually reached s′ after taking move a from position s.


What is the variant of the minimax tree with 3 types of nodes called?

The triangles pointing up are Max' nodes. We assume it starts. Then follows a random choice of moves at the circles, for instance, with a die. The triangles pointing down are from Min. This variant is called Expectiminimax, see https://en.wikipedia.org/wiki/Expectiminimax_tree.

At that circles you have to multiply the possibilities on the edges below that nodes to your current value and sum all products up. The circles in your picture mean that Min dices.


Changing the batch, mini-batch and learning rate for a previously-trained model

Yes as far as I know you only need to reset the optimizer.


Finding patterns in binary files using deep learning

Your network is essentially memorizing data but not extracting features. You need to apply CNN.

That said the CNN architecture will need to be kind of unusual, each bit will need to be turned into representing a positional element.


Which hyperparameters in neural network are accesible to users adjustment

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:

learning rate,
batch size,
epochs,
optimizer,
layers,
activation functions etc.

To answer your (a) part of your question, there are obsiously many frameworks and libraries, for example in python; TensorFlow, pytorch and so on. You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms. When we call from scatch networks, we mean that these networks are trained from scratch, with learnable weights and chosen hyperparameters; with no transfer learning.
To answer your (b) part of your question, I can understand from it that you mean when a net is good enough. Dependently of your data, of course, a neural network is good enough, when it is trained adequately on them. That is, you should be aware of overfitting, underfitting, and in general of the model you are trying to train with all its parameters and hyperparameters.
Since, you are at the very beginning with Machine Learning, I propose you read some books, in order to get everything needed, in terms of Mathematical and Computer Science aspects.


Is MIRI doing genuine high-quality research?

Whether or not it's produced good research is a question I can't answer.  But I find it interesting and it seems reliable.

For instance, this paper on Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agent is drawn from Cornell University's site.  Likewise this recent paper on Logical Induction.

That said, they also seem to publish papers that have not been subject to peer review, which is an issue not specific to the site, and seems to more widespread phenomenon per easy access to digital files in general.

In relation to MIRI specifically, possibly this is a function of an an applied field advancing so rapidly that publication in journals is not seen as a priority, or even particularly relevant.


Historical weakness of GOFAI in relation to partisan combinatorial games?

Nice question!

I think there are a couple of issues at work here.

  Is the historical weakness of GOFAI in relation to non-trivial
  combinatorial games partly a function of the structure of the games
  studied, where game states and token values cannot be precisely
  quantified?

I think the short answer is yes. The real issue is in the last part:

  token values cannot be precisely quantified

The most successful GOFAI approaches to these games were all some variation on A* search, combining combinatorial search with some form of heuristic function that estimated the value of the pieces and their positions in any given state. Piece counting is probably a better heuristic than not counting anything at all, but it's still clearly incorrect, because a player with less material may still have an overwhelming positional advantage. Some heuristics can try to estimate this positional advantage as well however.

The real problem that GOFAI encounters in these games is that positional advantage can be emergent in ways that require incredible heuristic power to detect. Checkers is a good example. In the 1990's, the Chinook project at the University of Alberta set out to solve it completely. Checkers is notable because it had the same world champion for more than 15 consecutive years, Marion Tinsley. Tinsley lost a total of 7 competitive matches over 40 years of play. This makes him an especially interesting person to examine when we look at combinatorial games. Figuring out how Tinsley plays can help us understand how human intelligence works in games like this. In the course of solving checkers, the researchers noted that Tinsley was making moves that required up to 42 move lookaheads to reveal an advantage (See Schaeffer et al., AI Magazine, Vol. 17, Issue 1).

This strongly suggests that Tinsley was not methodically considering each possible move. Instead, by his own admission, his thinking was guided by a combination of memory over his 40 year career (in one match against Chinook in 1992, he indicated he was trying to recall a sequence from a match 30 years prior when making a move (AI Magazine Volume 14, Number 2); and of attentional heuristics (i.e. not thinking about every move sequence, and being able to reliably rule out parts of the search space without looking at them).

The key is that for GOFAI to solve checkers without heuristics (i.e. to solve it exactly), required enormous amounts of computational power, because some moves yield positional advantages that require 40+ moves of followthrough. Even an incredibly simple game (branching factor of 2) would be hard under that constraint.

In contrast though, self-play techniques like those pioneered in Backgammon with TD-Gammon (Tesauro, Comm. of the ACM 1995) mimic the process through which Tinsley became so good: they play lots and lots of games, learn a good heuristic estimate of position and material value, and more importantly, can learn to remember odd circumstances that require careful play. TD-Gammon achieved worldclass play despite only explicitly looking 2 moves ahead. GOFAI search techniques weren't even close despite searching much more deeply.

Modern research on attention could salvage the GOFAI approach however. If you can learn to tell what's important, you might be able to get a lot more value out of deeper lookaheads. This seems even closer to how Tinsley played: great ability to estimate value was used to guide an explicit analysis of a specific chain of moves.


What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?

Yes that is a major benefit of using a replay buffer.
A secondary benefit is the ability to use the same sample more than once. This can lead to beter sample efficiency, although that is not guaranteed.

What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?

The algorithm is still valid, but the gradient estimate for the update step would be based on a single record of [state, action, reward, next state]. This would be a high variance update process, with many steps in wrong directions, but in expectation over many steps you should still see a correct gradient. You would probably need to compensate for the high variance per sample by reducing the learning rate.
In addition, assuming the standard approach of collecting one time step then making one update to DQN neural network, each piece of experience would only be used once on average before being discarded.
These two effects will likely combine such that the learning process would not be very sample efficient.
The size of the minibatch is one of many hyperparameters you can change in DQN. It might be the case for some problems that choosing a low minibatch size is helpful, provided other adjustments (such as a lower learning rate) are made along with it. If you are not sure, you mostly have to try and see.
In my experience on a small range of problems, a moderate size of minibatch - ranging from 10 to 100 - has worked the best in terms of end results of high scoring agents. However, I have not spent a long time trying to make low batch sizes work.


Getting Negative Information Gain (using Entropy as a measure of disorder)

It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.


Intuitively, how does it make sense to take an action $A'$ when the environment already ended?

Intuitively, how does it make sense to take an action A' when the environment already ended?

It doesn't make sense, in that nothing can happen once the agent reaches a terminal state. However, it is often modelled as an "absorbing state" where the action is unimportant (either null or value ignored) with value by definition of $0$.

And you can't set all the terminal values to zero because you are using function approximators

The value is zero by definition. There is no need to approximate it. So don't use function approximators for action values in terminal states.  When $S'$ is terminal, the update becomes:
$Q(S,A) \leftarrow Q(S,A) + \alpha(R - Q(S,A))$
Look at any implementation of Q learning and you will see a conditional calculation for the update value, that uses some variant of the above logic when $S'$ is terminal. For OpenAI Gym environments for instance, it will use the done flag.


Recent methods for Decision Support System (DSS)

In my opinion, the sucess level of the Deep Learning models is perfectly suited for DSS. For example, if you have data you can built perfect Deep Reinforcement Model to rank programmers. In general AI models, specifically Deep models, are not mature enough to use without supervision of human being. But they are very suitable to build support systems.


A few questions regarding the difference between policy iteration and value iteration

$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us how much reward do we expect to get by taking action $a$ in state $s$ onwards.

For the value iteration on the right side with this update formula:

$v(s) \leftarrow \max_\limits{a} \sum_\limits{s'}p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$

we have an implicit greedy deterministic policy that updates value of state $s$ based on the greedy action that gives us the biggest expected return. When the value iteration converges to its values based on greedy behaviour after $n$ iterations we can get the explicit optimal policy with:

$\pi(s) = \arg \max_\limits{a} \sum_\limits{s'} p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$

here we are basically saying that the action that has highest expected reward for state $s$ will have probability of 1, and all other actions in action space will have probability of 0

For the policy evaluation on the left side with this update formula:

$v(s) \leftarrow \sum_\limits{s'}p(s'\mid s, \pi(s))[r(s, \pi(s), s') + \gamma v(s')]$

we have an explicit policy $\pi$ that is not greedy in general case in the beginning. That policy is usually randomly initialized so the actions that it takes will not be greedy, it means we can start with policy that takes some pretty bad actions. It also does not need to be deterministic but I guess in this case it is. Here we are updating value of state $s$ according to the current policy $\pi$.
After policy evaluation step ran for $n$ iterations we start with the policy improvement step:

$\pi(s) = \arg \max_\limits{a} \sum_\limits{s'} p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$

here we are greedily updating our policy based on the values of states that we got through policy evaluation step. It is guaranteed that our policy will improve but it is not guaranteed that our policy will be optimal after only one policy improvement step. After improvement step we do the evaluation step for new improved policy and after that we again do the improvement step and so on until we converge to the optimal policy


How can I benchmark an application of ant colony optimization?

I would assume that in this case, they are asking you to compare the performance of your particular algorithm to other similar algorithms used in pathfinding. For example, A*


Have capsule neural networks been used to NLP problems?

For us to answer this question. First, we need to look at why capsule networks outperform convolution neural networks by as much as 45% in recognizing images that have been rotated, translated or are under a different pose. We can find Geof Hinton's paper on capsule networks here for reference https://arxiv.org/pdf/1710.09829v1.pdf

In a CNN architecture, a convolution layer is usually followed by a max-pooling layer. This is so that the lower levels can detect low-level features, like edges, while the high-level layers can detect abstraction like eyes. However, the application of max-pooling leads to the loss of important information regarding the location and spatial relationship between certain features.

On the other hand, this is where capsule networks excel, the way they represent certain features is locally invariant. This is why capsule networks can recognize images under different lighting conditions and deformations. They are likely to excel at applications such as video and object tracking but not necessarily NLP.

The current approach in NLP maps words and phrases to vectors. From there, we exploit the concept of vectors and distances between them (cosine, euclidean, etc.) to perform operations such as: finding the similarity between words and even documents, machine translation, and natural language understanding (NLU).

Capsule networks are unlikely to succeed in NLP. This is because algorithms that aim to find the hierarchical structure of natural languages or approaches that focus on grammar have met little success. Research by Stanford University aiming at finding the hierarchical structure of natural languages can be found here https://nlp.stanford.edu/projects/project-induction.shtml

Although conclusive research regarding other applications of capsule networks has not yet been conducted. They are likely to excel at applications such as video intelligence and object tracking but not necessarily NLP.


Why is tree search/planning used in reinforcement learning?

Some sources say MCTS (or planning in general) increases the sample efficiency.

If we're thinking purely about experiments run in simulations, then I'd estimate there may be cases where a combination of pure learning + MCTS (or some other form of planning / model-based aspect) may be more efficient, and there may be different cases where only a single one of those techniques on its own may perform better. So then I wouldn't say this is always necessarily true.
Often though, when saying "sample efficiency", we would only count the steps actually taken in a "real" environment, but maybe not count steps taken in a lookahead search or planning algorithm. In pure simulations this may seem like a weird distinction to make, but it's more sensible when you consider that we often use simulations just because they're convenient ways to evaluate and compare new algorithms, but often not the end goal. Often, the end goal would be to apply something in the "real world", for example on a robot or something. In such a situation, collecting pure learning experiences can be very expensive (time-consuming, maybe also risky because the robot may fall over and break, etc.). But you may be able to also provide that robot with a learned model, or simulator, and have the robot use that to also run its own search or planning algorithms on an approximated version of the real world. This is a clear case where performing such is much faster and cheaper and less risky than collecting true experiences in the real world for pure RL.

Assumed the transition model is known and the computational cost of interacting through planning is the same as interacting with the environment, I do not see the difference between playing many games versus playing a single game, but plan at each step.

Jumping back to the case where we actually are working purely in simulation, and where there's no difference in computational cost between steps taken for pure learning vs. steps taken in search/planning, there absolutely are still some differences. If you run search, you use extra time and have temporary extra memory usage (which frees up again after completing your search algorithm) to make one really good decision in the "main" environment. You could view the steps taken in MCTS simulations as a form of learning as well, but they have a different purpose from the steps taken in a pure RL setting; these steps are taken with the sole goal of learning how to act well in the root state. All search time and memory usage is dedicated to that single decision. This can enable smarter decision-making than if you're going purely off of what you learned through pure RL, due to 1) focusing more effort on just a single decision, and 2) not being constrained by the capacity of your learning algorithm to actually learn a good decision (simple function approximators may simply not be capable of accurately representing strong policies, and more complex function approximators will take a huge amount of time to learn). This smarter decision-making thanks to search can in turn also improve the quality of the experience used by your pure RL component.

Finally, since you started out by mentioning AlphaGo Zero, I'd like to emphasise that AlphaGo Zero and similar approaches are typically used in multi-agent adversarial domains (like zero-sum games). Pure RL approaches for such multi-agent settings do exist, but there has been significantly less research towards them than pure RL approaches for single-agent settings. Most of these pure RL approaches are really only applicable to single-agent settings, and can easily have poor performance when applied to these kinds of multi-agent settings. Search algorithms like MCTS on the other hand are very well-established techniques in these adversarial domains like games, and the combination of them with learning approaches appears to allow even for learning approaches to be used which are not explicitly "aware" of the fact that they're operating in such a multi-agent domain.


In ensemble learning, does accuracy increase depending on the number of models you want to combine?

Yes, you can. Let's say you have 5 classes named a,b,c,d,e. You fit your data into a SVM Classifier and a Random Forest Classifier. Assume that, SVM classified "a" and "b" class well and RFC classified "c","d","e" well. So, ensembling these two models is going to increase accuracy  dramatically. Ensemble learning is really good when a model generating many "False-Positive" and "False-Negative". You can also use weighted ensemble learning methods. You can set greater weights for reliable models & lower weights for untrustable models.


What does a consistent heuristic become if an edge is removed in A*?

Consistency is a property of heuristics. You can think of consistency as the common sense idea that our guess at the time to go from $A \rightarrow B \rightarrow C$ cannot be more than the time to go from $A \rightarrow B$, plus our guess of the time to go from $B \rightarrow C$.

Supposing we remove a given edge $c(n,m)$ from our graph, but that our heuristic function gives the same value for $h(n)$ as before. Can the function be inconsistent? Let's try a proof sketch:

Since the function was consistent before we removed this edge, it must already be the case that $h(n) < c(n, m') + h(m')$ for every other node $m'$ adjacent to $n$.
Removing the edge $c(n,m)$ doesn't change these relationships, because the other edges still have the same costs as before, and the heuristic function still gives the same value for every node.
The same argument may be applied to node $m$.
Since none of these relationships have changed, $h$ is still consistent.


Which unsupervised learning technique can be used for anomaly detection in a time series?

So if I understood correctly:
You have data from 2 sensors in time: Ar flow and BackGas Flow (SCCM, what is that?)
You have that data for multiple products.
1 - Since it is relatively low dimensional, you may try using raw data with K-Means or Self Organizing Maps.
2 - If you searching for anomalies in time, you might try using feature engineering with things like flow change:

i) Take 2 points in time and "derivate" (Point A - Point B)/(Time A - Time B)
ii) Take the result for A and repeat the process as a second derivative, try that for many levels of "derivative"
iii) Machine learning application for sensor failure detection in polymerization process. In: Simpósio Brasileiro de Telecomunicações e Processamento de Sinais.

3 - You might want to check other Time-Series related research, even on regression and classification, since they may give you ideas on relevant features or approaches:

i) An enseble approach to time dependent classification (10.1109/ICMLA.2018.00164)
ii) Multivariate Time Series for Data-driven Endpoint Prediction in the Blast Oxygen Furnace (DOI: 10.1109/ICMLA.2018.00231)

4 - Note that anomalies are outliers, so analysing data distant from the mean of any features you might engineer from the time. An model based on RBF kernels might pick up some information.
5 - Since you don't know the amount of clusters you can try hierarchical clustering.
6 - Don't forget to talk to people in your field about what to expect from this.
If you can elaborate on the type of anomalies you're looking for I may come with better ideas. This is pretty general tips.
I may have misunderstood what you wanted to cluster:
7 - If you want to detect anomalies in:

i) products behavior tips above were mostly for it and you might use all 23 features in short time intervals and cluster like this:

    For each product in dataset {

      samples = {} # empty set

      For each time interval in product {

        features in interval add to samples

      }

      perform clustering method using samples

    }

This algorithm would help cluster periods of time where the product behavied weirdly

ii) sensors behavior then for something like that you might try tip 8

iii) products itself create a feature vector with all avaiable information for each product, then cluster that. Since it is composed of variable time interval and 23 features per product (plus time) you might want to use a bit of dimensionality reduction (PCA should work fine, since it gives you the direction of greater variance and you're look for that). Using a regressor to model products behavior and cluster that could all be usefull (create a regressor in time for each product to model its behavior, then cluster the weights of the regressors as the product representation)

8 - Let's say you want to identify sensors behavior, it might be helpful to model the sensors output as a function f(time, extravariables) and do regression over it (try linear, then try non-linear). Points in time that the regressor had bad performance predicting the answer might indicate that that behavior is anomalous


Is the gradient at a layer independent of the activations of the previous layers?

Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.

It is however impacted by the gradient at $j_{n+1}$, which the back-prop algorithm also follows.


AFAIK, pain is produced by the nervous system (after the realease of chemicals), so no AI suffers. AI does not (yet) possess a life (and I don't think this will ever happen).

A definition of life from the dictionary:

  the condition that distinguishes animals and plants from inorganic matter, including the capacity for growth, reproduction, functional activity, and continual change preceding death.

The current AI definitely does not possess a life according to this definition. AI is inorganic (like a rock): it is based on silicon (not carbon atoms).


This classic problem exhibits a basic misunderstanding of what an artificial general intelligence would likely entail. First, consider this programmer's joke:

The programmer's wife couldn't take it anymore. Every discussion with her husband turned into an argument over semantics, picking over every piece of trivial detail. One day she sent him to the grocery store to pick up some eggs. On his way out the door,  she said, "While you are there, pick up milk."
And he never returned.

It's a cute play on words, but it isn't terribly realistic.
You are assuming because AI is being executed by a computer, it must exhibit this same level of linear, unwavering pedantry outlined in this joke. But AI isn't simply some long-winded computer program hard-coded with enough if-statements and while-loops to account for every possible input and follow the prescribe results.
while (command not completed)
     find solution()

This would not be strong AI.
In any classic definition of artificial general intelligence, you are creating a system that mimics some form of cognition that exhibits problem solving and adaptive learning (←note this phrase here). I would suggest that any AI that could get stuck in such an "infinite loop" isn't a learning AI at all. It's just a buggy inference engine.
Essentially, you are endowing a program of currently-unreachable sophistication with an inability to postulate if there is a solution to a simple problem at all. I can just as easily say "walk through that closed door" or "pick yourself up off the ground" or even "turn on that pencil" — and present a similar conundrum.

"Everything I say is false." — The Liar's Paradox


You are looking for incremental (or online) learning.

A CNN can be trained incrementally. For example, in the paper Incremental Learning of Convolutional Neural Networks, the authors propose an incremental learning algorithm (inspired by AdaBoost and Learn++, which is another incremental learning algorithm for supervised learning of neural networks) for CNNs.

However, note that incremental learning is a challenging task, given the  stability-plasticity dilemma: a completely stable model, in order to keep being stable, will attempt to preserve the existing knowledge, so it will not learn new knowledge; similarly, a completely plastic model, in order to keep being plastic, it will keep forgetting previously acquired knowledge so that to learn new information.


solving xor function using a neural network with no hidden layers

Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, exp(xi)/sum(exp(xj) for j in all x), where xi is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)).

Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space.
, [src: https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]

Now, whatever you do, you cannot separate the points with a single wx+b straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped wx+b.

Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be (w2(w1*x + b1)+b2, which again is a linear equation not able to separate the classes 0 and 1.

So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get w2*(1/+exp(w2*x+b1))+b2 which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.

Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?


What are the conceptual differences between regularisation and optimisation in deep neural nets?

You are correct.
The main conceptual difference is that optimization is about finding the set of parameters/weights that maximizes/minimizes some objective function (which can also include a regularization term), while regularization is about limiting the values that your parameters can take during the optimization/learning/training, so optimization with regularisation (especially, with $L_1$ and $L_2$ regularization) can be thought of as constrained optimization, but, in some cases, such as dropout, it can also be thought of as a way of introducing noise in the training process.
You should use regularisation when your neural network is big (where big, of course, is not well-defined) and you have little data (where little is also not well-defined). Why do you want to use regularisation in this case? Because big neural networks have more capacity, so they can memorize the training data more easily (i.e. they can over-fit the training data). If the training data is not representative of the whole data distribution (that you are trying to capture with your neural network), then your neural network may fail on other data from that data distribution, i.e. it may not generalize. Regularization techniques, such as $L_1$ and $L_2$ penalties and dropout, limit the complexity of the functions that can be represented by your neural network (remember that, for a specific set of weights $\theta$, a neural network represents a specific function $f_\theta$), so they prevent your neural network from learning a complicated function that would just over-fit the training data.
Of course, you can also use regularisation when you have a very large dataset or your neural network is not that big. However, in principle, the case mentioned above is the case where you will likely need some kind of regularisation. So, as a rule of thumb, the bigger your NN is and the smaller your training dataset is, the more likely you will need some kind of regularisation.
The typical way to visualize that your neural network is over-fitting is to look at the evolution of the loss function (a type of objective function, which you want to minimize) for the training dataset and the validation dataset (a dataset that you do not use for training, i.e. updating the weights of the neural network). If the training loss is very small, while the validation loss is bigger (stays constant or even increases), as you train more the neural network, that's a good sign of over-fitting, and it suggests that you may need some kind of regularisation. However, note that, even with regularisation, it is not guaranteed that you will find a good set of weights that is able to generalize to all data (not seen during training), but it's more likely.
There are other forms of regularization, such as the KL divergence in the context of Bayesian neural networks or variational auto-encoders, but these are more advanced topics that you don't need to know now. In any case, the KL divergence has the same role as the other regularisation techniques that I mentioned above, i.e., in some way, it restricts/limits the possible functions that you can learn.


Finding the optimal combination of inputs which return maximal output

If your model is gradient-based, such as a neural network, then may also be able to use gradient methods to drive virtual inputs:

Freeze all network weights to the trained version
Define a loss function that decribes how you want the output - or any internal measure - to behave. E.g. to maximise the output, the loss function can simply be the negative of the output, assuming you will perform gradient descent later. Some libraries will also support gradient ascent to maximise a function.
Define your input as a variable that can be optimised and instantiate an optimiser (details will vary depending on your library)
Start with a random or best-guess input, and iterate normal training routine (feed forward then backpropagate) to get better and better inputs

This is basically how Deep Dream and Style Transfer algorithms work - the detail that is different is definition of the loss functions. It is also a way to make adversarial attacks against known models, for example taking a picture of a car, modifying it such that a classifier returns that it is an ostrich (whilst it still looks like a car to a human).

It is not guaranteed to find the absolute best inputs for a given range, but should find good approximations of local minima or maxima far faster than a meshgrid when there are many dimensions. You could combine the idea with a coarse meshgrid or random search for start points for a better chance of finding the best results within input range constraints.

You should bear in mind that the ideal inputs you discover will only be as accurate as your trained model will let them be. If you discover maximising inputs in the model, that use inputs that are far away from any training examples, then in reality those inputs might not get anything like the result that the model predicts. Statistical models with many degrees of freedom are typically ok at interpolating between data points, but very bad at extrapolating beyond data that has been observed.


Should the input to the negative log likelihood loss function be probabilities?

This seems pretty reasonable to me. You can optimize for any function that is proportionate to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.

If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:

$$\sum_y (y-\hat{y})^2$$

This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.

Hope that helps!


Why do we have two similar action selection strategies for UCB1?

In the PDF of the original paper for UCB1 you linked, in page 242-243 the authors proves why non-optimal machines get played much less (in fact, logarithmically less) than the optimal ones. $c$ decides whether they indeed would, and $c=\sqrt{2}$ is the minimum choice of $c$.
We want to show that the number of runs for non-optimal machines ($n_i$, for non-optimal $i$s, in your notation) is asymptotically logarithmic. In other words, you may run them for a few times and well, it's fine, but not too often. We're devising some indicator value $a_i(t)=\hat \mu_i+(\epsilon...)$ such that the mistaken cases, where values of non-optimal ones surpass values of optimal ones ($a^*(t)<a_i(t)$), are minimized.
Think about the last inequality. We know that $\mu^* > \mu_i$ (again, optimal and non-optimal ones). Therefore, for that inequality to be true, it seems either the left-hand side should be quite small or the right-hand side should be quite large. But wait, $\hat \mu$s are actually some random trials for $\mu$, so we cannot claim directly from $\mu^* > \mu_i$ to $\hat{\mu^*} > \hat{\mu_i}$; it might be that we just need more trials.
The equations (7), (8) and (9) of the paper is the three conditions mentioned in the paragraph above; left-hand side is small, right-hand side is large or the trials are lacking. Well, in fact, as we stated the number of runs ... is asymptotically logarithmic at first, the third case can be eliminated(!), assuming that we've run this machine enough.
For the first and second case, since $\hat \mu_i$ is the average of some random variable in $[0, 1]$, we can use Chernoff–Hoeffding bound (or so called in the paper; stated as Hoeffding's inequality in Wikipedia). Now, a good choice of $(\epsilon ...)$ will guarantee (from Hoeffding's inequality) that the first two cases will occur sufficiently scarcely, or in other words, in the order of $t^{-4}$. To achieve this, we need $c \ge \sqrt 2$.
Now back to the third case, the enough number of runs is actually $l = \left \lceil 2 c^2 \ln n / (\mu^* - \mu_i)^2\right \rceil$. Thus, you may choose larger $c$ but receive longer convergence speed in penalty.
Funnily enough, after all the proofs the authors find $c=1/4$ to converge well and actually perform substantially better(!!) than $c=\sqrt{2}$. It seems they could not prove the bound as we did above.


Does this diagram represent several LSTMs, or one through several timesteps?

are the LSTM cells on a given row all the same cell, with time flowing forward from left to right?

Yes this is correct
The x-axis on this figure is basically the time axis. Essentially all pink boxes in the same row are the same LSTM cell, with different inputs from the same sequence. At each timestep, the cell takes an input and produces an output which is fed to the next layer. At the 8-th layer, the outputs over all timesteps are inputted at the same time to the attention layer.


Training an AI to play Starcraft 2 with superhuman level of performance?

StarCraft II is a real time strategy game that combines fast paced micro actions with the need for high level planning and execution. StarCraft II being a popular game with millions of users it proceeds that defeating top players becomes a meaningful and measurable long term objective in AI research.

Computer games provide a compelling solution to the issue of evaluating and comparing different learning and planning approaches on standardized tasks. They are an important source of challenges for research in AI.

Game playing AI agents i.e. deepmind's Atarinet and DQN alongside Open AI's Dota 2 bot represent the first demonstration of a General Purpose Agent that is able to continually adapt behavior without any human intervention, a major technical step forward in the quest for general AI (Source deepmind blog).

Computer games offer numerous advantages in AI research i.e:

They have clear objective measures of success.
Computer games typically output rich streams of observational data, which are ideal inputs for deep networks.
They are externally defined to be difficult and interesting for a human to play. Therefore they provide an excellent test for intelligence.
Games are designed to be able to run anywhere with the same interface and game dynamics. This enables running many simulations in parallel. Sharing and updating the same table throughout training.
In some cases pools of superb human players exist, making it possible to benchmark against highly skilled humans.

The Starcraft challenge for reinforcement learning, introduces a taxing set of problems because it is a multi-agent problem with multiple players interacting. There is imperfect information due to a partially observed map, it has a large state space, it has delayed credit assignment requiring long term strategies.

Tools

The SC2LE Environment

DeepMind and Blizard games have collaborated to release the SC2LE, which exposes StarCraft II as a research environment.

The SC2LE consists of three sub-components.

A Linux Starcraft II binary.
StarCraft II API which allows programmatic control of StarCraft II. The API can be used to start the game, get observations, take actions and review replays.
PySC25 which is an open source environment written in Python. It includes some mini-games and visualization tools

Open source Open AI RL environments

Universe - Universe is a software platform by Open AI for measuring and training an AI's general intelligence across games, websites and other applications.

Gym - Open AI Gym is a toolkit for developing and comparing reinforcement learning algorithms. It makes no assumptions about the structure of your agent and is compatible with any numerical computation library such as Tensorflow or Theano.

Supervised Classification Approach

Consider this, we could decide to screen capture game sessions from expert players and use it as input to a model. The output could be the direction in which the AI agent could move. This would be a supervised classification approach.

However, this is not an elegant solution because we are training a model not on a static dataset but a dynamic one (game environment). The training data from a game environment is stochastic/continuous meaning any number of events can occur.
Furthermore, humans learn most effectively by interacting with the environment. Not by watching others interact with the environment.

Markov Decision Process

Markov Decision Processes (MDPs) provide a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker i.e. game environments.

Reinforcement Learning with Deep Q-Learning

Q-Learning is a strategy that has been proven to find an optimal action selection policy for any Markov Decision Process (MDP). In Q-Learning we choose an action that maximizes future reward. The further in the future we go, the further the rewards can diverge, we resolve this by adding a discount in future rewards.

  Unlike policy gradient methods, which attempt to learn functions which directly map an observation to an action, Q-Learning attempts to learn the value of being in a given state, and taking a specific action there. (Arthur J 2016)

The formula for Q-Learning is:

  Where:

  R = Reward

  s = State

  a = Action

  Experience during learning is based on (s, a) pairs

  One has an array Q and uses experience to update it directly

  (Source wikipedia https://en.wikipedia.org/wiki/Markov_decision_process)

One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment.

For further reference, I recommend you look at Siraj Ravals tutorial on Deep Q-Learning

https://www.youtube.com/watch?v=79pmNdyxEGo and source code for the same available here https://github.com/llSourcell/deep_q_learning

Additionally I recommend the following references for more information on computer game playing AI agents.

StarCraft II: A New Challenge for Reinforcement Learning https://arxiv.org/abs/1708.04782

Playing Atari with Deep Reinforcement Learning https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Human-level control through deep reinforcement learning  https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf


How can a neural network learn when the derivative of the activation function is 0?

In a setup like the above where the derivat[iv]e is 0 is it true that an NN won´t learn anything?

There are a couple of adjustments to gradients that might apply if you do this in a standard framework:

Momentum may cause weights to continue changing if any recent ones were non-zero. This is typically implemented as a rolling mean of recent gradients.
Weight decay (aka L2 weight regularisation) is often implemented as an additional gradient term and may adjust weights down even in the absence of signals from prediction errors.

If either of these extensions to basic gradient descent are active, or anything similar, then it is possible for the neural network to move out of the stationary zone that you have created after a few steps and then continue learning.

Otherwise, then yes it is correct that the neural networks weights would not change at all through gradient descent, and the NN would remain unchanged for any of your input values. Your careful initialisation of biases and weights will have created a system that is unable to learn from the given data. This is a known problem with ReLU activation, and can happen to some percentage of artificial neurons during training with normal start conditions. Other activation functions such as sigmoid have similar problems - although the gradient is never zero in many of these, it can be arbitrarily low, so it is possible for parameters to get into a state where learning is so slow that the NN, whilst technically learning something on each iteration, is effectively stuck. It is not always easy to tell the difference between these unwanted states of a NN and the goal of finding a useful minimum error.


In variational autoencoders, what does p(x|z) mean?

Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities here, rather they are referring to density functions (though this should really be denoted with a lower case $p$ to avoid confusion such as this).
$p(x|z)$ is a conditional distribution, which is also allowed in the continuous case -- you can also 'mix and match', i.e. $x$ could be continuous and $z$ could be discrete, and vice-versa.
In the paper, all the authors are meaning when they write $p(x|z)$ is the density of $x$ conditioned on $z$; in VAE's with an image application this is the conditional density of the image $x$ given your latent vector $z$.


What is the best activation function for the embedding layer in a deep auto-encoder?

It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.
In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.
http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf


Overfitting nearly always occurs to some degree when fitting to limited data sets, and neural networks are very prone to it. However neither of your graphs show a major problem with overfitting - that is usually obvious when epoch counts increase, the results on training data continue to improve whilst the results on cross validation get progressively worse. Your validation results do not do that, and appear to remain stable.

It is usually pragmatic to accept that there will be at least some difference between measurements on the training set and cross validation or test sets. The primary goal is usually to get the best measurements in test that you can. With that in mind, you are usually only interested in how much you are overfitting if it implies you could improve performance by using techniques to reduce overfitting e.g. various forms of regularisation.

Without knowing your data set or known good results, it is hard to tell whether the difference you are seeing between test and train in accuracy could be improved. Your accuracy graph shows a train accuracy close to 100% and a validation accuracy close to ~96%. It looks a bit like MNIST results, and if I saw that result on MNIST I would suspect something was wrong, and it might be fixed by looking at regularisation (but it might also be something esle). However, that's only because I know that 99.7% accuracy is possible in test - on other problems I might be very happy with 96% accuracy.

The loss graph is not very useful, since the scale has completely lost any difference there might be between training and validation. You should probably re-scale it to show detail close to 0 loss, and ignore the earlier large loss values.


Why do DQNs use linear activations on cartpole?

Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.

DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to perform regression to predict the action values.

It is most common to use a linear final layer, and mean squared error loss in DQN, to match this regression task. So yes, you will find most control DQNs will make the same choice as the cartpole example that you are looking at.


Why was ELIZA able to induce "delusional thinking"?

I like your choice of "induce" instead of "produce," because the delusions came from the users. This means the answer has to do mostly with human psychology; people come equipped with lots of mental machinery specialized for dealing with other humans and not very much mental machinery specialized for dealing with software. So ELIZA behaved in ways that some people classified it as a person and behaved accordingly, and others didn't.

What features will trip up a person's internal person classification system seem like they vary heavily from person to person, and also with experience and familiarity. Going into more detail is, as mentioned in the comments, more appropriate for sites specializing on the human side of the keyboard.


Neural Nets: CNN confirming layer/filter arithmetic

Your first point is correct. The filters are stored in 4d arrays, with dimensions of (height, width, input channels, filter number) . The order may differ.
Your second point is correct too. The filtered result get stacked together so the output dimensions are (Height,width,filter numbers) the next layer's filters are of size( filter width, filter width, last layer's filter number). Your understanding of CNN is correct. If you want additional resources on CNN, you can try Andrew Ng's class on CNN in Coursera. Hope you can learn more about CNN.


Are there any applications of reinforcement learning other than games?

You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.

  Is there a way to teach reinforcement learning in applications other than games?

Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to Markov Decision Processes, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.

Here are some examples of possible uses for reinforcement learning outside of recreational games:

Control logic for motorised robot, such as learning to flip pancakes and other examples. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.
Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.
Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.

  is it possible to set this up with say a CAD software?

In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.

CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.


What should load_mask() return if an image doesn't have any objects? (Mask RCNN)

It should return an empty mask. Like

mask = np.empty([0, 0, 0])
class_ids = np.empty([0], np.int32)


Indoor positioning with variable number of distance measurements in tensorflow

The solution mentioned seems feasible but you'd probably encounter a lot of problems, such as -

Since you're outputting coordinates, each one of the input networks must be trained differently. Considering $N$ should ideally be a variable, how many input networks do you train?
You'd use an intermediate encoding of sorts as co-ordinate representation which would be averaged and then passed through the output network. - Interpolation properties of the encoding might not be too great, meaning a linear change in the representation might not lead to a linear change in the co-ordinate, causing an averaging function to give skewed results.

Just a suggestion but I think it would be better if instead of learning multiple estimates of co-ordinate representations and then averaging them it might be better if you tried to learn some sort of distance function.

Essentially, each receiver would have a fixed position (Assumption) and the input network (or two input networks preferably) would output the distance  ($r$) of the transmitter from that receiver and the angle ($\theta$) it makes with a common axis. The problem with this approach is you wouldn't be leveraging multiple receivers as they would not be learning together, and the advantage being that whatever $N$ is, the RSSI readings pass through the same network always.

Once an $N$ number of $(r, \theta)$ values are obtained - a robust algorithm
$f(r_1, r_2,..., \theta_1, \theta_2, ...) = (x_t, y_t)$ could be found to output the transmitter co-ordinates.

I'm not too confident if it would work too well but just a suggested direction I could think of! Hope this helped!


Is it possible that the model is overfitting when the training and validation accuracy increase?

I'll try to answer on more general questions

Is it ok that model performs better on validation, then on train?

It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.

I'm suspicious my model is too good. What could I do?

That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.
Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.
So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.
More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.
Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. http://karpathy.github.io/2019/04/25/recipe/


What are some examples of classical AI methods?

If you're looking for AI systems:

Deep Blue
Eliza

If you're looking for specific algorithms:

Bayesian Networks
Decision Trees
Any search algorithm


In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components and feeding that knowledge into the algorithm, then you place more emphasis on your understanding of the problem, and less on any learning that an agent might do.

Typically in a RL formulation of a simple board game, you would choose rewards or +1 for a win (the goal), 0 for a draw, and -1 for a loss. All non-terminal states would score 0 reward. The point of the RL learning algorithm is that the learning process would assign some nominal value to interim states due to observing play. For value-based RL approaches, such as Q learning or Monte Carlo Control, the algorithm does this more or less directly by "backing up" rewards that it experiences in later states into average value estimates for earlier states.

Most game-playing agents will combine the learning process, which will be imperfect given the limited experience an agent can obtain compared to all possible board states, with a look-ahead search method. Your heuristic scores would also make a reasonable input to a search method too - the difference being you may need to search more deeply using your simple heuristic than if you used a learned heuristic. The simplest heuristic would just be +1 for a win, 0 for everything else, and is still reasonably effective for Connect 4 if you can make it search e.g. 10 moves ahead.

The combination of deep Q learning and negamax search is quite effective in Connect 4. It can make near perfect agents. However, if you actually want a perfect agent, you are better off skipping the self-learning approach and working on optimised look-ahead search with some depth of opening moves stored as data (because search is too expesnive in the early game, even for a simple game like Connect 4).


What are the best machine learning models for music composition?

There are a few of them. The most recent I've found is from DeepMind: The challenge of realistic music generation: modelling raw audio at scale. This video is a great analysis of it.


What is the goal of the model and is the training data relevant to that?

Model means you can say a Prototype we make regarding to our task.
As we first train our model on some observed or you can say bench-marked data ; called as TRaining phase of model.

Then we apply that model to our problem (test data) you can say in order to evaluate how  much well you have trained your model.
Training data we use related to our task or use bench marked data for any model of AI.
hope you find this answer useful .


What are examples of problems where neural networks have achieved human-level or higher performance?

Here is an initial list of AI systems that used neural networks and have achieved human-level or superhuman performance. All of these systems are reinforcement learning systems that play videogames.

AlphaGo and AlphaGo Zero (an improved version of AlphaGo that does not use human knowledge but learns by playing against itself) have achieved superhuman performance in the game of go and, in the case of AlphaZero (a generalized version of AlphaGo Zero), also in the games of chess and shogi.
DQN has achieved human-level or superhuman performance in many Atari games
DeepStack has achieved human-level performance in poker
AlphaStar defeated a top professional player in the real-time strategy game StarCraft 2
OpenAI Five defeated world champions in the game of Dota


Dice loss gives binary output whereas binary crossentropy produces probability output map

The probability map / output isn't produced by your loss function, but your output layer, which is activated either by softmax or sigmoid.

In other words, your dice loss output is also a probability map. It's simply very confident in itself. If you forget about the problem with potential overfitting for a moment and train your binary crossentropy model longer, the probability values will eventually all converge to the 2 ends (0 and 1).

In my experience, dice loss and IOU tend to converge much faster than binary crossentropy for semantic segmentation, so if you stop the training early on, dice loss will produce a probability map that resembles a binarized output more so than binary crossentropy.


How does Microsoft use AI to make Windows 10 updates smoother

Hint :  According to antlersoft's comment which says

  ".....Microsoft  may well have used a neural network classifier to
  scan through it's telemetry  to find good  update candidates, it's
  clearly a classification problem.... "

This could be right,however,here questions (information) concerning the company's internal research and development of their software products is classified.

Any machine learning engineer; is not professionally  allowed to discuss nor give out information pertaining the company's product i.e. source codes,plan/design or even the flow of the conceptual idea on how the system is implemented,and this is inline with what the question is all about.

Therefore,if you're passionate about how Microsoft implements artificial intelligence in previous operating system software source codes,then by all means you have to check out its;

BetaArchive Discussion and Collection of Betas and Abandonware;

If you're a serious machine learning engineer,this community can help you as well,it releases them for developers and those who are passionately about open source project contribution.

The archive holds terabytes of files,which you can download for free.
So select which windows files have some machine learning algorithms in!
I wanted to give you a little bit of a glimpse,if you're a machine learning engineer.

For your information; answers to this question, will tend to be almost entirely based on opinions,rather than facts concerning Microsoft's software products.


Can you give me a piece of advise of the network sructure that would be suitable for my task?

Instead of NNs, you can use RANSAC algorithm to calculate homography matrix, but first you need to find feature points. However, if your images are blob-like, you may not get such a successful results. Here some presentations for better understanding: cs.umd notes and csail.mit notes
(Also, there might be better image processing tools.)


Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to reinforce unwanted behaviour.

For example, the way I read that reward function you propose, it provide a positive reward for any steps taken by the agent, with larger rewards for steps that get you closer to the goal (except for steps moving back into the start, those would have a reward of $0$. There does not appear to be any "compensation" with negative rewards for moves that take you back away from the goal; in fact, such steps also still seem to carry positive rewards! This means that the optimal behaviour that your agent can end up learning is to keep moving around in circles (somewhat close to the goal, but never quite stepping into the goal) for an infinite amount of time, continuously racking up those positive rewards.

The idea of adding some extra (heuristic) rewards to speed up learning is referred to as "reward shaping". Naive approaches to reward shaping often end up unintentionally modifying the "true" objective, as highlighted above. The correct way to implement reward shaping, which provably does not modify the optimal policy, is Potential-Based Reward Shaping. The basic intuition behind this is that, if you use reward shaping to encourage "movement" in one "direction", you should also provide equivalent (taking into account discount factor $\gamma$) discouragement for subsequent "movement" in the other "direction".

Now, there is this really cool paper named "Expressing Arbitrary Reward Functions as Potential-Based Advice" which proposes a method that can automatically convert from additional reward shaping functions specified in the more "natural" or "intuitive" manner like you did, into (approximately) a potential-based one that is more likely to actually function correctly. This is not quite straightforward though, and the approach involves learning an additional value function which makes additional predictions used to implement the "conversion". So... in practice, in a simple grid-world like yours, I think it's going to be simpler to just figure out the correct potential-based definition yourself than trying to learn it like this, but it's cool stuff nevertheless.


How to understand the concept of self-supervised learning in AI?

I don't think your interpretation is correct. Take images as example.

Supervised Learning

e.g. classification (maybe use CNN with a L2 loss function)

Assume you have many images with different labels. You wish to find a function to approximate the function $y=f(x)$ given a lot of $(\hat x, \hat y)$ sample pairs.
Unsupervised Learning

e.g. clustering (maybe use k-means)

Assume you have many images, but we don't have the labels or we just want to see if there's a way to categorize them into different categories. So we cluster the images by some  characteristic that isn't pre-defined.
Self-Supervised Learning

e.g. super resolution (maybe use CNN with a L2 loss function)

You have many high resolution images without labels, but, your target is to train a model to up sample a low resolution image. So you can have the high resolution images as target, and down size the image to be the input, and try to train the image pairs. So the target is not some manually tagged labels, but generated directly from the data.


How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should:

Be based primarily on the goals of the agent.

Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should not use other data than those four things, but you also do not have to use any of them. This is important, as using any other data stops your environment from being a Markov Decision Process (MDP).

Given the first point, as direct and simple as possible. In many situations this is all you need. A reward of +1 for winning a game, 0 for a draw and -1 for losing is enough to fully define the goals of most 2-player games.

In general, have positive rewards for things you want the agent to achieve or repeat, and negative rewards for things you want the agent to avoid or minimise doing. It is common for instance to have a fixed reward of -1 per time step when the objective is to complete a task as fast as possible.

In general, reward 0 for anything which is not directly related to the goals. This allows the agent to learn for itself whether a trajectory that uses particular states/actions or time resources is worthwhile or not.

Be scaled for convenience. Scaling all rewards by a common factor does not matter at a theoretical level, as the exact same behaviour will be optimal. In practice you want the sums of reward to be easy to assess by yourself as you analyse results of learning, and you also want whatever technical solution you implement to be able to cope with the range of values. Simple numbers such as +1/-1 achieve that for basic rewards.

Ideally, you should avoid using heuristic functions that reward an agent for interim goals or results, as that inserts your opinion about how the problem should be solved into the system, and may not in fact be optimal given the goals. In fact you can view the purpose of value-based RL is learning a good heuristic function (the value function) from the more sparse reward function. If you already had a good heuristic function then you may not need RL at all.
You may need to compare very different parts of the outcome in a single reward function. This can be hard to balance correctly, as the reward function is a single scalar value and you have to define what it means to balance between different objectives within a single scenario. If you do have very different metrics that you want to combine then you need to think harder about what that means:

Where possible, flatten the reward signal into the same units and base your goals around them. For instance, in business and production processes if may be possible to use currency as the units of reward and convert things such as energy used, transport distance, etc., into that currency.

For highly negative/unwanted outcomes, instead of assigning a negative reward, consider whether a constraint on the environment is more appropriate.

You may have opinions about valid solutions to the environment that you want the agent to use. In which case you can extend or modify the system of rewards to reflect that - e.g. provide a reward for achieving some interim sub-goal, even if it is not directly a result that you care about. This is called reward shaping, and can help in practical ways in difficult problems, but you have to take extra care not to break things.
There are also more sophisticated approaches that use multiple value schemes or no externally applied ones, such as hierarchical reinforcement learning or intrinsic rewards. These may be necessary to address more complex "real life" environments, but are still subject of active research. So bear in mind that all the above advice describes the current mainstream of RL, and there are more options the deeper you research the topic.

Is this purely experimental and down to the programmer of the environment. So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?

Generally no. You should base the reward function on the analysis of the problem and your learning goals. And this should be done at the start, before experimenting with hyper-parameters which define the learning process.
If you are trying different values, especially different relative values between different aspects of a problem, then you may be changing what it means for the agent to behave optimally. That might be what you want to do, because you are looking at how you want to frame the original problem to achieve a specific behaviour.
However, outside of inverse reinforcement learning, it is more usual to want an optimal solution to a well-defined problem, as opposed to a solution that matches some other observation that you are willing to change the problem definition to meet.

So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?

This is usually not the case.
Instead, think about how you want to define the goals of the agent. Write reward functions that encapsulate those goals. Then focus on changes to the agent that allow it to better learn how to achieve those goals.
Now, you can do it the way round, as you suggest. But what you are doing, in that case, is changing the problem definition, and seeing how well a certain kind of agent can cope with solving each kind of problem.


The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial.
What you are seeing is most likely the result of a greedy sampling - the most probable next word is chosen from the probability distribution. This quite often leads to infinite repletion loops as you experience.
Simplest way to solve the repetition problem is to actually randomly sample from the probability distribution of the next word - the distribution is usually augmented with so-called temperature parameter.
Finally, there's beam search - you just perform a tree search for a best N-gram sample in the future. That's the most sophisticated, and the most computationally hard one.


Is a dystopian surveillance state computationally possible?

You don't necessarily have to analyse it all. Just by having such data available you can achieve a lot in terms of surveillance, as long as you can retrieve relevant parts.

A few years ago there was a Radiolab podcast, "The Eye in the Sky" (there's a full transcript on the site). The basic idea is that you have a plane circling a city 24/7, and filming what goes on. If there was a crime somewhere, you retrieve the recordings after the event, and you can track back to where vehicles involved in the crime were coming from, and where they went after the crime. If nothing happens, you simply archive the data, and perhaps remove it after a month or so.

This method was used to solve a hit-and-run assassination of a police woman who was on her way to work. The gang who committed the attack were rather surprised when the police showed up at their secret hide-out a few days later, as they could see on the images where the cars involved went to later. At the time and place of the murder there were obviously no witnesses who could have done that. And this involved no computational processing at all.

The possibilities this opens up are just scary, as you can track pretty much anybody's movements without actually needing someone to follow them. Add to that street-level CCTV, and not much can happen without you being able to find out.

In this scenario there is no processing at all, but you could imaging simple processing steps, such as tracking vehicles or changes in the environment, which could be used to give clues about potentially 'interesting' events. So instead of using it 'passively' as a kind of memory, you could use that data to identify things that happened that you weren't aware of.

And this is without even any clandestine access to people's data. If you add that dimension, then you might even be able to identify crimes/etc before they even happen. Text processing can be quite fast, but is not easy to do, as presumably few people would openly communicate about things they were planning. So I guess we're still a long way away from that.

Of course there is the ethical dimension (which is mentioned in the podcast): who has access to that data, and who decides what it is used for? If you do, and you suspect your partner of being unfaithful, who/what would stop you from checking out their movements? Or check up on that politician who might have a secret affair, or a gambling issue, or who keeps being in the same locations as a well-known drug dealer. All rather scary.

While a complete analysis of all such data would be very heavy computationally, and fraught with false positives and recall problems, it might simply be enough to index it by time, location, and perhaps people involved (face recognition seems to be reasonably good, though still with a rather high error rate). This is enough already to make me feel worried about the future.


Where does entropy enter in Soft Actor-Critic?

In the answer I'll be using notation similar to the one from the SAC paper.
If we look at the standard objective function for policy gradient methods we have
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) +  \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\
&\cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)]
\end{align}
If you keep unwinding this $V(s_{t+i})$ you will get expected sum of rewards.
We can define soft state value as
\begin{align}
V(s_t) &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]
\end{align}
third equality comes from the fact that $\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.

In maximum entropy framework objective function would then be
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\
& \cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)]
\end{align}


How could we build a neural network that is invariant to permutations of the inputs?

Here is a few that might be what you are looking for:

Deep Sets, https://papers.nips.cc/paper/6931-deep-sets.pdf
BRUNO: A Deep Recurrent Model for Exchangeable Data, https://arxiv.org/pdf/1802.07535.pdf
Deep Learning with Sets and Point Clouds, https://openreview.net/pdf?id=HJF3iD9xe
Permutation-equivariant neural networks applied to dynamics prediction, https://arxiv.org/pdf/1612.04530.pdf


What is the purpose of the hidden layers?

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?".

And the answer to that latter question is an area of active research. Even top experts like Geoffrey Hinton and Andrew Ng will freely admit that we don't really understand why deep neural networks work. That is, we don't understand them in complete detail anyway.

That said, the theory, as I understand it goes something like this...  successive layers of the network learn successively more sophisticated features, which build on the features from preceding layers. So, for example, an NN used for facial recognition might work like this: the first layer detects edges and nothing else. The next layer up recognizes geometric shapes (boxes, circles, etc.). The next layer up recognizes primitive features of a face, like eyes, noses, jaw, etc. The next layer up then recognizes composites based on combinations of "eye" features, "nose" features, and so on.

So, in theory, deeper networks (more hidden layers) are better in that they develop a more granular/detailed representation of a "thing" being recognized.


What do the state features of KukaGymEnv represent?

Here's an incomplete answer, but it may help.
Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function getObservation() from this source code, gets a state, and extend this state with three components :

relative x,y position and euler angle of block in gripper space

But what are the 5 first components returned by getObservation()? From what I read, there are positions, then euler angles describing the orientation. But that would make 6 + 3 = 9 features, so there is either only 2 positions, or only 2 euler angles. You may know kuka better than me and know the answer of this one :).
So, to sum up :
state = [X, Y, (Z, ) , Alpha, Gamma, (Beta, ), gripX, gripY, gripAlpha]

(Either Z or Beta is absent)


What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper?

After a very brief look at the paper, I think that they are predicting the stock price for the next day, and not for the current day, which is quite common and reasonable: see equation (1), where they predict $x(t+1)$.
So I don't see any issue with this paper.

But I've only quickly looked at it, so I may have missed something, of course.


Can we use the Tierra approach to optimize machine code?

Yes it has been tried. In fact there is a whole field, dubbed Genetic Programming.

There is an annual competition to obtain "Human-Competitive" algorithms, and many instances of those have been found over the years.


Couldn't an AI cheat when trying to follow its goal?

These are two good questions. To address the first, this is a known problem in AI safety and is called "Wireheading". There is currently no known solution other than to somehow prevent an AI agent from being able to alter its internal state in this fashion.
If you are interested in this area in general then I recommend this paper as an entry point into the literature.