What would be a good internal language for an AI?

This is (even though it doesn't look like it at first glance) a deeply philosophical question about the nature of 'meaning'. This answer is necessarily limited in scope.

There are many ways of representing knowledge, and countless formalisms have been developed since the early days of AI. Many of them are based on some kind of predicate calculus, ontologies, semantic networks (providing eg inheritance of features and part-of relationships), and they seem to work fine for limited domains.

One problem is the grounding: if you have a predicate isGreen(x), what does that actually mean? How is it related to isBlue(x)? Do you want to treat them similarly? If so, you need to represent this somehow. You quickly come to the point where you will need to encode all the world's knowledge in some generalised way. An impossible task.

Linguists have struggled with this for decades: what is the meaning of a particular sentence? Apart from the fact that every individual human will interpret a given sentence differently (based on their own life experience and culture), there are many aspects to 'meaning' that need representing: the 'factual' meaning, but also pragmatic, evaluative, and all sorts of other nuances. An innocent utterance, That's a nice Apple you've got there, could have a whole raft of meanings packed into it, all implicit. For example, the person probably likes apples, that one in particular, that apple looks like a tasty piece of fruit, the other person is the owner of the apple, and it might also be a request which is intended to prompt the other person to offer it to you. How are you going to represent all that meaning?

One area that interests me personally is representing narrative events. This can — up to a point — be done using Conceptual Dependency, which uses a limited set of semantic primitives. While useful to encode basic stories, you cannot easily use it to represent the fact that grass is green.

So the answer is: there is no answer. AI is too broad a field, and you need to look at a particular application to decide which knowledge is relevant to it, and then how it can best be represented. There is a reason why there are so many ways of representing knowledge.

PS: You suggest this would be more precise. My personal view is that precision here is a red herring. The word green is not precise, as it covers a range of wave lengths, and different people would disagree on whether something is green or not. So a predicate isGreen(x) is not any more precise than that. Hence the appeal of fuzzy logic, which allows computation to be based on less precision.


Isolate the speech of two people in an audio record with two people only

The task of isolating 2 or more speakers is called speaker diarization,
here a list of softwares and useful resources.

Once you have the 2 or more audio files containing the individual voices, you could run some speech-to-text network that also outputs time information.


Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm

Seems this question was asked and answered in openai/baselines github issue. The issue has been closed for a while.
Below is an answer provided by @matthiasplappert which has the most "thumbs up":

To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.
However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.


Query regarding the minmax loss function formulation of the training of a Generative Adversarial Network (GAN)

I'll answer your questions one by one:

  In this equation are the $E_{z \sim p_z(z)}$ and $E_{x \sim p_{data}(x)}$ the means of the distributions of the mini batch samples?

So let's take the first part $E_{x \sim p_{data}(x)}[log \,D(x)]$. This is read as the "expected value of $log \, D(x)$, where $x$ is sampled from $p_{data}(x)$". So, in simpler terms this means that:

You have a distribution $p_{data}(x)$.
You sample a batch of samples $x$ from this distribution.
You feed the batch to the discriminator and get its output $D(x)$.
You compute the log for the batch of predictions.
You average over the samples in the batch.

  Also is the optimal case for the discriminator a maximum value of 0?

Yes, the goal of the discriminator is to maximize $V(D, G)$. This is achieved when $D(x) = 1$ and $D(z) = 0$, where $V(D, G)=0$.

  the optimal case for the generator a minimum value of log(small value) {basically a massive negative value}?

Yes, the generator wants to minimize $V(D, G)$, so it wants $D(x)$ and $D(z)$ to be as small as possible (though it can affect only the second term). The minimum value for $D(z)$ is $1 / M$, where $M$ is the number of classes in the dataset. So the minimum value for $V(D, G)$ is $\log{[(M-1) / M^2]}$ (I think).

  If so, what happens to the first term during the training of the generator - is it taken as a constant or is the discriminator performing badly considered optimal for the generator?

The first term is of no consequence to the Generator, because the generator can't affect it somehow (i.e. like you say it is taken as a constant). Consider the generator $G_{\theta}$ has parameters $\theta$ and the discriminator $D_{\phi}$ has parameters $\phi$. When training $G$:

$$
\frac{\log{D_{\phi}(x)}}{\partial \theta} = 0
$$

So the generator is trained of $V$.

  While putting this in code, for every step is the discriminator trained first for one step, keeping the generator constant, followed by the generator being trained for the same step, with the discriminator kept constant?

In theory, yes. In practice for every step the Generator takes, we train the Discriminator for more steps (e.g. $5$ steps).

  In every step of training are there multiple latent vectors sampled to produce multiple generator outputs for each step? If so is the loss function an average or sum of all V(D,G) for that step.?

Yes, at every step we sample a batch of latent vectors. The equation indicates that we average over these (i.e. the expectation) but in practice it doesn't have any difference (the average is the sum divided by a constant number, which doesn't affect the optimization very much).


Why do we need target network in deep Q learning?

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of the parameters that gets updated every $C$ updates, and the Q-Network with these parameters is the target network.
If you didn't use this target network, i.e. if your update target was $\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta)\right)^2$,  then learning would become unstable because the target, $r_t + \max_aQ(s_{t+1},a;\theta)$, and the prediction, $Q(s_t,a_t; \theta)$, are not independent, as they both rely on $\theta$.
A nice analogy I saw once was that it is akin to a dog chasing it's own tail - it will never catch it because the target is non-stationary; this non-stationarity is exactly what the dependence between the target and the prediction cause.


How do I restrict the neural network structure to be acyclic in NEAT?

The naive way is to generate connections randomly as you would for a cyclic graph, but then perform a test to reject any connections that form a cycle. This is the current approach in SharpNEAT and there has been some effort directed at improving the performance of the cycle test in the work-in-progress refactor branch.

One alternative would be to track the depth of all nodes, store a list of node IDs sorted by depth, and sample the connection endpoint nodes in such a way that the target node depth is always higher than the source node. Now I think about it that's probably the better method.


Machine Learning is a bad fit to this problem.

Even simple PRNGs that are not suitable for use in simulators (such as rand()) are varied enough that it is very hard to reverse engineer them statistically using generic techniques - essentially what 90% of ML does is fit a generic model to data statistically by altering parameters. The remaining 10% might do things in specialist manner, such as saving all the data and picking best option.

In theory most ML approaches would eventually solve a PRNG, however that would typically involve iterating through the entire state space of the PRNG multiple times. The statistical relationship between internal state, next state and output of a PRNG is complex by design, so that this is the only "black box" statistical approach, and this is clearly not feasible for any real implementation of a random number generator, which is going to have at least $2^{31}$ states on modern machines. Perhaps older 16-bit PRNGs, with a single value for state might be tractable.

An AI advanced enough to attempt to reverse engineer the output logically based on purely the data and researching how RNGs work is too advanced for current ML techniques to consider.

That leaves approaches that might try to construct a similar RNG, such as Genetic Programming (where the genome is converted to executable code). The trouble with this approach is there is no heuristic for a RNG that measures how close its output is to a target. A single bit of state difference or any tiny but meaningful change in generated RNG design will produce output that has no similarities with the target output whatsoever. Without such a measure you have no fitness function, and no way to attempt a guided search using the many discrete optimisation tools from AI.

Instead the usual approach to "breaking" a PRNG is to analyse the algorithm. Knowing the algorithm of many non-cryptographic PRNGs can allow predicting the internal state of the generator, sometimes in very few steps (for really simple Linear Congruential Generators that might be just a single step!).


Are most things generally discovered because they work empirically and later justified mathematically, or vice-versa?

There are definitely approaches that are theory driven (like SVMs), and others where the theory comes after the practice (like a lot of deep neural networks). I think it would be difficult to argue strongly that either direction is more common "in general" within AI, or indeed, within any other branch of science.

The approach that is currently in favor will tend to change over time as well. If we develop a new method that we don't understand, it will tend to attract interest from theorists. If they are able to understand it well, that understanding will likely lead to natural next steps that improve the model. As with other areas of science, the empirical cycle is at work.


What should the initial UCT value be with MCTS, when leaf's simulation count is zero? Infinity?

Assigning a value of $\infty$ to unvisited nodes is indeed the "default" or most basic choice, and it indeed ensures that the search never visits a node for a second time if it also still has siblings that have not had any visits. But many other kinds of values have been tried in the literature too.
Gelly and Wang, in "Exploration exploitation in Go: UCT for Monte-Carlo Go" referred to the parameter as "First-play Urgency" (FPU), and indeed really treated it as a hyperparameter; they simply tried various different values and some were found to work better than others.
Other more specific values that you may want to consider (but in most cases we can't really say much about which one will be better than which other ones, without empirical evaluations) are:

The value of a loss; treating nodes that have not had any visits as losing nodes is probably the most "pessimistic" initialisation you could pick. It's not very clearly described in the papers, but there is some evidence out there that this was used by AlphaGo Zero and AlphaZero. Those are not simply vanilla UCTs though, those programs had very strong trained Deep Neural Networks to make additional recommendations for actions. If you don't have such strong neural networks to introduce additional biases in your selection, I would not recommend treating unvisited nodes as losing nodes.
The value of a draw; treating unvisited nodes as draws means that you may prioritise re-visiting nodes that look like winning nodes, but you won't prioritise re-visiting nodes that look like losing nodes.
The value of a win; this probably produces similar behaviour in practice as a value of $\infty$
The average value backpropagated into the parent (correcting for differences in player-colour-to-move); you may prioritise re-visiting nodes that perform better than average in this subtree, but won't prioritise re-visiting nodes that perform worse than average in this subtree


How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just'  100K samples.  However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients).  For an application like this one, you would probably find most of the value from a few hundred words.


Will artificial intelligence cause mass unemployment?

The nuanced, boring answer is that it depends on your definition of AI. Most people wouldn't say that the rule-based systems designed in the 70's are AI. The amazing leaps in machine learning are almost taken for granted as well (think about how normal speech and facial recognition have become). This is known as the AI effect; when we become accustomed to the technology, it loses it's 'magical aspect' and is thus no longer labelled as AI.

Since AI is so diverse and difficult to define, the question becomes incredibly abstract. Did Siri cause all secretaries to become unemployed? Did TurboTax replace all accountants? Some parts of AI will affect jobs, or even make them redundant yes. On the other hand, it will give rise to new jobs as well. It is therefore impossible to generalize it as 'AI will cause massive unemployment'.

This is not a new phenomenon, however, it has been part of the human economy ever since the industrial revolution (probably even before that, but I am not a historian). The invention of the car crippled the horse-and-wagon industry, but it brought along new jobs as well.


There's some useful information in your description, but that's just a very vague description of how neural networks with sigmoid activation functions are trained.

Moreover, there are many other AI systems apart from neural networks (such as support vector machines, expert systems, etc.), which, of course, I cannot exhaustively list here.

  Is my understanding of how AI works correct?

I would say it's not completely incorrect, but, as I said, it's a very vague description and it only refers to a subset of techniques in the AI field. With that description, no newbie would probably understand how neural networks are really trained, apart from knowing that you will train them iteratively and there are sigmoids involved.


Is there any good app for iOS or Android to create, train and run neural networks?

You can use Neuroph to develop and train your network and add in app via NetBeans.

check this link

Creating Android image recognition application using NetBeans and Neuroph.


How to give each segment a new colour in image segmentation?

Deep learning based image segmentation is basically pixel-wise image classification.

So instead of predicting a $C$ dimensional vector $\vec{x}$ where $C$ corresponds to the number of class labels, you predict a tensor $x\in\mathbb{R}^{h\times w \times C}$ of the same height and width dimensions as your input image, and with $C$ channels which correspond to the different classes.

The paper "U-Net: Convolutional Networks for Biomedical Image Segmentation" provides a good overview of the current methods.


How can I create an artificially intelligent aimbot for a game like CS:GO?

Thare are many ways to approache this. One approach to start would be to try to describe the problem using a formalism closer to reinforcement learning.

Output:

Aiming in any shooter type game, as I recall, involves moving the mouse. So the output of your aimbot has two dimentions. Depending on the required accuracy, you can consider these two dimentions continous, with a limited range, or if you consider each pixel as a integer, you might be able to discretize your action space. (I assume mouse XY coordinates should be input to the game, not increments)

Input:

You  definetly need screen information. You can take the whole screen as input to a CNN, similarly to DeepQLearning for Atari.

Reward function

This might be tricky since your rewards need to be as dense as possible, however, the only feedback you get from the game is that someone was shot.
It might be ebough but this will definetly increase your training time.

Training data / Environemnt:

Your environment for training is the game itseld. Using a curriculum learning approach would probably make the training process more efficient.

You can also try an imitation learning approach, since I assume you are happy to provide expert training examples (in this case probably headshots in the game environment).

You can read more about how to apply reinforcement learning for games here. The Unity ML-Agents Library also includes sample tracking problems and their solutions.


Structured lighting basic principles for depth mapping

how, in the most simple-to-implement basic principle, does the light projection to depth map technique described here https://www.lightform.com/how-it-works actually functions?Is it some kind of an average based on the color of x pixel over all the patterns or what?

No it is not that simple , more details on that are described in https://kevinkarsch.com/publications/sa11-lowres.pdf

P.S the author of this paper went on to become CTO of lightform.

Is it some kind of an average based on the color of x pixel over all the patterns or what? How difficult would it be to code something that could do this, up?

As mentioned above it is not simple average based on color , coding would be a tough task , you could implement model if you are capable enough to code a ray-tracer from scratch. But replicating the exact results would require a lot of low-level knowledge and certain domain expertise as not much of the implementation details and hyperparameters are described.


What is wrong with equation 7.3 in Sutton & Barto's book?

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the $n$th step return. They would only be equal as $n \rightarrow \infty$.


Do all filters of the same convolutional layer need to have the same dimensions and stride?

It seems that a similar question has been raised here: https://stackoverflow.com/questions/57438922/different-size-filters-in-the-same-layer-with-tensorflow-2-0
Like answered in the link above, you could combine severall Conv2D ops with different kernel sizes on the same input. You would have to adapt each output with padding, or cropping, so that you could concatenate all of them.
Hope this helps!


Can we calculate mean recall and precision

For the precision metric for example you have:

$$
 Precision = \frac{TP}{TP+FP},
$$
with TP = True Positive and FP = False Positive.

Imagine you have the following values:
Image 1: $TP = 2, FP = 3$
Image 2: $TP = 1, FP = 4$
Image 3: $TP = 3, FP = 0$

The precision scores as you calculated will be:
Image 1: $2/5$
Image 2: $1/5$
Image 3: $1$
Your average will be: $0.533$

On the other hand if you sum them all up and then calculate the precision value you get:

$P = \frac{6}{6+7} = 0.462$

This proves that averaging the precision scores is not the same as calculating the total precision in one go.

Since what you want is to know how precise your algorithm is, independently of the precision for each image, you should sum all the TP and FP and only then calculate the precision for each model. This way you will not have a biased average. The average would give the same weight to an image with a larger number of objects as to another image which had fewer objects.


What is meant by "ground truth" in the context AI?

In the context of ML, ground truth refers to information provided by direct observation (empirical evidence). If you're training an algorithm to classify your data, then the ground truth will be the actual, true labels which could for example be manually annotated by an domain expert. Please note that the models prediction or the inferred labels, are not considered ground truth.


What is the mathematical definition of an activation function?

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases)

If you are using gradient descent as a training method, then differentiability is closest to being a requirement. However, even then as you noticed with ReLU, it is not an absolute requirement, provided behaviour close to discontinuities is good. For example $\frac{1}{x}$ or $log{x}$ make bad choices here due to how they behave for values near to $0$.

Wikipedia summarises some important desirable traits, plus compares popular activation functions in this table.

  Nonlinear – When the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator. The identity activation function does not satisfy this property. When multiple layers use the identity activation function, the entire network is equivalent to a single-layer model.

  Range – When the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights. In the latter case, smaller learning rates are typically necessary.

  Continuously differentiable – This property is desirable (RELU is not continuously differentiable and has some issues with gradient-based optimization, but it is still possible) for enabling gradient-based optimization methods. The binary step activation function is not differentiable at 0, and it differentiates to 0 for all other values, so gradient-based methods can make no progress with it.

  Monotonic – When the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex.

  Smooth functions with a monotonic derivative – These have been shown to generalize better in some cases.

  Approximates identity near the origin – When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

You can see that ReLU is an outlier on more than one of these traits. The reason that it is popular is that despite these weaknesses, the things it does well - speed of operation plus helping combat the vanishing gradient problem - make it a solid choice for very deep networks. In fact the success of ReLU has inspired use of a number of similar-looking activation functions that hope to keep its benefits whilst having more of these desirable traits.

Ultmately though, almost any non-linear function can be used successfully in a neural network. The more of the desirable traits it has, the more likely you can use it in general cases with existing approaches to training.


How fast is a GPU compared to a CPU for training deep learning models, and which GPU should I buy?

I think you should redirect your focus.
Learn and play with ML, and only when compute becomes the main bottleneck for your learning, invest in hardware.

I've recently engaged into a ML project for which I 've assembled a machine with gtx1080, installed gui-less ubuntu, configured ssh, drivers etc, and than spend 3 months on data collection. And the dataset was so small (30k images), that I could probably learn mobilenet v3 on my CPU instead.

I think that in most cases focusing on hardware is a form of procrastination when it comes to machine learning.
Also, a laptop with really strong GPU makes little sense (unless it's really well cooled like Alienware r3 that I use at work, but it's to heavy to be really considered a laptop...). It will throttle, It will be less durable due to constant overheating when you learn models.

So I'd say, go for gtx1060 laptop and only when you really need compute (you think you have a chance to actually win a competition due to a great idea you have, or you have a client that has seen a demo and really want's to pay for your model if you improve it), invest into standalone PC or cloud to tune your prototype model.


Non-Euclidian geometry can be generally boiled down to the phrase

the shortest path between 2 points isn't necessarily a straight line.

Or, put in a way that lends itself very much to machine learning,

things that are similar to each other are not necessarily close if one uses Euclidean distance as a metric (aka the triangle inequality doesn't hold).

You mention graphs and manifolds as being non-Euclidian, but, really, the majority of problems being worked on don't have Euclidian data. Take the below images for example:
Clearly, 2 of the images are more similar to each other than the third one is, but if we looked at the pixels alone, the Euclidean distance between the pixel values don't represent this similarity.

If there was a function, $F(\text{image})$, that mapped images to a space of values where similar images produced values that were closer together, we could better understand the data, infer some statistics about the distributions, and make predictions on data we have yet to see. This is what classic techniques of image recognition have done and it's also what modern machine learning is doing. Taking data and mapping it to a space such that the triangle inequality holds.
Let's look at a more concrete example, some points I drew in MSPaint.
On the left is some space that we are interested in where points have 2 classes (red or blue). Even though there are points that are close to each other, they may have different colors/classes. Ideally, we could have a function that converts these points to some space where we can draw a line to separate these 2 classes. In general, there would be many lines, or hyper-planes in dimensions > 3, but the goal is to transform the data so that it will be "linearly separable".

To conclude, non-Euclidian data is everywhere.


Is the derivative of the loss wrt a single scalar parameter proportional to the loss?

The derivative $f'(x)$ is correlated with $f(x)$ in a certain sense. In fact, $f'(x)$ is a function of $f$, so we could even say that there's a cause-effect relationship.

The derivative at a specific point $c$ of the domain, i.e. $f'(c)$, can either be negative or positive. If $f'(c) > 0$, then $f(c)$ is increasing (with respect to an increase of $x$). If $f'(c) < 0$, then $f(c)$ decreasing  (with respect to an increase of $x$).

This can easily be seen from an example. Consider $f(x) = x^2$, then $f'(x) = 2x$. Let $c = 2$, then $f'(2) = 4$, so the function is increasing. In fact, $f(1) = 2 \leq f(2) = 4 \leq f(3) = 9$. Similarly, let $c = -1$, then $f'(-1) = -2$, so the function is decreasing. In fact, $\leq f(-2) = 4 \geq f(-1) = 1 \geq  f(0) = 0$ (note that the function is decreasing as we increase $x$!).

Consider a model with only one parameter, then the partial derivative of the loss function with respect to that parameter corresponds to the derivative of the loss function. So, the reasoning above applies to this model. What about a model with more than one parameter? The same thing happens.

If the function decreases, does its derivative also decrease? In general, no, and this can easily be seen from a plot of a function and its derivative. For example, consider a plot of a parabola and its derivative (which is a linear function).

On the left of the y-axis, the parabola is decreasing, but its derivative is increasing, while, on the right of the y-axis, the parabola is increasing and the linear function is still increasing.

This is the same thing with a loss function of an ML model and its partial derivative.


What is the difference between active learning and online learning?

Active learning (AL) is a weakly supervised learning (WSL) technique where you can have both labelled and unlabelled data [1]. The main idea behind AL is that the learner (or learning algorithm) can query an "oracle" (e.g. a human) to label some unlabelled instances. AL is similar to semi-supervised learning (SSL), which is also a WSL technique, given that both deal with unlabelled and labeled data, but do that differently (i.e. SSL does not use an oracle).
Online learning are machine learning techniques that update the models as new data is collected or arrives sequentially, as opposed to batch learning (or offline learning), where you first collect a dataset of multiple instances and then you train a model once (although you can later update it when you update your dataset). Batch learning is currently the common way of training machine learning models, given that it avoids problems like the known catastrophic interference (aka catastrophic forgetting) problem, which can occur if you learn online. For example, neural networks are known to face this problem when learning online. There are incremental learning (aka lifelong learning) algorithms that attempt to address this catastrophic interference problem.


Do we train a logistic regression model using a dataset that is 3 times bigger than the validation dataset?

The predictions of each of your initial models will become a feature to feed the meta learner. If you use $n$ initial models, then for each example you will feed the meta learner $n$ features, each feature being a prediction from one of the initial models.

Note that this doesn’t mean the size of your dataset increases. Instead, each member of $X_{val}$ will be represented by $n$ features, with the $k$th feature being equal to the initial prediction of the $k$th initial predictor.

This medium link recommends a package that can help you build a pipeline.

Note: in a sense if you are caching the intermediate predictions of each initial predictor, you’ll end up with $n$ times as many data points as there are entries in $X_{val}$. But these are simply each going to be a feature of another data point, so the dataset hasn’t really increased in size.


Will AIs make most of the scientific discoveries in 50 years?

According to Ray Kurzweil, a prominent AI researcher, yes. In his book The Singularity is Near he predicts that AIs will take over developing other AIs in about 30 years, after which human intelligence will become marginalised.


How do I determine the genomes to use for crossover in NEAT?

The good thing about genetic algorithms is that they are exchangeable.
If you have the fitness of each individual, any algorithm (i.e. roulette, rank, tournament) will do.


Do we need an explicit policy to sample $A'$ in order to compute the target in SARSA or Q-learning?

Q-learning uses an exploratory policy, derived from the current estimate of the $Q$ function, such as the $\epsilon$-greedy policy, to select the action $a$ from the current state $s$. After having taken this action $a$ from $s$, the reward $r$ and the next state $s'$ are observed. At this point, to update the estimate of the $Q$ function, you use a target that assumes that the greedy action is taken from the next state $s'$. The greedy action is selected by the $\operatorname{max}$ operator, which can thus be thought of as an implicit policy (but this terminology isn't common, AFAIK), so, in this context, the greedy action is the action associated with the highest $Q$ value for the state $s'$.

In SARSA, no $\operatorname{max}$ operator is used, and you derive a policy (e.g. the $\epsilon$-greedy policy) from the current estimate of the $Q$ function to select both $a$ (from $s$) and $a'$ (from $s'$).

To conclude, in all cases, the policies are implicit, in the sense that they are derived from the estimate of the $Q$ function, but this isn't a common terminology. See also this answer, where I describe more in detail the differences between Q-learning and SARSA, and I also show the pseudocode of both algorithms, which you should read (multiple times) in order to fully understand their differences.


Picking a random move in exploitation in Q-Learning

It depends on the implementation of the software package that you are using. If you call a function that returns the maximum value and all values are the same then it might return the value at first index or some other one. The point is it doesn't matter which action is chosen since all of them are the best at the same time. So it's basically random but you should treat it as if you are trying to pick the best action.


If you read Steven Levy's book, Artificial Life,you will find, as I did,

the distinction between biological and "artificial" life blurred. If you think about it, what exactly is "life", anyway? A set of complex systems with emergent behavior capable of evolution and adaptation.

A prototypical biologist may not define life that way. Indeed, he would, not being focused or concerned with the computational aspect, define it in a way that would narrow it down to biological life.

Marvin Minsky mention the concept of luggage words, and I myself came up with the notion of mirage concepts. For the former, that which we don't understand gets lump into the word. For the latter, when you take a "mysterious" concept apart, it vanishes like a mirage does as you get closer.

So what is life? If we look at a "living" organism, we'd all that "life". If we remove a single cell from that organism, we'd still call that "life". But what if we remove a single organelle like, say, a ribosome? Lysosome? Contractile vacuole? Endoplasmic reticulum? Is that still "life"? What if we remove a macromolecule from that? Is that still "life"?

As you see, all becomes very murky, and I do this on purpose to illustrate just how arbitrary the very concept of "life" is.

So I think my definition is a good one, broad enough to encompass both in-silico and the organic versions. It bespeaks to algorithms, robots, viruses -- yes both computer and organic... anything that has complexity and the ability to adapt and evolve.


Is there any free dataset of source code along with natural language description?

StackOverflow answers with code snippets. This data needs some processing, because the description can be in the question (along with other notes) and along with the answer. But this dataset is very big.

Also take a look at CodeReview questions.


Input/output encoding for a neural network to learn a grid-based game

To represent the pieces, you should be able to use a single input matrix. Just designate an integer number for the different types of pieces. White stones can be positive integers and black stones can be negative.
You can use sigmoid for board position confidence and linear activation for piece identifier. pass would be another sigmoid output. I don't think you'll have to worry about pass being diluted. Since it is such a valuable action, the score will depend a lot on the pass output and it will have a large gradient. If you need to select the pass action with high frequency for reinforcement learning purposes, then just attribute a higher probability to the pass action in your random choice function.
The final score difference has a large impact on the desirability of the moves. A large score difference should result in a large impact on the function. Therefore you might want to include the magnitude of score difference in your loss function.

This is the type of job that Deep Q Learning does. Perhaps you'll want to look into that too.


How does an svm work? How does it perform comparisons between malignant and benign tumor

I will try to give you a simplified explanation of how SVMs work.

The data one works with can be of two types. Either it is very easily separable and there is a clear straight line boundary between data points of different classes. We call such data linearly separable(image) or the data points of the classes are mixed in a way that there is no clear straight line boundary that separates them. In other words the data is not linearly separable.(image).

One way to make data points linearly separable is to map them into a higher dimension where they would become linearly. If we have a set of data points which are not linearly separable in 2 dimensions, we could map them into 3 dimensions where the data becomes linearly separable(like so).

SVMs work by converting data from lower to higher dimensions where it is linearly separable and then tries to find the boundary that separates the data.

The same concept could be applied to images, where the images are converted into a very high dimension in which images of different types can be separated linearly.

This could mean that the SVM model trained is able to easily distinguish between a bottle and a glass or different tumour types but that would depend entirely on the data used to train the model. The problem arises when the distribution of the training data and testing data is completely different. So for example, you trained the SVM to distinguish between glasses and cups of a certain shape and colour and while testing you show it glasses of completely different shapes and colours, the SVM will not be able to distinguish between them.


How is transfer learning used to mitigate catastrophic forgetting in neural networks?

Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)

Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.

One implementation involves transferring a component and only training additional layers.
Another is retraining the entire system but at a lower learning rate?

In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant forget based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.

In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can forget)

These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above any learning system without lossless memory will have remnants of catastrophic forgetting, so making the generalized claim about transfer learning may not always fit the bill.


Do I need to know in advance all possible number of states in Q-Learning?

When you start off learning about Q-learning, you start with a simple example that has a few states. For each of the states, you try to estimate what the 'value' is of that state. Because there are so few states, it is possible to store these values in a table (it is also useful for the intuitiveness of the explanation).
However, if you start trying to solve more 'real-life' problems, the number of states can be insanely huge. The essence however will stay the same. You are trying to estimate what you want to do next, based on an estimation of how good each state is that you can end up in. However, now the values are 'most of the times' not stored in a table anymore, as the approach will often come down to using an ANN to estimate the value function.
Answer to your question: You are going to run into a lot of problems when training some model with Q-learning if your table is not able to store the values of possible states that it can come across. In practice, most implementations do not use a Q-table and just use an ANN, which alleviates the problem of having to 'define' how many states your problem consists of.


Is private research with low funds still feasible in the field of AI?

Yes, you can do research in Artificial Intelligence with low funds (but you need a lot of time!). Notably, because AI is not the same as applied machine learning (indeed running ML programs on big data requires a lot of computer power). For example, knowledge representation and reasoning or natural language processing (both are subfields of AI) generally don't require a lot of computer power. Even when interested in machine learning (which is not the same as AI, just a subfield of it among others), you can use a powerful PC or laptop for that.
A lot of recent papers (probably most of them) in journals like  Artificial Intelligence, or in conferences like IJCAI (see their past proceedings) are somehow theoretical, and when something is implemented, it runs on a laptop or desktop. Notice that both AI journal and IJCAI conferences are peer-reviewed with a very selective process.
Actually it is difficult and rare to find a research paper in AI mentioning that costly equipment was needed to do the research. Costly supercomputers used for research are generally not used by AI researchers (but by researchers in physics or bioinformatics), and AI researchers often don't even have access to such facilities.
For examples, recent IJCAI2016 papers such as Coco: Runtime Reasoning About Conflicting Commitments, Interdependent Scheduling Games, Control of Fair Division, Verifying Pushdown Multi-Agent Systems against Strategy Logics, etc. don't mention any costly computation.
Actually, it is likely that most recent papers don't use and don't need large scale costly cloud computing. And some of that research might not have been implemented (perhaps by some intern) in anything more than a toy prototype.

Please include in your answer examples of recent research results and the required infrastructure that was used to create them.

Here are some recent research publications, they all explicitly mention the needed computer equipment, follow the link to read them:
Event recent IJCAI experimental papers like A Multicore Tool for Constraint Solving, Compiling Constraint Networks
into Multivalued Decomposable Decision Graphs, External Memory Bidirectional Search, Multiple Constraint Acquisition, Completion of Disjunctive Logic Programs, Eliminating Disjunctions in Answer Set Programming by Restricted Unfolding and On the Empirical Time Complexity of Random 3-SAT at the Phase Transition mention at most a multi-core workstation (e.g. at most a dual Xeon socket workstation,  often a cheaper laptop or desktop). I guess that most authors need their GPU only as a graphics card, to use their display screen. BTW, papers in other journals, such as JAIR (like Improving the Efficiency of Dynamic Programming on Tree
Decompositions via Machine Learning), when they mention computer equipment, gives similar kind of machines (laptop or at most high-end desktop).
(Actually, I don't remember having read an AI paper mentioning costly computing equipment; and I believe the reason for that is that in the academic community access to supercomputers is nearly reserved to research in other domains: physics, bioinformatics, etc. For an AI researcher gaining such access is difficult and uncommon; BTW, in H2020 European research grants computing cost above 15% of the labor cost needs to be dully justified, so is exceptional)
However, you'll better publish your software as free software or open-source, and you need a lot of time (preferably full-time, or at least half-time) to do the research work, publish it, and follow outside progress in your area. BTW, contacting a nearby university could be helpful (you could attend some seminars, etc.)
You can find interesting blogs (e.g. the one of J.Pitrat) of AI researchers working on just a laptop or a desktop.
So, most research in AI (even in Machine Learning, read papers from JMLR) are done by researchers working on a laptop or desktop. I am not even sure you need a powerful GPU to do the research. You certainly can do interesting research with a desktop computer (probably running Linux, with 32Gb RAM, some AMD Ryzen or Intel i5 or i7) costing in 2017 less than 1500€ or maybe 2500€ (and sometimes a low end laptop is enough). In rare cases (a small minority of papers), you might need a dual-socket workstation (something costing perhaps 5000€).
Notice that Big Data is not considered as a field of Artificial Intelligence. I guess (but don't really know) that research in Big Data requires access to more computer power than a PC. BTW, I am not sure that AlphaGo is really a research project, it is more an industrial demo.
PS. See also http://norvig.com/21-days.html (notice who is the author!) and this Bismon draft report (developing programming techniques, in a GPLv3+ alpha-stage software prototype, notably frame-based or semantic-network related ones, with reflection and introspection, reusable and relevant to AI), and the RefPerSys project.


I know that in order to discuss your question, I must have a background in cryptography, which I don't have! But there is something that I know for sure:

First of all, a simple search gave me this. It might help.
To lots of us, ANN looks like a magic wand which can turn almost everything into anything. But the point is, ANN is only a ML algorithm, but absolutely a powerful one. You should be aware that there are random and multiple weight initializations in ANNs, and this means a fairly stochastic behavior of your network, which vanishes only in shallow networks with high amount of training data. Your network's specifications are even likely influenced by the order of feeding the instances in the training phase. So if what I think of the encryption/decryption process "certain rules for message transformation" is true, then a stochastic network is indeed a sophisticated option (but according to scientists in neural cryptography research area, completely possible)
You're probably thinking of auto-encoding networks, one kind that maps X to X and has a bottle neck, such that if successfully trained, it has developed the power of reducing the number of data features from say 10'000 to 500, and then successfully retrieving the original data with 10'000 features from that 500-features one; such that If I feed a new instance to a trained auto-encoding network, grab it in the bottle neck layer and send it to you, you must be able to feed the message to the bottle neck layer and receive the full decoded message from the output layer. However there's (always) a catch! and that is you need a really "BIG" dataset to train your deep network. Besides you cannot expect it to work well on every kind of data after being trained by a fixed amount. for example your network cannot successfully retrieve an image of a dog, if all you've fed in the training phase was cats' and lizards' images. So can you guarantee that new messages to be encoded have the same type as the ones in the training set?! If no, then you might also take this challenge into account.


How do I write a good evaluation function for a board game?

Based on your description, I'd maximize the following terms:

i
-max(f - 10 - (MAX_FIELD_INDEX - i), 0) - assuming consumption of one fuel per field; this becomes negative when you have too much fuel
a similar function of p, as spending them gets more important when approaching the goal

As having fuel is probably a good thing in the beginning, you could use a term like f. Similarly for the "power packs" (or are they rather "weakness packs"?).

I'd combine the terms using a linear function like you did and let it optimize. You may need more such terms. Maybe it's simpler to get rid of the power packs when you have enough fuel? Then something like -max(p-f, 0) may help.

You may generate some ad-hoc expressions or add some products of your terms as new terms. You may want to do this after the coefficients of the simpler terms have already been optimized (so you help the more complex optimization with a good staring point).


Which hyperparameter does Neural Architecture search (NAS) use?

Great question! It's not clearly stated (it's not stated at all on wikipedia), but after a bit of searching, I found an answer here about a third of the way down the page:

  The best performing architecture observed during the training of the controller is taken, and a grid search is performed over some basic hyperparameters such as learning rate and weight decay in order to achieve near STOTA (state of the art) performance.

So as a direct answer: The norm; A grid search.


Can we create a conscious agent by having two units: one rational and the other conscious?

Jürgen Schmidthuber has a similar concept of consciousness, maybe check out this article:

  I would like to claim we had little, rudimentary, conscious learning systems for at least 25 years. Back then, already, I proposed rather general learning systems consisting of two modules.

There you also have one module that is occupied with decision making. The other module's job is to create a model of the world.

Of course, a philosopher will tell you that we have no idea what consciousness is or "what it feels like to be ai with two units".


How can I fetch exploration decay rate of an iterable Q-table in Python?

Here is one way to calculate the exploration rate decay:
exploration_rate = min_exploration_rate + \ (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate*episode)


What does the notation sup dist mean in distributional RL?

It doesn't seem that it is a "proper" symbol.

I guess that $\sup$ simply refers to the supremum, that is, you want to select actions that maximize the quantity that comes to the right of $\sup$, while $\text{dist}$ is simply a proxy for any possible distance between distributions. For example, you can replace $\text{dist}$ with the Kullback-Leibler divergence or with the mutual information.


How can I simulate responses from the distribution of human intelligence?

The data you want to collect cannot be reliably simulated at this time. There is no current realistic simulator for a human performing reading comprehension.

The actual error rates and specific wrong answers chosen on the questions will depend to a large degree on specific humans, and the nature of questions. As you are hoping to get realistic results, the only method that will work for your ground truth data is to present 1000+ real humans with your sample questions. In addition, if you want to categorise your humans into "smart", "dumb" etc, you will need to run additional tests on them, such as an IQ test, in order to create those categories.

Depending on context, such as the nature of the questions you want to assess, you may be able to obtain some anonymised data from real-world exams that could help, instead of trying to generate that data yourself. In that case you could make an approximate model for humans answering multiple-choice questions - perhaps by training an LSTM-based natural language model. For best accuracy on your questions you would want the training set to include similar kinds of question. There is still a caveat that NNs do not really do logic or reasoning, they make statistical fits, so can easily get answers logically wrong or select nonsense answers. The best general NLP models still fail badly at semantics.

If you have specific set of questions to evaluate - and are not willing to ignore the content of the questions - then no machine can currently match distribution of human behaviour on such a task without extensive training data and significant effort.

If you don't care about the content of the questions, or assessing their difficulty, or indeed any feedback about the specific questions, then you could maybe use a dataset from any multi-choice questionnaire to get statistics of correct answer accuracy. With this approach your simulation could just be a simple distribution over "correct", "most plausible incorrect" etc answers where you choose "most plausible incorrect" either manually or randomly (but consistently). This will get you a distribution of responses similar to known real-world data. It could be used to unit-test a scoring system perhaps, or demonstrate some stats or visualisation software. But the question and answer text may as well be gibberish at that point.


How to add variation in the results of a neural networks?

I would suggest starting with Generative Adversarial Networks (GAN). They usually are capable of adding some randomness to the output to produce different variants. Moreover, Conditional GANs can generate outcomes regarding the observed condition. Therefore, as you change the condition (as an input to the network) you can get different results.

Some examples of NN for synthetic handwriting generators:

https://github.com/sjvasquez/handwriting-synthesis
http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
https://distill.pub/2016/handwriting/


What kind of problems cannot be solved using machine learning techniques?

At least you should be aware of two points:

P/NP/NP-hard (and all other class of complexities) are thoroughly valid for the machine learning area as well. Because these concepts are related to the fundamental of computations (theory of computation), and machine learning is not an exception here.
One of the useful concepts in the complexity of the learning problem is the VC dimension, PAC learnability, and their related concepts (such as sample complexity). Although these concepts can't be enough to measure the time complexity, they are useful for finding the learner model's capacity.


Given a non-admissible heuristic function, A* will always give a solution if one exists, but there is no guarantee it will be optimal.

I won't duplicate the proof here, but it isn't too hard to prove that any best-first search will find a solution for any measure of best, given that a path to the solution exists and infinite memory. A* is a best-first search algorithm, so it will always find a solution if one exists.

Which heuristics guarantee the optimality of A*? Is the admissibility of the heuristic always a necessary condition for A* to produce an optimal solution?

Admissibility is not a necessary condition. Take any admissible heuristic $h_1$ and make a new function $h(n) = h_1(n)+5$. This heuristic is not admissible, but if you run A* on it, it will still find optimal solutions.
But, we also have to ask what you mean by "the optimality of A*", because optimality can have two senses here. My point in the previous paragraph is in the sense of returning optimal paths. An alternate interpretation is that no algorithm performs fewer expansions that A* with the same information. This is probably not what was meant, and the answer in that context is far more complicated. But, with an inconsistent (but admissible) heuristic, A* can perform exponentially more expansions than other known algorithms, and thus is not the optimal algorithm to use.


What is "early stopping" in machine learning?

In some iterative learning methods the more iterations you apply the more specific your model becomes about the training set. If there are too many iterations, your model will become too specifically trained for the training samples and will score less on other samples that are not seen during the training phase. This is call over-fitting, though over-fitting is not specific to iterative learning methods.

One solution to prevent over-fitting in these iterative learning algorithms is early stopping. Normally a control group of samples called validation samples (validation set) are used to validate the model and notify when it starts to over-fit. The validation set is not used by the training algorithm, however, its corresponding outputs are known and after each iteration, its samples are employed to measure how good the model currently works. As soon as the performance on the validation set stops to grow and starts to drop we stop iterating the training algorithm. This is called early stopping which can help to maximize the generalization power of our learned model.

Note that if we use the training set itself for validation the performance will always increase because that is what the learning algorithm is designed to do. However, the learning algorithm does not know how specifically it should learn the training set and that is why we need methods like early stopping.


How is Google Translate able to convert texts of different lengths?

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can find two very good papers here, and here).
During training, to ensure the same size of the input and output they can just search for the longest sentence they have in the dataset or select a number that is high enough and pad all the other sentences with 0. In addition, they add a stop token where the sentence ends, so that the model is aware of this. When decoding (inference), the decoder is going to predict one word at a time until it predicts the stop token, which signals that the translation is done.
If you're interested to see an actual implementation, I would recommend looking at this tutorial which does a good job at explaining the code and how it works.


multi vs one prediction using Regression

I think the model will have no problem taking a multicolumn input. In fact, from your code, this is exactly how you trained it. It expects an input of size [k, 6], where k is k>=1.

Instead you are feeding it with [k, 1] sized data, which are the dimensions of y. So you if you run it like this it should work:

regressionModel.predict(X)


Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$
Which can be equivalently written as
$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$
Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities

Question 2
Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this
$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_
{a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$
The discounted return form is usually denoted
$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$
Where you notice the discounting terms aren't normalised which motivates cancelling it out


How long has it taken for autonomous driving cars to be being sold and used on the roads today?

There are a couple of things to consider with this question:

First, training time can be a deceptive measure. Neural network training is considered "trivially" parallelizable. This means that the more computers you have access to, the faster you can train (most of the time, one computation doesn't depend on another, so you can do them both at the same time). Since the start of the current phase of self-driving cars, GPU performance has increased by more than a factor of 5. Further, large companies like Google have migrated to AI-specific ASIC chips. These are faster still. This means any estimate of "time" is likely to be confusing or misleading. The amount of time required to train a network has dropped rapidly over time, even if the number of processing cycles required has stayed the same.

Second, it is very unlikely that self-driving car companies have spent that time training a single model. Instead, they are probably starting from a given model (say, last month's), and trying different approaches to improving it, with new data, alternative training methods, or "expert" domain knowledge. This makes it difficult to reason about what it means to "get the results that are used in today's self-driving vehicles." If we count all the aborted attempts, probably it's on the order of high hundreds or low thousands of person-years of researchers (6-7 years, and there aren't thousands of people working in research in this area). If you also count related development and engineering efforts, it's probably an order of magnitude or more than that.

That said, it might be more interesting to think about the amount of simulated driving time that is needed to train a network in this way. As discussed in this excellent blog post, "Deep Reinforcement Learning Doesn't Work Yet", the amount of training experience required to train a neural net for this kind of problem is extremely large. To learn to play simple Atari games better than human players required about 244 hours of exposure. These games are typically just a single screen, and most humans can pick them up in a couple of minutes or less. This site estimates average time for a human to learn to drive competently at just shy of 70 hours. Applying the same ratio, we can infer that a deep neural net would want something like 2 years of experience driving to achieve human level performance. This seems like the right ballpark, but probably a lot of that driving is done "offline" in a simulated environment, rather than operating the vehicle directly.

Of course, these are just ballpark estimates. The exact figures are likely to be proprietary. Further, there are some reports that modern systems are abandoning neural nets, and moving back to a more "rule-based" paradigm because of the difficulties in training them. I'm not sure how much credit to give those reports, but it again makes it difficult to pin down a training time.


Can you provide some pseudocode examples of what constitutes an AI?

AI is not a simple term. There are different types, ranging from the most simplistic rule-based AI to black-box AI's so complicated it's unreasonable for a human to understand exactly what they're doing.

There's no pseudocode that if used in a program automatically constitutes it as an AI. It's not that black and white. But I can give examples:

Here's a rule-based chess AI that forfeits if it's too far behind, and plays aggressively if it's far enough ahead.

if player.score - my.score > 10:
    forfeit
elif my.score - player.score > 10:
    agressive = True

for each piece of my.pieces:
    for each square of board.squares:
        if noThreats(square) and agressive is True:
            move(piece, square)
            return

This is considered an "AI" because it feigns intelligence - appearing to have a true understanding of chess while simply following a set of rules, making it an "Artificial" Intelligence.

Here's another more complicated AI:

decisionNet = NeuralNetwork(64 inputs, 2 outputs)

choice = decisionNet(board.squares) // Returns a chess square with one of my pieces and desitnation
move(choice)

This uses a neural network to make the decision, which could have been trained on a bunch of example games or against itself. Due to this "training phase", humans can't understand precisely what the network is doing without extensive effort, so it's an even gives an even more convincing understanding of chess. But if we want, we could still understand the nuances of this network, and show it doesn't possess an intelligence, it again only feigns it.

I should mention that virtually any code that has an if statement can be considered AI. The examples I provided are just easier to pass off as understanding a very complicated concept (chess), as opposed to, say, verifying a user login. They both have the same fundamentals, it's just one appears more complicated on the surface than the other.


Looping over Sarsa algorithm for better Q values

Given that your state space is continuous, then I would recommend using Deep Q-Learning. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.


In what RL algorithm category is MiniMax?

I think you are looking at it from the wrong direction, min-max is just a planning algorithm, decision strategy, in the sense that you are describing other algorithms/methods it does not have a category. For example, you have negamax algorithm which is in a sense the same thing the Monte Carlo Search Tree is to Monte Carlo. Min-max category is game theory really.

Now you should be thinking about RL algorithms in another way, and this is taxonomy:

So if you think about methods, you mentioned, let's put them in the right place:

TD methods in general - model free
Monte Carlo methods - model free
MinMax - model-based (that could be discussed but it definitely needs access to a world model)


What is the value of a state when there is a certain probability that agent will die after each step?

The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy.

According to the optimal policy, the agent would choose to stay in the square (1,1) every time, but since it has a 0.8 probability of actually staying (and 0.2 probability of dying), we can compute the value of the agent using the Bellman equation as:

$$
V(1,1) = 1 + 0.8 V(1,1) + 0.2 V(\text{death state}) \\
\implies V(1,1) = 1 + 0.8 V(1,1) \\
\implies V(1,1) = \frac{1}{1 - 0.8} \\
\implies V(1,1) = 5
$$

There are other ways of deriving the same number (value function has multiple definitions) but they are equivalent.


Are Neural Net architectures accidental discoveries?

Although there is a strong element of "try and see" that has driven successful architectures, the drivers for what to try are often inspired by underlying theory or knowledge from other disciplines.

Specifically for basic CNN, which led to AlexNet and many of the best image processing, the concept of using local receptive fields in layers was inspired by study of neurons in the cat visual system.

Modern RNNs also did not appear out of nowhere, there has long been an appreciation of the difference between a feed-forward network and a recurrently-connected one, and the different applications possible. The step change to LSTM was deliberate response to analysis of problems training simplest forms of RNN.

Like much of science, these things are also driven by success in the real world following the research. Many promising ideas have been tried and rejected. Some have been used for a while then superseded, e.g. using RBMs or stacked auto-encoders to pre-train deep networks before ReLUs and Xavier initialisation were discovered - although both RBMs and auto-encoders still have their niches.

Tweaks to architectures, such as variants of LSTM/GRU, may even be deliberately searched and assessed as part of research. That is done with the explicit knowledge that this part of finding a good design is best done as a search across possibilities.

Despite the evolution-like progress, presenting all such advances as completely random or pure GA-like search is ignoring the conscious effort and research that leads to the designs. If you search literature on any major successful design (such as the existence of RNNs or CNNs in the first place), and read the papers, you will often find that modern neural network architectures have deep roots in older research, plus have mathematical and/or scientific justifications for the choices made.


How much is currently invested in artificial general intelligence research and development?

In the last years, there have been big investments in AI technologies. For an overview, maybe take a look at this article Artificial Intelligence: Investment Trends and Selected Industry Uses (2019).
A few companies that have the long-term goal of creating an AGI, although, currently, they mainly do research on specific problems (e.g. video games) or AI techniques (e.g. reinforcement learning), have received many funds. I will only list a few (maybe the most well-known ones) of these companies below, but there are probably many other companies that have this long-term goal and have been funded by other companies or people.
DeepMind
In their site, they write

Like the Hubble telescope that helps us see deeper into space, we aim to build advanced AI - sometimes known as Artificial General Intelligence (AGI) - to expand our knowledge and find new answers. By solving this, we believe we could help people solve thousands of problems.

DeepMind was acquired by Google in 2015 for $500 million, given its success in playing games at a superhuman performance, which is a promising step towards the development of more AI techniques and maybe AGI.
The Wikipedia article on DeepMind contains some information about people or companies that have invested in DeepMind, which includes companies Horizons Ventures and Founders Fund, and people  Scott Banister, Peter Thiel, Elon Musk, and Jaan Tallinn, although I cannot give you the exact numbers in terms of capital. In any case, serious investments have been done in DeepMind, which is definitely one of the promising companies that could develop good insights into the development of AGI systems.
OpenAI
Another company that has a similar goal and is doing research on similar topics (such as reinforcement learning or natural language processing) is OpenAI, which also has the long-term goal of creating AGI systems, as they write in their website

OpenAI’s mission is to ensure that artificial general intelligence (AGI) — by which we mean highly autonomous systems that outperform humans at most economically valuable work — benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.

Their investors include Microsoft, Reid Hoffman's charitable foundation, and Khosla Ventures.
Vicarious
They write in their site

Artificial general intelligence is the finish line on our journey toward progressively more capable robots. Our approach leverages deep expertise in neuroscience and is shaped by a decade of research.

They are apparently backed by more than 150 million dollars from people like Jeff Bezos, Elon Musk and Mark Zuckerberg and companies like Samsung.


Expected behavior of adversarial attacks on deep NN?

This seems to be a known problem, and intuitively seems reasonable. You might be interested in the paper Adversarial Training Can Hurt Generalization.
The authors suggest that this might be because training on the perturbed data requires the model to learn more robust features, which means more samples are required to obtain performance comparable to a model that is not adversarially trained.
You could try collecting or generating additional samples to see if this leads to an improvement. They also mention that in their experiments on the MNIST dataset, using Xavier initialisation led to a significant benefit, so you could experiment with that too.


What's done towards AI learning new ways of learning?

I think the closest thing is building up knowledge using predictions like in the Horde Architecture. Research about what are good predictions to make is on going at the University of Alberta, Canada but the Horde architecture has the potential to ask new questions and generate new data based on the answers to those questions in the form of predictions.

One example is having a robot with a bump sensor and servos and asking questions like

  Will rotating the servos CW for 90 degrees make the bump sensor turn on?

This question can be phrased and it's answer estimated mathematically in the form of a General Value Function as defined in the paper.

Asking different questions based on predictions like this one is where the power comes from.


Pros and Cons of Seq2Seq vs Bidirectional RNN

Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in there classic form. Seq2Seq models are used to generate a sequence from another sequence, consider for example tranlation task from one language to another. In that sense Seq2Seq is more a family of models not an architecture, while Bidirectionnal RNNs is a neural network architecture and can be used to build several models including Seq2Seq, for example the encodign part of the Seq2Seq can be a Bi-RNN, but they also can be used for other tasks, for example sentence classfication, sentiment analysis...


What are the state-of-the-art meta-reinforcement learning methods?

One of the most recent papers on meta-RL is meta-Q-learning. This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-reinforcement learning (meta-RL). MQL builds upon three simple ideas.

Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory.

Using a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies.

past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates

Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with state-of-the-art meta-RL algorithms.
I think that other references to other work on meta-RL are present in the experiments part of the MQL paper.


What is the best variant of darknet to use?

I used Trieu's Darkflow and it was trained well up to around 30 to 50 epoch with maximum 30 GB dataset. At the same time, it was quite often crashed after these epochs.

Half precision...I don't remember Darkflow have that.

One way to estimate each fork reputation is by number of Star and Issues, for example Alexey's Darknet 5,000 and 2,500 and Darkflow 5,000, 539 respectively.

Reference: OS Ubuntu 16.04


How to distinguish between proper nouns and other words in NLP?

Essentially you would need to train a named entity recognizer (NER) to recognize the names out of the common words.

There are many works that try to use a similar language to the language in question as a pivot to train a full NER model (for example, Cheap Translation for Cross-Lingual Named Entity Recognition).

Your task might be slightly simpler, as you are interested only in one class: whether it is a named entity or not. But in general it is very similar to this setup.


RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected return (aka utility) $G$ from a starting state $s$ may be defined like this:

$$v(s) = \mathbb{E}_{\pi}[G_t|S_t=s] = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^kR_{t+k+1}|S_t=s] $$

Where $R_t$ is the reward distribution at time $t$, and $\mathbb{E}_{\pi}$ stands for
expected value given following the policy $\pi$ for action selection.

There are a few variations of this, depending on setting and which value function you are interested in. However, all value functions used in RL look at future sums of reward from the decision point when the action is taken. Past rewards are not taken into account.

An agent may still select to take an early high reward over a longer term reward, if:

The choice between two rewards is exclusive
The return is higher for the early reward. This may depend on the discounting factor, $\gamma$, where low values make the agent prefer more immediate rewards.

If your problem is that an agent selects a low early reward when it could ignore it in favour of something larger later, then you should check the discount factor you are using. If you want a RL agent to take a long term view, then the discount factor needs to be close to $1.0$.

The premise of your question however is that somehow a RL agent would become "lazy" or "complacent" because it already had enough reward. That is not an issue that occurs in RL due to the way that it is formulated. Not only are past rewards not accounted for when calculating return values from states, but there is also no formula in RL for an agent receiving "enough" total reward like a creature satisfying its hunger - the maximisation is applied always in all states.

There is no need to somehow decay past rewards in any memory structure, and in fact no real way to do this, as there is no data structure that accumulates past rewards used by any RL agent. You may still collect this information for displaying results or analysing performance, but the agent doesn't ever use $r_{t}$ to figure out what $a_t$ should be.

  I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of R

You have probably formulated the reward function incorrectly for your problem in that case. A cumulative reward scheme (where an agent receives reward $a$ at $t=1$ then $a+b$ on $t=2$ then $a+b+c$ on $t=3$ etc) would be quite specialist and you have likely misunderstood how to represent the agent's goals. I suggest ask a separate question about your specific environment and your proposed reward scheme if you cannot resolve this.


AI Knowledge Based System: I am having difficulties understanding proposition logic

The provided result is a non-sense for this input.

In a closed world paradigm (ie: prolog), where all non-provable facts are assumed as false, from "p2->p3 and p3->p1", p2 is false, and the program result should be "p2 ? No".

In an open world, as it is not possible to proof "p2" nor "not p2", the result must be "p2 ? unknown"

"p2 -> p3 and p3 -> p1" doesn't provides any information about "p2" in the same way that "if it is sunny then I'm happy" doesn't provides information about if it is sunny or raining.

The meaning of the number after the "yes" (the number of "models") is unknown. Even present two true tables for a single fact "p2 -> p3 and p3 -> p1" can be considered an error.


Where do the feature extraction and representation learning differ?

Feature extraction (FE) is not the same as representation learning (RL), but they are similar and related.
You describe accurately what feature extraction typically refers to, i.e. the process of extracting (new) features from existing ones or raw data (e.g. images). For example, let's say you have a dataset associated with a car. You have only two features in your dataset: distance and velocity. However, from these two, you can extract a third feature, e.g. the acceleration. So, feature extraction can be performed with a fixed algorithm (e.g. PCA for dimensionality reduction or SIFT) or manually.
Representation learning is the collection of all techniques that extract features automatically from the data (i.e. they learn the features or representations, hence the name representation/feature learning). So, for example, a convolutional neural network trained on ImageNet can (and/or needs to) learn general features in order to solve the corresponding classification task. (Chapter 9 of the book Deep Learning by Goodfellow et al. talks more about this topic.) These features are learned from the data (and that's why CNNs are data-driven), and they can later be exploited for transfer learning (TL), i.e. TL is based on the idea that neural networks learn general representations (which is thus a synonym for features) of data that can be exploited to solve other tasks (sometimes known as downstream tasks, especially in the context of self-supervised learning).
Yoshua Bengio et al. define representation learning as follows

learning representations of the data that make it easier to extract useful information when building classifiers or other predictors

So, RL is a subset of FE, given that RL also extracts features, but RL emphasizes the extraction of features automatically.


Why is non-linearity desirable in a neural network?

Consider what happens if you intend to train a linear classifier on replicating something trivial as the XOR function. If you program/train the classifier (of arbitrary size) such that it outputs XOR condition is met whenever feature a or feature b are present, then the linear classifier will also (incorrectly) output XOR condition is met whenever both features together are present. That is because linear classifiers simply sum up contributions of all features and work with the total weighted inputs they receive. For our example, that means that when the weighted contribution of either feature is sufficient already to trigger the classifier to output XOR condition is met, then obviously also the summed contributions of both features are sufficient to trigger the same response.
To get a classifier that is capable of outputting XOR condition is met if and only if the summed contributions of all input features are above a lower threshold and below an upper threshold, commonly non-linearities are introduced. You could of course also try to employ a quadratic function to solve the two-feature problem, but as soon as the number of variables/features increases again, you run into the same problem again, only in higher dimensions. Therefore, the most general approach to solving this problem of learning non-linear functions, like the XOR, is by setting up large models with enough capacity to learn a given task and equipping them with non-linearities. That simplifies training since it allows for using stochastic gradient descent for training the system/classifier, preventing one from having to solve Higher-Degree polynomial equations analytically (which can get computationally quite expensive quite quickly) to solve some task.
In case you are interested, here's one paper analyzing and dealing with the XOR problem (as one concrete instance of a problem where purely linear models fail to solve some task).
EDIT:
You can consider a layer in a network as a function $y = f(x)$, where $x$ is the input to some layer $f$ and $y$ is the output of the layer. As you propagate $x$, being the network's input, through the network, you get something like $y = p(t(h(g(f(x)))))$, where $f$ is the input layer and $p$ constitutes the output layer, i.e. a set of weights, by which the input to that respective layer gets multiplied.
If $h$, for example, is some non-linear activation function, like ReLU or sigmoid, then $y$, being the network's output, is a non-linear function of input $x$.


Why is the n-step tree backup algorithm an off-policy algorithm?

As for your first two questions: there is indeed a behaviour and a target policy, which can be different. In the example image of the $3$-step tree-backup update in the beginning of the section you mention, the actions $A_t$, $A_{t+1}$, and $A_{t+2}$ are assumed to be selected according to some behaviour policy, whereas a (different) target policy is used to determine weights for the different leaf nodes.

As for your third question, in the case where our target policy is greedy, lots of terms will indeed have $0$ weights and therefore entirely drop out. However, this is not always going to fall down to the same return as $n$-step Sarsa returns; that would not be correct, because ($n$-step) Sarsa is an on-policy algorithm.

In the case where the target policy is a greedy policy, the return will depend very much on how many actions happened to get selected by the behaviour policy which the greedy target policy also would have "agreed" with. If the behaviour policy already happened to have made a "mistake" (selected an action that the greedy policy wouldn't have selected) for $A_{t+1}$, you'll end up with a standard $1$-step $Q$-learning update. If the behaviour policy only made a "mistake" with the action $A_{t+2}$ (and agreed with the target policy on $A_{t+1}$), you'll get something that kind of looks like a "$2$-step $Q$-learning update" (informally here, because $n$-step $Q$-learning isn't really a thing). Etc.

You can see that this is the case by closely inspecting Equation 7.16 from the book:

$$G_{t:t+n} \doteq R_{t+1} + \gamma \color{blue}{\sum_{a \neq A_{t+1}} \pi (a \mid S_{t+1}) Q_{t + n - 1}(S_{t+1}, a)} + \gamma \color{red}{\pi (A_{t + 1} \mid S_{t + 1}) G_{t + 1 : t + n}}.$$

Suppose that the target policy $\pi$ is greedy, and the behaviour policy already selected action $A_{t+1}$ differently from what $\pi$ would have selected (meaning that $\pi(A_{t+1} \mid S_{t+1}) = 0$). Then we know that the $\color{red}{\text{red}}$ part of the equation will evaluate to $0$. In the $\color{blue}{\text{blue sum}}$, the $\pi(a \mid S_{t+1})$ term will only evaluate to $1$ for $a = \arg\max_a Q_{t+n-1}(S_{t+1}, a)$ (because that is the action $a$ to which the greedy target policy would assign a probability of $1$), and $0$ again for all other elements of the sum. So, in this example situation, we end up using the return $G_{t:t+n} = R_{t+1} + \gamma \max_a Q_{t+n-1}(S_{t+1}, a)$. This is exactly the return also used by the standard $1$-step $Q$-learning update rule.
Now suppose that the behaviour policy and a greedy target policy happened to "agree" with each other on $A_{t + 1}$, but disagree again on $A_{t + 2}$. In this case, the entire $\color{blue}{\text{blue sum}}$ for $G_{t:t+n}$ will evaluate to $0$, but the $\pi(A_{t+1} \mid S_{t+1})$ in the $\color{red}{\text{red part}}$ will evaluate to $1$. Then, "inside" the recursive $\color{red}{G_{t+1:t+n}}$, we'll get a very similar situation to the example above, with a single non-zero term in the blue sum and a red part that evaluates to $0$. In this case, the total return will be $G_{t:t+n} = R_{t + 1} + \gamma R_{t + 2} + \gamma^2 \max_a Q_{t + n - 1}(S_{t + 2}, a)$. Note that this is different from a $2$-step Sarsa return. $2$-step Sarsa would use a return with $\gamma^2 Q_{t+n-1}(S_{t+2}, A_{t+2})$ at the end (where $A_{t+2}$ is sampled according to the behaviour policy), rather than that term involving $\max_a$.

What you essentially end up getting in these cases is returns similar to those of $n$-step Sarsa, but they automatically get truncated as soon as there is disagreement between the behaviour policy and a greedy target policy (all subsequent reward observations get replaced by a single bootstrapping value estimate instead).

In the examples above, I assumed completely greedy target policies, since that is what you appeared to be most interested in in your question (and is probably also the most common use-case of off-policy learning). Note that a target policy does not have to be greedy. You can also have non-greedy target policies if you like, and then the returns will obviously change quite a bit from the discussion above (fewer $\pi(S, A)$ terms would evaluate to $0$, there'd be more non-zero terms).


Why they use KL divergence in Natural gradient?

The KL divergence has slightly different interpretations depending on the context. The related Wikipedia article contains a section dedicated to these interpretations. Independently of the interpretation, the KL divergence is always defined as a specific function of the cross-entropy (which you should be familiar with before attempting to understand the KL divergence) between two distributions (in this case, probability mass functions)

\begin{align}
D_\text{KL}(P\parallel Q)
&= -\sum_{x\in\mathcal{X}} p(x) \log q(x) + \sum_{x\in\mathcal{X}} p(x) \log p(x) \\
 &= H(P, Q) - H(P)
\end{align}
where $H(P, Q)$ is the cross-entropy of the distribution $P$ and $Q$ and $H(P) = H(P, P)$.

The KL is not a metric, given that it does not obey the triangle inequality. In other words, in general, $D_\text{KL}(P\parallel Q) \neq D_\text{KL}(Q\parallel P) $.

Given that a neural network is trained to output the mean (which can be a scalar or a vector) and the variance (which can be a scalar, a vector or a matrix), why don't we use a metric like the MSE to compare means and variances? When you use the KL divergence, you don't want to compare just numbers (or matrices), but probability distributions (more precisely, probability densities or mass functions), so you will not compare just the mean and the variance of two different distributions, but you will actually compare the distributions. See the example of the application of the KL divergence in the related Wikipedia article.


How to estimate the accuracy upper limit of any CNN model over a computer vision classification task

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set.  If your training set is "similar" in quantity and quality to what was used  for the accuracy achieved by the transfer learning model in some application you have a reasonable chance of coming close to that accuracy. By similar is mean roughly the same number of images per class. By quality I mean what percentage of the pixels in the images that are occupied by the "region of interest -ROI), same level of noise in the images etc. Also it depends on the nature of the classes. If the classes are widely different (elephants vs trees) the accuracy should be higher than if you are try to classify closely related images (human faces).


What is OpenCog?

OpenCog is a project with the vision of creating a thinking machine with human-level intelligence and beyond.

In OpenCog's introduction, Goertzel categorically states that the OpenCog project is not concerned with building more accurate classification algorithms, computer vision systems or better language processing systems. The OpenCog project is solely focused on general intelligence that is capable of being extended to more and more general tasks.

Knowledge representation

OpenCog's knowledge representation mechanisms are all based fundamentally on networks. OpenCog has the following knowledge representation components:

AtomSpace: it is a knowledge representation database and query engine. Data on AtomSpace is represented in the form of graphs and hypergraphs.

Probabilistic Logic Networks (PLN's): it is a novel conceptual, mathematical and computational approach to handle uncertainty and carry out effective reasoning in real-world circumstances.

MOSES (Meta-Optimizing Semantic Evolutionary Search): it implements program learning by using a meta-optimization algorithm. That is, it uses two optimization algorithms, one wrapped inside the other to find solutions.

Economic Attention Allocation (EAA): each atom has an attention value attached to it. The attention values are updated by using nonlinear dynamic equations to calculate the Short Term Importance (STI) and Long Term Importance (LTI).

Competency Goals

OpenCog lists 14 competencies that they believe AI systems should display in order to be considered an AGI system.

Perception: vision, hearing, touch and cross-modal proprioception

Actuation: physical skills, tool use, and navigation physical skills

Memory: declarative, behavioral and episodic

Learning: imitation, reinforcement, interactive verbal instruction, written media and learning via experimentation

Reasoning: deduction, induction, abduction, causal reasoning, physical reasoning and associational reasoning

Planning: tactical, strategic, physical and social

Attention: visual attention, behavioural attention, social attention

Motivation: subgoal creation, affect-based motivation, control of emotions

Emotion: expressing emotion, understanding emotion

Modelling self and other: self-awareness, theory of mind, self-control

Social interaction: appropriate social behavior, social communication, social inference and group play

Communication: gestural communication, verbal communication, pictorial communication, language acquisition and cross-modal communication

Quantitative skills: counting, arithmetic, comparison and measurement.

Ability to build/create: physical, conceptual, verbal and social.

Do I endorse OpenCog?

In my opinion, OpenCog introduces and covers important algorithms/approaches in machine learning, i.e. hyper-graphs and probabilistic logic networks. However, my criticism is that they fail to commit to a single architecture and integrate numerous architectures in an irregular and unsystematic manner.

Furthermore, Goertzel failed to recognize the fundamental shift that came with the introduction of deep learning architectures so as to revise his work accordingly. This puts his research out of touch with recent developments in machine learning


Why can't the XOR linear inseparability problem be solved with one perceptron like this?

It can be done.

The activation function of a neuron does not have to be monotonic. The activation that Rahul suggested can be implemented via a continuously differentiable function, for example $ f(s) = exp(-k(1-s)^2) $ which has a nice derivative $f'(s) = 2k~(1-s)f(s)$. Here, $s=w_0~x_0+w_1~x_1$. Therefore, standard gradient-based learning algorithms are applicable.

The neuron's error is $ E = \frac{1}{2}(v-v_d)^2$,
where $v_d$ - desired output, $v$ - actual output. The weights $w_i, ~i=0,1$ are initialized randomly and then updated during training as follows
$$w_i \to w_i - \alpha\frac{\partial E}{\partial w_i}$$
where $\alpha$ is a learning rate. We have
$$\frac{\partial E}{\partial w_i} = (v-v_d)\frac{\partial v}{\partial w_i}=(f(s)-v_d)~\frac{\partial f}{\partial s}\frac{\partial s}{\partial w_i}=2k~(f(s)-v_d)(1-s)f(s)~x_i$$
Let's test it in Python.

import numpy as np
import matplotlib.pyplot as plt

For training, I take a few points randomly scattered around $[0, 0]$, $[0, 1]$, $[1, 0]$, and $[1, 1]$.

n = 10
sd = [0.05, 0.05]

x00 = np.random.normal(loc=[0, 0], scale=sd, size=(n,2))
x01 = np.random.normal(loc=[0, 1], scale=sd, size=(n,2))
x10 = np.random.normal(loc=[1, 0], scale=sd, size=(n,2))
x11 = np.random.normal(loc=[1, 1], scale=sd, size=(n,2))

x = np.vstack((x00,x01,x10,x11))
y = np.vstack((np.zeros((x00.shape[0],1)),
               np.ones((x01.shape[0],1)),
               np.ones((x10.shape[0],1)),
               np.zeros((x11.shape[0],1)))).ravel()

ind = np.arange(len(y))
np.random.shuffle(ind)

x = x[ind]
y = y[ind]
N = len(y)

plt.scatter(*x00.T, label='00')
plt.scatter(*x01.T, label='01')
plt.scatter(*x10.T, label='10')
plt.scatter(*x11.T, label='11')
plt.legend()
plt.show()

Activation function:

k = 10

def f(s):
    return np.exp(-k*(s-1)**2)

Initialize the weights, and train the network:

w = np.random.uniform(low=0.25, high=1.75, size=(2))

print("Initial w:", w)

rate = 0.01
n_epochs = 20

error = []
for _ in range(n_epochs):
    err = 0
    for i in range(N):
        s = np.dot(x[i],w)
        w -= rate * 2 * k * (f(s) - y[i]) * (1-s) * f(s) * x[i]
        err += 0.5*(f(s) - y[i])**2
    err /= N
    error.append(err)

print('Final w:', w)

The weights have indeed converged to $w_0=1,~w_1=1$:

Initial w: [1.5915165  0.27594833]
Final w: [1.03561356 0.96695205]

The training error is decreasing:

plt.scatter(np.arange(n_epochs), error)
plt.grid()
plt.xticks(np.arange(0, n_epochs, step=1))
plt.show()

Let's test it. I create a testing set in the same way as the training set. My test data are different from my training data because I didn't fix the seed.

x00 = np.random.normal(loc=[0, 0], scale=sd, size=(n,2))
x01 = np.random.normal(loc=[0, 1], scale=sd, size=(n,2))
x10 = np.random.normal(loc=[1, 0], scale=sd, size=(n,2))
x11 = np.random.normal(loc=[1, 1], scale=sd, size=(n,2))

x_test = np.vstack((x00,x01,x10,x11))
y_test = np.vstack((np.zeros((x00.shape[0],1)),
               np.ones((x01.shape[0],1)),
               np.ones((x10.shape[0],1)),
               np.zeros((x11.shape[0],1)))).ravel()

I calculate the root mean squared error, and the coefficient of determination (R^2 score):

def fwd(x,w):
    return f(np.dot(x,w))

RMSE = 0

for i in range(N):
    RMSE += (fwd(x_test[i],w) - y_test[i])**2

RMSE = np.sqrt(RMSE/N)

print("RMSE", RMSE)

ybar = np.mean(y)

S = 0
D = 0
for i in range(N):
    S += (fwd(x_test[i],w) - y_test[i])**2
    D += (fwd(x_test[i],w) - ybar)**2

r2_score = 1 - S/D
print("r2_score", r2_score)

Result:

RMSE 0.09199468888373698
r2_score 0.9613632278609362

... or I am doing something wrong? Please tell me.


How to handle equality constraints in the mutation operation of evolutionary algorithms?

If X is your 6D vector and m(X) is the mutated version of X, then you can renormalise the mutant back to unity by dividing by the sum of X, i.e. X' = m(X)/sum(X).
However, I encourage you figure out how to mutate a vector while keeping the length of the vector at 1. One way to do this would be to randomly rotate your vector in 6D space. The length should stay the same, and you don't need to renormalise it afterwards.