How can thousand-robot swarm coordinate their moves without bumping into each other?

There has been quite a few approaches to achieve such kind of distributed coordination. I present here one of them, for its generality and simplicity (that makes it easy to remember too). But first, the general idea behind these approaches is pretty interesting, around a mechanism called stigmergy.

Stigmergy is a behaviour coordination mechanism mediated by the environment. It was first described for termites, and the most famous example pertains to ants. Ants form trails when going out for food, but they often do not interact directly. It turns out they leave pheromones on the ground as they walk away from their hills. The pheromone allows them to find their way home, and it also guides where they are going: If they find a pheromone trace from one of their peers, they follow it and their own pheromones add up, reinforcing the signal of the trail. In stage, more and more ants get "together as they move", forming a trail. Coordination has been achieved.

Among the various implementation derived from stigmergy, there is the "field-based motion coordination model" (FBMCM). The idea is to create a (maybe virtual) environment that maintains some states of the world. Each object registers in the environment a signal that is maximum at the object position (its edges) and then decreases with distance. Moving objects (e.g. robots) each emit a signal relayed by the environment. They can then sense each other's field and act accordingly: E.g., when signals are strong, move away; when weak, it is safe to get closer, etc. Several complex group moves have been demonstrated in software simulators (platoon formation in games, drill simulations) and with robots. The benefit of this approach is that it can be cheap to compute even complex behaviours. For example, avoiding clashes requires simple "logic" code based on summing-up nearby fields value. FBMCM is pretty slick, used in video-games, but hard to implement in physical settings (to my dating knowledge), as it can be challenging to build a reliable environment. See for example the work from Mamei and Zambonelli, as well as one of the first industrial implementations for robots by Weyns et al.. Note that the implementation for robots required significant work on the environment infrastructure, made somewhat more feasible as it was a controlled warehouse.

The advantage of stigmergy-like models is that they are often simple and resilient: You can lose an ant without impact on the food-finding trail. On the downside, these models are usually slow, as the coordination takes time to emerge from indirect interactions. This can be improved upon by adding extra direct interactions (e.g. empowering ants with a GPS and a grocery store map, or just a magnetic-North sense).

In practice, these models can collapse if the environment implementation is not reliable. It can be difficult for robots or, say, self-driving cars, if they expect some transponders put on their way, as these devices are expensive to set and maintain, and they can be broken or stolen. It would be better to endow robots with radars, sonars or other proximity sensors to implement stigmergic models. One related example is the decision by Tesla to add radars to its cars, instead of assuming reliable transponders on the road (note: This is just a parallel; there is no official relation).

Other implementations and related models are, for example, tuple-based coordination languages such as Linda, and network protocols like Chord. As you see, these works are not necessarily in the "AI domain".


Will artificial super-intelligence evolve to have selfishness inherent in biological systems?

AI will only "evolve" selfishness if it "evolves" in a competitive environment and has certain human-like faculties.

Self-protecting desires on the other hand are logical consequences of having any goal at all. After all, you can't reach your goal if you are destroyed.

The concern of "evil" super intelligences isn't that they literally turn evil and selfish and cruel. Those are human qualities.

Instead a superintelligence that has a certain goal, will logically pursue subgoals that help it to reach the ultimate goal. Such subgoals will be resources, power, safety.

So it will amass power and resources to reach its goal and exterminate any threat to its existence as long as its goal hasn't been reached, without being selfish at all.


Is running more epochs really a direct cause of overfitting?

The Problem of Overfitting
In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general a good way to avoid overfitting, except for fine-tuning, regularization, dropout etc, is to understand what you have from the learning curve. In most cases overfitting happens, after some epochs have passed, and, as a result the training error still decreases, whereas the validation error flactuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.
Methods: early-stopping, checkpoint.
useful link: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/


Is there any research on the identification of a person's feelings using features such as facial expressions or body temperature?

Yes, there is research on this topic. The field that studies it is known as affective computing (AC). Emotion recognition seems to be a specific problem in affective computing, i.e. the recognition of emotions, while AC is also concerned with giving machines the ability to convey emotions (in fact, this paper differentiates the two). There's also sentiment analysis, which refers to the analysis of the sentiment of people e.g. in social media (e.g. in comments) using natural language processing techniques, which doesn't seem exactly what you are looking for. These fields/tasks are all very related/similar, but I don't think they are exactly synonymous (although I'm not an expert in this topic).
Apart from the linked Wikipedia articles, you can find another list of related resources (including important papers and books, and software) here. If you're interested in an easy overview of the AC field, you should probably read the paper Affective computing: challenges, by Rosalind W. Picard (who is one of the leading researchers in this field). In this paper, she differentiates between emotions (what you observe/express with gestures, facial expressions, etc.) and feelings (internal states, which sometimes are not even clear to the person that is having them), and compares emotions to the weather: more precisely, when you say that the weather is windy, you're just giving a label to a fuzzy set of conditions: for example, there's the wind, it could start raining soon, etc., but these conditions are not always clear: for instance, how strong should the wind be in order for you to consider the weather windy? I found this is a nice analogy that conveys what emotions are and why interpreting emotions seems to be a difficult task.
Additionally, you may also be interested in the loving AI project.


How to fill missing values in a dataset where some properties can be inputs and outputs?

As you mentioned in the comments about a possible problem of using mean, median type of imputations naively could lead to wrong predictions. In such cases, you need to first check whether you have enough data.

If you have enough data

You can try using MICE (Multivariate Imputation By Chained Equations) algorithm on your missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. One note of caution with this method: It is a computationally expensive method, so it use it if you are not short on time.

The important thing to keep in mind is that in order to tackle such problems, you would be needing multiple iterations as a part of your algorithm. The conventional setup you are describing, does not seem to iterative in nature and hence you are running into the problem of features being input and output at the same time.

If by any chance you insist on finding the missing values just for solving a downstream task like classification or regression, you can try the XgBoost algorithm. It can be used as a classifier or as a regressor. This algorithm can handle missing values inherently. Source: this answer

If you don't have enough data

In such a case you would need to introduce bias in your model using your insights or domain knowledge about the problem. For, e.g., in the a possible problem of estimating weights using heights, you had an insight that your data comprises more of short people. So instead of naively using median values of the total dataset for weight, you can try to bin the data according to their heights, say 'S', 'M', 'L', 'XL', and estimate the weights of each bin separately using the median values of their respective bins. The thing to keep in mind is that when data is low, you need to provide knowledge to the model by enforcing some bias using your insights and domain knowledge about the problem.


Trying to understand why nonlinearity is important for neural networks by analogy

Your analogy is correct, except it is not really an "analogy". Sin is an activation function - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.
So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear neuron:
$$\sigma(x) = \sum_i V_i \sin\left(\sum_jW_{ij} x_j + b_i\right)+\beta$$
With all biases being zero $b_i=\beta=0$, output weights are $V_i = \left(A,B,C\right)$ and the inner weight matrix is diagonal:
$$W_{ij} = \begin{pmatrix}a&0&0\\0&b&0\\0&0&e\end{pmatrix}$$


How to estimate the cost and time to complete an AI Project

If you are a freelancer, when a client asks to create a website we can easily measure how much the total cost is needed based on the requirements of the client. (the backend, UI/UX design, features, etc.). We can even measure the estimated time of completion.

This is only the case when the full scope and design of the site is a well understood and relatively standard thing. With more experience, and perhaps a dedicated team, more complex sites and features can be addressed like this. It is just as common in my experience to view ongoing development of a site as a series of related projects, each of which can be estimated and costed as they become more feasible and closer to starting.

I have never seen a stand-alone site fully costed and estimated in my own work, because I work in-house on more complex features than this is possible for, but am aware this is a common approach, especially for agencies and freelancers. The difference is that the people involved are experts on producing well-defined work, and that in order to make a pre-estimated or even fixed price sale, they have put a lot of effort into risk management. Some of the risk management is contractual (e.g. "maximum of 3 major revisions" for web site design), some of it is in requirements gathering - often an initial pre-sales consultancy meeting is arranged, and/or a project proposal document generated where a lot of the possible misunderstandings between customer and developer are dealt with, and it is made clear what expectations are on both sides.

For a freelancer or agency, this de-risking is also spread over multiple projects as they will develop solutions that can be re-used including standardised contracts and templates for project planning etc.

There are also a few different project management approaches that deal with initial uncertainty in different ways. A "waterfall" approach of pre-planning as much in advance as possible requires very good knowledge of the work that will occur in later stages of a project, which of course depends heavily on earlier stages meeting all expectations of both the customer and developer. An "agile" approach will acknowledge that there are unknowns, and attempt to address them with some form of just in time planning. The problem with an agile approach is that "just in time" may be far too late if you need to budget up front (although it is still possible for instance by reducing scope to fit, provided that is agreed).

  What if the client asks you to make an AI project (image recognition, speech recognition, or NLP), how do you tell the client the estimated cost and time needed to complete the project in the beginning?

There is no difference in this respect just because it is an AI project. This would apply to any software project, including web development, where one or more of the following is true:

Scope of work is not clear
There are technical hurdles with unknown resolutions. I think it is likely you are focusing on this for your imagined AI project.
Customer expectations are not clear
Project success criteria is not measurable, or not known if achievable

As any software developer, if you are faced with project work with these features, you do some pre-project tasks focused on de-risking the project before attempting it. There are a few different approaches, than can be used in combination:

Gather more requirements, spend more time up front consulting on the project
Test assumptions before starting project. E.g. if you are not sure about quality of the data, take a look at it as early as possible. If you don't know if technology X can do Y, investigate it in advance.
Work on a timesheet basis, not fixed cost. Alternatively fix cost (or have a maximum), but leave scope of work open.
Have a flexible project plan, and re-estimate remaining work routinely.
Put any assumptions about customer support for the project, such as availablilty and quality of data, into project plan and in general form into your standard contract (e.g. something legal along the lines of "Project delivery depends on customer providing resources as outlined in the project plan document. If the resources are not provided in reasonable time, then the project may over-run or have additional costs, which will be covered by the customer, and not by the freelancer")

For additional up-front work on project feasibilty and scope, if you are concerned about risk to yourself as a freelancer (because customer expects these as a pre-sale effort), you could propose to bill for that as consultancy work, prior to starting project proper. You would likely need to produce something like a project proposal document in order that the paying customer received something for spending this money. Producing such a document might even show both you and the customer that the project is not feasible without additional work - in whch case you could note the steps needed to make it feasible (e.g. customer needs to source 10,000 more training images without the flaws noticed in existing images).


Is it recommended to remove stop words before named entity recognition?

It depends how you recognise the entities.

If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with.

However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if [work] at {organisation} is a rule you use to identify companies etc, then this wouldn't work if you take out the at.

You will also have problems if the stop words are part of an entity, eg the town Stoke-on-Trent. If hyphens are used to split tokens, then you won't be able to recognise it if the on is discarded.

In general I think stop words should mostly be kept; apart from information retrieval where they are not all that useful in an inverted index you will always lose something. If stop words were really pointless, then they would have been discarded from languages a long time ago. In the old days they were a useful way of reducing the demand on computational resources without losing too much, but nowadays I would say this is not really a problem anymore.


Why does the size reduce to $6 \times 6$ in the capsule networks?

After the first Conv layer, the size reduced to 20x20
for the primary caps which is convolutional caps layer
n final = (n + 2p -f )/s + 1
which gives a 6x6 output with 256 channels

6x6x256 is further encoded into capsules of 8 dimensions by reshaping the channels
i.e. 256/8 = 32
which gives 6x6x32 = 1152 capsules

try experimenting with the same hyperparameters first and then try to encode higher-level features by making suitable changes to the hyperparameters.


I propose you try this. It's about modern Natural Language Processing, Computational Linguistics και Speech Recognition, including Embeddings methods.


Which Openai Gym environment should I use to test a Temporal Difference RL Agent?

Cartpole will work just fine with a single-step TD method - e.g. Q learning - and a simple neural network. Configured well, a simple DQN model should solve it in a few minutes. You can also forgo the neural network and use discretised states, or tile coding etc.

What you cannot do is plug a neural network estimator into basic Q learning and make no other adjustments. That is because the bootstrap process in TD learning will create a runaway feedback driven from the random and incorrect return estimates in the initialised NN. You have to use something like DQN to get it to work at all. This is different to Monte Carlo approach, where you can generally just plug in a NN in place of a Q table.

The basics of DQN are:

Use experience replay. Store $(S, A, R, S')$ values from each step, and when training the NN, draw a small random batch (e.g. maybe 16 samples) from this table, re-estimate their Q values to calculate TD target e.g.  $R + \gamma\text{max}_{a'} Q(S',a')$ and use that to create a minibatch to train the NN
Use a "delayed" target network for estimating TD target. This can just be a snapshot of your learning network, taken once every N steps, where N is typically anything from 100 to 100000.

Further details of what configured well means varies from problem to problem. With DQN you will want to play with hyper parameters of experience history size, time to keep frozen copy of network (for bootstrap estimates), batch size of experience table samples, amount of exploration.

In CartPole though, there will be a wide range of acceptable hyperparameter values. So I suspect you have just plugged in a NN in place of the Q table and wonder why it doesn't work.

  I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.

Yes this is part of what makes some learning goals become harder challenges in sequential-decision control systems. However, this was also the case for your Monte Carlo method, and in general this relates to the credit assignment problem in RL.


How does one create a non-classifying CNN in order to gain information from images?

This can be thought of as a loss function design problem. If you optimize your network weights for something like multi-class classification, then expect your network to learn weights for this task (You will use cross entropy loss for this task). If you optimize your network to output a single value at the last layer and treat it as a regression problem for age prediction, then your network can learn weights for this particular task (You may use something like Mean Square Error loss here).

Let me give you a weak guidline on how to do this. Suppose say your input is images of chairs and you want to predict their age using a pretrained resnet, this is how you may do it in pytorch.

Definition
X: Input pictures
Y: List of ground truth values which is age here. i.e. : [1.4, 2.5, 2.2, ....]

Modify your neural network to give one output at the last layer
model = torchvision.models.resnet18(pretrained=True)
num_ftrs = model.fc.in_features
model.fc = torch.nn.Linear(num_ftrs, 1)

Design your loss function appropriately. You can treat age prediction as a standard regression problem. Of course, there are better loss designs here.
criterion =  torch.nn.MSELoss()

So use this loss during training
loss = criterion(output, target) where output is your neural network prediction and target is your ground truth values.

This is how you can modify an existing architecture for your task. Hope it helps.


How to make convnets aware what the image actually is, not what is depicted on it?

You seem to be wanting some description of the 'style' of an image.

To make that work in general, I'd guess that would actually require quite a lot of pre-processing to present 'texture elements' (rather than pixels) as the basic features.

This is quite speculative, but one approach might be to use Iterated Function Systems as a means of extracting these.

Whether 'spatial adjacency' (and hence CNN) is then the best approach to make higher-level decisions about these elements is (AFAIK) a matter for experiment.


Can someone explain R1 regularization function in simple terms?

Here is how I understand this regularization.
$R_1$ is simply the norm of the gradients, which indicates how fast the weights will be updated. Gradient regularization penalizes large changes in the output of some neural network layer.
$$
R_{1}\left(\psi\right) = \frac{\gamma}{2}E_{p_{D}\left(x\right)}\left[||\nabla{D_{\psi}\left(x\right)}||^{2}\right]\text{,}
$$
where $\psi$ is discriminator weights, $E_{p_{D}\left(x\right)}$ means that we sample data only form the real distribution (i.e. only real images) and $\gamma$ is a hyperparameter.
Since we don't know if $G$ can already generate data from real distribution, we apply this regularization to $D$ only on real data, because we don't want the discriminator to create a non-zero gradient without suffering a loss if we are already in a Nash Equilibrium. I guess this also prevents $G$ from updating if it generates data from the real distribution.
The authors also investigate which value is best for $\gamma$ by analyzing the eigenvalues of the Jacobian of the the associated gradient vector field, but in my opinion, this value is highly dependent on dataset and architecture.
"Gradient orthogonal to the data manifold" simply means zero gradients. From a GAN perspective, the data manifold is a lower-dimensional latent features manifold embedded in a higher-dimensional space and our goal is to approximate it. Since the gradient vector shows the direction in which we need to update our function, if it is orthogonal to this manifold, we do not need to update the function.


What is the time complexity of the upsampling stage of the U-net?

After further investigating the problem I have found the answer:
U-net generators' up-sampling stage consists of two steps:

Use UpSampling2D layer
Apply convolution on the output

The UpSampling2D layer is in the keras documentation described as:
Repeats the rows and columns of the data by size[0] and size[1] respectively.
From this information, we can calculate the time cost for UpSampling2D alone. Lets set size to (2,2), as is set in basic configuration of the U-net generator. The output of the UpSampling2D is then doubled. In case we started with (4,4,3), where the last index corresponds to number of channels, the output shape will be 8,8,3. We can see that each row and column need to be copied twice in each channel. From this we can define time complexity of a single up-sampling as:
$$
O\left(2 \cdot c \cdot n \cdot s\right)
$$
Where c corresponds to number of channels, n corresponds to input length (one side of a matrix) and s is equal to filter size. Assuming that length and filter size have square shape, the complexity is multiplied by 2. Since in this case the the filter size is known, equal to (2,2), the notation can be simplified to:
$$
O\left(4 \cdot c \cdot n \right) = O\left(c \cdot n \right)
$$
In my case, with only 1 channel, the complexity is simply
$$
O\left(n \right)
$$
Which means the up-sampling stage is linear, and the only important feature is input size, which is negligible to the complexity of the following convolutional layer and can be ignored.


How should I interpret this validation plot?

This is a sign of overfitting.

As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not exist. When you try out these patterns on new data (which is what cross-validation is imitating), then the patterns do not work, and your model fails to generalize.

The main piece of information to draw from this plot is that the optimum tree depth is about 15.


What is the space complexity for training a neural network using back-propagation?

I will not tell you what the exact space complexity of training an FFNN with GD and BP is (because that actually depends on the specific implementation of GD and BP and I don't want to dive into the details of some specific implementation now, maybe later!), but I will guide you towards the specific answer, which you should be able to figure out alone (although it may take some time because you need to understand all the details of the BP algorithm), if you understand this answer.
The space complexity of an algorithm is just the amount of memory that you need to use during the execution of the algorithm. The space complexity, like the time complexity, is typically expressed as a function of the size of the input (or the number of inputs that you have) and, usually, in big-O notation, i.e. in the limiting case. So, $n$ is not the same thing as $\mathcal{O}(n)$, $\Omega(n)$ or $\Theta(n)$. Moreover, you can also express the space/time complexity both in the worst, best, or average case, and this is orthogonal to upper (expressed with $\mathcal{O}$), lower ($\Omega$), or tight ($\Theta$) bounds (check this).
If you use gradient descent (GD) and back-propagation (BP) to train an FFNN, at each training iteration (i.e. a GD update), you need to store all the matrices that represent the parameters (or weights) of the FFNN, as well as the gradients and the learning rate (or other hyper-parameters). Let's denote the vector that contains all parameters of the FFNN as $\theta \in \mathbb{R}^m$, so it has $m$ components. The gradient vector has the same dimensionality as $\theta$, so we need at least to store $2m + 1$ parameters.
Depending on how you implement BP, you may need more memory. For example, if you need to store all the intermediate terms of the partial derivatives, that will require more memory. To compute exactly the amount of required memory, you will have to expand the gradient vector into all their components (which may not be a pleasant experience). As I just said, this only contributes to the space complexity if you need to store these intermediate components, so, ultimately, the space complexity of an algorithm depends on the specific implementation of the algorithm.
Moreover, to be precise, we cannot just say that the space complexity is $2m + 1$ or whatever the amount of memory that you require is (although many careless or ignorant people will just say that), because we are not expressing this complexity as a function of the size of the input in the limiting case (which is usually done when expressing the space complexity of an algorithm), the number of layers or the number of units per layer (and you probably want to express the space complexity as a function of these 3 possible variable hyper-parameters).
If you take a look at this answer, where I describe how to compute the time complexity of the forward pass of an MLP (or FFNN) as a function of the number of inputs and outputs, the number of layers, and units per layers, then you can express the space complexity for training an FFNN in the same way. Given that you are already familiar with how space and time complexities of an algorithm are calculated (and given that this answer is already quite long), I will not repeat the description here.
In any case, to answer one of your questions more directly, yes, the space complexity will depend on the number of inputs that you have, because the number of inputs will determine the number of weights in the first layer, which you need to store in memory. This is true in the case of FFNNs (or MLPs) but note that this would not be true in the case of CNNs (i.e. the number of parameters in the convolutional layers does not depend on the size of the input), and that's why CNNs are often said to be more memory efficient.


To address your description point by point,

There are many types of CNN architectures. You appear to be describing a fully-convolutional neural network built only from convolutional layers which is a very specialized type of network (typically used for segmentation tasks).

Most networks will include other layer types which have trainable parameters too, i.e. batch normalization layers or densely connected layers.
Yes, the training process operates on the same concept of error backpropagation.

If the model is performing a segmentation task (or any task which requires a 1-to-1 mapping of individually classified pixels), then yes its typically the same size.
The raw output of a fully-CNN is typically a probability map where each pixel is the probability between 0 and 1 of it being a member of the positive class. This is normally thresholded at 0.5 to obtain a binary mask which delimits the area of interest.
In essence, yes. But it may be more accurate to say that the goal is to minimise the loss function (reduce the error) which is done by learning optimal filter weights.
Overfitting is not about pixel location. In fact, CNNs are translationally invariant which means that the location of the feature has no impact on the output. There are some types of CNNs which take feature location into account but thats still not standard. Overfitting generally happens when a filter is an exact fit to a pixel patch, for example, the filter is no longer just looking for vertical lines but the exact pixel values of a section of vertical line in a specific image.
No, each convolution of a 3x3 filter takes 3x3 pixels and it is convolved over the whole image which means that the filter output will likely be around the same size of the image depending on stride length, padding and dilation. Have a look at one of my other answers for more details.


If the goal of training of a GAN is to have $P_g=P_{data}$, shouldn't this produce the exact same images?

Answers:

No, generally GANs model the joint distribution of pixel space. The space is so entangled that it would be very difficult to model a singular pixels PDF.
This question doesn't make sense given the first question's answer. You do NOT sample singular pixels, you sample the joint.
$P_g$ is trying to model $P_{data}$, so that's your goal, so yes it will find features in the training set. Ideally, though it can extrapolate usage of the features to generate new images. But, yes depending on your setup you can end up with overfitting or mode-collapse where you really only generate a small subset of images that trick the discriminator. So I would say, no, the GAN doesn't know what its doing (since its just a minimax optimization of a loss you setup), so your goal is to tell it what to do in a way to achieve your best result.


How do I optimize the number of filters in a convolution layer?

Am I right in thinking that because there are only newImageX * newImageY patterns in the 32 x 32 image, that the maximum amount of filters should be newImageX * newImageY, and any more would be redundant?

Your assumption is wrong. If you have a $32 \times 32$ images (so consider only grayscale images), then you have $256^{32 \times 32}$ possible patterns (i.e. a huge number of possible patterns!), given that, for each pixel, you can have $256$ values, and there are $32\times 32$ pixels. Of course, for a human, many of these patterns may be very similar to each other (or even indistinguishable), but the way you measure similarity, in this case, is another hard problem. Hence also the difficulty in choosing the number of filters.


If a neural network is a universal function approximator, can it have any prior beliefs?

I think your deduction is mostly correct.

Neural networks of depth are universal function approximators. This means that in principal, for any function of the form you describe, there's a NN that approximates it.

However, a particular NN architecture of fixed width and depth, with fixed connections is not a universal approximator for all functions. Only an infinitely wide NN is, and that's a theoretical construct, not something you can make in practice.

Typically, a practitioner using a NN injects their prior beliefs about the problem  by selecting an architecture. For example, it is common to use the ImageNet or ResNet architectures for image processing tasks. Those architectures are less effective on other types of tasks. For instance, it should be clear that they are ineffective on say, a time-series analysis task.


In Batch Normalisation, are $\hat{\mu}$, $\hat{\sigma}$ the mean and stdev of the original mini-batch or of the input into the current layer?

Your intuition is correct. We will be normalizing the inputs of the layer under consideration (just right before applying the activation function).

So, if this layer receives an input $\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$, the formula for normalizing the $k^{th}$ dimension of $\mathrm{x}$ would look as follows:
$$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$

Note that in practice a constant $\epsilon$ is also added under the square root in the denominator to ensure stability.

Source: The original Batch Normalization paper (Section 3).

Andrew Ng's video on this topic might also be useful for illustration.


Is it standard to say that an evaluation function estimates the “goodness” of a position?

Yes, "goodness" is a common description of the value generated by an evaluation function.
For example,
"Artificial Intelligence" p. 77;
"Knowledge-Free and Learning-Based Methods in Intelligent Game Playing" p. 15;
"Tenth Scandinavian Conference on Artificial Intelligence" p. 125; and
"Algorithms and Networking for Computer Games" p. 80.

The "position" is to the complete state of the game at some instant. A single position includes the locations of every piece on a game board at that instant, and also includes a few other bits of information -- who's turn it is; whether castling is still a legal option; etc.

I suspect that "the position" is a synecdoche, referring to the total state by a small part, the physical location (physical position) of some piece on the game board, based on earlier usage in military tactics of "positioning" troops relative to natural cover or relative to enemy troops.


Can an AI make a constructed (natural) language?

could an AI make construct it's own natural language, with words,
  conjugations and grammar rules?

Sure. This might be helpful: Simulated Evolution of Language: a Review of the Field

  Basically, a language that humans could use to speak to each other.
  (Preferably to communicate abstract, high-level concepts.)

I'm not sure how useful such a machine adapted language would be to human speech. I suspect not very. Perhaps it could be useful as a kind of "common byte-code format" to translate between multiple human languages... But English kind of already serves in that role. Doing so would probably be an academic exercise.

  Could it be based on existing natural languages or would it have few
  connections to existing natural languages?

You could probably generate languages in either direction. Languages not linked to human-natural languages will probably take shapes that reflect the problem spaces they work with. For instance, if this is an ant simulation, the generated words will probably reflect states related to food, energy, other ants, etc.

  Could it design a language that's easier to learn than existing
  languages (even Esperanto)?

Easier for a machine? Definitely. Easier for a human? Probably not. Our brains are somewhat adapted to the languages we use. And the languages we use are somewhat adapted to our brains.

What is your goal? To create a language that is easier to use for humans than existing human languages?

If your intention is to build a "universal language" that could be "the most efficient" for machines, humans and aliens - no such thing can exist. The space of all possible machines is infinite and therefor limits our ability to define communicative abstractions that have utility across all contexts.

If we make lots of assumptions about machines, like they have intentions, they exist in 3 dimensions, they differentiate between temporally linked events, they have eye balls, a need to consume foods and liquids, a need to carry the food from place to place, etc... Then yes, common communicative abstractions may have utility across the set of those kinds of machines. But then we're no longer dealing with the general case, but one much more specific.

These links also seem interesting and somewhat related:

Grammar Induction and Genetic Algorithms: An Overview. [pdf]
A Model of Children's Acquisition of Grammatical Word Categories Using an Adaptation and Selection Algorithm [pdf]
The processing of verbs and nouns in neural networks: insights from synthetic brain imaging


Is there has any method to train Tensorflow AI/ML that I focus on detecting background of image more than common objects?

Trying to address all the questions asked in the end in the same order

Most definitely possible.
I would say its best you approach this with segmentation to start with.
Just use a free GPU runtime notebook service such as Google Colab or Kaggle Kernels. But you would not directly be able to integrate with the device, you'd have to keep moving input and output from your drive (on Colab). There might be a better service for the needs described, but this is the best I know on this.
Your background can be segmented and the segment can work on transforms such as maybe convolutions or affine transforms to be able to get the relevant information regarding the background.

Hope this was helpful!


Small size datasets for object detection, segmentation and localization

There are various dataset available such as

Pascal VOC dataset: You can perform all your task with these.

size of the dataset is as  follows

ADE20K Semantic Segmentation Dataset: you can perform only segmentation here

COCO dataset: This is rich dataset but a size larger then 5 GB so you can try downloading using google colab in your drive and then make a zip file of data as less than 5 GB

You can download all these datasets using Gluoncv easily.here link.


How can I apply naive Bayes classifier for three classes (Positive, Negative and Neutral) in text data?

Use TextBlob to get the text sentiment polarity and subjectivity. polarity is negative or positive sentiment and subjectivity is how factual the sentiment is.
my_valance=TextBlob(sentence)
df.loc[key,'sentiment_polarity']=my_valance.sentiment.polarity
df.loc[key,'sentiment_subjectivity']=my_valance.sentiment.subjectivity


Can reinforcement learning algorithms be applied to computer vision problems?

Yes, there are a couple of ways to apply reinforcement learning in computer vision problems. This mainly employs the principle of "applying the algorithm -> evaluating the outcome -> adopting the best outcome".

The following are a couple of examples that use reinforcement learning in computer vision.

CAD2RL: Real Single-Image Flight Without a Single
Real Image.
End-to-End Training of Deep Visuomotor
Policies


Has there ever been a model (neuro-physical) proposed for the human unconscious brain?

The Unconscious in psychology is related to many aspects of behavior, but many levels of this exist, including low levels (simple tasks) like walking, eating, breathing etc.  This has been investigated in animal world, for example muscular memory, the insect brain model, and Nonlinear System Control Using Neural Networks . Many of these are related to body control.

See also: nested hierarchies of circular emotional control and cognitive regulation for “bottom-up” influences and “top-down” regulations


Does neuroevolution require a labelled dataset?

The GA will require a fitness function, which means you need labeled data for comparison.

That conclusion is wrong. Yes, sometimes your fitness function will use labeled data. For example, if you want to train an XOR gate or any other known function.

However, there is arguably no advantage of training a function with neuroevolution versus backpropagation, except for the fact that you might discover some new architectures which solve the solution very well.

You don't always need a labelled dataset for neuroevolution.

Take for example IAMDinosaur, which trains neural networks through a genetic algorithm - however, the optimal solution is not known. There is no labelling of input data, all it does is calculate the fitness from the score.


Could artificial intelligence cause problems for humanity after figuring out human behavior?

Will Artificial Intelligence some day become a problem to humanity
  after learning human behaviors and characteristics?

It can be answered in both ways, I think.

Yes, they may become a problem.

With the increasing integration of loads of apps and smart devices in our life, almost everything defining an individual human being is digitalised. For instance, our fingerprints, voice, facial image etc. Apart from these data, we use those apps and devices to track our health (heart rate, calorie intake etc), to plan our schedules, and most importantly to communicate. If some sort of AI engine is integrated into a chat application, for instance, it can learn our typing patterns, conversation style, and hundreds of other unforeseen parameters. Imagine what can be learned about a person if such AI is coded inside every device and every app in your day-to-day use.

We use smart devices and apps to harness their functionalities and features which ease our way of life, and we give them, unintentionally, our identity and sometimes, even our personality. For them, these are the parameters that can be input to some machine-learning algorithm and predict what we will do the next day, or what will happen to us the next day.

This sounds like a major problem, especially when these technologies are indispensable.

No, they may not become a problem.

Humans are really complex creatures and possess the most advanced intelligence technology called the brain. I think the brain can be thought of as a technology in this context. There are several tissues inside the brain that can learn to do certain things themselves. It was proven in a research that some tissues have the capability to perform the functions of other tissues using neuro-rewiring techniques. Imagine that the same tissue that has helped you see until now can be made to help you hear instead. Now imagine mimicking such a technology.

While it is not impossible, for an AI to achieve the brilliance of a human brain is a topic of ongoing research. To train a machine for the purpose, we would have to feed it with gazillion behaviors and characteristics, which it may not be able to handle! After learning some behavior and characteristics, the AI would be said to be smart, but it would still predate us.

So, they may not become a problem at all because of our brain.

The algorithms (like recommender systems) used by Amazon and Facebook influence us or even manipulate us. But, that manipulation is either very obvious (like viewing promoted products) or is in company's best interests (like viewing a certain news piece). It may be even possible that several external parameters are used by these systems to improve your experience. For instance, Google ads show us what we were looking for on an online store when we visit any random website. In most cases, what you see is a result of what you were looking for before. If any attempt to influence does happen, we may learn to avoid it through careful observation or even experience.


If I trained a model to perform semantic segmentation on images with only one object, would it also work on images with multiple objects?

Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model.

U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one single instance of an object and shade that region.

Now why does this depend on the model? Well let's say you are using something slightly different, where region proposals are generated by a determinisitc algorithm we've predefined, then these regions are run through a CNN to segment them. In this case, as each region is without context of the entire image, the difference between 2 objects in an image and 1 is indistinguishable to the network (as regions may overlap), and as such, only using images with 1 object will not pose any problems (there is a name for early models like these, though it escapes me).

So assuming I am correct, what should you do? The models that use a deterministic algorithm for region proposals are slow and old, so I wouldn't suggest that. Instead, I would think that you should first do some testing, to see if it actually does cause issues. Assuming it does, a good option could be to tamper with the training data and separate segments by a few pixels to sort of "force" multiple objects into existence.

Regardless of such, I would still suggest using U-net. Fixing this issue (if it does arrise) should be relatively easy to do, so there's little to lose by using U-net and just trying the training.


Which library / algorithm should I use to recognise a specific facial expression (C#)

Since you are

A C# developer already
Just getting started and not sure where to go next

I would suggest trying the Emotion API which is now part of the general Face API. This has the benefit of being pre-trained on a very large dataset. You can perform 30,000 recognitions/month for free.


Oscillating around the saddle point in gradient descent?

It's important to note that in a pragmatic instance of ML on a "real dataset", you likely wouldn't have a "strict" saddle point with precisely zero gradient. Your error surface won't be "smooth", so even though you would have something that resembles a saddle point, what you would obtain would in fact be a small region with near-zero gradient.

So let's say you are in a region with near-zero gradient. Assuming that the gradient is this area is normalized at 0 with small Gaussian distributed noise (thus gradient =  small Gaussian noise). You can then see that the algorithm can't quite escape the region (or at least, will spend a lot of time here) since 1. Gaussian random walks will more-or-less stay in place (unless for a long time) 2. small gradients means there is no obvious direction to leave the region.

In any case, SGD more or less solves this issue, and its usage is standard practice for reasons beyond this problem.


Where to publish a first article in Deep Reinforcement Learning?

I recommend you focus on quality over quantity. Publishing a paper will boost your reputation and make you more recognised within your academic field (AI); however, this is only if the paper provides useful insights into an important issue.
Your paper is more likely to be accepted if it is well written and easy to understand, stimulates new important questions, uses rigorous methods to explain why the data supports the conclusion and connections to prior work is made and serve to make your paper's arguments clear. (Elizabeth Z Elsevier blog)
Before submitting your paper, ask a mentor or a colleague to proofread it, so that you can make the relevant revisions and changes. Journal editors will look down on your work if it is poorly written or contains substandard grammar.
A way to get published is by writing reviews, especially for researchers in earlier stages of their careers. Most journal editors like to publish replies to previous publications since it stimulates debate.
Remember it is acceptable to challenge reviewers' suggestions with good justification. Many researchers fail to persevere when they are instructed to revise and resubmit their work. Don't give up, however, you can politely decline or even argue why a reviewer is wrong. Editors will accept a rational explanation if it is clear that you have considered all their feedback.
Getting published is never easy, especially in high ranking journals. If you focus on getting published quickly it could derail you from concentrating on the quality of your research. Yes, getting published can be expensive, however, it's much better for your career if you write a high-quality paper than a low-quality paper in a lowly ranked or ungraded journal since it will not be REFable.
Below is a list of Artificial Intelligence Journals that you can submit your papers to and possibly get published.

Artificial Intelligence (Journal) https://www.journals.elsevier.com/artificial-intelligence

Artificial Intelligence Review  https://link.springer.com/journal/10462

Applied Artificial Intelligence http://www.tandfonline.com/action/journalInformation?journalCode=uaai20&

Autonomous Agents and Multi-Agent Systems  https://www.springer.com/computer/ai/journal/10458

International Journal on Artificial Intelligence Tools http://www.worldscientific.com/worldscinet/ijait

International Journal of Pattern Recognition and Artificial Intelligence http://www.worldscientific.com/worldscinet/ijprai

Journal of Experimental and Theoretical Artificial Intelligence  http://www.tandfonline.com/action/journalInformation?journalCode=teta20

Journal of Machine Learning Research  http://www.jmlr.org/

IEEE Intelligent Systems  http://publications.computer.org/intelligent-systems/

Journal of Automated Reasoning  https://www.springer.com/computer/theoretical+computer+science/journal/10817

Minds and Machines  https://www.springer.com/computer/ai/journal/11023

Autonomous Agents and Multi-Agent Systems  https://www.springer.com/computer/ai/journal/10458

Applied Intelligence  https://link.springer.com/journal/10489

Journal of Intelligent Manufacturing  https://link.springer.com/journal/10845

Journal of Intelligent Information Systems   https://link.springer.com/journal/10844

AI & Society  https://link.springer.com/journal/146


How to use the LSTM layer in PPO architecture?

Not sure if you're using TensorFlow-Agents for this, but if you are, there's some nice functionality for this built out using the ActorDistributionRnnNetwork and ValueDistributionRnnNetwork that I've found to be very helpful.  The LSTM architecture is used in both the actor and critic networks.  See this page for more information - hope this helps!


is it possible to train several Neural Networks on different types of data and combine them?

The term you need  is “model ensembles”, that’s the way models are combined.  Pretty hard to be more specific since you don’t give a language or any other details.


There are projects out there attempting to apply Machine Learning / AI to cyber-security in different ways.  One that I'm familiar with is Apache Metron.  Another related project is Apache Spot.  I think if you read over the docs for these two projects respectively, they will probably give you some good insights on this subject.


How can we prevent AGI from doing drugs?

You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.
One thing I am sure though is that ML is known for "cheating" (see for exemple). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not "cheat". Thus, the answer is mostly contained in your assumptions.
Whether we are able to build an AGI, what "cheating" we would have to overcome to do so and how we will be able to do so is mostly undetermined.


Object Detection Algorithm that detects four corners of arbitrary quadrilateral, not just perpendicular rectangular

You can use OpenCV's cv2.minAreaRect() to detect oriented/rotated rectangular bounding boxes. Below's an example result from OpenCV-Python-tutorials:

Alternatively, you could train a supervised object detection model to output 8 co-ordinate values (x0,y0,x1,y1,x2,y2,x3,y3) of the quadrilateral by training with a labeled oriented-bounding-box dataset. You could also create the bounding box labels yourself for the same by using tools such as VGG Annotator Tool among others.


Why does k-means have more bias than spectral clustering and GMM?

I'm not an expert on clustering, but here's my take below. Note that this is only based on theoretical arguments, I haven't had enough clustering experience to say if this is generally true in practice.
K-means vs GMM
K-means has a higher bias than GMM because it is a special case of GMM. K-means specifically assumes the clustering is spherical (meaning each dimension is weighted equally important) and that the clustering problem is a hard clustering problem (each data point can only belong to one label). So, theoretically, K-means should perform equal to GMM (under very specific conditions) or worse. More info
K-means vs GMM (identity covariance matrix)
K-means has a higher bias than GMM (identity covariance matrix) because it is also a special case. K-mean specifically assumes the hard clustering problem, but GMM does not. Because of this, GMM has stronger estimates for the mean of the centroids. More specifically,

[GMM] estimates the cluster means as weighted means, not assigning observations in a crisp manner to one of the clusters. In this way it avoids the problem explained above and it will be consistent as ML estimator (in general this is problematic because of issues of degeneration of the covariance matrix, however not if you assume them spherical and equal).

In practice, if you generate observations from a number of Gaussians with same spherical covariance matrix and different means, K-means will therefore overestimate the distances between the means, whereas the ML-estimator for the mixture model will not.

So, theoretically, K-means should perform equal to GMM (identity covariance matrix) or worse. More info
K-means vs Spectral clustering
K-means has a higher bias then spectral clustering because spectral clustering effectively uses K-means after processing more information from the matrices.

Spectral clustering usually is spectral embedding, followed by k-means in the spectral domain.

So yes, it also uses k-means. But not on the original coordinates, but on an embedding that roughly captures connectivity. Instead of minimizing squared errors in the input domain, it minimizes squared errors on the ability to reconstruct neighbors. That is often better.

More info


How do I avoid an agent to tend to terminate in a negative state when time needs to be taken into account?

This is a common problem in reward shaping. You want a certain behavior from you agent but its challenging to describe it completely in terms of rewards. This situation you are describing is challenging specifically because as the grid world grows, the chance of randomly stumbling onto the goal state becomes less likely AKA the problem of exploration.
There are a few techniques that can be used to address this problem though, here are some.

0) This is an emergent property of the environment and gamma (0 based because its more immediate to your problem:p)

If gamma is small, your agent will value rewards that are in its immediate future more highly whereas as gamma approaches 1, the agent values rewards that are further in its future. In your grid world, the the size of the grid affects how this gamma affects your agent. Like in your example, if your grid was 100x100, if the trap was close to the agent, you would have to have a gamma closer to 0 in order for your agent to avoid the trap because its worse than moving to a cell that isn't a trap. This is interesting because the whole purpose of gamma is to increase the weighted value of temporally distant rewards but when you make the trap more favorable than the goal, going to the trap is the optimal strategy. :)

1) Include more observation data to your model

This isn't always a possibility but depending on your application and what you have available, you may be able to give your agent missing information that might be necessary to its ability to solve your task. For instance, in your infinite grid example, you may include the distance between the agent and the goal or the direction of the goal.

2) Include a reward that helps to shape the direction of progress.

One could easily create an infinite grid world where there isn't actually a goal but rather a continuing task where the agent has to cover distance in a desired direction while avoiding obstacles. How would you approach this problem? Perhaps a reward that specifically looks at the number of cells visited by the agent in the desired direction over time aka the discrete analog to velocity. Of course this is still dependent on a know direction but thinking about how to one might handle the "limit" of your environment as it grows (e.g., adding more and more cells to the grid world) helps to give an intuition of what your agent could be missing.

3) Use curiosity based approaches
Following from 2), if the direction isn't known, one thing to consider is rather than giving a penalty for each timestep, instead incentivize the agent to be faster, rewarding visiting "infrequently visited" states. As the task requires that the agent performs as quickly as possible, remaining or returning to a previously visited state clearly doesn't benefit the agent. Taking this notion of rewarding "visiting infrequently/unyet visited/ states" further results in the recent research topic of using curiosity to have RL agents that have novel exploration strategies.

Although there are many (often debated) ways to define curiosity, they all share an idea of giving the agent a bonus when it has entered a state that has not been visited before. A paper that gives a good recap of curiosity methods and also introduces a novel approach is Random Network Distillation from OpenAI.


Sorry but no one can give you a number.

As stated already in the comment by Manngo, the general rule: is as much as you can. I have also seen the 10x number being thrown around, see here. According to this "rule", you need  roughly 10 times as many examples as there are degrees of freedom in your model.

How much data you will need is something you can only answers to through empirical investigation. The amount of data you need to collect is influenced by the complexity of the data, the dimensionality of the data and the algorithm you intend to train(although this can be controlled somewhat through regulirazation, see this good blog post by Jake Vanderplas for more information).

This question  might be of interest to you.


DQN layers when state space and action space are multi dimensional

This is not necessarily the only way to do this but it would be the approach I'd take.
Assuming your agents position is a vector in $\mathbb{R}^d$, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the output so it is now also a vector in $\mathbb{R}^{d'}$. I would then concatenate these together so you have a vector in $\mathbb{R}^{d + d'}$ and pass this through some fully connected layers as usual.
As for how many layers and which activation to use this is something you'll have to do by trial and error as this really is problem specific.
Your output would typically be a score for each of the action combinations, though as one of your outputs is a matrix this could potentially make things expensive to compute as you would need $2^{n \times m}$ binary representations just for the matrix ($n$ is the number of rows and $m$ is the number of columns).


How is trajectory sampling different than normal (importance) sampling in reinforcement learning?

Here is my understanding:
In trajectory sampling as the book describes it, we use the current policy on the simulator to get (next-state, action) pairs. The advantage is that if some states occur more frequently than others in that environment, and if we take enough samples, the distribution among the samples would be similar to the actual distribution.
On the another hand, you can sample in a different manner if you have access to the model. Suppose you have access to the transition distribution. Then you can sample your start state uniformly, and use the transition distribution to get the (next-state, action) pairs. This can be useful if you want to force your algorithm to look at all states evenly, even if that's not the actual case.


How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)?

Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.

Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the tutorial. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.

for d in ['/gpu:2', '/gpu:3']:
    with tf.device(d):
        a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
        b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
        c.append(tf.matmul(a, b))

Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be distributed across processes and machines. Asynchronous algorithms (like A3C) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).

With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.


To my knowledge, there does not exist anything along the lines of model-based reinforcement learning with time-sensitive data. I think the best chance you have is to try to abstract the data that you have into a model which is not time-sensitive.
What would happen when you get past the timestamps of your original data? I am guessing that when testing this model, you will have states with time stamps past your original latest timestamp. Or are you only using your model in the time stamps of your training data? Is it going to be a useful model then?
Firstly, please note that the topics of this forum are mostly theoretical. This question leans on the boundary. If you frame your question a bit differently, ending your post with a clear question and also putting that question in the title, you will get more responses.
I feel like the question is 'What are the possibilities for model-based reinforcement learning for time-sensitive data?' But this question is inherently vague as it implies that you have a model according to the definitions of models in RL, which are not time sensitive. If you would like a better answer then please provide more information or ask your question on a more practical forum if it is more practical than what I guessed.
Disclaimer: would have just put some comments under the post as this is not a real sufficient answer, but my reputation is not high enough for that :P


How to scale all positive continuous reward?

I’ll try to find where I found it, but normalizing the rewards has always worked for me. Assuming you have a list of the discounted returns for each action, you subtract the whole list by its average value then divide it by its standard deviation. In Python with NumPy, that would look like:
returns -= np.mean(returns)
returns /= np.std(returns)

This puts the returns in a small and consistent range that keeps learning similar with different rewards.


What is the difference between a feed-forward neural network and a liquid state machine?

Liquid State Machines are used in the field of Neuroscience. What you have used is a variant of LSM called Echo State Network (ESN) used in the field of Machine Learning.

ESN's are pretty simple and superfast compared to normal ML paradigms like Feed Forward NN's or RNN's. ESN's are based on a relatively new paradigm called Reservoir Computing. ESN's are mainly used in Sequential ML problems, and it is used to overcome the problems RNN face like vanisihing gradient, exploding gradient, long training times, etc. The basic idea is that you have a reservoir of Neurons (basically a Neural Net with fixed and non-trainable weights, but it can be varied to complex structures) which will echo an input into some form based on which a simple Linear Regression model (or maybe a complex NN) will be trained.

The basic idea of an ESN is very simple, note it works on time series data and the conventions are similar to a RNN:

Just like RNN you have a hidden state $x(n)$ and an input $u(n)$ and some randomly initialized non-trainable weights are assigned to them (the weights with which you dot product these 2 terms).
$$a(n) = W_{in}[1:u(n)] + W[x(n-1)]$$
The next $x(n)$ is produced as some linear function of non-linear transformation of $x(n-1)$ and $u(n)$ multiplied by their respective weights and $x(n-1)$.
$$\tilde x(n) = tanh(a(n))$$
$$ x(n) = (1-\alpha).x(n-1) + \alpha\tilde x(n)$$
The prediction target $y(n)$ is done simply by multiplying $x(n)$ and $u(n)$ with $W_{out}$ which are trainable and the only trainable weights.
$$y(n) = W_{out}[1:u(n):x(n)]$$
Now you apply a suitable Loss Function on your predicted $y(n)$ vs actual target $t(n)$ to train weights $W_{out}$

($:$ means concatenation and follows the usual RNN concatenation rules)

This is a very simple overview of ESN's and it works surprisingly well for Sequential prediction tasks. The main idea and intuition behind the working is that the inputs are echoed in the reservoir of untrainable weights which basically means that the input is converted into a certain 'form' due to the random initialisation of weights and this 'form' trains the $W_{out}$ (the first few training $u(n)$ are run through the network without training for the network to assume a starting state where the input has already been echoed in the network and is now coming finally coming out). There are a few mathematical details for the ESN to train and not blow up by huge oscillations and can be found here in this simple intuitive tutorial.

In terms of difference of purpose, a few are:

Provides superfast training times.
Works on Sequential data mainly.
It is being used as a side network to initialise RNN weights which apparently improves the performance.

Although these networks have some very hard to tune hyperparameters, owing to its simple and superfast training one can easily check a lot of hyperparameters. The network also performs surprisingly well and it has resulted in Reservoir Computing being actually researched very actively. Check this very old question too.


Training an AI to recognize my voice (or any voice)

The human voice is based on the neural muscular control of vocal apparatus made up of many parts.

Diaphragm
Vocal cords
Throat (constrictors and anti-constrictors)
Nasal cavity
Cheek
Jaw
Tongue

These coordinated muscular manipulations produce envelopes (controlling) of audio that can be characterized by periodic and transient wave forms.

Volume
Pitch
Tone (relative volume of harmonics)
Consonant transients

Voices are unique to the learning state of neural activity and anatomic attributes, which is a way of saying that vocal habits and the physical attributes of the voice supports the distinguishing of vocal identity.

Strength of vocal muscles
Connectivity of muscles to bone, tendons, and cartilage
Shape of inner surface of vocal pathways
Neural coordination of those muscles
Neural production of phonetic control to produce linguistic elements
Neural serialization of semantic structures (ideas)

The detection of distinguishing features of voices by the ear is equally complex. In a room full of people talking, the brain can learn to track a single voice.

It is important to note that performing voice recognition to determine the identity of the human source is significantly different than performing voice recognition to produce text. To produce text accurately, the NLP must determine language elements and construct a semantic network that represents the vocal content or a text from that representation to be accurate in the case of like sounding words. Fortunately, the identification of the speaker is easier in some ways than the accurate voice to text. Unfortunately, the identification of the speaker has general limitations discussed below.

The first stage of hearing in the ear is mechanical, involving the length of hairs along the cochlear surface, which is like a radio tuner that discriminates all frequencies within a range simultaneously. The software equivalent is a spectrum derived by applying a root mean square to the result of an FFT (fast Fourier transform) to provide magnitudes.

$$ m_f := \sqrt{t_f^2 + {(it_f)}^2} $$

The phase component of the FFT results ($\, \arctan(t, it) \,)$ can be discarded, since it is not correlated with neural control of voice.

The application of the FFT to speech (as with any changing audio) requires windowing over the audio samples using one of the windowing tapers, such as the Hann window or Blackman window. The input is the audio stream or file contents as a sequence of pressure samples, the audio. The output is a sequence of spectra, each containing the volume of each frequency in the vocal range, from about 30 Hertz to 15 K Hertz.

This series of spectra can be fed into the initial layer of one of the more advanced RNNs (recurrent neural networks), such as the LSTM (long short term memory) networks, its bidirectional version, the B-LSTM, or a GRU (gated recurrent network), which is touted as training equally well with less time or computing resource consumption.

The identity of the speaker is the label. The series of spectra are the features.

Using the PAC (probably approximately correct) learning framework, it may be possible to estimate, in advance of experimentation, the minimum number of words the speaker must speak to produce a particular accuracy and reliability in use of the learned parameters from the network training.

It will take some study to set up the hyper-parameters and design the layers of the network in terms of depth (number of layers) and width sequence (number of cells per layer, which may vary from layer to layer).

The use case limitation of this system is that each speaker must read some text that provides adequate training example sequences of adequate length, so that there are sufficient number of overlapping windows for the FFT to transform into spectra so that the training converges reasonably.

There is no way around the individual user training as there is with recognition of linguistic content, which can be trained across a large set of speakers to recognize content somewhat independent of the speaker. The system can be adjusted and improved to minimize the amount of speech required, but information theory constraints keep that quantity from ever approaching zero.

No network, whether artificial or biological, can learn something from nothing. Claude Shannon and John von Neumann realized decades ago that there is a kind of conservation of information, just as there is a conservation of matter and energy in space below nuclear reaction thresholds. This led to the definition of a bit and the formulation of information as a quantity of bits corresponding to a narrowing of probability that the information provides.

$$ b_i = - \log_2 {\frac {P(x|i)} {P(x)}} $$


On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine):

  Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, and is set to zero for time-steps less than $1$.

I suspect the solution they write there would indeed work better than just repeating the starting position up to 8 times. Intuitively, you'll want the Neural Network to learn to primarily focus on the current game state. If the starting position is repeated a bunch of times in those planes, the Neural Network cannot distinguish between any of them in the learning process in the first few steps, and may start relying on them all equally. Only in later time steps will it "figure out" that they're sometimes not equal, and that the last one is probably the most informative one. If the "useless planes" during the first few steps are all-zero, they can be much more easily ignored in the start of the learning process.

Note that I suspect the difference really won't matter a whole lot at all, I suspect there'd just be a tiny difference in learning speed based on the intuition described above.


How to enforce covariance-matrix output as part of the last layer of a Policy Network?

Usually it is assumed that there is no correlation between different actions, so the covariance matrix will be zero everywhere except on the main diagonal. Diagonal will represent variances of actions. Diagonal covariance matrix will be positive semidefinite if all values on diagonal are $\geq$ 0 so you need to insure that output of final layer is $\geq$ 0, which can be done with ReLU activation for example.


If by "symbolic" you mean finding an analytical solution, that is, an equation for each weight, then the answer is no. The example you chose results in a system linear equations, which can be solved analytically. However once you introduce non linearities (by using activation functions with more than one layer), most non trivial cases will have no analytical solution and will need to be solved numerically. This is not a problem specific to tensorflow, it is  a mathematical issue, it will not be possible on any language, current or future. Unless there is some revolution in math first.


Does overfitting imply an upper bound on model size/complexity?

over-fitting does seem to imply an upper bound on model complexity.

As complexity of the model increases, its generality tends to decrease.  This is not an indisputable fact, but an identifiable trend, provided that the model is not inundated with redundancy.

For polynomial expressions, a Taylor series of 100 terms can always perfectly represent the position of 100 points in a time series, although no predictive general information can be ascertained from the 100 coefficients of the Taylor series.  This is also true for spectral analysis, Lyapunov exponent estimation, spline fitting, or artificial network convergence.

What is generally accepted as the definition of over-fitting does not come from the AI field.  It originates from the mathematics of surface approximation using closed forms.  The closed forms may be derived from a single differential equation or systems of them, but not usually, since closed forms do not exist for many such systems.

The information science principles behind feature extraction are closely aligned to the correlation between generality and predictive usefulness.

Still, we can't conclude, "There will always exist a model that is smaller and less complex than $M$ that will have a test error less than 30% on $S$."  Counter examples can probably be constructed with a little thought and effort.


First of all what you are trying to do can be achieved by simple logical programming. Secondly you are making things overly complex.

1 node in a Neural Net can predict as many values as you would like it to, as it outputs a real number no matter the activation function. It is us who round of the number to 0 or 1 or maybe 0, 0.5, 1 depending on the classification job at hand.

2 Neurons can predict 4 classes 00, 01, 10, 11 depending on how you use it. Now in your specific of quadrant detection 2 Neurons are sufficient, since not only can you decompose the problem into two separate problems, both the problems are linearly separable too. This is not always the case or it maybe too complex to find linearly separable classes, so in general one output node is reserved for 1 class only. If it is that class only that output node is activated while the rest is 0. Memory and process consuming, but easier.

Coming to your first approach. I don't exactly how hope to achieve 4 class classification with a single output node. Minimum of 2 nodes is required. The problem with your approach is that if you think critically your hidden nodes are not doing anything other than scaling the inputs, so almost the same type input is propagated and then you are asking a node to perform a a 4 task classification which is not linearly separable as per me. sin will not be able to separate it if you think critically.

The second idea is better, just remove the hidden node and you will be able to classify it based on 00, 01, 10, 11 outputs. Although this problem can be solved by simple logical computation and your Neural Net is basically doing nothing.

Please point out to me if you find any mathematical mistakes. Hope this helps!


In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?

That seems to be functional.
That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-learning.
By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely "good" ones, and you still want the network to try and improve on those.


What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

There is no difference. A policy decides which action to take in each state. This is usually split into deterministic policies of the form $\pi(s) : \mathcal{S} \rightarrow \mathcal{A}$ and stochastic policies of the form $\pi(s|a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}, \text{Pr}[A_t=a |S_t=s]$.

When we say that a policy is static, it means that the mapping from state to action - or distribution of actions - does not change over the time that we are interested in. This definition applies equally in the Reinforcement Learning (RL) and Contextual Bandit settings.

  But in a reinforcement learning problem, the history can be used to define a state.

It can be used this way, but is not required. What is required is the Markov property i.e. that identifying the state also determines all the allowed transitions and rewards, plus their probabilities of occurring given an action.

  In this case, does the history include the rewards revealed thus far as well?

If those can affect the future state transitions and rewards yes. If they generally do not, then you would usually exclude them from the state description.

  If so, the policy is not stationary anymore I guess.

What you are proposing is a system that changes its state depending on rewards seen so far. As above, this is not necessary to define a RL problem, but it is allowed in RL. Whilst in a contextual bandit you assume no rules apply to state transitions, and this is a key difference between contextual bandit and RL settings.

The policy is stationary if its mapping rules remain unchanged.

Your proposed addition to the state does not require changing the policy. The policy is a function of the state. It may choose a different action depending on this historical aspect of the state, but it can still remain a fixed function - its input may change, but the policy function itself does not need to change to account for that.


How to label edited images after data augmentation?

Yes, you should label it the same. But more importantly you need to make sure that each perturbation of the image doesn't change some important character of the image.

Consider training an apple classifier. If you plan to augment data by altering the RGB values, you need to be wary that you might cause issues in classification tasks where color is instrumental. Say, Granny apple / Fuji apple. If you still really want to augment data in this way, consider perturbing by smaller amounts each time.

Or consider an apple detector. An apple still "looks to be an apple" if looked at an angle, if looked through a mirror, if looked afar, but probably not if looked through a carnival mirror.

So ask yourself, if an image still NSFW if the colors are changed?

However, as a personal anecdote I don't think augmenting data by changing the color channel is a good idea. Also note that training on augmented data by altering the color channels as a whole should be more or less equivalent to training in B/W. Why?


How should I implement the backward pass through a flatten layer of a CNN?

Yes, a simple reshape would do the trick. A flattening layer is just a tool for reshaping data/activations to make them compatible with other layers/functions. The flattening layer doesn't change the activations themselves, so there is no special backpropagation handling needed other than changing back the shape.


Calculating Parameter value Using Gradient Descent for Linear Regression Model

X = np.array([1,2,3])
Y = np.array([2,1,2])

params = np.array([1, 0])

def loss(y, yhat):
    return ((y - yhat)**2).mean()

def model(x):
    return params[0] + params[1]*x

def loss_grad(y, yhat, x):
    return np.array([(2*(yhat-y)).mean(), (2*(yhat-y)*x).mean()])

lr = .1
for _ in range(3):
    yhat = model(X)
    l = loss(Y, yhat)
    g = loss_grad(Y, yhat, X)
    params = params - lr*g
    print(f'thetas are now {params} with new loss of {loss(Y, yhat)}')

outputs

thetas are now [1.13333333 0.26666667] with new loss of 0.6666666666666666
thetas are now [1.13333333 0.23111111] with new loss of 0.2696296296296296
thetas are now [1.14755556 0.22874074] with new loss of 0.262887242798354

I double checked this with keras too, but in numpy i explicitly wrote the gradients, i reccomend double checking your gradient or arithmetic


In 2016, can $1000.00 buy enough operations per second to be approximately equal to the computational power of a human brain?

The development of CPUs didn't quite keep up with Kurzweil's predictions. But if you also allow for GPUs, his prediction for 2009 is pretty accurate.
I think Moore's law slowed down recently and has now been pretty much abandoned by the industry. How much that will affect the 2019 prediction remains to be seen. Maybe the industry will hit its stride again with non-silicon based chips, maybe not.
And, of course, whether hitting Kurzweil's estimate of the computing power of the human brain will make an appreciable difference for the development of Artificial General Intelligence (AGI) is another question altogether.


After a model has been trained, how do I use it to address the real-world problems?

Using a machine learning or AI-powered model once it has been built and tested, is not directly an AI issue, it is just a development issue. As such, you won't find many machine learning tutorials that focus on this part of the work. But they do exist.

In essence it is the same as integrating any other function, which might be in a third-party library:

Package the new function so that it can be called from your production system (this may be the hardest part)
Decide where in the code of your production system to call the new function. In your example case maybe after form is completed describing an incident you could link the top recommended KB articles from the new ticket.
Design and check the associated user experience/UI as appropriate for your development team (in small projects you may skip this step and just implement)
Change your production code to call the packaged function. In any professional development team, this part will have multiple stages, but not really relevant to the question - if you are the ML specialist and delivering your new model to an existing team, you need to talk to them about both the packaging part and the steps involved here.

Often, machine learning models come with a whole bunch of dependencies that the rest of a system does not have. There are a variety of solutions for that, depending on the libraries that you are using, whether you are using a cloud PaaS service etc. You could just build a Docker image to hold all the AI parts and call it passing the input data.

Deploying ML models to production as a job is often a task for Machine Learning Engineer roles. There are courses and articles covering the practical aspects of these steps, if you search for them. Here are a couple:

Deploying Machine Learning Models from Coursera
A blog article on Medium: There are two very different ways to deploy ML models . . .

I have no affiliation with any of the above, have not read articles or taken the courses, and am not able to make any recommendation, even if you told me the technologies you were using for ML and in production currently.

After deploying to production . . .

Your work is not necessarily done. You will want to monitor behaviour of the system. During integration, you should of added logging, or some way to get feedback of performance in the wild.

For instance, if this is an ML system, does the accuracy seen in testing hold up against real life? Is the target population drifting? If the system makes interventions - by e.g. suggesting links, or showing categories to end users - how well are those working in practice? Is the performance and responsiveness of the system fast enough when the service is under load?

If it is more reactive AI system, that takes actions itself, you will similarly want to monitor what it is supposed to optimise, or sample its output for errors and quality control.

All this feedback can go back into a new iteration of design and integration. How to incorporate that will depend a lot on the nature of the system and what you discover, so could be the subject of further questions.


Is it possible to retrieve the optimal policy from the state value function?

You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment $p(s',r|s,a)$ - the probability of receiving reward $r$ and arriving in state $s'$ when starting in state $s$ and taking action $a$.
This looks like:
$$\pi^*(s) = \text{argmax}_a [\sum_{s',r} p(s',r|s,a)(r + \gamma v^*(s'))]$$
There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.
Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.


Is it necessary to standardise the expected output

It depends, as mentioned in comments, on your model and labels. For example how would you use standardisation on multi classification problem?
Generally, standardisation is more flavourable for input data as its mean is around 0.
I assume you have a regression model and in that case using standardisation could be better than normalisation.


Is increasing software complexity the most likely bottleneck to the AI singularity?

What if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?

If this turns out to be the case, then the complexity would indeed be the bottleneck.
There's a formal term for this: kolmogrov complexity or "k-complexity".  (The best formal definition of intelligence I've come across is Hutter & Legg's, which is broken down in the article "On the definition of intelligence" by our own nbro.  It will shed shed some light on the formal structure of the question at hand.

Any attempt to provide values to your d & l would speculative, since we're still so far from AGI. The question would be what is the effect of any given (d,l)

The subject can definitely be treated formally in a mathematical sense the and values analyzed.  (There may even been some formulae out there, although, they are likely to be more complex, per the Hutter&Legg.)
Sorry I can't give you a more formal answer--it's a good question!

Notes:
Another bottleneck would be the speed of light, essentially, "bits and bytes across a pipeline".  This would be a factor in both self-replication and networked processing and memory.  You can expand bandwidth/throughput, but not infinitely.
This relates to space complexity (how much volume/memory does the algorithm require?) in addition to the k-complexity, which itself is likely to be non-trivial for anything approaching AGI.
AI Threat:
From the standpoint of neoluddism, I'm skeptical that "infinitely" self-optimizing superintelligence is the greatest threat.  Evolution, and economic theory, suggest that the greater danger would come from not what is most expensive, complex, & capable, but that which is "just good enough", cheaper and more easily replicable.  This is because decision time becomes a critical factor in a competitive environment, as does replication time.
I always bring up grey goo because it only has to be good at 1 thing to eat everything else!


When does the mutation occur and how does it take place?

Finding a solution in NEAT algorithm is based on evolution strategy. It means that you have Neural Networks which are yours individuals, so mutations and crossing occurs in loop after phase of "fitnessing" (calculation fitness for every individual and removing bad ones).

  How is it chosen whether to add a node or to add a connection mutation? Furthermore, how is it chosen where the mutation is taking place in the network (between which connections)?

Randomly: just draw. You can read more about evolutionary algorithms here.

If it could somehow help, I include link to my repository with the implementation of NEAT.


Simple text recognition with neural network

If I'm reading it correctly, this question has nothing to do with optical character recognition. You want to create a system that takes a digital string of characters as input, then finds the best match from a predetermined list of words. That sounds like a task for if-then-else logic and dictionary lookup. It might be possible to use a neural net, but not easy.

A neural net takes a fixed number of inputs, each of which are a value between zero and one. A major hurdle is that you probably want variable-sized inputs. Another hurdle is that you'll need to code the inputs some way onto numbers.

These hurdles can be overcome but they are tipoffs that neural networks aren't well-suited for the task.


How can you represent the state and action spaces for a card game in the case of a variable number of cards and actions?

Instead of having the AI learn what action to take, you can alternatively train it to judge how "good" a position is. In order to determine what move to make, you don't ask the AI "This is the current state, what move should I make", you iterate through all possible moves, and feed the the resulting state into the AI asking "How good do you think this new state is?". You then chose the move with the resulting state that the AI liked best. (Or you probably even can combine this with a traditional MinMax approach)
I'm new to this area myself, but I'd guess you would use this approach when the action space is large, and in particular when most possible actions are not a legal option in most states.


How do GPUs faciliate the training of a Deep Learning Architecture?

GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector & matrix operations.

If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:

$$x_{i+1} = \sigma(W_ix_i + b_i)$$

where $x_i$ is the input vector, $W_i$ the matrix of weights, $b_i$ the bias vector in the $i^{th}$ layer, $x_{i+1}$ the output vector and $\sigma(\cdot)$ the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a LSTM cell you will notice that inside of it are multiple such operations.

Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.


What is the relationship between the Q and V functions?

Can we say that $Q^\pi(s, a) = V^\pi(s)$

No.

The correct relationship is this:

$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$

or, if you have a deterministic policy $a = \pi(s)$ you can instead write:

$$V^\pi(s) = Q^\pi(s, \pi(s))$$

Intuitively, this is because the $V^\pi(s)$ is the expected future return when following the policy $\pi$ from state $s$, whilst $Q^\pi(s, a)$ is the expected future return where it ignores the policy for only the next action $a$ (which will decide immediate reward $r$ and next state $s'$ independently of the policy), and thereafter follows $\pi$.

The above equations essentially show what happens when you apply the policy in state $s$ to decide which Q value(s) to use, they remove the independent choice of $a$ in $Q^\pi(s,a)$.

One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When any algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $s$ when making a free choice of $a$ for the next step, and thereafter strictly following the policy being evaluated.

What is different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different target policy. Both approaches have the same interpretation of what $Q(s,a)$ means otherwise, and have the same relationship between Q and V for their repsective policies.


Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true?

Let's say we have $a$ - constant and $\epsilon \sim \mathcal{N}(0,\sigma)$, then:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$
Expectations of constants are just the constants: $\mathbb{E}[a] = a$ and  $\mathbb{E}[a^2] = a^2$
The mean of $\epsilon$ is zero $\mathbb{E}[\epsilon] = 0$. And the expectation of $\epsilon^2$ is its variance:
$$ \mathop{\mathrm{Var}}(\epsilon)  = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$
Substituting, we get an expression for the original expectation:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{*}$$
Getting to the expectation in the book, we first substitute the values for $Y$ and $\hat{Y}$:
$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$
In the book it is assumed that $X$, $f$ and $\hat{f}$ are constant. So we can use the expression (*) with the constant being $a = f(X) - \hat{f}(X)$:
$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$


Which government agencies oversee development of new AI?

May not be quite what you’re looking for, but nonetheless helpful, I hope. The White House a year ago commissioned a report on AI that touches briefly on policy issues.


How can I handle overfitting in reinforcement learning problems?

Overfitting refers to a model being stuck in a local minimum while trying to minimise a loss function. In Reinforcement Learning the aim is to learn an optimal policy by maximising (not always but almost) a reward function rather than minimising a loss function, so you can't really talk of overfitting as in the supervised scenario, but you can definitely talk about sub-optimal policies.
If we think of a specific task like avoiding stationary objects, a simple sub-optimal policy would be to just stay still without moving at all, or moving in circles if the reward function was designed to penalise lack of movements.
The way to avoid an agent to learn sub-optimal policies is to find a good compromise between exploitation, i.e. the constant selection of the next action to take based on the maximum expected reward possible, and exploration, i.e. a random selection of the next action to take regardless of the rewards. Here's a link to an introduction to the topic:
Exploration and Exploitation in Reinforcement Learning
It is worth mentioning that sometimes an agent can actually outsmart humans though, some examples are reported in this paper The Surprising Creativity of Digital Evolution. I particularly like the story of the insect agent trained to learn to walk while minimising the contact with the floor surface. The agent surprisingly managed to learn to walk without touching the ground at all. When the authors checked what was going on they discovered that the insect leaned to flip itself and then walk using its fake 'elbows' (fig7 in the linked paper). I add this story just to point out that most of the time the design of the reward function is itself even more important than exploration and exploitation tuning.