How to choose evaluation functions for features, when network effects are in place (multi-agent systems)?

In a sense it seems like I am thinking some kind of "Deep Agent-Based Modeling", where it is okay to have network effects on the lowest layer (which would only evaluate the matching of the generated artifacts and their own preferences); the co-operation problem should be solved by a "Coach" layer, which would attempt to prevent situations, where some agents can not play well together.

The evaluation function of the "Coach" layer should be such, that it minimizes the amount of negotiation actions (energy efficient co-operation) required per artifact list generated and also minimizes the signaling distance regarding diversity (less popular artifacts / unorthodox solutions). The "Coach" layer should also be able to reorganize the lists contained by the Agents. The signaling distance can be easily evaluated in my use case, since the dataset has easily available popularity ranking per list artifact list item; the shorter the distance from popular artifact list to the least popular artifacts is, the better.

In practice this means, that I actually want to build composable Agents, which have lists of items (or other Agents), action space for generating artifacts from those items, evaluation functions and costs of specific actions, aggregateable popularity rank (which helps to evaluate delegation). All Agents would have generic default action space, which would help in solving the signaling problem; if the diameter of the co-operation network is too long (not within the scale-free network topology), the system should attempt to create a new layer or reorganize the existing layers (or do both).

All this might be computationally very expensive; however we tested a theory about building a "multi-dimensional" merge-sort algorithm by using entanglement / super dense encoding from quantum information theory and it would seem like it could work and be able to create a hash map, which would allow the Agents to reshape the scale-free network efficiently.

I might build some kind of demo out of this, since all this seems really interesting to me and complements nicely my past hobby projects.

I will not accept this answer until I get some verification, that this is a good answer. However, I will post this in hopes, that someone has already done similar work, gets inspired and can provide a better answer. This answer will probably be edited later, when I get to a more formal level with this.


What is the order of execution of steps in back-propagation algorithm in a neural network?

Both are feasible.
A generalization is to apply the modification of the weights after the presentation of $n$ examples. If $n=1$, this is online training. If $n=60$ ($60$, in your case, is the size of the dataset), this is batch training. In other cases, this is mini-batch training.
The main difference is not the computation complexity of the algorithm, but the theoretical speed of convergence to an "optimal" set of weights.
It is now generally admitted that online training is faster. Online training performs a stochastic approximation of the gradient descent method. The true gradient of the cost function over the whole training set is approximated by a gradient at each presentation of a training example.
Intuitively, with batch learning the weights are modified by an average of the gradient over the whole training set and averaging destroys information.
While with stochastic (or mini-batch learning), each example has his word to say. One after the other.
This has also been discussed here.


Are strong Chess AI's local on mobile devices?

Yes. In fact, I'm the developer of some mobile chess apps. [Disclaimer]

I would argue my chess engine products for the iOS (http://www.smallchess.com) are powerful and strong enough to beat any chess grandmaster. My apps run without network connection,


Hand-Signs Recognition using Deep Learning Convolutional Neural Networks

I feel your problem might not be with the model itself but with the dataset. If you only have $2500$ images for $24$ labels (hand-signs) that gets you roughly $104$ images per label. This is very little for the models I train (~$80K$ images in the smallest of cases). In my view you got a really decent accuracy at validation and test time for the size of your dataset.

But answering your questions:

Starting from few labels and extending is usually helpful when your model is too deep and suffers from convergence problems. Your model is simple enough not to suffer those problems so I would go learning the $24$ labels at once.
Transfer learning can help a lot at reducing training times. For example, if you start from a VGG classifier that detects hands there is a good chance that the weights of your convolutional layers are already almost configured for your use case.

Generally speaking, the easiest way to increase the accuracy of your model is to use one of these 2 methods:

Increase the dataset: I am not sure if it is possible in your case, maybe you can use image augmentation (rotation, zooming in/out, changes in the color space...).
Increase the depth of your model (provided that you have a big dataset).

If you already fulfilled those items, then you can go and make changes to the model architecture or loss function. Looking at your model and from the top of my head I would try to add Batch-Normalization for the convolutional layers.

Hope this helps a bit :)


Does it matter if it's a bot or a human generating text? Doesn't it come down to the content?

sometimes voice pitch also matters(and system designers forget about that) - if system is badly designed then you will feel that it is a bot, if design is great than you don't feel it's a bot at all


How can I convert the probability score between 0 to 1 to another format?

You could maybe do something like this, it's a bit hackish
\begin{equation}
y = C_1\cdot 1 + C_2 \cdot 0.5 + C_3 \cdot 0
\end{equation}
$y$ represents the output and its bounded $\in [0, 1]$. $C_i$ is probability for class $i$. This way when $C_1 \approx 1, C_2 \approx 0, C_3 \approx 0$ you have
\begin{equation}
 y \approx 1\cdot 1 + 0.5 \cdot 0 + 0 \cdot 0 \approx 1
\end{equation}
when $C_1 \approx 0, C_2 \approx 1, C_3 \approx 0 $ you have
\begin{equation}
 y \approx 1\cdot 0 + 0.5 \cdot 1 + 0 \cdot 0 \approx 0.5
\end{equation}
and when $C_1 \approx 0, C_2 \approx 0, C_3 \approx 1 $ you have
\begin{equation}
 y \approx 1\cdot 0 + 0.5 \cdot 0 + 0 \cdot 1 \approx 0
\end{equation}


Is GPT-3 an early example of Strong AI in a Narrow setting?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3".
Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).
While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning.
In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.
Check out this, this and the paper on Scaling Laws for Neural Language Models.


Why aren’t heuristics for Connect Four Monte Carlo tree search improving the agent?

It might be the case that if you perform a large number of random rollouts, the "best action" as chosen by the agent without the domain knowledge, is same as the agent with the domain knowledge. I guess what you can do is try to reduce the number of rollouts and see if the performance changes.


Has anyone attempted to train an AI to learn all languages?

There are approaches in machine translation that try to capture this kind of synergy between languages. The idea is that if you train your architecture to be able to translate English-Japanese, Japanese-English, Korean-English, English-Korean it will also be able to translate from Japanese to Korean without ever having seen a single such training example. Here, you can read about this so-called zero-shot translation.

It is also possible to train wordvectors on several languages at once, which might give you better wordvectors for a language with few training examples.

Of course for true language understanding you have to solve the grounding problem and I'm not sure using several language will help with that.


Is it a good idea to overfit on a small part of your data for faster model convergence?

The first thing which you first have to understand is that does your trained model is working efficiently with both the training and testing data. If yes then its not overfitting. There is only one case where over-fitting doesn't matter and that is when your testing data is same as training data and dealing with is very important or it could be disastrous. Therefore, to avoid overfitting you can try several methods and they are followings:

Optimization Algorithms like sgd, Adam, RMSprop, Adagard, are few gradient descents.
Loss function like Hinge loss could be useful.
Parameter initialization greatly influence the optimization process.

Overall, Large datasets are better for building ML models but there are some methods there is an interesting article that i would recommend you to read: Article Link

I Hope this Helps!!!


Is it possible to modify or replace the basic network of YOLO?

No, the original (or any) YOLO is for object detection. You can easily replace the feature extractor (DarkNet53, if I'm not mistaken) with any other, as long as you maintain the correct number of weights in the detection layer.


Do we have to use CNN for Deep Q Learning?

No. DQN and other deep RL methods work well with fully connected layers. Here's an implementation of DQN which doesn't use CNNs: github.com/keon/deep-q-learning/blob/master/dqn.py

DeepMind mostly use CNN because they use image as input state, and that because they tried to evaluate performance of their methods vs humans performance. Humane performance is easy to measure at games with image as input state, and that's why CNN based methods present so promptly in RL now.


Neural network architecture for line orientation prediction

A long shot: these guys have worked on a problem that might be relevant. They define "semantic" lines as lines delimiting significant regions or objects in an image. To detect such lines, they use the conv layers from a pre-trained VGG16 net and then add their own layers on top. The cool thing about their approach is that they run both classification and regression in parallel on the same network.

You might be able to adopt a similar technique to determine where the line is, and then run some simple analysis on the extracted line to determine the offset and the orientation.


To get a better understanding, let's compare a couple of games DOTA2 and Overwatch. At first, both seen similar in terms of multiplayer and strategy, but there is a significant difference.

You have mentioned it in the question. reaction time

Even though both the games require strategy and understanding of the surroundings and team play, FPS games are heavily dependent on reaction time and accuracy. As much as you try, it's next to impossible to reach the reaction times and accuracy of computer agent.

In games such as DOTA2, the game is more dependent towards strategy and how the characters interact with each other (Strengths and Weakness).

Finally, it doesn't make sense to train an AI based agent to play FPS games, because the optimal solution is already present and there is no way that AI can reach those levels. It's called AIM BOT. As good as you can get at strategy and planning, team work and reaction times, you can almost never beat it. It's next to 100% accurate and the opponent is basically dead as soon as he's in sight.


What kinds of techniques do autopilots of autonomous cars use?

They definitely don't use RL.

My guess is a mix of NN for segmentation, lidar, radar and how for perception.

Once object detection has been performed they use a trajectory generator and MPC.

There is an autonomous car class on Coursera that goes over this.


Why does the policy $\pi$ affect the Q value?

First of all, $Q_\pi(s, a)$ IS DEFINED AS the value (i.e. the expected return) of taking some action $a$ in some state $s$, AND THEN following some given policy $\pi$ (until e.g. the end of the game or your life). In other words, suppose that you take action $a$ in state $s$, AND THEN use the policy $\pi$ to behave in the world until you die, then $Q_\pi(s, a)$ would represent the value that you would obtain.

So, we are DEFINING $Q_\pi(s, a)$ in a certain way.  This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.

  From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$.

So, $Q_\pi(s, a)$ will not keep changing. You could say that $Q_\pi(s, a)$ (which is a function) is "affected by" $\pi$ ONLY in the sense that it is "defined in terms of" $\pi$. To be precise, $Q_\pi(s, a)$ is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.

  Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

Again, there's the distinction between the algorithm that you actually use to find the function $Q_\pi(s, a)$ and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book Reinforcement learning: an introduction by Sutton and Barto).


Recognising common short forms of appliances used daily

This is a pretty broad topic!

The question you're interested in has several parts.

The first part is Named Entity Recognition. This NLP task focuses on identifying the types of named objects. In your case, you'd want to be able to tell that something is an appliance (or perhaps even, that it is an air conditioner).

But that's not enough on its own. You also need to be able to tell which appliance they're talking about! This is usually studied as a separate problem, called Entity Recognition.

Both of these are considered pretty hard, and there isn't likely to be one "right" answer for your specific problem, but there are viable techniques. The wikipedia articles I linked to should be good starting places and suggest some out-of-the-box toolkits.


A mathematical explanation of Attention Mechanism

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them.

There is no one dominant attention approach and probably will not be. In fact, the earliest attention mechanism in common use in machines was likely the electromechanical fire alarm. In digital systems, it would be a vacuum tube electric eye driving an intruder alert followed by the first hardware interrupts in transistor microprocessor boards.

The sophistication of hardware interrupts in contemporary computer systems is probably higher than attention mechanisms in neural nets as of this writing, but that may change. Currently the dictionary definition of attention is the only constraint we can place on these newer approaches in artificial networks.

The four working links in the question (one is broken) lead to articles with some mathematical description, so the question half answers itself.
This article, Dynamic relevance: vision-based focus of attention using artificial neural networks,
Shumeet Baluja, Dean Pomerleau, 1995; revised 1996,
(abstract),
has only a few mathematical expressions but is filled with system
theory that could be described mathematically with a little reading
and thought.
Same with Self-Organization of an Artificial Neural Network
Subjected to Attention Shift Impairments and Familiarity
Preference, Characteristics Studied in Autism, Lennart Gustafsson and Andrew P. Papliński, 2004.
This one has clear and comprehensive mathematical treatment: Backpropagation Generalized Delta Rule for the Selective Attention Sigma-if Artificial Neural Network, Maciej Huk, 2012
This one too: Small and Dim Target Detection via Lateral Inhibition Filtering and Artificial Bee Colony Based Selective Visual Attention, Haibin Duan, Yimin Deng, Xiaohua Wang, Chunfang Xu, 2013
There is some here, but the visualizations and concepts are so theoretically relevant that one might not miss the more extensive mathematics: Using chaotic artificial neural networks to model memory in the brain, Zainab Aram, Sajad Jafari, Jun Ma, Julien C. Sprott, Sareh Zendehrouh, Viet-Thanh Pham
Solid math coverage here: Parallel Implementation of Artificial Neural Network Training, Stefano Scanzio, Sandro Cumani, Roberto Gemello, Franco Mana, P. Laface, 2010, (abstract)
Also here: Multiple Object Recognition with Visual Attention, Jimmy Lei Ba, Volodymyr Mnih, Koray Kavukcuoglu, 2015 ICLR
The mathematics for the very popular DiSAN is here: DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding, Tao Shen, Jing Jiang, Tianyi Zhou, Shirui Pan, Guodong Long, Chengqi Zhang, 2018
Mathematics and the algorithm is provided for this unique approach too: Graph Classification using Structural Attention, John Boaz Lee, Ryan Rossi, Xiangnan Kong, 2018

It would be interesting to develop a taxonomy of attention approaches in AI, as that has probably not yet been done. It would take quite a study to see if any of the above bullet items match up with either of the two articles referenced in the question.


Is the AI Effect caused by bad tests of intelligence?

I think it is mostly right. But not that intelligence is hard to define. In my opinion, it is simple: A is more intelligent than B if A achieves some purpose in less steps than B. It is functional/algorithmic efficiency.

What is difficult to define is human intelligence.

But when someone says, "No, X is not real intelligence," what they mean is that it does not satisfy what we would consider real human intelligence.

So when people discount new and amazing discoveries in machine intelligence, it is not because they are not amazing in their own way, but because those discoveries, while exhibiting intelligence, are actually not replicating human intelligence - which is what many people actually mean when they say "that thing isn't really intelligent."

In truth we are very far, in the science of artificial intelligence, algorithmically speaking, from an artificial human intelligence (AHI).

Additional note: What's funny is that we don't call the science of artificial intelligence just 'the science of intelligence.' That we add the "artificial" qualifier by necessity pegs the science to what the artificiality implicitly emulates: human intelligence. In other words, "artificial intelligence" must be by definition more specific to the thing it allegedly artificializes than a more general science of just "intelligence."


Searching an AI to recognize and locate persons in a factory

I am definitely not the most qualified person in order to answer this question, but I might be able to give you a couple of buzz words to go and do some more research on your own. A convolutional neural network is usually used for analyzing anything visual. (you can read about it here: https://en.wikipedia.org/wiki/Convolutional_neural_network)

There are a lot of frameworks that allow you to build recognition classifiers that can be loaded into a real-time video feed and give you feedback when it gets a hit.

Some of the ones that I know of are:

https://caffe2.ai/ or http://caffe.berkeleyvision.org/
https://pjreddie.com/darknet/
https://docs.opencv.org/2.4/modules/ml/doc/neural_networks.html

With Caffe being the most popular.

Some articles on how these work:

https://pjreddie.com/darknet/yolo/
http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

Hope this is at least able to get you on the right path.


(Of course, similar questions have been asked in the past and there are many sites, papers, video lessons, online that explain how CNNs work, but I think it's still a good idea to have a reference answer that hopefully will give you the main ideas behind CNNs.)
A convolutional neural network (CNN) is a neural network that performs the convolution (or cross-correlation) operation typically followed by some downsampling (aka pooling) operations.
The convolution operation comes from the mathematical equivalent operation, which is an operation that takes as inputs two functions $f$ and $h$ and produces another function $g$ as output, which is often denoted as $f \circledast h = g$, where $\circledast$ is the convolution operation (or operator).
In image processing, the convolution is used to process images in multiples ways. For example, it is used to remove noise (e.g. the convolution of a noisy image with a Gaussian kernel produces a smoother image) or to compute derivatives of the image, which can then be used e.g. to detect edges or corners, which are usually the main features of an image. For example, the Harris corner detector makes use of the partial derivatives (in the $x$ and $y$ direction) of the image to find corners (or interest points) in the image.
In the context of CNNs, $f$ is the image (in the case of the first layer of the CNN) or a so-called feature map (in the case of hidden layers), $h$ is the kernel (aka filter) and $g$ is also a feature map. (In this answer, I explain these concepts, including how an image can be viewed as a function, more in detail, so I suggest that you read it).
Here's an illustrative example of how the convolution works.

where the $\color{blue}{\text{blue}}$ grid is the image, the $\color{gray}{\text{gray}}$ grid is the kernel and $\color{green}{\text{green}}$ grid is the feature map (i.e. the output of the convolution between the image and the kernel). The white squares are around the images represent the padding that is added around the image so that the convolution operation produces a feature map of a specific dimension.
Essentially, the convolution is a series of dot products between the kernel and different patches (or parts) of the image. The convolution can even be represented as matrix multiplication, so the name convolution shouldn't scare you anymore, if you are familiar with dot products and matrix multiplications.
As opposed to many operations in image processing where the kernels are typically fixed, in the context of CNNs, the kernels are learnable parameters, i.e. they change depending on the loss function, so they are supposed to represent functions that when convolved with their respective input functions are useful to extract meaningful information (i.e. features) to solve the task the CNN is being trained to solve. For this reason, the convolution is often thought of as an operation that extracts features from images. In fact, the output of the convolution, in the context of CNNs, is often called feature map. Moreover, a CNN is typically thought of as a data-driven feature extractor for the same reason.
There are many different variations of the standard convolution operation. For example, there are transposed convolutions, dilated convolutions, etc., which are used to solve slightly different problems. For example, the dilated convolution can be used to increase the receptive field of an element of a feature map (which is typically a desirable property for several reasons). There are also upsampling operations which are particularly useful in the context of image segmentation. These different convolutions and their arithmetic are explained very well in the paper A guide to convolution arithmetic for deep learning by Vincent Dumoulin and Francesco Visin. Here's also the associated Github repository that contains all the images in this paper that show how the convolution operations work (from which I also took the gif above).
To conclude, CNNs are very useful to process images and extract features from them because they use convolution operations (and downsampling and upsampling operations). They can be used for image (or object) classification, object detection (i.e. object localization with a bounding box + object classification), image segmentation (including semantic segmentation and instance segmentation), and possibly many other tasks where you need to learn a function that takes as input images and needs to extract information from those images to get someone high-level (but also low-level) output (e.g. the name of the object in the image), given some training data.


How can I cluster based on the complementary categories?

Note: K-means does not assume an interpretation/label of the clusterings - in fact it is an unsupervised algorithm. The interpretations are a result of human analysis after running K-means.

For example, in the case of cats and dogs one would most definitely chose k = 2 - which provides an easy interpretation. However, what would it mean if we set k = 1000. We no longer have a "clean" interpretation of the centroids.

Note: how I keep saying "interpretation." The algorithm simply assigns a data point to a cluster and calls it a day. Humans then look at the results and try to understand them with an interpretation.

Continuing with the example where k = 2. One could easily interpret "is cat" as "not dog" and "is dog" as "not cat." The idea here is that the data is unlabeled beforehand and humans try to fathom the results retrospectively by assigning the resulting clusters with an understandable label.

I hope this clarifies the issue.


Is there any website that allows you to choose an algorithm, code it and visualise how it works?

Not planning, but this is a visual in-browser neural network for your interest:

http://playground.tensorflow.org/


How can I model and solve the Knight Tour problem with reinforcement learning?

Model your problem as an MDP

To solve a problem with reinforcement learning, you need to model your problem as a Markov decision process (MDP), so you need to define

the state space,
the action space, and
the reward function

of the MDP.

Understand your problem and the goal

To do define these, you need to understand your problem and define it as a goal-oriented problem.

In the knight tour problem, there's a knight that needs to visit each square of a chessboard exactly once. The knight can perform only $L$-shaped moves (as for the rules of chess). See the animation below (taken from the related Wikipedia article).

The goal is then, by making $L$ moves, to find a path through the squares such that each square is visited exactly once.

What is the state space?

You could think that the state space $S$ could be the set of all squares of the chessboard. So, if you have an $n \times n$ chessboard, then $|S| = n^2$, i.e. you will have $n^2$ states.

However, this can be problematic because a square alone doesn't tell you all the information that you need to know to take the optimal action. So, you need to define the states such that all available information is available to the agent, i.e. you need to define a state as the position of the current square and the position of the other available squares.

What is the action space?

The action space could be defined as the set of all actions that the knight can take across all states. Given that the knight can only take $L$ moves, whenever the knight is in state $s$, only $L$-shaped actions are available. Of course, it is possible that, for each state $s$, there's more thane one valid $L$-shaped action. That's fine. However, the chosen $L$-shaped action will definitely affect your next actions, so we need a way of guiding the knight. That's the purpose of the reward function!

What is the reward function?

The reward function is typically the most crucial function that you need to define when modeling your problem as an MDP that needs to be solved with an RL algorithm.

In this case, you could give a reward of e.g. $1$ for each found path. More precisely, you will let your RL agent explore the environment. If it eventually finds a correct path (or solution), you will give it $1$. You could also penalise the knight if it ends in a situation where it cannot take an $L$-shaped action anymore. Given that you don't really want this to happen, you could give it a very small reward e.g. $-100$. Finally, you could give it a reward of $0$ for each action taken, which could imply that you don't really care about the actions that the knight takes, as long as it reaches the goal, i.e. find a path through the chessboard.

The design of the reward function will highly affect the behaviour and performance of your RL agent. The above-suggested reward function may actually not work well, so you may need to try different reward functions to get some satisfactory results.

Which RL algorithm to use?

Of course, you will also need to choose an RL algorithm to solve this problem numerically. The most common one is Q-learning. You can find its pseudocode here.

How to implement this with OpenAI's gym?

You probably need to create a custom environment and define the state and action spaces, as well as the reward function. I cannot tell you the details, but I think you can figure them out.

Is RL the right approach to solve this problem?

RL isn't probably the most efficient approach to solve this problem. There are probably more efficient solutions. For example, there's a divide-and-conquer approach, which I am not familiar with, but that you may also try to use and compare with the RL approach.

You could also read the paper Solution of the knight's Hamiltonian path problem on chessboards (1994), especially, if you are already familiar with the Hamiltonian path problem (HPP). Note that the knight tour problem is an instance of the HPP.


Why is "-0.5 * torch.sum(1 + sigma - mu.pow(2) - sigma.exp())" in Pytorch equivalent to the KL?

The code is correct. Since OP asked for a proof, one follows.
The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: sigma is the natural logarithm of the variance, where usually a normal distribution is characterized in terms of a mean $\mu$ and variance. Some of the functions in OP's link even have arguments named log_var.$^*$
If you're not sure how to derive the standard expression for KL Divergence in this case, you can start from the definition of KL divergence and crank through the arithmetic. In this case, $p$ is the normal distribution given by the encoder and $q$ is the standard normal distribution.
$$\begin{align}
D_\text{KL}(P \| Q) &= \int_{-\infty}^{\infty} p(x) \log\left(\frac{p(x)}{q(x)}\right) dx \\
&= \int_{-\infty}^{\infty} p(x) \log(p(x)) dx - \int_{-\infty}^{\infty} p(x) \log(q(x)) dx
\end{align}$$
The first integral is recognizable as almost definition of entropy of a Gaussian (up to a change of sign).
$$
\int_{-\infty}^{\infty} p(x) \log(p(x)) dx = -\frac{1}{2}\left(1 + \log(2\pi\sigma_1^2) \right)
$$
The second one is more involved.
$$
\begin{align}
-\int_{-\infty}^{\infty} p(x) \log(q(x)) dx
&= \frac{1}{2}\log(2\pi\sigma_2^2) - \int p(x) \left(-\frac{\left(x - \mu_2\right)^2}{2 \sigma_2^2}\right)dx \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\mathbb{E}_{x\sim p}[x^2] - 2 \mathbb{E}_{x\sim p}[x]\mu_2 +\mu_2^2} {2\sigma_2^2} \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + \mu_1^2-2\mu_1\mu_2+\mu_2^2}{2\sigma_2^2} \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2}
\end{align}
$$
The key is recognizing this gives us a sum of several integrals, and each can apply the law of the unconscious statistician. Then we use the fact that $\text{Var}(x)=\mathbb{E}[x^2]-\mathbb{E}[x]^2$. The rest is just rearranging.
Putting it all together:
$$
\begin{align}
D_\text{KL}(P \| Q) &= -\frac{1}{2}\left(1 + \log(2\pi\sigma_1^2) \right) + \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} \\
&= \log (\sigma_2) - \log(\sigma_1) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac{1}{2}
\end{align}
$$
In this special case, we know that $q$ is a standard normal, so
$$
\begin{align}
D_\text{KL}(P \| Q) &= -\log \sigma_1 + \frac{1}{2}\left(\sigma_1^2 +
\mu_1^2 - 1 \right) \\
&= - \frac{1}{2}\left(1 + 2\log \sigma_1- \mu_1^2 -\sigma_1^2   \right)
\end{align}
$$
In the case that we have a $k$-variate normal with diagonal covariance for $p$, and a multivariate normal with covariance $I$, this is the sum of $k$ univariate normal distributions because in this case the distributions are independent.
The code is a correct implementation of this expression because $\log(\sigma_1^2) = 2 \log(\sigma_1)$ and in the code, sigma is the logarithm of the variance.

$^*$The reason that it's convenient to work on the scale of the log-variance is that the log-variance can be any real number, but the variance is constrained to be non-negative by definition. It's easier to perform optimization on the unconstrained scale than it is to work on the constrained scale in $\eta^2$. Also, we want to avoid "round-tripping," where we compute $\exp(y)$ in one step and then $\log(\exp(y))$ in a later step, because this incurs a loss of precision. In any case, autograd takes care of all of the messy details with adjustments to gradients resulting from moving from one scale to another.


Accuracy scores in a Deep Learning project

Precision is the number of true positives over the number of predicted positives(PP), and recall is the number of true positives(TP) over the number of actual positives(AP) you get. I used the initials just to make it easier ahead.

A true positive is when you predict a car in a place and there is a car in that place.

A predicted positive is every car you predict, being right or wrong does not matter.

A actual positive is every car that actually is in the picture.

You should calculate these values separately for each category, and then sum over the examples you sampled, if I am not mistaken.

So for the CAR category you have (assuming the predictions do match with the target, i.e., you are not predicting a truck as a car for example) :

model 1
line 1  -> 2 TP, 2 PP, 4 AP
line 2  -> 0 TP, 0 PP, 2 AP

So in total precision  is 2/2 = 1 and recall is 2/6 = 0.3(3).

You can then do the same for the other categories, and for the other models. This way you can check if a model is predicting one category better than the other. For example, model 1 can be better at finding cars in a picture whilst model 3 can be better at finding buses.

The important part is that you know if the objects the model predicted actually correspond to what is in the picture. A very unlikely example would be a picture with 1 car and 1 truck where the algorithm recognizes the car as a truck and the truck as a car. From the info that is in the table I cannot be sure if the 2 cars you predict are the actual cars in the picture, or in other words, if they are actually True Positives or are actually False Positives.


Are there any novel quantum machine learning algorithms that are fundamentally different from "classical" ones?

Generally, if one googles "quantum machine learning" or anything similar the general gist of the results is that quantum computing will greatly speed up the learning process of our "classical" machine learning algorithms.

This is correct. A lot of machine learning methods involve linear algebra, and it often takes far fewer quantum operations to do things in linear algebra than the number of classical operations that would be needed. To be more specific, for a matrix of size $N\times N$, if a classical computer needs $f(N)$ operations to do some linear algebra operation, such as diagonalization which can take $f(N)=\mathcal{O}(N^3)$ operations on a classical computer, a quantum computer would often need only $\log_2 f(N)$ operations, which inn (in and only in) the language of computational complexity theory, means exponential speed-up. The "only in" part is there because we have made here an assumption that "fewer quantum operations" means "speed-up", which for now is something we only know to be true in the world of "computational complexity theory".

However, "speed-up" itself does not seem very appealing to me as the current leaps made in AI/ML are generally due to novel architectures or methods, not faster training.

I disagree. Take vanilla deep learning for example (without GANs or any of the other things that came up in the last decade). Hinton and Bengio had been working on deep learning for decades, so why did interest in deep learning suddenly start growing so much from 2011-2014 after a roughly monotonic curve from 1988-2010? Not that this rise started before newer advances such as GANs and DenseNet were developed:

Notice also the similarity between the above graph and these ones:

These days pretty much everyone doing deep learning uses GPUs if they have access to GPUs, and what is possible to accomplish is extremely tied to what computing power a group has. I don't want to undermine the importance of new methods and new algorithms, but GPUs did play a big role for at least some areas of machine learning, such as deep learning.

Are there any quantum machine learning methods in development that are fundamentally different from "classical" methods?

I think you mean: "Most quantum machine learning algorithms are simply based on classical machine learning algorithms, but with some sub-routine sped-up by the QPU instead of a GPU -- are there any quantum algorithms that are not based on classical machine learning algorithms, and are entirely different".
The answer is yes, and more experts might be able to tell you more here.
One thing you might consider looking at is Quantum Boltzmann Machines.
Another things I'll mention is that a child prodigy named Ewin Tang who began university at age 14, discovered at around the age of 17 some classical algorithms that were inspired by quantum algorithms rather than the other way around, and
the comments on the Stack Exchange question Quantum machine learning after Ewin Tang might give you more insight on that. This is related to something called dequantization of quantum algorithms.

By this I mean that these methods are (almost*) impossible to perform on "classical" computers. *except for simulation of the quantum computer of course

Unfortunately quantum computers can't do anything that's impossible for classical computers to do, apart from the fact that they might be able to do some things faster. Classical computation is Turing complete, meaning anything that can be computed can be computed on a big enough classical computer.


Appropriate metric and approach for natural language generation for small sentences

I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the lengths of sentences are higher. So the value of 0.5 doesn't tell much. The fact that you improved accuracy by 0.1 for instance on the same dataset means one method is better than another. Also, people measure perplexity and it's more sensitive to small changes of loss than accuracy. That's why I suggest you to measure perplexity. This metric also depends on lengths of sentences, that's why you should only compare 2 models trained on the same dataset.
What about GPT and Transformer, they were invented to solve one of the issues of LSTMs (they're not capable to memorize a very large context), that's why transformers would be better for long sentences, but they would also be better for short I think due to their attention mechanist, which has many useful properties.


Why do my rewards reduce after extensive training using D3QN?

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science stack exchange, and reproduce the same answer here.

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.
If you trained a neural network to recognise cats and dogs and did the following:

Train it for many epochs on a full dataset until you got a high accuracy.

Continue to train it, but remove all the cat pictures.

Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.
Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.
Later on, when something bad happens and the NNs high predicted value is completely wrong, the error can be high. In addition the NN may have incorrectly "linked" features of the state representation so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.
There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.
For Cartpole, I found a very simple hack made the learning very stable. Keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper "The importance of experience replay database composition in deep reinforcement learning"


How to teach a model-based reflex agent for doing some task using machine learning methods?

The question is really broad---as stated by @thecomplexitytheorist---so difficult to give a meaningful answer. The following is about a clarification about the problem, and some directions.

A model-based reflex agent is a blueprint describing the key components necessary to build that agent. It is an abstract architecture to guide the creation of concrete agents. Whether software or hardware, the target agent should have a bit serving as "sensors" (a HW camera, or a software API"), "actuators", memory, etc. So if you aim at a HW agent, you need to decide what compenents, and collect them. A SW agent would be the definition of a software architecture, or the use of some modelling framework.

The Machine Learning part is about endowing the agent with some skill. It is the "smarts" programming part, what the "brains" will do in the instance of the blueprint. Please note that it does not have to be Machine Learning. It could be another "style" of programming, such as a rule-based system, or a plain hand-crafted program.

How to teach such agent is currently an open question. First steps usually start with drawing the agent as a black box, its inputs (e.g. symptom data), its desired output (diagnosis and alert). Then we detail the black box in terms of what sensors the agent needs to process the input, what actuators for the output, and what it needs to learn and memorize. Depending on the available input data and diagnosis/alert output system (email?), the next stage aims at refining step by step, until a good idea of what needs be implemented emerges.

At this point, all pieces are in place, except the internal model---the piece that really pertains to Machine Learning (if you choose ML for that). As the input data is available, and the output format decided, the "final" stage (before implementation) is to define the model. It really depends on the actual data and the goal (here we assume prediction). Labelled data (we can teach with input and output, as we already know them) usually leads to supervised learning. Unlabelled data (we can teach with the input only) leads to unsupervised learning or reinforcement learning. Once one understands the situation, we can choose some algorithm like SVM, decision trees, neural networks, etc. Note that the final decisions (before implementation) require studying the data beforehand (Is it regular? Are there missing bits? Is it in a format useful for ML? Etc.) to make appropriate choices.

Final note: The question may get closed on this site, because it is too broad. Way more useful to ask narrow questions, with a clear answer or set of answers. As you see here, this long answer is just the tip of how to teach an agent (I could not sleep anyway). And no implementation yet. In fact, all this could be summed up as "typical system development, with an ML component". Two guiding principles: (1) isolate the ML component(s), so specialists can dig and make the best of them, and (2) keep it simple.


Biological analogy for boosting and inhibition idea in Hierarchical Temporal Memory (HTM)

After doing some research, I found that "boosting" and "inhibition", in the context of HTM, are Numenta's implementation of homeostasis.


How do you distinguish between a complex and a simple model in machine learning?

Consider a continuum of complexity in models.

Trivial: $y = x + a$
Simple: $y = x \, \log \, (a x + b) + c$
Moderately complex: A wind turbine under constant wind velocity
Very complex: Ray tracing of lit 3-D motion scenes to pixels
Astronomically complex: The weather

Now consider a continuum regarding the generality or specificity of models.

Very specific: The robot for the Mars mission has an exact mechanical topology, materials call-out, and set of mechanical coordinates contained in the CAD files used to machine the robot's parts.
Somewhat specific: The formulas guiding the design of an internal combustion engine, which are well known.
Somewhat general: The phenomenon is deterministic and the variables and their domains are known.
Very general: There's probably some model because it works in nature but we know little more.

There are twenty permutations at the above level of granularity. Every one has purpose in mathematical analysis, applied research, engineering, and monetization.

Here are some general correlations between input, output, and layer counts.

Higher complexity often corresponds to larger layer count.
Higher i/o dimensionality corresponds to higher width to the corresponding i/o layers.
Mapping generality to or from specificity generally requires complexity.

Now, to make this answer even less appealing to those who want formula answer they can memorize, ...

Each artificial network is a model of an arbitrary function before training and a model of a specific function afterward.
Loss functions are models of disparity.
An algorithm is a model of a process created by spreading a recursive definition out in time to map into a model of centralized computation called a CPU.
The recursive definition is a model too.

There is almost nothing in science that is a not a model except ideas or data that are not yet modeled.


Should I use exploration strategy in Policy Gradient algorithms?

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.

Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C.

  Is it beneficial or is it common to use extra exploration strategy like UCB, Thompson sampling etc. n such algorithms?

It can be, but needs to be done carefully, as the gradient sampling for the policy function is different. Many policy gradient methods are strictly on-policy and will not work if you simply add extra exploration. It is relatively straightforward to adjust the critic part of actor-critic methods by using e.g.  Q learning update rules for it. However, the gradient of the policy function is trickier.

There are some policy gradient methods that do work with a separate, tunable, exploration function. Deep Deterministic Policy Gradient (DDPG) may be of interest to you - as per the title, it works with a deterministic policy function, and exploration is achieved by adding a separate noise function on top. The sampling for policy gradient is then corrected for being off-policy.


Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation.  This is based on training on a history of 45 input time steps and forecasting out 10 steps in the future.  I have assumed that your 'num_steps' is a stride through the training data.  Note that I have not tested this and you will most likely need to tweak a parameter or two:

{
    "data": {
        "sequence_length": 55,
        "train_test_split": 0.85,
        "normalise": true,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 100,
                "input_timesteps": 45,
                "input_dim": 2,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "lstm",
                "neurons": 100,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

Try this out and plot your model's loss and accuracy by epoch.  See Display Deep Learning Model Training History in Keras for some information and code in this area.  Let me know if you are not using Keras and I will point you to another article/code solution.


Are artificial intelligence learnings or trainings transferable from one agent to the other?

The simplest case is copying the software. That is instant duplication of the learnings. In a similar, but less trivial way you can adjust pretrained neural network classifiers to new datasets by simply re-initializing the last layer.

It gets more interesting when you have multiple agents and you want to combine their knowledge. This can be done with many techniques; the simplest is averaging the weights of identical neural network architectures. One example of this combined learning happened is Google Has a Room Full of Robot Arms Learning Hand-Eye Coordination. The relevant paper is Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection.

The most complicated case, where you want to transfer just the gist and prevent forgetting is -- up to my knowledge, and I'm relatively certain it's outdated in that area -- unsolved. The problem is post-hoc explanations. Humans decide to do something, observe the outcome and then explain why they did something / why it was good to do it. So we automatically generate hypotheses about the world. And sometimes we are able to formulate them in a way that others can understand them. That is not possible with current learning machines. We don't know yet how to abstract automatically from arbitrary problems.


In RL, the phrase "acting greedily" is usually short for "acting greedily with respect to the value function". Greedy local optimisation turns up in other contexts, and it is common to specify what metric is being maximised or minimised. The value function is most often the discounted sum of expected future reward, and also the metric used when defining a policy as "acting greedily".
It is possible to define a policy as acting greedily with respect to immediate expected reward, but not the norm. As a special case, when the discount factor $\gamma = 0$ then it is the same as acting greedily with respect to the value function.
When rewards are sparse (a common situation in RL), acting greedily with respect to expected immediate reward is not very useful. There is not enough information to make a decision.

I want to know the actions that will be taken in 2 cases:

$v_\pi(s)$ is known and $R_s$ is also known (only).

To act greedily in RL, you would use the value function $v_\pi(s')$ - the value function of the next states. To do so, you need to know the environment dynamics - to go with the notation $R_s$ you should also know the transition matrix $P_{ss'}^a$ - the probability of transitioning from $s$ to $s'$ whilst taking action $a$:
$$\pi'(s) = \text{argmax}_a \sum_{s'} P_{ss'}^a(R_{s} + \gamma v_{\pi}(s'))$$
Notes:

This assumes $R_{s}$ is your immediate reward for leaving state $s$. Substitute $R_{s'}$ if the reward matrix is for entering state $s'$ instead.

The policy gained from acting greedily with respect to $v_\pi(s)$ is not $\pi$, it is (a usually improved) policy $\pi'$

$q_{\pi}(s, a)$ is known and $R_s^a$ is also known (only).

To act greedily in RL, you would use the value function $q_{\pi}(s, a)$, and this is much simpler:
$$\pi'(s) = \text{argmax}_a q_{\pi}(s, a)$$
Again, the policy gained from acting greedily with respect to $q_\pi(s,a)$ is not $\pi$, it is (a usually improved) policy $\pi'$


Where can I find pre-trained language models in English and German?

Of course now there has been a huge development:
Huggingface published pytorch-transformers, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including many pretrained (mostly English or multilingual) models (docs here). It also includes one German BERT model. SpaCy offers a convenient wrapper (blog post).

Update: Now, Salesforce published the English model CTRL, which allows for use of "control codes" that influence the style, genre and content of the generated text.

For completeness, here is the old, now less relevant version of my answer:

Since I posed the question, I found this pretrained German language model:
https://lernapparat.de/german-lm/

It is an instance of a 3-layer "averaged stochastic descent weight-dropped" LSTM which was implemented based on an implementation by Salesforce.


Static or dynamic learning rate (Q-learning)

Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments.

However, this is something that is harder to manage than in supervised learning, and might not be as useful or necessary. The issues you need to be concerned about are:

Non-stationarity

The target values in value-based optimal control methods are non-stationary. The TD target derived from the Bellman equation is never quite a sample from the optimal action value until the very final stages of learning. This is an issue both due to iterative policy improvements and the bootstrap nature of TD learning.

Reducing learning rate before you reach optimal control could delay finding the optimal policy. In general you want the learning rate to be just low enough that inaccuracies due to over/undershooting the correct value don't prevent or delay differentiating between actions for whatever the interim policy is.

Policy indifference to accurate action values

Q-learning's policy is based on $\pi(s) = \text{argmax}_{a} Q(s,a)$. To obtain an optimal policy, what you care about is identifying the highest-valued action, $a$. As such, it is possible in some environments to achieve optimal control whilst Q values are far from accurate, provided the highest-valued action still has the highest estimate of expected return.

Whether or not you need highly accurate Q values to determine a policy depends on the environment, how similar actions are to each other. For long-term forecasts (i.e. $\gamma \approx 1$) where individual time steps make little difference, this is a more important detail, as there will be only minor differences between action values.

If your goal is to get highly accurate predictions of action values, then the above does not necessarily apply.

Some deterministic environments can use a high learning rate

This only applies to some, simpler environments, but you should definitely bear this in mind when working with tabular solvers.

For instance, it is possible to apply tabular Q-learning to Tic Tac Toe with a learning rate of $1.0$ - essentially replacing each estimate with a new latest estimate - and it works just fine.

In other, more complex environments, this would be a problem and the algorithm would not converge. But clearly, adding a learning rate decay is not a general requirement.

The learning rate decay schedule is a hyper parameter

There is no generic schedule that could apply to all environments and be equally effective in them. For an optimal approach, you would need to run a search over possible decay schedules, and the most efficient learning rate decay would apply only to the environment that you tested. It would likely apply well enough to similar environments that it could be an advantage to know it in future.

In complex environments, there are other working solutions

In practice, for environments where neural networks are used, an approach called Deep Q Networks (DQN) is used. It is common to use a conservative/low learning rate with this due to stability issues of combining neural networks and off-policy reinforcement learning.

In addition, you will quite often see an adaptive gradient optimiser - e.g. Adam or RMSProp - used with DQN. These already adjust gradient step sizes based on recent gradients observed in the neural network during training, so there is less need to add a dynamic learning rate schedule on top.

Supervised learning challenges, such as ImageNet do sometimes add a learning rate schedule over adaptive optimisers, resulting in further improvements. So it might help with DQN, but the other caveats above could still apply.


Why do we need weights when training a perceptron as an OR gate?

This would successfully simulate an OR gate.

Of course. In fact, hardware implementation of an OR gate needs just a few transistors. It may sound surprising, but the best python implementation of OR is a or b.

  So why do I need to "train" any weights?

It's one of the simplest functions out there, everyone understands how OR works just from the boolean table. The task shows that a neural network can learn it as well, just from examples and without specification what a logical operation is and how to wire it.


Are there real applications of fuzzy logic?

You've obviously never heard of fuzzy logic washing machines.

  ● Typically, fuzzy logic controls the washing process, water intake,water temperature, wash time, rinse performance, and spin speed. This optimises the life span of the washing machine. More sophisticated machines weigh the load (so you can’t overload the washing machine), advise on the required amount of detergent, assess cloth material type and water hardness, and check whether the detergent is in powder or liquid form. Some machines even learn from past experience,memorising programs and adjusting them to minimise running costs.

Fuzzy logic is used in a variety of control applications.  If your furnace can only be on or off, for example, you might use a probabilistic function of temperature to determine when to turn it on and off, rather than having fixed high and low temperatures activate your thermostat.  In some applications, that's been found to improve perceived comfort or efficiency.

For more sophisticated AI applications, you could use fuzzy logic for activations in a neural net, but I don't think it's offering much improvement over fixed, weighted activations.


$l_{2,1}$ is a matrix norm, as stated in this paper. For a certain matrix $A \in \mathbb{R}^{r\times c}$, we have  $$||A||_{2,1} = \sum_{i=1}^r \sqrt{\sum_{j=1}^c A_{ij}^2}$$
You first apply $l_2$ norm along the columns to obtain a vector with r dimensions. Then, you apply $l_1$ norm to that vector to obtain a real number. You can generalize this notation to every norm $l_{p,q}$.


What benefits can be got by applying Graph Convolutional Neural Network instead of ordinary CNN?

Generally speaking a graph CNN is applied to data represented by graphs, not images.

a graph is a collection of nodes and edges connecting them.
an image is a 2D or 3D matrix, in which each element denotes a pixel in space

If your data are just images, or something similar (e.g. some fMRI data), you usually cannot benefit from graph CNN compared with usual CNN.

Sometimes, the class labels of your images may be organized in a graph-like  (or tree-like) structure. In that case, you may have a chance to benefit from graph CNN.


I think you're looking for the minimization of false positives, that is, the instances that are classified as belonging to the desired class (the positive part of false positives) but that do not actually belong to that class (the false part of false positives). In practice, given your constraints, you may want to maximize the precision, while maintaining a good recall.

In this answer to the question How can the model be tuned to improve precision, when precision is much more important than recall?, the user suggests performing a grid search (using sklearn.grid_search.GridSearchCV(clf, param_grid, scoring="precision")) to find the parameters of the model that maximize the precision. See also the question Classifier with adjustable precision vs recall.


What is the relationship between these two taxonomies for machine learning with neural networks?

Could you please let me know which of the following classification of Neural Network's learning algorithm is correct?

  The first one classifies it into:

  supervised,
  unsupervised and
  reinforcement learning.

Those three forms of Machine Learning are not really different forms of a Neural Network learning algorithm (or, really, any learning algorithm); they are different forms of learning problems. They basically describe how much information we give our learning algorithm during a learning process / what kind of information we give it, regardless of what algorithm we're using to learn.

In supervised learning, we give very detailed information; we give example pairs of inputs + desired outputs.
In unsupervised learning, we still give many example inputs, but don't give any desired outputs. This can be, for example, because we simply don't know ourselves what the desired outputs would be. The most common example of unsupervised learning is learning to create clusters; we'd typically give some measure of similarity or distance between instances, and ask a learning algorithm to create clusters of instances for us such that instances within the same cluster are "close" to each other, and instances in different clusters are far away from each other. This is different from supervised learning because we do not directly tell the learning algorithm in which cluster each example instance would belong.
In reinforcement learning, we typically try to learn about policies (~= behaviours) in environments where an agent can take actions. We typically do not exactly know what the best complete policy would be, but can occasionally give "hints" (reinforcement) in the form of numerical rewards. This is not completely supervised learning because we don't tell exactly what the optimal action in a state would be. You can imagine this as giving a cookie to a dog if he's been a good boy.

Now, there also are indeed Supervised Learning / Unsupervised Learning / Reinforcement Learning algorithms, but it's generally a property of the problem first and foremost; once you know what type of problem you're trying to solve, you'll look for a matching algorithm that can handle that problem.

  However, the second one provides a different taxonomy on page 34:

  learning with a teacher (error correction learning including incremental and batch training),
  learning without a teacher (reinforcement, competitive, and unsupervised - memory-based learning, and
  Boltzmann learning.

Honestly, this seems a bit all-over-the-place to me. Learning with a teacher vs. learning without a teacher can be viewed as supervised vs unsupervised learning above. I suppose Reinforcement Learning would be kind of in-between.

What they describe as memory-based learning is also often referred to as instance-based learning. This is suddenly not anymore about properties of learning problems, this is a type of learning algorithm. I'm not aware of instance-based learning being common in Neural Networks at all, indeed the example given in your link (the most common example) is the k-nearest neighbours algorithm, which doesn't really have any relation with Neural Networks. This is normally used for Supervised Learning problems.

Boltzmann learning is a particular kind of learning algorithms for specific types of Neural Networks (with a specific architecture), and generally associated with unsupervised learning (or "generative" learning, learning probability distributions for given input data).


How is the convolution layer is usually implemented in practice?

I don't think that to understand convolution you need to dig into the nested code of huge libraries, since the code becomes quickly really hard to understand and convoluted (ba dum tsss!). Joking apart, in PyTorch Conv2d is a layer that applies another low level function, conv2d, written in c++.

Luckily enough, the guys from PyTorch wrote the general idea of how convolution is implemented in the documentation:

From this paragraph, we already have some important information like the input and output dimensions. The number of channels should be easy to understand, if we have an RGB image, for example, the channels are 3, one for each color, so they are just different matrices representing different features.

The next important element is the reference to the cross correlation, the function applied to our input images through the kernel k. Why cross-correlation? Because it is almost identical to a convolution, as you can see comparing their formulas:

The only difference consists in the way the indexing of the width is implemented, which causes the operation to start from the bottom right of the input matrix for the convolution and from the top left of the input image from the cross-correlation (the circles in the squares in the previous pic). Since, in most programming languages, matrix indexing starts from top left, cross-correlation is the most common choice to implement.

But how do these formulas work in practice? Here's another picture taken from Chapter 9 of Deep Learning (Goodfellow, Bengio, Courville), which I strongly suggest you to read.

Basically, from the input matrix, a submatrix is extracted, with the same dimension of the kernel, then the sub-matrix and the kernel are multiplied elementwise and all the resulting product summed together to produce a single output element that will form a 'pixel' of the resulting feature map (the output matrix).

Here's another example with fake numbers that I made. I hope the double notation for filter/kernel doesn't generate confusion, I actually found that sometimes it is inconsistent (in the chapter I linked they don't even use the filter at all). In practice they actually mean the same thing, I usually call kernel the actual matrix that is multiplied to the input and with filter I refer to the sliding window on the input image (that has, of course, must have the same dimension of the kernel).

Lastly, when you apply padding, the filter can actually move also outside the 'edges' of the input matrix, in which case all the elements outside would be considered to be zero. The computation is exactly the same, but since there are more splitting steps, the output matrix will have bigger dimension.

Please note that with multiple input channels you can perform either 2d convolution or 3d convolution, the difference relies in the filter dimension, in case of 2d convolution it would be a square whereas in 3d convolution it would be a cube. This means that for an RGB image a 2d convolution would treat each color layer independently, mixing the information from each channel with further computations like pooling (averaging the resulting feature maps of each color or select the max value among the feature maps for each pixel, etc..) while 3d convolution would mix together the color layers already during convolution thanks to the 3d kernel which will sum together elements from different layers.


What is a simplified way to explain why the AI researchers Bengio, Hinton, and Lecun, won the 2019 Turing Award?

The related ACM article describes a few specific technical contributions, which led the ACM to award them.

Geoffrey Hinton
Backpropagation: In a 1986 paper, "Learning Internal Representations by Error Propagation", co-authored with David Rumelhart and Ronald Williams, Hinton demonstrated that the backpropagation algorithm allowed neural nets to discover their own internal representations of data, making it possible to use neural nets to solve problems that had previously been thought to be beyond their reach. The backpropagation algorithm is standard in most neural networks today.
Boltzmann Machines: In 1983, with Terrence Sejnowski, Hinton invented Boltzmann Machines, one of the first neural networks capable of learning internal representations in neurons that were not part of the input or output.
Improvements to convolutional neural networks: In 2012, with his students, Alex Krizhevsky and Ilya Sutskever, Hinton improved convolutional neural networks using rectified linear neurons and dropout regularization. In the prominent ImageNet competition, Hinton and his students almost halved the error rate for object recognition and reshaped the computer vision field.
Yoshua Bengio
Probabilistic models of sequences: In the 1990s, Bengio combined neural networks with probabilistic models of sequences, such as hidden Markov models. These ideas were incorporated into a system used by AT&T/NCR for reading handwritten checks, were considered a pinnacle of neural network research in the 1990s, and modern deep learning speech recognition systems are extending these concepts.
High-dimensional word embeddings and attention: In 2000, Bengio authored the landmark paper, "A Neural Probabilistic Language Model", that introduced high-dimension word embeddings as a representation of word meaning. Bengio's insights had a huge and lasting impact on natural language processing tasks including language translation, question answering, and visual question answering. His group also introduced a form of attention mechanism which led to breakthroughs in machine translation and form a key component of sequential processing with deep learning.
Generative adversarial networks: Since 2010, Bengio's papers on generative deep learning, in particular the Generative Adversarial Networks (GANs) developed with Ian Goodfellow, have spawned a revolution in computer vision and computer graphics. In one fascinating application of this work, computers can actually create original images, reminiscent of the creativity that is considered a hallmark of human intelligence.
Yann LeCun
Convolutional neural networks: In the 1980s, LeCun developed convolutional neural networks, a foundational principle in the field, which, among other advantages, have been essential in making deep learning more efficient. In the late 1980s, while working at the University of Toronto and Bell Labs, LeCun was the first to train a convolutional neural network system on images of handwritten digits. Today, convolutional neural networks are an industry standard in computer vision, as well as in speech recognition, speech synthesis, image synthesis, and natural language processing. They are used in a wide variety of applications, including autonomous driving, medical image analysis, voice-activated assistants, and information filtering.
Improving backpropagation algorithms: LeCun proposed an early version of the backpropagation algorithm (backprop), and gave a clean derivation of it based on variational principles. His work to speed up backpropagation algorithms included describing two simple methods to accelerate learning time.
Broadening the vision of neural networks: LeCun is also credited with developing a broader vision for neural networks as a computational model for a wide range of tasks, introducing in early work a number of concepts now fundamental in AI. For example, in the context of recognizing images, he studied how hierarchical feature representation can be learned in neural networks - a concept that is now routinely used in many recognition tasks. Together with Léon Bottou, he proposed the idea, used in every modern deep learning software, that learning systems can be built as complex networks of modules where backpropagation is performed through automatic differentiation. They also proposed deep learning architectures that can manipulate structured data, such as graphs.


Does the "lowest layer" refer to the first or last layer of the neural network?

Lowest layer generally refers to the layer closest to the input. This comes from the idea that layers closer to the input represent low-level features such as gradients and edges, while layers closer to the output represent high-level features such as parts and objects.


Building Profile Based Off Speech Patterns

Yes, there is. An extremely quick search found this:
Multimodal Speaker Identification Based on Text_and_Speech.

Let me tl;dr for you: (My abstract addition in Italics)

Novel method for speaker identification based on both speech utterances and their transcribed text.

They first transcribed text of each speaker’s is processed by using probabilistic latent semantic indexing (PLSI) that models each speaker’s vocabulary which is closely related to his/her identity, function, or expertise.

The speech to text used by users is DARPA's Efficient, Affordable, Reusable Speech-to-Text (EARS) Program in MetadataExtraction (MDE).

By using Melfrequency cepstral coefficients (MFCCs) and dynamic range is quantized to a number of predefined bins in order to compute MFCC local histograms for each speech utterance, which is time-aligned with the transcribed text.

To test they used RT-03 MDE Training Data Text and Annotations corpus distributed by the Linguistic Data Consortium.

As for results: Identification rate versus Probe ID when 44 speakers are employed. Average identification rates for (a) PLSI: 69%; (b) MFCCs: 66%; (c) Both: 67%.

If you need more papers related, you could use a tool like https://the.iris.ai/ to find related papers.

Post edit: Hopefully now this post complies with the standards.


Would this neural network have short term memory?

Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field.
In this layer, you do aggregate $6$ values to one. Imagine two of them - it will be $36$ already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.


I think that by strict definition of the word inevitable, no, future AI Winter events are not inevitable. However likely or unlikely it may be, it is possible to control research spending and to create a more stable plan of funding research in Artificial Intelligence. Because it is possible to avoid an AI Winter event, an event is not inevitable.


Why is dialogue a hard problem in natural language processing?

Dialogue is a hard problem because it requires pretty advanced cognitive functions. Leaving aside all the lower levels of language analysis (phonology if dealing with speech, morphology and syntax), you quickly run into interpretation problems that require a lot of world knowledge.

Simple question and answer is fine, and restricted domains are somewhat easier as well. As soon as you get into a normal conversation, you will refer back to things you said before, so an NLP system would have to recognise that and resolve the reference accordingly. Typically in a conversation you would use variations of reference terms: the first time you mention an object you might describe it fully, but subsequently you will use shorter terms to refer to it.

There is also a structure to conversation. This is typically modelled as conversational moves, and usually moves will have corresponding response-moves. For example, a common sequence would be greeting - greeting. Then you might have question - response - feedback. This sounds fairly easy, but once you try to annotate a dialogue with such moves you will find that it is pretty hard. As far as I am aware, there is no 'grammar' equivalent of describing the structure of conversations.

Often, pragmatic meaning is interfering with the 'surface' meaning of utterances. A statement can actually function as a question, or a question can be a command (or a statement). The pragmatics of utterances depend on the context and also the relationship between the interlocutors. If I talk to my manager, I will use language differently than when I talk to my children.

Dialogue/conversations are hard to analyse. Because of that, descriptive frameworks are still fairly limited. You need to keep track of what has said before, as that can change the way an utterance has to be interpreted. Grammatical analysis is a fairly solved problem, and word sense disambiguation as well. But pragmatics and conversational structure are still on the bleeding edge of linguistic research; at least they were when I was still teaching Discourse Analysis at university a few years ago.

For that reason, chatbots are generally not very good. Sometimes they can fool people into believing they are human speakers, but this is usually done through trickery ("smoke and mirrors") rather than competent handling of conversational structures. It's all in the little box in nbro's answer labelled "DM"...


How does Wit.ai convert sentences into structured data?

I can't speak to wit.ai specifically, but I can tell you a little bit about how similar applications work. Specifically, I can talk a bit about Apache Stanbol which also converts free text into structured data.   That said, I should prefix this by saying there isn't just one way to "get there from here."  Many techniques could be part of a stack for accomplishing this goal.

Anyway, in the case of Stanbol, they run the text through multiple processing engines, sequentially, with different engines affecting the final output.  One engine simply does Named Entity Recognition using OpenNLP.  This identities discrete named "things" - people, places, companies, etc.  Another engine does entity matching with a pre-established database of entities - specifically (in the out-of-the-box configuration) a dump of entities from DBPedia.  Where a match is found, the text from the original input is assigned to the entity.  In the case of a collision, it assigns a weight to the mapping so any downstream consumers can use probabilistic techniques to select the "correct" mapping.

There are, of course, more details that I left out.  Before NER can happen there is parsing and tokenizing and other NLP activities.  But a big part of the basic process is doing NER and then doing the entity matching.

And in the case of Stanbol, you can add your own entities and corresponding structured data, as well as your own engines.  So, for example, if you wanted to write an engine based on neural networks / deep learning, and plug that in, you could.


Multi-label Classification with non-binary outputs

Technically, you're not really doing classification The outputs you have are not labels, they are real values. So I've got two (possible!) solutions, and you'll have to test them out for yourself:

Solution 1

Because each output is actually a numerical value, you can normalize them just like you would normalize all other numerical values. So you choose a maximum value that you would expect for the labels (you can also do this per label), and divide the actual value by that value to normalize it.

So if you don't expect more than 10 of the same amino acids in each peptide sequence, then you divide all amounts by 10:

0 becomes 0.0
1 becomes 0.1
2 becomes 0.2
3 becomes 0.3
etc.

Solution 2

This is more complex, but if you'd give it a try it would surely work. This solution requires recurrent networks. Just like LSTM networks are good for character-by-character text prediction, they will work for your 'peptide sequence language' as well.

This solution also allows an output sequence of any length.

How it works? You have a network with óne output (m/z), and 22 outputs (one-hot encoded amino acids) + 1 more, when this output has the highest value it tells you that the sequence has finished.

You keep on inputting the same value of m/z, until your network tells that the sequence has finished. The network will output different letters every activation based on the previous outputs (the LSTM model has some kind of memory).

This solution is kind of hard to explain, but if you provide me with a small list of peptide sequences (not too long, and not too much), then i'll make a working online example of you.

As promised, I made simple implementation of solution 2. However, I think you should go for solution 1 anyways. I didn't have any real data, so my model is basically overfitting, but it basically works: https://jsfiddle.net/ovpkL2xx/2/ (might take some time to run)

Made with Neataptic


How does a PDDL solver find a solution for a given problem?

The question doesn't really make sense: PDDL is a description language that is used to formulate a problem. This description then is the input to a planner; how the planner arrives at the intended solution is not related to the PDDL description.

There are a number of planning algorithms, and you can implement any of them to make use of a PDDL description. The output of the solver is a plan, which is usually an ordered sequence of actions, and a tree or graph structure might be a good way of capturing this.


Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?

The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs).
In Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Silver et al. say that the training process involved 5,000 first-generation TPUs generating self-play games and 64 second-generation TPUs for training. This is certainly far beyond what any practical gaming computer is likely to achieve, as you'll likely only have one GPU, and that might not even rival a single TPU. Training on the CPU will be substantially slower again than either a GPU or TPU. Training would be orders of magnitude slower; you might find these benchmarks by Wang et al. of interest.


What is the difference between vanilla policy gradient and advantage actor-critic?

The difference between Vanilla Policy Gradient (VPG) with a baseline as Value function and Advantage Actor Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:

The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.

The value estimates used in updates for A2C are based on temporal difference bootstrapped from e.g. a single step difference, and the Bellman function.

This leads to following practical differences:

A2C can learn during an episode which can lead to faster refinements in policy than with VPG.

A2C can learn in continuing environments, whilst VPG cannot.

A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allow for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.


In language theory, there are generally several admitted levels that can be studied in relation with one another or independently. The semantic level is the one dealing with the meaning of the text ("semantic" comes from the greek and means "to signify"). The semantic level is therefore generally independent from the syntax and even the language used to convey the message.
Here is an interesting picture I found on the internet to illustrate my point.

EDIT: I took some time reading the paper. I think "semantic levels" refers here to the different neural networks layers used for the exogenous features.

Here is a modified version of their figure I've drawned to make it clearer:

In particular, from what I have understood so far, the attention coefficients apply to the whole semantic level (which I find not really clearly indicated on their figure)

The LHS of their figure would then be better described by this new one:

Hope this helps!


Your probability hasn't been normalized!

In this case, you are computing the probability of being good, given that the other features have a fixed value. To obtain the correct probability, you need to normalize (divide) the value from your calculation by the probability that the features have taken on those fixed values.

You can calculate this as follows:
 $$P(Fem:dv/m/s, Young, own, Ex-credpaid) = \\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid, x) $$

by the marginalization rule.

Then, by the chain rule, you may write:

$$\\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x) $$

So the correct probability of 0.62 should be obtained by the equation:

$$ \frac{P(Fem:dv/m/s, Young, own, Ex-credpaid | good) * P(good)}{\sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x)}$$

You just need to calculate

$$P(Fem:dv/m/s, Young, own, Ex-credpaid | bad) * P(bad)$$

and it should be easy to compute the rest.


Trying to understand VGG convolution neural networks architecture

Good questions. Let me reply one by one.

1- Filter size can be increased. There is no limit for it. However, think two cases:

DNN part. Shape will be 1024 x 7 x 7 and it will map to 4096 features, it will cause 204M parameters at dense_1 layer. This change will cause two possible problem. Overfitting and inference/training speed.
Sparsity. You can implement 1024 features at 5th conv block and train the network and check the accuracy to see if increased feature count will make any impact or not.

So, to decide the number of con layers, it is a good practice to check sparsity of feature layers.

2- If I did not understand your question wrong, those layers are DNN classifier as a fully connected layer.

3- Maxpooling is not causing us to see a subset of the image. It is just an algorithm to resizing. What you see in that video is the filter response to some objects exist in the image. In deeper level, do not expect to have visual shape like in shallow layers. You may even get single white pixel at deepest layer and it may be a response to an object like human face.


Is it possible for a neural network to be used to compress data?

The auto-encoder (AE) can be used to learn a compressed representation (a vectorised hash value) of each observation in the training dataset, $z$, which can then be used to later retrieve the original (or similar) observation.

The variational auto-encoder (VAE), a statistical variation of AE, can also be used to generate objects similar to the observations (or inputs) in the training set.

There are other data compressor models, for example, Helmholtz machine, which precedes both the AE and VAE.


Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem

So, naturally, if you've observed something that contradicts the theoretical properties of Value Iteration, something's wrong, right?
Well, the code you've linked, as it is, is fine. It works as intended when all the values are initialized to zero. HOWEVER, my guess is that you're the one introducing an (admittedly very subtle) error. I think you're changing this:
state_value = np.zeros(GOAL + 1)
state_value[GOAL] = 1.0

for this:
state_value = np.ones(GOAL + 1)
state_value[GOAL] = 1.0

So, you see, this is wrong. And the reason why it's wrong is that both GOAL (which is 100 in the example) and 0 must have an immutable and fixed values, because they're terminal states, and their values are not subject to estimation. The value for GOAL is 1.0, as you can see in the original code. If you want initial values other than 0, then you must do this:
state_value = np.ones(GOAL + 1)
state_value[GOAL] = 1.0
state_value[0] = 0

In the first case (changing the initial values to 1) what you were seeing was, essentially, an "I don't care policy". Whatever you do, you'll end with a value of 1. In the second case, with the random values, you saw the classic effects of "garbage in, garbage out".


Strategic planning and multi dimensional knapsack problem

Here is a speculative cast of the problem to a travelling salesman problem, which would lead to shortest-path algorithms.

Please note this idea suggests different constraints to explore.

Given the knowledge vectors and efforts, build a acyclic directed graph (acyclic, as we are not supposed to unlearn). A vertex is an article, represented by its knowledge vector. An edge links two articles, weighted by the effort to "move" to the target article/vertex (i.e. acquire the knowledge of that article).
Assign a zero vector to a new participant. That is the starting point on the graph is vertex V0 = (0, ..., 0).
Define a learning objective as a vector V.
Use a shortest-path algorithm to find a (V0, V) plan.

This procedure is insufficient, as there are many ways to build the graph (in other words, the above is completely pointless as is). Extra constraints are necessary to make it practical. For example, we can order the vertices by ordering them along each dimension. Such setting would lead learners to start with "easy" articles (V[i] is low), and move step by step toward more complex topics ((V[i] gets higher).

The graph construction depends on the data available. For example, are knowledge vectors "absolute", or can they be relative? Relative can help in creating a path, as moving from V to W requires an effort that depends on your learner's initial conditions (V0 may not be 0 everywhere, afterall).

Is it an AI question? Definitely.


What do the values of the leaves of the decision tree represent?

Decision tree nodes are split bases on the number of data samples, these numbers indicate the number of data samples they are fit to.
In your case samples = 256. It is further split into two nodes of 154 and 102.


How to change the backward pass for an LSTM layer that outputs to another LSTM layer?

My understanding is that you have posted the last equation on p. 19 in the dissertation (please correct me if I'm wrong).

The derivation is indeed for a single LSTM layer as ek(t) is the error at the output (the assumed network topology is also mentioned just below Eq. (2.8) in the dissertation):

  Finally, assuming a layered network topology with a standard input layer, a hidden layer consisting of memory blocks, and a standard output layer[...]

Let's assume now that there are two stacked LSTM layers. In this case, the derivation in the dissertation applies to the last hidden LSTM layer (the one below the output layer), where Eqs. (3.10) -- (3.12) give you the partial derivatives for the weights at each gate for a cell in that layer. To derive the deltas for the hidden LSTM layer below, you have to compute the partial derivatives with respect to the portions of netcvj(t), netinj(t) and netfj(t) terms corresponding to the outputs of the preceding hidden LSTM layer, and then use those in the same way you used ek(t) for the current LSTM layer. Underneath all that unappealing notation is just the usual multi-layer backprop rule (well, with the truncated RTRL twist). If you stack more LSTM layers, just keep propagating the errors further down through the respective gates until you reach the input layer.

For a slightly more intuitive explanation, if you look at Fig. 2.1 in the dissertation, you can assume that in a multi-layered LSTM the IN in fact includes the OUT of the preceding LSTM layer.

Edit

There is a nice diagram of the flow of partial derivatives here (also see subsequent slides).

In this example, all xt terms represent the external input to that layer at time step t. In a multi-layer LSTM, this includes input from the LSTM layer below the current one. To propagate the error to the layer below, take the derivatives with respect to all gate weights corresponding to xt and apply the chain rule.

The reason why the derivation for the output gate weights is different is that the gating is applied after the cell state is updated, whereas the cell, input and forget gates are applied before that. This matters when computing the gradients.


What is the difference between machine learning and quantum machine learning?

The difference is much simpler than you might have anticipated: In the quantum computing community, machine learning algorithms designed to be used on quantum computers as opposed to classical computers, would fall under "quantum machine learning". There's really nothing more to it!
There is a short paper published in Nature called "Quantum Machine Learning" which was mentioned before, and it might give you all the answers you need about what "Quantum Machine Learning" is.


If we achieve sentience using mutable hardware, will it be possible to make a copy of that "brain" and its active state?

Theoretically, there shouldn't be a problem copying either of the artificial brains in any state.

Difficulty in measuring a state doesn't seem to really be a problem until you get down to the quantum level, where the means of measurement affect the state.

The configuration of the artificial brains, including pathway structures and states, should be reducible to a single string, which could then be used to reconfigure the artificial brain the information is being copied to.

Definitely look into the concept of a Turing Machine and Universal Turing Machine.