Is it possible to use AI to reverse engineer software?

Remarkably, more or less the scenario you describe is not only feasible and has already been demonstrated (detailed explanation and fascinating videos at link).

However, the fidelity of the copy is currently quite limited:

So for now, your copy will be quite low quality.  However, there is a big exception to this rule: if the software you are copying is itself based on machine learning, then you can probably make a high-quality copy quite cheaply and easy, as I and my co-authors explain in this short article.

Interesting question and I'm quite sure that the correct answer will change rapidly over the next few years.


Is unsupervised learning a branch of AI?

There is a problem with confining Artificial Intelligence to a single definition, because it has become an umbrella term encompassing many fields of science. It has come a long way from the "thinking machines" of the 50s. Actually, the term was coined in a summer workshop in 1956, whose proposal was:

  The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.

So even from its very first introduction the field of AI included learning. Personally, I've never seen a definition of AI, that doesn't include learning, of which unsupervised learning is clearly a part. So by any means clustering algorithms, which are unsupervised learning algorithms, can be considered AI. This can be confirmed in multiple sources, e.g. wikipedia.


Is there a notion of generalization in unsupervised learning?

In the paper Generalization in Unsupervised Learning (2015), Abou-Moustafa and Schuurmans develop an approach to assess the generalization of an unsupervised learning algorithm $A$ on a given dataset $S$ and how to compare the generalization ability of two unsupervised learning algorithms $A_1$ and $A_2$, for the same learning task.

They first provide a more abstract and general definition of an unsupervised learning algorithm and loss function. Then they define the expected risk, empirical risk and generalization gap in a similar way to the case of supervised learning. Finally, they derive an upper bound on $A$'s expected loss.

Of course, you should read the paper for more details. Specifically, section 2 (page 3) describes their setting in detail.


What tools are used to deal with adversarial examples problem?

However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?

I think it's difficult to tell whether or not there are any industrial strength defenses out there (which I assume would mean that they'd be reliable against all or most known methods of attacking). Adversarial Machine Learning is indeed a highly active, and growing, area of research. Not only are new approaches for defending being published quite regularly, but there is also active research into different approaches for "attacking". With new attack methods being discovered frequently, it's unlikely that anyone can already claim to have approaches that would work reliably against them all.

  The primary interest of this question, however, is whether any tools exist that can defend against some adversarial example attacks.

The closest thing to a ready-to-use "tool" that I've been able to find is IBM's Adversarial Robustness Toolbox, which appears to have various attack and defense methods implemented. It appears to be in active development, which is natural considering the area of research itself is also highly active. I've never tried using it, so I can't vouch personally for the extent to which it's easily usable as a tool for industry, or if it's maybe really only still suitable for research.

Based on comments by Ilya, other frameworks that may be useful to consider are Cleverhans and Foolbox.


In machine learning, the idea behind Hebbian learning is to strengthen (or weaken) the connection (the weight) between the neurons that have similar (or, respectively, dissimilar) outputs, where "similar" can be defined in different ways (e.g. it could be based on the sign of the output of the neurons).

Hebbian learning is a more biologically plausible way of learning than back-propagation, because it is a "local learning strategy" (you locally update the connections and not all the connections of the model at the same time), as opposed to back-propagation, which is a "global learning strategy" (where all connections are usually updated at once, given a "global error" of the network).

There are several neural networks (or models) that can learn in a Hebbian fashion: for example, the Hopfield network or Numenta's temporal memory.


Are biological neurons organized in consecutive layers as well?

Really short answer: yes

Slightly longer answer: kinda

Long answer:

Convolutional neural networks (CNNs), which are now a standard in image processing models, were inspired from work done by Hubel and Wiesel in the 1950-60s. They showed that the visual cortexes of cats and mokeys contain neurons which individually respond to small regions of the visual field.

To give some background, we have to first start from the rods and cones in eyes. These photosensitive cells are connected to a few layers of cells before even leaving the retina via ganglion cells.

These ganglion cells are then connected to several regions of the brain but primarily the Occipital lobe located at the back of the brain. The Occipital lobe is responsible for visual processing and is separated into cortical layers, the first named V1 which is the primary visual area. Most of the work by Hubel and Wiesel involved cells in V1 and showed how these cells were sensitive to orientation and color from their respective receptive areas on the retina.

The cells in V1 are connected to the cells in V2 which are sensitive to even more specific stimulus such as movement with orientation and this trend of specific sensitivity continues up from V2 to higher regions in the brain.

This layered approach to vision has been heavily exploited in CNNs, so much so that when the sensitivity of neurons in trained CNNs is displayed, similar responses (orientation) are found.

There is clear evidence of layers in biological optical systems and similarly layered structures in the other senses. Although there are many connections between different brain structures, the main structure of layers in the brain has helped understand what different areas of the brain do and has helped inspire many (if not all) advances in neural network research.


How could I use reinforcement learning to solve a chess-like board game?

I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic but I am still quite confused.

Be warned: Reinforcement learning is a large complex subject. Although it might take you on a detour from game-playing bots, you may want to study RL basics. A good place to start is Sutton & Barto Reinforcement Learning: An Introduction

  What other reward is there in a game a part the win-or-lose output (1 or 0)?

Depending on you game, that is usually it. Actually for a win/draw/lose game like chess then the reward from each action is 0 except for winning (+1) or losing (-1) at the end. In a zero-sum game then this aligns nicely to minimax, alphabeta pruning etc.

Reinforcement Learning is intended to address environments with delayed rewards. Adding "helper" rewards for interim non-goals is usually counter-productive.

  If I use other rewards, like the output from the evaluation function at each turn, how can I implement it?

Typically you don't. What applying self-playing RL will do is learn a return (sometimes called utility) function which predicts the expectation of your total +1/0/-1 reward by the end of the game. You would use this in place of your current heuristic for minimax search. Or, potentially you would adjust your current heuristic function to output in the same range and use RL to optimise its weights to make the best approximation to the true optimal play return function (which is probably too complex to calculate exactly).

  How do I modify the evaluation function to give better rewards iteration after iteration?

That is what the different RL approaches all attempt to do, there are a variety of different solvers. There is no short way to explain it. You could start with a simple method such as Q-Learning. Q-Learning learns estimates of Q(s,a) (called the action value) which is the expected return when in state s and taking action a, and thereafter following an optimal policy. It makes an arbitrary guess to start with and refines it closer to the true value with each step made in the learning environment. Simple tabular Q-learners do this refinement simply by storing a big table of all states and actions with the best estimate so far of the true value, and averaging in each new estimate as it is experienced.

It is also possible to combine an RL method for heuristics with look-ahead minimax search - that is what the original AlphaGo did, and what AlphaGo Zero does during training. This is a powerful approach because minimax search will work to double-check the RL-generated heuristics. Although for simple enough games, RL can learn perfect heuristics and you would only need local search (what the next move should be).

Unless your game is very simple (all possible states would fit in memory), you will need some kind of function approximator inside the RL algorithm. Neural networks are a standard choice. Having something for that part is unavoidable - although another good choice is to define a bunch of proxy features (that you might use to construct a heuristic by hand) and use a linear approximator - just a weighted sum of all the features. This can work well enough, and has been used for example in checkers (draughts) players trained using RL.

In fact, provided your own heuristic function is not too unusual, you can probably treat it just like a linear approximator and use RL to learn the best weights for it.


What's the difference between poker with public cards and without them?

It depends a little on what you mean by "the same rounds, just with no public cards dealt."

If you mean that each player will just be dealt 2 cards, and no public cards exist, then really we're playing a sort of "high card" game. The best hand is just a pair of aces, CFR will solve this quickly, because the number of possible game states is extremely small compared to a full poker game (especially if we exploit the symmetry of suits, since flushes aren't possible).

If you mean that each player will be dealt 5 cards, with several rounds of betting as before, CFR will probably do less well. The state space will be larger, since there are more cards in play (10 instead of 9). Betting may become more complex, and more complex betting expands the state space enormously.

If you mean that the cards are dealt as before, but the program simply will not look at the cards, then you've kept the state space of the game the same size, but radically reduced the number of information sets. Playing against an opponent who can look at the cards on the table, your program would be at an enormous disadvantage. For instance, imagine the cards on the table are "3 3 8 8 5", and that you have a pair of 2's in your hand. You would want to play very differently from if the cards on the table were "2 2 10 4 7", but an AI without access to the table cards would have to act the same in both situations.


Matrix Dimension for Linear regression coefficients

I have used W as beta and Y_pred as Y_hat.

Apparently, as far as conventions go both feature vector X and weight vector W are assumed as column vectors. Although this is not important in your case but when we use Neural Nets, this is particularly important, and a weight vector for a layer is given by p*N_n where p is the number of features being input from the previous layer and N_n is the number of nodes in the layer. Check a NN structure and you will understand what I am saying.

As far as the special case in your current question, the book is correct, although comments say Y_predicted should be vector, it is not, since we are doing dot product between X and W (although it is represented in a vectorised form).

Also  how can W  have dimensions p*K (assuming it is not a NN), it means multiple solutions to the same problem? Instead it is X which will have dimensions p*K and W will have p*1. It means you have K training examples with p features.

If its a NN with K nodes then dimensions for X like p*N and W as p*K is totally possible. The only downside of this notation is that you have to again take a transpose of the result.

Here is the explanation of the notation used:

Notation for NN


What techniques are used to make MDP discrete state space manageable?

tl:dr
Read chapter 9 of an Introduction of Reinforcement Learning

There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem to a much harder case, continuous variables, and see how we deal with it.

Mood: range [-1, 1]   // 1 is Happy, 0 is Neutral, -1 is Sad

Hunger: range [0, 1]  // 0 is Very Hungry, 1 is Full

Food: range [0, 100] // amount of food, capping at 100 for numbers sake

Time of day: [0, 23] // Hours, 9.5 would be 9:30 am

Position x: range [-10, 10] // assuming the area the agent stays in is 10x10 km

Position y: range [-10, 10]

Money: range [0, 2000]

Now its impossible to even count the number of states the agent could be in. An example of one state would be:

Mood, Hunger, Food, Time, PosX, PosY, Money
.5, .3, 67.4, 4.5, 0, 0, 5

This would mean that our agent has a neutral mood, is kind of hungry, has 67.4% food, the time is 4:30 am, they are in the center of the city (0,0), and have 5 dollars. Which has happened to me once or twice before too so this is reasonable. If we we still wanted to generate a transition matrix and a reward matrix we would have to represent this state uniquely from other possibly similar states such as if the time was 5 pm with all other things equal.

So now how would we approach this?
First lets discretize or breakup each range into little chunks. Then we could have an array of 0s representing our chunks and assign a 1 to the index of the chunk we are in. Lets call this array our feature array.
If we did this for each dimension of our state information, we would end up with some state similar to what you originally proposed. I've chosen some arbitrary numbers to break up the state space and lets see what it does to the example state.

If we broke the Mood range up into 8 chunks of .25 we would have these ranges:

m = Mood

-1.0 ≤ m < -.75
-.75 ≤ m < -.50
-.50 ≤ m < -.25
-.25 ≤ m < 0.0
0.0 ≤ m < .25
.25 ≤ m < .50
.50 ≤ m < .75
.75 ≤ m < 1.0

If we took the mood from the example state (.5), it would land in the 7th range, so then our feature vector for mood would look like:

[0,0,0,0,0,0,1,0] <- represents a mood of .5

Lets do the same for hunger by splitting it into 8 chunks of .125:

[0,0,1,0,0,0,0,0] <- represents a hunger of .3

We could then do something similar for each other state variable and break each state variable's range up into 8 chunks of equal sizes as well. Then after calculating their feature vectors, we would concatenate them all together and have a feature vector that is 56 elements long (7 state variables * 8 chunks each).

Although this would give us 2^56 different combinations of feature vectors, it actually doesn't represent what we wanted to represent. We need features that activate (are 1) when events happen together. We need a feature that is 1 when we are hungry AND when we have food. We could do this by doing a cross product of our 56 element feature vector with itself.

Now we have a 3136 element feature vector that has features like:
  1 if its 3pm and am happy
  1 if am full and out of food
  1 if at coordinate -3, 4 (Position x, Position y)

This is a step in the right direction but still isn't enough to represent or original example state. So lets keep going with the cross products!

If we do 6 cross products we would have 30,840,979,456 features and only 1 of them would be on when our agent has a neutral mood, is kind of hungry, has 67.4% food, the time is 4:30 am, they are in the center of the city (0,0), and have 5 dollars.

At this point you are probably like "Well.. thats a bit much", to which I would agree. This would be the curse of dimensionality, and is the reason transition tables stop being fun (if they ever were).

Instead lets take a different approach, rather than trying to represent an individual state with an individual feature and saying whether that feature is good, lets go back to our nice 56 element feature vector. From here lets instead give a weight (w_i) to each of our 56 features. We have now entered the world of linear function approximation. Although I could explain it here, I think its better explained in Chapter 9 of the Introduction of Reinforcement Learning.


Is it possible to teach an AI to edit video content?

I believe the answer is yes, but the video edition is mostly programmatic. The AI part comes with detecting the right spots to cut.

You want to detect the right portion when someone picks a wooden skittle
you want to detect when the Skittles stop moving on the ground (to detect this, also when they start moving)

These will give you timestamps on the video. The next step is to add some padding and run the drop-down frame operations in the video, which can all be done with a simple script.

You might find this video very similar in concept, and maybe even as code source: https://youtu.be/DQ8orIurGxw

For the recognition of the objects in the video frames, there are lots of options. You might look into computer vision or object motion detection.


How can I make the kernels non-learnable and set them manually?

In most modern neural network frameworks, the update rules for training can be selectively applied to some parameters and not others.

How to do that is dependent on the framework. Some will have the concept of "freezing" a layer, preventing parameters in it being updated. Keras does this for example. Others will do the opposite and expect you to provide a list of trainable parameters - these typically come with helpers that will list all parameters in a neural network, so you would need to add some kind of filter after collecting that data to exclude your pre-trained layer. PyTorch does this (although the linked example is slightly more complex in that it stops calculating gradients too).

If your framework of choice does not allow you to select and isolate layers in the training process, then you still have a couple of options:

You could store a copy of layer parameters that you want to keep and force your learning network to re-load these parameters after each mini-batch. This does depend on you having a method that can selectively set parameters.
If your pre-trained layers are the first ones, immediately next to the input, then instead of including them in your learning network model, you can pre-process all your training data with just the fixed layers (build a model using only those layers), save the output and use that as an alternative input for the learning layers (build a second model with only the learning layers). Later, once training is complete, you can build a combined neural network out of the fixed layers and the learning layers.


Can reinforcement learning be used for tasks where only one final reward is received?

RL can be used for cases where you have sparse rewards (i.e. at almost every step all rewards are zero), but, in such a setting, the experience the agent receives during the trajectory does not provide much information regarding the quality of the actions.
Games can be often formulated as episodic tasks. For example, you could formulate a chess match as an episode and you could give a (non-zero) reward only at the end of the match. However, in this specific case, it will be hard for the RL to understand which moves have mainly contributed to the reward received, which is known as the credit assignment problem.
You can solve the issue of sparse rewards with reward shaping (in particular, potential-reward shaping).
The term "delayed rewards" may also refer to the cases where you receive only one reward at the end of the episode, although it may more usually refer to scenarios where the reward at one-time step is only received later (for some reason).


Monte Carlo updates on policy gradient with no terminal state

Naturally, one might let the MDP run for 1000 periods, and then terminate as an approximation. If we feed these trajectories into a monte carlo update, I imagine that samples for time period t=1,2,...,100 would give very good estimates for the value function due to the discount factor. However, the time periods 997, 998, 999, 1000, we'd have an expected value for those trajectories far different than V(s) due to their proximity to the cutoff of 1000.

I think your intuition is... partially right here, but not entirely precise. Recall that a value function $V(S)$ is generally defined as something like (omitting some unimportant details like specifying the policy):

$$V(S_i) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t R_{i+t} \right].$$

The "samples for time period $t = 1, 2, \dots, 100$ that you mention are not estimates for this full value function. They're estimates for the corresponding individual terms $R_{i+t}$. Indeed, in general, your intuition is right that the closer they are to the "starting point" $i$, the more likely they'll be to be accurate estimators. This is because larger $t$ are typically associated with larger numbers of stochastic state-transitions and stochastic decision-making, and therefore often exhibit higher variance.

  Should we even include these later-occurring data points when we update our function approximation?

Theoretically, you absolutely should. Suppose you have an environment where almost every reward is equal to $0$, and only after 1000 steps do you actually observe a non-zero reward. If you don't include this, you'll learn nothing! In practice, it can often be a good idea to give them less importance though. This already happens automatically by picking a discount factor $\gamma < 1$.

  Is it usually implied that the final data reward in the trajectory is bootstrapped in these cases (i.e., we have some TD(0)-like behavior in this case)?

  OR

  Are monte carlo updates for policy gradient algorithms even appropriate for non-terminating MDPs due to this issue?

It would be possible to do some form of bootstrapping at the end yeah, cut off and then have a trained value function predicting what the remainder of the rewards would be. TD($\lambda$) with $\lambda$ close to $1$ would be much closer in behaviour to true MC updates than TD($0$) though. Either way, it would be technically incorrect to still call it Monte-Carlo then, it would no longer be pure Monte-Carlo. So yes, strict Monte-Carlo updates in the purest sense of the term are not really applicable to infinite episodes.


What is the time complexity of the value iteration algorithm?

The update equation for value iteration that you show is time complexity $O(|\mathcal{S}\times\mathcal{A}|)$ for each update to a single $V(s)$ estimate, because it iterates over all actions to perform $\text{max}_a$ and over all next states for $\sum_{s'}$.

The sources you have found are probably counting an entire sweep through the state space as an "iteration" i.e. $\forall \space s \in \mathcal{S}: V_{k+1}(s) \leftarrow \text{max}_a \sum_{s'} T...$ That adds another factor of $|\mathcal{S}|$ making the overall complexity $O(|\mathcal{S}\times\mathcal{S}\times\mathcal{A}|)$ or $O(|\mathcal{S}|^2|\mathcal{A}|)$

This definition of iteration makes sense, as the basic value iteration algorithm is required to sweep through the whole state space in order to converge. This also matches the standard test for convergence, which is made after each full sweep, and checks what the largest absolute update was at the end of the sweep - if it is below some target value for accuracy then the process is declared complete.


How does dimensionality reduction occur in Self organizing Map (SOM)?

SOM (Self-Organinizing Map) is a type of artificial neural network (ANN), introduced by the Finnish professor Teuvo Kohonen in the 1980s, that is trained using unsupervised learning to produce a low-dimensional, discretized representation of the input space of the training samples, called a map, and is therefore a method to do dimensionality reduction.
SOM produces a mapping from a multidimensional input space onto a lattice of clusters, i.e. neurons, in a way that preserves their topology, so that
neighboring neurons respond to “similar” input patterns.
It uses three basic processes:

Competition
Cooperation
Adaptation

In competition, each neuron is assigned a weight vector with the same
dimensionality d as the input space. Any given input pattern is compared to the weight vector of each neuron and the closest neuron is declared the winner.
In cooperation, the activation of the winning neuron is spread to neurons in its immediate neighborhood, and as a result this allows topologically close neurons to become sensitive to similar patterns. The size of the neighborhood is initially large, but shrinks over time, where an initially large neighborhood promotes a topology-preserving mapping and smaller neighborhoods allows neurons to specialize in the latter stages of training.
In adaptation, the winner neuron and its topological neighbors are adapted to make their weight vectors more similar to the input pattern that caused the activation.
So, a Self-Organizing Map (SOM) is to encode a large set of input vectors $\textbf{x}$ by finding a smaller set of “representatives” or
“prototypes” or “code-book vectors” $\textbf{w}$ that provide a good approximation to the original input space. This is the basic idea of vector quantization theory, the motivation of which is dimensionality reduction or data compression. Performing a gradient descent style minimization on SOM's loss function (eg. the sum of the Euclidean distances between the input sample and each neuron) does lead to the SOM weight update algorithm, which confirms that it is generating the best possible discrete low dimensional approximation to the input space (at least assuming it does not get trapped in a local minimum of the error function).
To answer your question, you should take into consideration that the Dimensionality reduction takes place in fields that deal with large numbers of observations and/or large numbers of variables. Thus, SOM helps finding good "prototypes", in a way that each input pattern belongs to exactly one of them. As a result, the training instances are mapped to the training "prototypes" and the whole training set is mapped to a new one with less instances.
In addition, the "prototypes" neurons resulted by SOM can often be used as good centers in RBF networks or to classify patterns with the LVQ family algorithms.


Feasibility of sorting a basket of clothes in the real world

Peter Abbeel does work in deep learning for robotics, and one of the projects they've tackled is manipulating clothes. Here's a video from 2011 of their robot folding laundry (one piece at a time).

There are also companies attempting to market this; seven dreamers makes the Laundroid and Foldimate claims that it will start taking pre-orders in 2017.


There are different variants of LSTM, in most ML packages now of days you'll probably see what's shown in the bottom picture. For more details, intuition and motivations please see this paper.
It is not reshaped and there is no softmax layer. That is all done outside the LSTM.


What is the theoretical basis for the use of Cross Validation set?

I think Cross-Validation serves a completely different purpose.

From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model parameters after cross validation are closer to the parameters of the test data).

In fact, we use CV to get an estimate of generalization error while keeping our test set outside the training process. That is, we use it to answer the question "What is the size of the difference between my training and testing performance likely to be?". If you have an estimate of this that you are confident in, you can be confident that when you deploy a model to your customers, the model will actually work as you expect.

If you're only going to build a single model, then you don't need cross validation. You just train the model on the training data, and test it on the test data. Then you have an unbiased estimate of generalization error.

However, we might want to try out many different kinds of models, and many different parameters (broadly, we might want to do hyperparameter tuning). To do this, we need to understand how generalization error changes as we change our hyperparameters, and then use this information to pick hyperparameter values that we think will minimize the actual error when we deploy the model.

You could do this by training different models on the training set, and then testing them on the test set, recording the difference in model performance on the two sets. If you use this as a basis to pick a model though, you have effectively pulled the test set inside your training process (model parameters were implicitly selected using the test set, since you picked the parameters with the lowest test error). This bias will make your true generalization error much larger than what you observed.

As a stop gap, you could split your training set into a 'real training' set and a validation set. You could train models on the 'real training' set, and then measure their performance on the validation set. The difference would be a biased (but hopefully still useful) estimate of generalization error. You could then test against the test set just once (at the end) to get an unbiased estimate that you can use to decide whether or not to deploy the model.

A better workflow is to use CV on the training set to get an estimate of generalization error during hyperparameter optimization. You get K samples for k-fold cross-validation, so you can do statistical testing to see whether one model truly has better generalization error than another, or whether it's just a fluke. This decreases the degree of bias in your estimates of generalization error. Then, once you've completed hyperparameter optimization, you can run your final model against the test set once to obtain a truly unbiased estimate of your final generalization error.


How should I weight the factors that affect the choice of an action in a strategy board game with multiple actions?

You could use a genetic algorithm to optimise the parameter settings. Here you don't choose random parameters all the time, but only at the beginning. Each AI (which is a vector of parameter settings) plays each other one for a ranking (you can probably reduce the number of total games by using a ladder-style ranking where only neighbours play against each other). Also, you don't need 100 to start with, as you might come across better combinations than those present initially throughout the processing.

Then you discard the worst AIs (ie parameter vectors), and recombine the best ones, adding some random mutations into the mix. In theory this should converge faster, as you preserve good parameter settings (depending on how interdependent they are) and remove bad ones from the 'pool'.

You can also have a higher rate of mutations initially, which slowly goes down as you progress through the generations.


Compute Jacobian matrix of Deep learning model?

In the paper The Limitations of Deep Learning in Adversarial Settings, Papernot et. al., 2016, IEEE the chain rule is used, "To express $\nabla F(X∗)$ in terms of $X$ and constant values only."

Earlier is stated, "Our understanding of how changes made to inputs affect a DNN’s output stems from the evaluation of the forward derivative: a matrix we introduce and define as the Jacobian of the function learned by the DNN. The forward derivative is used to construct adversarial saliency maps indicating input features to include in perturbation $\partial X$ in order to produce adversarial samples inducing a certain behavior from the DNN;"

And later, "We define the forward derivative as the Jacobian matrix of the function F learned by the neural network during training. For this example, the output of F is one dimensional, the matrix is therefore reduced to a vector (below). Both components of this vector are computable using the adversary’s knowledge, and later we show how to compute this term efficiently."

$\nabla F(X) = \Big[ \frac {\partial F(X)} {\partial X_1}, \frac {\partial F(X)} {\partial X_2} \Big] \quad\quad (2)$

Between equation (2) and equation (6) your questeion is answered, resulting in an equation where the chain rule has already been applied.

$\frac {\partial F_j (X)} {\partial x_i} = \Big(W_{n+1,j} \, . \, \frac {\partial H_n} {\partial x_i} \Big) \times \frac {\partial f_{n+1,j}} {\partial x_i} \;(W_{n+1,j} \, . \, H_n + b_{n+1,j}) \quad\quad(6)$

It is (6) that you must implement in code, but that is done after the DNN is converged as the paper states above.

Applying the Chain Rule

If $z = f(y)$ and $y = g(x)$, then $\frac {dz} {dx} = f'(g(x)) g'(x)$

Applied twice, if $w = e(z)$, $z = f(y)$, and $y = g(x)$, then $\frac {dw} {dx} = e'(f(g(x)) f'(g(x)) g'(x)$

Applied indefinitely we multiply all the first order derivatives of intermediate values, which has the following first order predicate logic expression.

$\forall \; 0 \le n \lt N \land v_{n + 1} = f_{n + 1}(v_n) \land F_{n+1}(x) = f_{n+1}(F_n(x)) \land g(a) = \dfrac {d} {d a} \, f(a)$
$\exists \; g_n(v_0) = \prod_{\,n = 1}^{\,n < N} {g_n} (F_n(v_0))$

Derivatives of convolution kernel values are the values themselves because they represent attenuation of the form $y = ax + b$.  This is true of first degree linear activation functions too.  The derivative normally used for ReLU (although the x = 0 case is actually undefined) is as follows:

$\dfrac {d} {dx} R(x) = {\begin{cases}x < 0 & 0\\x = 0 & 1\\x > 0 & 1 \end{cases}}$

The derivative of a max function used in max pool (although the u = v case is actually undefined) is as follows:

$\dfrac {d} {dt} M(u, v) = {\begin{cases}u < v & \frac {dv} {dt}\\u = v & \frac {dv} {dt}\\u > v & \frac {du} {dt}\end{cases}}$


Why do we minimise the loss between the target Q values and 'local' Q values?

The loss function is designed in a way to approximate the bellman optimality for $Q^*(s,a)$. Given an optimal policy $\pi^*$, $Q^*(s,a)$ satisfies the equation $$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$
At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ and the max $Q$ values at the next state.
You can see in the loss function that the DQN tries to attain a $Q^*(s,a)$ value that closely approximates the equation above.
On a side note, in model based RL setting, the bellman optimality for $Q^*$ is expressed as $$Q^*(s,a) = r(s) + \gamma \sum_{s'}P(s'|s,a)V^*(s)$$ $V^*(s')$ is used in the model based case because $V^*(s)$ by definition represents the highest possible value attainable at state $s'$ following $\pi^*$. In model free setting, $V^*$ is replaced by $Q^*$ because computing $V^*$ is not useful in achieving model free control without a transition model $P(s'|s,a)$


How do I select the (number of) negative cases, if I'm given a set of positive cases?

Short answer
To select the proper dataset to construct, you should first figure out a metric to use to compare, and then select the dataset construction that gives the better metric. There is no single best metric, it depends on the task and your interpretation on what type of error is more important.
If you believe it is important that errors should not be normalized across class, then use the overall accuracy, and keep your dataset distribution same as the natural distribution (so 1-2% positive cases).
If you believe it is important that errors should be normalized across class, then use PR-AUC or ROC-AUC, and re-balance your dataset so that the samples a little more closer to 1:1. The exact ratio will only be determined after testing and comparing the PR-AUC or ROC-AUC metrics.
How to select the best metric?
Two popular metrics are ROC-AUC and PR-AUC. ROC curves (Receiver Operating Characteristics) plot the true positive rate vs false positive rate, while PR curves (Precision and Recall) plot the precision vs recall. AUC stands for "area under curve", because you can achieve any single point in the curve by specifying the classifier threshold, so the sum of all points (i.e. the entire area under the curve) is the most general way of comparing if one model is doing better than another.
Although both ROC curves and PR curves equalize class imbalance at some capacity, PR curves are more sensitive to class imbalance. The paper The Relationship Between Precision-Recall and ROC Curves concludes that if the PR-AUC is good then the ROC-AUC will always also be good, but not the other way around. The difference is due to the fact that if the dataset has huge class imbalance, a false positive hurts PR curves significantly more than ROC curves.
On the other hand, total accuracy does not normalize class imbalance at all, so therefore favors the majority class.
As a result:

if you do not care about normalizing the measures of class imbalance, choose total accuracy, which will optimize for the most # of correct cases (regardless of class)
if you want to normalize your metric across class imbalance, and normalizing false positive errors across classes is at all important to you, choose PR-AUC
if you want to normalize your metric across class imbalance, and don't care about normalizing false positive errors, PR-AUC or ROC-AUC may both be good for you

If it helps, for most imbalance problems, people usually go for PR curves.
By the way, (this paper) studies class imbalance in neural networks by optimizing the ROC curves, and show that you should definitely have equal numbers of positive and negative examples. So if you want the best performance in terms of ROC-AUC, you should do the 50:50 split. I haven't read any similar study that optimizes for PR-AUC, but my intuition tells me that it will have the same conclusion (you should do 50:50 split to optimize for PR-AUC as well).


AI to recognize keyword in a phrase within a context

Here is a glimpse for you.

I once contributed to an open source project via Github, and the open source artificial intelligence library/API which helped a lot and I think will as well help you out is wit, it provides location data.

And the good thing about "wit", is it provides wit_location entity which provides information on the captured location for instance "time-zone, locality, region / location"

For example: in sentence "How far is Russia?", it captures "far" as the distance to the target location entity before this returns

{"value":"location_name",Russia"}

In the entity value. (don't forget)

To further improve the quality and search engine racking to return values in future, that's if your project goes well, you can simply provide an optimal "coords" map in context when calling the "wit-API"

This will also help you to factor into your search engine racking of your returned values, which will as well take the location's proximity to the "coords" into  account.

Hope this can give you some insights.


Are hill climbing variations always optimal and complete?

No, they are prone to get stuck in local maxima, unless the whole search space is investigated.
A simple algorithm will only ever move upwards; if you imagine you're in a mountain range, this will not get you very far, as you will need to go down before going up higher. You can see that going down a bit will have a net benefit, but the search algorithm will not be able to see that.
Random restart (and similar variations) allow you to do that, up to a point. Imagine you have ten people that you parachute over your mountain range, but they can only go upwards. Now you've got a better chance of finding a higher peak, but there's still no guarantee that any of them will reach the highest one.


Can optimizing for immediate reward result in a policy maximizing the return?

If we train a policy to maximize the immediate reward, will it also perform well in the long term?

In general, no. The delay of long term reward in real world problems, and often a lack of easy-to-compute heuristics, is a key motivation for developing reinforcement learning in the first place.

It is easy to construct a counter-example to demonstrate this. Any state where the transitions into it are high and positive, but the transitions out of it are higher and negative would "trap" an agent that only considered immediate reward. More complex traps include high immediate gains but ending an episode vs lower gains that continue for longer.

Many real-world environments have sparse rewards where it is not possible to tell the difference between two action choices by immediate reward, but the consequences of being in one part of the state space rather than another early in a trajectory are critical. Consider any two-player strategy board game for instance, where the only goal is to win at the end. Only the last move in such a game is associated with an immediate reward, but there are often important differences between early moves.

  What properties would the reward function need to fulfill?

In all states, the expected immediate reward for taking the correct long term action would need to be higher than the expected immediate reward for any other action choice.

Solving a problem framed in this way could be done with discount factor $\gamma=0$. If the action choices were always the same and valid in each state, then the problem could also be simplified to a contextual bandit, where the fact that the choices exist within a larger trajectory is not relevant.

In practice you can construct environments like this. Simple ones are possible to do manually. Doing that is similiar to adding a heuristic function for search, but with different restrictions. For many search algorithms, admissible heuristic functions are allowed to over-estimate future gains (or under-estimate costs), because a planning/search algorithm will resolve longer-term differences. In your case, you can maybe consider stochastic reward functions, but the expected reward for the correct action must always be highest.

Needing to know the correct optimal action in the first place is clearly a circular problem - if you knew it already you would have no need to perform reinforcement learning to discover the optimal policy. An exception might be if you constructed an easy environment in order to test an algorithm, and prove that it could find the optimal policy. Although even then usually you are interested in the algorithm solving a harder variant of your problem than one you have deliberately constructed to be easy.

In brief, there is no way to create a shortcut here and avoid the need to solve a harder RL problem.


Does it make sense to train images (for object detection algorithms) with cameras that will not be used to collect future data?

I think this can only be used for pretraining/some kind of transfer learning. This would be useful if the ratio of real training data to pretraining data is really low. You could then pretrain on digital data, and fine-tune on your UAV data.
How much of this is really useful I can't really say, this depends on how close the digital data is to UAV data. If it is significantly different, you are training on a different distribution and sample space than your UAV images, which is pointless.


To model 2048 (or any problem) for search, you need a only a few pieces of information.

Note first though, that 2048 is not suitable for minimax, because there's only one player! Instead, you can treat this as a Markov decision process. The techniques to solve it are pretty similar though. Basically, you'll do search for one player, and insert "chance" nodes at each ply of the search. The value of a chance node is the expected value of its children. Note that this will reduce the effectiveness of pruning, so it might mean the problem is not tractable for search-based approaches.

What does an end state look like? Usually you have some function G(s) that accepts a state s, and produces true if and only if it's an end state. In 2048, end states would be states where the player loses (no moves possible), or where the player wins (a 131072 tile is present), so this should be fairly easy to write.
What are the payoffs? This is usually given by a utility function U(e) that accepts an end-state e, and produces a numeric value indicating the utility the player will receive.
What actions can the player take in each state? In 2048, these are always the same (up, down, left and right)
How are new states generated from old states and player actions (in this case, the tiles slide according to the rules of the game, and then a new tile is inserted at a random empty location.

Although search might work here, since 2048 is a relatively simple MDP, you might be happier using techniques from reinforcement learning, which were specifically designed for this kind of problem. Russell & Norvig have a good set of chapters on both approaches (14-17).


The issue is likely in how you estimate wellness, how the error function is constructed and from what data, since you have used two known good pieces of software and probably known good derivatives for your activation functions.  The second most likely is in now the components of wellness is aggregated.  Summing squares is sometimes not representative of a solid aggregation strategy.

I'm a bit confused about the game for three reasons.

Stars enter the description without telling us whether they are not obstacles, are obsticles, or the only obstacles
At one point bitcoins collection is the objective and obstacles are the challenge and at another point bitcoins are obstacles themselves
Your inputs have distance but not direction (Is this 2-D game?)

Based on textual hints, I'm going to assume five simple things, and you can correct any misconceptions I list.

Collecting all bitcoins is part of the objective
Running into bitcoins mid-jump is considered a crash
There is some radial tolerance to landing in at a bitcoin location with some imprecision
There are other things to crash into
Your outputs are jump magnitude and direction

I see that when the bitcoins are not in the vicinity of other things, the net can be trained effectively to jump to them, but when in the vicinity of other things, the training converges on a behavior that repeatedly fails prior to completion.  I'm assuming that the failure location is not 100% repeatable because the genetic algorithm has a pseudo-random seed that changes.  Again, correct me if I misunderstand in my piecing together the scenario.

One should consider the possibility that the distance to non-bitcoin obstacles is part of the error function and the difference between the jump destination and the bitcoin is also part of the error function.  (This second one is why I prefer to call the error contour a wellness measure.)

If the bitcoin incentive does not seem to be the crash cause, in that, had the other obstacle been moved, the bitcoin would have been collected, then the first of the two wellness criteria needs a higher order contribution from the distance to the other obstacle.

There are two simple functional forms that come to mind, which could be tried that increase the alarm represented in back propagation when the probability of collision is heightened to more effectively train against collision.  Both involve determining the direct jump line to the nearest bit coin and the distance from that line to its nearest other obstacle; call that x.

$x^y$, where $y > 1.0$
$e^{kx}$


Can an evolutionary algorithm adapt to a changing environment?

The core question to whether or not an AI is adaptable or not is whether or not it supports online learning. That doesn't mean using the Internet to learn things; that means continuing to accept training data during the functioning of the system.

This is (mostly) independent of the underlying architecture; in evolutionary approaches one can continue to breed generations with a shifting fitness function or with neural networks one can continue to backpropagate errors, and so on with other approaches.


What are the best classifiers for this type of data?

Well, it depends on the structure of the data. The best way is to try all the intelligent models like naive bayes, random forest, svm with different parameters by grid search. There is no model works best all the time for classification. However,  neural network (named Multilayer Perceptron on weka) is supposed to be better if it is set correctly.


Difficulty understanding Keras LSTM fitting data

Yes, intuition says that RNNs like LSTM or GRU will work better in your case, because predicted values might depend on input patterns corresponding to much earlier time intervals.
There is no reason to create samples shifted by a single measurement because many of the samples will contain pretty much the same information for your model. A viable approach is to shift by sample size. Keeping some overlap between samples is feasible as well.

Keep in mind that as a general rule for processing audio data, it makes sense to convert raw audio data into vectors representing audio spectrum before feeding it into LSTM RNN (see this video for e.g. https://www.coursera.org/learn/nlp-sequence-models/lecture/sjiUm/speech-recognition).

Batch size is different from sample or window size in your case.


TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In practice, there are issues with the prevailing approach, motivating an emerging perspective on 'whitebox' hyper-heuristics.

In more detail:

Metaheuristics are methods for searching an intractably large space of possible solutions in order to find a 'high quality' solution. Popular metaheuristics include Simulated Annealing, Tabu Search, Genetic Algorithms etc.

The essential difference between metaheuristics and hyper-heuristics is the addition of a level of search indirection: informally, hyper-heuristics can be described as 'heuristics for searching the space of heuristics'. One can therefore use any metaheuristic as a hyper-heuristic, providing the nature of the 'space of heuristics' to be searched is appropriately defined.

The application area for hyper-heuristics is therefore the same as metaheuristics. Their applicability (relative to metaheuristics) is as a 'rapid prototyping tool': the original motivation was to allow non-expert practitioners to apply metaheuristics to their specific optimization problem (e.g. "Travelling-Salesman (TSP) plus time-windows plus bin-packing") without requiring expertise in the highly-specific problem domain. The idea was that this could be done by:

Allowing practitioners to implement only very simple (effectively,
randomized) heuristics for transforming potential solutions. For
example, for the TSP: "swap two random cities" rather than (say) the more
complex Lin-Kernighan heuristic.
Achieve effective results (despite using these simple heuristics) by combining/sequencing them in an intelligent way, typically by employing some form of learning mechanism.

Hyper-heuristics can be described as 'selective' or 'generative' depending on whether the heuristics are  (respectively) sequenced or combined. Generative hyper-heuristics thus often use methods such as Genetic Programming to combine primitive heuristics and are therefore typically customized by the practitioner to solve a specific problem. For example, the original paper on generative hyper-heuristics used a Learning Classifier System to combine heuristics for bin-packing. Because generative approaches are problem-specific, the comments below do not apply to them.

In contrast, the original motivator for selective hyper-heuristics was that researchers would be able to create a hyper-heuristic solver that was then likely to work well in an unseen problem domain, using only simple randomized heuristics.

The way that this has traditionally been implemented was via the introduction of the 'hyper-heuristic domain barrier' (see figure, below), whereby generality across problem domains is claimed to be achievable by preventing the solver from having knowledge of the domain on which it is operating. Instead, it would solve the problem by operating only on opaque integer indices into a list of available heuristics (e.g. in the manner of the 'Multi-armed Bandit Problem').

In practice, this 'domain blind' approach has not resulted in solutions of sufficient quality. In order to achieve results anywhere comparable to problem-specific metaheuristics, hyper-heuristic researchers have had to implement complex problem-specific heuristics, thereby failing in the goal of rapid prototyping.

It is still possible in principle to create a selective hyper-heuristic solver which is capable of generalizing to new problem domains, but this has been made more difficult since the above notion of domain barrier means that only a very limited feature set is available for cross-domain learning (e.g. as exemplified by a popular selective hyper-heuristic framework).

A more recent research perspective towards 'whitebox' hyper-heuristics advocates a declarative, feature-rich approach to describing problem domains. This approach has a number of claimed advantages:

Practitioners now need no longer implement heuristics, but rather simply specify the problem domain.
It eliminates the domain-barrier, putting hyper-heuristics on the same 'informed' status about the problem as problem-specific metaheuristics.
With a whitebox problem description, the infamous 'No Free Lunch' theorem (which essentially states that, considered over the space of all black box problems, Simulated Annealing with an infinite annealing schedule is, on average, as good as any other approach) no longer applies.

DISCLAIMER: I work in this research area, and it is therefore impossible to remove all personal bias from the answer.


How accurate are neuroevolution algorithms in modelling organism evolution?

Modelling genetic code persistence, mutation, and meiosis for neurological features or any other biological features is not the greatest of challenges. Modelling the rest of the organism that can support neural growth and change, the electrochemistry of the many types of neurons, and the environment that favors the emergence of learned neural behaviors is challenging and still far beyond the current level of technology.


What is regression layer in a spatial transformer?

Basically, it means that the "localization network" should output a set of real valued parameters (typically 6 numbers). The word "regression" doesn't bear any specific meaning.

Any network that relies on the original image as input (directly or indirectly) and outputs 6 numbers, would work. And its last layer would qualify as "regression layer" as long as it is not restricted in the real domain (not normalized, softmaxed, etc)


DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as
$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu)  \right]\;.$$
We are interested in learning about the policy parameters of $\mu$, denoted by $\theta$, but we take expected with respect to some discounted state distribution induced by a policy $\beta$, which we will denote as $\rho^\beta$.
To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.
Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.


How close are we to having an AI that can play Magic: The Gathering objectively well?

This is a very specific task, with clearly defined parameters, so it would already theoretically be within the scope of current AI technology to do this.

The AI would need to learn how to make decisions, and the best way to do this is the approach taken to teaching an AI to play Go - seeing thousands of example games by experts, and playing itself thousands of times.

The AI won't necessarily "understand" what a card represents, the way a human would, but it can learn to make the same kinds of decisions as the human would.

The difficulties would be purely practical - Go is very easy to represent digitally, because the options for action are limited to placing a stone on one of the intersections, and there is only one opponent. Magic is more complex, so the developers would need to spend sufficient resources to be quite sure they had captured all the relevant variables in the digital representation. Then, of course, they would need to encode thousands of games of Magic (or deck-building processes) into that digital format, so the AI could study them.


Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists!

I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $l$th layer is a function of the previous output $x_l = H(x_{l-1})$, while in the dense net each layer is a function of all the previous outputs $x_l = H([x_0, x_1, \dots, x_{l-1}])$.

However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.

The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.

If you're interested you should definitely check out their paper Densely Connected Convolutional Networks (2018).


Will a .h5 file trained with Xception model work with Resnet50?

If I understand your question correctly, you are asking whether you could load the saved weights of a trained model with the Xception architecture on a a Resnet50 architecture.

Short answer: No

Long answer: Xception and Resnet50 have very different architectures.

Here is a paper comparing multilple CNNs including Xception and Resnet50:
https://www.researchgate.net/publication/330478807_Deep_Feature-Based_Classifiers_for_Fruit_Fly_Identification_Diptera_Tephritidae/figures?lo=1

As you can see, the architectures are quiet different between Xception and ResNet50. Thus when you train a model, you are changing the weights and bias of the different layers of the model. If you switch models, and they are similar like let's say VGG16 and VGG19, you could import part of the weights for the layers which are similar between the two models.
As far as I know, I don't know of a function which can do this operation in tensorflow or keras.
But in your case, the architectures are extremely different between Xception and ResNet and there seem not to have any layers in common.

In general, one train a model, save the weights of the training, import back the weights, and use the same model for the next round of training/testing/predictions.

Hope that helps!


How to train a model for 1 image class to detect anomaly?

It sounds like you only have "normal" examples with which to train your model, so this makes the problem feel like an application for outlier detection algorithms. There are a variety of approaches here. You could indeed take an autoencoder approach and then use the reconstruction error to determine if a new image is normal or not, on the presumption that normal images will have lower error. You could also take the activations from the bottleneck layer and build an explicit outlier model using an algorithm like isolation forest. And you are not limited to just autoencoders if you take the latter approach -- other models pretrained for other tasks like image net classification could also provide good features for outlier detection.


"A full-fledged self-aware artificial intelligence may have come to exist in a distributed environment like the internet"

The question implies that this artificial intelligence has surpassed human intelligence (full fledged) and therefore, due to the concept of the intelligence explosion resulting from such a state, the AI you are looking for is no doubt superintelligent.

The question states that this AI is

"unwilling to reveal itself"

and therefore does not intend to be discovered.

Strong (or Superintelligent) AI

Given these two factors (its superior intellect and its unwillingness to be discovered), we can conclude that there is no way in which we would be able to detect such an AI under conventional conditions.

A possible solution may involve employing a second superintelligent AI system, but this is precarious in more ways than one.

Weak AI

Detection of a simpler AI would rely on tracking the pattern of activity and the traces it leaves behind in this distributed network in order to find it, then subjecting it to some form of testing to verify its intelligence.

There are a very large number of possible indicators and these would vary widely depending on the specific AI concerned. This would depend especially on how the AI exists within the framework (here, the web).
Note: a superintelligent AI would not only be able to disguise its activity, but also find a way around any test we can develop - these abilities are what renders this level of AI perhaps the greatest threat to mankind, but also our greatest asset if we do develop it and find a way to gain its alliance.


Multi class text classification when having only one sample for classes

Siamese networks may be useful in your case.
http://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
https://en.wikipedia.org/wiki/Siamese_neural_network
https://link.springer.com/protocol/10.1007%2F978-1-0716-0826-5_3


How to derive matrix form of the Bellman operators?

There's not much to derive here it's simply a definition of Bellman operator, it comes from Bellman equation. If you're wondering why
\begin{equation}
Q^{\pi} = (I - \gamma P^{\pi})^{-1}r \tag{1}
\end{equation}
they state that $Q^{\pi}$ is a fixed point which means if you apply Bellman operator to it you get the same value
\begin{equation}
T^{\pi}(Q^{\pi}) = Q^{\pi}
\end{equation}
You can easily check that since from $(1)$
\begin{equation}
r = (I-\gamma P^{\pi})Q^{\pi}
\end{equation}
if you plug it in definition of Bellman operator you get
\begin{align}
T^{\pi}(Q^{\pi}) &= r + \gamma P^{\pi} Q^{\pi}\\
&= (I - \gamma P^{\pi})Q^{\pi}+  \gamma P^{\pi} Q^{\pi}\\
&= Q^{\pi}
\end{align}


How do 3 channels affect a network when detecting human skin (CNN)?

3 channels will give you more information about the color of the object and surroundings which might be important in certain cases. For example if you want to classify blue cars and red cars then color of the object is very important. Since your problem is to classify hand gestures then color might not be that relevant. You're not very interested in the color of the hand you only care about it's position so there is a good chance that grayscale images might be enough. You should try with grayscale first and if that works good, if not, try with 3 channels.


Are there clever (fitness-based) crossover operators for binary chromosomes?

It's not obvious what you mean by "intelligent crossover".

However, it is common to use fitness-based selection of parents: individuals in the current population who have higher fitness are assigned a higher probability of being selected to mate and produce offspring.  This will increase the likelihood that "good" combinations of genes in members of the current population will be passed along to the next generation, and that independent "good" combinations will be combined in some members of the next generation.

The "best" crossover operator depends dramatically on the structure of the problem being solved, and on the mapping of gene "vectors" to the salient features of a solution.

Edit #1:
In some cases it is important to increase diversity in order to avoid convergence to a local optimum.  In that case, an "intelligent" GA might for example select a first parent for its high fitness, and a second parent at random.  In "Generator", a GA I sold for a while about 25 years ago, mate selection worked that way, and it was often very effective.  Generator also replaced any duplicate individuals in the population with entirely random individuals.  I have also structured genetic algorithms specifically to evolve multiple separate populations of individuals, with minimum gene flow between the populations, in order to evolve multiple solutions corresponding to "regional" fitness optima.

Edit #2:
In genetic algorithms it is not common to directly seek the best combination of genes from both parents. The assumption is that higher-fitness parents are more likely to produce even higher-fitness offspring than lower-fitness parents.  Sometimes there is a local search (hill climbing) operation where the offspring of two parents is mutated in various ways and the best of the mutants is put in the next generation.

And, sometimes a crossover operation involves producing a larger number of offspring than the parent generation, followed by culling low-fitness individuals to keep the population size constant.  This is vaguely analogous to a local search via mutation, but uses random crossover instead of random mutation for its search.


How could an AI be used to improve the teaching and learning of mathematics?

Fundraising is probably the most realistic answer with the biggest effect. Similar to simply adding blockchain to the company name helps.

For the question of real application, I should first say that I'm a machine learning engineer who works as a consultant. I had mathematics as a minor subject during bachelor's and Master's degree, which is like 80% of a bachelor's degree in mathematics. Mainly the bachelor's thesis is missing. I also gave private mathematics lessons.

I would distinguish learning mathematics at school from University. It's completely different. However, for this question it does not make much of a difference. It's also not super important that you ask it for math. It's about learning in general.

Machine learning tools can support for several minor tasks:

Figuring out how a symbol is called (I build write-math.com for that as a bachelor's project)
Figuring out what a symbol means
Getting relevant papers / parts of the lecture notes for a given problem
Helping to digitalize stuff for easier sharing / content creation
Learning things by heart: there are some nice apps like Anki which help you to learn arbitrary stuff by heart. The learning algorithm could potentially be improved by AI


ValueError: Error when checking target: expected dense_3 to have shape (1,) but got array with shape (2,)

Problem is in the output layer and you are using categorical_crossentropy for a loss function. Quoting Keras documentation:

  Note: when using the categorical_crossentropy loss, your targets
  should be in categorical format (e.g. if you have 10 classes, the
  target for each sample should be a 10-dimensional vector that is
  all-zeros except for a 1 at the index corresponding to the class of
  the sample). In order to convert integer targets into categorical
  targets, you can use the Keras utility to_categorical:

I would check if the target values are defined consistently. They seem to be defined for sparse_categorical_crossentropy, which requires a dense layer where number of neurons = number of classes of the problem. If they are defined as integers - as required for sparse_categorical_crossentropy, you can convert them to categorical formatting using to_categorical function, also defined in the documentation page linked above; or you can keep them as integers, and change the number of neurons in the final layer to 2, and loss function to sparse_categorical_crossentropy.


When should I create a custom loss function?

Using two value and using MSE is probably a better approach. I'd you combine the value to one value, like the case of summation, the network may fits to output 0 on one axis and the value on the other. The method you propose also have the same issue. There are many combination to the real distance, but only one is correct.
For a neural network to learn faster, one value will not help it learn faster. Instead, accuracy is often increased if the predicted value is a one hot vector of labels instead of a single value. Hope this can help you.


Why do we average gradients and not loss in distributed training?

The whole idea behind those distributed optimization methods is that data should be local in every node/worker. Thus, if you only send the loss value to the central node, this node can't compute the gradients of this loss, and thus can't do any training. However, if you don't want to send gradients, a family of distributed optimization algorithms called consensus-based optimization can be used to only send the local weight of the model to neighbouring nodes, and those nodes use their local gradient and the models from their neighbours to update their local models.


Why doesn't Q-learning converge when using function approximation?

Here's an intuitive description answer:

Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $1$, and the true Q-function is $Q(s, 0) = s^2$, and $Q(s, 1)= 2s^2$, for all states. If your function approximator is $Q(s, a) = m*s + n*a + b$, there exists no parameters which can accurately represent the true $Q$ function (we're trying to fit a line to a quadratic function). Consequently, even if you chose a good learning rate, and visit all states infinitely often, your approximation function will never converge to the true $Q$ function.

And here's a bit more detail:

Neural networks approximate functions. A function can be approximated to greater or lesser degrees by using more or less complex polynomials to approximate it. If you're familiar with Taylor Series approximation, this idea should seem pretty natural. If not, think about a function like a sine-wave over the interval [0-$\pi/2$). You can approximate it (badly) with a straight line. You can approximate it better with a quadratic curve. By increasing the degree of the polynomial we use to approximate the curve, we can get something that fits the curve more and more closely.
Neural networks are universal function approximators. This means that, if you have a function, you can also make a neural network that is deep or wide enough that it can approximate the function you have created to an arbitrarily precise degree. However, any specific network topology you pick will be unable to learn all functions, unless it is infinitely wide or infinitely deep. This is analogous to how, if you pick the right parameters, a line can fit any two points, but not any 3 points. If you pick a network that is of a certain finite width or depth, I can always construct a function that needs a few more neurons to fit properly.
Q-learning's bounds hold only when the representation of the Q-function is exact. To see why, suppose that you chose to approximate your Q-function with a linear interpolation. If the true function can take any shape at all, then clearly the error in our interpolation can be made unboundedly large simply by constructing a XOR-like Q-function function, and no amount of extra time or data will allow us to reduce this error. If you use a function approximator, and the true function you try to fit is not something that the function can approximate arbitrarily well, then your model will not converge properly, even with a well-chosen learning rate and exploration rate. Using the terminology of computational learning theory, we might say that the convergence proofs for Q-learning have implicitly assumed that the true Q-function is a member of the hypothesis space from which you will select your model.


Is there a reason evolutionary algorithms are language-bound in research material?

I would say there are quite a few different reasons for this, with the proportion of each dependent on a given researcher.

For example, I use python for the vast majority of what I do. And for me, it is due to a few different factors:

I was already familiar with python, and it is a simple, high-level abstraction. This is probably the reason a lot of people set it and forget it, particularly with python. It allows them to focus on the ideas and implementation of said ideas, without worrying about all the junk that comes with trying to write a program in a faster language like C++
The vast majority of ML/DS packages are only or primarily supported via python. I think this is probably the main reason for most in the field, as even if one can implement the architecture in a faster language, the time to do so would likely even out when taking into account the time required to prototype a given model. Tensorflow and others are supported for other languages but do not see the same level of dev support.
The ability to deploy models to multiple platforms without headache. When working in an environment where the work is also applied, the ability to deploy a given model without too much debugging cannot be understated.

These are just a few of the main ones, and as I said the reasons for a particular language over another is primarily a preferential one, and can even vary by requirements(i.e speed)


Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete?

Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in the range $[0,1]$ and the sum across all allowed actions is $1$. The two concepts of determinism and discrete actions are entirely separate.
The optimal policy in many situations can be deterministic. If there is only one deterministic optimal policy, your learned policy should be also close to deterministic if the learning process has been successful. That is, the probabilities of optimal actions should all be close to $1$, all the rest close to $0$.
If there is more than one possible optimal policy, your learning agent may have learned a stochastic "mix" of them where in some states it is equally good to take more than one action and the probabilities may be split between those good actions. This will still be optimal and not a problem. If that is the case you should expect to see many action choices close to $0$ and in each state a select few (maybe one) that sum to close to $1$ between them.
In the case of discrete actions, you can derive a deterministic policy from your neural network function by taking the argmax of the action probabilities. This is worth trying. It will round away bad actions that are close to 0 probability due to approximation in the neural network.
In practice sometimes a little randomness in a policy works better for real-world problems with imprecise measurements or other unknowns. It may even be necessary for adversarial environments or where there is key information missing. The only way to find out though is to try both stochastic and deterministic interpretations of the neural network output for your policy.


What does "immediate vector-valued feedback" mean?

By "immediate vector-valued feedback", they probably mean exactly the label in the "labeled examples" you mentioned.


There is not one answer to this question, but one could argue that transformers heavily rely on

transforming each input into latent subspaces of queries, keys and values in order to generate attention score
a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input


Over- and underestimations of the lowest and highest values in LSTM network

RNN is a deeply non-linear function over time, how the black linear line is deduced?

Assuming you are doing just linear regression if the least square error was used as the loss function, it will have a probabilistic interpretation

$$y^{(i)}|x^{(i)};\theta \sim \mathcal N(\theta^Tx^{(i)}, \sigma^2)$$

$Y$ is conditioned on $X$ parameterized by $\theta$, with a Gaussian distribution, thus for every data point $x$, there is a corresponding $y$ which if you are doing maximum likelihood estimation is just the mean of the Gaussian, hence variance is introduced to express the noise. if you are doing mini-batch is not guaranteed to reach a global minimum.

Side note: If you normalize the data with min-max scaling, please make sure to use only the train set, if you include the dev/test set, you are doing some kind of data snooping, generalization error will be biased.


Image dataset for pomegranate plant disease

You can find the dataset in the following links:

Pomegranate Disease Detection Using Image Processing
fruits 360 datasets


Could the normalisation of the inputs make the neural network insensitive to changes in the inputs?

When using neural networks (NNs), we often normalized the inputs. I
think this is done to equally capture the changes in any input
feature, that is, if any feature takes huge values and other features
take small values, we don't want the NN not to be able to "see" the change
in the smaller value.
However, what if we cause the NN to become insensitive to the input,
that is, the NN is not able to identify changes in the input
because the changes are too small?

We don't normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).
An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have only $256$ distinct values. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input more sensitive to changes.
Instead, normalization is done to help with the model's convergence.


A learning agent is any entity that, over time, improves its performance (which can be defined in different ways depending on the context) based on the interaction with the environment (or experience).

The human is an example of a learning agent. For example, a human can learn to ride a bicycle, even though, at birth, no human possesses this skill.


What are some online courses on artificial general intelligence?

There are at least three free courses.

MIT 6.S099: Artificial General Intelligence, organized by Lex Fridman, is a series of lessons and talks primarily given by a diverse set of guest appearances, such as

Josh Tenenbaum (researcher and professor in computational cognitive science),
Nate Derbinsky (who gives a lesson on cognitive architectures, Soar, etc.),
Stephen Wolfram (creator of Mathematica, Wolfram Alpha and the Wolfram Language; he talks about his work throughout the years, especially, the development of Wolfram Alpha and Language),
Marc Raibert (CEO of Boston Dynamics, who gives a lesson on his work at Boston Dynamics and the robots they have developed),
Lisa Feldman Barrett (professor of psychology, who gives a very insightful lesson on emotions and feelings, with possibly different perspectives),
Rosalind Picard (a professor at MIT, director of the Affective Computing Research Group at the MIT Media Lab, and co-founder of two companies, Affectiva and Empatica; in her talk with Lex, she talks about affective computing and other interesting concepts and issues)
Marcus Hutter (inventor of a mathematical theory of general intelligence, AIXI; in his talk with Lex, he talks about Occam's razor, Solomonoff induction, Kolmogorov complexity, the definition of intelligence, AIXI, rewards, bounded rationality and consciousness)

Knowledge-Based AI: Cognitive Systems (taught by Ashok Goel and David Joyner)

The Society of Mind (taught by Marvin Minsky, who also wrote a book and developed a theory of natural intelligence with the same name)

CS 294-149: Safety and Control for Artificial General Intelligence (Fall 2018) (see also these notes)


Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?

I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the different "factors" they mention in the paper).
As an example, imagine a game in which you can both move and jump at the same time. In that case, you might have a 4-dimensional discrete action space for moving (NSWE), and a 2-dimensional discrete action space for jumping (yes/no jump), both of which will require a Categorical distribution which has the size of that factor ($N_k$ in the paper).
So in this case, you would need to have a categorical distribution for each factor, unless you were to turn these two factors into 1 joint 4*2-dimensional factor and learn a single categorical distribution on that (which would likely be less efficient).


Understanding the proof that A* search is optimal

The key phrase here is

  because heuristics are admissible

In other words, the heuristics never overestimate the path length:

$$cost(n) + heuristic(n) \le cost(\text{any path going through n})$$

And since the frontier is ordered by $\textbf{cost + heuristic}$, when a completed path $p$ is dequeued from the frontier, we know that it must necessarily be $\le$ any path going through some other frontier node $q$, because

$$cost(p) = cost(p) + heuristic(p) \le cost(q) + heuristic(q) \le cost(\text{any path going through q})$$


How do I predict the occurrence of rare events?

It might be more informative to:

Label each combination of location, type, and time of crime with a crime rate. For example, theft, in Crystal City, at 11pm at night, occurs 20 times per year, or 0.4 times per resident per year.
Predict the crime rate, rather than individual events.

This avoids the need to have explicit examples of "non-crime", and lets you instead directly learn something related to the probabilities of crimes being committed (the rate).


Developing character tactics via repeated trials

There are a few ways to tackle this. You could make an AI that is simply a series of IF statements, or you could actually make an AI that would actually take in the situation and come up with a sensible solution.

IF Approach - You make a series of IF statements that come up with a sensible action to execute. This is the method that Minecraft uses. The resulting action were recorded from some of the best players.
True AI - Have your Character execute random actions and learn the consequences of them. The, train it to execute various actions for certain scenarios.

The main difference between these two approaches is that IF statements have a constant and predictable behavior, while the AI approach has a very bad startup value but ends up improving over time.

There is no "best" method, it is up to you to choose one or the other or a mix of both.


How does A* search work given there are multiple goal states?

Yes. If you leave A* running (i.e. do not impose a goal condition on a newly-encountered state), all states will be explored, just as they would be in breadth- or depth- first search.


How well can a CNN distinguish an object from its class?

To help you understand the feasibility of your project these posts could be a good start:
https://datascience.stackexchange.com/questions/13181/how-many-images-per-class-are-sufficient-for-training-a-cnnenter link description here

https://stats.stackexchange.com/questions/226672/how-few-training-examples-is-too-few-when-training-a-neural-network

That being said, the short answer would be it depends. It depends how precise you want to be, what is the difficulty of the task, the infrastructure you have for the training etc.

For the images, you should not worry, you could start with the ImageNet dataset:
For construction and buildings:
http://www.image-net.org/explore?wnid=n04341686

For mosquees:
http://www.image-net.org/synset?wnid=n03788195

You can then use data augmentation techniques to enhance the size of you training set. Here is a library I have used in the past which helped me greatly to achieve this task: https://github.com/aleju/imgaug

Hope that helps!


The accuracy of a model is defined as the number of correct predictions divided by the total number of predictions. The expression "prediction accuracy" refers to the accuracy of the model at doing predictions.

For example, suppose that you observe the sequence of numbers $1, 1, 2, 3$. What would be a likely next number of this sequence? The operation of guessing this number is called a prediction. These are numbers of a Fibonacci sequence, so a model could predict $5$ as the next number of this sequence. However, it could also predict e.g. $2$ because the complete sequence could be $1, 1, 2, 3, 2, 1, 1$. Which prediction is correct? It depends on the problem. Do we want the model to predict the numbers of the Fibonacci sequence or of another sequence? Suppose that we want the model to learn the Fibonacci sequence and the models predicts another number different than $5$, then this will be considered an incorrect prediction.

A model that produces incorrect predictions is useless. So, we want models that have high accuracy. However, accuracy is not always the best measure of a model's performance. See e.g. Why is accuracy not the best measure for assessing classification models?.

There are several related terms and concepts, such as inference, forecasting or deduction. They are sometimes used interchangeably, even though they might have slightly different meanings or definitions.


How to identify whether images contain driver's licenses or ID cards

Why not make a detector/classifier to look for the text "Drivers Licence" or some very generic keywords related to license? As you mentioned images of people may be present in any sort of documents. Looking for text which is super specific to IDs/drivers license seem to be a better way.


What is convergence in machine learning?

When formulating a problem in deep learning, we need to come up with a loss function, which uses model weights as parameters. Back-propagation starts at an arbitrary point on the error manifold defined by the loss function and with every iteration intends to move closer to a point that minimises error value by updating the weights. Essentially for every possible set of weights the model can have, there is an associated loss for a given loss function, with our goal being to find the minimum point on this manifold.

Convergence is a term mathematically most common in the study of series and sequences. A model is said to converge when the series $s(n) = loss_{w_n}(\hat y, y)$ (Where $w_n$ is the set of weights after the $n$'th iteration of back-propagation and $s(n)$ is the $n$'th term of the series) is a converging series. The series is of course an infinite series only if you assume that loss = 0 is never actually achieved, and that learning rate keeps getting smaller.

Essentially meaning, a model converges when its loss actually moves towards a minima (local or global) with a decreasing trend. Its quite rare to actually come across a strictly converging model but convergence is commonly used in a similar manner as convexity is. Strictly speaking rarely exists practically, but is spoken in a manner telling us how close the model is to the ideal scenario for convexity, or in this case convergence.


How can I train a neural network to describe the characteristics of a picture?

The term you are looking for is multi-label classification, i.e. where you are making more than one classification on each image (one for each label). Most examples you'll find online are in the NLP domain but it is just as easy with CNNs since it's essentially defined by the structure of the output layer and the loss function used. It's not as complicated as it might sound if you are already familiar with CNNs.

The output layer of a neural network (for 3 or more classes) has as many units as there are targets. The network learns to associate each of those units with a corresponding class. A multi-class classifier normally applies a softmax activation function to the raw unit output, which yields a probability vector. To get the final classification, the max() of the probability vector is taken (the most probable class). The output would look like this:

                 Cat    Bird   Plane   Superman  Ball   Dog
Raw output:      -1     2      3       6         -1     -1
Softmax:         0.001  0.017  0.046   0.934     0.001  0.001
Classification:  0      0      0       1         0      0

Multi-label classification typically uses a sigmoid activation function since the probabilities of a label occuring can be treated independently. The classification is then determined by the probability (>=0.5 for True). For your problem, this output could look like:

                 Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
Raw output:      -1        -2         3           6         -1        10
Sigmoid:         0.269     0.119      0.953       0.998     0.269     1.000
Classification:  0         0          1           1         0         1

The binary crossentropy loss function is normally used for a multi-label classifier since a n-label problem is essentially splitting up a multi-class classification problem into n binary classification problems.

Since all you need to do to get from a multi-class classifier to a multi-label classifier is change the output layer, its very easy to do with pre-trained networks. If you get the pre-trained model from Keras its as simple as including include_top=False when downloading the model and then adding the correct output layer.

With 13000 images, I would recommend using Keras' ImageDataGenerator class with the flow_from_dataframe method. This allows you to use a simple pandas dataframe to label and feed in all your images. The dataframe would look like this:

Filename  Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
0001.JPG  0         0          1           1         0         1
0002.JPG  1         0          1           0         1         1
   .      .         .          .           .         .         .

flow_from_dataframe's class_mode parameter can be set to raw or multi_output along with x_col to 'Filename' and y_col to ['Big nose', 'Long hair', 'Curly hair', 'Superman', 'Big ears', 'Sharp Jawline'] (in this example). Check out the documentation for more details.

The amount of data you need for each label depends on many factors and is essentially impossible to know without trying. 13000 sounds like a good start but it also depends on how many labels you have and how evenly distributed they are between the labels. A decent guide (one of many) on how to set up a multi-label classifier and how to implement it with Keras can be found here. It also covers imbalances on label frequency and is well worth a read. I'd highly recommend that you become as intimately familiar with your dataset as possible before you start tuning your neural network architecture.


To obtain the edges, you need to use an edge detector. Many exists, more or less efficient depending on the input image. Among the most broadly used are the Sobel and the Canny edge detectors that are implemented in many libraries for various programming languages. (Matlab, scikit-image in Python)

To obtain straight lines from these edges, I suggest that you start by using the Hough transform also available in many libraries for various programming languages. (Matlab, scikit-image in Python)

Lines output from the Hough transform are usually given in there parametric form. From there, it is easy to find the angles between the different lines. I imagine that some rules can be derived from there to identify a regular polygon that is in the image.

Regarding obtaining 3D information from an image, note that there isn't enough information in a single 2D image to reconstruct its 3D coordinates, since we are projecting information into a lower dimensional space.


Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.

Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.

Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to  do. You just repeat and repeat, until you see the test error no longer improve.


What event would confirm that we have implemented an AGI system?

It is a difficult question to answer, as — for a start — we still don't really know what 'intelligence' means. It's a bit like Supreme Court Justice Potter Stewart declining to define 'pornography', instead stating that [...]I know it when I see it. AGI will be the same.

There is no single event (almost by definition), as that's not general. OK, we've got machines that can beat the best human players at chess and go, two games that were for centuries seen as an indication of intelligence. But can they order a takeaway pizza? Do they even understand what they are doing? Or, even more fundamental, know what they means in the previous sentence?

In order for a machine to show a non-trivial level of intelligent behaviour, I would expect it to interact with its environment (which is more social intelligence, an aspect that seems to be rather overlooked in much of AI). I would expect it to be aware of what it's doing/saying. If I have a conversation with a chatbot that really understands what it's saying (and can explain why it came to certain conclusions), that would be an indication that we're getting closer to AGI. So Turing wasn't that far off, though nowadays it's more achieved with smoke and mirrors rather than 'real' intelligence.

Understanding a story: being able to finish a partial story in a sensible way, inferring and extrapolating the motives of characters, being able to say why a character acted in a particular way. That for me would be a better sign of AGI than beating someone at chess or solving complex equations. Jokes that are funny; breaking rules in story-telling in a sensible way.

Writing stories: NaNoGenMo is a great idea, and throws up lots of really creative stuff, but how many of the resulting novels would you want to read instead of human-authored books? Once that process has generated a best-seller (based on the quality of the story), then we might be getting closer to AGI.

Composing music: of course you can already generate decent music using ML algorithms. Similar to stories, the hard bit is the intention behind choices. If choices are random (or based on learnt probabilities), that is purely imitation. An AGI should be able to do more than that. Give it a libretto and ask it to compose an opera around it. Do this 100 times, and when more than 70-80 of the resulting operas are actually decent pieces of music that one would want to listen to, then great.

Self-driving cars? That's not really any more intelligent (but a lot sexier!) than to walk around in a crowd without bumping into people and not getting run over by a bus. In my view it's much more a sign of intelligence if you can translate literature into a foreign language and the people reading it actually end up enjoying it (instead of wondering who translated that garbage).

One aspect we need to be aware of is anthropomorphising. Weizenbaum's ELIZA was taken for more than it was, because its users tried to make sense of the conversations they had and built up a mental model of Eliza, which clearly wasn't there on the other side of the screen. I would want to see some real evidence of intentionality of what an AGI was doing, rather than ascribing intelligence to it because it acts in a way that I'm able to interpret.


What beginner-friendly machine learning method should I use to make teams for my pickup ultimate frisbee club fairly balanced?

What you have on your hands is an attribution problem (and that’s a good keyword to help in your Googling). Two common approaches are computing Shapley values or a Markov chain.

In your case, I think Shapley values would be a good approach. To over-simplify, this approach attempts to first determine the total “surplus” of value created by different combinations of players, and then estimate how much was due to each player in the combination. Also under consideration are constrained combinatorial effects: e.g. having two great goalies doesn’t add much over one great goalie since they can’t play at the same time.

Depending on how hands-on you want to be you could implement yourself or find existing libraries by Googling e.g. “Shapley values python”.