Tuning of PPO metaparameters: a high level overview of what each parameter does

Some investigation about the memory dict: The current type is latest, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
Also, include_next_state means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to False, otherwise an error is raised.
Your interpretation of capacity looks OK.

About the update mode spec dict, your current settings mean that : every 10 (frequency) episodes (unit), you pull a batch of 10 (batch_size) episodes (unit) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the unit defines both the unit of the optimization frequency and the type of the object fetched from the memory.


I am assuming the images you gave the model all contain sheep. This is what i understand from your question.

Any model that you build will be based on the data that you give (training data) and your code. In your case, if you only give images that contains sheeps, and then you test it with no sheep and a background the model hasn't seen, it will search through all of its node, derived from your training data, to see which one is closest to the image you gave. Based on the information given, the only 'route' your cnn model can take is the one with sheeps on it, because you only gave images of sheeps for it to learn.

Here are a few suggestions that i can give you:

Give your model images with no sheep and different backgrounds so it can handle cases where there are no sheeps
Or you can add a piece of code to your model that tells it to default to some value if certain conditions aren't met (If the model doesn't 'see enough' sheep and background, it defaults to 'unknown image' or whatever you choose)

You are on the right path though! Since you are playing around with a binary classifier, you should definitely feed images with and without sheeps so that it can identify the two cases. Remember, a binary classifier is optimal when you give it 2 things to look out for, such as identifying images with and without sheep.

Here are some reading materials that you can brush up on to get a better basic understanding of how CNN works, personally i found the video helpful on the second link:

https://becominghuman.ai/what-exactly-does-cnn-see-4d436d8e6e52
https://brohrer.github.io/how_convolutional_neural_networks_work.html

Also, i find google images helps me alot in terms of visualizing binary classifier as a concept.


What are the state-of-the-art Person-Detektion / Human-Segmentation?

You can use any state of the art object detection such as DETR, EfficientDET, YOLO etc. since they have an object 'person' in them. But, if the density of people is high in the images, then, you should look at 'Finding Tiny Faces' which is good for that problem (this will help you detect faces only, think group photos). If you want to find the whole person even if they are occluded, then, it will be difficult (you can then think of segmentation techniques and then, try to find out the people but it will be more taxing).


How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation
In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network.
In reinforcement learning, one could map (state, action)-pairs to Q-values with a neural network. Then GD and BP can be used to update the weights of this neural network.
How to design the neural network?
In this context, a neural network can be designed in different ways. A few are listed below:

The state and action are concatenated and fed to the neural network. The neural network is trained to return a single Q-value belonging to the previously mentioned state and action.

For each action, there is a neural network that provides the Q-value given a state. This is not desirable when a lot of actions exist.

Another option is to construct a neural network that accepts as input the state. The output layer consists of $K$-units where $K$ is the number of possible actions. Each output unit is trained to return the Q-value for a particular action.

Q-learning update rule
This is the Q-learning update rule

So, we choose an action $a_t$ (for example, with $\epsilon$-greedy behavior policy) in the state $s_t$. Once the action $a_t$ has been taken, we end up in a new state $s_{t+1}$ and receive a reward $r_{t+1}$ associated with it. To perform the update, we also need to choose the Q-value in $s_{t+1}$ associated with the action $a$, i.e. $\max _{a} Q\left(s_{t+1}, a\right)$. Here, $\gamma$ is the discount factor and $\alpha$ is the learning rate.
Back-propagation and Q-learning
If we use a neural network to map states (or state-action pairs) to Q-values, we can use a similar update rule, but we use GD and BP to update the weights of this neural network.
Here's a possible implementation of a function that would update the weights of such a neural network.
def update(self, old_state, old_action, new_state,
  reward, isFinalState = False):

  # The neural network has a learning rate associated with it.
  # It is advised not to use two learning rates
  learningRate = 1

  # Obtain the old value
  old_Q = self.getQ(old_state, old_action)

  # Obtain the max Q-value
  new_Q = -1000000
  action = 0
  for a in self.action_set:
    q_val = self.getQ(new_state, a)
    if (q_val > new_Q):
      new_Q = q_val
      action = a

  # In the final state there is no action to be chosen
  if isFinalState:
    diff = learningRate * (reward - old_Q)
  else:
    diff = learningRate * (reward + self.discount * new_Q - old_Q)

  # Compute the target
  target = old_Q + diff

  # Update the Q-value using backpropagation
  self.updateQ(action, old_state, target)

In the pseudocode and in the Q-learning update formula above, you can see the discount factor $\gamma$. This simply denotes whether we are interested in an immediate reward or a more rewarding and enduring reward later on. This value is often set to 0.9.


How do neural networks weigh multiple inputs/features of different dimensionality?

As stated in your example, the three features are: an image, a price, a rating. Now, you want to build a model that uses all of these features and the simplest way to do is to feed them directly into the neural network, but it's inefficient and fundamentally flawed, due to the following reasons:

In the first dense layer, the neural network will try to combine raw pixel values linearly with price and rating, which will produce features that are meaningless for inference.

It could perform well just by optimizing the cost function, but the model performance will be nowhere as good as it could perform with a good architecture.

So, the neural network doesn't care if the data is a raw pixel value, price, or rating: it would just optimize it to produce the desired output. That is why it is necessary to design a suitable architecture for the given problem.
Possible architecture for your given example :

Separate your raw features, i.e. pixel value, and high-level data, i.e. price and rating

Stack 2-3 dense layers for raw features (to find complex patterns in images)

Stack 1-2 dense layers for high-level features

Combine them together in a final dense layer

If you want to de-emphasize the importance of the image, just connect the first dense layer 16,384 to another layer having fewer connections, say 1024, and have more connections from the high-level data, say 2048.
So, again, here's the possible architecture

Raw features -> dense layer (16384) -> dense layer (1024)
High-level features -> dense layer (2048)
Combine 1 and 2 with another dense layer


Is it possible to train a neural network with 3 inputs and 12 outputs?

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers)

The better question is should you?. In all honesty, it depends on the data that you have, but, usually, with only 120 examples you'll either overfit completely or do relatively well if the true solution is a very simple function, but, in general, in the common situations where that isn't the case I find myself more likely or not using Bayesian approaches, so I can actually consider confidence (with little data, this is really nice)


Where would consciousness reside in an artificial intelligence?

There are still many different interpretations of the terms intelligence and consciousness. For the sake of this discussion I would define intelligence as the problem solving capabilities and consciousness as the state of being aware of and responsive to one's surroundings, including the experience of qualia.

If I understand your question correctly, you are focused on consciousness and not intelligence (based on the definitions above).

In the case of humans - the only species where we have first hand knowledge about what consciousness feels like - it feels like the consciousness resides inside the respective body. Most people experience it like "being" inside their head. Although debated, consciousness is most likely a simulation created by the human brain. Qualia is heavily - if not exclusively - influenced by the sensory input from the human senses. Therefore it only makes sense that it feels like the consciousness resides in the center of the sensors (eyes, ears, etc.).

Science hasn't figured out the nature of consciousness yet, so there is a lot of speculation involved. Applying above this logic to artificial consciousness and qualia, it would make sense that the machine will feel like being at the location if its sensors as well. If there is no dominant area of sensors, maybe a distributed system, it might be a split consciousness spread across several locations.

If the creation of consciousness happens accidentally - like it did during evolution - emergence at the sensors feels most likely. Should the creator of artificial consciousness do it on purpose, he should be able to shape the simulation in other ways as he sees fit.


What are the consequences of layer norm vs batch norm?

This is how I understand it.
Batch normalization is used to remove internal covariate shift by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:
$$
\textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
$$
where $\gamma$ and $\beta$ are affine parameters learned from data; $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $\gamma$ and $\beta$.
This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:
$$
\textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
$$
but now $\mu(x)$ and $\sigma(x)$ are computed across all channels for each individual sample. Here's an illustration of the difference:

So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.
In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.
In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.
I strongly encourage you to read the original BN paper and also:
Adaptive instance normalization
Group Normalization


Generate Image with Artificial intelligence

For recreating an image exactly the same as the original, you can use an autoencoder. This basically use AI Layers to encode the image raw pixel values to a vector of floats, drastically decreasing the representing vector. Afterwards another AI Layer increases the dimensions back to the original image. The method does not required labels, as it only refer to teh image to encode it to a vector of features. For implementing in java, there is not a lot 9f resources. However, you can check this library out: https://deeplearning4j.org/
For the implementation, see this: https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/unsupervised/variational/VariationalAutoEncoderExample.java
For the method, you can see this python tutorial and implement it in java. https://towardsdatascience.com/autoencoders-in-keras-c1f57b9a2fd7

For generating completely new image, you can try GAN(Generative Adverserial Network). This generates completely new images from a random noise image. The noise image is passed through a generator which is a CNN(convolutional neural network) and get a result image. The result image is then feed to  a discriminator(CNN as well) to classify if that image is fake or real. The generator and discriminator compete and slowly gets better. For java implementation, see this: https://github.com/wmeddie/dl4j-gans

Hope I can help you and have a nice day!


Effect of adding an Independent Variable in Multiple Linear Regression

Yes, the addition of the independent variable ($X_2$) will have an impact on the beta coefficient of $X_1$.
Maybe, it depends on the relationship between $X_1$ and $Y$, and whether you are using any regularization or not.
If the independent variable $X_2$ is more correlated than $X_1$, then the value of $r^2$ should be higher (but I am not sure on this)


How can abstract graphs be recognized by neural nets?

The OP's (i.e. my) assumption is wrong. It is not true that for almost all random graphs under no permutation a pattern is visible, as can be  seen in Tiago P. Peixoto's paper Bayesian stochastic blockmodeling, p. 4:

The opposite seems to be correct: For almost all random graphs there are permutations such that patterns are visible.


Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

That is the definition of return.
In fact when applying a discount factor this should formally be called discounted return, and not simply "return". Usually the same symbol is used for both ($R$ in your case, $G$ in e.g. Sutton & Barto).
There are also other variations, such as truncated return (sum up to a given time horizon). They all share the feature that a return is a sum of reward values. You cannot really change that and keep the formal term "return", that's how it has been defined.
You can however define the value function to be something other than the expected return. Rather than looking for alternative definitions of return as your title suggests, you could be looking for alternative metrics to use as value functions.
You do go on to ask about computing "the value of a state" without mentioning the word "return", but it is not 100% clear whether you are aware that the way to resolve this is to not use return, but something else.

Would it not make more sense to compute the value of a state as the following:
$V^π(s)$ = $(r_{t+1} + γr_{t+2} + (γ^2)_{t+3}, + ...    \mid s)/k = {E}(∑γ^kr_{t+k+1}\mid s)/k $ where k is the number of elements in the sum, thus giving us the average reward for being in state s.

Your example would nearly always result in zero for long-running or non-episodic problems, as you are summing a decreasing geometric series possibly up to very large $k$, then dividing by the maximum $k$. Notation-wise you are also using $k$ to be an iterator and the maximum value of the same iterator, that would need fixing.
However, this is very close to a real value metric used in reinforcement learning, called the average reward setting.
The expected average reward value function for a non-episodic problem is typically given by
$$V^\pi(s) = \mathbb{E}[\lim_{h \to \infty}\frac{1}{h}\sum_{k=0}^{h}r_{t+k+1}|s_t = s]$$
Note there is no discount factor, it is not usually possible to combine a discount factor with the average reward setting.
Sutton & Barto point out in Reinforcement Learning: An Introduction chapter 10, section 10.4, that when using function approximation on continuing tasks, then a discount factor is not a useful part of the setting. Instead average reward is a more natural approach. It is also not so different, and quite easy to modify the Bellman equations and update rules. However, many DQN implementations still use discounted return to solve continuing tasks. That is because with high enough discount factor $\gamma$, e.g. $0.99$ or $0.999$, then the end result is likely to be the same optimal solution - the discount factor has moved from being part of the problem formulation to being a solution hyperparameter.


In reinforcement learning (RL), an agent interacts with an environment in time steps. On each time step, the agent takes an action in a certain state and the environment emits a percept or perception, which is composed of a reward and an observation, which, in the case of fully-observable MDPs, is the next state (of the environment and the agent). The goal of the agent is to maximise the reward in the long run.

The (temporal) credit assignment problem (CAP) (discussed in Steps Toward Artificial Intelligence by Marvin Minsky in 1961) is the problem of determining the actions that lead to a certain outcome.

For example, in football, at each second, each football player takes an action. In this context, an action can e.g. be "pass the ball", "dribbe", "run" or "shoot the ball". At the end of the football match, the outcome can either be a victory, a loss or a tie. After the match, the coach talks to the players and analyses the match and the performance of each player. He discusses the contribution of each player to the result of the match. The problem of determinig the contribution of each player to the result of the match is the (temporal) credit assignment problem.

How is this related to RL? In order to maximise the reward in the long run, the agent needs to determine which actions will lead to such outcome, which is essentially the temporal CAP.

Why is it called credit assignment problem? In this context, the word credit is a synonym for value. In RL, an action that leads to a higher final cumulative reward should have more value (so more "credit" should be assigned to it) than an action that leads to a lower final reward.

Why is the CAP relevant to RL? Most RL agents attempt to solve the CAP. For example, a $Q$-learning agent attempts to learn an (optimal) value function. To do so, it needs to determine the actions that will lead to the highest value in each state.

There are a few variations of the (temporal) CAP problem. For example, the structural CAP, that is, the problem of assigning credit to each structural component (which might contribute to the final outcome) of the system.


Is Expected SARSA an off-policy or on-policy algorithm?

Expected SARSA can be used either on-policy or off-policy.

The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.

Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one $r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a')$ term to add with a probability $1$.

You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions $r_{t+1} + \gamma \sum_{a'} \pi(a'|s_{t+1}) Q(s_{t+1}, a')$

You can construct Expected SARSA updates where $\pi(a|s)$ is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using $\epsilon$-greedy with $\epsilon=0.1$ and update the value function with $\epsilon=0.01$.


Why don't people always use TensorFlow Lite, if it doesn't decrease the accuracy of the models?

This partly answer to question 1.
There is no general rule concerning accuracy or size of the model. It depends on the training data and the processed data.
The lightest is your model compared to the full accuracy model the less accurate it will be. I would run the lite model on test data and compare to the accuracy of the full model to get an exact measure of the difference.

Tensor flow has different options to save the "lite" model (optimized in size, latency, none and default).

The following mostly answer question 2.

Tensor flow lite is intented to provide the ability to use the model to on line predict only and load the model not to train the model.

On the other hand Tensor flow is used to build (train) the model off line.

If your edge platform support any of the binding language provided for TensorFlow (javascript, java/kotlin, C++, python) you can use Tensorflow for prediction. The accuracy or speed options you might have selected to create the model will not be affected whether you use Tensor Flow or Tensor Flow Lite. Typically Tensor flow lite can be used on mobile devices (Ios, Android). There are other supported target, see this link


Can I combine two classifiers that make different kinds of errors to get a better classifier?

Yes, you can. There are a lot of different techniques, usually called Ensemble Methods.

A better approach might be to use something like AdaBoost along with a cheaper method like the decision trees you looked at. AdaBoost explicitly tries to train classifiers to correctly handle different parts of the data, rather than hoping that different methods turn out to do so by chance.


In GradCAM, why is activation strength considered an indicator of relevant regions?

I think you are misreading the relevant passage here.
Since you do not specify exact excerpt(s), I take that by "implicit assumption" you refer to the equation (2) (application of a ReLU) and the corresponding text explanation (bold emphasis mine):

We apply a ReLU to the linear combination of maps because we are only
interested in the features that have a positive influence on the class
of interest, i.e. pixels whose intensity should be increased in order
to increase $y^c$. Negative pixels are likely to belong to other
categories in the image. As expected, without this ReLU, localization maps sometimes highlight more than just the desired class and perform worse at localization.

The first thing to notice here is that this choice is not at all about activations close to zero, as you seem to believe,  but about negative ones; and since negative activations are indeed likely to belong to other categories/classes than the one being "explained" at a given trial, it is very natural to exclude them using a ReLU.
Grad-CAM maps are essentially localization ones; this is apparent already from the paper abstract (emphasis mine):

Our approach – Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’  in  a classification  network  or  a  sequence  of  words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept.

and they are even occasionally referred to as "Grad-CAM localizations" (e.g. in the caption of Fig. 14); taking a standard example figure from the paper, e.g. this part of Fig. 1:

it is hard to see how including the negative values of the map (i.e. removing the thresholding imposed by the ReLU) would not lead to maps that include irrelevant parts of the image, hence resulting in a worse localization.

A general remark: while your claim that

After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means

is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a trained network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases & weights matter, and we cannot transform them arbitrarily.

UPDATE (after comments):

Are you pointing out that it's called "localization" and therefore must be so?

It is called "localization" because it is localization, literally ("look, here is the "dog" in the picture, not there").

I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?"

It is not at all like that; positive means X and negative means not-X in the presence of class X (i.e. a specific X is present), in the specific network, and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.

Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? [...] why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would hold, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side (and thus broken the symmetry)...


Can neurons in MLP and filters in CNN be compared?

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer
Differences
The neurons of these two types of layers have two key differences. These are that the convolution layers implement:

Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole.
Weight sharing, i.e. similar connections end up having the same weights. This is usually visualized as the same filter traversing the image.

Besides these two key differences, there are some other technical details, e.g. how the biases are implemented. Other than that they perform the same operation.
What causes some confusion is that the input of a CNN is usually 2 or 3-dimensional, while a FC is usually 1-dimensional. These aren't mandatory however. To better help visualize the differences between the two I made a couple of figures illustrating the differences between a conv-layer and a FC one, both in 1D.
Sparse connectivity
On the left are two FC neural networks. On the right, are layers with sparse connections.

Weight sharing
On the left is a sparsely connected network. The colors represent the different values of the weights. On the right is the same network with weight-sharing. Note that similar weights (i.e. arrows with the same direction in each layer) have the same value.

To answer your other questions:

Are filters not the same in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layers use these new "images"?

Yes, if the input of a convolution layer is an image, so will the output. The next layer will also operate on an image.
However, I'd like to note that not all convolution layers accept images as their inputs. There are 1D and 3D convolutional layers as well.


How are mean and variance calculated in Bayesian curve fitting?

Since you only mention that their Gaussian, here is a screenshot depicting the relationship of gaussian conjugate priors

I took this from wikipedia. If you check with what ever your likelihood and priors parameters are it probably all works out (you did not add these in the question)

From a quick google, here's a link to someone going through the proof: link


What approach would work well for predicting earthquake intensity based on historical data?

You can do prediction like this with regression. Take a look at scikit-learn models for regression: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model
Here's an example:
# This is made up data!
#        epicenter magnitude, depth_m, distance_km, local_magnitude
data = [[3.5,                 6000,    61.3,        2.1],
        [9.1,                 16000,   261.0,       6.0],
        [9.1,                 16000,   96.5,        8.2],
        [3.8,                 6000,    80.0,        2.6],
        [2.5,                 4000,    101.3,       0.4],
        [7.0,                 9000,    81.9,        6.1],
        [5.1,                 7000,    21.3,        4.1]]

from sklearn.linear_model import LinearRegression
import numpy as np

data = np.asarray(data) # to numpy array
X=data[:,:3] # the input variables
y=data[:,3] # the output variable (local magnitude)
reg = LinearRegression().fit(X, y) # fit the model based on the data
reg.predict(np.array([[6.6, 8000, 27]])) # Get a prediction based on new inputs

>>> 6.1511027


An approach on reading musical notes from photos

refer to MNIST on kagle
to train model to recognize given set of notes in your case.

if you start training model will understand pattern depending on bits in your image.

issue comes with blurry images when last 2 sets of pics with minute change. will be difficult task for computer to distinguish.

there is one sure shot method which works almost every time but is resource intensive and needs plethora of data to train. is DNN.

this is my take on your question.


What happens if the opponent doesn't play optimally in minimax?

What happens if the opponent plays irrationally or sub-optimally? Do you still have a guarantee that you are going to win?

If your search is deep enough to guarantee optimal play in all cases, then yes. Optimal play is such that the opponent's decision causes least impact to your agent. In fact, if the opponent makes a mistake, often that can make the search easier/faster, and the agent will win more convincingly.

What it may mean is that optimisations you may have taken - e.g. pruning game tree segments that lead to non-optimal decisions by either player - might not be as useful. This might impact decision time if you keep some partial game tree or cache branch evaluations between moves to help speed up the agent.

Actual optimal agents for games as complex as chess are not possible. In these games, you will not have a truly optimal agent, but approximately optimal. You will be relying on some heuristic to guide the minimax search when it cannot force an end game win. If the opponent manages to control play into a state where the heuristics are not accurate, they could cause minimax search to fail and misdirect your agent into making mistakes.

A combination of effects is also possible if you have implemented a tree caching mechanism for performance improvements and made the system playable by limiting planning time - e.g. you limit computer search time to 3 seconds max - an irrational opponent may cause your agent's performance to degrade to the point where it too starts to make mistakes. Whether or not this is enough for a smart opponent to take advantage of it and beat an agent which is capable of otherwise playing a "perfect" game depends on details of the game, and how narrow the agent's measure of "perfect" is.

An extreme case might be an agent that has memorised a single perfect game (by scoring high heuristics for any state on a single tarjectory through the game that has been pre-calculated for perfect moves by both players), and has poor heuristics otherwise - once state moves away from what it can evaluate directly, the agent will be limited by how well it can search for completed game wins, and can easily be manipulated by a more generally smart opponent into a losing position that is beyond the depth of its search.

In practice, well-coded agents will not suffer too much from this effect. If you make a mistake, or try a random probably bad move in an attempt to confuse the agent when playing against Stockfish, you will lose.


Find anomalies from records of categorical data

First of all, you mention that you have categorical data. I don't see how you can define similarity so that you can also define the distance between the predicted value and the ground truth (error). You can do that only if the data are ordinal.

If you want to just classify between normal and anomalous points (binary classification), without caring about further classification of the anomaly types themselves, one of the most common algorithms is the One-Class Support Vector Machine (OC-SVM).

Anomalies are unpredictable in nature and sometimes hard to replicate and record. Therefore, there is usually lack of anomalous data and supervised learning approaches suffer because if you sacrifice some "precious" anomalous points to train the algorithm, you cannot use them to test it.

The main advantage of OC-SVM is that it is semi-supervised learning, meaning that you train it only with normal data and then it can detect samples that deviate from the trained behaviour during testing and classify them as anomalous. Thus, you "save" all the rare anomalous points for testing purposes!

Take a look at this short Python example, it has all you need :)


In this VAE formula, why do $p$ and $q$ have the same parameters?

I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.

  In $logp_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$ why is $\theta$ and param for $p$ and $q$?

In a few words, your equation is wrong because it uses the letters $\phi$ and $\theta$ inconsistently.

If you look more carefully at the right-hand side of your equation, you will notice that $q_{\theta}$ has different parameters, i.e. $\theta$, than $p_{\phi}$, which has parameters $\phi$, so $p$ and $q$ have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. However, the left-hand side uses $\theta$ as the parameters of $p$ (while the right-hand side uses $\phi$ to index $p$), so this should already suggest that the equation is not correct (as you correctly thought).

In the case of the VAE, $\phi$ usually represents the parameters (or weights) of the encoder neural network (NN), while $\theta$ usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in the VAE paper, in equation 3, the authors use $\phi$ to represent the parameters of the encoder $q$, while $\theta$ is used to denote the parameters of the decoder $p$.

So, if you follow the notation in the VAE paper, the ELBO can actually be written something like

\begin{align}
\mathcal{L}(\phi,\theta; \mathbf{x})
&=
\mathbb{E}_{\tilde{z} \sim q_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p_{\theta}(\mathbf{z}) \right) \tag{1} \label{1}
\end{align}

The ELBO loss $\mathcal{L}(\phi,\theta; \mathbf{x})$ has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations $\mathbf{x}$ (for simplicity), while, in the VAE paper, they are present. Furthermore, note that, both in \ref{1} and in the VAE paper, we use bold letters (because these objects are usually vectors), i.e. $\mathbf{x}$ and $\mathbf{z}$, rather than $x$ and $z$ (like in your equation).

Note also that, even though $p_{\theta}(\mathbf{z})$ is indexed by $\theta$, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean $0$ and variance $1$), i.e. not a family of distributions. The use of the index $\theta$ in $p_{\theta}(\mathbf{z})$ comes from the (implicit) assumption that both $p_{\theta}(\mathbf{z})$ and $p_{\theta} (\mathbf{x} \mid \mathbf{z})$ come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then $p_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ also belongs to that family. But $\theta$ and $\phi$ are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see https://stats.stackexchange.com/a/370048/82135).

  Why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$

This is wrong, in fact. If you look at equation 1 of the VAE paper, they use $\theta$ to denote the parameters of $p(\mathbf{x})$, i.e. $p_{\theta}(\mathbf{x})$, while the parameters of the encoder are $\phi$, i.e. $q_{\phi}(\mathbf{z} \mid \mathbf{x}$).

  Cause $p$ is just the probability of the observed data and $q$ is the approximation of the posterior so shouldn't they be different distributions and their parameters different?

Yes.


Why is no activation function needed for the output layer of a neural network for regression?

In regression, the goal is to approximate a function $f: \mathcal{I} \rightarrow \mathbb{R}$, so $f(x) \in \mathbb{R}$. In other words, in regression, you want to learn a function whose outputs can be any number, so not necessarily just a number in the range $[0, 1]$.
You use the sigmoid as the activation function of the output layer of a neural network, for example, when you want to interpret it as a probability. This is typically done when you are using the binary cross-entropy loss function, i.e. you are solving a binary classification problem (i.e. the output can either be one of two classes/labels).
By default, tf.keras.layers.Dense does not use any activation function, which means that the output of your neural network is indeed just a linear combination of the inputs from the previous layer. This should be fine for a regression problem.


How to study the correlation between GAN's input vector and output image

Let me first provide a brief introduction first

GANs as VAEs are generative models which means they learn exactly what you described: to map a typically small dimensional vector/tensor into a higher dimensional one, which in your case is (interpreted as) an image.

These 2 generative models differ in the actual learning strategy which results typically in making GAN outperform VAE in realistic image generation (but it’s better not to get into the reasons for this here)

However in principle the Generator Network in GAN works in a similar way to the Decoder Network in VAE so as their output space is interpreted as an image, you can interpret their input space as a sort of semantic space hence the values you are setting define the semantic of the output image (in fact you observe the features change)

It is possible to say the generative model learns a  Semantic to Appearance Mapping hence it works the opposite way of the classifier model which learns an Appearance to Semantic Mapping

Unfortunately as it typically happens in NN this mapping are definitely hard to understand for humans, nevertheless this is an active area of research

So after this introduction, aimed at giving you an essential understanding of the problem, I can suggest you some papers to get into more details like this one

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

In case you found it not that clear, I’m planning to write a summary of this paper and publish it on my Medium so you could check it there (I’ll update this answer or add a comment to notify about it)


Is there 1-dimensional reinforcement learning?

From what I can find, reinforcement algorithms work on a grid or 2-dimensional environment.

A lot of teaching materials use a "grid world" presentation to demonstrate basic reinforcement learning (RL). However, the underlying Markov Decision Process (MDP) theory works on an arbitrary graph of connected states. This graph could be based on subdividing a metric space of any dimensions into a grid of the same dimensions (and using tiles of any shape that worked in that dimension). However, it is not limited to that, the state space does not need to be a metric that represents distances or physical properties.

In practice, the set of states can be arbitrary objects, connected via state transitions in any consistent way. Provided the transition probability function $p(s'|s,a)$ is consistent, the environment could be used in a RL problem.

A very common state description is that the state is a vector of numbers that capture all the variables relevant to the problem. The environment can then be measurements taken in the real world of those variables, or the same quantities provided by a simulation. That state vector can be of any size, and have arbitrary constraints on individual components. This is no different from numerical representations of other machine learning problems, such as the inputs allowed to a neural network.

  The environment would be a lighthouse beam, the state would be the brightness seen at the sensor for a given orientation, and the agent would be the approximate brightness/orientation?

Something not quite right about the description there. There does not seem to be any action that the agent takes.

  What would the reward be?

It would be whatever measure of reaching a goal or maintaining a "good" result that is appropriate for the problem. You do not give any information about the goal in your description.

If your goal is to light up a moving sensor with the highest brightness, then the brightness measured at the sensor, transformed into suitable units, would seem to be a good candidate for a reward function (you would also need the state to give information about the target - where it had been seen last for instance). Assuming the problem is continuous, you would also need a discount factor.

  What reinforcement learning algorithm would I use to approximate the lighthouse orientation given sensor brightnesses?

Generally RL algorithms estimate rewards, or generate policies. If the lighthouse orientation is the action you wanted to take, then pretty much all RL algorithms can do one or the other to allow you to do this. The differences are in things like complexity or speed of the algorithm, what approximations you are willing to take etc.

You don't give enough information about the problem to even nearly suggest a "best" algorithm. Before you start, you will need to determine a more thorough description of state, action and rewards, that will define the problem. Once you have a more formal description of the problem, that may suggest which algorithms would be good starting points.


What does "stationary" mean in the context of reinforcement learning?

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that the policy is not being updated by a learning algorithm.

If you are working with a stationary policy in reinforcement learning (RL), typically that is because you are trying to learn its value function. Many RL techniques - including Monte Carlo, Temporal Difference, Dynamic Programming - can be used to evaluate a given policy, as well as used to search for a better or optimal policy.

Stationary dynamics refers to the environment, and is an assumption that the rules of the environment do not change over time. The rules of the environment are often represented as an MDP model, which consists of all the state transition probabilities and reward distributions. Reinforcement learning algorithms that work online can usually cope and adjust policies to match non-stationary environments, provided the changes do not happen too often, or enough learning/exploring time is allowed between more radical changes. Most RL algorithms have at least some online component, it is also important to keep exploring non-optimal actions in environments with this trait (in order to spot when they may become optimal).

Stationary data is not a RL-specific term, but also relates to the need for an online algorithm, or at least plans for discarding older data and re-training existing models over time. You might have non-stationary data in any ML, including supervised learning - prediction problems that work with data about people and their behaviour often have this issue as population norms change over timescales of months and years.


Is there a possibility that there is no relationship between some inputs and outputs?

Of course, it's possible to define a problem where there is no relationship between input $x$ and output $y$. In general, if the mutual information between $x$ and $y$ is zero (i.e. $x$ and $y$ are statistically independent) then the best prediction you can do is independent of $x$. The task of machine learning is to learn a distribution $q(y|x)$ that is as close as possible to the real data generating distribution $p(y|x)$.

For example, looking at the common cross-entropy loss, we have
$$
\begin{align}
H(p,q) = -\mathbb{E}_{y,x \sim p}\left[\log q(y|x)\right] & = \mathbb{E}_{x\sim p}\left[\text{H}(p(y|x)) + \text{D}_{\text{KL}}(p(y|x)\|q(y|x))\right]  \\
& = \text{H}(p(y)) + \mathbb{E}_{x \sim p}\left[\text{D}_{\text{KL}}(p(y)\|q(y|x))\right],
\end{align}
$$
where we have used the fact that $p(y|x)=p(y)$ since $y$ and $x$ are independent. From this, we can see that the optimal predicted distribution $q(y|x)$ is equal to $p(y)$, and actually independent of $x$. Also, the best loss you can get is equal to the entropy of $y$.


Why does the policy network in AlphaZero work?

The output of the policy network is as described in the original paper:

  A move in chess may be described in two parts: selecting the piece to move, and then
  selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
  stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
  positions identifies the square from which to “pick up” a piece. The first 56 planes encode
  possible ‘queen moves’ for any piece: a number of squares [1..7] in which the piece will be
  moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The
  next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible
  underpromotions for pawn moves or captures in two possible diagonals, to knight, bishop or
  rook respectively. Other pawn moves or captures from the seventh rank are promoted to a
  queen.

So each move selector scores the relative probability of selecting a piece in a given square and moving it in a specific way. For example, there is always one output dedicated to representing picking up the piece in A3 and moving it to A6. This representation includes selecting opponent pieces, selecting empty squares, making knight moves for rooks, making long diagonal moves for pawns. It also includes moves that take pieces off the board or through other blocking pieces.

The typical branching factor in chess is around 35. The policy network described above always calculates discrete probabilities for 4672 moves.

Clearly this can select many non-valid moves, if pieces are not available, or cannot move as suggested. In fact it does this all the time, even when fully trained, as nothing is ever learned about avoiding the non-valid moves during training - they do not receive positive or negative feedback, as there is never any experience gained relating to them. However, the benefit is that this structure is simple and fixed, both useful traits when building a neural network.

The simple work-around is to filter out impossible moves logically, setting their effective probability to zero, and then re-normalise the probabilities for the remaining valid moves. That step involves asking the game engine for what the valid moves are - but that's fine, it's not "cheating".

Whilst it might be possible to either have the agent learn to avoid non-valid moves, or some clever output structure that could only express valid moves, these would both distract from the core goal of learning how to play the game optimally.


Which API can I use for tracking the position of animal in one or more images?

FastAI is the most “out of the box” API for this type of task.

For video examples (and a little theory) check out the MOOC section of their site.

Practical Deep Learning and Cutting Edge Deep Learning are the two sections most relevant to you.

But if you want a working implementation check out this GitHub repo that implements SSD for your purposes. I can’t say how simple the API is but it does what you are seeking (in pure PyTorch).

note: FastAI was originally built on top of PyTorch (although they are expanding out now), so you’ll be using PyTorch and need a rough idea of how to work with tensors. Most of the challenges are already implemented in their api. Installing their library will automatically install PyTorch.


Why feed actions in later layer in Q network?

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Yes.

  Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.

So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.

Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.

I can try a few guesses:

Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.
Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.
You don't want the neural network to construct a reverse map $\pi(a|s) \rightarrow Q(s,a)$, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).

  Is this just a small little tweak or a more significant improvement?

I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.


What is the intuition behind the number of filters/channels for each convolutional layer?

The channel sizes 32, 128, etc. are used because of memory and efficiency. There is nothing holy about these numbers.
The intuition behind choosing the number of channels is as follows-
The initial layers extract low-level features- they consist of edge detectors, etc. There aren't many such features. So, we won't gain much by adding a lot of filters(of course, if we use a 3x3 filters on an RGB image, we would have 2^27 different filters even if our neurons have only 0 and 1 as their values. However, most of them are quite similar/meaningless for our job). Using a lot of filters might even lead to overfitting.
The latter layers are responsible for detecting more nuanced features, like elbows/nose shape from the lower level features extracted previously. So, we might do better if we increase the number of channels. Also, note that the resultant layers become more and more sparse as we go deeper.
Though it might differ in applications like super resolution image, in general, the number of channels stays the same or increases when we go deeper.
A nice experiment would be to try and increase the number of channels until you get no more benefit from it. I believe there was a paper that did exactly this(please cite it if someone remembers). You could even try to visualise the filters at this stage and see if the filters are similar or not.
Hope it helps.


How is dropout applied to the embedding layer's output?

It doesn't drops rows or columns, it acts directly on scalars. The Dropout Layer keras documentation explains it and illustrates it with an example :

The Dropout layer randomly sets input units to 0 with a frequency of rate

After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, in your case, you should have rate * (16 * input_length) = 0.2 * 20 * 16 = 64 inputs set to 0 out of the 320 scalars inputs. These 64 dropped inputs are randomly selected in the 20x16 grid. Note that the Dropout rescales the non dropped inputs by multiplicating them by a factor $\frac{1}{1-rate}$.


Support Vector Machine Convert optimisation problem from argmax to argmin

So actually I managed to get hold of my lecturer to explain the argmax to argmin conversion.
Generally speaking maximising $\frac{1}{||w||}$ is identical to minimising $||w||$. As $||w||$ in $\frac{1}{||w||}$ decreases, the overall value increases, i.e. we maximise it. The reason for choosing $\frac{1}{2}||w||^2$ turns out to be a less mathematic and more practical one as our optimisation algorithms used perform better on quadratic functions and the $\frac{1}{2}$ seems to be a rather arbitrary scaling choice.
If anyone has something to add, I'd love to hear any details though!


Does augmenting data changes the distribution of augmented data?

Yes you do change the distribution of your training data if you modify it (for example by augmenting it with rotated versions of images that were in an original training set of images).

This is fine because, typically, our goal of training is not to get a model with high performance on the dataset we happened to collect as training data (e.g. a bunch of natural images). Our goal is to train a model that generalizes well to new data outside of the training data's distribution.

Typically, the training data is only a sample of the distribution we're actually interested in. For example, we'll be interested in making accurate predictions for all natural images in the entire world. That is a distribution that would likely include rotated variants of all images in our training set. So, if we augment our training set by adding such rotated variants, we expect to modify our training data distribution in such a way that it actually gets a little bit closer to the distribution we're interested in (all natural images).


Could I just choose the other (non-predicted) class when the accuracy is low?

The short answer is no, you shouldn't do that.
There is a "distribution shift" thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but what about novel test data? Will it be like a train? Will it be like validation? You don't know and your model is worthless in fact.
What you can do

You should redefine the train/validation scheme. I would recommend making cross-validation with at least three splits
As mentioned in a comment, your model seems to overfit; try to apply some regularization techniques, like dropout, batch norm, data augmentation, model simplification and so on
Add some data if it's possible, that would better cover all the possible cases.


How does adjusted fitness penalize large species in NEAT?

Consider what would the outcome be if offspring was assigned on the basis of fitness (unadjusted). Sum of A fitnesses would be 40 and B=18. Fitness ratio for both species would be 2.(2):1. In case of adjusted fitness the numbers are A=15 and B=9, which gives ratio of 1.(6):1, thus A is assigned less offspring based on adjusted fitness then unadjusted.

Also note that every new genome assigned to a species decreases all it's members adjusted fitness. In your case members of species A are more successful than members of B, so it should grow. The mechanism is designed to hinder growth of successful species, not to block it entirely. This allows more diversity, which is important when a successful species reaches a dead end and a previously less successful one can take over.


What is the difference between strong-AI and weak-AI?

The terms strong and weak don't actually refer to processing, or optimization power, or any interpretation leading to "strong AI" being stronger than "weak AI". It holds conveniently in practice, but the terms come from elsewhere. In 1980, John Searle coined the following statements:

AI hypothesis, strong form: an AI system can think and have a mind (in the philosophical definition of the term);
AI hypothesis, weak form: an AI system can only act like it thinks and has a mind.

So strong AI is a shortcut for an AI systems that verifies the strong AI hypothesis. Similarly, for the weak form. The terms have then evolved: strong AI refers to AI that performs as well as humans (who have minds), weak AI refers to AI that doesn't.

The problem with these definitions is that they're fuzzy. For example, AlphaGo is an example of weak AI, but is "strong" by Go-playing standards. A hypothetical AI replicating a human baby would be a strong AI, while being "weak" at most tasks.

Other terms exist: Artificial General Intelligence (AGI), which has cross-domain capability (like humans), can learn from a wide range of experiences (like humans), among other features. Artificial Narrow Intelligence refers to systems bound to a certain range of tasks (where they may nevertheless have superhuman ability), lacking capacity to significantly improve themselves.

Beyond AGI, we find Artificial Superintelligence (ASI), based on the idea that a system with the capabilities of an AGI, without the physical limitations of humans would learn and improve far beyond human level.


Why did fuzzy logic fall out of fashion?

Fuzzy logic is not down trending.  It's a common architecture selection for systems that require the representation of uncertainty in changing rules.  The domains include work-flow control, aeronautics, chemical plant engineering, automated defense of cyber-attack, building systems, and business intelligence.

In natural language processing, it's up-trending under other names.  Semantic nets are a logical structure with strengths assigned between nodes of the net based on meta rules, which is reminiscent of early fuzzy logic software.

For instance, the rule might be that a word that ends with -ly before a word associated with action may be an adverb modifying a verb in English, but it is not always the case.  The rule's application is not smooth, meaning non-probabilistic.  It is fuzzy.

There is no way to create a grammar for common speech or writing in any natural language.  Language is dynamic and as practically non-deterministic as wind vectors.  That's why grammar is a down-trending word among linguists and people working in NLP.

Cognition is expected to require fuzzy valuations to the black and white nature of early production systems developed at MIT, Cambridge, the U.S. Navy, Yale U, and other locations in Eurasia.  People develop rules in their cortex and they rise and drop in probability when decisions are made after the rule seemed to have worked or didn't.

The logic that works to prove a math theorem may require discrete inference, but people are smart when they can weigh alternatives and apply rules in unique ways to partial data and limited experience.


What is the use of the regular convolutional layer in expansion path of U-Net?

The point is that in the expansive path you have two forms of information:

the information from the contracting path, which includes all high-level features extracted from the original image.
the information from the skip-connections, which copy a cropped version of the feature maps in the contracting path. Because, as we go forward through the expansive path these originate from earlier stages in the contracting path, these are progressively richer in detail.

Intuitively you can think of it as this: high-level features help the network tell which areas to group together, while details help it tell where each group starts and ends at the pixel level.
The idea is to combine these two forms of information, i.e. the high-level features and the details, optimally.  To do this you need trainable layers that learn this optimal combination. Here is where the convolution layers come to play.


Has it been theoretically proven or dis-proven that a machine can generally think and communicate on its own?

We have not been able to create a truly intelligent AI yet, according to your definition. So we have no real life proof-of-concept that shows that it actually works.

But based on the current research, there is no known property of the human brain that cannot be modeled in software/hardware. We do not understand the human brain enough yet - and most likely lack the computational power and required storage - to create an artificial copy. But as long as we do not identify a property of the human brain that cannot be translated into a piece of software/hardware, it is most likely possible. I honestly cannot even imagine such a property within the bounds of physics that would create such a barrier.

Of course this is no formal proof, but without a clearly identified show-stopper there is no reason to believe that it is not possible to create a thinking machine.


Data-set values feature scaling: sigmoid vs tanh

Yes it applies, and no it shouldn't matter that much between the two activation functions.


Method to compute the sum when the activation is a continuous function?

In terms of the normal use cases for machine learning, the equation does not have much utility, because:

  Consider we have a curve $f(x)$ now if one wishes to  . . .

In most AI problems, we don't usually have such a curve as input that can be treated analytically. For instance, there is no such input curve to describe a natural image received by a sensor.

In the vast majority of cases in AI problems, the form of inputs - whether it is images, text, mapping data, robotic telemetry, is going to be multi-dimensional discrete samples from highly complex functions where we don't know the analytical form and can only construct approximations from a set of basis functions. The resulting combination of basis functions could be treated as continuous, integrated etc, but as it would have been constructed from discrete data, the end result would be a lot of computation to end up with something probably less accurate than working direct with the discrete samples. In a lot of cases, the raw data is discrete by definition (e.g. whether someone clicked on a link or replied to a message), so the form of $f(x)$ would be discrete by definition of the problem, and calculus not really applicable.

There might be some interesting use cases in analog signal processing. Using your formula or a variation of it for instance it should be possible to create an analog neural network learning system - a robotic brain that worked with continuous signals and only contained analog components. Such a system would have some interesting advantages - speed of processing, probably low power consumption compared to digital approach, but the accuracy and precision might be lower. E.g. imagine a bot that could steer towards/away from light sources (compared to a digital one that might recognise the faces of people and steer towards/away from them). I would not be at all surprised to find that someone had done just that already, although I am not sure how to search for it.


Is there a proof to explain why XOR cannot be linearly separable?

Before proving that XOR cannot be linearly separable, we first need to prove a  lemma:
Lemma 1
Lemma: If 3 points are collinear and the middle point has a different label than the other two, then these 3 points cannot be linearly separable.

Proof: Let us label the points as point $A$, $B$, and $C$. $A$ and $C$ have the same label, and $B$ has a different label. They are all collinear with line $\mathcal{L}$.
Assume the contradiction, so a line can linearly separate $A$, $B$, and $C$. This means a line must cross between segment $AB$, and segment $BC$ to linearly separate these three points (by definition of linear separability). Let us label the point where the line crosses segment $AB$ and point $Y$, and the point where the line corsses segment $BC$ and point $Z$.
However, since segments $AB$ and $BC$ are collinear to line $\mathcal{L}$, points $Y$ and $Z$ also falls on line $\mathcal{L}$. Since only one unique line can cross 2 points, it must be that the only line that passes segments $AB$ and $BC$ and (therefore separates points $A$, $B$, and $C$) is line $\mathcal{L}$.
However, line $\mathcal{L}$ cannot linearly separate $A$, $B$, and $C$, since line $\mathcal{L}$ also crosses them. Therefore, no line exists can separate $A$, $B$, and $C$.
Main proof

Consider these 4 points that represent a XOR table. Let us label them clock-wise, so the top-left point as $A$, top-right point as $B$, bottom-right point as $C$, and bottom-left point as $D$. So $A$ and $C$ have the same label, and $B$ and $D$ have the same label. We want to show that points $A, B, C$ and $D$ cannot be linearly separable.
Assume the contradiction, and that there is a line that can separate these 4 points.
Imagine a fifth point that lies in the center, and let us label this as point $E$.
Since $E$ lies in the center, the three points $A, E$ and $C$ are collinear. Similarly, since $E$ lies in the center, the three points $B, E$ and $D$ are collinear.
Because we assume a line can linearly separate $A, B, C$ and $D$, then this line must label point $E$ as some label. If $E$ shares the same label as $A$ and $C$, then the points $B, E$ and $D$ will become "collinear points where the middle point has a different label", which by Lemma 1 cannot be linearly separable. Likewise, if $E$ shares the same label as $B$ and $D$, then the points $A, E$ and $C$ will become "collinear points where the middle point has a different label", which by Lemma 1 cannot be linearly separable.
Therefore it is impossible to give a label to $E$ while satisfying linear separability. As a result, our assumption must be false, and the four points $A, B, C$ and $D$ cannot be linearly separable.


Why is the status of artificial life software so under-developed?

I don’t know for certain, but I can make a guess. This is just my opinion, some others may disagree.
The field of ALife has four branches that I’m aware of:
Self-Organizing/self assembly behavior. This is the application you refer to, another context it’s useful is swarm control (for drone swarms, for example). While this is technically ALife, as far as I’m aware it’s not really where most of the emphasis is. Swarm control and self assembly are seen as “different” problems, as machines that can work together and also build more of themselves is interesting (and potentially dangerous), but is missing out on the diversity, the open-endedness that life on earth has. Much of ALife research is focused on trying to formally define this open-endedness and coming up with systems that achieve that. Self assembly and swarm control are interesting and difficult problems, just different. This leads to the other three sides of ALife research:
Coming up with environments, and running tests on them. This is a constant game of coming up with a definition that seems to capture open-endedness, then coming up with ALife sims that meet that criteria but fall short of our expectations. So new definitions are made and we repeat. Geb is a classic example: Geb has passed pretty much every test so far, but it’s fairly uninspiring to watch. Most of those programs you reference chose a particular ALife paradigm, but that paradigm may not be the right one, and is often disappointing. Because we still haven’t found something that really “looks like life”, new paradigms and programs are constantly being created and abandoned when they fail to work (Or perhaps some would have already worked, but the computing time is too much). That’s what you’re seeing. Without any unifying theory or sim that is really convincing, I suspect it’ll stay this way for a while. And because:

we still haven’t made much progress since Karl Sims in the 90s, or since Geb (this point is debatable)
these sorts of sims don’t really have much commercial use aside from games

the direction of making new simulators seems to be lacking funding and research interest, as far as I can tell. Commercial sim games seem to push the boundary these days.
Fortunately there’s a sub field of cellular automata life that’s pretty interesting, its software is slightly more developed due to the overlap with cellular automata and ease of implementation, and research seems to be progressing there at an okay rate.
Realistically, there seem to be two things people want: novel behavior, and novel bodies. My two cents is that these are separate problems, and achieving both is more expensive than just achieving one. But most of these sims end up not balancing development happening in both of these factors (doing this is very difficult), so one factor develops much further than the other, and this disconnect is disappointing to the sim creator. For example, Geb does behavioural diversity really well, while Karl Sims does body diversity well. Sensitivity to small details like mutation rate or genetic encoding also can be quite frustrating. Fortunately, eventually we’ll sorta get behavioural diversity for free in any sim once RL/AI is really understood well.
The third piece of ALife research I’m aware of is the theoretical side, which right now mostly isn’t really far enough along to warrant practical implementation. One big branch of this is the learning theory side, represented by Valiant’s Evolvability theory and followups. Essentially this talks about what functions are possible to evolve, and using stuff like PAC Learning theory they are able to prove some things. Some of these models are more natural than others, but it’s an interesting perpendicular approach to coming up with sims and seeing if they do what we want. Maybe eventually these two approaches will meet in the middle at some point, but they haven’t yet.
The fourth piece is Artificial Chemistry. I recommend this paper as a somewhat dated overview. While this is technically a field of ALife, and is centered around understanding a chemical system that has the necessary emergent properties, it has broken off into applications that may have industrial relevance. For example, robust self repairing and self assembling electronic systems, DNA computing (DNA is capable of simulating arbitrary chemical reaction networks which are capable of arbitrary computing), and artificial hormone systems for automatic task assignment. This has some software developed, but much of that software isn’t really considered ALife anymore since it has branched off into its own domain.


What exactly is averaged when doing batch gradient descent?

Introduction

First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't completely answer the original question. In fact, I leave some unanswered questions at the end (that I will eventually answer).

The gradient is a linear operator

The gradient operator $\nabla$ is a linear operator, because, for some $f : \mathbb{R} \rightarrow \mathbb{R} $ and $g: \mathbb{R} \rightarrow \mathbb{R}$, the following two conditions hold.

$\nabla(f + g)(x) = (\nabla f)(x) + (\nabla g)(x),\; \forall x \in \mathbb{R}$
$\nabla(kf)(x) = k(\nabla f)(x),\; \forall k, x \in \mathbb{R}$

In other words, the restriction, in this case, is that the functions are evaluated at the same point $x$ in the domain. This is a very important restriction to understand the answer to your question below!

The linearity of the gradient directly follows from the linearity of the derivative. See a simple proof here.

Example

For example, let $f(x) = x^2$, $g(x) = x^3$ and $h(x) = f(x) + g(x) = x^2 + x^3$, then $\frac{dh}{dx} = \frac{d (x^2 + x^3)}{d x} = \frac{d x^2}{d x} + \frac{d x^3}{d x} = \frac{d f}{d x} + \frac{d g}{d x} = 2x + 3x$.

Note that both $f$ and $g$ are not linear functions (i.e. straight-lines), so the linearity of the gradients is not just applicable in the case of straight-lines.

Straight-lines are not necessarily linear maps

Before proceeding, I want to note that there are at least two notions of linearity.

There's the notion of a linear map (or linear operator), i.e. which is the definition above (i.e. the gradient operator is a linear operator because it satisfies the two conditions, i.e. it preserves addition and scalar multiplication).
There's the notion of a straight-line function: $f(x) = c*x + k$. A function can be a straight-line and not be a linear map. For example, $f(x) = x+1$ is a straight-line but it doesn't satisfy the conditions above. More precisely, in general, $f(x+y) \neq f(x) + f(y)$, and you can easily verify that this is the case if $x = 2$ and $y=3$ (i.e. $f(2+3) = 6$, $f(2) = 3$, $f(3) = 4$, but $f(2) + f(3) = 7 \neq f(2+3)$.

Neural networks

A neural network is a composition of (typically) non-linear functions (let's ignore the case of linear functions), which can thus be represented as $$y'_{\theta}= f^{L}_{\theta_L} \circ f^{L-1}_{\theta_{L-1}} \circ \dots \circ f_{\theta_1},$$ where

$f^{l}_{\theta_l}$ is the $i$th layer of your neural network and it
computes a non-linear function
${\theta_l}$ is a vector of parameters associated with the $l$th layer
$L$ is the number of layers,
$y'_{\theta}$ is your neural network,
$\theta$ is a vector containing all parameters of the neural network
$y'_{\theta}(x)$ is the output of your neural network
$\circ $ means the composition of functions

Given that $f^l_{\theta}$ are non-linear, $y'_{\theta}$ is also a non-linear function of the input $x$. This notion of linearity is the second one above (i.e. $y'_{\theta}$ is not a straight-line). In fact, neural networks are typically composed of sigmoids, ReLUs, and hyperbolic tangents, which are not straight-lines.

Sum of squared errors

Now, for simplicity, let's consider the sum of squared error (SSE) as the loss function of your neural network, which is defined as

$$
\mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^N  \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) = \sum_{i=1}^N (\mathbf{y}_i - y'_{\theta}(\mathbf{x}_i))^2
$$
where

$\mathbf{x} \in \mathbb{R}$ and $\mathbf{y} \in \mathbb{R}$ are vectors of inputs and labels, respectively
$\mathbf{y}_i$ is the label for the $i$th input $\mathbf{x}_i$
$\mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) = (\mathbf{y}_i - y'_{\theta}(\mathbf{x}_i))^2$

Sum of gradients vs gradient of a sum

Given the gradient is a linear operator, one could think that computing the sum of the gradients is equal to the gradient of the sums.

However, in our case, we are summing $\mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i)$ and, in general, $\mathbf{x}_i \neq \mathbf{x}_j$, for $i \neq j$. So, essentially, the SSE is the sum of the same function, i.e. $S_{\theta}$, evaluated at different points of the domain. However, the definition of a linear map applies when the functions are evaluated at the same point in the domain, as I said above.

So, in general, in the case of neural networks with SSE, the gradient of the sum may not be equal to the sum of gradients, i.e. the definition of the linear operator for the gradient doesn't apply here because we are evaluating every squared error at different points of their domains.

Stochastic gradient descent

The idea of stochastic gradient descent is to approximate the true gradient (i.e. the gradient that would be computed with all training examples) with a noisy gradient (which is an approximation of the true gradient).

How does the noisy gradient approximate the true gradient?

In the case of mini-batch ($M \leq N$, where $M$ is the size of the mini-batch and $N$ is the total number of training examples), this is actually a sum of the gradients, one for each example in the mini-batch.

The papers Bayesian Learning via Stochastic Gradient Langevin Dynamics (equation 1) or Auto-Encoding Variational Bayes (in section 2.2) use this type of approximation. See also these slides.

Why?

To give you some intuition of why we sum the gradients of the error of each input point $\mathbf{x}_i$, let's consider the case $M=1$, which is often referred to as the (actual) stochastic gradient descent algorithm.

Let's assume we uniformly sample an arbitrary tuple $(\mathbf{x}_j, \mathbf{y}_j)$ from the dataset $\mathcal{D} = \{ (\mathbf{x}_i, \mathbf{y}_i) \}_{i=1}^N$.

Formally, we want to show that

\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y})
&=
\mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}\left[ \nabla_{\theta} \mathcal{S}_{\theta} \right] \label{1} \tag{1}
\end{align}

where

$\nabla_{\theta} \mathcal{S}_{\theta}$ is the gradient of $\mathcal{S}_{\theta}$ with respect to the parameters $\theta$
$\mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}$ is the expectation with respect to the random variable associated with a sample $(\mathbf{x}_j, \mathbf{y}_j)$ from the uniform distribution $\mathbb{U}$

Under some conditions (see this), we can exchange the expectation and gradient operators, so \ref{1} becomes
\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta}(\mathbf{x}, \mathbf{y})
&=
\nabla_{\theta} \mathbb{E}_{(\mathbf{x}_j, \mathbf{y}_j) \sim \mathbb{U}}\left[  \mathcal{S}_{\theta} \right]
\label{2} \tag{2}
\end{align}
Given that we uniformly sample, the probability of sampling an arbitrary $(\mathbf{x}_j, \mathbf{y}_j)$ is $\frac{1}{N}$. So, equation \ref{2} becomes
\begin{align}
\nabla_{\theta} \mathcal{L}_{\theta} (\mathbf{x}, \mathbf{y})
&=
\nabla_{\theta} \sum_{i=1}^N \frac{1}{N} \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i) \\
&=
\nabla_{\theta} \frac{1}{N}  \sum_{i=1}^N \mathcal{S}_{\theta}(\mathbf{x}_i, \mathbf{y}_i)
\end{align}

Note that $\frac{1}{N}$ is a constant with respect to the summation variable $i$ and so it can be taken out of the summation.

This shows that the gradient with respect to $\theta$ of the loss function $\mathcal{L}_{\theta}$ that includes all training examples is equivalent, in expectation, to the gradient of $\mathcal{S}_{\theta}$ (the loss function of one training example).

Questions

How can we extend the previous proof to the case $1 < M \leq N$?
Which conditions need exactly to be satisfied so that we can exchange the gradient and the expectation operators? And are they satisfied in the case of typical loss functions, or sometimes they aren't (but in which cases)?
What is the relationship between the proof above and the linearity of the gradient?

In the proof above, we are dealing with expectations and probabilities!

What would the gradient of a sum of errors represent? Can we still use it in place of the sum of gradients?


How can I design a hierarchy of agents each of which with different goals?

In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called hierarchical reinforcement learning, which gives rise to concepts such as semi-Markov decision processes and options (aka macro actions). The article The Promise of Hierarchical Reinforcement Learning presents and describes the topic quite well, so I suggest you read it.
However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper Neural Programmer-Interpreters (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.
Another example is genetic programming with automatically defined functions (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.
So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. MAXQ-OP) have been successfully used to solve the RoboCup challenge.


How should I encode the structure of a neural network into a genome?

Section 4.2 of "Essentials of Metaheuristics" has a wealth of information on alternative ways of encoding graph structures via Genetic Algorithms.

With particular regard to evolving ANNs, I would personally not be inclined to implement this sort of thing 'from scratch':

The field of neuroevolution has been around for some time, and the implementation some of the methods, such as Neuroevolution of Augmenting Topologies (NEAT) now incorporate the results of much practical experience.

According to the above link:

  We also developed an extension to NEAT called HyperNEAT that can evolve neural networks with millions of connections and exploit geometric regularities in the task domain. The HyperNEAT Page includes links to publications and a general explanation of the approach.


What is the calcium equivalent role in neural networks

Neural networks don't model biological neurons.

They are at best inspired by biological neurons, in that they get excited by certain inputs and fire once the excitation crosses a threshold. And this second point even holds only approximately because the backpropagation algorithm needs smoothed out steps to learn by gradient descent. And backpropagation is not even inspired by biology, that would rather be hebbian learning.

Generally in machine learning people do what works. The connection to biology is tenuous at best. You will usually not find one to one correspondence of low level details between machine learning setups and biological neurons. For that you'll have to turn to brain simulations.


Approaches to poker tournament winner prediction?

This looks like it maps very nicely onto Association Mining. In association mining, you are trying to find items from a discrete set that often co-occur in transactions. For instance, you might want to find the items that most commonly appear in an online shopping cart together.

In your case, the problem amounts to:

Split up the data into sub-problems by who won.
Perform association mining on the sets of players in each sub-problem, using, e.g. the apriori algorithm.

The resulting rules will then have associated confidence factors. When you want to predict who will win, you can take the winner suggested by the rule with the greatest confidence.

The other approach suggested by your problem is to model it as classification. Here you are trying to assign labels (who won) to input vectors (who played). The process would look like:

Map the sets of who played into binary features. So the subset P1,P2, P4,P8 might be represented with {1,1,0,1,0,0,0,1} if there are only 8 players. Map the winning player onto a numeric class (e.g. the numbers 1-8).
Run any classification algorithm to create a model that predicts the class from the binary features. A decision tree learner might be a interesting starting point if you want to understand which factors are important to the model. There are many other techniques however.

You can use the model you train to make predictions about future games with the same set of players.


How Machine Learning Studies Correlation?

Welcome to AI.SE @Laura!

It sounds like you have a series of data points, each with 10 related measurements, and you want automatically assess which of the measurements are most closely related to each other.

You are right that the correlation coefficient is a good choice for this.

Other techniques used in some AI algorithms include the Information Gain measurement (where you measure the reduction in entropy of one variable that follows from partitioning on another one), and embedded feature selection approaches, like the one in this paper.


Is there any artificial intelligence that possesses "concentration"?

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of competing possibilities, tend to maintain interest in the one that is most compelling. Although the salience value of (part of) a solution is ultimately represented numerically, the means by which it determined is broadly intended to correspond (at least functionally) to the way 'selective attention' might operate in human cognition.


Can neuro-evolution methods be combined with A3C?

Is such an idea viable?

Yes.

One approach that should work in terms of underlying theory is to start with population of NEAT-generated networks that describe the policy, and instead of measuring fitness of them on the task whilst keeping all weights static, measure fitness whilst applying any policy gradient algorithm like A3C. In addition, the final weights of the networks could be fed into the next generation. A bit Lamarckian* perhaps, but that is already a thing in evolutionary algorithms.

  Has it been tried?

Yes, a recent paper (July 2018) is "NEAT for large-scale reinforcement learning through evolutionary feature learning and policy gradient search".

I suspect there are more, including hobby efforts without academic publishing, but that is the first paper that popped up on a brief search.

* Jean-Baptiste Lamarck believed that offspring of animal parents could inherit traits based on parental behaviour and desires, such as a giraffe's neck evolving over time as the animals strived to reach higher food sources. Interestingly, although the core of this theory is not generally accepted, recent theories around phenotypic plasticity and discoveries in epigenetics show that biological systems can make use of the idea - although very far from the idea of children using parental memories directly to aid in tasks (which has more in common with science fiction for humans, but possible in neural networks where we can "copy brains").


How to show Sauer's Lemma when the inequalities are strict or they are equalities?

I am not good with math notations so I'll stick to language more.

The first equality holds when all hypotheses shattering a set of certain dimension (whose superset has not been shattered) are non overlapping with other hypotheses shattering similar sets.

Informally let' see with an example what this means. Let's say $\mathcal H = \{h_1, h_2,h_3, h_4, h_5, h_6\}$ and all have unique behaviours on $A$ which implies $|\mathcal H_A| = |\mathcal H|$. Now if $\{h_1, h_2,h_3, h_4\}$ shatters a subset $B_1$ of size $2$ then it also shatters all subsets of $B_1$ and number of subsets including $B_1$ itself of $B_1$ is $2^{|B_1|} = 4$ (in this case). Similarly, $\{h_5 h_6\}$ must shatter another subset $B_2$ with size $=1$ and using similar argument as before we get subsets $=2$. Which shows $2+4=6$ (i.e $B_1$ and $B_2$ and all its subsets form the RHS of the equation whose total number is $6$).

NOTE: This will also hold true for hypothesis having same behaviour on subset $A$ i.e if another hypothesis $h_7$ exist which give same output on $A$ as $h_6$. There will be no effect since the former derivation is based on $\mathcal H_A$ and not $\mathcal H$

The case during which $|\mathcal H_A| < |\{B \subseteq A: \mathcal{H} \text { shatters } B\} |$ , is when we have overlapping hypothesis i.e if certain hypothesis exists which helps in shattering $2$ different subsets none of which is subset of each other. Which means now if  $\{h_1, h_2,h_3, h_4\}$ shatters some $B_1$ and $\{h_4, h_5\}$ with the overlap of $h_4$ shatters another hypothesis then this is the case $<$ will hold.

Now coming to the second part of the equation, we can see equality will hold when the hypothesis shatters all possible subsets of size $\leq d$. If we define a hypothesis as a set of points where the values is $1$ and $0$ otherwise, then we can verify this. Clearly, if $\mathcal H$ shatters all sets of size $\leq d$ then we can pick any set of size $i \leq d$, and total number of such sets will be $\sum_{i=0}^d \begin{array}{c}{|A|} \\ {i}\end{array}$. In all other cases $<$ will hold.

NOTE: A hypothesis class having VC dimension $d$ does not necessarily shatter all sets of size less $\leq d$ and hence the inequality. Check tis answer.

You can now play with the different scenarios to get $4$ cases as required, and as can  be clearly seen the assumptions for equality in both the part of the equations are independent of each other. One deals with overlapping behaviour, while other deals with shattering of subsets $\leq d$. But note only under a certain condition will the equality hold on both sides, namely the hypothesis class shatters all possible subsets of size $d$ non overlappingly.

Sidenote: The first part of the equation can probably be proven by formal number theory, but I am not sure how.


Could an AI think laterally while avoiding "ethically suboptimal" choices?

No, with a but. We can have creative yet ethical problem-solving if the system has a complete system of ethics, but otherwise creativity will be unsafe by default.

One can classify AI decision-making approaches into two types: interpolative thinkers, and extrapolative thinkers.

Interpolative thinkers learn to classify and mimic whatever they're learning from, and don't try to give reasonable results outside of their training domain. You can think of them as interpolating between training examples, and benefitting from all of the mathematical guarantees and provisos as other statistical techniques.

Extrapolative thinkers learn to manipulate underlying principles, which allows them to combine those principles in previously unconsidered ways. The relevant field for intuition here is numerical optimization, of which the simplest and most famous example is linear programming, rather than the statistical fields that birthed machine learning. You can think of them as extrapolating beyond training examples (indeed, many of them don't even require training examples, or use those examples to infer underlying principles).

The promise of extrapolative thinkers is that they can come up with these 'lateral' solutions much more quickly than people would be able to. The problem with these extrapolative thinkers is that they only use the spoken principles, not any unspoken ones that might seem too obvious to mention.

An attribute of solutions to optimization problems is that the feature vector is often 'extreme' in some way. In linear programming, at least one vertex of the feasible solution space will be optimal, and so simple solution methods find an optimal vertex (which is almost infeasible by nature of being a vertex).

As another example, the minimum-fuel solution for moving a spacecraft from one position to another is called 'bang-bang,' where you accelerate the craft as quickly as possible at the beginning and end of the trajectory, coasting at maximum speed in between.

While a virtue when the system is correctly understood (bang-bang is optimal for many cases), this is catastrophic when the system is incorrectly understood. My favorite example here is Dantzig's diet problem (discussion starts on page 5 of the pdf), where he tries to optimize his diet using math. Under his first constraint set, he's supposed to drink 500 gallons of vinegar a day. Under his second, 200 bouillon cubes. Under his third, two pounds of bran. The considerations that make those obviously bad ideas aren't baked into the system, and so the system innocently suggests them.

If you can completely encode the knowledge and values that a person uses to judge these plans into the AI, then extrapolative systems are as safe as that person. They'll be able to consider and reject the wrong sort of extreme plans, and leave you with the right sort of extreme plans.

But if you can't, then it does make sense to not build an extrapolative decision-maker, and instead build an interpolative one. That is, instead of asking itself "how do I best accomplish goal X?" it's asking itself "what would a person do in this situation?". The latter might be much worse at accomplishing goal X, but it has much less of the tail risk of sacrificing other goals to accomplish X.


Is there any system that generates website designs?

The closest research that I am aware of is in artificial intelligence designed user interfaces(1,2).

The scope of which vary tremendously. Some teams are trying to generate UIs based on some user-defined parameters, others are trying to generate based off of images(as in the second link). I think part of the reason research is focused here is two-fold: One, we aren't very good at generating UIs automatically yet, and 2, it is an exponentially harder problem to develop a backend that has integrated calls and features required for a fully-featured website or application.

With that being said, one could adapt this research into things like one-page sites or other simple implementations that might bear fruit.


Can a NN be configured to indicate which points of the input influenced its prediction and how?

Yes there definitely is, and research into this has actually resulted in some really cool behaviour.

One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.

Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).

You can also determine what each filter in a convolutional layer is looking at using similar techniques.

It's a super interesting field of machine learning, and I highly recommend taking a look at this lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.


Do GANs come under supervised learning or unsupervised learning?

The terms Supervised Learning and Unsupervised Learning predate the invention of the application of artificial networks to a generative and discriminative network pair, which was the first popular generative topology.  The existence of labeling is the key distinction between the two.  Even partial labeling indicates supervision, as odd as that jargon is, since the supervisor does no learning and the labels are constants.

Based on the original description of the discriminative network in a GAN, that it consumes examples without labels, GANs are a type of unsupervised learning.  That fact does not eliminate the use of labels as part of an extension of the original design that generates based on some labeled element in the examples or the use of other labels to indicate the fitness of each image for a class into which generated images are expected to fall.


How do I turn this formula of the average degree of a graph into Python code?

Each node is a position in the arrays

values = value of the node
conn = indexes of connected nodes

If its an undirected graph, each node must have all the nodes to which they are attached.
Instead, in directed graphs, only the start node has the index.
For your image:

values = ['A','B','C','D','E']
conn = [[1,2,3],[0,4],[0,3,4],[0,2,3],[1,3]]

Example = 'A' -> 1 ('B') ,2 ('C') ,3 ('D')

def averageGraph(conn):
    if conn.all() != None:
        average = 0
        for node in conn:
            average = average + len(node) #len(node) = nº nodes connected = degree
        return average / len(conn)
    else:
        return None