Can the C4.5 algorithm learn a GOAP model?

You're going to run straight into the qualification problem, the frame problem and ramification problem on these. Plus you'll need to get your head round what an action is and means.

Qualification Problem

A more formal "STRIPS" would be Situation Calculus (logical equivalency demonstrated by Reiter), which can be defined in FOL or SOL, depending on who you talk to. In Situation Calculus you have your actions and you define when they are possible (S is the current Situation):

poss(open_door, S) → holds(door_position(closed) ^ door_lock(unlocked), S).

This is true, when it is possible to open the door it's both closed and unlocked. The difficulty arises when we try to flip that implication around:

poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked), S).

This is not true, because there must also be nothing blocking the door, the door can't be wedged too tightly into the door frame, and a whole bunch of other stuff that we might not think of. This is the qualification problem, which we get around in STRIPS by ignoring it and just flipping the implication.

In your case you'll end up learning a whole bunch of irrelevant qualifications:

 poss(open_door, S) ← holds(door_position(closed) ^ door_lock(unlocked) ^ weather(sunny) ^ grass(green), S).

You'd need some way to trim them down, by observing gameplay for your data you'll only learn the conditions when players choose to do an action, not when it is possible to do that action: the old correlation vs causation problem. You can immediately prune things that aren't fluents, but as far as I'm aware there's no good solution to this without letting the computer simulate to try actions in subsets of the fluents that hold in-order to prune.

Frame Problem

The result of an action in the language of Situation Calculus is that it causes some fluent to hold. So in our example we have a door_position fluent, we can say that opening the door causes door_position(open) to hold, but what about some other action, such as making tea? We know it has no effect on the position of the door, but for the computer to know that we need to tell it:

holds(door_position(open), S) → holds(door_position(open), do(make_tea, S)).

This creates far too many axioms. Reiter has a working solution to this for Golog in his book "Knowledge in Action". In your case it will manifest as missing axioms, i.e. not knowing what the result of an action will be for some fluent. Try beginning with the assumption that an action has no effect unless proven otherwise.

Ramification Problem

The ramification problem is very similar to the frame problem, it means you might not attribute all of the effects of an action to that action. Opening a door can cause changes in air-pressure through a house that can cause another door to slam, such an effect is easy to miss. For this you'd need a sensitive algorithm that can correctly recognise rare events that are important, which is a big ask. The solution is likely to be to ignore it.

Closely related and not usually a problem for such systems, you could also attribute effects to an action incorrectly. How will you be able to distinguish what the action caused vs occurred simultaneously? In some Mario game data it could likely be induced that jumping causes a turtle to move closer to Mario, again correlation vs causation and needing a method to distinguish. The solution here is likely to come from increasing the volume of data.

Definition and Granularity Problem

OK, it's probably possible that with enough data and a sensitive training algorithm you can overcome the above problems. The definition one is merely a conceptual one and easily overcome.

Humans tend to define an action by it's name, say for example "turning on a light". However, in Situation Calculus the action is defined by when it is possible and the result of that action, the name is merely a label. So "turning on a light" where the bulb is broken and when it is not may share the same label, but they are different actions.

Furthermore, in STRIPS and Situation Calculus we're defining "types" or "classes" of actions, the actual execution is an "instance" of that "class". Your problem is to derive the abstract from the concrete.

For you this problem will arise in terms of defining the granularity, as each instance of an action is possible in the unique situation that it occurred in, and caused the unique situation that came after it.

How do you group these actions together to abstract the most similar ones? How do you know how many actions there are to create these groups? If it's not based on a limited set of user-inputs to some computer game, then you might want to draw some ideas from the research into clustering, particularly heuristics of how many clusters to use.

Alternative to a decision tree

If I were to tackle this problem (fund a Post-Doc for me and I'd be tempted), I'd first investigate ILP as a method to learn the relations using Golog as the foundation of my action axioms. Then I'd look at clustering based on a space where each fluent was a dimension. I imagine it would be a rather useful tool, even if it merely created a pretty accurate template for manual review.


Why are Q values updated according to the greedy policy?

The Q values are updated using a "greedy policy" because, in the Q-learning algorithm, the $\max$ operator is used to determine the "target", which is denoted by
$$\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$$
Intuitively, the $\max$ operator is used to take the "greedy" action, that is, the action associated with the highest Q value: $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ means that we are selecting the $Q$ value, associated with the (next) state $S_{t+1}$, which corresponds to the action $a$, such that $Q(S_{t+1}, a)$ is the highest (with respect to other possible actions from $S_{t+1}$).
Note that the $Q$ function receives as input a state and an action. So, for each state $s$, we have an action (among all possible actions, $a_1, a_2, \dots$, which we can take from the state $s$), denote it by $a^*$, such that $Q(s, a^*) > Q(s, a_1)$, $Q(s, a^*) > Q(s, a_2)$, etc. In the expression $\color{blue}{\max_{a}Q(S_{t+1}, a)}$, we are basically selecting $Q(s, a^*)$ for $s = S_{t+1}$.
To explain all the components of this target, I will first have to explain other parts of the Q-learning algorithm. Hopefully, after this explanation, you will be able to understand why Q-learning uses the greedy policy to update the Q values. I will first describe the Q-learning algorithm, you will then, hopefully, understand why it is called a "off-policy" algorithm (which is, IMHO, a quite unintuitive and confusing term to describe what "off-policy" actually means).
Here's the Q-learning algorithm

The Q-learning algorithm estimates (and returns) the $Q$ function associated with a policy $\pi$ (which needs to be passed as input). Intuitively, it does that by simulating an "agent" which takes actions in the environment, "observes" the impact (in terms of rewards received and states visited, after having taken those actions) of those actions on the environment, and then attempts to infer the optimal state-action value (or $Q$) function (that is, the optimal Q function that is associated with the optimal policy, which would give the agent the "highest amount of reward" in the long run), given its interaction with the environment.
Q-learning proceeds in episodes. So, initially, you need to pass the number of episodes as input. You can think of episodes as iterations (like in any iterative optimisation algorithm). However, in the context of RL, an episode is a little bit more specific: the start and end of an episode are associated with specific states of the environment: the episode starts when the agent is in a "starting state" and ends when it is in an "ending (or goal) state". In the pseudocode above, $S_0$ is the starting state.
In the pseudocode above, at the beginning of each episode, we initialise $t=0$, where $t$ represents the time step of a specific episode. Note also that the agent "moves" to the starting state, $S_0$, at the beginning of each episode.
We then have the following loop, which terminates when the agent reaches a goal state (denoted, in the pseudocode, by "terminal", i.e. "terminal state" or "goal state" are synonyms in this context):

So, at each episode, we run the loop above. The block of code inside this loop contains the main logic of the Q-learning algorithm. On each iteration of this inner loop, the agent chooses an action $A_t$ (the action at time step $t$ of the current episode) using any policy (which "ensures that all states are sufficiently visited": you can ignore this for now, given that this answer is already quite long!). In this case, the $\epsilon$-greedy policy is used. How does this $\epsilon$-greedy policy work? If you look at the pseudocode above, $\epsilon$ is initialised at beginning of each episode. In the pseudocode above, $\epsilon$ can change from episode to episode, but assume, for simplicity, that, at every episode, it is a fixed small number (e.g. $0.01$). The statement "Choose action $A_t$ using policy derived from $Q$ (e.g., $\epsilon$-greedy)" means that, with probability $1 - \epsilon$, the "greedy action" is chosen (from the current state, which is, at the beginning of the episode, $S_0$), and, with probability $\epsilon$, a random action is taken. What is the greedy action in this case? It is the action, in the current state, which is associated with the highest Q value (given the current estimate of the Q value). The greedy action is exactly the same action as the action $a^*$ (as I explained above). The difference is that, in this case, we choose $A_t$ using the $\epsilon$-greedy policy: so, most of the times, we choose the greedy action, but, sometimes, we can also choose a random action. Note that the $\epsilon$-greedy policy is indeed a policy, because it allows the agent to choose an action, given a state: this is roughly the definition of a policy.
The agent then executes the just chosen action $A_t$ in the environment, and it observes the "impact" of this action on the environment, which is determined by how the environment responds to this action: the response consists of a reward, $R_{t+1}$, and a next state, $S_{t+1}$.
To recapitulate, the agent chooses an action using the $\epsilon$-greedy policy, executes this action on the environment, and it observes the response (that is, a reward and a next state) of the environment to this action. This is the part of the Q-learning algorithm where the agent interacts with the environment in order to gather some info about it, so as to be able to estimate the Q function.
After that, the agent can update its estimate of the Q function associated with the policy given as input $\pi$. It does that using the following update rule
$$\color{orange}{Q(S_t, A_t)} \leftarrow \color{red}{Q(S_t, A_t)} + \alpha ([\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)})$$
where $S_t$ is the current state (of the current episode) the agent is in, $A_t$ is the action chosen using the $\epsilon$-greedy policy (as described above), and $S_{t+1}$ and $R_{t+1}$ are respectively the next state and rewards, which, collectively, are the response of the environment to the just taken action $A_t$.
So, how is the estimate of this $Q$ function updated?
First of all, I would like to note that, if you look at the beginning of the pseudocode above, $Q(s, a)$ is initialized arbitrarily for all states $s \in \mathcal{S}$ and for all actions $a \in \mathcal{A}$: it can e.g. be initialised to $0$. $Q(s, a)$ can e.g. be implemented as a matrix (or 2-dimensional array) $M \in \mathbb{R}^{|\mathcal{S}| \times |\mathcal{A}|}$, where $M[s, a] = Q(s, a)$, $|\mathcal{S}|$ is the number of states in your problem and $|\mathcal{A}|$ the number of actions.
Furthermore, note that the symbol $\leftarrow$ means "assignment" (like assignment to a variable, in the context of programming). So, in the update rule above, we are assigning to $\color{orange}{Q(S_t, A_t)}$ (which will be the next or updated estimate of the Q value for the current state $S_t$ and the just taken action from that state $A_t$) the value $\color{red}{Q(S_t, A_t)} + \alpha (\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} - \color{red}{Q(S_t, A_t)})$. Let's break this value down.
$\color{red}{Q(S_t, A_t)}$ (on the right side of the assignment) is the estimate of the Q value for the state $S_t$ and action $A_t$ before the assignment. So, we are summing $\color{red}{Q(S_t, A_t)}$ and $\alpha (\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} - \color{red}{Q(S_t, A_t)})$, and then we assign it to $\color{orange}{Q(S_t, A_t)}$ again.
$\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is what is often called the "target". Q-learning is a "temporal-difference" (TD) algorithm, and TD algorithms update estimates of the value or action-value functions based on the difference between the current estimate, in the case of Q-learning it is denoted by $\color{red}{Q(S_t, A_t)}$ (on the right side of the $\leftarrow$), and a "target". So, in the Q-learning algorithm, $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is the target. We can roughly think of it as "the value that $\color{red}{Q(S_t, A_t)}$ should have been". So, in a certain way, we are performing "supervised learning", where $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ would be the "ground-truth" label and $\color{red}{Q(S_t, A_t)}$ the current estimate, and so $[\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)}$ would be the "error" (or "loss"): in fact, it is often called the "TD error". However, note that this is not really supervised learning, because $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is not a ground-truth (it is partially an estimate, because of the part $\gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} $, and it partially a ground-truth, because of $\color{green}{R_{t+1}}$).
To recapitulate, $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is the "target", $\color{red}{Q(S_t, A_t)}$ is the current estimate, and $[\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)}$ is the "error". We are thus summing the "error" (weighted by the hyper-parameter $\alpha$, which is, in this case, often called the "learning rate") and the current estimate $\color{red}{Q(S_t, A_t)}$ in order to produce the new estimate $\color{orange}{Q(S_t, A_t)}$.
In the target, you can see that we are multiplying the $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ by $\gamma$. This is a hyper-parameter (a parameter which often needs to be chosen by the programmer before the algorithm is executed). It controls the contribution of $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ to the "target": that is, how much of $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ we want to include in the target. Recall that I've just said above that the "target" is composed of the reward $\color{green}{R_{t+1}}$ (which is a "ground-truth" or real-world "experience", because it is directly received from the environment) and $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ (which actually uses an estimate of the Q function, that is, it uses $Q(S_{t+1}, a)$). So, $\gamma$ controls the contribution of an estimate to the "ground-truth".
As I said at the beginning of this answer, $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ can be thought of as the $Q$ value associated with the next state $S_{t+1}$ (which was observed by the agent after he has taken the action $A_t$) and associated with the action $a$, such that $Q(S_{t+1}, a)$ is the highest among all other possible actions from state $S_{t+1}$. In other words, $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ can be thought of the estimate of the Q value associated with the next state $S_{t+1}$ and the "greedy action" taken from that same state.
I would like to note that, at this point, the agent is not really taking actions in the environment. Nonetheless, people, in the RL community, often call Q-learning an "off-policy" algorithm because

It uses the $\epsilon$-greedy policy to interact with the environment (this is often called the "behaviour policy"). In this case, actions are really taken, and the responses of the environment are really produced, observed and used to update estimates of the $Q$ function.

It uses a "target" that is based on an estimate which is "greedy" (i.e. it uses $\color{blue}{\max_{a}Q(S_{t+1}, a)}$).

Given that Q-learning uses estimates of the form $\color{blue}{\max_{a}Q(S_{t+1}, a)}$, Q-learning is often considered to be performing updates to the Q values, as if those Q values were associated with the "greedy policy", that is, the policy that always chooses the action associated with highest Q value. So, you will often hear that Q-learning finds a "target policy" (i.e. the policy that is derived from the last estimate of the Q function) that is "greedy".
I would like to emphasize that, in the Q-learning algorithm, we have several episodes, and, at each episode, the agent interacts with the environment by taking actions and observing the responses of the environment, so as to produce an estimate of the action-value function (i.e. the $Q$ function).


How could an AI interpret the requests of a human without the human needing to specify every detail?

This is basically the problem of commonsense knowledge. It is AI-complete.

If we knew how to solve it, Siri and Cortana wouldn't be as limited as they are.


What is the status of the capsule networks?

ML is full with things that supposed to work better (in theory).

Sigmoid function seems better than ReLu.

L1 seems way better than L2.

Spikes neural network seem to be better than standard neural network.

A shallow neural network with a lot of neurons has more parameters than a deep one with the same amount of neurons. So, in theory, has to be more powerful.

Is common to forget the big impact of the training capacity. And easy to get charmed by its theoretical capacity.
Training process are still in diapers. We lack a good understating of it.
Today, Capsules neural network are a very powerful architecture, hard to train.


Is the pattern recognition capability of CNNs limited to image processing?

Convolutional Nets (CNN) rely on mathematical convolution (e.g. 2D or 3D convolutions), which is commonly used for signal processing. Images are a type of signal, and convolution can equally be used on sound, vibrations, etc. So, in principle, CNNs can find applications to any signal, and probably more.

In practice, there exists already work on NLP (as mentioned by Matthew Graves), where some people process text with CNNs rather than recursive networks. Some other works apply to sound processing (no reference here, but I have yet unpublished work ongoing).

Original contents: In answer to the original title question, which has changed now. Perhaps need to delete this one.

Research on adversarial networks (and related) show that even deep networks can easily be fooled, leading them to see a dog (or whatever object) in what appears to be random noise when a human look at it (the article has clear examples).

Another issue is the generalization power of a neural network. Convolutional nets have amazed the world with their capability to generalize way better than other techniques. But if the network is only fed images of cats, it will recognize only cats (and probably see cats everywhere, as by adversarial network results). In other words, even CNs have a hard time generalizing too far beyond what they learned from.

The recognition limit is hard to define precisely. I would simply say that the diversity of the learning data pushes the limit (I assume further detail should lead to more appropriate venue for discussion).


Is it possible with stochastic gradient descent for the error to increase?

Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot without passing it through smoothing filter, so noisy it is. Specific depend on the problem of cause. It's not only for SGD but for any optimizer.


How to find the optimal number of neurons per layer?

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.

There exist more advanced techniques such as Gaussian processes, e.g. Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.


Flatten image using Neural network and matrix transpose

Yes, if you have 3 images (and by images I assume you mean samples) the flatten layer will be of the shape $12288*3$ ($64*64*3=12288$). The size of $W$ however does not change, and nor does the size of $b$ as these are parameters and are independent of the amount of samples passed through the network.
ETA: I only answered the "Am I right?" part of your question because that's the only part of your questions that's actually a question. I don't know what you're trying to ask in the second half of your question


Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

There is no special calculation you can do to determine the optimal batch size for any situation, so you kinda have to do a bit of testing to determine what batch size will work best. But there are some common trends you can take into account to make your testing easier.

How to choose your batch size
According to the paper Accelerated Methods for Deep Reinforcement Learning you get the best performance from DQNs (on average) with a batch size of 512. The problem with this is that is is much slower than the usual batch size of 32 and most of the time the performance improvement doesn't warrant it.

If you are just trying to test out your agents it is generally best to stick with a batch size of 32 or 64 so that you can train the agent quickly yet still get an idea of what it is capable of. But if getting the best performance is your top priority and waiting longer isn't a problem, then you should go for a batch size of 512 (higher can actually lead to worse performance) or something near that.


Why is it possible to eliminate this branch with alpha-beta pruning?

You can't prune the nodes that are cross out if we search from left-to-right in the tree using alpha-beta pruning. To do this analysis we can pretend the right branch of the tree doesn't exist. (Branch C from the root.)

In the left branch (A) of the root Helen will get 2 or more.

In the middle branch (B) from the root after going down the left, Stavros will get 7 or less.

Now, we can ask what happens if we put different values for the crossed out branches. If putting different values doesn't change the value at the root, then pruning is correct.

Suppose that after the 7 on branch A at the second level we make the value of branch B to be -10. In this case the value of the second branch will be -10, and Helen will prefer the first branch at the root.

Suppose that after the 7 on branch A at the second level we make the value of branch B to be 100. In this case the value of the second branch will be 7, and Helen will prefer this branch.

Thus, the value of the crossed off nodes matters, and they cannot be pruned.

You may find this tool useful for exploring alpha-beta on binary trees:

https://movingai.com/ab/


Why everyone is using CNN for image segmentation?

CNN is used since it is effectively an optimized use case for dealing with image data.

CNN effectively automatically extracts features from images. Other techniques are more likely to not take full advantage of the data. CNN is able to make full use of the data by also including information from adjacent pixels and downsample through layers.

Here is a paper on the performance of CNN on image data
Here is a paper comparing CNN's to other methods


What is the most sophisticated AI ever made?

In my opinion, this would be Phaeaco, which was developed by Harry Foundalis at Douglas Hofstadter's CRCC research group.

It takes noisy photographic images of Bongard problems as input and (using a variant of Hofstadter's 'Fluid Concepts' architecture) successfully deduces the required rule in many cases.

Hofstadter has described the related success of CopyCat as being 'like a little kid doing a somersault': i.e. it doesn't have the flashy appeal of systems like AlphaGo. What it does however have is a much more flexible (i.e. not precanned) approach to perception of problem structure than other systems, which Hofstadter claims (and many including Peter Norvig agree) is the really hard problem.


Why is the Universal Declaration of Human Rights not included as statement on the AI?

If I understand what you are asking, I think the simple answer would be that AI is nowhere near having demonstrated sentience, thus they do not qualify for any type of rights.

We won't have to "cross this bridge" until an AI demonstrates self-awareness and human-level-or-beyond intelligence, but it sure is interesting to think about!

(Also, the UDHR dates to the 1940's and seems to have had its last additions in 1966.  Computers weren't very "smart" back then so likely no on was even considering the question ;)

Value Alignment is a huge looming issue in regards to Artificial General Intelligence

Although you may also want to look at the grey good scenario, which posits inadvertent destruction of homo sapiens sapiens not as a factor not of too much, but of too little, intelligence.

The problem with an Asimov approach is highlighted by his book I, Robot, which is the potential pitfalls of pure logic.  The philosophy of Neo-Luddism is preoccupied with these problems in relation to technology--specifically that the threats posed by technology cannot be predicted.

The problem with the UDHR today is that there is no algorithm smart enough to understand it--we're not even close.  (There is something called the symbol grounding problem which demonstrates that meaning and understanding in relation to algorithms is still unsolved.)


Should we feed a greater fraction of terminal states to the value network so that their values are learned first?

The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known.

This may be true in some environments. Many environments do not have a terminal state, they are continuous. Your statement may be true for instance in a board game environment where the goal is to win, but it is false for e.g. the Atari games environment.
In addition, when calculating the value of the terminal state, it is always zero, so often a special hard-coded $0$ is used, and the neural network is not required to learn that. So it is only for deterministic transitions $(S,A) \rightarrow (R, S^T)$ where you need to learn that $Q(S,A) = R$ absolutely.

Shouldn't it make sense to feed the model a greater proportion of terminal states initially, to ensure that the predicted value of a step in terminal states (zero) is learned first?

In a situation where you have and know the terminal state, then yes this could help a little. It will help most where the terminal state is also a "goal" state, which does not have to be the case, even in episodic problems. For instance in a treasure-collecting maze where the episode ends after a fixed time, knowing the value of the terminal state and transitions close to it is less important to optimal control than establishing the expected return for earlier parts of the path.
Focusing on "goal" states does not generalise to all environments, and is of minimal help once the network has approximated Q values close to terminal and/or goal states. There are more generic approaches than your suggestion for distributing knowledge of sparse rewards including episode termination:

Prioritised sweeping. This generalises your idea of selectively sampling where experience shows that there is knowledge to be gained (by tracking current error values and transitions).

n-step temporal difference. Using longer trajectories to calculate TD targets increases variance but reduces bias and allows assignment of reward across multiple steps quickly. This is extended in TD($\lambda$) to allow parametric mixing of multiple length trajectories and can be done online using eligibility traces. Combining Q($\lambda$) with deep neural networks is possible - see this paper for example.


Why was Go a harder game for an AI to master than Chess?

The branching factor is important, as it limits the effectiveness of search.

However, the branching factor in chess is already too high to effectively search without techniques that reduce the size of the search space. Even with millions of tests per second, a computer can only check a small fraction of the possible future games in order to find results in its favour.

One key factor is heuristics - approximate measures of the value of each game state. A good heuristic can guide and improve search by orders of magnitude. There are some effective heuristics possible in chess, from weighted values of the pieces in play, scores for areas of the board that a position controls etc.

Heuristics for Go are much harder to find. Here's a sample paper from a few years ago that makes an attempt, there are several similar ones available online. Although plenty of options have been tried, and many were partially successful, none managed to bring the quality of computer play up to the standard of best human players.

One of the major achievements of AlphaGo was training a neural network that had good position evaluation - the "value network". The technology that made generating this approximate function of board game positions possible was deep learning, which has been developed very strongly since about 2010.

It is still possible a more analytical heuristic approach could be found that challenges deep learning models driven by self-play reinforcement learning on more raw board data. However, in some regards the reverse has been shown, with AlphaZero taking the same learning technique into chess and demonstrating its effectiveness against "old school" tuned expert heuristics.


Will AI always depend on models and thus approximations?

There's a neuroscience theory, known as predictive coding, which roughly states that the (human) brain is constantly generating and updating a model of the world.

The brain is constantly confronted with a wealth of sensory information that must be processed efficiently to facilitate appropriate reactions. One way of optimizing this processing effort is to predict incoming sensory information based on previous experience so that expected information is processed efficiently and resources can be allocated to novel or surprising information. Theoretical and computational studies led to the formulation of the predictive coding framework (Friston 2005, Hawkins and Blakeslee 2004, Mumford 1992, Rao and Ballard 1999). Predictive coding states that the brain continually generates models of the world based on context and information from memory to predict sensory input. In terms of brain processing, a predictive model is created in higher cortical areas and communicated through feedback connections to lower sensory areas. In contrast, feedforward connections process and project an error signal, i.e. the mismatch between the predicted information and the actual sensory input (Rao & Ballard, 1999). The predictive model is constantly updated according to this error signal.

This theory should not be surprising or unintuitive, given that every person possesses a slightly different perspective (or model) of the world, which is based on her (or his) personal experiences. Of course, this is just a theory, which may not be the most precise one that describes our brain, but this theory is already being validated by a number of brain imaging studies investigating predictive feedback and the processing of prediction errors.
Therefore, artificial intelligence may not be the only entity that is based on or will be limited by a model of the world. To answer your question more directly, yes, the AI will always be limited by its model and environment (e.g. hardware), in a similar way that flatlanders are limited by their 2-dimensional nature and world, but this doesn't necessarily mean we will not be able to create useful (and even sophisticated or human-like) AI systems.


What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics?

It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal distribution.


Evaluate model multiple times in loss function? Is this reinforcement learning?

Can my loss function be evaluating the model until it dies? 1/survival time could be the loss value to be minimized by gradient descent.

In order to use backpropagation and gradient descent, you have to relate the loss function directly to the output of the neural network. Your proposed loss function is too indirect, it is not possible to turn it directly into a gradient that could be used to alter the neural network weights.
In addition, the specific function of time you have chosen will be difficult to optimise, as incrememental improvements from e.g. 10 to 11 time steps surviving will provide a much lower signal for adjusting behaviour than the improvement from  e.g. 2 to 3 lifetime. If the environment has enough randomness (and typically these kind of a-life scenarios do), then the signal here could be swamped by random events and very hard to optimise, requiring a larger number of samples in order to extract expected improvements to the loss function.

Is this a reinforcement learning problem (I don't think so..?)

It is very close to a definition of reinforcement learning (RL) problem. For a RL problem you need the following things:

An environment in which an agent exists, and which has a measurable state.

A set of actions = sequential decisions that need to be made, and that have consequences.

The consequences of any action are:

A consistent (but allowing for some randomness) change to the state of the environment. Here you have change to agent's location and resource levels.
A consistent scalar reward value that measures how well the agent is achieving a goal. This can also be partially random, and can also be sparse, only achieved in certain specific states.

In your problem definition you don't have a reward signal, but it would be easy to add one. A suitable one would be $+1$ per time step.
Technically your problem would also be partially observable (sometimes called a POMDP), in that the agent does not get to see resources available in other locations. It only knows its current location, its internal state and resources available at its current location. This is not a major issue, although you should note that adding some kind of memory (either open-ended memory as in a recurrent neural network, or explicitly added to the state) would allow for more efficient agents. That's not a RL issue as such, any learning process without ability to form or use memories would be limited in this environment, and you might want to look into that as a later experiment.
How RL helps is that it provides a framework to convert your problem definition into measurements and gradients for the neural network to improve its performance.
As you have set up your neural network to predict best action choice, this would naturally lend itself to policy gradient methods in RL, such as REINFORCE. As it is a simple problem, I would expect REINFORCE with baseline to perform well enough for it.
I will not explain the algorithm here in full detail, but the basic approach is to have the agent act with the current network for a few episodes, collecting data on its choices and performance. You will get a dataset of (state, action, return = sum of rewards to end of episode). You then use that as a labelled dataset to train a minibatch as if the action choice was correct ground truth, but multiply each gradient by (return - baseline) where the baseline is typically the average return seen so far from that state. You may need a second neural network during training to estimate that expected return (aka state value). After using a minibatch once, you will need to discard it, as it represents results for the previous iteration of the network before the weight updates. There are ways around this, but not typically done in REINFORCE - instead the approach is to just keep generating data and train on the new mini-batch as fast as you can go.
RL offers other methods which may work just as well to solve your problem, but I suspect REINFORCE will serve you well since it will allow the agent to randomise direction choice which is important when it cannot see where the resources are and has no memory of where it has searched.
You don't have to use RL for this problem. An alternative that may work for you is using genetic algorithms to tune the network architecture and weights. It avoids using gradients, but I would still recommend a simple fitness function equal to number of time steps survived. There is a framework called NEAT which is ideal for this sort of a-life control problem.


Why do RL implementations converge on one action?

If the optimal policy shouldn't always select the same action in the same state, i.e., if the optimal policy isn't deterministic (e.g., in the case of the rock paper scissors, the optimal policy cannot be deterministic because any intelligent player would easily memorize your deterministic policy, so, after a while, you would always lose again that player), then there are a few things that you can do to make your policy more stochastic

Change the reward function. If your agent ends up selecting always the same action and you don't want that, it's probably because you're not giving it the right reinforcement signal (given that the agent selects the action that apparently will give it the highest reward in the long run).
Try to explore more during training. So, if you're using a behavior policy like $\epsilon$-greedy, you may want to increase your $\epsilon$ (i.e. probability of selecting a random action).
If you estimated the state-action value function (e.g. with Q-learning), maybe you derived the policy from it by selecting the best action, but, of course, that will make your policy deterministic. You may want to use e.g. softmax to derive the policy from the state-action value function (i.e. the probability of selecting an action is proportional to its value), although Q-learning assumes that your target policy is greedy with respect to the state-action value function.

If the optimal policy is supposed to be deterministic, then, if you find the optimal policy (which isn't probably the case), you will end up with an agent that always selects the same action. In that case, obviously, it's not a problem that the RL agent selects always the same optimal action.


In CNNs, why do we sum the filter derivatives w.r.t the loss function to get the final gradient?

I really liked the question. Yes, we sum over derivatives. First of all think what backpropagation is trying to do: finding the affect of each parameter on the loss.
So as you said:

the same filter is used multiple times on the input while convolving

meaning that each kernel affects the final loss in several ways, so those affects should be summed together, not averaged.


What exactly is an interpretable machine learning model?

In a simple linear model of the form $y = \beta_0 + \beta_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta_1$. Here we can completely determine what the effect on the models prediction will be by increasing $x$. With more complex models such as neural networks it is much more difficult to tell due to all the calculations that a single data point is involved in. For instance, in a CNN as you mentioned, if I changed the value of a pixel in an image we were passing through the CNN you wouldn't really be able to tell me exactly the effect this would have on the prediction like you can with the linear model.


Immediate reward received in Atari game using DQN

To answer your questions in order:

  What is $r_s $ used in the case where the score has not changed ?

It is $0$.

  Therefore what is the immediate rewards being modelled for temporal credit assignment problems in atari game ?

Rewards can be re-modelled to aid speed of learning. This is called "reward shaping", and is typically done by domain experts who can adjust numbers to reward known good intermediate states and actions.

For DQN Atari, this was not done. Instead, the researchers performed a reward normalisation/scaling so that games which used moderate scoring system in single digits could be handled by the same neural network approximator as games that handed out thousands of points at a go.

Using sparse rewards is standard practice in reinforcement learning, and the credit assignment problem is solved to some degree by all reinforcement learning methods. Essentially the value functions work as a prediction mechanism theoretically whatever the reward sparsity, so if they are correct, thay can be used to drive policy whether the next reward is 1, 50 or 1000 time steps away. The reward backup updates in everything from Value Iteration, through Monte Carlo Control, SARSA, Q-Learning and Actor-Critic all backup values to states/actions seen in earlier time steps. This value backup is a basic mechanism that addresses credit assignment in principle. The credit assignment problem is then a matter of degree and difficulty of different environments, such that sometimes it is readily solved, and other times it is a major hurdle.

In the case of video games, especially older arcade games, it is often not a very hard part of the problem. The games are designed to reward human players by incrementing scores frequently, with very many sub-goals already within the game. In fact this is one of the attractions of video games as toy environments for developing new algorithms.

For example the classic space invaders does not simply score +1 for surviving a wave of enemies, but adds points for every player missile that hits. Although the score does not increment on every frame, the reward sparsity is relatively low for games like this, and simple single-step Q learning with experience replay can solve the credit assignment problem readily for that environment (experience replay does help a little with credit assignment). This was what was demonstrated with the original DQN Atari paper, there were no extra allowances made for reward sparsity.

  If $r_s$ is indeed 0 until score changes, would it affect the accuracy of the DQN ?

Not directly, the DQN predicts future expected rewards and can in principle account for delay and sparsity in its estimates. However, if this becomes very sparse you get two problems:

Discovering the positive rewards within the environment may take a long time, and may require more advanced techniques, such as reward shaping to encourage searching behaviour (a small negative reward per time step) or "curiousity"
Credit assignment becomes much harder as the possible number of combinations that could of contributed to success can grow exponentially with temporal distance between rewards. Resolving whch ones are important, especially early in a trajectory leading to reward, can take many samples.

  What are some of the current methods used to solve the temporal credit assignment problem ?

As noted above, this is core problem in RL, so there are many approaches in the literature. Some standard approaches are:

Background planning as used by DynaQ, or experience replay. This re-evaluates states and actions seen before whilst using latest estimates, and can backup values to where important decisions are made within a trajectory. Prioritised experience replay helps even more by focusing on updates that make the most difference to the current estimates.
TD($\lambda$) with eligibility traces. Eligibility traces are intuitively a credit-assignment mechanism, they track state features that were active recently and multiply value updates based on the trace vector. Again, this causes state or state action values to backup to earlier parts of trajectories faster.
Reward shaping as discussed above. In some research settings this might be seen as "cheating" - for instance when facing a standardised environment test for developing algorithms, adding in domain knowledge to help the agent just demonstrates the core algorithm is weaker than claimed. However, when the challenge presented is to solve an environment with the best agent possible, it is fine to use any knoweldge (or of course not to use RL at all).


How is it that AI can become biased, and what are the proposals to mitigate this?

Lately with my Google searches, the AI model keeps auto filling the ending of my searches with:
“...in Vietnamese”

I can see how this would be annoying.
I don't think Google's auto-complete algorithm and training data is publicly available. Also it changes frequently as they work to improve the service. As such, it is hard to tell what exactly is leading it to come up with this less-than-useful suggestion.
Your suspicion that it has something to do with Google's service detecting your heritage seems plausible.
The whole thing is based around statistical inference. At no point does any machine "know" what Vietnamese - or in fact any of the words in your query  - actually means. This is a weakness of pretty much all core NLP work in AI, and is called the grounding problem. It is why, for instance, that samples of computer generated text produce such surreal and comic material. The rules of grammar are followed, but semantics and longer term coherence are a mess.
Commercial chatbot systems work around this with a lot of bespoke coding around some subject area, such as booking tickets, shopping etc. These smaller domains are possible for human developers to "police", connecting them back to reality, and avoiding the open-ended nature of the whole of human language. Search engine text autocomplete however, cannot realistically use this approach.
Your best bets are probably:

Wait it out. The service will improve. Whatever language use statistics are at work here are likely change over time. Your own normal use of the system without using the suggestions will be part of that data stream of corrections.

Send a complaint to Google. Someone, somewhere in Google will care about these results, and view them as errors to be fixed.

Neither of these approaches guarantee results in any time frame sadly.

We already have enough issues in the US with racism, projections of who others expect us to be based on any number of things, stereotyping and putting people in boxes to limit them - I truly believe AI is adding to the problem, not helping.

You are not alone in having these worries. The statistics-driven nature of machine learning algorithms and use of "big data" to train them means that machines are exposing bias and prejudice that are long buried in our language. These biases are picked up by machinery then used by companies that don't necessarily want to reflect those attitudes.
A similar example occurs in natural language processing models with word embeddings. A very interesting feature of LSTM neural networks that learn statistical language models is that you can look at word embeddings, mathematical representations of words, and do "word math":
$$W(king) - W(man) + W(woman) \approx W(queen)$$
$$W(he) - W(male) + W(female) \approx W(she)$$
This is very cool, and implies that the learned embeddings really are capturing semantics up to some depth. However, the same model can produce results like this:
$$W(doctor) - W(male) + W(female) \approx W(nurse)$$
This doesn't reflect modern sensibilities of gender equality. There is obviously a deep set reason for this, as it has appeared from non-prejudiced statistical analysis of billions of words of text from all sorts of sources. Regardless of this though, engineers responsible for these systems would prefer that their models did not have these flaws.

How can we fix this. Moreover, how can we use AI to bring out peoples true self, talents and empower and free them them to create their life how they like ?

Primarily by recognising that statistical ML and AI doesn't inherently have prejudice or any agenda at all. It is reflecting back ugliness already in the world. The root problem is to fix people (beyond scope of this answer, if I had solid ideas about this I would not be working in software engineering, but in something more people-focussed).
However, we can remove some of the unwanted bias from AI systems. Broadly the steps toward this go:

Recognise that a particular AI system has captured and is using unwanted gender, racial, religious etc bias.

Reach a consensus about how an unbiased model should behave. It must still be useful for purpose.

Add the desired model behaviour into the training and assessment routines of the AI.

For instance in your case, there are possibly some users of Google's system who would prefer to read articles in Vietnamese, or have English translated into Vietnamese, and are finding it awkward that the default assumption is that everything should be presented in English. These users don't necessarily need to use the search text for this, but presumably are for some reason. A reasonable approach is to figure out how their needs could be met without spamming "in Vietnamese" on the end of every autocomplete suggestion, and perhaps in general move suggestions to localise searches by cultural differences out of autocomplete into a different part of the system.
For the case of gender bias in NLP systems, Andrew Ng's Coursera course on RNNs shows how this can be achieved using the embeddings themselves. Essentially it can be done by identifying a bias direction from a set of words (e.g. "he/she", "male/female"), and removing deviations in that direction for most other words, preserving it only for words where it is inherently OK to reflect the differences (such as "king" and "queen" for gender bias).
Each case of unwanted bias though needs to be discovered by people and oversight of this as a political and social issue, not primarily a technical one.


Can feature engineering change the selection of the model according to the minimum description length?

I think the wrong assumption here is that you've forgotten the cost of encoding the new features!

MDL should be considered relative to the original or raw dataset. The idea is that you want to find an expression you could send to someone else that encodes the structure of the dataset in terms of the original variables. If you define new features, you need to send a description of those features along with your model.

To make this clearer, imagine you call me up on the phone, and we're both looking at your left hand image. You say to me 'Yeah, you just draw a line through it at $p=a$'. The natural question for me to ask is 'What's p?'. If you have to tell me what $p$ is, then it's part of the description length.

A circular model for your lefthand image does have a MDL for its decision boundary of something like $(x−a)^2+(y−b)^2=c$. However, the feature transformation you've selected has a description length of $p=(x-a)^2 + (y-b)^2$. It should be clear that the description lengths for the linear model through $p$ and the circular model through $x$ and $y$ are identical.


I couldn't add a comment because of my low reputation but you can check this. It is about the state space. https://math.stackexchange.com/questions/485752/tictactoe-state-space-choose-calculation


In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?

I think that this is an intentional subtle detail of the algorithm that ensures the convergence property.  The claim in the book is that for any $b$ that provides us with "an infinite number of returns for each pair of state and action" the target policy $\pi$ will converge to optimal.
Imagine now that we have such a bad policy $b$ that it never aligns with the target policy action at the last step $t=T-1$ of each generated episode: $A_{T-1} = \pi(S_{T-1})$. In that case the weight value will stay $W=1$ and the the algorithm will be reduced to (ignoring $t$ indices for the last $S,A,R$ triplet):
$$\begin{array}{l} C(S,A) \leftarrow C(S,A) + 1 \\
Q(S,A) \leftarrow Q(S,A) + \frac{1}{C(S,A)}\left[R - Q(S,A)\right]
\end{array}$$
Which is just the tabular incremental averaging for the Q values (see for example eq. (2.3)).
If we bail from the for loop before these updates, then no updates would happen at all. And authors won't be able to claim convergence to optimal policy for all those "good" sampling policies $b$.


How can the multiple intelligences model be incorporated into AI?

If anything, multiple intelligences are much more obvious in AI than in other fields, because we haven't yet unlocked how to do transfer between domains.

As an example, AlphaGo is very, very good at playing Go, but it's got basically nothing in the way of bodily-kinesthetic intelligence. But other teams have built software to control robots that does have bodily-kinesthetic intelligence, while not being good at the tasks that AlphaGo excels at.

This sort of modular intelligence is typically referred to as 'narrow AI,' whereas we use the term 'general AI' (or AGI, for Artificial General Intelligence) to refer to intelligence that we've built that can do roughly as many different kinds of things as people can do.


Should we fear an AGI if it will be equivalent to a human?

There are basically two worries:

If we create an AGI that is a slightly better AGI-programmer than its creators, it might be able to improve its own source code to become even more intelligent. Which would enable it to improve its source code even more etc. Such a selfimproving seed AI might very quickly become superintelligent.

The other scenario is that intelligence is such a complicated algorithmic task, that when we finally crack it, there will be a significant hardware overhang. So the "intelligence algorithm" would be human level on 2030 hardware, but we figure it out in 2050. In that case we would immediately have superintelligent AI without ever creating human level AI. This scenario is especially likely because development often requires a lot of test runs to tweak parameters and try out different ideas.


What is asymmetric relaxation backpropagation?

I'll give you my initial $0.02 for symmetric relaxation or relaxation in general in working with neural networks. The book covers 'Weight perturbation' and this is a basic outline of that. Say you want to host a wedding and every person gives you a 'must-have' list of requirements for them to attend. You can abide by all the requirements of each wedding guest or start 'uninviting' guests whose restrictions cause too many complications.
There are several kinds of relaxation. I've only used Lagrangian relaxation, so my experience is biased to that application. Think of it like this: you are traveling from New York to LA and you want to optimize for time, if you 'relax' the constraints, you can just fly instead of driving. This, however, creates an increased cost of the air ticket. By relaxing the constraints you remove the isolating requirement that you must travel by car.
Symmetric relaxation can be a challenging subject, so I'll include a few links academic research
Academic research arxiv.org is another site I use for research. Hope this helps.
I also found a link on Medium which is another good source for application, theory, and implementation of algorithms. Medium Lagrangian Relaxation


How machine learning can help with sustainable development and biological conservation?

There are a variety of aspects where AI can help for the public good. Future studies of computational methods can contribute to a sustainable management ecosystem by its data acquisition, interpretation, integration and model fitting.
Prof. Tom Dietterich is a leader in combining computer science and ecological sciences to build the new discipline of Ecosystem Informatics which studies methods for collecting, analyzing and visualizing data on the structure and function of ecosystems.
His group is involved in many aspects of the ecosystem, such as:

Models that can predict species distribution and their presence/absence elsewhere in order to create species distribution and migration/dispersal maps (such as DataONE Datanet, eBird project, BirdCast) .
Bio-economic models require solving large Spatio-temporal optimization problems under uncertainty.
Ecosystem prediction problems require integrating heterogeneous data sources.
Algorithms for deployment (sensor placement), cleaning and analysis of sensor network data of resulting data to increase agricultural productivity (Project TAHMO), like deployment of 20,000 hydro-meteorological stations in Africa (e.g. computational problem where to place it).
Systems for capturing, imaging, and sorting bugs combined with general image processing/machine learning/pattern recognition tools for counting and classifying them (BugID project). The goal is to develop algorithms for automating biodiversity based on visual pattern recognition by using the computer vision method.

For further information about this work, check the following resources, see:

(video) "Challenges for Machine Learning in Computational Sustainability" (CRCS) (2013)
(slides) Tom Dietterich, Understanding and Managing Ecosystems through AI (2013)
(study) Adapting environmental management to uncertain but inevitable change (2015)


Is a genetic algorithm an example of artificial intelligence?

An ability that is commonly attributed to intelligence is problem solving.
Another one is learning (improving itself from experience).
Artificial intelligence can be defined as "replicating intelligence, or parts of it, at least in appearance, inside a computer" (dodging the definition of intelligence itself).
Genetic algorithms are computational problem solving tools that find and improve solutions (they learn).

Thus, genetic algorithms are a kind of artificial intelligence.

Regarding scale, I don't see it as an important factor for defining G.A. as A.I or not. The same way we can simply classify different living forms as more or less intelligent instead of just saying intelligent or not intelligent.

Finally, let's just make an important distinction: our brains are the product of natural selection, but the brains themselves don't use the same principle in order to achieve intelligence.


There is no easy way to play around with the hyper-parameters (number of layers, layer configuration, number of outputs per layer) of a CNN and get an accurate view of how these will affect the resulting performance of your model. However there are a few things that you can do to avoid wasting too much time training and re-training.

Why?

When training a CNN we aim to minimize a loss function, thus a better CNN model is defined as one which converges to a set of model parameters with a lower loss function. Identifying the minimum of the loss function for a given CNN is already very difficult, and there is no guarantee that the true minimum will every be reached by fault of gradient descent.

Each variation of the CNN resulting from the different hyper-parameters will still result in a very large number of model parameters. There is no way to know how the loss function will look in this hyper-dimensional space and even harder to estimate its minimum.

What to do?

First, you should try and understand how each layer in the network affects its inputs. You should know what kind of layers to use for what kind of data. You should also know what kinds of activation functions to use for different data distributions. You should also know how many model parameters per layer to try in order to sufficiently compress your data whilst not losing significant information.

You can get a lot of this intuition by reading papers which have found successful models for specific tasks.

In addition, you can train with smaller amounts of data and estimate the potential minimum loss function by seeing how fast the loss function moves towards its minimal value (momentum). Usually a lower minimum is achieved when the loss function decreases faster. However, this is in no way always true. A loss function can converge slowly at first and then speed up later. This is entirely possible. But, you can get some sense of the potential of your model in this way.


Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.
But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.


How to construct input dependent convolutional filter?

Not a tensorflow expert but I may be able to offer some conceptual advice. Since you do not care to learn the filter, but instead want to fix a discrete set of possible values for a discrete set of cases, you can use tensor operations (i.e convolutions) rather than neural network layer operations. Essentially, in framework-agnostic pseudocode this would look like:
# layers with learned parameters
output1 = layers1(input)

# apply unlearned but changeable layer convolutions
kernel_val = kernel_val_selection_function(output1)
output2 = convolve_2D(output1,kernel_val)

# more layers with learned parameters
output3 = layers3(output2)

...

The function graph will treat kernel_val as a constant for purposes of backpropagation, so as long as your convolution operations are done within the framework used to create the function graph (i.e. tensorflow) you shouldn't have any problems with backprop.


How do I choose the size of the hidden state of a GRU?

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which links the size of the hidden state and the problem at hand. But, there are some rules of thumb like to start out with the size of the hidden state to be a power of 2, etc. Keep tuning the hyperparameter until you get very good predictions.


How to embed/deploy an arbitrary machine learning model on microcontrollers?

There are a few possible approaches to deploying a ML model to a microcontroller.
The main limiting factor to deployment on microcontollers is that ML models are usually a representation of a set of parameters that are intended to be used as input to a prediction algorithm alongside a new datapoint. Most such models assume the presence of an accompanying library that implements the algorithm in question. However, a microcrontoller may use an exotic chip architecture, or have very several or unusual resource constraints that prevent these standard libraries from being deployed easily.
Presumably you will already have some way to get input into your microcontroller and to program it in order to call some function that you can write. If not, you will need to first figure out how to do that, and the right methods depend on your microcontroller. A common approach is to write assembly code or code in a very limited subset of C or another language. An alternative is to find a distribution of an interpreter for another language (e.g. Java, Python) that has been compiled to work on your chip. Either way, you will need some way to program the chip.
Presuming you can program the chip, you have two fundamental challenges in deploying the model:

Most models are trained with very wide floating point numbers for their parameters. For example, 128-bit numbers may be used. On a standard computing environment, the CPU or GPU will be equipped to perform operations on wide datatypes efficiently. On a microcontroller, you may be limited to 8-bit or 16-bit integers. To work with your model parameters in an environment like this, you will need to either make the parameters smaller (usually by rounding them to fit in a much smaller numeric format, a process called "quantization"), or by finding or writing software that can simulate the operations you want (probably addition and multiplication) on a large datatype that is represented as a collection of smaller datatypes. The first approach may make the model perform poorly. The second may make model prediction very slow.

You need an implementation of the algorithm. Some algorithms like linear regression, linear discriminant analysis, or even decision trees are extremely easy to implement prediction for, and may require only addition, multiplication, and/or comparison. You might be able to write these yourself in a simple subset of C, or even in assembly (for example, prediction with linear regression should be just a simple loop). Other algorithms, like deep neural networks, may contain more complex operations, and may contain many such operations performed in complicated sequences. For these, you generally will need to find an distribution of a library that implements the algorithms, or compile one yourself. Compiling one yourself will require setting up a build toolchain for your specific microcontroller, and can be quite involved.


How can I formulate the map colouring problem as a hill climbing search problem?

First of all you need an initial solution. You will then improve this solution with hill climbing.

For your initial solution, you can color the map randomly using the K colors. This will most likely result in conflicts (adjacent regions of the same color).

Then the hill climbing part: Find a region which has conflicts and swap its color for another color, making sure that the new color does not incur more conflicts than the old one. With each iteration your solution should slowly improve.

Note that hill climbing is not perfect and that you may not find a feasible solution in the end.


How to input dataset with multi-value properties?

You could use scikit-learn's MultiLabelBinarizer. It's essentially the multi-label equivalent of one-hot encoding. For each movie, create a vector of zeros, where each zero is associated with a particular actor. If an actor is in that movie, change their zero to a one. In the context of a neural network, think of it as each actor having their own input neuron, which will fire only if they are in that movie.
The caveat is that to represent all actors, you'd need a rediculously long vector. In such cases, it's often sufficient to only look at the most common, say, 100 and ignore the rest. It intuitively makes sense that having a big actor who's done a lot of movies says more about the quality of a movie than whoever's playing Unimportant Bystander #3. This is actually how natural language processing represents words in the English language - take the top n and ignore the rest.


Is there any way can teach AI creative painting (not convert photo to paint)?

This paper (that was featured in another question), StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks describes some techniques similar to what you described in the question.  Instead of comics, the system in the paper is trained on a database of nature photos (birds or flowers) combined with text descriptions of each photo.  It will then draw an "imaginary" bird or flower based purely on a text description.  It does this in two steps: it the first, it picks a posture for the depicted subject (I believe randomly...) and roughs out patches of color that match the described features of the subject in the selected posture.  It then refines the sketch, again using the text description for parameters, until it appears "photo-realistic" in the style of the training photos.  It does not just recreate its training images; instead, it is able to recreate features of its training images with different positions/colors.


Why do knowledge-based agents only add a sentence to the knowledge base when it is 100% sure the sentence is true?

Mostly because traditional "knowledge based systems" are based purely on deductive logic and that's just the way deduction works.  It only deals with what consequences must follow from the premises.  Traditionally these systems didn't deal with probabilistic knowledge or other less strict forms of reasoning, like abduction.

That said, it is possible to build a knowledge based system that includes probabilistic reasoning, and other reasoning systems (like abduction).

I'll add some more on that point when I have a little more time.


How to incorporate a symmetry constraint in the loss function to train a CNN?

If you know it is symmetric, then you could do a couple things.

Zero out a half.

Don't bother learning both halves of the image. Just put a zero mask over the upper or lower half of the output matrix and just have the network regress the other half. Just don't make the network do more work than it needs to do.

Learn both, but add symmetric loss

In your case, it looks like you could create two loss functions added together.
$focal(x, y) + focal(x^T, y)$
This will help the network learn both halves equally.

L1 between $x$ and $x^T$

This might be silly but adding a Huber loss between $x$ and $x^T$ might help promote symmetry, but I'm not as much if a fan of it. Personally I'm more partial to (1).

Combine (1) and (2)

You could take the loss function of (2) at train time but at test time just use whatever half had better metrics and copy that to the bottom half.

Post Processing

Add a post processing custom layer that takes the average of the two halves so that you guarantee symmetry.
$x' = \frac{1}{2}(x + x^T)$
Then do your normal focal loss. Personally I like this the best since it always guarantees a symmetric output and is a pretty easy custom layer.


Why do layered neural nets struggle with continous data?

They struggle because if your network have an inductive bias towards modeling datasets which are described with ODEs well, you will learn faster, and with smaller dataset. I think, this what the authors of the original article meant.

In a similar way, CNNs recognize images much better, because their features are translation invariant whereas fully connected net needs to learn to recognize a cat in each different position from scratch.


How can we derive a Convolution Neural Network from a more generic Graph Neural Network?

Spectral Graph Convolution
We use the Convolution Theorem to define convolution for graphs. The Convolution Theorem states that the Fourier transform of the convolution of two functions is the pointwise product of their Fourier transforms:
$$\mathcal{F}(w*h) = \mathcal{F}(w) \odot \mathcal{F}(h) \tag{1}\label{1} $$
$$ w * h = \mathcal{F}^{-1}(\mathcal{F}(w)\odot\mathcal{F}(h))  \tag{2}\label{2}$$
Here $w$ is the filter in spatial domain(time domain) and $h$ is the signal in spatial domain(time domain). For images this signal $h$ is a $2D$ matrix and for other cases this $h$ can be a $1D$ signal.
Assume we have $n$ number of node in a graph. In graph fourier transform the eigenvalues carry the notion of frequency. $\Lambda$ is the $ n \times n$ egenvalue matrix and it is a diagonal matrix. We can write equation 2 as:
$$w * h = \phi(\phi^{T}w \odot \phi^{T}h) = \phi\hat{w}(\Lambda)\phi^{T}h \tag{3}$$
Here $\phi$ is the eigenvector matrix of graph Laplacian $\in R^{n \times n}$,
$\hat{w}(\Lambda)$ is the filter in spectral domain(frequency domain) $\in R^{n \times n}$ a diagonal matrix, $h$ is the $1D$ graph signal $\in R^{n}$ in spatial domain and w is the filter in spatial domain $\in R^{n}$.
Vanilla Spectral GCN
We define the spatial convolutional layer such that given layer $h^{l}$ , the activation of the next layer is:
$$h^{l+1}=\sigma(w^l*h^l)  \tag{4}\label{4},$$
where $\sigma$ represents a nonlinear activation and $w^l$ is a spatial filter and $h$ is the graph signal.
We can perform the above equation in terms of spectral graph convolution operation as:
$$h^{l+1}=\sigma(\hat{w}^l*\hat{h}^l)  \tag{5}\label{5},$$
where $\hat{w}$ is the same filter but in the spectral domain(frequency domain). In case of vanilla GCN this equation yeild to:
$$ h^{l+1} = \sigma(\phi\hat{w}^{l}(\Lambda)\phi^{T}h)  \tag{6}\label{6}$$
Now, we will learn the $\hat{w}$ using backpropagation.
This vanilla GCN has several limitations, like larger time complexity and this does not guarantee localization in the spatial domain that we get from CNN's filter.
In next works, such as SplineGCNs, ChebNet, Klipf and Welling's GCN, and many other works address those issues, and try to solve them.
Note that we can think of ChebNet and Klipf and Welling's GCN as a message-passing system, but, in the background, they are computing spectral convolution and also they use some standard assumption that's why we do not need any eigenvector and we implement them in the spatial domain, but still they are spectral convolution.
There is also another branch in graph convolution called spatial graph convolution. I only talked about the spectral graph convolution.


What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?

The first-visit and the every-visit Monte-Carlo (MC) algorithms are both used to solve the prediction problem (or, also called, "evaluation problem"), that is, the problem of estimating the value function associated with a given (as input to the algorithms) fixed (that is, it does not change during the execution of the algorithm) policy, denoted by $\pi$. In general, even if we are given the policy $\pi$, we are not necessarily able to find the exact corresponding value function, so these two algorithms are used to estimate the value function associated with $\pi$.

Intuitively, we care about the value function associated with $\pi$ because we might want or need to know "how good it is to be in a certain state", if the agent behaves in the environment according to the policy $\pi$.

For simplicity, assume that the value function is the state value function (but it could also be e.g. the state-action value function), denoted by $v_\pi(s)$, where $v_\pi(s)$ is the expected return (or, in other words, expected cumulative future discounted reward), starting from state $s$ (at some time step $t$) and then following (after time step $t$) the given policy $\pi$. Formally, $v_\pi(s) = \mathbb{E}_\pi [ G_t \mid S_t = s ]$, where $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ is the return (after time step $t$).

In the case of MC algorithms, $G_t$ is often defined as $\sum_{k=0}^T  R_{t+k+1}$, where $T \in \mathbb{N}^+$ is the last time step of the episode, that is, the sum goes up to the final time step of the episode, $T$. This is because MC algorithms, in this context, often assume that the problem can be naturally split into episodes and each episode proceeds in a discrete number of time steps (from $t=0$ to $t=T$).

As I defined it here, the return, in the case of MC algorithms, is only associated with a single episode (that is, it is the return of one episode). However, in general, the expected return can be different from one episode to the other, but, for simplicity, we will assume that the expected return (of all states) is the same for all episodes.

To recapitulate, the first-visit and every-visit MC (prediction) algorithms are used to estimate $v_\pi(s)$, for all states $s \in \mathcal{S}$. To do that, at every episode, these two algorithms use $\pi$ to behave in the environment, so that to obtain some knowledge of the environment in the form of sequences of states, actions and rewards. This knowledge is then used to estimate $v_\pi(s)$. How is this knowledge used in order to estimate $v_\pi$? Let us have a look at the pseudocode of these two algorithms.

$N(s)$ is a "counter" variable that counts the number of times we visit state $s$ throughout the entire algorithm (i.e. from episode one to $num\_episodes$). $\text{Returns(s)}$ is a list of (undiscounted) returns for state $s$.

I think it is more useful for you to read the pseudocode (which should be easily translatable to actual code) and understand what it does rather than explaining it with words. Anyway, the basic idea (of both algorithms) is to generate trajectories (of states, actions and rewards) at each episode, keep track of the returns (for each state) and number of visits (of each state), and then, at the end of all episodes, average these returns (for all states). This average of returns should be an approximation of the expected return (which is what we wanted to estimate).

The differences of the two algorithms are highlighted in $\color{red}{\text{red}}$. The part "If state $S_t$ is not in the sequence $S_0, S_1, \dots, S_{t-1}$" means that the associated block of code will be executed only if $S_t$ is not part of the sequence of states that were visited (in the episode sequence generated with $\pi$) before the time step $t$. In other words, that block of code will be executed only if it is the first time we encounter $S_t$ in the sequence of states, action and rewards: $S_0, A_0, R_1, S_1, A_1, R_2 \ldots, S_{T-1}, A_{T-1}, R_T$ (which can be collectively be called "episode sequence"), with respect to the time step and not the way the episode sequence is processed.  Note that a certain state $s$ might appear more than once in $S_0, A_0, R_1, S_1, A_1, R_2 \ldots, S_{T-1}, A_{T-1}, R_T$: for example, $S_3 = s$ and $S_5 = s$.

Do not get confused by the fact that, within each episode, we proceed from the time step $T-1$ to time step $t = 0$, that is, we process the "episode sequence" backwards. We are doing that only to more conveniently compute the returns (given that the returns are iteratively computed as follows $G \leftarrow G + R_{t+1}$).

So, intuitively, in the first-visit MC, we only update the $\text{Returns}(S_t)$ (that is, the list of returns for state $S_t$, that is, the state of the episode at time step $t$) the first time we encounter $S_t$ in that same episode (or trajectory). In the every-visit MC, we update the list of returns for the state $S_t$ every time we encounter $S_t$ in that same episode.

For more info regarding these two algorithms (for example, the convergence properties), have a look at section 5.1 (on page 92) of the book "Reinforcement Learning: An Introduction" (2nd edition), by Andrew Barto and Richard S. Sutton.


What is the best AI or Machine Learning SaaS application?

From my understanding you are referring to two different but related innovative workplace systems. These are Automated Data Capture and Entry and Document Automation.

Businesses generate massive amounts of documents such as receipts, invoices, contracts, memo's, minutes, reports e.t.c. The information from the above documents is then to be entered into different software i.e. Accounting, Payroll, Excell and the web portal. Without a doubt data entry and document management can be a time consuming and tedious task in many modern offices.

Automated Data Entry and Document Automation can help reduce the amount of time that employees spend on data entry and managing documents. This improves the speed and accuracy of work at the workplace.

The above systems work by combining OCR technology with AI. When a document is scanned, it is initially just an image file because computers cannot distinguish text characters from the background image. With OCR software you can extract text from a document. As a result the document becomes searchable and easier to work with. Document automation software can then enter the information into a spreadsheet or other software automatically.

Some players in this field include Softworks (Automated Data Entry), Xtracta (Automated Data Entry and Document Automation) and Dynamic AI. Intuit also has a paperless productivity bundle that includes equipment such as the ScanSnap scanner (Automated Data Capture) and Intuit Hubdoc (Document Automation).


What are the differences between stability and convergence in reinforcement learning?

Sometimes when training, particularly in reinforcement learning, the model can become unstable due to the amount of variance that exists in the training data that the agent generates by interacting with the environment.  This is certainly a problem at the start of training as you can get huge outliers in the data because the agent is behaving randomly.  You can find that just one update to the policy could potentially make it collapse because it moves the policy into some obscure region, e.g. so the agent always take a particular action.  You can make training more stable by using larger batches and a smaller learning rate so it takes smaller steps at a time, but the downside to that is training is slower.  So you need to test different hyperparameters to find a good trade-off between the two.  You can also use an training architecture such as Proximal Policy Optimization (PPO) which clips the amount the policy can move in any given update to try and maintain some stability.

Convergence is a term used to describe when the model has found an optimal policy and isn't learning any further, usually demonstrated when the reward plateaus for a certain number of episodes.  Of course, it may have settled on a local optima, and other global optima may exist; the data you present and the way you train your model may yield better results - again, all part of testing and experimentation.


Medical diagnosis systems based on artificial neural networks

Yes, there are many, actually. A Google search turned this paper Artificial Neural Networks in Medical Diagnosis (2011) by Al-Shayea up.

Not only are they used in disease diagnosis, but even with things like prescribing medicines. In fact, the top project for a hackathon at my school analysed thousands of research articles, and took a patient's medication history as input, to best recommend them specific medicines. Check it out.


Machine learning to detect wrong address data

The comments are off base.  Having worked in validation of data as a consultant for Nasdac, Amex, and Lexis Nexis I can tell you that using the UNIX sed -r or pcrelib is insufficient to do a stellar cleansing of address data.

Although none of those companies did this at the time I was consulting, the application of current machine learning is easy to infer from the basic characteristics of data.  What is needed is a good record of incoming data, cleaned data, and rejected records to constantly use as a reference.

The manual process would be a nightmare.  For instance, misspellings are more indicative of true data than properly spelled ones in some but not all cases (unless the falsified data generator is programmed to synthesize the existing distribution of misspellings in authentic data sources).

Try to profile that in static code such that the data validation will adapt to data trends without requiring a maintenance workforce.  Those kinds of things never get addressed in a typical IT environment.  Non-adaptive validation would be a gross mistake, especially if the volume is high, like for M16 or Liberty Cross or the NSA or Interpol.

What you want is an extremely fast index of well authenticated good addresses (perhaps corroborated by public records with local authentication policies) and a feature extraction from it using auto encoding or some similar and perhaps better methodology and its associated algorithms.

Then you can train the classification of fake and authentic addresses based on extracted feature profiles.

You will also want reinforcement, because the attackers (those trying to create false identities using plausible addresses) will adapt to the training of the current system.  The authentication system must stay steps ahead of those trying to defeat it, which they will likely try to do once the existence of authentication automation is detected.

One can fend of attackers by placing misinformation strategically and then tracing the input sources back to the attacker based on the misinformation seeds.  That only works if you have law enforcement in your camp.


What is the time complexity of the forward pass algorithm of a feedforward neural network?

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only represent the inputs (so they do not perform any operation).

Each hidden neuron performs a linear combination of its inputs followed by the application of a non-linear (or activation) function. So, each hidden neuron $j$ performs the following operation

\begin{align}
o_j = \sigma \left(\sum_{i}^{15} w_{ij}x_i \right),\tag{1}\label{1}
\end{align}

where $i$ is the input coming from the input neuron $i$, $w_{ij}$ is the weight of the connection from the input neuron $i$ to the hidden neuron $j$, and $o_j$ is used to denote the output of neuron $j$.

There are $20$ hidden neurons and, for each of them, according to equation $\ref{1}$, we perform $15$ multiplications (ignoring any multiplications that might be associated with the activation function), so $15*20 = 300$ multiplications are performed at the (only) hidden layer. In general, if there are $n$ inputs and $m$ hidden neurons, then $n*m$ multiplications will be performed in the first hidden layer.

Now, each neuron $j$ at the next layer (in this case, the output layer), also performs a linear combination followed by the application of an activation function

\begin{align}
o_j = \tau \left(\sum_{i}^{20} w_{ij}x_i \right),\tag{2}\label{2}
\end{align}

where $\tau$ is another activation function which might or not be equal to $\sigma$, but we ignore all multiplications that might involve the application of the activation functions (we just want to count the ones in the linear combinations). Of course, in this case, $x_i$ corresponds to the activation of neuron $i$ (of the hidden layer).

Similarly to the previous reasoning, there are $2$ output neurons and, to compute the output of each of them, $20$ multiplications are performed (in the linear combination), so there are a total of $2*20 = 40$ multiplications at the output layer.

So, an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons will perform $15*20 + 20*2 = 340$ multiplications (excluding activation functions). Of course, in this case, the number of multiplication depends not only on the number of neurons but also on the input size.

In general, an MLP with $n$ inputs, $M$ hidden layers, where the $i$th hidden layer contains $m_i$ hidden neurons, and $k$ output neurons will perform the following number of multiplications (excluding activation functions)

\begin{align}
nm_{1} + m_{1}m_{2} + m_{2}m_{3} + \dots + m_{M-1}m_{M} + m_{M}k = nm_{1} + m_{M}k + \sum_{i=1}^{M-1} m_{i}m_{i+1}
\end{align}

which, in big-O notation, can be written as

\begin{align}
\Theta\left(nm_{1} + m_{M}k + \sum_{i=1}^{M-1} m_{i}m_{i+1} \right)
\end{align}

where $\Theta(\cdot)$ is used (as opposed to $\mathcal{O}(\cdot)$) because this is a strict bound. If you have just one hidden layer, the number of multiplications becomes

\begin{align}
\Theta\left(nm_{1} + m_{1}k \right)
\end{align}

Of course, at each layer, the number of multiplications can be computed independently of the multiplications of the other layers (you can think of each layer as a perceptron), hence we sum (and not e.g. multiply) the multiplications of each layer when computing the total number of multiplications of the whole MLP.

In general, when analyzing the time complexity of an algorithm, we do it with respect to the size of the input. However, in this case, the time complexity (more precisely, the number of multiplications involved in the linear combinations) also depends on the number of layers and the size of each layer. The time complexity of a forward pass of a trained MLP thus is architecture-dependent (which is a similar concept to an output-sensitive algorithm).

You can easily include other operations (sums, etc.) in this reasoning to calculate the actual time complexity of a trained MLP.


If mankind can create artificial life in a machine, when would we define it's death?

Death as we know it for natural life is terminal. That is once dead, natural life cannot come back (at least in the current understanding and with current technologies---some people believe otherwise).

Death for AI is trickier. There may be only one scenario: Global destruction: Extreme scenario where everything supporting the existence of an AI disappears. This is equivalent to death in natural life, and low probability. It means all AIs die at once (as well as us).

We also do not know the degree and form of embodiment necessary for AGIs. We can assume now that hardware is replaceable indefinitely, thus "limiting" death to the above extreme scenario. But AGIs "body" may not be indefinitely replaceable. Then a definition closer to natural life death may be necessary.

We see arguments for two other scenarios, that I refute below:

"Static Death": An AI is still "defined" or "saved" somewhere (whatever it means actually), but it is not authorized or able to use resources. Assuming an AI is made of hardware and software, it is like a program stored on a disk, but without permission to run.

"Dynamic Death": Under the same characterization of AI as hardware and software, dynamic death is the invalidation of progress akin to strong liveness properties, where an AI is trapped in an infinite loop (or a void loop), in a form of "active death", as what happens to Sisyphus in Greek mythology. This is different from static death, as the AI still uses dynamic resources, although it cannot make progress. Continuing under the same assumptions, such AI could be "loaded" in main memory, or locked waiting for inputs or outputs to complete.

Note that in these two scenarios, rebirth is possible, and they also subsume that there is an entity that can decide conditions for rebirth, or preventing it completely. Would this entity be an "admin", a god, other AIs, or a human is another question, really.

The terms "death" and "rebirth" here could just be changed for "imprisoning", where the dynamic version would be like our human prisons, and the static version would be like SciFi cryogeny. This is a bit of a stretch, but we can see an equivalence, and no good reason to qualify these two scenarios as deaths.

In conclusion, death for AI seems to be an exceptional, singular scenario, so AI cannot die in practice, except if we are wrong on how we think we can make AGIs. AI can however be imprisoned forever.

Note: The terminology above is completely made-up for the post. I do not have citations to back some claims, but it is based on readings and personal work (including in software verification).


If an heuristic is not admissible, can it be consistent?

For a heuristic to be admissible, it must never overestimate the distance from a state to the nearest goal state.

For a heuristic to be consistent, the heuristic's value must be less than or equal to the cost of moving from that state to the state nearest the goal that can be reached from it, plus the heurstic's estimate for that state.

What this means is that, as you move along the sequence of nodes from start to goal that the heuristic recommends, a consistent heuristic should monotonically decrease in value. A consistent heuristic is thus also always admissible.

Notice that this means that if a heuristic is not admissible (like yours), it is also not consistent (by the contrapositive).

Therefore, if you already know your heuristic is not admissible, you should not be surprised that it is not consistent.

It seems most likely that you may have confused the definition of consistent for monotone. A consistent heuristic is both monotone and admissible.

As Neil Says, if you want to know why your specific heuristic is inadmissible, you should post another question about it, or modify this one.


In DQN, updating target network every N steps or slowly update every step is better?

Most of papers and my experience support hard update, once per N steps. N is usually very bug, range in between 10^4 to 10^6. DQN training is slow. But that depend on the problem. If you DQN converge with soft update with weight ~.1 (N~10) your problem could be very simple.


How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG:

I added a reward component based on the actuator movement, i. e. the delta of actions from one step to the next.

Excessive action changes are punished now and this could mitigate the tendency to oscillate. The solution is nnot really perfect, but it works for the moment.

This finding is detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into https://github.com/opt12/Markov-Pilot/tree/master/thesis

I'll be glad to get feedback on it. And I'll be glad to hear better solutions than adding a delta-punishment.

Regards, Felix


What is feature embedding in the context of convolutional neural networks?

The term feature embedding appears to be a synonym for feature extraction, feature learning etc. I.e. a form of embedding/dimension reduction (with the caveat the goal may not be a lower dimensional representation but one of equal dimensionality, but more meaningfully expressed):

  Feature embedding is an emerging research area which intends to transform features from the original space into a new space to support effective learning.

Generalized Feature Embedding for Supervised, Unsupervised, and Online Learning Tasks (2019)

  Feature embedding aims to learn a low-dimensional vector
  representation for each instance to preserve the information
  in its features.

Large-Scale Heterogeneous Feature Embedding (2019)


Understanding Generalized Advantage Estimate in reinforcement learning

Bootstrapped estimate is biased because it based on $V(s_{t+1})$ which is usually a biased estimate by some estimator such as neural network.

I don't think this statement is totally correct in the context of the paper. Quoting the paper:

Taking $\gamma < 1$ introduces bias into the policy gradient estimate, regardless of the value function’s accuracy. On the other hand, $ \lambda < 1$ introduces bias only when the value function is inaccurate.

So the initial $\gamma$-discounting introduces bias regardless of the accuracy of the $V(s)$.
Same point is reiterated in the footnote on the page 3:

Note, that we have already introduced bias by using $A^{\pi,\gamma}$ in place of $A^\pi$; here we are concerned with obtaining an unbiased estimate of $g^\gamma$, which is a biased estimate of the policy gradient of the undiscounted MDP.

To summarize the above:

the original MDP under consideration is undiscounted
authors introduce $\gamma$-discounting to reduce variance (at cost of bias) of the PG estimate - this happens even if we had accurate $V(s)$
later, authors introduce another $\lambda$-discounting which also controls bias-variance tradeoff - now due to inaccuracy in $V(s)$

As for why lower $\gamma$ reduces variance (p.2 above) - your explanation is on point: smaller $\gamma$ $\Rightarrow$ smaller weights for distant rewards $\Rightarrow$ less variance (but more bias).


Is there an AI technology that can predict human behaviour?

GPT2 predicts the next word that people will say. https://openai.com/blog/better-language-models/
Facebook predicts what will make you keep using their site.
Youtube predicts what videos you will click on.


CNN output generally has more than one category in one-hot categorization?

When you use sigmoid activation it is applied independently to all outputs so outputs won't sum up to 1. Sigmoid is usually used for binary logistic regression where you only have 2 classes. In your case you should use softmax activation at the output which will squash outputs to range [0,1] and additionally make them sum up to 1.