Is there an online tool that can predict accuracy given only the dataset?

If I get correctly what you're asking, the answer is no, there is no way of knowing in advance how good a model would perform in a dataset without training a model on it. That's the whole point of data science, you try, you analyse the results, you try again using the knowledge you got from your previous attempts. It would be nice to hack the whole field and know in advance what to do to get the perfect model but it is rather unrealistic.

Anyway, there are some standard steps that usually help understanding if you're going on the right direction. For example creating a random benchmark to see how much your model is better than a random one. To create such benchmark you definitely don't need a specific tool, all main programming languages provide built in function to generate random numbers, and that's basically all that you need. For example in python you could do something like:

import numpy as np

# create 30 true labels
y_true = [1,2,3]*10

# generate 30 random labels
y_random = np.random.randint(0,3,size=30)

# calculate accuracy for random model
acc_random = sum(y_true==y_random)/len(y_true)

Another thing that is worth to mention is that in the academy people are pushing more and more to use the same datasets in order to have comparable results. for example if you're trying to train an architecture that should be used for images classification, some golden dataset you must test your models on are the MNIST ones. It is well known that on these datasets a good model should achieve more than 99% accuracy, therefore this is an establish lower bound for these datasets.


TensorFlow estimator DNNClassifier fails to fit simple data

Normalise your inputs.

Neural networks work poorly outside of relatively small numerical ranges on input. An ideal range is for each feature to be drawn from $\mathcal{N}(0,1)$ i.e. a Normal distribution with mean $0$ and standard deviation $1$. In your case, divide both parts of $\mathbf{x}$ by $25$ and subtract $1$ would probably suffice.

Your neural network architecture is completely overblown for the problem at hand. That may be because you were trying to force it to fit this data (and failing because of lack of normalisation). Try something more like: hidden_units = [20,10]


How to handle multiple types of decisions?

Your intuition is correct, neural networks are a no go (except see bottom).

It seems like you'd want to look into the ML sub-field called Reinforcement Learning.
In a nutshell, RL offers a set of methods to learn what is the best action to take in a given situation.

More formally, in RL settings the algorithm learns from experience by observing a reward ($R$) associated to an action ($a$).

RL problems can be conceptualized as Markov Decision Processes (MDPs). From Sutton & Barto:

  MDPs are a classical formalization of sequential decision making, where actions influence not just immediate rewards, but also subsequent situations, or states, and through those future rewards.

MDP can be faced with several methods, mainly Monte Carlo Methods and Temporal-Difference Learning.

In short, both use the Bellman equation in different ways. A key component of the eq. is the discounting term ( $\gamma$ ). Values for rewards associated with future states are discounted (multiplied) by $\gamma<1$ in order to modulate the importance of future rewards.

Since you're concerned with strategy, another way to influence the agent's learning wrt. the environmental rewards is the adoption of an epsilon value $\epsilon<1$. When defining an $\epsilon$-greedy policy, the agent will choose the action with highest value $Q$ only with a probability equal to $1-\epsilon$. This enables the agent to balance exploitation of previous experience with the exploration of the environment. Very useful in cases where an immediate low reward compromises the achievement of a much higher reward later during the episode.

There's many other ways to influence the behaviour of your learner of choice to respond differently to its environment. For a complete view, I recommend Sutton and Burton who offer their book in pdf format for free at the given link.

Edit: almost forgot. Neural networks can be useful when combined with more classical RL methods, say Q-learning (a type of Temporal-difference learning), taking on the name of deep Q-learning.
Alpha go also uses a combination of Monte Carlo methods combined with 2 NNs for Value evaluation and policy evaluation Wikipedia.
I wouldn't necessarily venture in this domain without first having a clear overview of classical RL methods.

Hope this helps.


How do evolutionary algorithms have advantages over the conventional backpropagation methods?

Unlike backpropagation, evolutionary algorithms do not require the objective function to be differential with respect to the parameters you aim to optimize. As a result, you can optimize "more things" in the network, such as activation functions or number of layers, which wouldn't be possible in the standard backpropagation.

Another advantage is that by defining the mutation and crossover functions, you can influence how the parameter search space should be explored.


Should forecasting with neural networks only be treated as a supervised learning (regression) problem?

I think the choice of technique strongly depends on how fine-grained your forecast-predictions need to be.
When it comes to forecasting by Reinforcement Learning (RL), one prominent example is the stock-trading RL agent. The agent must decide which stock to buy or sell, thereby drawing upon predictions concerting the expected future development of some stock.
Given this approach, you would not necessarily let the RL agent explicitly generate estimates of how stock prices are gonna develop at any point, but instead you would only observe the predicted decision concerning whether to buy or sell etc.
But if you think hard enough, I am certain that you could come up with setups of RL agents that would allow you to explicitly generate future estimates of values to be forecast. In this case, the final buy/sell decision would have to depend on the explicit future stock price predictions to enforce accurate predictions.
Concerning unsupervised learning, you could cluster data points (training samples) with respect to how some value(s) of interest changed $t$ time steps in the future (after having observed the training sample). Then, you could associate clusters with rough forecast-estimates. After all, you would treat the forecast value as a label associated with data points.
Afterwards, you could use some kind of nearest neighbor approach to determine which cluster is closest to some novel data sample. Then, you take as a prediction for the new data sample the forecast prediction (i.e. label) that is associated with the closest cluster/prototype etc. But strictly speaking, as soon as you start turning forecast values (which were previously part of some unlabeled time-series dataset) into labels, you turn the training procedure of course into a supervised technique again.
How well especially the latter training approach would work, I can't tell since I have never heard anyone using this method. But if training data is too scarce to employ some deep learning method, why not giving it at least a try if accuracy doesn't have to be too precise?
After all, it's just a matter of creativity and testing which method works best given your specific machine learning problem at hand.


In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

Okay - the answer is here  https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
Gradient Vector/Matrix/2D tensor of Loss function wrto Weight
$$
C = \frac{1}{2} \sum_j (y_j-a^L_j)^2
$$
Assuming a neural net with 2 layers, we have the final Loss as
$$
C = \frac{1}{2} \sum_j (y_j-a^2_j)^2
$$
Where
$$
a^2 = \sigma(w^2.a^1)
$$
We can then write
$$
C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)
$$
Where
$$
v= y-a^2
$$
Partial Derivative of Loss function wrto Weight
For the last layer, lets use Chain Rule to split like below
$$
\frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
$$
$$
 \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
$$
$$
\frac{\partial v}{\partial w^2}=  \frac{\partial (y-a^2)}{\partial w^2} =
0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
$$
$$
\frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$

Now we need to find $\frac{\partial a^2}{\partial w^2}$
Let
$$
a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2)
$$
$$
z^2 = sum(w^2 \otimes a^1 )
$$
$$
z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1
$$
We now need to derive an intermediate term which we will use later
$$
\frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}
$$
$$
=\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
$$
$$
\frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
$$
How the above is, you need to check this  in https://explained.ai/matrix-calculus/#sec6.2
Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.
Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)
Going back to  $Eq \;(B.2)$
$$
\frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}
$$
Using $Eq \;(B.3)$ for the term in left
$$
=  \frac{\partial a^2}{\partial z^2} * (a^{1})^T
$$
$$
=  \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T
$$
$$
\frac {\partial a^2}{\partial w^2} =   \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)
$$
Now lets got back to partial derivative of Loss function wrto to weight
$$
\frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$
Using $Eq \;(B.4)$ to substitute in the last term
$$
= v(0- \sigma^{'}(z^2) * (a^{1})^T)
$$
$$
= v*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
= (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
\frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T
$$


If I understood correctly, the model is a polynomial equation

No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a polynomial, but, for example, in a neural network, you can combine many sigmoids to build complicated functions that may not necessarily be polynomials.
We usually distinguish between linear (straight-lines) and non-linear functions (rather than polynomials and non-polynomials). In some cases, it is straightforward to visualize the function that your model computes: for example, in the case of linear regression, once you learned the coefficients (i.e. the slope and y-intercept), you can plot the learned straight-line function. In other cases, for example, in the case of neural networks, it is not fully clear how to visualize the function that your model computes, given that it is the composition of many non-linear functions (typically, ReLUs, sigmoids or hyperbolic tangents).
If you are interested in solving problems with polynomials, take a look at polynomial regression.

and its weights are calculated by training the model with data sets.

Yes, in machine learning, we want to find a function that "fits the given data", and the specific meaning of "fitting the data" depends on the specific machine learning technique.
For simplicity, let's focus on supervised learning, a machine learning technique where we are given a labelled dataset, i.e. a dataset of pairs $D = \{(x_1, y_1), \dots, (x_N, y_N)\}$, where we assume that $f(x_i) = y_i$, for some typically unknown function $f$, and $y_i$s are the labels (the outputs of $f$) and $x_i$ the inputs of $f$. The goal is to find function $g_{\theta}$ that approximates well $f$. I will soon describe what the subscript $\theta$ represents.
For simplicity, let's assume that $f$ is a linear function (i.e. a straight-line). So, we can define a linear model $g_{\theta}$ that we can use to find a function that approximates well $f$. Here is the linear model
$$g_{\theta}(x) = ax + b,$$
where

$g_{\theta}(x)$ is the output
$x$ is the input
$ax + b$ is the linear function (a straight-line)
$a$ is the slope (a parameter, aka weight)
$b$ is the $y$-intercept (another parameter)
$\theta = \{ a, b \}$ (the set of parameters of the linear model)

Why is this a model? I call this a model because, depending on the specific values of the parameters $\theta$, we have different specific functions. So, I am using the term "model" as a synonym for a set of functions, which, in this case, are limited by the definition $ax + b$ and the specific values that $a$ and $b$ (i.e. $\theta$) can take.
So, what do we do with this linear model? We want to find a specific set of parameters $\hat{\theta}$ (note that I use the $\hat{ }$ to emphasize that this is a specific configuration of the variable $\theta$) that corresponds to a linear function (a straight-line) that approximates $f$ well.  In other words, we need to find the parameters $\hat{\theta}$, such that $g_\hat{\theta} \approx f$, where $\approx$ means "approximately computes".
How do we do that? We typically don't know $f$, but we know (or assumed) that $f(x_i) = y_i$, so the labeled dataset $D$ contains information about our unknown function $f$. So, the idea is that we can use the dataset $D$ to find a specific set of parameters $\hat{\theta}$ that corresponds to some function that approximates $f$ according to the information in $D$.
This process of finding $\hat{\theta}$ based on $D$ is often denoted as "fitting the model to the data". There different ways of fitting the model to the data that differ in the way they compute some notion of distance between the information in $D$ and $g_{\hat{\theta}}$. I will not explain them here because this answer is already quite long. If you want to know more about it, you should take a book about the topic and read it.

What are some examples of functions that machine learning models compute?

I don't have specific examples, but you can easily try to fit a linear regression model to some labelled data, then plot the function that you found. You could use the Python library sklearn to do that.


How is the percentage or the probablity calculated using Loss function in Facenet Model?

The facenet model is just the head of the model. The architecture is similar to the enocdr part of an autoencoder, but it uses supervised learning instead of unsupervised learning. The network is called a siamese network The triplet loss helps make the embeddings more representative of the input image/person, with the embedding distance being as large as possible for difference people and vice versa.

However the embeddings is just the representation of the people. It doesn't contain information directly mapping to what person it is. The classification head is used after the Face Net feature extraction top is used.

One-shot learning method for person classfication

This method saves the calculated embeddings of people in a database. The face recognition system first calculates the embeddings of the new, unknown image to be classified. The system loop through the embeddings database and calculates the Euclidean distance of the unknown embeddings and the embeddings in the database.

After computing, it chooses the smallest distance of all and compares it to a threshold set by you. If the distance is larger than the threshold, the classified class is unknown , or else the resulting class will be the embeddings which results to the lowest distance.

Example code from the deeplearning.ai coursera course (great course on AI btw):
Example code of one shot face recognition

Advantages and disadvantages of the system

The system have advantages ranging from no training needed and can classify unknown class. However, once the number of people in the database becomes huge, the system will run slower. It's performance may also differ depending on the quality of the reference image.

SVM (Support Vector Machine) method

This is the method used in the github repository of your previous posts. It uses a one layer nn classifying head to output the classified class.

Advantages and disadvantages of the system

The system have the flexibility of adapting to multiple photo environment and no affecting of performance due to reference images. This method also works well for larger database of people. Triplet loss can also be removed using this method. You can train the network directly from propagating the loss with the probability predicted. However, This method requires re-training when a new person is added to the system. Multiple images of a person is also needed.


What is the difference between artificial intelligence and swarm intelligence?

Swarm intelligence (SI) is a sub-field of or an approach to artificial intelligence (AI), where you have multiple individuals (for example, artificial ants), which collectively can produce what we (or most of us) would intuitively call intelligent behaviour.
SI is sometimes categorized as a sub-field of evolutionary computation (which also includes evolutionary algorithms, such as genetic algorithms, genetic programming, evolution strategies, and so on), which is often considered a sub-field of AI or techniques to produce artificial intelligence, because all these techniques are often based on the use of multiple individuals/solutions (that either compete or collaborate with each other).
One of the most commonly used SI techniques is ant colony optimization algorithms (proposed by M. Dorigo and further developed by other people like Luca M. Gambardella), which have been successfully applied to solve the non-decision version of the NP-complete problem (in simple words, it's a combinatorial problem that may require exponential time to be solved in the usual case) known as the travelling salesman problem. There are other SI techniques, which are somehow similar to ACO algorithms, such as particle swarm optimization or the artificial bee colony algorithm.
Occasionally, SI may also be categorised as a sub-field of computational intelligence, which often refers to specific techniques to create artificially intelligent systems (i.e. programs that exhibit what we would call intelligence) that are more based on or inspired by the biology, such as neural networks, genetic algorithms, or, in fact, SI algorithms, such as ACO algorithms. However, CI can also be considered a sub-field of AI, given that it studies techniques to produce artificial intelligence, so, in the end, as I said above, SI is an approach to AI, which includes other approaches, such as evolutionary algorithms, rule-based systems, deep learning or other machine learning techniques.


What are state-of-the-art ways of using greedy heuristics to initially set the weights of a Deep Q-Network in Reinforcement Learning?

You can check out Bootstrapped DQN, with a demonstration video. Without reading much of the paper, it seems the authors use a different sampling strategy and an action-guide for specific instances.

Another way to initially set weights for the network is to create a dataset of moves (correct, incorrect, etc, as long as they are relevant) and have the network initially learn the dataset. This also helps debugging, as you can see whether the network can actually learn the policy used in the dataset. After learning the dataset, use the same learned network with DQN and start with a smaller exploration rate (like 0.5 instead of 1.0).


What artificial intelligence strategies are useful for summarization?

The following post has a bit of math, which I hope helps to explain the problem better. Unfortunately it seems, this SE site does not support LaTex:

Document summarization is very much an open problem in AI research. One way this task is currently handled is called "extractive summarization". The basic strategy is as follows: Split this document into sentences and we will present as a summary a subset of sentences which together cover all the important details in the post. Assign sentence $i$, $1 \leq i \leq n$, a variable $z_i \in \{ 0, 1 \}$, where $z_i = 1$ indicates the sentence was selected and $z_i = 0$ means the sentence was left out. Then, $z_i z_j = 1$ if and only if both sentences were chosen. We will also define the importance of each sentence $w_i$ for the sentence $i$ and interaction terms $w_{i,j}$ between sentences $i$ and $j$.

Let $x_i$ be the feature vectors for sentence $i$. $w_i = w(x_i)$ captures how important it is to include this sentence (or the topics covered by it) while $w_{i,j} = w(x_i,x_j)$ indicates the amount of overlap between sentences in our summary. Finally we put all this in a minimization problem:

\begin{aligned}
\underset{z_i}{\text{maximize }} & \sum_{i} w_i z_i - w_{i,j} z_i z_j \\
\text{s.t. } & z_i = 0 \text{ or } 1
\end{aligned}

This tries to maximize the total weight of the sentences covered and tries to minimize the amount of overlap. This is an integer programming problem similar to finding the lowest weight independent set in a graph and many techniques exist to solve such problems.

This design, in my opinion, captures the fundamental problems in text summarization and can be extended in many ways. We will discuss those in a bit, but first, we need to completely specify the features $w$. $w_i = w(x_i)$ could be a function only of the sentence $i$, but it could also depends on the place of the sentence in the document or its context (Is the sentence at the beginning of a paragraph? Does it share common words with the title? What is its length? Does it mention any proper nouns? etc)

$w_{i,j} = w(x_i,x_j)$ is a similarity measure. It measures how many repetitions there will be if we include both words in the sentence. It can be defined by looking at common words between sentences. We can also extract topics or concepts from each sentence and see how many are common among them, and use language features like pronouns to see if one sentence expands on another.

To improve the design, first, we could do keyphrase extraction, i.e. identify key phrases in the text and choose to define the above problem in terms of those instead of trying to pick sentences. That is a similar problem to what Google does to summarize news articles in their search results, but I am not aware of the details of their approach. We could also break the sentences up further into concepts and try to establish the semantic meaning of the sentences ( Ponzo and Fila are people P1 and P2, a mall is a place P, P1 and P2 went to the place P at time T (day). Mode of transport walking.... and so on). To do this, we would need to use a semantic ontology or other common-sense knowledge databases. However, all the parts of this last semantic classification problem are open and I have not seen anyone make satisfactory progress on it yet.

We could also tweak the loss function above so that instead of the setting the tradeoff between the sentence importance $w_i$ and the diversity score $w_{i,j}$ by hand, we could learn it from data. One way to do this is to use Conditional Random Fields to model the data, but many others surely exist.

I hope this answer explained the basic problems that need to be solved to make progress towards good summarization systems. This is an active field of research and you will find the most recent papers via Google Scholar, but first read the Wikipedia page to learn the relevant terms


MCTS: How to choose the final action from the root

By far the most commonly used strategy is to select the child with the highest number of visits. This is as described in the 2008 paper you linked. It's also what's referred to as the "robust child" in the 2012 paper you linked.

In algorithm 2 of the 2012 paper, they actually use the highest average reward, which corresponds to "Max child". It looks like they're using the UCB1 policy, but they actually use a value of $0$ for the exploration parameter $c$, which makes the entire square root term drop out. This is also explained in the text at the end of your quote. But usually, a robust child / max visit count performs better.

Progressive Strategies for Monte-Carlo Tree Search is a different paper from 2008, in which these strategies are experimented with a bit. Usually, they perform similarly, but a robust child tends to be the best if there is any difference at all.


keras ValueError: Error when checking model target: expected activation_4 to have shape (None, 19) but got array with shape (100, 1)

Try to use the model like this, for example:

model = Sequential()
model.add(Dense(50, input_shape=(2,)))
model.add(Activation("relu"))
model.add(Dense(50, activation='softmax'))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='sgd', loss='mse', metrics=["accuracy"])

This means that first layer will have 50 neurons and can receive data in form of matrix with 2 columns and an unspecified number of rows.
So you can prepare your data in this form – 2 numbers for adding in each row.

Dense(50, input_shape=(2,))

At the end, you need a layer with 1 neuron and the 'linear' activation, because you expect one simple number as a result.

Dense(1, activation='linear')

And finally, use 'mse' loss function or something similar. 'categorical_crossentropy' is needed for classification tasks, not regression as needed for you.
See: https://keras.io/objectives/


Denoising Images When Training a Classification Model

You can denoise a certain fraction of images (preferably 0.25-0.3) randomly for each epoch.
Adding Gaussian noise to images gives better results.Refer this : Link


How does maximum approximation of the posterior choose a distribution?

Introduction: MAP finds a point estimate!

As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a point estimate (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a Bayesian version of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.

Maximum likelihood estimation (MLE)

Let $\theta$ be the parameters you want to find. For example, $\theta$ can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) \tag{1}\label{1}
\end{align}

where $p(X \mid \theta)$ is the likelihood of the data $X$ given the parameters $\theta$. In other words, we want to find the parameters $\theta$ such that $p(X \mid \theta)$ is the highest, where $X$ is your given training data, so $X$ is fixed.

The notation $p(X \mid \theta)$ can be confusing because, in a conditional probability distribution, $p(a\mid b)$, we often assume that $b$ is given and $p(a\mid b)$ is a distribution over $a$. However, in the case of MLE, $\theta$ in $p(X \mid \theta)$ is not fixed, but it is a variable, while $X$ is given and fixed. Hence we call $p(X \mid \theta)$ a likelihood rather than a probability density or mass function. Moreover, we often denote the likelihood as $\mathcal{L}(\theta; X) = p_{\theta}(X)$ (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable $\theta$. However, this is notation can also be confusing because we equate a function of a variable $\theta$ to a probability distribution over $X$. However, you should note that $p_{\theta}(X)$ is parametrized by $\theta$.

Therefore, the MLE estimation \ref{1} can also be written as follows

\begin{align}
\theta^*
&=
\operatorname{argmax}_\theta \mathcal{L}(\theta; X) \\
&=\operatorname{argmax}_\theta p_{\theta}(X)
\tag{2}\label{2}
\end{align}
where $\theta^*$ is the point estimate of the objective function.

This notation emphasizes the fact that we want to find $\theta$, such that the probability of the given data $X$ is maximized.

Maximum a posteriori (MAP)

MAP is similar to MLE, but the objective is slightly different. First of all, we assume that $\theta$ is a random variable, so we have an associated probability distribution $p(\theta)$.

Recall that the Bayes' rule is the following

\begin{align}
p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
\tag{3}\label{3}
\end{align}

The objective function in MAP estimation is

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
&= \operatorname{argmax}_\theta p(\theta \mid X) \tag{4}\label{4}
\end{align}

Given that $p(X)$ does not depend on $\theta$, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes

\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) p(\theta) \\
&= \operatorname{argmax}_\theta p(\theta \mid X)
\tag{5}\label{5}
\end{align}
which is the MAP objective.

What is the relationship between MLE and MAP?

In MAP, the objective is \ref{5}, which includes a prior over $\theta$, while, in MLE, equation \ref{1}, there is no such thing.

Therefore, in MAP, we can assume that the parameters $\theta$ follow a certain distribution, thanks to the usage of $p(\theta)$.

In both MAP and MLE, we want to find a point estimate (which can be a number, if you have just one parameter, or a vector of size $N$, if you have $N$ parameters).
MAP is equivalent to MLE if you use a uniform prior, that is, if $p(\theta)$ is a uniform distribution.

Which distribution fits the data $X$?

In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters $\theta$ follow a Gaussian distribution, so $p(\theta)$ will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.

How do you find $\theta$?

To find $\theta^*$, you can use an optimization method like gradient descent or, in certain cases, you can find a closed-form solution. See also Which distributions have closed-form solutions for maximum likelihood estimation?.

Resources

The following blog post MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation, by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.


Is there any research on the development of attacks against artificial intelligence systems?

Yes, there is some research on this topic, which can be called adversarial machine learning, which is more an experimental field.

An adversarial example is an input similar to the ones used to train the model, but that leads the model to produce an unexpected outcome. For example, consider an artificial neural network (ANN) trained to distinguish between oranges and apples. You are then given an image of an apple similar to another image used to train the ANN, but that is slightly blurred. Then you pass it to the ANN, which unexpectedly predicts the object to be an orange.

Several machine learning and optimization methods have been used to detect the boundary behaviour of machine learning models, that is, the unexpected behaviour of the model that produces different outcomes given two slightly different inputs (but that correspond to the same object). For example, evolutionary algorithms have been used to develop tests for self-driving cars. See, for example, Automatically testing self-driving cars with search-based procedural content generation (2019) by Alessio Gambi et al.


Research for misbehavior detection in WiFI networks

One popular technique for doing this is to use Artificial Immune Systems, an evolutionary computation approach which maintains a population of pattern detectors.

Here is a survey paper.


What are "function terms" in the context of an ontology?

If you are looking for the functional verbs used in VA/VE (value analysis / value engineering) look at:

https://www.valueanalysis.ca/functionanalysis.php
http://www.brcommunity.com/articles.php?id=b679
https://www.washoeschools.net/cms/lib08/NV01912265/Centricity/Domain/253/Social%20Studies/Marker%20Verbs.pdf


Yes you can, a few years ago I made a simple CNN for a single Arabic phoneme classification. You can use spectogram or using MFCC / MFSC as features, as long all data has the same size (use padding or cropping if needed).

You may need RNN if you want to combine some phonemes to recognize a single word or longer.


What is the difference between the state transition of an MDP and an action-value?

Transition Probabilities: Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s_{1}'$ or $s_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).

Action value function $Q_{\pi}(s, a)$: It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$.

  Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.


How do we choose the kernel size depending on the problem?

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.


Is There A Need For Stochastic Inputs To Mimic Real-World Biology And Environment?

Quite an interesting question, this sort of computational representation of biological systems are at the forefront of what we are trying to accomplish algorithmically.

This can be tackled via a variety of coordinate frames and problem formulations and could be thought of quite philosophically, as it pokes at some deep questions such as free-will and consciousness more generally.

For example, we could take the assumption that based on everything we know physically, the universe is completely deterministic. In this case, implementing deterministic functions for these variables makes quite a bit of sense and seems like the obvious choice. However, we can easily imagine that for at least some of this dynamic variables a stochastic representation would be both simpler(computationally) as well as giving better approximations to what we see in the real-world.

So it really is a question of how much value you see from implementing a system like this, and which representation you think would be most effective.


Can neural networks be better than human experts at prediction of greyhound racing results?

Practically, no.

In greyhound racing (or horse racing), there is no definite underlying pattern that can be associated with the outcome of the race. There are far too many variables to record and code as features. Most of which cannot be accessed by the public. This includes eating, sleeping, and training patterns. Furthermore, there are variables that cannot be readily quantified, such as the trainer's techniques, training effort, health history, and genes. A mere history of racing results and age won't be that helpful.

A neural network can only be as good as the features that are used to represent the instances. If the features don't capture the necessary characteristics of the instances that are associated with the problem, then the learner cannot generalize to predict the real world outcome.


What is a bad local minimum in machine learning?

The adjective bad isn't mathematically descriptive. A better term is sub-optimal, which implies the state of learning might appear optimal based on current information but the optimal solution from among all possibilities is not yet located.

Consider a graph representing a loss function, one of the names to measure disparity between the current learning state and the optimal. Some papers use the term error. In all learning and adaptive control cases, it is the quantification of disparity between current and optimal states. This is a 3D plot, so it only visualizes the loss as a function of two real number parameters. There can be thousands, but the two will suffice to visualize the meaning of local versus global minima. Some may recall this phenomenon from pre-calculus or analytic geometry.

If pure gradient descent is used with an initial value to the far left, the local minimum in the loss surface would be located first. Climbing the slope to test the loss value at the global minimum further to the right would not generally occur. The gradient will in most cases cause the next iteration in learning algorithms to travel down hill, thus the term gradient descent.

This is a simple case beyond just the visualization of two parameters, since there can be thousands of local minima in the loss surface.

There are many strategic approaches to improve the speed and reliability in the search for the global minimum loss. These are just a few.

Descent using gradient, or more exactly, the Jacobian
Possibly further terms of the multidimensional Taylor series, the next in line being related to curvature, the Hessian
Injection of noise, which is at an intuitive (and somewhat oversimplified) level is like shaking the graph so a ball representing the current learning state might bounce over a ridge or peak and land (by essentially trial and error) the global minimum — Simulated annealing is a materials science analogy and involves simulating the injection of Brownian (thermal) motion
Searches from more than one starting position
Parallelism, with or without an intelligent control strategy, to try multiple initial learning states and hyper-parameter settings
Models of the surface based on past learning or theoretically known principles so that the global minimum can be projected as trials are used to tune the parameters of the model

Interestingly, the only way to prove that the optimal state is found is to try every possibility by checking each one, which is not nearly feasible in most cases, or by relying on a model from which the global optimal may be determined. Most theoretical frameworks target a particular accuracy, reliability, speed, and minimal input information quantity as part of an AI project, so that no such exhaustive search or model perfection is required.

In practical terms, for example, an automated vehicle control system is adequate when the unit testing, alpha functional testing, and eventually beta functional testing all indicate a lower incidence of both injury and loss of property than when humans drive. It is a statistical quality assurance, as in the case of most service and manufacturing businesses.

The graph above was developed for another answer, which has additional information for those interested.

If digital values are mere estimates, why not return to analog for AI?


What are the current theories on the development of a conscious AI?

To answer this question, first we need to know why developing conscious AI is hard.
The main reason is that there is no mathematically or otherwise rigorous definition of consciousness.
Sure you have an idea of consciousness as you experience it and we can talk about philosophical zombies but it isn’t a tangible concept that can be broken down and worked on.
Moreover, the majority of current research in AI is primarily a pragmatic approach in that one is trying to construct a model that can perform well according to some desired cost function. This is a very very big and exciting field and encompasses many research problems and every new finding is based either on mathematical theory or empirical evidence of a new algorithm/model construction/etc.
Because of this, progress is based on and compared against previous progress as it is the scientific method.

So to answer your question, no one is trying to actually make a “conscious” AI because we don’t know what that word means yet, however that doesn’t stop people talking about it.


Would AlphaGo Zero become perfect with enough training time?

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do have some convergence guarantees, but there are some caveats:

Theories of convergence that apply to value and policy functions learned by RL assume unrealistic timescales and decays of learning parameters. If you have actual infinite time and resources then it is possible to explore all board states and learn their values accurately. But then, if you have those resources at your disposal, a brute-force search would work too.
Using neural network approximations to true value functions can put bounds on how well value functions are learned, as they rely on generalisation, and are characterised by an error metric. The values that they calculate are in fact guaranteed to not be mathematically perfect, and in part that is by design (because you want learning from similar states to apply to new unseen states, and as part of that need to accept the compromise that most state values will be slightly incorrect). This is especially true of the fast policy network used to drive Monte Carlo Tree Search (MCTS).
Running longer MCTS during play improves performance at cost of spending more time per move decision. Given infinite resources, MCTS can play perfectly, even from very crude heuristics.

The difference between AlphaGo Zero and a more traditional game search algorithm is to do with optimal use of available computing resources, as set by available hardware, training time and decision time when playing. It is orders of magnitude more effective to use the RL self-play approach combined with the focused MCTS as in AlphaGo Zero, than any basic search. At least for similar puzzles and games, it is more efficient than any other game playing technique that has been explored. We are likely still more orders of magnitude of effort away from a perfect Go player, and there is no reason yet to suspect that this will ever become practical, and Go become "solved".


What is the space complexity of breadth-first search?

The space complexity of the breadth-first search algorithm is $O(b^d$) in the worst case, and it corresponds to the largest possible number of nodes that may be stored in the frontier at once, where the frontier is the set of nodes (or states) that you are currently considering for expansion.
You can take a look at section 3.5 (page 74) of the book Artificial Intelligence: A Modern Approach (3rd edition, by Norvig and Russell) for more info about the time and space complexity of BFS.


Action recognition using video stream data

If you want to build a stream recognition pipeline than you can use OpenCV, Kafka, and Spark together.how to build pipeline of video stream

and for Action recognition using ConvLstm. This link will help you.action recognition using Convlstm


Research into social behavior in Prisoner's Dilemma

I've asked this on rl-list and got lots of interesting references to research papers.
So far the most promising one I've seen is this:
Nicolas Anastassacos, Stephen Hailes and Mirco Musolesi. Partner Selection for the Emergence of Cooperation in Multi-Agent Systems using Reinforcement Learning. In AAAI 2020. New York City, NY, USA. February 2020.


What is the relation between Monte Carlo and model-free algorithms?

In Reinforcement Learning (RL), the use of the term Monte Carlo has been slightly adjusted by convention to refer to only a few specific things.

The more general use of "Monte Carlo" is for simulation methods that use random numbers to sample - often as a replacement for an otherwise difficult analysis or exhaustive search.

In RL, Monte Carlo methods are generally taken to be non-bootstrapping sample-based approaches to estimating returns. This is a labelling convention within RL - probably because someone called an initial model-free learner a "Monte Carlo method", and the name stuck whilst many refinements and new ideas have since been published under different names.

The historical use of the term is important, since if you mention you are using "Monte Carlo Control", it usually means a very specific subset of methods within RL, to most readers.

  So, couldn't -learning also be considered a MC method?

In the general sense perhaps. The argument is perhaps stronger for it if the environment is being simulated on a computer for the agent to learn from.

However, if you start to unilaterally call Q-learning a MC method, you are probably going to just confuse people who have learned the conventions in RL.

  Can an MC method be model-based?

In general, yes, because the model can be sampled for planning, and a policy can also be separately sampled - so it is possible to run an MC method with or without a model - depending on whether you sample from the model taking "virtual actions" (for e.g. planning or refining your agent) or from the environment taking real actions. Many RL techniques blur the line between online learning and planning. For instance, using a simulated environment or historical data can be framed as planning for the real environment.

Monte Carlo Tree Search is an example of a model-based technique using the term "Monte Carlo" within RL frameworks. It is used famously within DeepMind's AlphaZero in order to refine a policy and value estimates during self-play.

  What is the relation between Monte Carlo and [other] model-free algorithms?

In the context of RL, Monte Carlo is presented as one way to estimate expected utility (or return) - by sampling from the environment and policy until the a complete trajectory is available:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{T-t} \gamma^k R_{t+1+k} | S_t = s ]$$

MC is contrasted with Temporal Difference (TD) approaches, such as Q-learning, which sample bootstrap estimates using the Bellman Equation:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1}) | S_t = s ]$$

The two approaches can be combined in various ways, including TD($\lambda$) methods. With  TD($\lambda$), if you set $\lambda = 0$ then the algorithm is identical to single-step TD learning, and if you set $\lambda = 1$ then it is very similar to a Monte Carlo method. Often setting it to some intermediate value is more efficient than either extreme.


What is the relation between Q-learning and policy gradients methods?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).

Both methods are theoretically driven by the Markov Decision Process construct, and as a result use similar notation and concepts. In addition, in simple solvable environments you should expect both methods to result in the same - or at least equivalent - optimal policies.

However, they are actually different internally. The most fundamental differences between the approaches is in how they approach action selection, both whilst learning, and as the output (the learned policy). In Q-learning, the goal is to learn a single deterministic action from a discrete set of actions by finding the maximum value. With policy gradients, and other direct policy searches, the goal is to learn a map from state to action, which can be stochastic, and works in continuous action spaces.

As a result, policy gradient methods can solve problems that value-based methods cannot:

Large and continuous action space. However, with value-based methods, this can still be approximated with discretisation - and this is not a bad choice, since the mapping function in policy gradient has to be some kind of approximator in practice.
Stochastic policies. A value-based method cannot solve an environment where the optimal policy is stochastic requiring specific probabilities, such as Scissor/Paper/Stone. That is because there are no trainable parameters in Q-learning that control probabilities of action, the problem formulation in TD learning assumes that a deterministic agent can be optimal.

However, value-based methods like Q-learning have some advantages too:

Simplicity. You can implement Q functions as simple discrete tables, and this gives some guarantees of convergence. There are no tabular versions of policy gradient, because you need a mapping function $p(a \mid s, \theta)$ which also must have a smooth gradient with respect to $\theta$.
Speed. TD learning methods that bootstrap are often much faster to learn a policy than methods which must purely sample from the environment in order to evaluate progress.

There are other reasons why you might care to use one or other approach:

You may want to know the predicted return whilst the process is running, to help other planning processes associated with the agent.
The state representation of the problem lends itself more easily to either a value function or a policy function. A value function may turn out to have very simple relationship to the state and the policy function very complex and hard to learn, or vice-versa.

Some state-of-the-art RL solvers actually use both approaches together, such as Actor-Critic. This combines strengths of value and policy gradient methods.


TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier

Advice from Neil, yes, output targeting class labels is still classification.

Output range is in contiguous range:

This is regression
For example: Linear activation, it has full numeric range of outputs.

Output targeting class labels is classification:

Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
Multiple output neurons (probabilities of classes) with sigmoid-like functions (softmax is mainly used). This will classify to 2 or more classes.

Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.

For example, this combination should work:

loss_fn   = tf.losses.LogCosh();
optimizer = tf.optimizers.RMSprop(1e-3);

From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):

                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
CategoricalHinge         --        --       --    --      --    --     --       --
CosineSimilarity         --        --       --    --      --    --     --       --
Hinge                    Yes       Yes      --    --      --    --     --       Yes
Huber                    Yes       Yes      --    --      --    --     --       Yes
LogCosh                  Yes       Yes      --    --      --    --     --       Yes
Poisson                  Yes       Yes      --    --      --    --     --       Yes
SquaredHinge             Yes       Yes      --    --      --    --     --       Yes

KLD: lambda a,b: KLD(a,b)
MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
SparseCategorialCrossentropy: Seems not working with single output neuron.


What is the relationship between MLE and naive Bayes?

And that's all, we can infer P(x|y=c) and P(c) from the data. I don't see where the MLE shows its role.

Maximum likelihood estimate is used for this very purpose, i.e. to estimate the conditional probability $p(x_j \mid y)$ and marginal probability $p(y)$ .

In Naive Bayes Algorithm ,using the properties of conditional probability, we can estimate the joint probability

$$
p(y, x_1, x_2, \dots, x_n) =  p(x_1, x_2, \dots, x_n \mid y) \; p(y)
$$

which is the same as
$$
p(y=c|x)\propto p(x|y=c)p(y=c)
$$

What we need to estimate in here, are the conditional $p(x_j \mid y)$ and marginal $p(y)$ probabilities, and we use maximum likelihood for this.

To be more precise: Assume that the $x$ is a feature vector $(x_1, x_2, ...x_n)$ - which means nothing more than the data point is a vector. In Naive Bayes Classification the assumption is that these features are independent of each other conditional on the class. So, in that case this term
$$p(x|y=c)$$ is replaced by $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- this is because of the independence assumption that lets us simply multiply the probabilities.

The problem of estimating the class to which the data point belongs reduces to the problem of maximizing the  likelihood $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- which means assigning the data $(x_1, x_2, ...x_n)$  to the class $k$ for which this likelihood is highest. This will give the same result as MLE which takes the form  of maximizing this quantity -- $$\prod_{i=1}^{n} p(x_i|\theta)$$ where $\theta$ are the assumed parameters.


How is creativity generated in a currently rule based neural network?

Your question is a bit broad. Some examples of creativity and how it's achieved (in all of them there is an ordinary convolutional network trained via supervised learning, but applied differently):

Google Deep Dream uses backpropagation to the image to tweak the image towards maximum activation in the convolutional layer. As a result, it finds lots of "things" in the image and enhances them:

Generative Adversarial Networks use two competing networks (generator and discriminator) to generate samples from the training manifold:

Neuro style method captures the style of one image via Gram matrix of convolutional layer activations and trains (another) neural network with a special loss function that minimizes the joint distance to the incoming image (content) and to the Gram matrix (style). The result:


If use the weights from previous iteration of a k-fold cross validation to seed a neural network classifier would I be overfitting?

To answer directly your question: estimating a model's performance on data previously used for fitting will overestimate performance.

When your dataset is "small" you are faced with a bias-variance dilemma with regards to how much data should be used for train and test sets:

Too much train data, you end up with little test samples and your
performance estimate has high variance.
Too much test data, your train samples do not represent well the
population you are trying to model/target, so your average
performance will be much lower than what could be achieved.

K-fold CV is a compromise in the evaluation of performance of certain procedure. Once you have settled on a certain model or hyperparameters, when moving to production you can:

Choose to take one of the k models you have trained, perhaps using the "one std variation rule of thumb".

Re-train on all of your data and you can expect the resulting model
to be at least as good as you have estimated.
Use all k models to form an ensemble and you can expect the resulting
model to be at least as good as you have estimated.


What reinforcement learning algorithm should I use in continuous states?

I would recommend looking at Deep Q-Learning.


Can artificial intelligence applications be hacked?

To answer your question, it really depends on the purpose of the Artificial Intelligence program.

For example, 4Chan has hacked a number of "Artificial Intelligent" bots, most notably was Microsoft's Twitter bot Tay. The general purpose of the bot was to parse what was tweeted at it and respond in kind, learning and evolving with each and every interaction.

Within 24 hours, 4Chan had corrupted Tay beyond repair, by teaching it racist and sexist terminology, ironic memes, sending it to shitpost tweets, and otherwise attempting to alter its output so much so that Microsoft had to remove it.

Now, the flaw with Tay was that it accepted any input, and learned off of that exclusively, without any interaction from the developers. Other bots have similar features, but they have checks in place that require human intervention to determine what is "quality" information to learn, and what is "bad" information to learn as to not pollute the global knowledge base of the bot.

These are just two examples of how Artificial Intelligence can be "hacked", but it ultimately comes down to how the programs are implemented.

You mention in one of your comments about Cellphone Artificial Intelligence such as Siri, and whether this technology can be hacked. The answer is - not really.

Siri learns based off of her global interactions - with limited user input allowed. You can ask Siri how to pronounce a name. When she pronounces it incorrectly, you can say to her "Siri that's now how you pronounce that". And she will provide you with a limited set of options of how you pronounce that name, and you have to choose which option sounds the best.

There was no option to allow a user to give Siri "bad" information, as she already populates the results for you, and you have to teach her based off of her list of options. To give Siri bad input, you would have to have access to Siri's global learning base, which we do not have access to, and alter how she accepts human interactions within the program - which would never happen due to too many moving parts within the iPhone update process, and would be caught before you were able to deploy your update.


One 'easy' way would be to have some sort of conversational memory, where you track what the user has said already. I don't know how complex your patterns are, but if you could recognise names and track references, you could try and build up a mental model of the user's relationships with other people, and perhaps refer to that in your bots responses.

The latter will be quite advanced, but keeping track of things said earlier and referring back to them on occasion might make it appear a lot more capable.

As an added bonus, track changes in the user's sentiment scores, and see if you spot a pattern in the conversation (maybe over the course of multiple conversations) to see which bot utterance have the biggest (positive or negative) effect on the user's mood.


Using NEAT, will the child of two parent genomes always have the same structure as the more fit parent?

Yes, if you follow the original implementation the children will inherit the topology from the most fit parent.

Keep in mind that the goal is to obtain a good population, maintaining the genetic diversity high but at the same time selecting the best individuals from the population; so, in theory you are allowed to give the topology you prefer to the children.

Here there is an example of an alternative topology inheritance in which a child gets the genes that lead to an excess node while another child gets only the genes that create a new connection.


Are there any public real-life code examples of ML applications in Python?

The best way is probably to Google it with "[org name] tensorflow github" and look what you get.

For instance I found:

Microsoft

Nvidia

Intel


What are some best practices when trying to design a reward function?

Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any other situation).
However, in certain cases, the specification of the reward function can be a difficult task [1, 2, 3] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?
So, often, designing a reward function is a trial-and-error and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance (for example, in terms of observable behavior, so not in terms of the collected reward; otherwise, this would be an easy problem: you could just design a reward function that gives infinite reward to the agent in all situations!). For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that penalizes the agent for every non-win move (maybe it will hurry up!).
Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.
Misspecification of rewards
It is well known that the misspecification of the reward function can have unintended and even dangerous consequences [5]. To overcome the misspecification of rewards or improve the reward functions, you have some options, such as

Learning from demonstrations (aka apprenticeship learning), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to

learn the policy directly (known as imitation learning [8]), or
learn a reward function first to later learn the policy (known as inverse reinforcement learning [1] or sometimes known as reward learning)

Incorporate human feedback [9] in the RL algorithms (in an interactive manner)

Transfer the information in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL [10])

Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.
Reward shaping
Regarding the common pitfalls, although reward shaping (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms, [4] states that reward shaping (and progress estimators) should be used cautiously.  If you want to perform reward shaping, you should probably be using potential-based reward shaping (which is guaranteed not to change the optimal policy).
Further reading
The MathWorks' article Define Reward Signals discusses continuous and discrete reward functions (this is also discussed in [4]), and addresses some of their advantages and disadvantages.
Last but not least, the 2nd edition of the RL bible contains a section (17.4 Designing Reward Signals) completely dedicated to this topic.
Another similar question was also asked here.


What you could do is just try and bypass the rest of the network after the LSTM if it isn't the 5'th minute.
Depending on your framework this can be easy or a painstakingly task compared to the alternative.
The alternative is just running and throwing away the output that isn't the next 5'th minute. While the last may seem inefficient it's rather easy to implement and takes just a bit more execution time. If execution time isn't a problem it's the easiest to get started with, if it doesn't work for your task you can always change it.


What's wrong with getting a dynamic RNN's output at the end of a padded sequence?

There is nothing wrong in theory. It is reasonable to expect that for some problems it will work worse than taking the immediate answer on the last known part of the sequence, some it will have roughly the same effect, and for others it might result in a better result.

A trained RNN is essentially a learned arbitrary program. If a computer program that processed arrays would get a better result by running more iterations at the end of an array, then so might a RNN. One example that comes to mind is predicting the end point of a trajectory of an observed object, when you have a variable number of observations at the start. A program that simulated each timestep in a simple fashion could well perform better than one that attempted to analyse and project directly from whatever the last observed state was.

On the other hand, you may face some practical problems:

One practical detail of not padding sequences is that it may reduce computation time since the network is run less often.
A padded sequence will only get a gradient to learn from with the last output, after the padding. This could make learning harder than it needs to be with vanishing or exploding gradients (these are more of a problem, the more time steps you need take back through the network in order to assign correct weights due to influence of first input).

The usual way to find out is to try it in an experiment. See which approach does better on a hold-out test set when training with or without padding. Bear in mind that whatever the answer it will not hold in general, but is likely to hold for problems similar to the one you tested with.


Why are tree-based models more widely used in Medical Diagnosis?

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article.  It presents a case study of a hospital whose policy was to send asthma sufferers to an intensive care unit; the intensive care meant they were less likely to develop pneumonia and therefore the data showed that people with asthma were less likely to have pneumonia.

Essentially, since machine learning models learn false relationships if the data are in any way flawed, it is beneficial to be able to "debug" them.  The processes by which decision trees make their decisions, and the reasons for making them, are more readily visible than in other models - particularly neural networks - which makes errors such as the example given in the article more likely to be picked up and corrected.


Are Graph Neural Networks generalizations of Convolutional Neural Networks?

Excuse my lack of rigor. Although I believe this could be rigorously proven for certain definitions of GNN, the term is still too loose for me to honestly claim one way or another on this. Hopefully the following thoughts will be helpful anyway.
I prefer the term Message Passing Networks as a generalization of many things people like to call GNN. In a generic Message Passing Network, every node has an associated vector.
Each node's vector, $x_i$, is then updated as a function of itself and its neighbors. (I've left off time indices for readability...)
For instance:
$$ x_i := Update(x_i, \sum_{x_j\in N(x_i)} Message(x_i,x_j, e_{ij})) $$
where $N(x_i)$ is the set of vectors associated with nodes neighboring $x_i$ and $Update$ and $Message$ are arbitrary parameterized functions.
In the case where $Message(x_i, x_j, e_{ij}) = e_{ij} x_j$ and $e_{ij}$ is a scalar, and $Update(a,b) = b$, this becomes the following:
$$ x_{i} := \sum_{x_j\in N(x_i)} e_{ij} x_j $$
This means that each node becomes a weighted sum of its neighbors and this is precisely convolution (given that you have a stride of 1 ... and that you have padding ... and that the graph is a grid where all upward edges are equal, all leftward edges are equal, all rightward edges are equal and all downward edges are equal.)


Can we combine Off-Policy with On-Policy Algorithms?

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017).
Citing the paper:

  The core idea is to use the first-order Taylor expansion of the critic
  as a control variate, resulting in an analytical gradient term through
  the critic and a Monte Carlo policy gradient term consisting of the
  residuals in advantage approximations. The method helps unify policy
  gradient and actor-critic methods: it can be seen as using the
  off-policy critic to reduce variance in policy gradient or using
  on-policy Monte Carlo returns to correct for bias in the critic
  gradient.

The authors provide an open source implementation of it in https://github.com/shaneshixiang/rllabplusplus

There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NIPS 2017).

The Related Work section in both papers might be also worth looking at.


How close are we to creating Ex Machina?

We are absolutely nowhere near, nor do we have any idea how to bridge the gap between what we can currently do and what is depicted in these films.

The current trend for DL approaches (coupled with the emergence of data science as a mainstream discipline) has led to a lot of popular interest in AI.

However, researchers and practitioners would do well to learn the lessons of the 'AI Winter' and not engage in hubris or read too much into current successes.

For example:

Success in transfer learning is very limited.
The 'hard problem' (i.e. presenting the 'raw, unwashed environment' to the machine and having it come up with a solution from scratch) is not being
addressed by DL to the extent that it is popularly portrayed: expert human knowledge is still required to help decide how the input should be framed, tune parameters, interpret output etc.

Someone who has enthusiasm for AGI would hopefully agree that the 'hard problem' is actually the only one that matters. Some years ago, a famous cognitive scientist said "We have yet to successfully represent even a single concept on a computer".

In my opinion, recent research trends have done little to change this.

All of this perhaps sounds pessimistic - it's not intended to. None of us want another AI Winter, so we should challenge (and be honest about) the limits of our current techniques rather than mythologizing them.


Can a neural network be used to predict the next pseudo random number?

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random.

When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course be easily predicted by a neural network with little training. But in the real world things look different.

The neural network could be trained to find certain patterns in the history of random numbers generated by a PRNG to predict the next bit. The stronger the PRNG gets, the more input neurons are required, assuming you are using one neuron for each bit of prior randomness generated by the PRNG. The less predictable the PRNG gets, the more data will be required to find some kind of pattern. For strong PRNGs this is not feasable.

On a positive note, it is helpful that you can generate an arbitrary amount of training patterns for the neural network, assuming that you have control over the PRNG and can produce as many random numbers as you want.

Because modern PRNGs are a key component for cryptography, extensive research has been conducted to verify that they are "random enough" to withstand such prediction attacks. Therefore I am pretty sure that it is not possible with currently available computational resources to build a neural network to successfully attack a PRNG that's considered secure for cryptography.

It is also worth noting that it is not necessary to exactly predict the output of a PRNG to break cryptography - it might be enough to predict the next bit with a certainty of a little more than 50% to weaken an implementation significantly. So if you are able to build a neural network that predicts the next bit of a PRNG (considered secure for cryptography) with a 55% success rate, you'll probably make the security news headlines for quite a while.


Are there known error bounds for TD(0) with a constant learning rate?

The paper "Bias-Variance" Error Bounds for Temporal Difference Updates (2000) by M. Kearns and S. Singh provides error bounds for temporal-difference algorithms, i.e. TD($k$) and TD($\lambda$) (see theorem 1 and theorem 2, respectively). Note that both TD($k$) and TD($\lambda$) include TD($0$) as a special case.


How are LSTM's trained for text generation?

You compute the error in the same way as normal - treat the actual output as "ground truth" even if other options are possible. The LSTM will output probabilities of next character, it will not learn to associate a single "true" output except for rare circumstances (such as completing a word that can only be done in one way according to the training data). In practice that means that understanding acceptable loss values in NLP training is different to understanding them in other supervised learning environments, typically being higher than for datasets where ground truth is more predictable from the data.

In this way, it is a lot like a Markov or n-gram model, except that it learns its own internal state representations (they are not prescribed by the model, but are limited by structure and data available). The gated RNN memory models like LSTM and GRU include a mechanism for handling long term memory so can relatively easily learn things like quote, paren pairs etc.

  What I'm getting at is that I don't get how it could LEARN any sort of structure more than a Markov model.

A Markov model can in theory represent any sequence knowledge that an LSTM can. The difference is in terms of efficiency. RNNs have more compact internal representations than explicit state representation in say HMMs. This makes them harder to work with in some ways (it is hard to see what a RNN state vector represents), but means that they can store more complex knowledge about state transitions in less memory. It also means that RNNs can generalise (although not guaranteed correctly) more readily, they will not fall back to random behaviour given a brand new state without any training history.


How can artificial intelligence (including deep learning algorithms) find suspicious patterns in the body’s biochemistry?

I wouldn't focus only on "deep learning" unless you have some specific reason for doing so.  There may be other techniques which could be as effective, or more effective.

One approach I've seen used for something similar was Inductive Logic Programming.  For one example of using ILP to reason about elements of biochemistry, see this paper

That's not exactly about detecting disease, but it does sort of illustrate the broad idea of reasoning about states and reactions involving metabolic pathways in biochemistry, using ILP.   Possibly the basic idea could be adapted more towards detecting disease.


What is the name of an AI whose primary goal is to create a better AI?

I don't think there is a single standard word or phrase that covers just this concept. Perhaps recursive self-improvement matches the idea concisely - but that is not specific AI jargon.

Very little is understood about what strength this effect can have or what the limits are. Will 10 generations of self-improvement lead to a machine that is 10% better, 10 times better, or $2^{10}$ times better? And by what measure?

Some futurologists suggest this might be a very strong effect, and use the term Singularity to capture the idea that intelligence growth through recursive self-improvement will be strong, exceed human intelligence, and lead to some form of super-intelligent machine - the point at which this goal is reached is called The Singularity. Ray Kurzweil is a well-known proponent of this idea.

Specifically, use of the term Singularity implies more than just the basic recursion that you suggest, and includes assumptions of a very large effect. Plus technically, it refers to a stage that results from the recursion, not the recursion itself.

However, despite the popularity of it as a concept, whether or not such self-improving system will have a large impact on the generation of intelligent machines is completely unknown at this stage. Related research about general intelligence is still in its infancy, so it is not even clear what would count as being the first example system x.


Is there actually a lack of fundamental theory on deep learning?

There is a paper called Why does Deep Learning work so well?.

"However, it is still not fully understood why deep learning works so well. In contrast to GOFAI (“good old-fashioned AI”) algorithms that are hand-crafted and fully understood analytically, many algorithms using artificial neural networks are understood only at a heuristic level, where we empirically know that certain training protocols employing large data sets will result in excellent performance. This is reminiscent of the situation with human brains: we know that if we train a child according to a certain curriculum, she will learn certain skills — but we lack a deep understanding of how her brain accomplishes this."


Q-learning in gridworld with random board

From your linked description of the game, we can see it has a key property when used normally in AI teaching:

Partially observable: The Wumpus world is partially observable because the agent can only perceive the close environment such as an adjacent room.

This makes sense, the problem of avoiding hazards would be trivial if the full map was revealed to the agent. The problem is also not about learning a specific map.
You could use the agent's perception to construct a simple state table based on current observations:

Current location
Current facing
Boolean flags for stench, breeze, glitter, bump, scream

This would be relatively easy to build into a Q table, and might have some success. However, it would likely perform a lot worse than the propositional logic and planning suggested suggested at https://www.javatpoint.com/the-wumpus-world-in-artificial-intelligence for two reasons:

Agent will not take account of knowledge specific to the current map that it has gathered. This is critical, because the stench and breeze sensors only tell you that at least one of the adjacent rooms has a hazard. In theory you know that this will not be any of the rooms that the agent has already visited, but a simple state representation based on current observations will not capture this.

Agent will not plan using the deterministic rules of the game that you know and could code for.

There are a few different approaches you could take to improve on these issues. Sticking with Q learning and trying to solve this by improving the state representation, you could look at the suggested knowledge-building structure from the java T point site, and replicate something like it as input features:

Current facing
Whether or not the last move forward resulted in a bump
Whether or not there has been a scream
For each allowed room, a binary flag set to true if the room has been visited by the agent
For each allowed room, a set of binary variables covering the three observations that are about the room: stench, breeze, glitter
Either the current location separately, or another binary variable added to each grid square set to true for the one room that the agent is in

When multiplied by 16 to cover each room, you will end up with ~84 binary flags. Although many combinations will not be possible, this is still going to be far too large a state space to use with a Q table. You would probably use a simple neural network and DQN agent to solve this problem using Q learning.


Can we teach an artificial intelligence through sentences?

I think that what you're really asking about is the question of knowledge representation.  Regardless of how you train your AI, one of the most fundamental questions is how do you represent "knowledge" and especially when it exists at different levels of abstraction, may be mutually recursive, etc.  Along with that goes the question of belief revision which deals with how you update existing beliefs/knowledge in the light of new information.

Both of these areas are still subject to plenty of active research and neither has entirely settled answers to the core questions.  But progress has been made in both areas.

Personally I suspect that something like semantic networks or conceptual graphs will be the best answer to the KR problem.  Dealing with belief revision seems even fuzzier to me, although there are known strategies (like the AGM postulates) that work to a point. Something like Bayesian Belief Networks may also prove useful.


Why not replacing the simple linear functions that neurons compute with more complex functions?

It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a computational graph and the standardised structures used in most neural networks are a subset of all possible (and maybe useful) computational graphs.

When adding complex and non-linear functions into a neural network, this is usually alternated with simpler linear layers using the weights. A generalised function of a single neuron as used in most neural networks looks like this:

$$a = f(\sum_i w_i x_i + b)$$

Where $i$ indexes all inputs to the neuron, $x_i$ are the input values, $w_i$ the weights associated with each input, $b$ is a bias term and $f()$ is a differentiable non-linear activation function. Training process learns $w_i$ and $b$. The output $a$ is the the neuron's activation value, that may be taken as output of the neural network, or fed in to some other neuron as one of the next neuron's $x_i$.

A simple feed-forward network using this basic neuron function, with at least one hidden layer which has a non-linear activation function can already learn approximations to any given function - a result proved in the universal approximation theorem.

The practical result of the universal approximation theorem is that you need motivation other than increasing diversity in order to make neural network functions more complex. If you were considering altering one of the $w_i x_i$ multiplications, and replacing with a more complex learnable function, you can effectively achieve the same thing by adding another neuron whose output $a$ is used as $x_i$ - or simply adding a layer in most neural network libraries.

In some situations there may be good reasons to make lower-level changes:

If you know the function you are learning relates to a theoretical model with a specific mathematical form, you can deliberately set up functions that mirror that with learnable parameters. Typcially that is done as transforms on inputs, but could also be part of a more complex computational graph if necessary.
In neural network architectures, you can consider things such as gate combinations in LSTM cells, or skip connections in residual networks as examples where the functions have deliberately be made more complex to achieve a specific goal - in both those cases in order to increase effectiveness of backpropagation in deep structures.


Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

Is the method itself defective or anything wrong with my code?

There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust their writing too :) ).

The first issue I see, and likely the most important, is that the update() calls of DynamicPBA frequently update the contents of self.q_table to those of self.q_table_, but the contents of self.q_table_ are never updated. So, your q_table is essentially always filled with a bunch of $0$ values that never get a chance to properly learn.

I did not check the rest of the implementation in detail, but at a glance it all looks fine to me. So I guess that changing the last line of learn() to update an entry in self.q_table_ rather than self.q_table should go a long way towards fixing your issue.


How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?

You will want to look into Contextual Multi-Armed Bandits. These are MAB problems that additionally involve feature vectors in some way.

You'll sometimes see researchers considering problems where you get to see a single feature vector per timestep (like an "environment state" you're in) which may provide useful information. You'll also sometimes see researchers considering problems where you observe a single feature vector per arm/action (per timestep). That second case is pretty much what you're describing; every arm/action has a feature vector (model, year, mileage of car), and you can use those to make predictions and generalize across arms.

To give a little bit of flavour for these algorithms, the most common ones simply assume that the rewards you observe after pulling arms are given by a linear combination of features, plus some noise. Then, they simply try to find (through online learning) a parameter vector such that observed rewards can be accurately approximated by the dot product of the parameter vector and a feature vector. Of course, there are lots of different variants of algorithms, and not all are linear, but this gives an idea of what they generally look like.


How can I remove the noise from an EEG signal?

This might be more of a signal-processing question, rather than a artificial intelligence question, but I will try my best to be of help.

Do you know what the noise you are trying to remove is? How it behaves/where it stems from?
Or, do you know how your output signal should look, post processing?

If you know these things and you are familiar with MATLAB or any other matrix multiplication software, they come with great prebuilt toolboxes for traditional approaches to remove noise from signals.

If you are not exactly sure what patterns you are looking for, I suggest perhaps looking into Autoencoders to discover the hidden patterns. Though it is important to note that the origin of the noise may greatly effect its abilities.
If you plan on using such a technique it is important that you have a sufficiently large dataset of the signals available.

Without the clarifications to these questions, along with @nbro's questions, it is hard to be more specific.


Adding Class Weights for imbalanced dataset?

The paper A systematic study of the class imbalance problem
in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning":

Cost sensitive learning. This method assigns different cost to misclassification of examples from different classes [44]. With respect to neural networks it can be implemented in various ways. One approach is threshold moving [22] or post scaling [23] that is applied in the inference phase after the classifier is already trained. Similar strategy is to adapt the output of the network and also use it in the backward pass of backpropagation algorithm [45]. Another adaptation of neural network to be cost sensitive is to modify the learning rate such that higher cost examples contribute more to the update of weights. And finally we can train the network by minimizing the misclassification cost instead of standard loss function

Without further context, "Adding Class Weights for an imbalanced dataset" can mean many things as enumerated by the above. But if I had to guess, the most common meaning is that they weigh the misclassification cost differently per label by multiplying it with a different weight variable.
For example, maybe label 1, 2, and 3, when misclassified, gets a 1x multiplier in the training loss (the standard weight), but label 4 gets a 3x multiplier because it is roughly 3x more important, and label 5 gets a 0.5x multiplier because it is 0.5x less important.
Here is an example of how to do this with Keras.
(FYI the paper above recommends you over-sample the minority class rather than using the cost-sensitive learning approach to help with class imbalance)
As for missing labels, the quickest solution is just to skip those training instances when training for that particular missing label. If you want to still use the training instances somehow, a common approach is to fix the missing label with semi-supervised learning approaches.
One way to do semi-supervised learning is self-training, where your model is trained on only the labeled instances, and then makes predictions on the missing labels. High-confidence predictions of the missing labels are then added to the training data, and the model is trained on the new training data. This process repeats until convergence of the training data.


Why can neural networks generalize at all?

A neural network is composed of continuous functions. Neural networks are regularized by adding an l2 penalty on the weights to the loss function. This means the neural network will try to make the weights as small as possible. The weights are also initiallized with a N(0, 1) distribution so the initial weights will also tend to be small. All of this means that neural networks will compute a continuous function that is as smooth as possible while still fitting the data. By smooth I mean that similar inputs will tend to have similar outputs when run through the neural network. More formally, $||x-y||$ small implies $||f(x)-f(y)||$ small where f represents the output from the neural network. This mean that if a neural network sees an novel input $x$ that is close to an input from the training data $y$, then $f(x)$ will tend to be close to $f(y)$. So the end result is that the neural network will classify $x$ based on what the labels for the nearby training examples were. So the neural network is actually a little like k-nearest neighbors in that way.

Another way for neural networks to generalize is using invariance. For example, convolutional neural networks are approximately translation invariant. So this means that if it sees and image where the object in question has been translated then it will still recognize the object.

But it's not giving us the exact function we want. The loss function is a combination of classification accuracy and making the weights small so that you can fit the data with a function that is as smooth as possible. This tends to generalize well for the reasons I said before but it's just an approximation. You can solve the problem more exactly using minimal assumptions with a Gaussian process but Guassian processes are too slow to handle large amounts of data.


What if we took a recursive approach and built a smallest possible robot?

Lost Article and Found Articles

MIT Review had an article on nanotechnology for disease eradication, DNA repair, and microsurgery in the 1990s that's probably somewhere among the thousands of entries resulting from a web search of, "MIT Review nanotechnology cell repair," or the few hundred resulting from an academic article search for the same.1 The article I can't find described a recursive nano-robot scheme like the one described in this question.

What seems plausible given the current state of technology is to find a recursive algorithm that will command a 3D printer to make a smaller 3D printer that can print a still smaller one.

Extending the Normal Meaning of Recursion

The algorithm will have to take a step in capability beyond mere recursion.  It doesn't call itself.  It must load itself into the machine it printed and then boot the copy of itself there.  As it loads itself, at each level in size, it must parametrize its child for that geometrically reduced size for each progressive reduction.  It must stop when the desired size is achieved.

Such a paradigm could be called, "Printail Recursion," from the synthesis of, "Printer," and the algorithmic principle of, "tail recursion," from the dawn of LISP.

Applying the Decorator Pattern

Once Printail Recursion works, other robotic or algorithmic features could piggy back on the appropriate components of the progressive micro-children.

Notes

[1] The later of those two may even provide some departmental contacts to open options for partnering between academic institutions, something usually compelling for research oriented students and a great catalyst for scientific collaboration among the next generation of researchers.


What does the Fourier transformed image mean?

Have a look at this explanation of the two dimensional Fourier Transform applied to an image.

The way I read it is that it somewhat mirrors the diagonal structure of the 'X', as that is where most of the 'image energy' is, so that are the lines you highlighted. Because the image is fairly uniform in colour, most of the components are low-frequency (ie close to the centre of the 2DFT image).

Disclaimer: I'm not really that familiar with a 2D version either, as I mainly have experience with using FFTs in speech processing.


What is the best way to smoothen out a loss curve plot

You can use the Exponential Moving Average method. This method is used in tensorbaord as a way to smoothen a loss curve plot.

The algorithm is as follow:

However there is a small problem doing it this way. As you can see S_t is initialized with the starting value, which makes the starting curve inaccurate.

The green curve is the ideal curve for the algorithm, but the purple curve is the predicted curve. The curve is not correct on the start. To solve this, a correction factor is added in, thus making the algorithm this:

This introduces WeightedCount which decreases over time to 0.

Exponential Moving Average is also used is other areas of deep learning, the most notable being some optimization algorithms. It is used in Adam, RMSProp and other similar optimizers to smooth out the gradients to make the path to minimal loss a more direct and straightforward path.


What are the differences between a knowledge base and a knowledge graph?

Based on the related Wikipedia, a knowledge base (KB) is:

  a technology used to store complex structured and unstructured information used by a computer system. The initial use of the term was in connection with expert systems which were the first knowledge-based systems.

As there are different representation model for a KB, we can find different terminology in different domains. For example, in some AI articles, it's called ontology.

Knowledeg graph (KG) is another object model to KB realization which is introduced by Google for its search engine (as you have mentioned). Hence, KG is a specification of KB. You can find more information in the paper Knowledge Graphs, such as more history about the KG or a formal definition of that:

  knowledge graph is a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent relations between these entities.

Moreover, you can find some articles about contextual KG (CKG) in the paper Learning Contextual Embeddings for Knowledge Graph Completion and KG$^2$: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings.


You need a bidirectional LSTM. The temporal direction on the original LSTM and on GRU type cell connectivity is forward in time. Fortunately, the bidirectional class is available in Keras.

from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import TimeDistributed
from keras.layers import Bidirectional
...
model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(Bidirectional(LSTM(64)))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

Now the network has causality from both history and forecast. This is only useful if the forecasts reduce entropy, which means they have to be known effective forecasts that when fed to earlier cells (in the time domain) will enhance accuracy and confidence metrics.

When the loss function (for its derivative) is provided, it must return the error at $t = t(i)$, where $i$ is the index of the last sample in the sequence before the values become forecasts. This is the main distinction when the sequence includes both history and forecast.


How can I test my trained network on the next unavailable hour?

You need to have access to the 696th hour (or successive hours), otherwise, you cannot test your model. An alternative would be, for example, to train your model on the first 693 hours, validate it on the 694th hour, and test it on the 695th hour.


How to generate the original image from feature set?

The model (that I know of) which most resembles your description is the auto-encoder, which is trained to learn a compact representation (a vector) of the input, which can later be used to reconstruct the original input. In a certain way, this compact representation (implicitly) encodes the most important features of the input. In particular, you may be looking for denoising auto-encoders.


This sounds a little bit like a Blackboard Architecture approach.  One of the biggest challenges in these is figuring out how to handle coordination between the various "experts" (ie, algorithms).  Some people talk about an "executive" which manages the coordination, and that leaves you needing to figure out how to train the "executive".  Alternatively, you may be able to do this without a master "executive", but nobody knows exactly what the best way to do this is.

You may also find some inspiration / useful ideas in the realm of Multi-agent Systems if you model each component of your systems as a discrete "agent".