Can the (sparse) categorical cross-entropy be greater than one?

Both the sparse categorical cross-entropy (SCE) and the categorical cross-entropy (CCE) can be greater than $1$. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors.
Here is the explanation with examples.
Let $(x, y) \in D$ be an input-output pair from the labelled dataset $D$, where $x$ is the input and $y$ is the ground-truth class/label for $x$, which is an integer between $0$ and $C-1$. Let's suppose that your neural network $f$ produces a probability vector $f(x) = \hat{y} \in [0, 1]^C$ (e.g. with a softmax), where $\hat{y}_i \in [0, 1]$ is the $i$th element of $\hat{y}$.
The formula for SCE is (which is consistent with the TensorFlow implementation of this loss function)
$$
\text{SCE}(y, \hat{y}) = - \ln (\hat{y}_{y}) \label{1}\tag{1},
$$
where $\hat{y}_{y}$ is the $y$th element of the output probability vector $\hat{y}$ that corresponds to the probability that $x$ belongs to class $y$, according to $f$.
Actually, the equation \ref{1} is also the definition of the CCE with one-hot vectors as targets (which behave as indicator functions). The only difference between CCE and SSE is really just the representation of the targets, which can slightly change the implementation under the hood. Moreover, note that this is the definition of the CE for only $1$ training pair. If you have multiple pairs, you have to compute the CE for all pairs, then average these CEs (for a reference, see equation 4.108, section 4.3.4 Multiclass logistic regression of Bishop's book PRML).
Let's have a look at a concrete example with concrete numbers. Let $C=5$, $y = 3$, $\hat{y} = [0.2, 0.2, 0.1, 0.4, 0.1]$, then the SCE is
\begin{align}
\text{SCE}(y, \hat{y})
&=
- \ln (0.4) \approx 0.92,
\end{align}
If $\hat{y} = [0.2, 0.2, 0.2, 0.1, 0.3]$, so $\hat{y}_{y} = 0.1$, and we still have $y = 3$, then the CCE is $2.3 > 1$.
You can execute this Python code to check yourself.
import numpy as np
import tensorflow as tf # Install TensorFlow 2.3!

y = 1
y_true = [3] # sparse label (integer)
y_true2 = [0, 0, 0, 1, 0] # one-hot vector

for y_y in [0.4, 0.1]:
    sce_np = -(y * np.log(y_y))
    print("SCE (NumPy) =", sce_np)

y_preds = [[0.2, 0.2, 0.1, 0.4, 0.1],
           [0.2, 0.2, 0.2, 0.1, 0.3]]
for y_pred in y_preds:
    sce_tf = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
    cce_tf = tf.keras.losses.categorical_crossentropy(y_true2, y_pred)

    print("SCE (TensorFlow) =", sce_tf)
    print("CCE (TensorFlow) =", cce_tf)

To answer the following question more directly.

However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?

Yes. It can happen, as explained above. (However, this does not mean that you do not have mistakes in your code.)


How can I constraint the actions with dependent coordinates?

At first glance, I thought this was similar to "continuous-discrete" action selection (https://arxiv.org/pdf/1810.06394.pdf). However, I think your problem is different.

I am assuming that each $a_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a_1,a_2,\dotso,a_n)$ and not an individual $a_i$. Then you could treat it like a hierarchical problem. If you want $a_1 + a_2 < 2$ for example, then you could sample $a_1 \sim U(0,2)$ and $a_2 | a_1 \sim U(0, 2-a_1)$ and have $p(a) = p(a_2 | a_1)p(a_1)$. The specifics of how you do this depends more finely on how your problem is set up.

Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.


How can people contribute to AGI research?

OpenCog is an open source AGI project. But it is is also incredibly complex and IMHO not a good idea (I have not fully read his theories). You can learn the essential ideas behind OpenCog from the co-founder Ben Goertzel site as well.

Or, you can participate in the philosophical discussion regarding AGI. For strictly AGI, decision theory, logic, and math material (they are all related), you can look up stuff from http://yudkowsky.net/ or https://arbital.com/. But, in some sense, every branch of philosophical inquiry can be tied back to AGI and consciousness (ethics, metaphysics, etc.), so if it fancies you it depends on how you'd like to tackle it.

You could also study the psychology end of things. The following papers and related ideas are quite important in the field of study of consciousness and cognition (but keep in mind this is pretty much a random list, the literature is massive!):

Global Workspace: A Theory of Consciousness (1997)
Relevance Realization and the Emerging Framework in Cognitive Science (2009)
The free-energy principle: a unified brain theory? (2010)

Recently, (in mathematical time), progress in category theory shows promise of being a unified framework of much of the existing math. I know next to nothing about it, but the people that do are applying to many new fields of study (including AI, apparently). Category theory requires a lot of background mathematical knowledge before its vocabulary begins to make sense, though, so beware. You can read about it on the nCatLab and occasionally on John Baez's blog: Azimuth

Of course, "regular" techniques in Machine Learning such as neural networks, reinforcement learning, statistical methods, and others are very powerful as well, but due to certain regards in their construction, they are generally understood as only being capable of being "narrow AI" in the sense that they can only complete a single task very well, but perhaps you can find some research that changes this?


Is batch normalization not suitable for non-gaussian input?

So batch-normalization helps descent based learning have an easier time traversing the loss manifold, but in your case you use it along with a relu as a final activation is problematic, it means the output is relatively associated with the other samples in the batch.

Remove that last BN and you get better results, but also understand BN is inherently problematic for this task. Think of DNNs as featurizers, and BN in this case takes out the 2 batch-wide statistics which if they don't align to the initial distribution will cause error, which will lead to error in the output. In theory if BN gets the perfect statistics of the gaussian it should not matter too much, so one thing I tried with your code was remove last BN and increase N to 100,000 while increasing the batch size to 10000 and you see a huge boost in performance.


How would you intuitively but rigorously explain what the VC dimension is?

Shattered set. First we need a concept of a shattered set. I'll work from a shattered set example in Wikipedia adjusting it to your notation.
The statement that $\mathcal{H}$ shatters $C$ means that for every subset $A \subset C$ there is a set $B\in\mathcal{H}$ such that $B$ "separates" $A$ from $C \backslash A$. Writing this formally:
$$\text{shatters}(\mathcal{H},C) = \forall A \subset C\; \exists B\in\mathcal{H}\;.\;A = B\,\cap A $$
As an example consider the set $C$ of four points on $\mathcal{X} = \mathbb{R}^2$:
$$C = \left\{(0,0); (0,1); (1,0); (1,1)\right\}$$
And the classification class $\mathcal{H}$ being all possible 2D discs on $\mathcal{X} =\mathbb{R}^2$. (Notice that people use the world "class" here, because we are dealing with "set-of-sets" stuff and that might get tricky). Note that a disk can be represented as the set of all points inside the circle.
Now, it turns out that this $\mathcal{H}$ doesn't shatter $C$. The counterexample would be the subset $A$ of "diagonal" points:
$$A = \left\{(0,0); (1,1)\right\} \subset C$$
There is no 2D disk $B\in\mathcal{H}$ (in the context of learning theory, an element/set of the hypothesis class is a hypothesis, which can also be viewed as a function) that satisfies $A = B\,\cap A$.  Intuitively, this means that you cannot use a 2D disk to classify your pair of points $A$ from the rest of the set $C$.
The VC dimension of $\mathcal{H}$ is the maximal cardinality $d$ of the set $C$ that it can shatter. For $d=3$, we can provide tree points $C'$ for which we can easily find all discs that cover all eight possible subsets of $C'$:
$$C' = \left\{(0,0); (0,1); (1,0)\right\}$$
Above we've shown that for a particular set $C$ of $d=4$ points $\text{shatters}(\mathcal{H}, C)$ is false. But we need to prove that it is false for all 4-point sets. To prove this, we  consider a convex hull of an arbitrary set of four points $C = \left\{a;b;c;d\right\}$. In general position, the convex hull is either a triangle or a quad:

In the case of a triangle, we choose the outermost three points as a counterexample set $A$. So, with this (labeling) configuration, you cannot find a disk that covers (i.e. classifies) the outermost three points correctly while excluding the point inside the triangle. In the case of a quad, we choose the pair on the longest diagonal. If any three points lay on a single line, then we choose the pair of outermost points.
This sketches a proof that no $d=4$ point set can be shattered by $\mathcal{H}$, but we've shown that there is a $d=3$ set $C'$ that can be. Concluding that $VCdim(\mathcal{H}) = 3$
Another example is considered on the next page (pg. 71) of the book you've referenced. It again considers 2D plane  $\mathcal{X} =\mathbb{R}^2$ and the classification class $\mathcal{H}$ is all possible axis-aligned rectangles. Authors show a configuration $C$ of four points that can be shattered by $\mathcal{H}$ on the left of figure 6.1. And then provide a proof that no $d=5$ points can be shattered by axis-aligned rectangles. Concluding that VC dimension of their  $\mathcal{H}$ is four.
Hope these examples help. (BTW, note that, it seems that Deep Learning has quite bad VC dimension but it still works somehow - which is rather puzzling).


In adversarial machine learning, how does an attacker have access to the test and training dataset in order to poison it?

They don't have acces to the original training or test dataset. Machine learning environments are build on the premise of a benign environment. The models are trained on real data (real inputs). When someone sends a made up input (fake input) it is very easy to fool the model.

This is used for example in image recognition. Imagine a fotograph of a panda. the model may correctly identify this fotograph as a panda. With knowledge of the model you can now alter some pixels in the fotograph. To the human Eye, the fotographs will appear exactly the same, but the model can be fooled to believe the fotograph is actually of a gibbon.

This is all done after the training of the model and doesn't require the original datasets.

For more info, visit this site:
https://medium.com/@ml.at.berkeley/tricking-neural-networks-create-your-own-adversarial-examples-a61eb7620fd8


What is a good way of identifying volatile positions for a checkers game?

When combatting the horizon affect, you want to consider any short term actions that will greatly affect your position evaluation. Thus, in addition to captures, you will also want to include:

When the opponent can make a king next move
When the current player can make a king next move
When the only legal moves left will lead to capture the turn after for the opponent
When the only legal moves left will lead to capture the turn after for the current player

3 and 4 are commonly called Zugzwang and play a very prominent role in high level checkers, but may be a bit more difficult to implement. However, they will contribute perhaps most of all when combatting the horizon affect.


Why do we need 10 bits to represent the 1000 classes in AlexNet?

You need 10-bits ($2^{10} = 1024$) to represent 1000 classes.


Why use the output of the generator to train the discriminator in a GAN?

The main reason that the discriminator is trained concurrently with the generator is to provide (at least in theory) a smooth and gradual learning signal for the generator.

If we trained the discriminator on only the input data, then, assuming our training algorithm converges well, it should quickly converge to a fixed model. The generator can then learn to fool this fixed model, but it will likely still be easy to spot the generator's fakes for a human. For example, the discriminator may have learned that, coincidentally, in the sample you provided, all images of trucks have a fully white pixel in the top left corner. If it learns that pattern, the generator can fool it by generating noise with a white pixel. Once the generator has learned this pattern, all learning stops.

In contrast, suppose that the discriminator is repeatedly re-trained on a mixture of real and generated examples. The discriminator will be forced to learn more complex patterns than "white pixel in upper left", which improves its quality beyond the raw patterns in the sample data.

The converse relationship is also true. If the generator is trained only on the training data, it will also likely pick out only the most obvious patterns. These patterns are likely to create many local minima in the weight space for the network. However, if the error signal from the discriminator is fed to the generator, then the generator must adapt: in effect, we are telling it "making the top left pixel white is not good enough to fool observers. Find more complex patterns".


What to do when PDFs are not Gaussian/Normal in Naive Bayes Classifier

The relationship between the axes of graph (1) and your variables $x$ and $y$ is not clear, so this generalized answer may be helpful or useless.

From graph (1) it appears that the correlation coefficient $\mathcal{C}$ of a quadratic fit of data set $\mathcal{S}$ would be much better.  Consider $y_1$ and $y_2$ approximations of $y$.

$$
\mathcal{C} (y_2, a, b, c, \mathcal{S}) > \mathcal{C} (y_1, a, b, \mathcal{S}) \\
y_2 = ax^2 + bx + c \\
y_1 = ax + b
$$

To achieve a more nearly uniform distribution, perform a least squares fit for $y_2$ against $y$ on $\mathcal{S}$ to obtain $(a, b, c)$.  Then find a mapping function that produces $y'$ and use it where the uniform distribution is desired.  A reasonable approximation is simply this.

$$y' = \frac{y}{y_2(x)}$$


Neural networks with internal dynamics in the state-space form

I think that the book "Neural Networks and Learning Machines" of Haykin can help you. In his book the chapter 13 is about neural dynamics, and there are some examples of how analise the dynamics of network.


Can an artificial intelligence be unbeatable at simple games?

There is actually a github project about 'solving' Nim that implements certain type of Q-learning reinforcement algorithm (described in undergraduate thesis of Erik Jarleberg (Royal Institute of Technology) entitled "Reinforcement learning on the combinatorial game of Nim") that supposedly finds that optimal strategy lying down there inside in the game that also human can find out.

The project uses a code in Python, which is 'only' 114 lines of code, so it can be run on your own machine if you got interested and want to test it. Github page tells also:

  There is a known optimal strategy for the game which is credited to Charles L. Bouton of Harvard which can serve as a benchmark for evaluating the performance on our Q-learning agent.

(That is the quote I refer for it getting enough good / optimal results).

Q-learning is part of the reinforcement learning algorithm family where the so called agent learns how to play the game by getting rewards from its actions and it tries to maximize the amount of total reward with following strategy:

  The goal of the agent is to maximize its total reward. It does this by adding the maximum reward attainable from future states to the reward for achieving its current state, effectively influencing the current action by the potential future reward. This potential reward is a weighted sum of the expected values of the rewards of all future steps starting from the current state.

Quote and more information in Q-learning Wikipedia-page.

For further interest on easy games tackled with Reinforcement Learning, please have a look on following paper: Playing Atari with Deep Reinforcement Learning. There is also an Udemy class on that paper and its findings.


What model structure I should use to train on low res and blurry images?

Yes, you may use the small model you have but you need to tune it.
The models you have used are not too big if any overfitting happens you can just use dropout to alleviate that.
You need to improve the quality of your images. One way is to reconstruct them using autoencoders or GANs (Super-resolution GANs). Then, you can classify them.


How to use BERT as a multi-purpose conversational AI?

Think of BERT (or similar models) as as good starting place for understanding context.

A couple options to make BERT contextualize dialogue:

Concatenate all messages with a seperator embedding and finetune a language model like BERT

This has shown good results in this paper, but understand it has weaknesses like struggling to determine order or author comprehension or conversation disentanglement. Additionally as threads grow longer this becomes more and more expensive

Use A BERT like model as an RNN and pass along author/memory information as the hidden state

Weaknesses here is that you are putting a lot of strain on the hidden state of the model, and ideally you want to train each comment block separately so the gradient wouldnt need to unroll or traverse the network in backprop (I actually implemented a variant of this for my work, and had good results)

The 2 modes above are just baseline examples that you can alter for your own needs, and im sure there is also others if you brainstorm about it.

Note if you want generation capabilities, you would want to use GPT2 or XLNet instead of BERT for unidirectional embeddings. Hope this helped.


It might be that your dataset of images is to small. Your discriminative network might hardlearn these images at which point your generative network can only produce good images if it copies the same images of your dataset.


What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network weights directly, at least. That would defeat the purpose of using backpropagation to optimize the weights and find the optimal parameters and Q-values.
2) Is there any way to limit the set of valid actions per state?
Again, not directly through the network. That would imply different network architectures for different states, which would imply optimizing several different networks. Unless you used some convoluted approach like Net2Net to keep the networks synced.
However, you can bypass the network itself.
You can, however, on top of the network, filter its outputs. Let's say you have a set of states, and in state X, only 2 out of all 5 actions are valid. When getting the Q-values for all the actions in DQN, you can detect if you're in state X, and instead of choosing the greedy action out of all actions, choose the greedy action out of all valid actions. Better yet, have your environment just send a list of all valid actions for whatever state you're in. Makes this more general, as it removes your need for discrete states, having lists of valid actions for each state, etc.
You can also really just ignore invalid actions.
You claimed

However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case.

If your environment has no time limit, then the best action really is to stand still and avoid wasting resources. However, if there is a time-limit, eventually the agent should receive some sort of penalty for not completing the task. Otherwise, if there is no time limit, you can also give him an award for completing the task. Imagining you have 10 resources, instead of having your reward interval be from [-10; 0], shift it to [0, 10] by awarding 10 points at the end of the game. This gives your agent the motivation to complete the task instead of doing useless actions, and discards your need for action-filtering at all.


Some lines above the author says

  By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution

hence we can assume $X \sim \mathcal{N}(0,1)$ with the $X$ domain being continuous

Then he says the input vector is assumed to be binary which changes the domain from continuous to discrete so we can discretize it assuming $-1 \le X \le 1$ is mapped into zero and $X< -1 $ and $X > 1$ are mapped to 1

Finally according to the 68-95-99.7 Rule we can compute

$$ E(X^2) = P(X=1) \cdot 1^2 + P(X=0) \cdot 0^2 = 0.32 $$

Finally probably the author rounds this up to $\frac{1}{3} \simeq 0.33$


Just for clarification: your description (1 sample per minute) does not match the example data (far fewer data points which is understandable, but also two data points in one minute which contradicts the initial assertion.) If your actual measurements are like that you should first work on the sampling process to get reliable data.

For creating predictions, you need to have a reasonable model of the observed process. If you're measuring environmental temperatures, you will basically have three causes of variation:

A day/night cycle
A seasonal (summer/winter) cycle
Local weather fluctuation

From only one day of samples, the only thing you can reasonably predict is that the next day will look mostly the same. If you collect more data over a year, you will be able to extract a seasonal cycle and estimate the deviations caused by local weather. "You" means either you as a researcher or any machine learning system that you program. Without sufficient historical data it is impossible to make good predictions (and even with sufficient data it's hard.)


Analysis of Training Loss and Validation Loss Graph

Simply model 1 is a better fit compared to model 2.

Graph for model 1

We notice that the training loss and validation loss aren't correlated. This means the as the training loss is decreasing, the validation loss remains the same of increases over the iterations. This means that the model is not exactly improving, but is instead overfitting the training data. This isn't what we are looking for.
Graph for model 2

In this case, there is clearly a health correlation between training loss and the validation loss. They both seem to reduce and stay at a constant value. This means that the model is well trained and is equally good on the training data as well as the hidden data.

You should stick with model 2. In case you're going ahead with model 1, make sure to use the chechpoint where both the losses are at a similar value (at around 100 -150 epochs)


Decision boundary figure in Least square GAN paper


Is this ExpectiMinimax Tree correctly drawn?

No, your tree is not an accurate representation of the problem.

Hint: Consider the case where both players flip an H on their first coin. Player 1 decides not to flip a second coin. Is the game over?


Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

Generally we can write for $R_c$ the total reward with added constant $c$ of a policy as
$$
R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i
$$
So if we have two policies with the same total reward (without added constant)
$$
\sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i
$$
but with different lengths $K_1 \neq K_2$ the total reward with added constant will be different, because the second term in $R_c$ ( $\sum_{i=0}^K c \gamma^i$ ) will be different.

As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as:
$$
10 + 0 + 0 + 0 = 10
$$
and
$$
0 + 10 = 10
$$
But when we add 100 to every reward:
$$
110 + 100 + 100 + 100 = 410
$$
and
$$
100 + 110 = 210
$$
Thus, now the first one is better.

In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R_c$ stays the same.


Are methods of exhaustive search considered to be AI?

If one thinks of intelligence as a continuous measure of optimization power (that is, how much better are outcomes for any unit of cognitive effort expended), then exhaustive search has non-zero intelligence (in that it does actually give better outcomes as more effort is expended) but very, very low intelligence (as the outcomes are better mostly by luck, and the amount of effort expended can be impossibly large).


Why did ML only become viable after Nvidia's chips were available?

There is a lot of factors for the boom of AI industry. What many people miss though is the boom has mostly been in the Machine Learning part of AI. This can be attributed to various simple reasons along with their comparisons during earlier times:

Mathematics:  The maths behind ML algorithms are pretty simple and known for a long time (whether it would work or not was not known though). During earlier times it was not possible to implement algorithms which require high precision of numbers, to be calculated on a chip, in an acceptable amount of time. One of the main arithmetic operations division of numbers still takes a lot of cycles in modern processors. Older processors were magnitude times slower than modern processors (more than 100x), this bottleneck made it impossible to train sophisticated models on contemporary processors.
Precision: Precision in calculations is an important factor in ML algorithms. 32 bit precision in processor was made in the 80's and was probably commercially available in the late 90's (x86), but it was still hella slow than current processors. This resulted in scientists improvising on the precision part and the most basic Perceptron Learning Algorithm invented in the 1960's to train a classifier uses only $1$'s and $0$'s, so basically a binary classifier. It was run on special computers. Although, it is interesting to note that we have come a full circle and Google is now using TPU's with 8-16 bit accuracy to implement ML models with great success.
Parallelization : The concept of parallelization of matrix operations is nothing new. It was only when we started to see Deep Learning as just a set of matrix operations we realized that it can be easily parallelized on massively parallel GPU's, still if your ML algorithm is not inherently parallel then it hardly matters whether you use CPU or GPU (e.g. RNN's).
Data: Probably the biggest cause in the ML boom. The Internet has provided opportunities to collect huge amounts of data from users and also make it available to interested parties. Since an ML algorithm is just a function approximator based on data, therefore data is the single most important thing in a ML algorithm. The more the data the better the performance of your model.
Cost: The cost of training a ML model has gone down significantly. So using a Supercomputer to train a model might be fine, but was it worth it? Super computers unlike normal PC's are tremendously resource hungry in terms of cooling, space, etc. A recent article on MIT Technology Review points out the carbon footprint of training a Deep Learning model (sub-branch of ML). It is quite a good indicator why it would have been infeasible to train on Supercomputers in earlier times (considering modern processors consume much lesser power and gives higher speeds).
Although, I am not sure but I think earlier supercomputers were specialised in "parallel+very high precision computing" (required for weather, astronomy, military applications, etc) and the "very high precison part" is overkill in Machine Learning scenario.

Another important aspect is nowadays everyone has access to powerful computers. Thus, anyone can build new ML models, re-train pre-existing models, modify models, etc. This was quite not possible during earlier times,

All this factors has led to a huge surge in interest in ML and has caused the boom we are seeing today. Also check out this question on how we are moving beyond digital processors.


What is the computational complexity of the forward pass of a convolutional neural network?

What is the time complexity?
The time complexity of an algorithm is the number of basic operations, such as multiplications and summations, that the algorithm performs. The time complexity is usually expressed as a function of the input's size $n$ (but this does not always have to be the case: for instance, you can express the time complexity as a function of the output's size).
Example
Rather than giving you a full answer to your question, I will try to help you by explaining, with the simplest example, how you should calculate the time complexity.
For simplicity, let's assume that we have a kernel $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ and input image $\mathbf{I} \in \mathbb{R}^{3 \times 3}$ (i.e. the kernel has the same dimensions as the input), we use a stride of $1$ and no padding. If we convolve $\mathbf{I}$ with $\mathbf{H}$, how many operations will we perform? The convolution is defined as a scalar product, so it is composed of multiplications and summations, so we need to count both of them. We have $9$ multiplications and $8$ summations, for a total of $17$ operations.
\begin{align}
\mathbf{I} \circledast \mathbf{H}
&=
\begin{bmatrix}
i_{11} & i_{12} & i_{13} \\
i_{21} & i_{22} & i_{23} \\
i_{31} & i_{32} & i_{33}
\end{bmatrix}
\odot
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & h_{33}
\end{bmatrix}\\
&=
\sum_{ij}
\begin{bmatrix}
i_{11} h_{11} & i_{12} h_{12} & i_{13} h_{13} \\
i_{21} h_{21} & i_{22}  h_{22} & i_{23} h_{23} \\
i_{31} h_{31} & i_{32} h_{32} & i_{33} h_{33}
\end{bmatrix}\\
&=
i_{11} h_{11} + i_{12} h_{12} + i_{13} h_{13} +
i_{21} h_{21} + i_{22}  h_{22} + i_{23} h_{23} +
i_{31} h_{31} + i_{32} h_{32} + i_{33} h_{33}
\end{align}
Time complexity
What is the time complexity of this convolution? To answer this question, you first need to know the input's size, $n$. The input contains $9$ elements, so its size is $n = 9$. How many operations did we perform with respect to the input's size? We performed $17$ operations, so the time complexity $\mathcal{O}(2*n) = \mathcal{O}(n)$, i.e. this operation is linear. If you are not familiar with the big-O notation, I suggest that you get familiar with it, otherwise, you will not understand anything about computational complexity.
To calculate the time complexity in the case the input's dimensions are different than the kernel's dimensions, you will need to calculate the number of times you slide the kernel over the input. You can't ignore this (as I ignored the constant $2$ above) because the number of times you slide the kernel over the input depends on the input's size, so it's a function of the input. Anyway, the paper A guide to convolution arithmetic for deep learning contains a lot of information about convolution arithmetic, so it will be helpful.
Non-linearities, pooling and fully connected layers
Note that, in the above example, I ignored the non-linearities and pooling layer. You can easily extend my reasoning to include these operations too. I also ignored the operations in the final fully connected layers. You can find how to calculate the number of operations in an MLP in this answer. If you also perform other operations or have other layers other than convolutional, pooling and fully connected, of course, you will also need to take them into account.
Forward pass
Moreover, the time complexity of the forward pass of a CNN depends on all these operations in these different layers, so you need to compute the number of operations in each layer first. However, once you know how to compute the number of operations for one convolutional layer, one pooling layer, and one fully connected layer, you can easily compute the number of operations for the other convolutional, pooling, and fully connected layers. Then you just need to sum all these operations and express your time complexity as a function of the input (and probably number of layers).
Space complexity
If you also want to compute the space complexity, you just need to do the same thing, but as a function of the space that you use, i.e. how many variables you use to perform the convolution.


Examples of uses of artificial intelligence at work

This is an interesting question in that it addresses how AI becomes a part of our mundane existence.

@JadenTravnik's pithy comment is quite correct--spam filtering is a form of AI that that it so fundamental to the information eco system that it has become nearly invisible.

It's useful to be able to define what we mean by Artificial (or Algorithmic) Intelligence, and to distinguish it from computation in general.  A fundamental distinction is that AI makes decisions as opposed to merely crunching numbers.

My feeling is that applied AI is still a very specialized field, so that people using it dynamically to solve problems at work today are those who work in the field of AI, as opposed to workers in general.

You may find this related question useful: Is AI programming useful in everyday programs?

This may change in the future, where a type of expert system, for instance, could assist a worker by presenting it's own analysis and conclusions on a decision problem. But in most office settings, that type of reasoning is still the domain of humans.

My advice would be to continue on Jaden's track, looking for examples of where AI underlies everyday functions.


Can this tic tac toe program be considered AI?

This is basically reinforcement learning. The state space contains your moves, and the value function are the value you store at the end. And your rewards are the end results. And you have episodic game. It is an AI method. Consider looking at value iteration, policy iteration, SARSA, Q-learning. The difference between neural network method and yours is you are not doing function approximation with neural network for the value function, you are doing tabular method.


Why are the rewards of my RL agent for the Atari Breakout game decreasing after a certain number of episodes?

It seems that decaying the learning rate solved my problem. I changed learning_rate from 0.001 to 0.0001


Do I need an encoder-decoder architecture to predict the next item of a sequence?

You don't need to Encoder-Decoder here. When using seq2seq learning for text (for example, for translation) you need encoder-decoder to encode the words into the numeric vectors and decode the vectors into the words. Therefore, for your numerical case, you don't need an encoder or decoder to train the RNN.


L2 normalization in Facenet paper acting as a Constraint?

The constraint is enforced with bespoke code. If FaceNet was implemented in NumPy, and the embedding layer vector (pre-constraint) was in the NumPy array h, then the code might look like:

e = h / np.linalg.norm(h)

The variable e would then contain the desired embedding with L2 distance of 1. In practice, even with NumPy, the code might be more complex due to handling mini-batches. More likely this will not be implemented in NumPy.

In this Keras/TensorFlow-based FaceNet implementation you can see how it may be done in practice:

# L2 normalization
X = Lambda(lambda  x: K.l2_normalize(x,axis=1))(X)

This scaling transformation is considered part of the neural network code (it is part of the Keras model building routine in the above snippet), so there needs to be corresponding support for back propagation through the embedding. When using an automatic differentiation framework such as Theano or TensorFlow, then no extra code is required.


Is the following the correct implementation of the Q learning algorithm for a neural network?

Your pseudocode is fairly close to something that could be implemented. There are a couple of omissions, where it is not clear whether your implementation would do the right thing. There are a few non-general assumptions that are worth noting - they don't prevent you implementing for a specific purpose, but do prevent you writing the most generic DQN that you could. There is one mistake that prevents it working at all.

Your mistake is here:

  Do gradient step where error = (y - max(q1))^2, only the max of q1 gets any gradient

The loss function and gradient must be applied to the state/action pair s1, a1, i.e. the action that was taken. This is an important distinction for when the agent takes exploratory (non-greedy) actions, as it must if it is going to learn about their values.

A corrected statement might read: Do gradient step where error = (y - Q(s1,a1))^2, only the output for a1 gets any gradient.

Regarding the rest of the pseudocode I could make the following observations:

  The state, s1, is initialised randomly

This is fine if it is possible. However, it is more usual in Q learning to initialise the state to whatever the normal start of the episode would be. That might be from some randomised starting rules, but that is not quite the same as
picking a random state from all possible states.

  action a1 is chosen based off the max of q1 (or randomly)

It is not quite clear what you mean by "or randomly". For Q learning - and most RL - exploration is important. So this is not a design time choice for your algorithm, but a runtime choice that must itself be made so that all actions can be tried. Typically you would use an $\epsilon$-greedy policy here that took the argmax action by default, but with probability $\epsilon$ took a random sample from actions with same chance of each possible action.

  state s2 is found by progressing s1 based on a1

This is one of a few places where you have not quite captured the nature of a time step in a general MDP. You appear to have made the assumption that rewards are directly associated with states (with language like "reward at s2"). Whilst some MDPs are like that, it is not true in general.

A better, more general statement might be:

s2, reward, done are found by taking a step from s1, using a1 as the action

Typically the environment (or a wrapper for it that establishes state representations and rewards) will provide a step method that returns the tuple $s_{t+1}, r_{t+1}, (s_{t+1} = s_T)$, although you could also hold these values in the environment model for the current time step and return them as required.

This statement:

  (The "otherwise" doesn't match bellmans equation as α=1)

I don't know which Bellman equation you are referring to, but they don't include a learning rate, $\alpha$ in any version I have seen. Perhaps you are referring to a tabular Q learning update? In any case, it doesn't seem relevant, as whatever learning rate you are using is now inside the gradient step, and does not need to be part of the pseuodcode.

Finally, regarding this:

  I am also not taking into account storing network states (in this case, the network is called Q) for proper learning to avoid catastrophic forgetting, as I'm more concerned on getting the specifics right before good practice.

DQN will very likely not work at all, for most environments, unless you implement experience replay i.e. training the neural network on a random sample of stored s1,a1,r,s2,done taken from a table that you build up on each time step. Usually you cannot simply use the latest data from the current step. This is not just "good practice", but an almost-always necessary stability improvement.


How to classify data which is spiral in shape?

There are many approaches to this kind of problem. The most obvious one is to create new features. The best features I can come up with is to transform the coordinates to spherical coordinates.

I have not found a way to do it in playground, so I just created a few features that should help with this (sin features). After 500 iterations it will saturate and will fluctuate at 0.1 score. This suggest that no further improvement will be done and most probably I should make the hidden layer wider or add another layer.

Not a surprise that after adding just one neuron to the hidden layer you easily get 0.013 after 300 iterations. Similar thing happens by adding a new layer (0.017, but after significantly longer 500 iterations. Also no surprise as it is harder to propagate the errors). Most probably you can play with a learning rate or do an adaptive learning to make it faster, but this is not the point here.


Number of classes vs number of parameters/layers?

You are correct about your first assumption, but not about your second assumption.
More layers does not always mean better pattern detection. The analogy that in the deeper layers the network learns more complex features is an oversimplified one. It is true to some extent, although it is not enough to explain very complex architectures like GoogLeNet.
Moreover, using more layers than neccessary increases the risk of observing vanishing gradients. Because the network is too deep i.e. has too many layers, the gradient will be vanishingly small when backpropagating so many layers in the beginning will not get any significant update during training.


How does the Alpha Zero's move encoding work?

Let's do the code, so all the details are down.
Encoding dictionary:
codes, i = {}, 0
for nSquares in range(1,8):
    for direction in ["N", "NE", "E", "SE", "S", "SW", "W", "NW"]:
        codes[(nSquares,direction)] = i
        i += 1

You'll see that the codes dictionary will have 56 entries in it for each (nSquares,direction) pair.
The knight moves we'll encode as the long "two"-cell edge move first and the short "one"-cell edge second:
for two in ["N","S"]:
    for one in ["E","W"]:
        codes[("knight", two, one)] , i = i , i + 1
for two in ["E","W"]:
    for one in ["N","S"]:
        codes[("knight", two, one)] , i = i , i + 1

Now we should have 64 codes. As I understand, the final 9 moves are when a pawn reaches the final rank and chosen to be underpromoted. It can reach teh final rank either by moving N, or by capturing NE, NW. Underpromotion is possible to three pieces. Writing the code:
for move in ["N","NW","NE"]:
    for promote_to in ["Rook","Knight","Bishop"]:
        codes[("underpromotion", move, promote_to)] , i = i , i + 1

We get 73 codes as described.
Policy
The distribution over actions is a (8,8,73) tensor (it is not formally a "policy", since policy should also depend on state, but lets cut this corner for this discussion):
policy = np.zeros((8,8,73))

Let's also do codes for columns for convenience:
columns = { k:v for v,k in enumerate("abcdefgh")}

How would one numerically represent the move 1. e4

The first two dimensions choose the figure you are moving. So, that'd be e2 pawn. And we move north 'N' by 2 cells.
So, we put 1 into the tensor at the appropriate indices. Note that you have to subtract 1 from the row index, to make it zero-based.
e4policy = np.zeros((8,8,73))
e4policy[ columns['e'] , 2 - 1 , codes[(2 , "N")]] = 1

How would one numerically represent the move 1. NF3

The first two dimensions choose the figure you are moving. So, that'd be g1 knight. And we perform north-west N,W knight jump.
NF3policy = np.zeros((8,8,73))
NF3policy[ columns['g'] , 1 - 1 , codes[("knight", 'N' , 'W')]] = 1

Generally, the policy is a probability distribution over all possible moves, so the policy tensor would have several non-zero probability values in it. For example an opening policy that does 1.e4 or 1.Nf3 with 50/50 probability would be:
openingPolicy = (e4policy + NF3policy) / 2

Hope this clears things up.


Agent in toy environment only learns to act optimally with small discount factors

This is an episodic problem, and there should be no issue in theory with most learning algorithms coping without a discount factor (or setting gamma = 1).

  Are there any explanations to why the agent only works for small discount factors in this example?

The most likely explanations are:

You have a mistake in your implementation or use of DQN.
You have an incorrect setting of a neural network hyperparameter. My initial thought would be you have mistakenly put a non-linearity like softmax or sigmoid on the NN outputs (it needs to be a linear output). Or it could just be that 10 hidden neurons is not enough for this representation
Convergence requires far longer to train than you thought, once you introduce non-linear relationships between state and expected reward.

It is not surprising that the simpler puzzle trains more easily, as the agent can reduce it down to a linear counting puzzle without really needing to "look" at the representation. I would expect the Q values to be very poor approximations to the correct values after so few iterations, but the problem is simple enough that the incorrect values still produce an optimal behaviour.


How to "forward" updated NN model to a transferred model?

I think there is no simple way to transfer knowledge changes between different models.
If you take your initial model and create a new version of it which you use to learn some other task (like "Walk to a specific location"), then the values copied from the first (original) model change in the second model.
From that moment on, training the former model on another task will have different effects on its weights than continuing the training of the second model, whose parameter have been changed already.
Consider, for example, that you had changed the mass of the robot and trained the initial model on that new task already. Then, if you took all the re-trained parameters from the first model and implanted them into the second model (trained on walking to a certain location), then you would essentially overwrite the additional knowledge the second model had gained already during its initial transfer-learning-process (not even taking into consideration any additional parameters appended to the list of parameters in the second model).
So, you will have to re-train all three models (the original one and the two transfer-learning models) if you change the mass of the robot.
Edit:
There might be an option to apply the same knowledge changes to another model architecture if you refrain from pure transfer learning. This can be achieved with a more modularized model architecture.
Consider that you train your first model on walking straight head. Let's call this model $m_{walk}$.
Then you intend to recycle $m_{walk}$ for another task, like walking straight ahead to a given location. Such a model architecture could be realized in two ways:

You apply real transfer learning, retraining a copy of $m_{walk}$ to walk straight ahead until it reaches a certain location
You take $m_{walk}$, don't change it, but add a second model (let's call it $m_{navigator}$), which is trained on predicting $go$ vs. $stop$.

In the second case, your overall model architecture (let's call it $m_{go\_to}$) consists of both one model used for walking (i.e. $m_{walk}$) and one model architecture which is used for predicting $go$ vs. $stop$ (i.e. $m_{navigator}$).
The idea then is that the robot executes the actions suggested by $m_{walk}$ until $m_{navigator}$ suggests stopping, upon which prediction the suggestions by $m_{walk}$ will be ignored.
Then, whenever you retrain your model $m_{walk}$ (e.g. because the mass of some robot changes), you can simply apply the changes to $m_{go\_to}$ by replacing $m_{walk}$ by a new version, leaving the rest of $m_{go\_to}$ intact.
If you generalize $m_{walk}$ to not only walk straight ahead, but also to take turns etc. and you generalize $m_{navigator}$ to predict going $left$, $right$, $straight\ ahead$, $back$, or $stop$ (being predicted when a certain destination has been reached), you can generalize $m_{go\_to}$ to walk where-ever you want it to go.


In the Markov chain, how are the directions to each successive state defined?

It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is arbitrary; it depends on what you want to model with it.

As DuttaA says in his comment, you can imagine that all nodes are linked with all other nodes, but those links have a transition probability of 0.0; so the probability of observing the sequence ACDF is actually 0.0 as well. In order to make the diagram more easy to understand, zero-probability transitions are generally not shown.

There is no prescribed method of populating the transition probabilities. You can define the probabilities manually, randomly, or however you like. You could derive them from observable training data.

Re updating: this depends on the application. If you are modelling a process with known probabilities, you would not update the probabilities; if you are modelling a dynamic process which changes over time, then this is something you might want to consider. However, from my own experience (HMMs in speech recognition), once they have been assigned, they are generally kept as they are.

A Markov chain is really a fairly basic model; it gets more complex with a Hidden Markov Model, where you would generally use a learning algorithm to assign the transition and emission probabilities.


How should I implement the state transition when it is a Gaussian distribution?

I figured this out by going to the author's publicly available github code. It turned out  the authors were just generating the transition probability $p$ from $\mathcal{N}(\mu,\sigma^2)$ at the beginning of each episode for some reason. Answering it myself for the sake of not leaving this question unanswered.


The problem you describe is a matter of perspective; although you want to predict the inputs, defining the outputs of the BB (blackbox) as the inputs to a new system (an inverse BB) whose outputs are that of the inputs of the original BB, then one can use previous literature. There are some assumptions using this method such as "each input can be calculated from an inverse function using 1 or more outputs" so take precaution.

I'd recommend reading through this paper by Richard Sutton. Its introduction defines the problem mathematically and references some common approaches as well as its own method.


Measuring Object size using Deep Neural Network

Yes! This most certainly can be done. Since you have a labeled dataset, that makes it all the more simple!

I would take a look at this project and that should get you where you need to go.

The implementation details should be pretty straightforward. Let me know if I can help further.


Why isn't the implementation of my policy evaluation for a simple MDP converging?

The issue is that in your list comprehension in def V_pi(state) you have

return sum(prob * (reward + mdp.discount*V[newState]) for prob, reward, newState in
        mdp.succProbReward(state))

whereas with the way you have defined the succProbReward output, it should be

return sum(prob * (reward + mdp.discount*V[newState]) for newState, prob, reward in
        mdp.succProbReward(state))

When I run this it converges immediately with a reward of 0 for all states, which I believe is correct for the policy you specified. If I change the policy it also seems to give reasonable results.


How do self-driving cars construct paths?

As you say, GPS is not precise enough for the purpose (until recently it was only accurate within 5m or so, since 2018 there are receivers that have an accuracy of about 30cm). Instead, autonomous vehicles have a multitude of sensors, mostly cameras and radar, which record the surrounding area and monitor the road ahead. Due to them being flat, mostly one colour, and often with lines or other markers on them, roads are usually fairly easy to spot, which is why most success has been made driving on roads as opposed to off-road. Once you know exactly where you are and where you want to go, computing the correct trajectory is then just a matter of maths and physics.

For an academic paper on the subject of trajectory planning see Local Trajectory Planning and Tracking of Autonomous Vehicles, Using Clothoid Tentacles Method.

It quickly becomes more complex when other road users and obstacles are taken into account; here machine learning is used to identify stationary and movable objects at high speed from the sensor input. Reacting to the input is a further problem, and one reason why there aren't any self-driving cars on the roads today.

This is all on driving automation level 2 and above; on the lower levels things are somewhat easier. For example, the latest model Nissan LEAF has an automatic parking mode, where the car self-steers, guided by camera images and sonar, but still requires the driver to indicate the final position of the vehicle. Apart from that, it is fully automatic.


It is not reverse-artificial-intelligence .not only escher , almost all paintings can be interpreted into abstract thought , in fact that is what the brain does. Even when you read something your brain processes this into its mental language ("hermaneutics").

for more on "mental language" and "abstract thought" you should check https://drive.google.com/file/d/0B8i61jl8OE3XdHRCSkV1VFNqTWc/view (geoff hinton is a noted researcher in this field (ai))


How to combine input from different types of data sources?

Handling Multiple Input Types

Multiple input types are common as learning technology moves from academic labs and open source examples into the real world.  The design process includes the below four steps, which related directly to this question.  There are other steps that don't centrally related not mentioned here.

First Step — List Inputs in Most Usable Form

In the case of this question, there are three in the list.

Monophonic or multichannel audio data as an array of samples or sample vectors acquired at a fixed sampling rate
Acceleration data as an array of samples in $\mathbb{R}^n$ where $n \in \{1, 2, 3\}$ acquired at a fixed sample rate.
Visible light or IR detection data as an array of samples of $c$ channels at a fixed sample rate.

Fixed sample rate is listed with each type.  Variable sample rates pose additional burden on learning.  If any of the three are acquired with a variable sample rate, they can be converted to fixed sample rate arrays provided there is a time stamp or a time-since-last-sample channel available with each sample vector.

Variable to fixed sample rate conversion can be accomplished easily via ffmpeg, programmatically via its library, or through any equivalent software designed for this purpose.  The acceleration and visible light signals can also be adapted for ffmpeg's input.

When a specific dynamic characteristic must be preserved, finer control over the interpolation can be obtained using cubic spline algorithms.  Such algorithms were originally written in FORTRAN or C and have been ported to Java and Python.

Second Step — When Spectral Data is Practically Required

With time series data, the phenomena being learned is often largely a function of vibration frequencies and amplitudes.  This is almost always the case with audio data.  The exception is when transient detection is the primary teller of the conditions being measured rather than more continuous sounds.

A note is spectral in description.  A knock is a transient and both spectra and actual rise characteristics are indicative of source.

Presenting digital samples from the transducer circuit (in this case the microphone diaphragm's position in response to pressure waves) to the learning system directly is not a best practice.  Unconverted vibration information forces the learning system to learn transformations for which optimized algorithms already exist.  This slows learning and generally pushes complexity beyond the learning system's capacity.

When the spectral features best characterize the phenomena being learned, as in hums, screeches, vowel sounds, music, horns, or beeps, the best practice is to convert the samples in the time domain to frequency domain data.

This is usually done using one of the forms of the FFT (fast Fourier transform) algorithm in combination with a RMS (root mean square) calculation of amplitude and windowing.  Windowing attenuation forms, such as the Hamming window or the cosine window, are available in most packages that have FFT capabilities.

The sample rate is also important.  Nyquist's sample rate criteria must be met by the A-to-D converter (in the sound card or instrumentation circuitry) to ensure that the spectral range includes the best indicators of the phenomena to be learned.  In machine learning, such indicators are called features.

In most cases, audio or vibration information is presented to a learning system or sub-system as spectral frequencies and amplitudes.  Vibration recognition systems and speech to text systems generally execute FFTs (with RMS and windowing) on the raw input.  In human ears, the conversion to spectral information is mechanical, using the tapering geometry of the ear's cochlea.

There also exists algorithms that can directly develop spectral information from variable sample rate data to amplitudes and frequencies, combining two transformations into one for efficiency of computing resources and additional input accuracy.

In summary of this section, whether light, IR, acceleration, or other data is treated as audio and converted to spectral information is based on whether

The signals are vibrations that have rises and falls within specific frequency ranges like hums, notes, and speech or
They are transients (as with tapping the mechanical package on which the accelerometer is mounted) or
They are binary in nature (as in the case of linear encoder signals).

Third Step — When Topology of Types

The topological orientation of the input source types to one another is key to feeding the information into any learning system.  These are the two pure extremes.

The signals from the source types are describing the same physical or data mining property.
The signals are measuring independent properties that together describe the state of the system being measured.

In the case of (1), a normalized sample from one type of input channel is, from a learning point of view, indistinguishable from a normalized sample from another.  This would be the case if the accelerometer is measuring the same vibration that the microphone is measuring.  This is the less usual case, but is more common when the data is for study of vibration to project mechanical or materials failures in advance.

With Background Clarified, the Q&A Becomes Clearer

The question asked this.

  How [does one best] combine input from different types of data sources?

Whether the source data is case (1) or case (2) or some midpoint between them is key to answering this question.  Samples from different source types that measure single phenomena can be phase adjusted, spectrally adjusted, and otherwise normalized to present a single data type at the input.  Merging into a single serial data stream would be the optimal pre-processing for training in case (1).

More often, the topological orientation of the different input source types conforms to case (2) above.  In that case, different input types should be presented in parallel to the learning system inputs.  The merging would be counterproductive in this case, for a reason that will be clarified in the next section below.

To avoid sparsity in input data, which creates additional training burden, the sample rates should be equalized for all channels of all data source types.  This may require conversion of the sample rate, which can be done using the same techniques (above) for converting from variable to fixed sample rates.

Images of Data

The question stated this.

  Right now, the approach I [was considering is] to convert all data into images and combine them into a single image and train my neural network.

The attempt to normalize and combine into one input stream is the correct approach if (1) above is the case.  However, presenting data as images requires the learning system restore the data to a less redundant, more feature encoded form, so conversion to images is not recommended.

If (2) above is the case, the provision of unique and property-independent data through the same input tensors requires the artificial network sort the input samples by type, which can be done with recurrent or attention based artificial networks, but it is unnecessary.  Here's the rule.

  If the signals from different sources are measuring independent properties to which learning will be applied, do not merge the input streams.  Instead, present them in parallel to the learning system so that the learning system does not need to also learn how to discriminate sample types.

The reason topology is mentioned in the subtitle of this section is that there is a third case, (3), which is a hybrid of (1) and (2) as hinted above.

Many excellent detection and control systems process the input channels in relation to one another prior to the actual learning system components.  This pre-processing takes advantage of known relationships in the data and the topology of signals in phase space.

The phase space of two or three signals can be visualized in 2D or 3D where each channel is assigned to a Cartesian coordinate system's axis and the time domain is not represented by any axis.  Mechanical fault detection can be optimized using phase space analysis.

An example of this more complex case (3) is in the human recognition of both consonant and vowel sounds and various other non-speech audio phenomena.  After the cochlear conversion to spectra, the transient information (as from a 'k' sound or a click) and tone information (as from an 'e' or a whistle) is presented to speech recognition nets in the brain in an integrated form.

This decreases the learning time newborns take to learn voice and household sounds and probably improved evolutionary viability in times of global stress or local predatory activity.

In automated vehicles and smart robotics, the same is true.  Some sounds, accelerations, and visible light changes are transient and some are vibrations.  Studying the topology of the surface created by the input signal types and their spectra in phase space provides much information.  Armed with the results of such studies allows excellence in the preparation of input to a learning system.

Fourth Step — Normalizing and Standardizing

Once the above is complete, understanding how to present the data to the input layer of an artificial network becomes clearer.  The goal of this design step is to have relatively equal ranges for the components of each input tensor, whether presented serially or in parallel depending on the above criteria.

Doing so reduces the burden on the learning system.  All signals are presented at full strength but within the clipping range of the signal.  This is a general principle and is important for both data center (batch) learning and real time learning and adaptive systems.


Yes, it does and at many parts of the solution. For one of the core components - intent detection - Intento did a benchmark comparing IBM Watson and other similar products: https://www.slideshare.net/KonstantinSavenkov/nlu-intent-detection-benchmark-by-intento-august-2017

Outside of intent detection, there are other areas where AI techniques help - e.g. disambiguation, bootstrapping a bot from chat logs etc. Specifically for IBM Watson, you can learn more here: https://www.ibm.com/cloud/watson-assistant/features/


Can exogenous variables be state features in reinforcement learning?

Including exogenous variables in your state representation certainly can be useful, as long as you expect them to be relevant information for determining the action to pick. So, state features are not only useful if you expect your agent (through application of actions) to have (partial) influence on those state variables; you just want the state variables themselves to be informative for your next action to take / prediction of expected future rewards.

However, if you only have exogenous variables, i.e. if you expect your agent to have no influence whatsoever on what states you'll end up in next... then the full problem definition typically used in RL (Markov decision processes) may be unnecessarily complex, and you may prefer to look into the Multi-Armed Bandits (MAB) problem formulation. If you're already familiar with RL / MDPs, you may think of MAB problems as (sequences of) single-step episodes, where you always just look at the current state and don't care at all about future states (because you expect to have 0 influence on them).

In theory, the RL / MDP framework is more general and is also applicable to those MAB problems, but RL algorithms that support this framework may perform worse than MAB algorithms in practice, because they (informally speaking) still put in effort trying to "learn" how their actions affect future states (a waste of effort when you expect there to be no such influence from the agent).


Do convolutional neural networks perform convolution or cross-correlation?

Short answer
Theoretically, convolutional neural networks (CNNs) can either perform the cross-correlation or convolution: it does not really matter whether they perform the cross-correlation or convolution because the kernels are learnable, so they can adapt to the cross-correlation or convolution given the data, although, in the typical diagrams, CNNs are shown to perform the cross-correlation because (in libraries like TensorFlow) they are typically implemented with cross-correlations (and cross-correlations are conceptually simpler than convolutions). Moreover, in general, the kernels can or not be symmetric (although they typically won't be symmetric). In the case they are symmetric, the cross-correlation is equal to the convolution.
Long answer
To understand the answer to this question, I will provide two examples that show the similarities and differences between the convolution and cross-correlation operations. I will focus on the convolution and cross-correlation applied to 1-dimensional discrete and finite signals (which is the simplest case to which these operations can be applied) because, essentially, CNNs process finite and discrete signals (although typically higher-dimensional ones, but this answer applies to higher-dimensional signals too). Moreover, in this answer, I will assume that you are at least familiar with how the convolution (or cross-correlation) in a CNN is performed, so that I do not have to explain these operations in detail (otherwise this answer would be even longer).
What is the convolution and cross-correlation?
Both the convolution and the cross-correlation operations are defined as the dot product between a small matrix and different parts of another typically bigger matrix (in the case of CNNs, it is an image or a feature map). Here's the usual illustration (of the cross-correlation, but the idea of the convolution is the same!).

Example 1
To be more concrete, let's suppose that we have the output of a function (or signal) $f$ grouped in a matrix $$f = [2, 1, 3, 5, 4] \in \mathbb{R}^{1 \times 5},$$ and the output of a kernel function also grouped in another matrix $$h=[1, -1] \in \mathbb{R}^{1 \times 2}.$$  For simplicity, let's assume that we do not pad the input signal and we perform the convolution and cross-correlation with a stride of 1 (I assume that you are familiar with the concepts of padding and stride).
Convolution
Then the convolution of $f$ with $h$, denoted as $f \circledast h = g_1$, where $\circledast$ is the convolution operator, is computed as follows
\begin{align}
f \circledast h = g_1
&=\\
[(-1)*2 + 1*1, \\
(-1)*1 + 1*3, \\
(-1)*3 + 1*5, \\
(-1)*5+1*4]
&=\\
[-2 + 1, -1 + 3, -3 + 5, -5 + 4]
&=\\
[-1, 2, 2, -1] \in \mathbb{R}^{1 \times 4}
\end{align}
So, the convolution of $f$ with $h$ is computed as a series of element-wise multiplications between the horizontally flipped kernel $h$, i.e. $[-1, 1]$, and each $1 \times 2$ window of $f$, each of which is followed by a summation (i.e. a dot product). This follows from the definition of convolution (which I will not report here).
Cross-correlation
Similarly, the cross-correlation of $f$ with $h$, denoted as $f \otimes h = g_2$, where $\otimes$ is the cross-correlation operator, is also defined as a dot product between $h$ and different parts of $f$, but without flipping the elements of the kernel before applying the element-wise multiplications, that is
\begin{align}
f \otimes h = g_2
&=\\
[1*2 + (-1)*1, \\
1*1 + (-1)*3, \\
1*3 + (-1)*5, \\
1*5 + (-1)*4]
&=\\
[2 - 1, 1 - 3, 3 - 5, 5 - 4]
&=\\
[1, -2, -2, 1] \in \mathbb{R}^{1 \times 4}
\end{align}
Notes

The only difference between the convolution and cross-correlation operations is that, in the first case, the kernel is flipped (along all spatial dimensions) before being applied.

In both cases, the result is a $1 \times 4$ vector. If we had convolved $f$ with a $1 \times 1$ vector, the result would have been a $1 \times 5$ vector. Recall that we assumed no padding (i.e. we don't add dummy elements to the left or right borders of $f$) and stride 1 (i.e. we shift the kernel to the right one element at a time). Similarly, if we had convolved $f$ with a $1 \times 3$, the result would have been a $1 \times 3$ vector (as you will see from the next example).

The results of the convolution and cross-correlation, $g_1$ and $g_2$, are different. Specifically, one is the negated version of the other. So, the result of the convolution is generally different than the result of the cross-correlation, given the same signals and kernels (as you might have suspected).

Example 2: symmetric kernel
Now, let's convolve $f$ with a $1 \times 3$ kernel that is symmetric around the middle element, $h_2 = [-1, 2, -1]$. Let's first compute the convolution.
\begin{align}
f \circledast h_2 = g_3
&=\\
[(-1)*2 + 1*2 + (-1) * 3,\\ (-1)*1 + 2*3 + (-1) * 5,\\ (-1)*3 + 2*5 + (-1) * 4]
&=\\
[-2 + 2 + -3, -1 + 6 + -5, -3 + 10 + -4]
&=\\
[-3, 0, 3]
\in \mathbb{R}^{1 \times 3}
\end{align}
Now, let's compute the cross-correlation
\begin{align}
f \otimes h_2 = g_4
&=\\
[(-1)*2 + 1*2 + (-1) * 3, \\ (-1)*1 + 2*3 + (-1) * 5, \\ (-1)*3 + 2*5 + (-1) * 4]
&=\\
[-3, 0, 3]
\in \mathbb{R}^{1 \times 3}
\end{align}
Yes, that's right! In this case, the result of the convolution and the cross-correlation is the same. This is because the kernel is symmetric around the middle element. This result applies to any convolution or cross-correlation in any dimension. For example, the convolution of the 2d Gaussian kernel (a centric-symmetric kernel) and a 2d image is equal to the cross-correlation of the same signals.
CNNs have learnable kernels
In the case of CNNs, the kernels are the learnable parameters, so we do not know beforehand whether the kernels will be symmetric or not around their middle element. They won't probably be. In any case, CNNs can perform either the cross-correlation (i.e. no flip of the filter) or convolution: it does not really matter if they perform cross-correlation or convolution because the filter is learnable and can adapt to the data and tasks that you want to solve, although, in the visualizations and diagrams, CNNs are typically shown to perform the cross-correlation (but this does not have to be the case in practice).
Do libraries implement the convolution or correlation?
In practice, certain libraries provide functions to compute both convolution and cross-correlation. For example, NumPy provides both the functions convolve and correlate to compute both the convolution and cross-correlation, respectively. If you execute the following piece of code (Python 3.7), you will get results that are consistent with my explanations above.
import numpy as np

f = np.array([2., 1., 3., 5., 4.])

h = np.array([1., -1.])
h2 = np.array([-1., 2., 1.])

g1 = np.convolve(f, h, mode="valid")
g2 = np.correlate(f, h, mode="valid")

print("g1 =", g1) # g1 = [-1.  2.  2. -1.]
print("g2 =", g2) # g2 = [ 1. -2. -2.  1.]

However, NumPy is not really a library that provides out-of-the-box functionality to build CNNs.
On the other hand, TensorFlow's and PyTorch's functions to build the convolutional layers actually perform cross-correlations. As I said above, although it does not really matter whether CNNs perform the convolution or cross-correlation, this naming is misleading. Here's a proof that TensorFlow's tf.nn.conv1d actually implements the cross-correlation.
import tensorflow as tf # TensorFlow 2.2

f = tf.constant([2., 1., 3., 5., 4.], dtype=tf.float32)
h = tf.constant([1., -1.], dtype=tf.float32)

# Reshaping the inputs because conv1d accepts only certain shapes.
f = tf.reshape(f, [1, int(f.shape[0]), 1])
h = tf.reshape(h, [int(h.shape[0]), 1, 1])

g = tf.nn.conv1d(f, h, stride=1, padding="VALID")
print("g =", g) # [1, -2, -2, 1]

Further reading
After having written this answer, I found the article Convolution vs. Cross-Correlation (2019) by Rachel Draelos, which essentially says the same thing that I am saying here, but provides more details and examples.


Using graph searching to solve peg solitaire?

Your approach seems reasonable to me. The edges do not necessarily have to be numbers, but, if you wish, you could also encode the actions as numbers. For example, the weight of an edge could represent the "cost" of the corresponding action. If there's no natural cost associated with an action, then you can add a unit cost for each action.


Non-differentiable reward function to update a neural network

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?

In reinforcement learning (RL), the reward function is often not differentiable with respect to any learnable parameters. In fact it is quite common to not know the function at all, and apply a model-free learning method based purely on sampling many observations of state, action, reward and next state. For reward, you don't need to know the reward function as a function of your parameters, but you do need to know how to caclulate it from each observation. That means that the reward is either provided as part of the environment, or it is clear how to calculate it from the initial state, action or next state. In the case of an agent that creates a neural network then reports the accuracy, you can probably view the NN training process and its results including accuracy on a validation set as a "black box" that reports back an arbitrary reward signal.

  In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network

Policy gradient methods are one broad family of RL methods which are also often model-free. Alternatively, value-based methods are broadly the other choice (e.g. Q-learning) or can be used in combination with policy gradient methods (e.g. Actor-Critic). All of these can be model-free, and often are.

Unless you want to apply a model-based RL method, then you have no need to find the reward function in an explicit form based on any parameters. Even if you did want to use a model-based approach, the reward function does not need to be differentiable.

  I was wondering if I can use another value, for example silhouette score (in a different scenario) as the reward signal?

Yes this would likely be viable, and could be made to work similarly to the paper, provided there is some meaningful connection between the parameters you manipulate and the end result.

RL can be made to work as a generic way of optimising numeric measure influenced due to a vector of parameters. That includes indirect optimisation of neural networks by non-differentiable metrics such as accuracy. There is nothing special about the choice of accuracy here, the metric just needs to be related to the parameters being learned.

There is a catch. Indirect learning using RL can produce very noisy training data, due to e.g. covariance between all the parameters, and it can take many samples to get a clear gradient signal through all the noise and make meaningful updates. It can be inefficient compared to other optimisation methods if they are available - however, the ability to extract a gradient for the parameters even when rules are complex and non-differentiable is a nice feature of RL.


Is there a way to get landmark features automatically learned by a neural network?

No, you can't. In CNN, if you want to detect landmark, you need to prepare data with region box, it's coordinates, width, height, than number of points that should be detected and points coordinates. Then your target vector should be,

This is your target vector. Optionally you can use YOLO algorithm.


How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
\begin{equation}
    T = P
\end{equation}

\begin{equation}
    W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
\end{equation}

Applying a prototype vector as input :

\begin{equation}
    \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
\end{equation}

Because they are orthogonal we have :
\begin{equation}
    \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
\end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation}
    W \cdot p_k= (R - Q \cdot I) \cdot p _k
\end{equation}
prototype vectors
continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
ability to generalize or may increase the biological
plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used


How should I decay $\epsilon$ in Q-learning?

The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit in the latter case I imagine you would want to start with a smaller $\epsilon$ value and then decay by a larger amount.


Is it possible to design an AI with two inputs and a Boolean output?

Yes... But...  you have a lot of hours.. possibly days or weeks.. of work before you are at that point.
Your bigger problem is apparent from your concerns in the second paragraph.  It doesn't sound as though your org has a solid grasp of the problem at the moment.  For that reason, it seems that some first steps are in order.
Data Exploration
Begin by collecting all of the data points.  I'd recommend that you perhaps begin with some statistical analysis of the data as a whole, including some basic visualization and generating a covariance matrix.  From there, begin to look at using some clustering methods to identify possible patterns.  Along the way, you will almost certainly go down the path of some dimensionality reduction, either via something like PCA or possibly identifying useless features.
Feature Selection
Based on the exploration work above you should now have a much better understanding of your data and relationships within it.  Based on this, it's time to start thinking about how to generate a model that produces the desired output.  Frankly, you may discover that something as simple as a random forest classification or even a clustering method such as DBSCAN can be used to initially train and then continuously fit your data, producing either a binary classification with the random forest or a yes/no cluster with the clustering technique.
Is More Required?
Of course, something more sophisticated might be required, but if it is you would know have a better handle on the problem and likely be able to intuitively generate a large dataset that could be used for a neural network.
Oh... And as a concluding thought... It might turn out that after all of this analysis the problem cannot be solved with the data that you have.  At that point you have to go back and see if there are other data points that could be gathered.


What is the concept of channels in CNNs?

Say you have a colored image that is 200x200 pixels. The standard is such that the input matrix is a 200x200 matrix with 3 channels. The first convolutional layer would have a filter that is size $N×M×3$, where $N,M<200$ (I think they're usually set to 3 or 5).

  Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be 200x3 or 3x200. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

The different dimensions (width, height, and number of channels) do have different meanings, the intuition behind them is different, and this is important. You could rearrange your data, but if you then plug it into an implementation of CNNs that expects data in the original format, it would likely perform poorly.

The important observation to make is that the intuition behind CNNs is that they encode the "prior" assumption or knowledge, the "heuristic", the "rule of thumb", of location invariance. The intuition is that, when looking at images, we often want our Neural Network to be able to consistently (in the same way) detect features (maybe low-level features such as edges, corners, or maybe high-level features such as complete faces) regardless of where they are. It should not matter whether a face is located in the top-left corner or the bottom-right corner of an image, detecting that it is there should still be performed in the same way (i.e. likely requires exactly the same combination of learned weights in our network). That is what we mean with location invariance.

That intution of location invariance is implemented by using "filters" or "feature detectors" that we "slide" along the entire image. These are the things you mentioned having dimensionality $N \times M \times 3$. The intuition of location invariance is implemented by taking the exact same filter, and re-applying it in different locations of the image.

If you change the order in which you present your data, you will break this property of location invariance. Instead, you will replace it with a rather strange property of... for example, "width-colour" invariance. You might get a filter that can detect the same type of feature regardless its $x$-coordinate in an image, and regarldess of the colour in which it was drawn, but the $y$-coordinate will suddenly become relevant; your filter may be able to detect edges of any colour in the bottom of an image, but fail to recognize the same edges in the top-side of an image. This is not an intuition that I would expect to work successfully in most image recognition tasks.

Note that there may also be advantages in terms of computation time in having the data ordered in a certain way, depending on what calculations you're going to perform using that data afterwards (typically lots of matrix multiplications). It is best to have the data stored in RAM in such a way that the inner-most loops of algorithms using the data (matrix multiplication) access the data sequentially, in the same order that it is stored in. This is the most efficient way in which to access data from RAM, and will result in the fastest computations. You can generally safely expect that implementations in large frameworks like Tensorflow and PyTorch will already require you to supply data in whatever format is the most efficient by default.


Why does the adversarial search minimax algorithm use Depth-First Search (DFS) instead of Breadth-First Search (BFS)?

The primary reason is that Breadth-First Search requires much more memory (and this probably also makes it a little bit slower in practice, due to time required to allocate memory, jumping around in memory rather than working with what's still in the CPU's caches, etc.). Breadth-First Search needs memory to remember "where it was" in all the different branches, whereas Depth-First Search completes an entire path first before recursing back -- which doesn't really require any memory other than the stack trace. This is assuming we're using a recursive implementation for DFS -- which we normally do in the case of minimax.

You can clearly see this if you look at pseudocode for the two approaches (ignoring the minimax details here, just presenting pseudocode for straightforward searches):

BreadthFirstSearch(start):
    Q = new queue()
    Q.append(start)
    while Q is not empty:
        node = Q.pop()
        if node is leaf:
            do something with leaf
        else:
            for each child of node:
                Q.append(child)

DepthFirstSearch(start):
    if start is leaf:
        do something with leaf
    for each child of start:
        DepthFirstSearch(child)
        // probably do something with return value from the recursive DFS call

You see that the BFS requires a queue object that explicitly stores a bunch of stuff in memory, whereas DFS doesn't.

There's more to the story once you get to extensions of Minimax, like Alpha-Beta pruning and Iterative Deepening... but since the question is just about Minimax, I'll leave it at that for now.


How are the parameters of the Bernoulli distribution learned?

I heard back from the authors of the paper.

As expected the bernoulli sampler is non-differentiable, so as an approximation they use the expectation of the samplers gradient.

$
\begin{align*}
\frac{dL}{dv_i} &= \frac{dL}{dBern(\sigma(v_i))} * \frac{dBern(\sigma(v_i))}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
&\approx \frac{dL}{dBern(\sigma(v_i))} * \frac{dE[Bern(\sigma(v_i))]}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
&= \frac{dL}{dBern(\sigma(v_i))} * \frac{d\sigma(v_i)}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
&= \frac{dL}{dBern(\sigma(v_i))} * 1 * \frac{d\sigma(v_i)}{dv_i} \\
\end{align*}
$

So the answer ended up being as simple as that.


What makes Google Translate fail on the Latin language?

I don't know what model Google is using for their translations, but it's highly likely that they're using one of today's SOTA deep learning models.
The latest NLP models are trained on data scraped from the web, e.g. OpenAI's GPT-2 was trained on a dataset of 8 million web pages, Google's BERT was trained on the BookCorpus (800M words) and English Wikipedia (2.500M words) pages.
Now think about the amount of latin web pages and notice that there are over 6 million english wikipedia articles but less than 135.000 in latin (see here).
As you can see, massive amounts of data are crucial for neural machine translation and I assume there is simply not enough out there for latin. Plus latin is one of the most complex and complicated languages, this makes the task not easier. Maybe Google and Co also focus less on a 'dead' language which is not spoken anymore and has it's right to exist more for educational purposes.


How would you encode your input vector/matrix from a sequence of moves in game like tasks to train an AI? e.g. Chess AI?

The core of the question seems to really be: "how to approach thinking about this", where "this" is the input of an AI player.

Modern attempts at game playing AI players try to replace a human player "as is". No advantage whatsoever. This implies that we want to feed the same "raw input" to the software player and to the human player. For a video game, the raw input is usually the screen we see, but it could also include sound, vibration feedback from a game controller, force feedback from driving gears, etc. Basically any sensor input available. For a physical board game like Go, the raw input could be a video feed, as done for AlphaGo. Doing so, we fall back to the video game approach. So the input to the AI player is often a tensor, where each element is the intensity of a screen pixel, or some other sensor signal (note there is nothing here about the shape of the tensor, please see below).

The way of thinking here is actually pretty simple, and neat: We want an AI player to replace a human player. So we list up what the human player can sense, and model each input as a tensor (this includes vectors and matrices).

There is perhaps a subtlety here. We face choices not so obvious to make, and there are usually trials and errors (many errors). For example, the Deep-Q Network from DeepMind to play Atari games takes as input differences between two consecutive video frames. Some other approach could solve the same problem with actual sequences of inputs (e.g. 10 consecutive frames). The two approaches are valid---they just have different tradeoffs we must evaluate to find the best configuration.

Another subtelty is the "shape" of the input. Should a single (W, H) input frame be a (W*H) vector, or a (W, H) matrix, or something else? The ramifications can go far. If the frame is an RGB frame with transparency, the input could well be a tensor of shape (W, H, 4), a "cubic volume" with 4 "slices" of shape (W, H)---one for each RGB channel, and a fourth one for transparency. Imagine now if you also have sound, and can add a fifth "slice"---and more if you have stereo or Dolby system channels.

Last point: There is no reference to the architecture of the AI player beyond the input here. Although most exemples refer to recent architectures based on neural networks, this "thinking" also applies whatever can leverage the input. For example, the Big Blue system from IBM, who won Chess against Gary Kasparov, is an AI player using exclusively (if my memory's good) clever tree search. Yet its input is a tensor representing the Chess board.


I need to select the image from a predefined dataset that are the closest to the input, is this possible or do I even need to use ML/AI?

You will only need to use ML or AI if
a) the dataset is very big.
b) It is difficult to get the meaning or value of the image.(ex: group of ants, photos of stars in night sky)
And many more.

Below is the way I thought it can be done through a machine(with deeplearning, CNN)
1. Train the model
        arrange the images in dataset into clusters (the number of clusters can
        be decided based on the criticality of the application, more critical the
        application more should be the number of clusters) .

Predict about input
    Then predict the nearest image in the dataset to the provided image.

I recently listened a podcast in which, a problem is approached similarly. The podcast is about youtube predictions and the result is to provide closest video from the youtube videos ( here the dataset).Below is the podcast.
    ( https://open.spotify.com/episode/6PcMtVXR58i7iu8kLH40Wd?si=feyu0LjESoiE479xHkaqSA )


What are the real world uses for SAT solvers?

Instead of talking about just SAT solvers, let me talk about optimization in general. Many economic problems can be cast as optimization problems: for example, FedEx may have a list of packages and the destinations for those packages, and must decide which packages to put on which trucks, and what order to deliver those packages in.

If you write out a mathematical description of this problem, there are a truly stunning number of possible solutions, and a well-defined way to evaluate which of two solutions is better. A solver is an algorithm that will evaluate a solution, come up with another solution, and then evaluate that one, and so on.

In small cases and simple problems, the solver can also terminate with a proof that it is actually the best solution possible. But typically instead the solver just reports "this is the best solution that I've seen," and that's used. An improvement in the solver means you can reliably get lower-cost solutions than you were seeing before.

For the SAT problem specifically, the Wikipedia page on SAT gives some examples:

  Since the SAT problem is NP-complete, only algorithms with exponential worst-case complexity are known for it. In spite of this, efficient and scalable algorithms for SAT were developed over the last decade[when?] and have contributed to dramatic advances in our ability to automatically solve problem instances involving tens of thousands of variables and millions of constraints (i.e. clauses).[1] Examples of such problems in electronic design automation (EDA) include formal equivalence checking, model checking, formal verification of pipelined microprocessors,[12] automatic test pattern generation, routing of FPGAs,[14] planning, and scheduling problems, and so on. A SAT-solving engine is now considered to be an essential component in the EDA toolbox.


A very efficient approach to what you are trying to do is velocity obstacles.

Assuming two agents use constant velocity motion vectors, a velocity obstacle models a geometric region in which if the endpoint of the velocity vector for agent 1 falls, it will collide with agent 2 (and vice versa). Hence you can predict what velocity vectors will lead to collisions between the agents and choose velocity vectors that do not collide.

There are very good examples and tutorials here:
http://gamma.cs.unc.edu/RVO/


Which loss function should I use for binary classification?

There are several loss functions that you can use for binary classification. For example, you could use the binary cross-entropy or the hinge loss functions. See, for example, the tutorials Binary Classification Tutorial with the Keras Deep Learning Library (2016) and How to Choose Loss Functions When Training Deep Learning Neural Networks (2019) by Jason Brownlee. Have also a look at Keras documentation of its available loss functions.


What are the conditions for the convergence of SARSA to the optimal value function?

I have the conditions for convergence in these notes SARSA convergence by Nahum Shimkin.

The Robbins-Monro conditions above hold for $α_t$.
Every state-action pair is visited infinitely often
The policy is greedy with respect to the policy derived from $Q$ in the limit
The controlled Markov chain is communicating: every state can be reached from any other with positive probability (under some policy).
$\operatorname{Var}{R(s, a)} < \infty$, where $R$ is the reward function


What is the exact output of the Inception ResNet V2's feature extraction layer?

Due to this article: https://arxiv.org/pdf/1512.00567v3.pdf?source=post_page--------------------------- ,

I try to flatten the 3-d tensor in to 1d vector: 8*8*2048, because in the article, the pool layer of inception resnet v2 at page 6 is Pool: 8 * 8 * 2048.

But at the end, my code showed the error:
 ValueError: cannot reshape array of size 33423360 into shape (340,131072)

This is all my code:

from keras.applications.inception_resnet_v2 import InceptionResNetV2
from keras.applications.inception_resnet_v2 import preprocess_input
from keras.models import Model
from keras.preprocessing.image import load_img
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from imutils import paths
from keras.applications import imagenet_utils
from keras.preprocessing.image import img_to_array
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from keras.preprocessing import image
import random
import os
import numpy as np
import cv2

# Path to image
image_path = list(paths.list_images('/content/drive/My Drive/casia-299-small'))
# Random image path
random.shuffle(image_path)
# Get image name
labels = [p.split(os.path.sep)[-2] for p in image_path]

# Encode face name in to number
le = LabelEncoder()
labels = le.fit_transform(labels)

# Load model inception v2, include_top = Fale to ignore Fully Connected layer
model = InceptionResNetV2(include_top = False, weights = 'imagenet')

# Load images and resize into required input size of Inception Resnet v2 299x299
list_image = []
for (j, imagePath) in enumerate(image_path):
    image = load_img(imagePath, target_size = (299, 299, 3))
    image = img_to_array(image)

    image = np.expand_dims(image, 0)
    image = imagenet_utils.preprocess_input(image)

    list_image.append(image)

# Use pre-trained model to extract feature
list_image = np.vstack(list_image)
print("LIst image: ", list_image)
features = model.predict(list_image)
print("feature: ", features)
print("feature shape[0]: ", features.shape[0])
print("feature shape: ", features.shape)
features = features.reshape((features.shape[0], 8*8*2048))

# Split training set and test set n ratio of 80-20
x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size = 0.2, random_state =42)

params = {'C': [0.1, 1.0, 10.0, 100.0]}
model = GridSearchCV(LogisticRegression(), params)
model.fit(x_train,y_train)
model.save('/content/drive/My Drive/casia-299-small/myweight1.h5')
print('Best parameter for the model {}'.format(model.best_params_))

preds = model.predict(x_test)
print(classification_report(y_test, preds))
```