Can we define the AI singularity mathematically?

I found someone that has done this thing! You can hear a good explanation in Marcus Hutter's answer to this question about rewards given to AIXI. He describes a work that seems to be referring to this paper:

Universal Knowledge-Seeking Agents for Stochastic Environments

I'll edit this answer later with a full explanation of the approach, but essentially the idea is that you use an AIXI model that does optimal reinforcement learning, giving it a reward that is based on information gained (phrased in a careful way to avoid a few common pitfalls). As a result, it learns to choose actions that give it the most information possible to predict the impacts of it's actions. This results in a "scientist" like behaviour, and you could imagine it doing things like turing the entire earth into a supercollider to better understand some physics laws if it decides that is the best approach for gaining maximum information. It would probably also do plenty of very unethical psycology experiements, for example, if it ended up deciding that human actions were important to predict and understand.

It's not a "safe" singularity in that sense, but that's okay, I didn't require that. It's at least a formal definition. It requires doing some uncomputable things, but the hope of future research is that we can make close enough approximations to those uncomputable things to be good enough anyway.

I feel this theory is lacking any explanation of how feasable such a task is since it uses uncomputable agents, so I won't accept it yet, but it's the best answer I've seen so far. And I'll be watching future research closely to see if they can get a better handle on feasability, there seems to be quite a bit of work that has gone on in finding computable approximations to AIXI. The reason I care about feasability is because it is very relevant for mathmatically deciding how plausable something like an "intelligence explosion" actually is. So if a theory doesn't talk about feasability, it is missing out on a big piece of this question. Still this theory seems hopeful. For example, maybe there are fundamental computational limits to maximizing some reward functions, and we can prove that even certain levels of approximations for this reward function aren't computable. That would be a really interesting negative result.

In general, I think the idea of using reinforcement learning, then choosing a reward function that tries to capture something instrinsic (such as "curiosity") is a very good approach to trying to formally define the singularity. I look forward to seeing other potential reward functions defined in the future, I don't expect this to be the only one.


Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence.
Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on previous iterations squared. https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c.
The loss surfaces of the neural networks is a difficult and poorly understood topic in the present. I suppose, that the poor convergence of SGD is caused by the roughness of loss surface, where the gradient makes big leaps, and jumps over the mimima. The adaptive learing strategy of the Adam, on the other hand, allows to dive into the valleys.


Which algorithm to use to solve this optimization problem?

A possible approach would be starting by grouping the actions into all possible groups.
For 30 actions grouped in a maximum of 6 this would mean:
$$
C^{30}_6 + C^{30}_5 + C^{30}_4 + C^{30}_3 + C^{30}_2 + C^{30}_1 = 768 211
$$
Operations.

Then, I would define the constraints to simplify the problem, as in a Constraint Satisfaction Problem (CSP). The main constraint that you have is that one action can only be represented in one Operation. So, if you choose 1 of the 768ks operations to start with, the second option will be restricted by the condition.

Then I would do some sort of planning and sum the costs until the end doing a depth-first search. Whenever I find a total cost better the previous best, you would update the best value. (Remember that the search ends when between all the operations used, all the actions were explored once.)

While doing the search the costs are added up on the go. You can precompute a vector with the same size as the number of operations where each value corresponds to the total cost of the operation, and then sum it as you are going down the path.

You should also add some heuristic to prevent checking the same operations in different orders. For example doing operating [2,3,4] followed by [7,10], where each number represents an action, is the same as doing: [7,10] followed by [2,3,4].


Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?

Yes, RNN can work on the functions you have mentioned. In fact, neural networks can approximate anything (Universal Approximation Theorem). This question also reminds me of Neural Turing Machine.
But, it would be a complete waste to use RNNs or NNs for such a task.


A good way to understand the mathematical details of variational autoencoders through implementation?

I even don't know what exactly back-propagation is used for, if this would help.

Programming backprop manually is one of the best exercises in machine learning. In fact, to do this you don't need a special package, like Tensorflor or Keras, just a general computing library like NumPy (and a whiteboard).

Though I dare to say that VAE is not the best thing to implement, if you are not 100% familiar with backprop. It's pretty advanced technique and you risk to mix up fundamental (e.g. backprop) and specific ideas (e.g. deliberately crafted loss function) and slow down your understanding.

I really think that writing a simple feed-forward neural net (including forward and backward passes) is a very good starting point, after which you can move on to VAE details, if you'd like.

  Any suggestions on where/how to start?

If you feel confident with algebra and calculus, I highly recommend this Stanford course, especially its assignments. The course itself is about computer vision and convolutional neural networks, but the first assignments are very general and will give a lot of insight of what and how neural networks are doing.


Single AI-person in software firm about to adopt AI, good idea?

I think you should be fine if you are using other people's tried and tested algorithms and have a reasonable understanding of machine learning best practices - how to clean and balance your dataset and how to effectively measure performance. Maybe if you are trying to come up with new algorithms / DNN architectures you will struggle.


What is the difference between LSTM and fully connected LSTM?

Based on the citations in the ConvLSTM paper I have come to the conclusion that they mean the Peephole LSTM when they say fully connected LSTM. In the paper that they have taken the encoder-decoder-predictor model from, where they refer to a fully connected LSTM, a Peephole LSTM is used. Also they take their fully connected LSTM definition from this paper, which again uses the Peephole LSTM.

With that the difference would be the added "peephole connections", that lets the gate layers look at the cell states and access the constant error carousel.


Relation between size of parameters and complexity of model with overfitting

Regarding your first question, this is domain/task specific, and not always the case. My guess of why it happens in your situation (You did not specify your domain, so ill assume it's sometimes outside $(-1,1)$) is that higher order polynomials increase much faster than lower order ones so it may fall under the trap where the coefficients have to be large to handle it. (e.g. if $x=2.0$ then $ 2^5*x^2 = x^7 \rightarrow 32*x^2 = x^7$)

Now to answer why more points will cause less overfitting is sheerly because you have more equations. Your loss is generally some form of average distance accross points and predictions. Given $N$ points I can always find an $N-1$ order polynomial that fits it exactly. Instead of a formal proof I will provide a form in which you can easily generate this polynomial from $N$ $(x_i, y_i)$ pairs
$$ f(x) = \sum_{i=1}^{N}y_i\prod_{j=1\atop j \ != i}^N\frac{(x-x_j)}{(x_i-x_j)}$$

As you can see, for each $x_i$ it will zero out all terms in the sum except one where the product become 1 making $f(x_i) = y_i$ by construction. So adding more points will prevent whatever optimization process you are learning to directly solve each point exactly, but instead try to find an algorithm that best generalizes to each of the points.


Will quantum computing have any kind of effect on the development of AI?

Preface

  "Before answering this question, let me preface by stating that the
  following is simply MY answer as a Machine Learning Researcher and
  "Hobbyist" Theoretical Physicist, although I have strong feelings that
  my answer will most certainly be proven as true, I am more than sure
  that others will have differing opinions as with everything else in
  this world!" ~ Me

Answer

That being said, I would say that the answer to your question, is unequivocally YES! I could write a short book on the subject (that might be a good idea actually) but I will keep this response brief, although I am happy to answer any further questions your may have to which I possess the answers in the comments!

The primary reasons that I believe the newly blossoming field of Quantum Computation will have a massive impact on the field of Machine Learning in general, are as follows:

The most simple reason for my belief is that the primary goal of Machine Learning, is to create an entity which is capable of coherent, self-aware thought much like we exhibit as human beings. We know that the brain is what allows us to be capable of such feats, and thus I view the field something like brain counterfeiting. Without going into esoteric detail, there are many subtleties of the brain's workings which are thought to be quantum mechanical in operation, and thus would suggest that the path of least resistance to replicating the system would require a quantum mechanical computational medium.
The second primary rationale which solidifies my position, is the point which you briefly touched upon by stating that,

  "We know that according to Shor's Algorithm Quantum Computing can reduce the complexity > of breaking an encryption from O(2n) to O(2n2) (not sure)."

This is primarily due to the quantum phenomena referred to as Super Position, which allows for a multiple qubit gate to not only work with the options of 00, 01, 10 and 11 (assuming a two qubit gate) but to also work with any combination in-between, during the computation. Although when the result is obtained (this is what is referred to as collapsing the wave function) you will only still have a resulting state space  with 2^n possibilities, where n is the number of qubits. This being said, there are very clever ways, by which one can design their algorithms to make full use of this technically infinite computational space before observing the final results.

Conclusion

I hope that my answer is helpful to you in some way, although I am aware that it is not a very in-depth answer, I feel that it hits the primary reasons why my personal belief is that there will be a wall which is hit, in the pursuit of a general AI, while we are limited to classical computation faculties; and thus will require quantum based computation before we are able to truly mimic the brain's most well kept secrets! The next couple decades should be VERY interesting in the fields, keep a close eye on the latest happenings!


Understanding GLIE conditions for epsilon greedy approach

For (Q1) we mathematically define $\mathbb{E}[exploit(T)]$ as:
$$\mathbb{E}[exploit(T)] = \sum_{t=1}^T (1-\epsilon_t)$$
where $\epsilon_t$ is the exploration probability at round $t$.
The equation is basically telling us the expected number of times you pull a greedy arm over the time horizon $T$.
A more useful metric would be:
$$\mathbb{E}[explore(T)] = \sum_{t=1}^T \epsilon_t$$
It os obvious that:
$$\mathbb{E}[explore(T)] + \mathbb{E}[exploit(T)] = T$$
First, I would like to gloss over the idea why this formulation produces sublinear regret. Although there are additional assumptions that have clearly not been mentioned in the lecture, I will say it here. There is a paradigm of a 'consistent' algorithm in online learning, which in simple terms mean that the actual best arm will be identified eventually as $T \to \infty$ and by the design of your algorithm, you will be pulling this arm greedily (or with a high probability) after some time giving you a regret better than linear regret.
Now in this case, if you assume the algorithm is 'consistent' it is clear that the number of exploration steps (which is decided by exploration probability) at each step will slowly have a reduced growth as compared to the linear case i.e you are now slowly reducing exploration as your 'consistent' algorithm gurantees you have with high probability identified the best arm. From this definition/intuition given above it is clear that this means that $\mathbb{E}[explore(T)]$ should grow at a slower rate while $\mathbb{E}[exploit(T)]$ should grow at a faster rate with $T$.
From this inutitive idea rises that:
$$\lim_{T\rightarrow\infty}\frac{\mathbb{E}[exploit(T)]}{T}=1 $$
or
$$\lim_{T\rightarrow\infty}\frac{\mathbb{E}[explore(T)]}{T}=0 $$
For (Q2), it is not. It might be in expectation, but in general, it is not. I am unsure of the type of bandits used here so I will not do this calculation. In anyways, whenever sampling is involved, it means you have sampled a trajectory or a particular instance of a joint probability distribution and its expectation can never be equal to one sample (we need $\infty$ samples).(Although I am not clear on this one, the slides are not very clear to me).
For (Q3) Yes. The instructor states an inequality which I could not reproduce (I tried to do it, but it reverses the inequality sign of the instructor, you can try it yourself using the below set of equations and using the fact that $\epsilon_T \leq \epsilon_t, \quad \forall T \geq t$). Now consider the following set of equations:
$$\mathbb{E}[explore(T)] = \sum_{t=1}^T \epsilon_t$$
(a good approximate summation can be found using integrals like this)
$$\mathbb{E}[explore(T)] \approx k \sqrt T$$
where $k$ is a constant.
$$-\mathbb{E}[explore(T)] \approx -k \sqrt T$$
$$T-\mathbb{E}[explore(T)] \approx T - k \sqrt T$$
$$\frac{\mathbb{E}[exploit(T)]}{T} \approx 1-k \frac{1}{\sqrt T}$$
which answers your (Q4) that:
$$\lim_{T \to \infty} \frac{\mathbb{E}[exploit(T)]}{T} \approx \lim_{T \to \infty} 1-k \frac{1}{\sqrt T} = 1$$


Why are documents kept separated when training a text classifier?

Common words should not be ignored.  For instance, there may be a correlation between document classes and whether more indefinite articles like a or an are included or more definite ones like the or this.

Neither is word count is a good substitution for word distribution. (Frequency though it is commonly (not necessarily frequently) used is an incorrect term.  Words do not normally occur periodically in natural language.)  Representing word distribution throughout the concatenated text of multiple documents of a single category.

The distribution of use of individual words and the distribution of language units and word pair proximity may correlate well to document classification, but the linguistic pattern distribution may vary from document to document within the same class, so the distribution of distributions is relevant.  The deviations between distributions is information needed for proper statistical analysis.

In ten documents of differing sizes, the distribution of words may differ for some words and correlate for others.  Concatenating them hides the distribution of distributions.  From an information theory perspective, concatenating and totaling across the concatenation would represent an information loss with respect to separate documents analysis.

The naive Bayesian machinery produces the most reliable categorization when all information is integrated into the probability calculations.


How powerful are the computers that power the most advanced artificial intelligence nowdays

From the Deep Stack paper:

This seems to be for training:

  For the turn network, ten million poker turn situations (from after the turn card is dealt) were generated and solved with 6,144 CPU cores of the Calcul Quebec MP2 research cluster, using over 175 core years of computation time. For the flop network, one million poker flop situations (from after the flop cards are dealt) were generated and solved. These situations were solved using DeepStack’s depth limited solver with the turn network used for the counterfactual values at public states immediately after the turn card. We used a cluster of 20 GPUS and one-half of a GPU year of computation time. For the auxiliary network, ten million situations were generated and the target values were obtained by enumerating all 22,100 possible flops and averaging the counterfactual values from the flop network’s output.

And this for actual play:

  The re-solving computation and neural network evaluations are both implemented in Torch7 (53) and run on a single NVIDIA GeForce GTX 1080 graphics card.

For comparison: The distributed version of AlphaGo took 1.920 CPUs and 280 GPUs to run.


Let me preface this by acknowledging that this question is prone to opinion. This answer is, in so far as it is possible, primarily observation based.

My understanding is that, when it comes to publishing in a journal, doing so as an individual (without backing from an institution) is, in general, going to be frowned upon.

As you may know: scientific research, including AI research, is generally subject to peer review. When it comes to journals, this is mandatory. The review process attempts to enhance and preserve the integrety of the information published. As an additional safeguard, submissions will often expected to be backed by endorsement from someone within the academic community (typically an institution).

However, and as Pasaba correctly points out (in his comment on the OP), research and publishing are not neccessarily the same thing. Furthermore, not having an endorsement does not not stop you from making a contribution to the field.

For example, you can publish code and/or articles on websites such as Github, and engage with communities of professional and hobbiest researchers around the web, (e.g. this Stack Exchange).

Note that there is also some scope for endorsement without being a direct member of a research institution. For example: arXiv, whilst not strictly a journal, is an open archive that supports endorsement by request.

Without knowing your circumstances, it's hard to know exactly what to do. However, my general advice is to find and engage with communities, and build a network of collaborative peers, rather than trying to succeed in an isolated fashion.


Is there a way of converting a neural network to another one that represents the same function?

To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. number of nodes in a layer), and some other structural aspects, such as recurrent layers, convolution layers, pool layers, etc.

Theory

In terms of the theory, there are a few ideas that provide insight into your question. First is the commonly cited universal approximation theorem, which states that a single hidden layer, feed forward neural network with a sigmoid activation and finite width (i.e. the number of nodes in single the hidden layer) can approximate any n-dimensional real-valued function. Superficially, this tells us that we can do a whole lot with one of the most simple architectures and could (in theory) approximate more complex architectures such as LSTMs, CNNs, etc.

Yet, while this seems to indicate that reductions are always possible, research on the asymmetry in representational power between depth and width hints that things may not be so simple (for example, see this paper and this paper). Specifically, network depth provides much greater representational power relative to width. Intuitively, this makes sense – depth allows us to build hierarchies and stack multiple non-linearities. Indeed, this is where we begin to see the outlines of what the answer looks like in practice – depth provides a clear, easy way to bake structure into our network.

So, the upshot from the theoretical perspective is that the universal approximation theorem answers in the affirmative; yes, you could reduce a complex network architecture to a single hidden layer network, but the basis for doing this is a bit naïve – it doesn't consider the efficiency of the network relative to the number of parameters.The two papers linked above imply this.

Practice

In practice, there are issues with performing such a reduction. To date, the most successful models have deep architectures. More importantly though, many of these models are specialized – we have CNNs for computer vision, RNNs for sequence-based data, etc. These architectures were conceived as a way to integrate knowledge about the structure of the problem into the structure of the network to improve learning of the desired representation, and this too requires depth.

In fact, last year a paper took this idea to the extreme. Weight agnostic neural networks are specifically constructed such that their architecture is in essence the representation with a single parameter (weight) that can be varied to generate multiple distinct models with different performance characteristics. The WANNs that were developed in this research weren't especially deep, but they also weren't single layered.

The upshot here is that, for complex problems, depth is necessary for efficient learning and gradient computation and that it can be used to encode problem structure into a network. While it may be possible to reduce a learned representation to a single hidden layer network, it's not inconceivable that you may end up with more parameters condensed into a single hidden layer with very high dimensional gradients. In terms of implementation, this isn't desirable.


How should continuous action/gesture recognition be performed differently than isolated action recognition

This question actually includes many. I try to answer a couple.

First of all, you need to make sure for your use case to know the non-functional requirements. It is helpful to know that you have a soft real-time case (if harm is done in case of too late predictions it might also be hard real-time)

Latency: how much time may pass after the action was made until the prediction is there?
Stability of prediction: how fast it's the system allowed to change it's prediction? Once every 0.5s?
General hardware, especially memory usage
Optimization metric: maybe accuracy does not tell you enough?
Evaluation setup: likely, you need to have multiple "evaluation points" per video, e.g. Once after the system had one second, once after it had 1.5,... You will also want to consider the stability of the prediction.

Now to the question about how you can do that. If you didn't try it so far, I recommend looking at optical flow.


What is the difference between human brains and neural networks?

One incredibly important difference between humans and NNs is that the human brain is the result of billions of years of evolution whereas NNs were partially inspired by looking at the result and thinking "... we could do that" (utmost respect for Hubel and Wiesel).

Human brains (and in fact anything biological really) have an embedded structure to them within the DNA of the animal. DNA has about 4 MB of data and incredibly contains the information of where arms go, where to put sensors and in what density, how to initialize neural structures, the chemical balances that drive neural activation, memory architecture, and learning mechanisms among many many other things. This is phenomenal. Note, the placement of neurons and their connections isn't encoded in dna, rather the rules dictating how these connections form is. This is fundamentally different from simply saying "there are 3 conv layers then 2 fully connected layers...".
There has been some progress at neural evolution that I highly recommend checking out which is promising though.

Another important difference is that during "runtime" (lol), human brains (and other biological neural nets) have a multitude of functions beyond the neurons. Things like Glial cells. There are about 3.7 Glial cells for every neuron in your body. They are a supportive cell in the central nervous system that surround neurons and provide support for and insulation between them and trim dead neurons. This maintenance is continuous update for neural structures and allows resources to be utilized most effectively. With fMRIs, neurologists are only beginning to understand the how these small changes affect brains.

This isn't to say that its impossible to have an artificial NN that can have the same high level capabilities as a human. Its just that there is a lot that is missing from our current models. Its like we are trying to replicate the sun with a campfire but heck, they are both warm.


Let's first express a network of arbitrary topology and heterogeneous or homogeneous cell type arrangements as

$$ N(T, H, s) := \, \big[\, \mathcal{Y} = F(P_s, \, \mathcal{X}) \,\big] \\
   s \in \mathbb{C} \; \land \; s \le S \; \text{,} $$

where $S$ is the number of learning states or rounds, $N$ is the network of $T$ topology and $H$ hyper-parameter structure and values that at stage $s$ produces a $P$ parameterized function $f$ of $\mathcal{X}$ resulting in $\mathcal{Y}$. In supervised learning, the goal is that $F(P_s)$ approaches a conceptually ideal function $F_i$ as $s \rightarrow S$.

The popular loss aggregation norms are not quite as the question defines them. The below more canonically expresses the level 1 and 2 norms, which systematically aggregate multidimensional disparity between an intermediate result at some stage (epoch and example index) of training and the conceptual ideal toward which the network in training is intended to converge.

$$ {||F-\mathcal{Y}||}_1 = \sum{|F_i - y_i|} \\
   {||F-\mathcal{Y}||}_2 = \sqrt{\sum{(F_i - y_i)}^2} $$

These equations have been mutated by various authors to make various points, but those mutations have obscured the obviousness of their original relationship. The first is where distance can be aggregated through only orthogonal vector displacements. The second is where aggregation uses the minimum Cartesian distance by extending the Pythagorean theorem.

Note that quadratic loss is a term with some ambiguity. These are all broadly describable as quadratic expressions of loss.

Distance as described in the second expression above
RMS where the sum is divided by the number of dimensions
$\text{count}(i)$
Just the sum of squared difference by itself

Cross entropy is an extension of Claude Shannon's information theory concepts based on the work of Bohr, Boltzmann, Gibbs, Maxwell, von Neumann, Frisch, Fermi, and others who were interested in quanta and the thermodynamic concept of entropy as a universal principle running through mater, energy, and knowledge.

$$ S = k_B \log{\Omega} \\
   H(X) = - \sum_i p(x_i) \, \log_2{\, p(x_i)} \\
   H(p, \, q) = -\sum_{x \in \mathcal{X}} \, p(x) \, \log_2{\, q(x)} $$

In this progression of theory, we begin with a fundamental postulate in quantum physics, where $k_B$ is Boltzmann's constant and $\Omega$ are the number of microstates for the quanta. The next relation is Shannon's adaptation for information, where $H$ is the entropy in bits, thus the $\log_2$ instead of a natural logarithm. The third relation above expresses cross-entropy in bits for features $\mathcal{X}$ is based on the Kullback-Leibler divergence.  the p-attenuated sum of bits of q-information in .

Notice that $p$ and $q$ are probabilities, not $F$ or $\mathcal{Y}$ values, so one cannot substitute labels and outputs of a network into them and retain the meaning of cross entropy. Therefore level 1 and 2 norms are closely related, but cross-entropy is not a norm; it is the dispersion of one thing Cartesian distance aggregation scheme like them. Cross-entropy is remotely related but is statistically more sophisticated. To produce a cross-entropy loss function of form

$$ {||F-\mathcal{Y}||}_H = \mathcal{P}(F, y) \; \text{,} $$

one must derive the probabilistic function $\mathcal{P}$ that represents the cross entropy for two distributions in some way that is theoretically sound on the basis of both information theory and convergence resource conservation. It is not clear that the interpretation of cross entropy in the context of gradient descent and back propagation has caught up with the concepts of entropy in quantum theory. That's an area needing further research and deeper theoretical consideration.

In the question, the cross-entropy expression is not properly characterized, most evident in the fact that the expression is independent of the labels $\mathcal{Y}$, which would be fine if for unsupervised learning except that no other basis for evaluation is represented in the expression. For the term cross-entropy to be valid, the basis for evaluation must include two distributions, a target one and one that represents the current state of learning.

The derivatives of the three norms (assuming the cross entropy is properly characterized) can be studied for the case of $\ell$ ReLU layers by generalizing the chain rule (from differential calculus) as applied to ReLU and the loss function developed by applying each of the three norms to aggregate measures of disparity from optimal.

Regarding the inference in sub-question (1) nothing of particular value can be assumed about the Jacobians of level 2 norms from level 1 norms, both with respect to parameters $P$ or vice versa, except the retention of sign. This is because we cannot determine much about the correlation between the output channels of the network.

There is no doubt, regarding sub-question (2), that some constraint, set of constraints, stochastic distribution applied to initialization, hyper-parameter settings, or data set features, labels, or number of examples have implications for the reliability and accuracy of convergence. The PAC (probably approximately correct) learning framework is one system of theory that approaches this question with mathematical rigor. One of its practical uses, among others, is to derive inequalities that predict feasibility in some cases and produce more lucid approaches to learning system projects.


Since I am working with audio synthesis projects, I can suggest you to go through the recently released paper "MelNet: A Generative Model for Audio in the Frequency Domain", in this architecture human like speech is synthesized, I think this might be better than WaveNet where the generated audio does contain robotic nature, I think this will definitely help you with Emotional speech synthesis as it generates human-like voices.
You can check for online demos as well.

Demo

Here Is the Paper!

Related YouTube Video

Hope this helps!


Why do we need both encoder and decoder in sequence to sequence prediction?

(Old question, I know...)

It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.

Example for Sequence-to-sequence without two RNNs

To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.

Approaches like Eager Translation break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.

Why encoder-decoder works better very often

Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.

This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.

The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.

In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:

At every time step, read one source token, then write one target token: Previous elements in the source sequence are represented only in lossy recurrent states, while future elements cannot be accessed at all.
First read all source tokens, then write all target tokens.: the meaning of the entire source sentence has to be compressed into a fixed-size recurrent state vector.


Without any additional information, lean towards the vote of the best performing classifier when it comes to ties.

However, as others have stated already, it is best to analyze the performances in more detail (e.g. confusion matrices).

For instance, it could be that model B almost always classifies class X correctly (hardly any false positives). In that case, you could lean towards the prediction of model B if it predicts class X and model A does not. In other words; you could weigh the votes of the models based on how well they did in similar, previous predictions.


A successor function is a function that generates a next state from the current state, plus the choices that affect state changes. In e.g. the 8 queens problem, a state might be the location of 5 queens so far, the choice might be where to put the next queen, and the successor function would return the resulting state with 6 queens on the board.

Typically a solver will store/cache a current state, make a choice, and use the successor function to discover what the next state is. Then it may call the heuristic function on the next state and make a decision whether to continue that search deeper into the search tree (or recursion) or try another choice at the current state.


Do self-driving cars resort to randomness to make decisions?

Driving Priorities

When considering the kind of modeling needed to create reliable and safe autonomous vehicles, the following driving safety and efficacy criteria should be considered, listed in priority with the most important first.

The safety of those inside the vehicle and outside the vehicle
Reduction of wear on passengers
The safety of property
The arrival at the given destination
Reduction of wear on the vehicle
Thrift in fuel resources
Fairness to other vehicles
The thrift in time

These are ordered in a way that makes civic and global sense, but they are not the priorities exhibited by human drivers.

Copy Humans or Reevaluate and Design from Scratch?

Whoever said that the goal of autonomous car design is to model the portions of a human mind that can drive should not be designing autonomous cars for actual manufacture.  It is well known that most humans, although they may have heard of the following safety tips, cannot bring them into consciousness with sufficient speed to benefit from them in actual driving arrangements.

When the tires slip sideways, steer into the skid.
When a forward skid starts, pump the breaks.
If someone is headed tangentially into your car's rear, immediately accelerate and then break.
On an on ramp, accelerate to match the speed of the cars in the lane into which you merge, unless there is no space to merge.
If you see a patch of ice, steer straight and neither accelerate nor decelerate once you reach it.

Many collisions between locomotives and cars are because a red light causes a line in multiple lanes across the tracks.  Frequently, a person will move onto the railroad tracks to gain one car's length on the other cars.  When others move to make undoing that choice problematic, a serious risk emerges.

As absurd as this behavior is to anyone watching, many deaths occur as a fast traveling 2,000 ton locomotive hits what feels like a dust speck to the train passengers.

Predictability and Adaptability

Humans are unpredictable, as the question indicates, but although adaptability may be unpredictable, unpredictability may not be adaptive.  It is adaptability that is needed, and it is needed in five main ways.

Adaptive in the moment to surprises
Adaptive through general driving experience
Adaptive to the specific car
Adaptive to passenger expression
Adaptive to particular map regions

In addition, driving a car is

Highly mechanical,
Visual,
Auditory,
Plan oriented
Geographical, and
Preemptive in surprise situations.

Modelling Driving Complexities

This requires a model or models comprise of several kinds of objects.

Maps
The vehicle
The passenger intentions
Other vehicle
Other obstructions
Pedestrians
Animals
Crossings
Traffic signals
Road signs
Road side

Neither Mystery nor Indeterminance

Although these models are cognitively approximated in the human brain, how well they are modeled and how effective those models are at reaching something close to a reasonable balance of the above priorities varies from driver to driver, and varies from trip to trip for the same driver.

However, as complex as driving is, it is not mysterious.  Each of the above models are easy to consider at a high level in terms of how they interact and what mechanical and probabilistic properties they have.  Detailing these is an enormous task, and making the system work reliably is a significant engineering challenge, in addition to the training question.

Inevitability of Achievement

Regardless of the complexity, because of the economics involved and the fact that it is largely a problem of mechanics, probability, and pattern recognition, it will be done, and it will eventually be done well.

When it is, as unlikely as this sounds to the person who accepts our current culture as permanent, human driving may become illegal in this century in some jurisdictions.  Any traffic analyst can mount heaps of evidence that most humans are ill equipped to drive a machine that weighs a ton at common speeds.  The licensing of unprofessional drivers has only become widely accepted because of public insistence on transportation convenience and comfort and because the workforce economy requires it.

Autonomous cars may reflect the best of human capabilities, but they will likely far surpass them because, although the objects in the model are complex, they are largely predictable, with the notable exception of children playing.  AV technology will use the standard solution for this.  The entire scenario can be brought into slow motion to adapt for children playing simply by slowing way down.  AI components that specifically detect children and dogs are likely to emerge soon, if they do not already exist.

Randomness

Randomness is important in training.  For instance, a race car driver will deliberately create skids of various types to get used to how to control them. In machine learning we see some pseudo random perturbations introduced during training to ensure that the gradient descent process does not get caught in a local minimum but rather is more likely to find a global minimum (optimum).

Deadlock

The question is correct in stating that, "A dose of unpredictability could have its uses."  The deadlock scenario is an interesting one, but is unlikely to occur as standards develop.  When four drivers come to a stop sign at the same time, they really don't.  It only seems like they did.  The likelihood that none of them arrived more than a millisecond before the others is astronomically small.

People will not detect (or even be honest enough) to distinguish these small time differences, so it usually comes to who is most gracious to wave the others on, and there can be some deadlock there too, which can become comical, especially since all of them really wish to get moving.  Autonomous vehicles will extremely rarely encounter a deadlock that is not covered by the rule book the government licensing entity publishes, which can be programmed as driving rules into the system.

On those rare occasions, the vehicles could digitally draw lots, as suggested, which is one place where unpredictability is adaptive.  Doing skid experimentation like a race car driver on Main Street at midnight may be what some drunk teen might do, but that is a form of unpredictability that is not adaptive toward a sensible ordering of the priorities of driving.  Neither would be texting or trying to eat and drive.

Determinism

Regarding determinism, in the context of the uses discussed, pseudo-random number generation of particular distributions will suffice.

Deadlock release or
Training speed-ups and improved reliability when there are local minima that are not the global minimum during optimization,

Functional tests and unit testing technologies are not only able to handle the testing of components with pseudo-randomness, but they sometimes employ pseudo-randomness to provide better testing coverage.  The key to doing this well is understanding of probability and statistics, and some engineers and AI designers understand it well.

Element of Surprise

Where randomness is most important in AV technology is not in the decision making but in the surprises.  That is the bleeding edge of that engineering work today.  How can one drive safely when a completely new scenario appears in the audio or visual channels?  This is perhaps the place where the diversity of human thought may be best adept, but at highway speeds, it is usually too slow to react in the way we see in movie chase scenes.

Correlation Between Risk and Speed

This brings up an interesting interaction of risk factors.  It is assumed that higher speeds are more dangerous, the actual mechanics and probability are not that clear cut.  Low speeds produce temporally longer trips and higher traffic densities.  Some forms of accidents are less likely at higher speeds, specifically ones that are related mostly to either traffic density or happenstance.  Other forms are more likely at higher speeds, specifically ones that are related to reaction time and tire friction.

With autonomous vehicles, tire slippage may be more accurately modeled and reaction time may be orders of magnitude faster, so minimum speed limits may be more imposed and upper limits may increase once we get humans out of the driver's seats.


How can I determine if an input sentence is consistent with a certain subject?

If you have lots of training data, ANN's (deep learning) could quite possibly get you there.  But I have a hunch you might also get some mileage out of using something like a rule induction approach.  Maybe something like CN2. I'd suggest at least reading up on those and see if you can see a way to apply that to your system.


What is the state-space complexity of Spades?

I have calculated an upper bound and modified a calculation of a lower bound for a similar game.  I assume the real size is closer to the upper bound.
$$ 7.36 \cdot 10^{27} \leq |SPADES| \leq 3.09 \cdot 10^{72} $$
Upper bound
The State-space complexity of Spades can be computed by counting for each possible starting state (deal), all the possible sequences of actions.  While that calculation could be hard, an upper bound is easy to achieve by multiplying the number of initial positions $|S_0|$ by an upper bound on the number of different possible histories $|\bar{H}|$.
$$ |SPADES| = \sum_{s \in S_0} |H_{s}| \leq |S_0| \cdot |\bar{H}| $$
The number of possible starting positions $|S_0|$ is the number of different ways to deal a 52 cards deck into 4 players
$$|S_0| = \frac{52!}{13!^4} \approx 5.3\cdot10^{28}$$
An information set in Spades contains two objects: the player's hand, and the sequence of actions each of the players have made from the start of the round.  Before any card is played, each information set (a single hand) can be completed into a full game-state (4 hands) in the following number of ways
$$ |\bar{S}| = \frac{39!}{13!^3} \approx 8.45 \cdot 10^{16} $$
During a round, each player first decide upon a bid, then choose one out of her 13 cards, than choose one out of 12, etc. Thus, the size of the decision tree, which is the number of histories is bounded by
$$|H| \leq {14!}^4 \approx 5.77 \cdot 10^{43}$$
This is an upper bound since some of those histories violates the rules of the game.
Therefore, the number of legal positions in Spades is bounded by
$$ |SPADES| = \sum_{s \in S_0} |H_{s}| \leq |S_0| \cdot |H| \leq \frac{52!}{13!^4} \cdot {14!}^4  \approx 3.09 \cdot 10^{72} $$
Lower bound
In the paper: Understanding the Success of Perfect Information Monte Carlo Sampling in Game Tree Search, Sturtevant et al. describe a loose lower bound for the size of Skat:

Skat is a 3-player card game with 32 cards, from which 10 are dealt
to each player. There are $$ {32 \choose 10} = 364, 512, 240 $$hands each
player can have and
$$ H := {22 \choose10} \cdot {12\choose10} = 42, 678, 636 $$
hands for the other players which is what constitutes an information set at the start of a game. At the beginning of each trick, the trick leader can choose to play any of his remaining cards. Therefore, there are at least
$$ 10!H \approx 1.54 · 10^{14}$$
information sets. But, from an opponents perspective there
are actually 20 or 22 unknown cards that can be lead, so this
is only a loose lower bound on the size of the tree. This
should clearly establish that even when using short decks it
is infeasible to solve even a single game instance of a trick-based
card game.

Using the same logic to produce a loose lower bound on the size of Spades tree produce the following:
$$ H := {39 \choose 13} \cdot {26 \choose 13} = 8.45 \cdot 10^{16} $$
At the start of the round (before bidding has been made) the number of information sets is lower bounded by
$$ 14!H \approx  7.36 \cdot 10^{27}$$


What is eager learning and lazy learning?

What is eager learning or lazy learning?

Eager learning is when a model does all its computation before needing to make a prediction for unseen data. For example, Neural Networks are eager models.
Lazy learning is when a model doesn't require any training, but all of its computation during inference. An example of such a model is k-NN. Eager learning is also known as instance-based learning [1, 2, 3].

How does eager and lazy learning help me build a neural network system? And how can I use it for any target function?

To answer your second question, you can't employ lazy learning to train a neural network, because they are inherently eager models.


Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?

The rhetorical point of the Turing Test is that it places the 'test' for 'humanity' in observable outcomes, instead of in internal components. If you would behave the same in interacting with an AI as you would with a person, how could you know the difference between them?

But that doesn't mean it's reliable, because intelligence has many different components and there are many sorts of intellectual tasks. The Turing Test, in some respects, is about the reaction of people to behavior, which is not at all reliable--remember that many people thought ELIZA, a very simple chatbot, was an excellent listener and got deeply emotionally involved very quickly. It calls to mind the Ikea commercial about throwing out a lamp, where the emotional attachment comes from the human viewer (and the music), rather than from the lamp.

Turing tests for specific economic activities are much more practically interesting--if one can write an AI that replaces an Uber driver, for example, what that will imply is much clearer than if someone can create a conversational chatbot.


How do we classify an unrecognised face in face recognition?

Specifically for face recognition (and other identification algorithms) there are better approaches than using classifiers directly.

Most identity recognition algorithms generate some kind of metric - typically an "embedding" of the original image into an abstract space i.e. a vector of real numbers. The space might be based on real-world biometrics e.g. normalised measurements of eye distance, eyebrow arch etc, which would be trained as a regression algorithm. The problem with this is that it requires a lot of labelled data, and the biometrics are not necessarily good at differentiating between identities. An alternative is to get the neural network to find the best abstract space for identities. You can do this if you have at least two images of each identity, and using triplet loss to train the network - the loss function directly rewards embedding the same identity close and different identity far apart.

Once you have an embedding, you no longer directly classify identities using the neural network. Instead, you base identity on distance between measured embedding and stored embeddings. This requires implementing a search function that looks at known embeddings and sorts by distance.

  how do we ensure the network tells us when it encounters a new person?

Embeddings don't solve this problem directly, but give a useful heuristic - distance in embedding space. Typically a maximum allowed distance is set as a cutoff to consider an image as showing a new identity. This is a hyperparameter of the model. This is an area that triplet loss helps with, since it is trained to make the distance as large as possible between images that show different identities. If it has generalised well during training, then it should ignore differences due to lighting, pose, makeup etc, but still be able to differentiate similar looking people.

As the embedding is approximate, any such system may make mistakes, and needs to be carefully tested. The quality and quantity of training data are important, and it should match images used in production. But that is no different to the pure classifier, which must in addition be re-built and re-trained for every new class added.

Whether to use a more basic classifier or something like triplet loss is a question of scale - if the number of identities that need to be tracked is high, or the rate of change in identities is high, then embeddings trained on triplet loss (or similar) are more practical.


BERT: After pretraining 880000 step, why fine-tune not work?

change
bert_output = bert_model.get_pooled_output()

to
bert_output = tf.reduce_mean(bert_model.get_sequence_output()[:,1:,:],1)


According to Open AI's Greg Brockman, the Gym website never had a big impact and so was never maintained. This is the reason he gives for shutting down the website.

A read only export of the site was archived at https://gym.openai.com/read-only.html and if you attempt to access the old website through the url https://gym.openai.com you will be redirected to the Open AI gym github repository.

For a while a static copy of the Open AI leader-board was maintained on the below url https://scoreboard-site-1764008611.us-west-2.elb.amazonaws.com/ . However as of present the static site is also unreachable.

The Canadian AI startup montreal.ai had offered to maintain the gym website. However this offer was not taken up by open ai.

This issue was discussed in depth on the following threads

https://www.reddit.com/r/MachineLearning/comments/6zvlm2/d_openai_closing_down_gym_toolkit_website/?st=jca6ia3n&sh=48a0d8f3

https://github.com/openai/gym/issues/718


Clarifying representation of Neural Nerwork input for Chess Alpha Zero

For anyone wondering, I believe to have found the answer:

Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress).

There are two repetitions planes (for each position from the most recent T=8 positions):
a) The first repetition plane will be a plane where all the entries are 1's if the position is being repeated for the first time. Else 0's.
b) The second repetition plane will be a plane where all the entries are 1's if the current position is being repeated for the second time. Else 0's.


Is unsupervised disentanglement really impossible?

The impossibility is referring how to learn the disentangled representations from the observed distribution or to know whether you have a disentangled representation in the first place.

Basically, an unsupervised learning agent tasked with learning a disentangled transformation of some features $\mathbf{z}$ needs to infer a set of features from the data which are not entangled, but the supplied data will always have many equally valid entangled solutions - valid from the point of view of describing the distribution of $\mathbf{z}$ accurately.

An analogy would be "I have observed the value 50, and know it is the sum of 3 numbers. What are those numbers?". Whilst the correct answer exists, and it is possible to guess it, it cannot be inferred from the supplied information.

The part of the proof you quote shows that the multiple equivalent entangled feature sets exist, and theoretically cannot be separated from a "true" disentangled feature set on the basis of knowing the distribution. Once you accept this, it is indeed just a short hop in logic to say that the disentangled features are not learnable - there is no way for a learning system to differentiate between the entangled and disentangled features that explain the distribution, and a large (infinite) set of valid entangled features are guaranteed to exist, which will confound attempts to find a perfect solution.

It is worth noting that the impossibility refers to learning perfect solutions, and that the proof does not rule out useful or practical approximate solutions, or solutions that augment unsupervised learning by applying some additional rules or a semi-supervised approach.


Use AI or Neural Network for logo detection

To perform image recognition you have to find a way to represent an image with certain features.

One of the defining characteristics of a good image recognition algorithm are it's ability to detect salient regions, that is, regions which contain the most information

There is a lot of attention on deep learning for content-based image classification at the moment. You can achieve decent results by implementing deep learning having three or more layers of CNN's where each layer is responsible for extracting one or more feature of the image.


What are some academic AI podcast out there?

Also a few others options (that I listen to) include:

Learning Machines 101
Machine Learning Guide
Machine Learning - Software Engineering Daily


DQN not learning and step not stepping towards target

In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save("filename"+".h5") ).
Then, load this saved model file and do a test run to see if it worked. In this case, you should notice pretty easily if it learned or not).


Why is my fine-tuned YOLO model detecting other objects as a human?

So you have a network pretrained on 80 classes. I also assume that one of these classes are human (or else this is just not the way to go*) I suspect that the final layer contains 80 labels, correct? Then you then 'rescale' this layer to 1 label and then train on some data you possess? Then you're basically trying to teach the network that it shouldn't care about the 79 other classes, which is just nonsense I think.

What you could do, and I do not recommend this, but if you feel like you have to use this exact network, you just keep the 80 outputs and only look at the label correspondning to the human.

You shouldn't do this because the network is WAY bigger than it needs to be to only classify human/non human, which will make it slower than it needs to be.

What you rather want to do is either to train your own network (If you have lots of training data, I suspect this wouldn't be be the hardest thing to train) or obtain a CNN that is pretrained on human classification.

(*I've heard rumours that you can do pretty well on retraining a class on a pretrained network. I just don't know if the rumours are true or how to go about it.)


How to properly resume training of deep Q-learning network?

Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.
Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.
From your description, I am assuming that you are monitoring the average reward per training episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.

Is this behavior expected?

I would expect it if you re-set $\epsilon$ to a high value.

If not, how do I properly resume training of a deep q-learning network?

I recommend that you look at the following things:

Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.

Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.

Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.

You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.


How to create a fully connected(matrix) layer with vector input

You can use tf.reshape() method (tensorflow doc) to reshape (2048) dimensional tensor to (100,300). Here's one way to do this:

input1 = tf.reshape(input1, [100,300], name="reshaped_tensor")

If you're not using TensorFlow but using Numpy, here's an implementation:

input1 = np.array(input1)
input1 = np.reshape(input1, (100,300))

Note: You might want to follow up this layer with tf.nn.conv2d layers to "densify" the sparse matrix/values obtained from the above step.


How to process data in a data stream for a LSTM

One thing I might add is not to expect the ML model to take care of everything, that is where ML engineering comes into play. One suggestion I have, without knowing much about your data stream, is that you need to implement a real-time data ingestion/pipeline, like Apache Kafka Stream (so similar implementation), where you can decompose your stream into output that your model was trained for. You could split by dictionary, pattern or whatever you need, then push the split output into another real-time stream that can be ingested by your model. Not a lot to go for, but hopefully it helps a little.


Why does a fully connected layer only accept a fixed input size?

A convolutional layer is a layer where you slide a kernel or filter (which you can think of as a small square matrix of weights, which need to be learned during the learning phase) over the input. In practice, when you need to slide this kernel, you will often need to specify the "padding" (around the input) and "stride" (with which you convolve the kernel on the input), in order to obtain the desired output (size). So, even if you receive inputs of different sizes, you can change these values, like the padding or the stride, in order to produce a valid output (size). In this sense, I think, we can say that convolutional layers accept inputs of (almost) any size.

The number of feature maps does not depend on the kernel or input (sizes). The number of features maps is determined by the number of different kernels that you will use to slide over the input. If you have $K$ different kernels, then you will have $K$ different feature maps. The number of kernels is often a hyper-parameter, so you can change it (as you please).

A fully connected (FC) layer requires a fixed input size by design. The programmer decides the number of input units (or neurons) that the FC layer will have. This hyper-parameter often does not change during the learning phase. So, yes, FC often accept inputs of fixed size (also because they do not adopt techniques like "padding").


What is the best XOR neural network configuration out there in terms of low error?

The initialization of the weights has a big impact on the results. I'm not sure specifically for the XOR gate, but the error can have a local minimum that the network can get "stuck" in during training. Using stochastic gradient descent can help give some randomness that gets the error out of these pits. Also, for the sigmoid function, weights should be initialized so that the input to the activation is close to the part with the highest derivative so that training is better.


Can two planning PDDL actions be taken simultaneously?

First place to look is how the preconditions/effects of different actions interact.


Are Markov Random Fields and Conditional Random Fields still used in computer vision?

In the Image-to-Image Translation with Conditional Adversarial Networks paper (popularly known as pix2pix), they used a Markovian Discriminator to effectively model the image as a Markovian Random Field.
There were some papers in the last 5 years concerning Markov Fields. Here are some of them:-

Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks


What is the difference between a receptive field and a feature map?

I hope this illustration will help you:

Receptive field(s): is a small portion of the input to produce only one node in a feature map.

Feature map(s): is a convolutional process output, a feature map can be said as a feature representation of filter's input. One feature map consists of many filter's outputs (from different receptive fields) from one kernel. The number of feature maps depends on the number of the kernel.

So even feature maps are the next filter's input, but the next receptive fields is not a feature map. The next receptive field consists of a small portion node from different feature maps (not only one feature map).

And also, we can see from the illustration above, a feature map has two-dimensional size $(46 \times 46)$, then a receptive field size will always three-dimensional $(5 \times 5 \times \text{Number Of Feature Maps})$.


What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

This expression: $|\mathcal{A}(s)|$ means

$|\quad|$ the size of

$\mathcal{A}(s)$ the set of actions in state $s$

or more simply the number of actions allowed in the state.
This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.


How do I segment each part of a DICOM image?

I looked at some picture from DICOM, and it seems that these images usually don't have many local minima, so I guess that's why watershed isn't optimal.
However, it seems that in DICOM the color gradients are often strong, so I suggest to use techniques that works well with gradients:

Direct Gradient computations, either convolutional gradients or morphological gradients (selects shape of your different objects)
Normalized Cut on Similarity Graphs

If you want to know more about these you can read this post.


What approach should I use to detect faces in video game footage?

At first, you can find lots of information as pedestrian detection. As you are trying to localize game characters, the face is not the best option. You need to look for the character in general.

About HAAR Cascades, the algorithm is one of the fastest face localization solutions in the market. The reason is, it applies all the feature classifications layer by layer, by starting from the wider feature. So, if it fails, it does not spend the time to check computationally intensive features. It was good until DNN overcome its success rate. However, it is not the best approach for recognizing game characters/pedestrians.

Also, it seems like you are overfitting the cascade so it stops and does not learn anything valuable. You can search methods about how to reduce overfit problem.

In 2005, a new method has been proposed, HOG (Histogram Of Gradients). You can use this one and classify output features to get what you desire. If you would like to go for deep learning version, I would suggest you to investigate how DNN is working, which kind of input images you need, what are the localization networks (i.e. YOLO, Faster R-CNN), how they are working.


What is the intuition behind the Xavier initialization for deep neural networks?

Weight initialization is one of the most critical factors for successfully training a deep neural network. This explanation by deeplearning.ai is probably the best that one could give for the need for initializing a DNN with Xavier initialization. Here is what it talks about in a nutshell:

The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: $1.1^{50} = 117.390$, while at the same time, $0.9^{50} = 0.00515$. Note that the difference between the two numbers is just $0.1$ but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.

The standard weight initialization methods come into the picture here. They reinforce what are the de-facto rules of thumb when it comes to weight initializations: (1) the mean of the activations should be zero, and (2) the variance of these activations across all the layers should be the same.

Note: The link given above has complete mathematical justification for why Xavier initialization works, along with an interactive visualization for the same.


GPFlow: Gaussian Process Uncertainty Quantification

Not sure about which function to call in GPFlow, but the principle is simple. If your $Y$ data was scaled:
$$Y_{scaled} = {(Y - mean(Y)) \over {sd(Y)}}$$
So now if you have any prediction (from the scaled model) $\hat{Y}_{scaled}$, with stand error $\sigma_{\hat {Y}_{scaled}}$, then it is a normal distribution ${\cal N}(\hat {Y}_{scaled}, \sigma^2_{\hat {Y}_{scaled}})$. Then you can very easily rescale it back to the original scale ${\cal N}(\hat {Y}, \sigma^2_{\hat {Y}})$, where:
$$\hat {Y} = \hat {Y}_{scaled}\cdot sd(Y) + mean(Y)$$
$$\sigma_{\hat {Y}} = \sigma_{\hat {Y}_{scaled}} \cdot sd(Y)$$


The learning rate used in SGD and in most convergence strategies used in artificial network designs is much like the use of voltage dividers in electronic feedback. The differential equations for control theory, developed by Norbert Wiener and others, demonstrated that insufficiently attenuated negative feedback will cause a circuit to destabilize.

Back-propagation with gradient descent is conceptually similar. The main difference is that artificial networks use discrete examples or batches of examples rather than continuous signals. Back-propagation is a strategy to distribute negative feedback throughout the layers of an artificial network using the chain rule, but that is also common in electronic control circuitry.

It is hypothesized, and the evidence is accumulating in support, that more sophisticated strategies to achieve stability using feedback exist in the neural networks of the brain and in the gut of animals. That is likely, and part of the reason arrays of activation functions are called artificially neural.

Those that have studied the mathematics field of chaos understand that discrete systems can fall into oscillation with insufficiently attenuated negative feedback too. In a worse case, a chaotic pattern will emerge at what is called a bifurcation point. Those that work with convergence in machine learning are always working against factors that lead to divergence. However, too much attenuation, which corresponds to too low a learning rate will be highly stable and extremely costly in terms of computing resources, time to converge, or both.

All of this was established in theory in the late nineteenth and early twentieth century. Goodfellow is explaining it to university level students in the context of deep learning, but it is not specific to SGD. In fact SGD is less likely to diverge or go into chaotic states than what the question is calling vanilla gradient descent.

From a mathematicians stoic point of view, calling chaos violent is anthropomorphic, placing human qualities on non-human things. They might point out that some forms of chaos are more stable than constants. There are many forms of mental, metabolic, molecular, and biospheric stasis that have what chaoticians call attractors, and never converge yet maintain a statistical adjacency to their attractor in phase space. For example, using the above list, emotion, glucose, electron shells, and populations show this type of non-constant stability.

That does not help the person using a multilayer-perceptron or convolution kernel hoping to get some ideal set of parameters during training to deploy into a working system in the field. What is desired is reliable, relatively fast, and accurate convergence results.

What practitioners try to avoid is a cost function that diverges. Keep in mind that cost, error, and loss functions are terms used in various contexts in various articles but all represent disparity from some ideal. The function is the mathematical model for that ideal or optimum, not to be confused with the use of the term model in other aspects of AI. Models are used in many places. In SGD, increases in the cost function are indications of instability and reduced likelihood that convergence will occur.

This is the interesting part. Since the cost function represents a surface and the curvature of the surface has curvature (otherwise it could be solved with linear algebra technique without a network) it may not be monotonic. In other words, one cannot determine the mismatch of output values and corresponding labels from the result of the function. The information flow is constrained to be unidirectional.

This means that the existence of local minima can interfere with the search for the global minimum. Mounds, ridges, saddle points, mazes of troughs, and circular ridges may separate the current state of network parameters from those that represent the optimal state sought.

Therein lies the caveat. The cost surface has curvature.

The statement in the question, because of this, is not generally true.

  The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient.

It is true only in specific cases identified by the theory. In the general case, the cost is not guaranteed to decrease and the algorithm is not guaranteed to converge unless a number of constraints are maintained, few of which normally are.

If not sufficiently attenuated, the next estimation to radically overshoot or undershoot the global minimum. To see how this works, look into chaos simulations using the logistic function. As the next estimate amplification increases, the system first goes into oscillation and then into modes of chaos. In the case of artificial networks, the values in the algorithm used can also exceed the range of IEEE floats and throw an exception.

If one studies the theoretical guarantees in the literature and some of the overview theory such as the PAC (probably approximately correct) Learning Framework, and then try to find a business, marketing, defense, industrial, robotic, or financial application for AI that meets all the constraints required to benefit from the theoretical guarantees, the challenge will become clear. A few easy problems exist. Most require expertise, further investigation, proofs of concept, architecture and design skills, and significant development effort to produce satisfactory results.

Once the results are published with the appropriate references to theoretical foundations and the various design choices presented with hindsight, it looks much easier than it was.


If you're doing deep learning (which I assume you are, if you say you want to learn "AI"), then Python is a MUST. Virtually all the big frameworks are Python wrappers over a C++ core.

C# has no real deep learning frameworks. There are a couple such as the Microsoft Cognitive Toolkit, but they are on a completely different level from PyTorch or Tensorflow. No serious ML practitioner would do the majority of their research in a framework like that.

For more information, see: Why is Python the most popular language in the AI field?


Can OpenAI simulations be used in real world applications?

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.

The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.

For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation.

Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.

So, for instance, in CartPole environments, the following values are fixed:

Size of time step
Mass of cart
Mass and length of pole to be balanced
Force that cart motor pushes with

There are a couple of approaches you could use to work around this:

Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.
Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident can learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.

You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.


How to train a neural network with few weights and biases held constant?

The strategy is turning b2 into a separate model, initialise b2 the way it should be, and train your network without b2 as usual.
In the middle of the main network, combine the output of b1 layer and b2 network using concatenation function, for example in TensorFlow:
# Axis 0 is the batch dimension, axis 1 for the dimension of value in every sample
tf.concat([b1out,b2out],axis=1)

Example source code (paste to Google Colab to test):
%tensorflow_version 2.x
%reset -f

import tensorflow                   as     tf
from   tensorflow.keras             import *
from   tensorflow.keras.layers      import *
from   tensorflow.keras.activations import *
from   tensorflow.keras.models      import *
from   tensorflow.keras.callbacks   import *

class MyModel(Model):
    def __init__(self):
        super(MyModel,self).__init__()
        self.dense1 = Dense(200, activation=relu)
        self.dense2 = Dense(1,   activation=tf.identity)

    @tf.function
    def call(self,x):
        # MIND THE CONCATENATION IN THIS PART:
        h1a = self.dense1(x)
        h1b = b2(x) # b2 won't get trained, so it stays fixed

        # CONCATENATION AT AXIS 1, COZ AXIS 0 IS BATCH DIMENSION
        h1  = tf.concat([h1a,h1b],axis=1)
        u   = self.dense2(h1)
        return u

class ModelB2(Model):
    def __init__(self):
        super(ModelB2,self).__init__()
        self.dense1 = Dense(200)
        # self.dense2 = ...
        # Init weights of B here or B is pre-trained model

    @tf.function
    def call(self,x):
        u = self.dense1(x)
        return u

# PROGRAMME ENTRY POINT ========================================================
if __name__=="__main__":
    inp = [[1,2,3,4,5,6,7],[7,6,5,4,3,2,1]] # Example values
    exp = [[0],            [1]            ]

    mm = MyModel()
    b2 = ModelB2()

    mm.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    b2.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here

    print("\nbefore training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"...")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss)

    mm.fit(x=inp,y=exp, batch_size=len(inp), epochs=500, verbose=0)
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1)
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1)

    print("\nafter training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"... <-- UNCHANGED AS WANTED")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss,"<-- UNCHANGED AS WANTED")
# EOF

Google colab: https://colab.research.google.com


Genetic Algorithms: Trade-off between time and variance with regards to fitness function

You can probably get away with a relatively low X for two reasons:

The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of the number of games played.
In a GA, you don't need an absolute ranking of individuals, because your selection mechanism (see "related articles" here) typically isn't completely elitist. For example, if individuals in the top half of the population are allowed to breed, then your fitness function really just needs to separate the good from the bad reasonably well. It need not be perfect to work.

The correct value of X will still depend on the variance in the scores an agent might receive from playing the game, but this is easy to track. A good approach might be to incorporate this directly into your estimate. Compute the variance, and then prefer agents that not only score highly, but do so with low variance.


What is the physics engine used by DeepMimic?

Bullet physics engine

Their paper says

Physics simulation is performed at 1.2kHz using the Bullet physics engine [Bullet 2015].


Does the observation function for POMDP always add up to 1?

$O(a, s', z) = \mathbb{P}(z \mid a, s')$ is a conditional probability distribution, so it always needs to sum up to $1$. You should interpret $O(a, s', z)$ as the probability of observation $z$, given that the agent took action $a$ and landed in state $s'$.

$O(a, s', z)$ is thus not a joint distribution, even though the notation $O(a, s', z)$ might suggest it. In this case, $O(a, s', z)$ simply means that $O$ is a function of $a$, $z$ and $s'$.

If you want to see a proof that conditional probability distributions sum up to 1, have a look at this post.


Is Nassim Taleb right about AI not being able to accurately predict certain types of distributions?

Yes and no!

There's no inherent reason that machine learning systems can't deal with extreme events. As a simple version, you can learn the parameters of a Weibull distribution, or another extreme value model, from data.

The bigger issue is with known-unknowns vs. unknown-unknowns. If you know that rare events are possible (as with, say, earthquake prediction), you can incorporate that knowledge into the models you develop, and you'll get something that works as well or better than humans in that domain. If you don't know that rare events are possible (as with, say, a stock market crash produced by correlated housing defaults), then your model will reflect that as well.

I tend to think Taleb is being a bit unfair here: AI can't handle these kinds of events precisely because its creators (us) can't handle them! If we knew they were possible, then we could handle them pretty well, and AI could too.


How to calculate the output of this neural network?

The neural Network in the image is a "Recurrent Neural Network"(RNN). Because of the connection leading backward from h10 to h01, h10 has to be a "memory node" (mn), meaning it can store its value from the previous input. The basic functionality of an RNN can be seen in this animation:

Your example:
In the beginning, the storage of the mn is initialized with a value, probably 0.
Now the first input is fed into network:

i0 = 2
i1 = 3
h00 = (i0 * 0.4) = 0.8
h01 = (i1 * -0.9) + ("the stored value of h10" * 1.2) = -2.7
("the stored value of h10" in the first run is 0.)
h10 = (h00 * 0.85) + (h01 * -0.2) = 1.22
out = (h10 * 0.3) + (h01 * 0.1) =  0.096

Now you can feed the next input through the network and use -2.7 as "the stored value of h10" and so on. You can also add an activation function as you would for any other NN.


Is DDPG just for deterministic environments?

I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean  deterministic environment.

Stochastic policy: Probabilistic(random) action choice for a given state.
Deterministic policy: one action is chosen for a given state.

Deterministic Policy Gradient algorithm still can handle a stochastic (and continuous, of cause) environment, but the policy will be deterministic.

Reference

"in DDPG, the Actor directly maps states to actions (the output of the network directly the output) instead of outputting the probability distribution across a discrete action space"  -towards Data Science

Deterministic Policy Gradient Algorithms by Silver et al. PDF


Understanding the reconstruction loss in the paper "Anomaly Detection using Deep Learning based Image Completion"

$F$ in this context is the output of the Convolutional Neural Network that's being trained, which is of the same size as $X$.


If you mean to binarize the vector such that all positive values become 1 and the rest becoming 0, then you can do this.

bin_arr = np.zeros_like(FV_R)
bin_arr[FV_R > 0] = 1.

As an example,

In [7]: arr
Out[7]:
array([[ 0.15, -0.52],
       [ 1.  , -0.43]])

In [8]: bin_arr = np.zeros_like(arr)

In [9]: bin_arr[arr > 0] = 1

In [10]: bin_arr
Out[10]:
array([[1., 0.],
       [1., 0.]])
```


Why does the classifier network in RPN output two scores?

I just want to provide this intuition

this NN consists of a 2 steps detection pipeline (the region proposal and regression + classification in parallel) exploring a certain range of scales and aspect ratios for proposals
as the proposed region is rectangular and the objects of interests have not a rectangular appearance strictly speaking, but a mostly rectangular one (with different ARs), considering the resulting detection bboxes areas some pixels will belong to a certain relevant object appearance (e.g. pixels of a pedestrian, car, …) while other pixels in that BBox won’t be related to that semantic
as this is not semantic segmentation we can now know what pixel is actually relevant, however the objectness and backgroundness can provide a rough measure of this in an aggregated way

Let’s assume at some point in your processing pipeline, you have 2 regions containing the same car but at different scales: so ideally both of them should be associated with good confidence to CarID but it’s clear one will contain much more background pixels than the other one and you'd like this to be represented as an additional measure


Is it possible to spot photoshoped or edited photos using AI?

It is possible! Here is an article by Adobe where they explain how they do it: https://theblog.adobe.com/spotting-image-manipulation-ai/

The algorithm for this would almost certainly be a Convolutional Neural Net trained on a dataset of real and manipulated images (labeled as such).


How can I do video classification while taking into account the temporal dependencies of the frames?

Look at spatio-temporal CNNs which extend the image-based CNN in 2D to 3D to handle time.  These are commonly used to detect or classify action in a video.  People have used them to identify specific actions in various sports such as kicking a soccer ball, throwing a baseball or dribbling a basketball.  They have been used to identify fire, smoke, deep fakes, and violence in videos.  Below are some articles to help get started.

Introduction to Video Classification, by Connor Shorten
Are you ready for a Video Classification Challenge? Preparation Guide for Video Classification, by Neha Goel
MiCT-Net for Human Action Recognition in Videos, by Florent Mahoudeau

Source code:

Video Classification
Video Classification Using 3D ResNet


Why do we need learning in unsupervised learning?

Imagine you have a dataset of people who have cancer.
You have information about their age, physique, diagnosis, treatments, and results.

Using this data, you want to prescribe a set of treatments for a new patient, P.

Obviously, if there is someone in the dataset that has very similar traits as P and had a positive result with their treatments, you could prescribe the same set of treatments. However, this is incredibly unlikely and becomes more infeasible as more information about P is observed (e.g. Has brown hair and hates pasta).

A better option is to cluster the dataset into groups that have positive outcomes for treatment results. For example, perhaps patients with lung cancer who smoke and are given treatment A do better than patients with lung cancer who didn't smoke and are given the same treatment A. These patients should then be divided based on this outcome.

Once these different clusters are found, patient P can be evaluated against each of the clusters and a set of treatments can be prescribed (e.g. Most of the treatments from cluster A, but 1 treatment from cluster B).

Unsupervised learning is the method of finding these clusters, which helps find structure to the data to better answer questions.


Detecting Keypoint of 3D model, and distance between them

It depends on the format of your 3D model dataset , if your dataset is made of cad models you could voxelize your dataset and train a convolutional neural network on it , but it could be very time consuming to train 3D convnets . instead you could use a multi view 2D CNN https://arxiv.org/abs/1505.00880?context=cs


TensorFlow 2.0 - Normalizing input to DNN (on structured data)

One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.


If you're looking for an existing solution, the best approach I found was using a TF-IDF model, check out the links below which have similar examples which should be easily adapted for your dataset.

https://www.kaggle.com/selener/multi-class-text-classification-tfidf#targetText=Text%20classification%20(multiclass)&targetText=With%20the%20aim%20to%20classify,one%20of%20the%20product%20categories).

https://github.com/susanli2016/Machine-Learning-with-Python/blob/master/Consumer_complaints.ipynb

However if you specifically want to go for a DNN approach, there are a few options you can take for a multi-class text classification. Try looking into a simple CNN classifier, which is a relatively lightweight approach, computationally speaking, yet showing pretty good results:

https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f

Alternatively, you can use a word2vec or a doc2vec model to map your sentences to unique vectors, and then put them through a regression algorithm.

https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568


Is there any computer vision technology that can detect any type of object?

TL;DR

This is possible. You need a correctly labeled dataset. Your dataset has two labels:

$y\in \{\text{background},\text{object in frame}\}$ or simply $y\in \{0,1\}$

This labelling avoids needing to know what object is in frame only that there is an object in frame.

Examples With Similar Objectives

Here as a paper (link) that was seeking to classify animals in a frame. The first part of their pipeline needed to know whether there was an animal in frame or not. Excerpt:

  Task I: Detecting Images That Contain Animals. For this task, our models take an image as input and output two probabilities describing whether the image has an animal or not (i.e. binary classification)

Here is another paper (link) regarding detecting animals in images. Excerpt:

  Wildlife detection, which is actually a binary classiﬁer capable of clasifying input images into two classes: “animal” or “no animal”

The methods used in these papers could be adapted to fit the conveyor belt use case.

Here is a system (link) that detects moving objects using background extraction. The method they use to check whether or not the image is a background image or not could be adapted to your use case.

Ideas for Implementation

In the conveyor belt use case, a very simple model might work. Thus, it is recommended that you try a simple model first. For example, you could flatten your inputs to a vector and feed these into a shallow feed forward network. A simple logistic regression classifier might even work on flattened inputs for this use case.

If your desired performance is not reached with a simple model you can analyze their performance using methods like learning curves to decide how to proceed.

If you find that your model has high bias:

You can either use a larger/deeper model like a custom tuned CNN or you can use a pre-trained network like VGG16 and  use transfer learning to achieve your goal.

Here is a post that describes using a CNN to classify cats vs dogs (link). Simply replace the cat/dog dataset with your object/no-object dataset. The post even describes using transfer learning with VGG16.

If you find that your model has high variance

More data can help. Furthermore, in the conveyor belt example, hand labeling data would be easy and quick. Thus, it would not take very much time to get many more examples.

I hope this helps.


Why is description logic decidable but first order logic is not decidable?

This has to do with the fact that you can define arithmetic inside the axiomatic system or not. In description logic you cannot speak about arithmetic sentences and in first order logic you do.

if you look at the proof of incompleteness you will understand this in depth. This demonstration depends on an arithmetic coding of statements, and this representation is fundamental to obtain the conclusion.