What are possible functions assigned on decision nodes for decision tree prediction?

For a binary split, there are only three possible operations (or arguably only two if you consider one-hot encoding). Any other kind of split would simply not be binary. Almost every tree-based model is restricted to binary splits, due to the combinatorial explosion when considering ternary or even more complex splits.

Of course you could write your own algorithm that uses recursive non-binary splits. Beware though that you'll be facing the same difficulty that has led the vast majority of algorithms to be limited to strictly binary splits.

Have a look at this related question.


Is there an LSTM-based unsupervised learning algorithm to label a dataset of curves?

You can try to work with Gated Recurrent Units or GRU. This will solve your problem of with too much latency time that LSTM required. LSTM also doesn't give preference on newer data too. For more information, you can follow more on this great article

GRU Cells


First of all, efficiency and convergence are two different things. There's also the rate of convergence, so an algorithm may converge faster than another, so, in this sense, it may be more efficient. I will focus on the proof that policy evaluation (PE) converges. If you want to know about its efficiency, maybe ask another question, but the proof below also tells you about the rate of convergence of PE.

What is the proof that policy evaluation converges to the optimal solution?

To provide some context, I will briefly describe policy evaluation and what you need to know to understand the proof.
Policy evaluation
Policy evaluation (PE) is an iterative numerical algorithm to find the value function $v^\pi$ for a given (and arbitrary) policy $\pi$. This problem is often called the prediction problem (i.e. you want to predict the rewards you will get if you behave in a certain way).
Two versions: synchronous and asynchronous
There are (at least) two versions of policy evaluation: a synchronous one and an asynchronous one.
In the synchronous version (SPE), you maintain two arrays for the values of the states: one array holds the current values of the states and the other array will contain the next values of the states, so two arrays are used in order to be able to update the value of each state at the same time.
In the asynchronous version (APE), you update the value of each state in place. So, first, you update the value of e.g. $s_1$, then $s_2$, etc., by changing your only array of values (so you do not require a second array).
SPE is similar in style to the numerical method called Jacobi method, which is a general iterative method for finding a solution to a system of linear equations (which is exactly what PE is actually doing, and this is also explained in the cited book by Sutton and Barto). Similarly, APE is similar in style to the Gauss–Seidel method, which is another method to solve a system of linear equations.
Both of these general numerical methods to solve a system of linear equations are studied in detail in Parallel and Distributed Computation Numerical Methods (1989) by Bertsekas and Tsitsiklis, which I haven't read yet, but provides convergence results for these numerical methods.
The book Reinforcement learning: an introduction by Sutton and Barto provides a more detailed description of policy evaluation (PE).
Proof of convergence
I will provide a proof for the SPE based on these slides by Tom Mitchell. Before proceeding, I suggest you read the following question  What is the Bellman operator in reinforcement learning? and its answer, and you should also get familiar with vector spaces, norms, fixed points and maybe contraction mappings.
The proof that PE finds a unique fixed-point is based on the contraction mapping theorem and on the concept of $\gamma$-contractions, so let me first recall these definitions.

Definition ($\gamma$-contraction): An operator on a normed vector space $\mathcal{X}$ is a $\gamma$-contraction, for $0 < \gamma < 1$, provided for all  $x, y \in \mathcal{X}$
$$\| F(x) - F(y) \| \leq \gamma \| x - y\|$$
Contraction mapping theorem: For a $\gamma$-contraction $F$ in a complete normed vector space $\mathcal{X}$

Iterative application of $F$ converges to a unique fixed point in $\mathcal{X}$ independently of the starting point

at a linear convergence rate determined by $\gamma$

Now, consider the vector space $\mathcal{V}$ over state-value functions $v$ (i.e. $v \in \mathcal{V})$. So, each point in this space fully specifies a value function $v : \mathcal{S} \rightarrow \mathbb{R}$ (where $\mathcal{S}$ is the state space of the MDP).
Theorem (convergence of PE): The Bellman operator is a $\gamma$-contraction operator, so an iterative application of it converges to a unique fixed-point in $\mathcal{V}$. Given that PE is an iterative application of the Bellman operator (see What is the Bellman operator in reinforcement learning?), PE finds this unique fixed-point solution.
So, we just need to show that the Bellman operator is a $\gamma$-contraction operator in order to show that PE finds this unique fixed-point solution.
Proof
We will measure the distance between state-value functions $u$ and $v$ by the $\infty$-norm, i.e. the largest difference between state values:
$$\|u - v\|_{\infty} = \operatorname{max}_{s \in \mathcal{S}} |u(s) - v(s)|$$
Definition (Bellman operator): We define the Bellman expectation operator as
$$F^\pi(v) = \mathbf{r}^\pi + \gamma \mathbf{T}^\pi v$$
where $v \in \mathcal{V}$, $\mathbf{r}^\pi$ is an $|\mathcal{S}|$-dimensional vector whose $j$th entry gives $\mathbb{E} \left[ r \mid s_j, a=\pi(s_j) \right]$ and $\mathbf{T}^\pi$ is an $|\mathcal{S}| \times |\mathcal{S}|$ matrix whose $(j, k)$ entry gives $\mathbb{P}(s_k \mid s_j, a=\pi(s_j))$.
Now, let's measure the distance (with the $\infty$-norm defined above) between any two value functions $u \in \mathcal{V}$ and $v \in \mathcal{V}$ after the application of the Bellman operator $F^\pi$
\begin{align}
\| F^\pi(u) - F^\pi(v) \|_{\infty}
&= \| (\mathbf{r}^\pi + \gamma \mathbf{T}^\pi u) - (\mathbf{r}^\pi + \gamma \mathbf{T}^\pi v)\|_{\infty} \\
&= \| \gamma \mathbf{T}^\pi (u - v)\|_{\infty} \\
&\leq \| \gamma \mathbf{T}^\pi ( \mathbb{1} \cdot \| u - v \|_{\infty})\|_{\infty} \\
&\leq \| \gamma (\mathbf{T}^\pi \mathbb{1}) \cdot \| u - v \|_{\infty}\|_{\infty} \\
&\leq \gamma \| u - v \|_{\infty}
\end{align}
where $\mathbb{1} = [1, \dots, 1]^T$. Note that $\mathbf{T}^\pi \cdot \mathbb{1} = \mathbb{1}$ because $\mathbf{T}^\pi$ is a stochastic matrix.
By the Bellman expectation equation (see Barto and Sutton's book and What is the Bellman operator in reinforcement learning?), $v^\pi$ is a fixed-point of the Bellman operator $F^\pi$. Given the contraction mapping theorem, the iterative application of $F^\pi$ produces a unique solution, so $v^\pi$ must be this unique solution, i.e. SPE finds $v^\pi$.
Notes
I didn't prove the contraction mapping theorem, but you can find more info about the theorem and its proof in the related Wikipedia article.


Are there any better visual models for transfer rather than ImageNet?

Why is ImageNet so popular for transfer learning?

Models pre-trained on the ImageNet datasets have been the de-facto choice for many years now. Many popular reasons as to why people think that ImageNet is so effective for transfer learning are the following:

ImageNet is a truly large-scale dataset that contains over 1 million images, each of which has a decent resolution.
ImageNet has a wide and diverse set of classes (1000 in total) ranging from animals to humans, cars, trees, etc. Most Computer Vision tasks operate in similar domains, however there are some notable exceptions (e.g. medical images).
For example, an object detection model for autonomous driving would benefit from ImageNet transfer learning, as the pre-trained model has seen images with similar content (e.g. roads, people, cars, street signs), even though it tries to solve a different task (i.e. object detection not classification).
The above two reasons allow models trained on ImageNet to identify and extract very generic features, especially in their initial layers, that can be effectively re-used.
ImageNet has a lot of similar classes. This is an interesting argument because it contradicts the second one. Due to the closeness of some classes (e.g. multiple breeds of cats), networks learn to extract more fine-grained features.

Another overlooked reason I find very important is that:

ImageNet has been the benchmark for performance for image classifiers for years now. When, for example, you are selecting a re-trained ResNet to use, you know that that model is guaranteed to operate at a high level of performance. Other datasets don't have such notable challenges as the ILVRC. That challenge is what make the VGG and ResNet popular in the first case, so it comes natural that people would want to use those weights.

In practice, due to the way in which CNNs identify and extract features from images, they can easily be "transferred" from task to task.

Is it actually better than other datasets?

This question was widely explored by Huh et al., who tried to identify the reasons that made the ImageNet dataset better than other ones for transfer learning.

In short they found out that most of the reasons that people thought made ImageNet so good (i.e. the ones I mentioned above) weren't necessarily correct. Furthermore, the amount and diversity of images and classes required to effectively train a CNN has been highly overestimated. So there is no particular reason people should choose this specific dataset.

Now, to answer your questions:

  I was wondering if such a thrust exists in Computer Vision?

No, ImageNet is currently established as the de-facto choice, evident by the fact that all 10 keras.applications models offer weights only for ImageNet.

  But relative to the data we have access too, shouldn't there exist something stronger?

This is an interesting question, as the consensus things that deep learning models keep getting better with more data. There is, however, evidence that indicates otherwise (i.e. that CNN models con't have as much capacity as we thought). You can read the aforementioned study for more details. In any case, this is still an open research question.

Even if models could get better, though, with more data, it is possible that it still wouldn't matter because ImageNet pre-trained models are strong enough.

  classification as a sole task has its own constrictions on domain transfer

There have been numerous cases where models initialized from pre-trained ImageNet weights have done well in settings other than classification (e.g. regression, object detection). I'd argue that initialization from ImageNet is almost always better than random initialization.

  Are there any better visual models for transfer rather than ImageNet successes? If no, why? Is it because of the domains fluidity in shape, resolution, etc., in comparison to text?

Partly, yes. I think that in comparison to text, images have some useful properties that are exploited through CNNs, which makes their knowledge more transferable. This claim, however, is based on intuition; I can't back this up somehow.


Applications of AI for creatives and artists

The machine learning (~AI) is all about code and data. If you planning to create something really unique, you'll probably need to, well, work with code and data.

You'll probably should look at github and try to get inspiration from opensourced projects. If you found some AI-related artwork, try to google it's name with 'github'

Another approach would be to look at medium, it's usually have some post/tutorial.

The art is very broad topic and it's hard to cover it all. I try to provide examples of different resources that could help you to start.

Google Research made some examples, where you run code in colab without installing anything locally. It helps you to try basic stuff.
There is tensorflow.js, that allows to launch code at browser. You can look at demos with code
There is intro course for visual artist on kadenze.
Here it is some more links on different courses.
DCGAN - very popular image generation library.
TouchDesigner, very popular tool among visual artists, provides python interface. For example, here is usage of aforementioned DCGAN with it

Also, there is creative section on NIPS(one of the biggest ML conferences) http://nips4creativity.com/. Some of the work provides detailed explanation, like this one

I know, starting to code could be intimidating, but many github repos provides some description about how to launch them and you don't even have to know how they work exactly to use them. Also, it would help to have a person with programming experience, who could help you to deal with code.


Is there any way to classify Document Image without OCR?

It is possible to classify invoice scans without passing through an OCR component if they are visually different (they demonstrate different visual features). On the other hand, if the invoices look very similar, then the classifier might not be very accurate.

Another challenge would be the number of images you need to train a deep network for image classification (you may start with pretrained models and only perform the finetuning of you do not have enough images). On the other hand, the combination of pretrained OCR models and NLP-based document classifier may not need that many samples for training (for this specific task).


Why teaching only search algorithms in a short introductory AI course?

There is lots of misconceptions about AI, specifically the idea that it is about making computers "think" like humans, simulating brain, the sci-fi robots taking over the world, all the philosophical discussions around brain as machine etc. The practice/reality of AI is about "using computing to solve problems" which basically means you take any problem, represent it as a computing problem and then design the algorithm to solve the computing problem which lead to solving the original problem. These search algorithms are general purpose algorithms for general purpose computing problems i.e any real world problem can be represented by these general purpose computing problem and then these algorithms can be used to solve them.

Remember, its about problem solving and its about general purpose computing problems that can represent any real world problem.


Why is a Nvidia Single Board computer less than ideal for AI model training?

Jetson Nano specs Price:$99:

  128-core Maxwell GPU

  Quad-core ARM A57 @ 1.43 GHz

  472 GFLOPS max
   compute

Titan V(for example) Price:$2500 :

  640 tensor cores

  5120 cuda cores

  110000 GFLOPS max compute

Some simple maths gets us to a speed of 11,800 GFLOPS for 25 of these perfectly synced in parallel(not likely or gonna be fun). Vs the 110000 it just isn't even a question.


In a MCTS (Monte Carlo tree search) the resulting tree is expanded by two policies:

tree-policy (selects next and unknown child nodes by a exploration-exploitation tradeoff)
default policy (selects known child nodes till terminal state e.g. with random uniform moves).

The term terminal state is often considered to be the end a tree. If you consider designing an agent with MCTS that is able to play chess, then the terminal state would a win, lose or draw. One game like this is called a playout (sometimes also roll-out). The main concept of MCTS relies on repeating this playouts many times and updating the tree according to the scores. The tree is built up in an incremental and asymmetric manner.

Imagine designing a chess-playing agent. The tree, or rather how the tree is built up can be like this:

Root Node: Starting with an empty board
Child Nodes: Board configuration after each move (opponent and agent)
Selection of Child Nodes: If the opponent is a human, then the selection of child nodes is performed by his/her own strategy. The agent is selecting moves according to a policy.
Terminal State: Score of the game (win/lose/draw).

So after each move (opponent or agent) the current child node/root node is expanded by the resulting child nodes.
If you want to dive deeper into MCTS, then I would recommend to take a look at A Survey of Monte Carlo Tree Search Methods.


What is the best way to integrate unchangeable ethics into a chatbot

Basic Fairness
If by ethics the question means that the system design is to foster basic ethics in the context of a conversation, then at least three attributes can be designed into the system.

Egalitarianism — no favor or disregard on the basis of gender, age, beliefs, financial status, ancestry, or culture, even if those features of the conversant are known and useful to know in personalizing the conversation
Temporal fairness — ensuring that the interaction with new correspondents and veteran correspondents provide these with equal access to relevant information, even if the establishment of trust is an evidence based process controlled by the system (recommended)
Sharing — balancing the objectives of the system to serve the interests of others as much as the interests of the system and its stakeholders — The Golden Rule: Treat others as you wish to be treated.

These attributes can be encoded in fuzzy rules, incorporated into the value function of an ongoing learning algorithm, or expressed in the loss function used during training, directing what is learned by a network.  All of these strategies require the tracking of metrics and the uses of them in adjusting behavior, a kind of social awareness.
There are obvious ways to do this in the mathematical expressions of values, assets, advantages, affirmations, attentions, losses, errors, pains, or other learning feedback approaches are assembled.  In generative network topologies, a discriminative network would need to be able to recognize unfairness and selfishness using the above principles.  Such places specific requirements on the designs of metrics and how fairness and the balance of interests are quantified.
A sum of squares across the dimensions (features) of potential inequity that can develop during a conversation is a possibility.  Using the fourth power instead of squaring would imbue permissiveness for smaller inequities but a drastic aversion to greater inequities.
$$\ell = \sum {(b_i - b_j)}^4 + \dots$$
The variable $\ell$ is the loss function result and $b$ is the benefit to conversation participants $i$ and $j$ where the two might be a human and the chatbot, a human and a stakeholder of the chatbot production server, or two humans that are in separate conversations with the chatbot, or two humans that are in the same conversation with the chatbot.  Note the ellipsis indicates more terms that affect loss as needed.
Cognitive Ethical Sophistication
If by ethics the question means the kind of ethics passed from families that value honor and truth to their children or taught at Harvard Law, the system design may have to wait until cognitive abilities are further developed in computing machinery.  Otherwise the question author will have to further AI to achieve the ethical sophistication.
Research into semantic acquisition, abstraction, automated assembly of causal models, application of these abstractions and models in planning and execution, and the integration of these with natural linguistics is ongoing.  Broad cognition and What some might call wisdom escapes the products of AI as of this writing.
A sufficiently deep network may approximate dialog that demonstrates what appears to be ethical awareness if trained with data that represents stories and conversational actions deemed to be ethical in those stories.  However, compacting learning the layers of comprehension that humans acquire through books, movies, other media, family life, and community events over a few decades (if they acquire it at all) into a computing project may be an overoptimistic objective at this time.
Although a collection of classic works might demonstrate much of a high ethical standard, it is not labeled data.  One would need to identify which character was transformed in a way that demonstrates the valuable ethical standards and then configure the learning process to learn from the character arcs of ethical characters in the stories.
Inherent Feedback

It has no feedback loops of any kind as of right now, though reinforcement learning with a loop might be helpful.

There is no way to adapt without feedback.  If the system is adapting, some feedback mechanism is correcting conditions that are maladaptive, even if the mechanism is opaque to the observer or identified by some other name.  In the case of GANs, the feedback creates a balance through the discriminative and generative network components.
$$\Bigg(\Big((G, E_1) \Rightarrow D\Big) \Rightarrow G, E_2\Bigg) \Rightarrow D \dots$$
That does not exclude the possibility that additional feedback paths may provide additional design advantages.
Hard Coding Ethics
Can judicious behavior be hard coded?

The idea is that there would be some sort of unchanging knowledge base for the chatbot to use that has hard-coded ethical statements and values that the bot has no ability to change.
Will this even be effective at allowing it to generate its own responses yet still be confined to the ethical standards given it?

It is not unchanging knowledge in the domain of ethics that leads to justice, but unchanging principles that guide the detection of what is not ethical that guides a continuous improvement of ethical standards.  That is why even constitutions have rules for amendment.  In probabilistic terms, without the balance of doubt and certainty, no ethical growth can occur.
No one would read a book or watch a movie where the protagonist has no character arc because it is unremarkable.  When a story is remarkable it is because ethics are adaptive and the adaptation must be a new kind that is not already covered many times in previous books or movies.  People like to be surprised by these adaptations.  They must occur against all fears and opposing external forces that are trapping the protagonist in a prison of mediocrity.
The story climax is when the protagonist, against all the building forces that keep them unremarkable, takes a path that discards all previous planning and forges a new radically different but fundamentally better one.
Excellent Objective in the Question
The intention to develop a system that is ethical is honorable and ethical in itself, so the question author may imbue some of those qualities into the system being designed.  In my opinion, such would be a triumph of greater value to the field of AI than all other progress up until now.
To make such an advancement, the framing of the problem will require thinking in new directions.  Consider the stated concern.

It may begin to ignore these "facts" over time, and they may become irrelevant.

Two principles may prove elucidating in regard to this concern.

Every language in common use has the equivalent of the English words should and ought, indicating a universal sense that something is awry.  This detection and identification of imbalance is central to the definition of heroism and honor, which is making at least an attempt to re-balance it.  Even the Oracle in the Matrix, after creating imbalance, sits on the park bench talking to Sati in the warmth of a new balance.  Identifying what needs disruption and re-balancing and constructing a strategy to do so is the one thing in common across all ethical behavior.  Ethics is remedial.  Some ethical choices require unlearning, and that is the most timeless feature of ethics.
Some of the most ancient principles, such as the Golden Rule or the idea of transparency – that right thought and right action are inescapably coupled – are more pertinent to this project than anything one can find in a book about machine learning.  The computer cannot hide a malicious intention and then converse ethically.  The objectives of the stakeholders of the system will need to also be ethical.

First Decision in Approach
The decision to make is the level of abstraction to apply.  Should this chatbot be ethical or be ethically minded?
The first would require the imbuing of ethical character into the bot.  The second would require the imbuing of the acquisition of ethical character into the bot.  As difficult as the more abstract second choice may seem, the first may be unachievable.  We don't have an example of it.  We do, however, have one proof of concept for the second choice: Good people.
The other sub-question to address is regarding selection of AI components.

So which would likely be better [generative or reinforcement]? Or perhaps I should try a combination of the two?

If you are serious about this objective beyond the simple tracking of basic fairness metrics, then you may have to use elements of both and possibly new things that extend the list of AI components and principles to use.
Take One Step Back
One thing to understand about generative topologies is that they are not really adversarial, even if that word is in the name of a seminal paper.  They are usually highly symbiotic and involve equilibria.  What would the discriminative network be without the generative one?  What would the generative one generate without the discriminative one?
The two collaborate to create a feedback system exhibiting a balanced equilibrium.  This is like biological stasis, which is the genius of it.  It is also like chemistry.  Salt in saturated suspension in water involves both dissolving and crystallization in a constant expression of opposing reactions at a molecular level.  Similarly, all ethical decisions are decisions are made within the context of at least one equilibrium, and the details matter.
Every court case where a judge or jury is to apply the public perception of ethics to the case has two sides presented.  They may appear to be adversaries, but the entire legal process is a sophisticated and symbiotic design to force collaboration in the interest of justice.
Every educated journalist understands the importance of avoiding sensational reporting.  Dramatically presenting only one side of the story sells news and is a constant temptation.  Journalistic integrity demands striking an equilibrium.  The result of the ethical choice determines whether the news item educates the public or adds polarization to a culture that may already be partially crippled by extremism.
Take Another Step Back
Reinforcement is an odd name to give what is essentially Agile planning via the probabilistic projection of expected advantages.  Should lean methods for strategic approaches to conversation be in this ethical chatbot?  If ethical behavior is defined to include telling stories that convey wisdom, then yes.  Roger Schank's work is of great interest in that case.  His work in story based reasoning may be a worthy direction of study.
Adapting the foundations of ethics to the ever changing story inherent in every significant conversation may require planning.  It is possibly unavoidably inherent every conversation when the intention of the participants is to do more than exchange pleasantries.


Can a neural network be used to predict a sequence of integers based on dataset of previously produced random numbers?

The post you linked to clearly states that pseudo random number cannot be predicted. Their randomness is made to be nearly perfect, and if you ever found a way to even predict a pseudo random number with 20% chance of correct, the security of the entire world would be vulnerable to attacks, as things ranges from cryptocurrency and secure data transfer is all protected by pseudo random number.


How do I compute the structural similarity between sentences?

Firstly, before we commence I recommend that you refer to similar questions on the network such as https://datascience.stackexchange.com/questions/25053/best-practical-algorithm-for-sentence-similarity and https://stackoverflow.com/questions/62328/is-there-an-algorithm-that-tells-the-semantic-similarity-of-two-phrases

To determine the similarity of sentences we need to consider what kind of data we have. For example if you had a labelled dataset i.e. similar sentences and disimilar sentences then a straight forward approach could have been to use a supervised algorithm to classify the  sentences.

An approach that could determine sentence structural similarity would be to average the word vectors generated by word embedding algorithms i.e word2vec. These algorithms create a vector for each word and the cosine similarity among them represents semantic similarity among words. (Daniel L 2017)

Using word vectors we can use the following metrics to determine the similarity of words.

Cosine distance between word embeddings of the words
Euclidean distance between word embeddings of the words

Cosine similarity is a measure of the similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. The cosine angle is the measure of overlap between the sentences in terms of their content.

The Euclidean distance  between two word vectors provides an effective method for measuring the linguistic or semantic similarity of the corresponding words. (Frank D 2015)

Alternatively you could calculate the eigenvector of the sentences to determine sentence similarity.

Eigenvectors are a special set of vectors associated with a linear system of equations (i.e. matrix equation). Here a sentence similarity matrix is generated for each cluster and the eigenvector for the matrix is calculated. You can read more on Eigenvector based approach to sentence ranking on this paper https://pdfs.semanticscholar.org/ca73/bbc99be157074d8aad17ca8535e2cd956815.pdf

For source code Siraj Rawal has a Python notebook to create a set of word vectors. The word vectors can then be used to find the similarity between words. The source code is available here https://github.com/llSourcell/word_vectors_game_of_thrones-LIVE

Another option is a tutorial from Oreily that utilizes the gensin Python library to determine the similarity between documents. This tutorial uses NLTK to tokenize then creates a tf-idf (term frequency-inverse document frequency) model from the corpus.  The tf-idf is then used to determine the similarity of the documents. The tutorial is available here https://www.oreilly.com/learning/how-do-i-compare-document-similarity-using-python


Is my understanding of how the convolution with stride 2 works in this example correct?

Yes, you're right, after the green one, it should also move two steps (because stride = 2) to the right once more. Note that in the $3 \times 3$ output volume picture, there's also still a white cell in the top right corner. That cell would get filled with whatever colour you choose to move to the right after the green one.
The blue one would then follow after what I described above, as the fourth square. I guess the author simply didn't feel like drawing a third square to the right of the green one, because the red + green squares already illustrate how the pattern works. The blue one was probably additionally drawn to illustrate that stride works the same way vertically as it does horizontally, e.g. blue also moves down two rows (because stride = 2).
See the bottom of the picture in the second answer here.


What model to use for fully unbalanced data?

You can balance your data-set.
Many models work with batches of samples. If you have a very unbalanced dataset, you can simply split it and ensure your batches are balanced (for example, for a Neural Network, using minibatches of 32 samples, you could draw 16 from your fraud users, and 16 from non-fraud users).
During the learning phase, this ensures the model doesn't just output the most common class, but instead tries to learn to distinguish both.


Are there some guidelines for designing the architecture of neural networks?

In general no. The research area that you seem to be looking for is called generalization, which is very much so still an active area of research. Actual architecture design strongly depends on the dataset itself and available resources, and thus there isn't a general rule of thumb that works for every case.


Using DDPG for control in multi-dimensional continuous action space?

First, is it even possible to use DDPG for multi-dimensional
continuous action spaces?

Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here.

I have not found any code examples to learn from and many of the
papers I have read are near the limit of my understanding in this
area.

You can check kerasRL ddpg implementation it is quite easy to understand and can solve your problem of a model with multiple outputs..

why might my actor network be outputting values clustered near its
max/min values, and why would the values in either cluster all be the
same?

I think it is just a convergence problem try out small learning rates and/or small target_model_update param.
If this doesn't work for you check TRPO and/or PPO..


Is ReLU a non-linear activation function?

Short Answer: Yes

Visually:

if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia:

  "In calculus and related areas, a linear function is a function whose graph is a straight line"

Mathematically:

Linear function of one variable can be defined as:

$ f(x) = ax + b $

If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:

$ f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b $

If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:

$ f(x_1, x_2, ..., x_n) = w_1x_1 + w_2x_2 + ... + w_nx_n + b $

Adding more layer of linear functions doesn't make the function become "complex" for example, if you have $f(x)$ like below and then you put another layer of linear function $g(x)$ on top of it:

$f(x) = ax + b$

$g(x) = cf(x) + d = cax + cb + d$

as the neural network is trained to find the value of $a,b,c,d$, we can group the constant from the formula above, and then rewrite to:

$h(x) = mx + n$

with $m=ca$ and $n=cb+d$. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function

ReLU formula is a $f(x)=max(0,x)$, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.


Are the shortcomings of neural networks diminishing?

Neural Networks have other short comings as well.

It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing techniques too.
The effort involved in learning how to architect and train a NN is still much higher than competing methods, like an SVM. People who are just starting out in Data Science should probably use other techniques to learn about the nuances of fitting data before getting involved in neural networks. And although simple NNs with only one or two hyperparameters are often available in
many data science libraries, they don't perform any better than other techniques so are just another ML black-box technique really.
While we have made a lot of progress in understanding how neural networks do their magic, they are still less accessible and dissectible than most competing methods. So while NNs might solve the problem, they might not give you as many insights as easily as other techniques do.

Looking forward to what other people have to say here.


What are the differences between Bytenet and Wavenet?

My conclusion is the same as yours that there doesn't seem to be any published comparison of the two models.  ByteNet is computationally expensive and requires a lot of parameters.  WaveNet improves on ByteNet's efficiency, as you mentioned, and I believe that is the main difference.


What do the vectors of the center and outside word look like in word2vec?

No, the word vectors are not one-hot encodings. Yes, they are learned.

The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space says something about what that word means. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of distributional similarity.

The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.

A naive objective function to maximize for word2vec is
$$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$

where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}_{t+j}$ is an outside word, and $\textbf{v}_t$ is a center word.

If we let $p(\textbf{u}_{t+j}|\textbf{v}_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}_{t+j}^T\textbf{v}_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.

This lecture from Stanford's CS224N goes into more detail.


Would the people of the 19th Century call our conventional software today artificial intelligence?

They would probably have followed the same sequence we do:

be amazed at the capabilities,
ask how it is done,
wonder whether this is really intelligence and (or) point out our narrow the performance was,
require more next time to be impressed again.


Can a neuron have both a bias and a threshold?

I assume you're talking about a perceptron threshold function. One definition of it with an explicit threshold is
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} > t\\
0& \text{otherwise}
\end{cases}.$$

Another form with a bias is
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} + b > 0\\
0& \text{otherwise}
\end{cases}.$$

But these forms are of course equivalent if you set $b=-t$.

There's nothing stopping you from using a perceptron definition with both a bias and a threshold:
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} +b > t\\
0& \text{otherwise}
\end{cases}.$$

But this is also equivalent to the other two forms. We can rewrite this as
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} > t'\\
0& \text{otherwise}
\end{cases}$$
where $t'=t-b$ is the new threshold. Or, we could rewrite it as
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} + a> 0\\
0& \text{otherwise}
\end{cases}$$
where $a=b-t$ is the new bias.

You never see definitions of this function with both a threshold and a bias because it has simpler forms.


Is it possible to have a dynamic $Q$-function?

I have a function that generates possible actions, so that I would have a Q-table in form of a nested dictionary, with states (added as they occur) as keys whose values are also dictionaries of possible actions as keys and q-values as values. Is this possible? How would it affect learning? What other method can I use?

(Disclaimer: I provided this suggestion to OP here as an answer to a question on Data Science Stack Exchange)

Yes this is possible. Assuming you have settled on all your other decisions for Q learning (such as rewards, discount factor or time horizon etc), then it will have no logical impact on learning compared to any other approach to table building. The structure of your table has no relevance to how Q learning converges, it is an implementation detail.

This choice of structure may have a performance impact in terms of how fast the code runs - the design works best when it significantly reduces memory overhead compared to using a tensor that over-specifies all possible states and actions. If all parts of the state vector could take all values in any combination, and all states had the same number of allowed actions, then a tensor model for the Q table would likely be more efficient than a hash.

  I want to update Q-value, but the next state is one that was never there before and has to be added to my nested dictionary with all possible actions having initial Q-values of zero; how do I update the Q-value, now that all of the actions in this next state have Q-values of zero.

I assume you are referring to the update rule from single step Q learning:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r + \gamma \text{max}_{a'} Q(s',a') - Q(s,a))$$

What do you do when you first visit $(s,a)$, and want to calculate $\text{max}_{a'} Q(s',a')$ for the above update, yet all of $Q(s',a') = 0$, because you literally just created them?

What you do is use the zero value in your update. There is no difference whether you create entries on demand or start with a large table of zeroes. The value of zero is your best estimate of the action value, because you have no data. Over time, as the state and action pair are visited multiple times, perhaps across multiple episodes, the values from experience will back up over time steps due to the way that the update formula makes a link between states $s$ and $s'$.

Actually you can use any arbitrary value other than zero. If you have some method or information from outside of your reinforcement learning routine, then you could use that. Also, sometimes it helps exploration if you use optimistic starting values - i.e. some value which is likely higher than the true optimal value. There are limits to that approach, but it's a quick and easy trick to try and sometimes it helps explore and discover the best policy more reliably.


In deep learning, do we learn a continuous distribution based on the training dataset?

Well, there are some questions here...

Does it (Deep Learning) try to learn a continuous distribution based
on the training-set and its corresponding mappings, and map unseen
examples from this learned distribution?

Yes. Talking about Deep Artificial Neural Networks, they try to learn continuous distribution using continuous activation functions in each neuron. Therefore, the output is also a continuous function to represent a continuous probability distribution. The issue with the unseen examples is the need for similar examples in the training set; otherwise, the weights and bias of the network will not be tuned in the regions of space around the unseen example. Imagine a Neural Network learning a function y = x, if we only present values between 0 and 10 during training, we should expect it to only make good predictions for y for values of x ranging from 0 to 10. It doesn't mean that it won't predict for other values, but the predictions will not be so accurate or nowhere close to the expectations. That is because the network is not trying to guess what was the function used to generate y, but it is simply trying to adjust its parameters to make its internal functions generate the expected y for the given x. That is why Deep Neural Networks require a lot of data. In a unidimensional space is easier to provide examples that cover the subset of the domain we want our network to learn. When we use multidimensional space, we need a lot more examples to have a good representation of the hyperspace used as domain.

Could this (map unseen examples) be said about all predictive algorithms?

Yes, it should. Otherwise, the algorithm would not be able to generalize well. A good predictive algorithm is the one that can predict unseen examples using fewer training samples.

Could the idea of classification problems using hyper-planes be
extended in general to any Deep Learning problem learning a mapping?

In the case of Deep Neural Networks, the result is more like, for a given input value, return the probability of it belonging to a class. For binary classification, the network will have a single output. The sigmoid function modulates this output to ranges between 0 and 1. We can interpret the output as the probability of belonging to one out of two possible classes. To know the probability for the other class, we subtract it from 1. For three or more classes, we will need three or more outputs ranging from 0 to 1, and each output is the probability of belonging to one of the classes. In this case, the outputs are also normalized by a softmax function, that guarantees that the sum of all outputs is equal to 1, as a probability distribution.

Would also explain why Deep Learning needs a lot of data and why it
works better mostly than other Learning algorithms for simple
problems.

Already partially explained... The need for a lot of data is to have a good representation of the hyperplane used as the domain.
The Deep Neural Networks work well because of their power to represent different models. They are a very 'flexible' functions that can be bent to approximate the relation existent between the data in the training set and the expected target. Simpler algorithms, as linear models, for instance, have less representation power, they are limited to a smaller set of models. Even though many models can be linearly approximated (because the input and output almost follow a linear relation), the neural network will be able to learn the nuances of the dataset better. This can also be the curse of neural networks, because they may try to learn every detail of the training set that wasn't really relevant and true for other cases and this concept is called overtraining... but is a discussion for another topic.


How is it possible that deep neural networks are so easily fooled?

First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network.

  The deep neural network is the pre-trained network modeled on AlexNet provided by Caffe. To evolve images, both the directly encoded and indirectly encoded images, we use the Sferes evolutionary framework. The entire code base to conduct the evolutionary experiments can be download [sic] here. The code for the images produced by gradient ascent is available here.

Images that are actually random junk were correctly recognized as nothing meaningful:

  In response to an unrecognizable image, the networks could have output a low confidence for each of the 1000 classes, instead of an extremely high confidence value for one of the classes. In fact, they do just that for randomly generated images (e.g. those in generation 0 of the evolutionary run)

The original goal of the researchers was to use the neural networks to automatically generate images that look like the real things (by getting the recognizer's feedback and trying to change the image to get a more confident result), but they ended up creating the above art. Notice how even in the static-like images there are little splotches - usually near the center - which, it's fair to say, are triggering the recognition.

  We were not trying to produce adversarial, unrecognizable images. Instead, we were trying to produce recognizable images, but these unrecognizable images emerged.

Evidently, these images had just the right distinguishing features to match what the AI looked for in pictures. The "paddle" image does have a paddle-like shape, the "bagel" is round and the right color, the "projector" image is a camera-lens-like thing, the "computer keyboard" is a bunch of rectangles (like the individual keys), and the "chainlink fence" legitimately looks like a chain-link fence to me.

  Figure 8. Evolving images to match DNN classes produces a tremendous diversity of images. Shown are images selected to showcase diversity from 5 evolutionary runs. The diversity suggests that the images are non-random, but that instead evolutions producing [sic] discriminative features of each target class.

Further reading: the original paper (large PDF)


Is a trained model in keras is saved with the weights for max accuracy?

The weights update by default, they do not get saved anywhere, they are just in the programs memory.

But you can choose to save the weights manually or define a callback to do so. You should checkout the ModelCheckpoint callback. It will give you the power to decide what criterion you want to save the weights on.


What's the optimal policy in the rock-paper-scissors game?

For this, we will need game theory.
In game theory, an optimal strategy is one that cannot be exploited by the opponent even if they know your strategy.
Let's say you want a strategy where your move selection is not based on what happened before (so you are not trying to model your opponent, or trick them into believing you will always play scissors and then throw them off, anything like that).
A strategy will look like $(P, S, R)$, where $P, S, R \in [0, 1], P+S+R = 1$. You select paper with probability $P$, scissors with probability $S$, rock with probability $R$.
Now, if your probabilities are a bit uneven (for example $(0.5, 0.2, 0.3)$) an opponent can abuse that strategy. If your opponent plays with probabilities $(p, s, r)$, their expected reward (counting +1 for win, -1 for loss, 0 for draw) would be $0.5(s - r) + 0.2(r - p) + 0.3(p - s) = 0.1p + 0.2s - 0.3r$. If they wish to maximize their wins, they would play scissors all the time against you, and expect to have a distinct advantage over you.
In general, for a strategy $(P, S, R)$ for you and $(p, s, r)$ for your opponent, your opponent's winnings would be $P(s - r) + S(r - p) + R(p - s) =  p(R-S) + s(P-R) + r(S - P)$. If all the partial derivatives of this, with respect to $p$, $s$ and $r$ are 0, the opponent has no way to maximize his winnings; they would have no incentive to play a particular move over any other move.
This occurs when $P = S = R = \frac13$.
That's basically how to approach game theory: find a strategy so your opponent has no incentive to choose one action over another. The approach seems a bit counter-intuitive at first (you're trying to find the optimal strategy for your opponent instead of for yourself) but it works for many similar problems.


How does reinforcement learning with video data work?

In reinforcement learning, to learn off policy control, you need data on the states, actions and rewards at each time step. If, in addition to a recorded video, you had a recording of controller inputs, and could add reward data by hand, then you could use a standard reinforcement learning method, e.g. DQN. Simply run the DQN training loop as normal, but skip the parts where it acts in the environment, and only train on batches of recorded experience.

With only video data, your options are limited. However, it might still be useful, because a significant part of the challenge is a machine vision task. For a DQN agent, it will need to convert frames from the video (e.g. last 4 frames) into a prediction of the different rewards that it could get depending on which controller buttons are pressed. If you can teach a separate neural network to perform a vision task on relevant video data, it may may help. You could use the learned weights from the first layers of this network as the starting point for your Q values network, and it will likely speed up a DQN figuring out the relationship to its predictions. This sort of task switch following learning is called transfer learning, and is often used in computer vision tasks.

A possibly useful starting task if you have a video, but no controller or reward data, would be to predict the next frame(s) of the video, given say four starting frames (you need more than one so that the neural network can use velocity information). It should be possible to generate the training data using opencv or ffmpeg from your recordings.


Regarding the output layer's activation function for continuous action space problems

the use of Tanh is purely because it fits the described problem (Especially for values that are min-max normalized). I have worked on couple of professional RL projects ( specifically with actions in the continuous space) and I did not use tanh at all. Hope that helped :)


I have not touched Obj-C, but I've played with evolution in PHP, which wasn't designed for that at all. If a slow script language on my 10-year old desktop PC can do it, Obj-C should be able to handle that.

Some tricks:

This is a game, so - I assume - you've disabled all the graphics. A headless program is ideal for training. Waste no CPU cycles on stuff you don't need!
Use more threads/instances. Threads around the number of cores should be ideal.
Watch out for memory leaks! Even languages with GC can have problems with this when used in a wrong way. But you'll need to fix these bugs anyway. You can monitor memory consumption when you run it, and if it increases more-or-less steadily, that's a problem. There are various tools against memleaks under different systems/languages, I guess Obj-C has something similar. (these tools log every allocation and release, and list everything at exit that's left)


What is the difference between success rate and reward when dealing with binary and sparse rewards?

Page 6 of the paper describes the exact reward functions, and why they were used:

  Goals: Goals describe the desired position of the object (a box or a
  puck depending on the task) with some fixed tolerance of $\epsilon$ i.e. $G = \mathcal{R}^3$
   and $f_g(s) = [|g − s_{object}| ≤ \epsilon]$, where $s_{object}$ is the position of
  the object in the state s. The mapping from states to goals used in
  HER is simply $m(s) = s_{object}$.

  Rewards: Unless stated otherwise we use binary and sparse rewards $r(s, a, g) = −[f_g(s 0 ) = 0]$ where $s'$ is
  the state after the execution of the action a in the state s. We
  compare sparse and shaped reward functions in Sec. 4.4.

So, at least in the base version (which I believe is your fetchreach-v1), the agent receives a reward of -1 for every timestep spent more than $\epsilon$ from the goal state, and a reward of 0 for every timestep spent within $\epsilon$ of the goal state. Thus, a score of -5.0 would seem to correspond to the agent moving directly to the goal and staying there, while a score of -50.0 would correspond to the agent failing to reach the goal state entirely.


Training a classifier on different datasets with different image conditions for different labels causes the model to infer using the background

I am afraid that the model will infer from the background information that it shouldn't use to predict the plant diseases, what makes the problem worse is that some plant diseases only exist in one dataset and not the other

I am afraid that when I use the model on real-life conditions (say someone capturing an image with their phone) the model will be heavily biased towards predicting labels in the second dataset (the one with actual background) since the background is similar and it somehow associated the background with the inference process.

You are right to be concerned about this. However, you are not alone in having a neural network model that relies heavily on the background or other confounding factors for your classification task.
A small paper that addresses a similar issue is Context Augmentation for Convolutional Neural Networks. Table 1 from this paper shows how the model trained and tested on only the background performs better than model trained and tested on only the foreground. They address this by making a training set where the foreground objects are given many backgrounds. Given your imbalanced datasets, with some diseases only present in one dataset and not the other, I think context augmentation might work for you. The main barrier here is that you do not have segmentation maps for these. Are your datasets too large to segment yourself? You might consider automatic segmentation networks - especially for the images with white backgrounds.
As for your interpretability methods, you should not expect too much from them though some groups are able to get good results with them using GradCAM.
Here is an alternative idea that may also solve your problem: If you are trying to make a system that can be used in the field, is it possible to have the technician or photographer put a white board under the diseased leaf?


Recurrent Neural Network to track distance from origin

This kind of problem does not really have a name other than "toy problem" since no-one needs to teach an AI to add up, multiply or divide* - there are already far more reliable and far faster ways to achieve that on any computer. What you are doing here is essentially vector addition, applying a distance metric then setting a true/false value based on a comparison. It would be 2 or 3 lines of code in most high-level programming languages.

Neural networks can learn any function though, so in theory what you want to do is possible. You should not expect results to be perfect, a statistical learner never actually learns the analytical form of a function or process, just the rough "shape" of it.

I have not done your experiment. However, your idea to use a RNN seems reasonable. With the details you have given, I can offer a few pieces of general advice:

Use a modern RNN gated architecture, either LSTM or GRU. That's because the point in the sequence of moves where you want to set a "distance exceeded" toggle could be many steps away from the end of a game. The simplest RNNs (with direct loop backs from output to input within a layer) can easily suffer from vanishing gradients in this situation, whilst LSTM and GRU architectures are designed to deal with it.
Generate a lot of training data. You will need many examples of both categories before any neural network will home in on what is causing one class or the other to occur. The learning is based on statistics, not reason.
Take a look at related LTSM examples that learn to add binary numbers. Repeat those even simpler experiments first, then move on to your own problem. This will avoid some beginner mistakes with poor choices of hyper-parameters, bad implementations etc.

* Humans of course do use intelligence, reasoning and logic to learn reliable procedures for addition, multiplication and division. No doubt someone could be interested in how an AI can replicate that, without starting from built-in capabilties of a CPU (which of course the humans designed and built those procedures into the system at a low level). However, that's at a higher level of AI research than we are dealing with here.


Can CNNs be applied to non-image data, given that the convolution and pooling operations are mainly applied to imagery?

Usually, you need to ensure that your convolutions are causal, meaning that there is no information leakage from the future into the past. You could start by looking at this paper, which compares Temporal Convolutional Networks (TCN) with vanilla RNNs models.


Yes, this is possible.

There is actually a pretty easy way that doesn't even require machine learning and can be implemented with a small amount of code. You just use a framework for image processing (e.g. PIL for Python), find the marks by going over your image with an appropriate filter and use the implemented crop function, that the framework hopefully provides (or write i yourself with 2 nested for loops copying the area). Your filter will react strongest in the areas of your image that look like the mark you defined. This could actually be implemented like a single layer conv+RELU layer in a CNN (see this introduction for details about conv and RELU layers).

Because you are asking explicitly about machine learning, I will also give you the hard version to solve this problem using a full CNN to identify the bounding box. Once you have found the bounding box, the actual cropping can be done like in the example above and is not part of the CNN. You could call it post processing if you like.

The technique you are looking for is called Object Localization and Detection. The linked article should tell you all the details you need. You basically build and train your CNN to localize and identify interesting objects in your image. The easy way is to train your CNN to identify the 4 marks individually and return the coordinates of the 4 areas with the highest likelihood to contain your marks. You calculate the center of those 4 individual points and feed it to your cropping function. The hard way is to find the smallest bounding box containing all 4 marks. This is a much harder problem, because the neural network needs to learn that it should ignore everything except those 4 marks during detection, although they will be a small portion of the bounding box. It can be done, but looking for the 4 marks individually would be much easier.

Which approach is best for you? Depends on your goal. If you need a robust and efficient algorithm for this problem, forget about ML and implement it with the straight forward approach I described first. No need for fancy learning CNNs. You can implement this with a few dozen lines of code.

If this is a research or training project to study ML, take my second approach. The easy and the hard way are both fine, the hard one obviously more challenging for your CNN. If you are new to CNN, I recommend following this Stanford Course. It teaches you all you need for your project in 25 to 30 hours (excluding homework).

Edit: Concerning your question, how do you add your own marks. I would simply do it using with the same framework you use for cropping. Your program needs some kind of image manipulation capabilities and this will allow pixel manipulation as well (at least the framework PIL which I suggested has this capability). Just draw your mark using this method. You can also copy a sample mark with a transparent background using a tool like Gimp manually, if you prefer that.


Do we have to use the IOB format on labels in the NER dataset? If so, why?

There's nothing stopping you from training a model with whatever tags you want.
Using what you describe as "usual" format means you would have approx half as many tags as using the IOB format. In theory this means your model will develop higher accuracy faster and with less training data. On the downside, you will need to do more work when interpreting the results in order to be confident where one named entity ends and another one begins.
I made a notebook to do some empirical tests on this which uses 17 tags in the IOB format and 9 in the "usual" format.
TL;DR using the "usual" format did not produce a noticeable improvement in model quality.


What techniques would allow a model to be trained to produce subjectively pleasant 'artistic' output?

You will need a large training set to be able to train your machine to produce subjectively pleasant 'artistic' output.

If you can't produce your own dataset with real people, you can try the following:

search for existing datasets on the internet
produce a new dataset using a spider bot that perform mining of artistic output on websites that have voting features
use crowdsourcing


How many parameters are being optimised over in a simple CNN?

Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions.

In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.

So if you had Conv -> pool -> Conv -> pool -> ... , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.

In your network you start with a 28*28 image, and you use 32 3*3 filters. so number of parameters is (3*3 + 1) * 32 = 320.

In the dense layer you have as input a 13*13*32 and use a 100 FC layer. so n_parameters is (13*13*32 + 1)*100 which is 540900.

Then you get (100+1) * 10 FC which is 1010 more.

total = 320 + 540900 + 1000 which is 542230, as expected.

The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.

Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.


What are some books/paper that deal with fundamental and philosophical issues of ML and relate it to the global discourse of AIs?

I'll recommend two sources:

The venerable Russell & Norvig book, which is a common text in AI courses. Russell & Norvig end each chapter with a summary of the history of the developments of the techniques they have just discussed. These sections are often skipped by novice readers, but are almost exactly what you are looking for. The ones in the back half of the book should, together, give you a good sense of the order in which developments occurred, why techniques were developed, and what advances techniques enabled.
R&N does a good job of covering what happened, but not always why. For that, you want a philosophy of AI book. I recommend Mind Design II as a starting place. This book is a chronologically organized collection of papers and academic essays by the big thinkers in philosophy of mind and in AI research. Often, the papers are responses to one another. With some side reading about the history of each author, you can begin to get a good sense of the big philosophical movements in the field over the last 70 years, and why things have changed.

If you don't want to read the book, I can give you a summary (spoilers ahead!):

1920's: behaviorists, and others, propose that Mind = Brain, and specifically focus on intelligent behavior.
1950: Turing proposes that a computer could be programmed to exhibit intelligent behavior.
1960's: Cognativists in AI, Psych, and Linguistics argue that behavior is not enough. Minds think, and thought takes the form of reasoned algorithms. The lynchpin of their argument is domains like language understanding, which they claim cannot be modeled without logic. Their work produces search algorithms, and early rule-based planning and language systems.
1970's Searle argues that computers can't think because of Phenomenology. Most AI folk ignore him, and go on working, but Philosophy & Psych take greater notice.
1970's Dryfus (and others) argue compelling that logic and reason cannot explain human thought, by dissecting the rule based systems of the day through the Frame Problem. AI researchers take some notice.
The Connectionists, especially Hinton (AI) & Churchland (Philosophy) propose the first post-Cognativist theories of mind. These focus on the idea that Mind is in the Connections (specifically, the firing patterns) of neurons, not in the brain itself. This view of minds spurs (re)-development of Neural Networks. In the 1980's, this is mostly ignored.
During the 1990's, Connectionists, and others working on probabilistic methods in AI demonstrate systems for language that are substantially better than rule based methods. Cognativists begin to decline, because the domains they claimed needed reasoned algorithms are actually better handled by statistical algorithms. Connectionist views gain support in AI and elsewhere.
Today, with rising computational power, statistical techniques come to the forefront of many AI domains. Simultainiously, the Churchlands, Rodney Brooks, and others propose further Post-Cognativist schools of thought based around dynamical systems theory and embodied cognition, which were somewhat influential in robotics. Cognativism continues to enjoy some support within the AI, Psych, and Linguistics communities, but this is much diminished. Some hybrid systems use a mix of statistical and rule-based techniques.


Which paper introduced the term "softmax"?

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (1989, NIPS) by John S. Bridle.
As a side note, the softmax function (with base $b = e^{-\beta}$)
$$\sigma (\mathbf {z} )_{i}={\frac {e^{-\beta z_{i}}}{\sum _{j=1}^{K}e^{-\beta z_{j}}}}{\text{ for }}i=1,\dotsc ,K {\text{ and }}\mathbf {z} =(z_{1},\dotsc ,z_{K})\in \mathbb {R} ^{K}$$
is very similar to the Boltzmann (or Gibbs) distribution
$$
p_i=\frac{e^{- {\varepsilon}_i / k T}}{\sum_{j=1}^{M}{e^{- {\varepsilon}_j / k T}}}
$$
which was formulated by Ludwig Boltzmann in 1868, so the idea and formulation of the softmax function is quite old.


Are the training loss and validation loss plotted per sample or per batch?

You want to compute the mean loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!

In your case:

You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So the way you compute your training loss, that is what you need to divide by.

Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses

training_loss = []
validation_loss = []
training_steps = num_samples // batch_size
validation_steps = num_validation_samples // batch_size

for epoch in range(num_epochs):

    # Training steps
    total_loss = 0
    for b in range(training_steps):
        batch_loss = ...  # compute batch loss
        total_loss += batch_loss
    training_loss.append(total_loss / training_steps)

    # Validation steps
    total_loss = 0
    for b in range(validation_steps):
        batch_loss = ...  # compute batch validation loss
        total_loss += batch_loss
    training_loss.append(total_loss / validation_steps)

# Plot training and validation curves
plt.plot(range(num_epochs), training_loss)
plt.plot(range(num_epochs), validation_loss)

Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.

...

for epoch in range(num_epochs):

    list_of_batch_losses = []  # initialize list that is going to store batch losses

    # Training steps
    for b in range(training_steps):
        batch_loss = ...  # compute batch loss
        list_of_batch_losses.append(batch_loss)  # store loss in a list

    epoch_loss = np.mean(list_of_batch_losses)
    training_loss.append(epoch_loss)

    ...

plt.plot(range(num_epochs), training_loss)


Is it possible to have the latent vector of an auto-encoder with size 1?

According to various experimentation on autoencoders, it is very possible to have latent vector of size 1. Various layers can help the downsizing of the original input to a very small size of 1. But an issue may arise during decoding. If you're expecting that through one or two or maybe five layers in decoder you can achieve an accurate reconstruction, it is highly unlikely abd the result will turn out to be blurry. Maybe a great network with various parameters may help the reconstruction without considering generalization as asked by you.


classification of unseen classes of image in open set classification

I have one possible solution for you inspired by real-time face recognition system. It is a similar case to you.

Create embeddings for each class i.e. person or in your case, a class. (Using Siamese network with ArcFace loss)
When a new image comes, take L1 or cosine distance with the embeddings. You will need a threshold above which you create a embedding for the new guy and give it a new class.

One issue you will think at this point of time is how do you do that when have billions of data points. The search for the matching embeddings can then be done using Hierarchical Navigable Small World graphs (see https://github.com/nmslib/hnswlib).
This solution is loosely connected to the plausible techniques you have mentioned here. And if you still  want something in that regard, I shall add the keyword Active Learning.


Is it meaningful to give more weight to the result of monte carlo search with less turn win?

Traditionally (when not considering your idea), the evaluation function for terminal game states would be implemented to return $1$, $0$, or $-1$ for wins, draws, or losses, respectively.

Changing that in a naive/straightforward way to make short-term wins more rewarding, long-term wins less rewarding, short-term losses more negative, and long-term losses less negative can be dangerous, it may change the objective that your agent is ultimately optimizing for (i.e. may lose the guarantee of converging towards optimal play given an infinite amount of time) if not done very carefully.

There is definitely value in considering the idea though, especially because in the Play-Out phase of MCTS, trajectories of (semi-)random moves introduce uncertainty in the evaluations at the end of those simulations, and this uncertainty increases as the length of the trajectories increases (due to increased number of uninformed decisions being made along the trajectory). Note that it is especially important to take into consideration here the number of moves played in the Play-Out phase, not necessarily including the number of moves made in the Selection phase (which are selected according to a much more informed strategy).

One paper I know of that investigates ideas along these lines is "Quality-based Rewards for Monte-Carlo Tree Search Simulations".


Which is more memory efficient uninformed or informed search algorithms?

Uninformed Search Techniques

Breadth-First Search needs to store a frontier of nodes to visit next (where "visit" basically means: see if it's the goal, generate its children and add to frontier otherwise). You can visualize the memory requirements of this as a pyramid; initially you just have the root node in there. As the search process continues and you keep going further down, it will become wider and wider (more memory required). The amount of memory required in the worst case to find a node at depth d can be expressed as O(b^d), where b is the branching factor (see details here). Intuitively, whenever you go one level deeper, your memory requirements multiply by a factor of b (your pyramid becomes b times wider; actually this means pyramid is not quite the right shape, it should rapidly curve outwards and become wider much faster than a triangle/pyramid would).

Depth-First Search also stores a frontier (or stack in the case of DFS) of nodes to visit next. However, in the case of DFS this generally grows less quickly. Whenever you visit a node (pop it off of the stack), you generate all of its children and push them on top of the stack. However, whereas BFS would then continue "to the right" by visiting a relatively "old" node and generating all of its children again, DFS continues by going "down" and visiting one of the children that only just got pushed onto the stack. Once DFS has finished a part of the search tree (you can visualize this in your head as DFS having completed searched for example the left half of the pyramid), that entire section no longer needs to be stored in memory. Assuming you have a search tree of finite depth d, the worst case space complexity is O(bd) (you have to search one path all the way down to a level of d, and for each of those levels have b nodes in the stack).

Note that, based on the above, it is not sufficient to characterize your comparison as "uninformed" vs "informed" search. We've only just looked at uninformed search techniques, and already have two different memory requirements.

In the above, I also did not consider subtleties such as the question of whether or not you're additionally memorizing which nodes of a graph you have already previously visited so that you can avoid visiting them again. This is not necessary in a tree, but may be necessary in a graph with cycles.

Informed Search Techniques

A* is probably one of the most canonical examples of informed search algorithms. In terms of worst case memory requirements, it's really similar to BFS; it also stores a frontier of nodes to visit next, but prioritizes those based on some estimate of "goodness" rather than Breadth-First order. In cases where you the information you're using (typically a combination of known/incurred costs + heuristic estimate of future costs) is of extremely poor quality, this can regress to the level of Breadth-First Search, so the worst-case space complexity is the same as that for BFS (see details). In practice, if you have high quality information (good heuristics), the memory requirements will be much better though; in the case of an ideal/perfect heuristic, your search algorithm will pretty much go directly to the goal and hardly require any memory.

There is also an algorithm called Iterative Deepening A* which has a much better space complexity, at the cost of often being slower. It is still an informed search algorithm using exactly the same kind of information as A* though. So, really, memory requirements cannot be characterized in terms of informed vs. uninformed search; in both cases there are algorithms that require a lot of memory, and algorithms that require less memory.


How can we find find the input image which maximizes the class-probability for an ANN?

In deep networks there is actually a wide variety of solutions to the problem, but if you need to find one, any easy way to do this is just through normal optimization schemes
$$\hat x = argmin_x \ L(y,x)$$
where $L(y,x)$ is your loss function. Since ANN's are generally differentiable you can optimize this iteratively with some form gradient descent scheme:
$$x^{i+1} \leftarrow x^{i} - \lambda \nabla_{x^i}L(y,x^i)$$
where $\lambda$ is your learning rate.


Why do the inputs and outputs of a convolutional layer usually have the same depth?

Keeping the same channel size allows the model to maintain rank but i would say the main reason is convenience. Its easier book keeping.

Also in many model cases output features need some form of alignment with the input (example being all models using residual units -- $\hat{x} = F(x) + x$


How are filters weights updated for a CNN?

You have made many wrong assumptions in this question. First theoretically speaking,

Filters do not work in the way to 'pick up elements' (they work on principle of edge detection).
You have assumed only a single combination of filter weights will give the desired output (assuming continuous weights not binary). This is especially in prominence in the problem of Regularization where we want to choose a set of weights without over-fitting data.
The error you used looks very similar to Perceptron update rule (squared error gives the same derivative, but make sure you are not confusing the two).
Backpropagation through 'dead ReLu's' is not possible (see this answer for more details).

Now, let us check mathematically:

  Input Volume:

$$
    \begin{matrix}
    1 & -1 & 0 \\
    0 & 1 & 0 \\
    0 & -1 & 1 \\
    \end{matrix}
$$

  Desired Output:

$$
    \begin{matrix}
    1 & -1 \\
    0 & 1  \\
    \end{matrix}
$$

Note, in this step you desire an output which is negative (element (0,-1)), but you are forward propagating through a ReLu which is cutting off the negative part, thus the gradients have no way to communicate or update the required negative.
Basically,

$ wx \rightarrow ReLu \rightarrow y$ is happening and if 'x' is a negative number then $y$ is always $0$ thus $(target-y)$ is always $target$ and hence whatever the value of $x$ error remains constant, and if we want to backpropagate (assuming squared error) then:

$\frac{d}{dw} (target - y)^2 = 2*(target - y)*\frac{d}{dw}y = -2*(target - y)*0 = 0$ (Remember from the ReLu output graph slope is $0$ in the negative region).

  Now, you randomise a filter:

$$
    \begin{matrix}
    1 & -1 \\
    1 & 1  \\
    \end{matrix}
$$

apply ReLu and get the following:

$$
    \begin{matrix}
    3 & 0 \\
    0 & 1  \\
    \end{matrix}
$$

again you have chosen you target to have a negative number which is not possible in the case of ReLu activation.

But continuing you get the error as:

$$
    \begin{matrix}
    -2 & -1 \\
    0 & 0  \\
    \end{matrix}
$$

use error to compute gradients (which again you have calculated wrong,by Backpropagating through ReLu's with ) values and also missed the minus sign associated with the output, but you have compensated it by adding it to the $w$ whereas the convention is to subtract):

$$
    \begin{matrix}
    1 & 2 \\
    1 & -2  \\
    \end{matrix}
$$

And get the new filter as:

$$
    \begin{matrix}
    0.5 & 0 \\
    0.5 & 0  \\
    \end{matrix}
$$

This is a pretty good approximation of the desired filter (even though the previous steps have wrong assumptions, but it does not matter much, since what you essentially did was use a linear activation function which will work, if you go through enough iterations). So basically you are using a Linear Filters and the details are too hodge podge for me to go into, so I will suggest some resources for you to see ReLu backpropagation:

Deep Neural Network - Backpropogation with ReLU


Is it possible to do face recognition with just the eyes?

Yes, it must be possible as retina scanners have been used as a method of personal identification for some time. The difference is, if you have a retinal scanner you are probably controlling the focal distance of the picture you are taking and using suitably high resolution. Your mileage may vary as these things decrease in quality.


Were there known tests done on two or more AI interacting together?

One example might be self-play in games. Since neural networks and deep learning depend on massive amounts of data, one way to generate data is to have two virtual machines play each other and record the experience. An example discussion can be found at http://www.cs.cornell.edu/boom/2001sp/Tsinteris/gammon.htm which uses reinforcement learning. I believe AlphaGo also uses this technique of self-play, and uses two independent neural networks, one reducing the search space and the other deciding on the best move in the remaining space, that in a sense cooperate to decide on the next move.


I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and  9000th epoch's loss is greater about 0.2 alternate of 0.09

I also tried to change optimizer and learning rate but no loss didn't improve.

you can explore the modified code may help you with another idea


What kind of problem is "email text extraction"?

It's certainly possible to treat this as a natural language processing problem, basically you're looking to assign "salience" scores to the text.

Really, though, that's overkill for this kind of problem. Writing a regex or a CFG parser (or better: finding an existing parser) is likely to be easier and more reliable.


How can AI help developer to develop things

An umbrella term for the application of heuristic techniques to software development is 'Search Based Software Engineering' (SBSE).

SBSE emerged as a distinct activity around the turn of the century, with a strong initial focus on automating the generation/prioritization of test cases.

With respect to some of your specific queries:

1.2 Paper on Automated refactoring

Automatically choosing screen colour to minimize energy consumption.
This sort of thing is not usually done heuristically, since it needs platform-specific code.
Automated refactoring to patterns.
AFAIK, penetration testing has yet to be successfully automated.
As stated, this doesn't really require AI. More generally, I don't know of any specific work automating for HCI preferences, but something like 'Interactive Genetic Algorithms' could be used.
There's a lot of SBSE literature on testing. See this paper for a general overview.


Thoughts on Apple Mac Pro vs GCP/AWS for Deep Learning?

The Mac Pro uses AMD GPUs. These don't support CUDA, but instead support the OpenCL framework.

TensorFlow, the most popular deep learning library, uses CUDA to run on GPUs, although OpenCL support is in the works. That said, one of the main approaches to OpenCL support, SYCL, isn't planning to support OSX:

  We have no plans to support OSX in near future. I know sad face.

The current advice is to avoid AMD GPUs:

  Currently, if you want to do DL and want to avoid major headaches, choose Nvidia.


What should we do when we have equal observations with different labels?

The problem you are portraying looks like a modified XOR problem. You can't throw away the lines with a label of 1 because a the model won't be able to learn this class.


Why can't we train neural networks in a peer-to-peer manner?

Data management and bandwidth are key issues for interconnecting multiple GPUs.  These are such big issues that it is hard to think about other challenges like neural network architecture, metrics, etc.  The key to success for interconnecting multiple GPUs on a single computer is NVIDIA's NVLink:
NVLink is a wire-based communications protocol for near-range semiconductor communications developed by Nvidia that can be used for data and control code transfers in processor systems between CPUs and GPUs and solely between GPUs. NVLink specifies a point-to-point connection with data rates of 20 and 25 Gbit/s (v1.0/v2.0) per differential pair.
Compare 25 Gbit/s to a typical peer to peer connection over the web of 100Mbps.  NVLINK provides a 250x advantage assuming everything else is equal which it is not.  This means that, considering bandwidth only, a neural network which takes one day to train on a computer with two GPUs connected with NVLINK could take 250 days over the internet using two computers with the same GPU!


Which approach can I use to generate text based on multiple inputs?

Generally, text generators work by modeling the joint distribution of the text by its Bayesian forward decomposition

$
\begin{align*}
p(w_1, w_2, ..., w_n) &= p(w_1) * p(w_2|w_1) * p(w_3|w_2, w_1) *\ ...\ * p(w_n|\{w_i\}_{i<n})\\
&= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i})\\
\end{align*}
$

From a modeling perspective, this looks right up RNN's ally, where you can have a state holding information from $\{w_k\}_{k<i}$ to learn a representation of $w_i$

Now, in your specific case, you're interested in a conditional text-generator, so you are trying to model $p(w_1, w_2, ..., w_n | \{v_j\}_j)$, but this same tactic works.

$
\begin{align*}
p(w_1, w_2, ..., w_n| \{v_j\}_j) &= p(w_1|\{v_j\}_j) * p(w_2|w_1, \{v_j\}_j) * p(w_3|w_2, w_1, \{v_j\}_j) *\ ...\ * p(w_n|\{w_i\}_{i<n}, \{v_j\}_j)\\
&= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i}, \{v_j\}_j)\\
\end{align*}
$

So, in your RNN or forward-based model, you can use the exact same approach just additionally embed the conditional inputs you have and somehow infuse it into the model (in practice, I have seen this through attention, concatenation, or some other common approach).

My recommendation (depending on the computational power you have) is to take advantage of the recent fad of pre-trained language models. Specifically, ones trained on next word prediction will probably do the job best. A good example is gpt-2, and, if you check out their GitHub, their code is very readable and easy to adjust for adding conditional input in the ways I have described.


Can a machine learning approach solve this constrained optimisation problem?

Sounds more like a optimization problem than a deep learning / machine learning problem to me.

For machine learning you would have the features of every child / vehicle and the optimal amount of pizza / fuel already given, but you don't know how exactly the optimal amount is computed. So the target is to find a function which maps features to target.

However in your case you don't know the optimal value, you just have a number of constraints. So my suggestion would be to use linear programming.

Here is an simple example: drive the maximum distance with a given amount of fuel

$$\text{max }\ m_1 \cdot x_1 + ... + m_n \cdot x_n$$
$$\text{s.t. }\ x_1 + ... + x_n \leq F$$
$$m_i \cdot x_i \geq T_i\  \forall\  1 \leq i \leq n$$

$x_i$ is the amount of fuel (in gallons) given to car $i$, $m_i$'s are miles-per-gallon for car $i$. $F$ is the total amount of fuel available, this is our first constraint, we cannot distribute more fuel than we have. The second constraint say's that we like to drive at least $T_i$ miles with car $i$. This ensures we do not just give all the fuel to the most efficient car.

You can come up with more constraints for drivers competency, engine horse power etc. In order so solve your problem with linear programming, you just have to make sure it is still linear and convex. Problems of this kind can be solved using the Simplex algorithm (scipy.optimize.linprog).
If your objective or your constraints are more complex you can use numerical methods for non-linear constrained optimization problems (see scipy.optimize for algorithms).


What makes GAN or VAE better at image generation than NN that directly maps inputs to images

The only disadvantage and difference between these generative models and the method you describe, is the input. You describe inputting tags, where as for a GAN, or VAE, the generation segment of the model takes in some representation of a probability distribution. For a GAN, it's mostly random noise, and for a VAE it is some latent space (see nbros answer).
Your described method prevents the network from properly learning fluid generation. If you have a discrete input, the network will attempt to perform a sort of classification on the input, rather then generation, and so when you try to generate a new image, you will most likely get the image equivalent of gibberish.
In fact, that's why a standard auto-encoder (not variational) doesn't work very well for generation. You would think that you could feed in your own custom input into the latent space:
But if you tried this, you would end up telling the network to try and generate something from a latent space it can't interpret.
Hence where the "variational" part of the VAE comes in. This helps the network learn to generate from a continuous distribution, so no matter what input you use, the network will be able to interpret it and give an appropriate output.
As for a GAN, it is simply fed random noise at each training step, so it too generates based on a continuous distribution.
If you were to try and train your method of generation, I would predict that you would find an average of all images that share similar tags (say you have the tags of cat, dog and brown haired, if you input "dog=1, cat=0, brown haired=1" you would get an average of all brown haired dogs), but if you tried to input a combination of tags the network has not seen, as it has not learnt from a continuous distribution, the resultant image would not be anything like what you'd expect from those tags.


What kind of optimizer is suggested to use for binary classification of similar images?

I have consistently found Adam to work very well but to tell you the truth I have not seen all that much difference in performance based on the optimizer. Other factor seem to have much more influence on the final model performance.In particular adjusting the learning rate during training can be very effective. Also saving the weights for the lowest validation loss and loading the model with those weights to make predictions works very well. Keras provides two callbacks that help you achieve this. Documentation is at
https://keras.io/callbacks/. The ReduceLROnPlateau callback allows you to adjust the learning rate based on monitoring a metric. Typically validation loss is monitored. If the loss fails to reduce after N consecutive epochs(parameter patience) the learning rate is adjusted by a factor(parameter factor). You can think of training as descending into a valley which gets more and more narrow as you approach the bottom. If the learning rate does not adjust to this "narrowness" there is no way you will get to the very bottom.
The other callback is ModelCheckpoint. This allows you to save the model( or just the weights) based on monitoring of a metric. Again usually validation loss is monitored and the parameter save_best_only is set to true. This saves the model with the lowest validation loss. That model can than be used to make predictions.


In what situations ELUs should be used instead of RELUs?

ELU does not suffer from from dying neurons issue, unlike ReLU. While ELU can help you to acheive a better accuracy, it is a slower than ReLU because of its non linearity in its negative range.
Choosing a right activation function totally depends on situations but you need to also consider other similar types of activation function such as leaky ReLU.
Check this link out. It could be useful.


How to reduce amount of species in NEAT?

Your species count will increase as the chance of mutation increases. This is because in every generation, so many genes will be mutated that they have little resemblance of each other, and the distance function doesn't factor in historical markings / innovation numbers.

Try lowering the mutation rates.

Below is the distance function from here page 110

$$\delta = \frac{c_1E}{N} + \frac{c_2D}{N} + c_3 \cdot \overline{W}.  $$

If your fitness vary a lot, try ranking the fitnesses in each specie and setting the survival chance based on its rank.

If you mean a large action space by changing environment, you can set the number of output nodes to the total number of actions, and rank each action, best to worst, then pick the best available action for the state.


How are exploding numbers in a forward pass of a CNN combated?

The most effective way to prevent both the forward and backward propagation of exploding is keeping the weights in a small range. The main way this is accomplished is through their initialization.

For example in the case of He initialization, the authors show (given some assumptions) that the variance of the output of the final layer $L$ of the network is:

$$
Var[y_L] = Var[y_1] \left( \prod_{i=2}^L{\frac{1}{2} \, n_l \, Var[w_l]} \right)
$$

where $n_l$ and $w_l$ are the number of connections and weights of layer $l$. In order to keep the outputs from exploding the product above should not exponentially magnify its inputs. In order to do this the authors elect to initialize the weights so that:

$$
\frac{1}{2} \, n_l \, Var[w_l] = 1
$$

Now this is helps keeping the outputs from exploding. They then go and prove that the same strategy helps preventing the gradients from exploding.

Another similar strategy is the so-called Glorot (or Xavier) Initialization.
These techniques are extremely effective in helping the models converge!


Neural network cost function implementation

So, firstly, for h _{Theta}(x) to be 1, the weighted sum of x (after you dot product it with Theta) would have to be literally infinity, if you're using the sigmoid function. Doesn't happen in practice, even with the rounding computers do, as we don't use big numbers to initialize our Theta matrices.

Intuitively, that'd mean you're basically more certain than one can possibly be in this universe that the label of this example should be '1'.

So if (1 - h_{Theta}(x)) = 0, 'y' is certainly 1, and so 1-y will be zero.

Secondly, as the comments note, the convention is to drop the entire right-hand-side term when y^(i) is 1. This will not cause problems when programming, due to the first point I made above.


Does IBM Cloud Private for Data run on public clouds like AWS or Azure?

Yes, ICP for Data can run on any public cloud infrastructure as long as it supports Docker containers and Kubernetes.

Already tested: Amazon AWS, IBM Cloud, Azure, and Red Hat OpenStack


This is an idea that I used for my model -
Try using two RNN (GRU) Networks, one of them to manage current output state and the other to maintain context

Say we are at timestamp $t$ and the two GRUCells are represented as $GRU_c$ and $GRU_s$ for GRU context network and state network. (Your output coming from the state network)

At time stamp $t$ , the input $GRU_s(t) = concat(input, att(all~GRU_c~from ~[0, ~t-1]))$ where $att$ is an attention mechanism to give importance to specific parts of the conversation uptil that point (This is what maintains context)  and input $GRU_c(t) = learned~representation~of~GRU_s(t)$ , hence updating $GRU_c$ for that timestamp, which along with the historical information can be used for $GRU_s(t+1)$

Hope this helped!


Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?

You grasped a bit of the answer.

  In the latter two cases the optimisation gets stuck in suboptimal local minima.

When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.

The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.


Is there a convention on the order of multiplication of the weights with the inputs in neural nets?

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.
Some examples include:

In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by
$$ f(x) = \sigma(Wx + b)$$
for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.

In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$ f(x) = \max \{ 0, W^T x + c \},$$
where the $\max$ function is applied componentwise and gives the ReLU activation.

In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$
where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.
Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.