What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. Also, using bold letters to represent vectors. In that notation, the update rule for basic gradient descent would be written as:

$$\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_w J$$

This cannot be extended to momentum and Adam update rules whilst keeping it as a single line and modifying the right hand side. That is because these variations of gradient descent maintain running statistics of previous gradient values, which have their own separate update rules. When implemented on a computer, these become additional terms, mainly vectors the same size as the weight vector being updated. These variables also require initialisation before use.

Momentum

Momentum maintains a "velocity" term which essentially tracks a recency-weighted average of gradients. However, the classic form of momentum given here does not normalise the resulting vector, and you often have to adjust the learning rate down when using it. Momentum has a parameter $\beta$ which should be between 0 and 1, and typically is set at $0.9$ or higher.

Intitialisation

$$\mathbf{m} \leftarrow \mathbf{0}$$

Update rules

$$\mathbf{m} \leftarrow \beta \mathbf{m} + \nabla_w J$$
$$\mathbf{w} \leftarrow \mathbf{w} - \alpha \mathbf{m}$$

There are some variants of these update rules in practice. An important one is Nesterov momentum.

Adam

The Adam optimiser maintains a momentum term, plus a scaling term, and also corrects these terms for initial bias. Adam has three parameters $\beta_m$ for momentum (typically 0.99), $\beta_v$ for scaling (typically 0.999) and $\epsilon$ to avoid divide by zero and numerical stability issues (typically $10^{-6}$).

Initialisation

$$\rho_m \leftarrow 1$$
$$\rho_v \leftarrow 1$$
$$\mathbf{m} \leftarrow \mathbf{0}$$
$$\mathbf{v} \leftarrow \mathbf{0}$$

Update rules

$$\rho_m \leftarrow \beta_m \rho_m$$
$$\rho_v \leftarrow \beta_v \rho_v$$
$$\mathbf{m} \leftarrow \beta_m \mathbf{m} + (1-\beta_m) \nabla_w J$$
$$\mathbf{v} \leftarrow \beta_v \mathbf{v} + (1-\beta_v) (\nabla_w J \odot \nabla_w J)$$
$$\mathbf{w} \leftarrow \mathbf{w}- \alpha(\frac{\mathbf{m}}{\sqrt{\mathbf{v}}+\epsilon} \frac{\sqrt{1-\rho_v}}{1-\rho_m})$$

The symbol $\odot$ stands for element-wise multiplication. Here that essentially means to square each term of the gradient to calculate terms in $\mathbf{v}$. The square root and division of $\mathbf{m}$ by $\sqrt{\mathbf{v}}+\epsilon$ in the last update step are also handled element-wise.

The variant I show here has an "optimisation" to the bias correction so that you don't need to calculate high powers of either of the parameters. You may see variants that don't have $\rho_m$ and $\rho_v$ (or equivalents), but instead use $\beta_{m}^t$ or $\beta_{v}^t$ directly, which is exactly what $\rho_m$ and $\rho_v$ represent.

* $\nabla_w J$ is the gradient of $J$ with respect to $\mathbf{w}$. By writing it this way, it also describes the goal of the update explicitly within the notation i.e. to minimise a function that is parameterised by $\mathbf{w}$.


How do you encode a chess move in a neural network?

The number is 4672 from Google.

  https://arxiv.org/pdf/1712.01815.pdf

A move in chess may be described in two parts: selecting the piece to move, and then
selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
positions identifies the square from which to “pick up” a piece.

4672 might sound a big number but it's nothing to what Google (and many other competitors) has been doing to deep learning for image analysis.


What are the biggest barriers to get RL in production?

There is a relatively recent paper that tackles this issue: Challenges of real-world reinforcement learning (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.

Batch off-line and off-policy training

One current solution is importance sampling

Learning on the real system from limited samples (sample inefficiency)

Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches

High dimensional continuous state and action spaces

Solutions: AE-DQN, DRRN

Satisfying safety constraints

Solutions: constrained MDP, safe exploration strategies, etc.

Partial observability and non-stationarity

Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.
Solutions to non-stationarity: domain randomization or system identification

Unspecified and multi-objective reward functions

Solutions: CVaR, Distributional DQN

Explainability
Real-time inference
System delays (see also this and this answers)

There's also a more recent and related paper An empirical investigation of the challenges of real-world reinforcement learning (2020) by Gabriel Dulac-Arnold et al, and here you have the associated code with the experiments.
However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [1, 2]. See also this answer.


Is AI already being used in the drug industry to combat the COVID-19?

A global race is underway to discover a vaccine, drug, or combination of treatments that can disrupt the SARS-CoV-2 virus.

The problem is, there are more than a billion such molecules. A researcher would conceivably want to test each one against the two dozen or so proteins in SARS-CoV-2 to see their effects. Such a project could use every wet lab in the world and still not be completed for centuries.

Computer modelling is a common approach used by academic researchers and pharmaceutical companies as a preliminary, filtering step in drug discovery. However, in this case, even every supercomputer on Earth could not test those billion molecules in a reasonable amount of time.

Folding@home is a distributed computing project run by Stanford University. The aim of the project is to examine how proteins fold and it does this using spare computing power. however, there is a lot of research in progress that are harnessing the potential of Artificial intelligence to develop the potential treatment to combat the COVID-19.

Check this recent article By Tyler Orton in biv focus on how artificial intelligence is used to accelerate the process of Drug Discovery:
Drug research turns to artificial intelligence in COVID-19 fight

Here are the list some companies that are using AI-driven approach for Drug discovery

BlackThorn Therapeutics
xscientia
Insilico Medicine
Insitro
Notable Labs
Standigm
Recursion Pharmaceuticals

The Hong Kong-based company Insilico Medicine, a developer of a comprehensive drug discovery and biomarker development platform GENTRL, and a pioneer in the application of generative adversarial networks (GANs) to drug discovery.

Insilico Medicine, Publishes a paper in September last year titled,
Deep learning enables rapid identification of potent DDR1 kinase inhibitors," in a most reputed journal Nature Biotechnology. The paper describes a timed challenge, where the new artificial intelligence system called Generative Tensorial Reinforcement Learning (GENTRL) designed six novel inhibitors of DDR1, a kinase target implicated in fibrosis and other diseases, in 21 days.

Four compounds were active in biochemical assays, and two were validated in cell-based assays. One lead candidate was tested and demonstrated favourable pharmacokinetics in mice.


Is AI entirely a part of Computer Science?

AI is an amalgamation of many fields, Computer Science plays a major role in imparting "Intelligence" to the machine. Following is a quote from the best selling AI book Artificial Intelligence: A Modern Approach by
Stuart J. Russell and Peter Norvig.

Artificial Intelligence (AI) is a big field, and this is a big book.
We have tried to explore the full breadth of the field, which
encompasses logic, probability, and continuous mathematics;
perception, reasoning, learning, and action; and everything from
microelectronic devices to robotic planetary explorers.

So, the answer to your question is yes, there are other fields that AI depends including mathematics (to optimize AI algorithms), electronic components (sensors, microprocessors, etc.), mechanical actuators (hydraulic, pneumatic, electric, etc.)
I highly recommend the book if you are looking for a starting point.


Backpropagation: Chain Rule to the Third Last Layer

I finally solved it out but it's long, it's not only chain rule, it includes quotient rule too. And, this is only the third last layer, once a DNN has more layers then it's more complex.

$\dfrac{dLoss}{dW_1} = \dfrac{d}{dW_1}(Out-Y) = \dfrac{d}{dW_1}Out = \dfrac{d}{dW_1}(H_4W_{13} + H_5W_{14}) \\= \dfrac{d}{dW_1}H_4W_{13} + \dfrac{d}{dW_1}H_5W_{14} \\= W_{13} \times \dfrac{d}{dW_1}H_4 + W_{14} \times \dfrac{d}{dW_1}H_5 \\= W_{13} \times \dfrac{d}{dW_1}(H_1W_7 + H_2W_8 + H_3W_9) + W_{14} \times \dfrac{d}{dW_1}(H_1W_{10} + H_2W_{11} + H_3W_{12}) \\= W_{13} \times \dfrac{d}{dW_1}H_1W_7 + W_{14} \times \dfrac{d}{dW_1}H_1W_{10} \\= W_{13}W_7 \times \dfrac{d}{dW_1}H_1 + W_{14}W_{10} \times \dfrac{d}{dW_1}H_1 \\= (W_{13}W_7 + W_{14}W_{10}) \times \dfrac{d}{dW_1}(X_1W_1 + X_2W_2) \\= (W_{13}W_7 + W_{14}W_{10}) \times X_1$


How to use computer vision to find corners of a soccer field based on location coordinates?

Finding lines in an image often leads to the Hough line transform. Many libraries implement it, including OpenCV. Getting the lines should answer subsequent questions (and it doesn't, please consider having one question per post, and some other sites on StackExchange may be better suited than AI.SE).

Alternative approaches based on Machine Learning may also exist. It may be interesting at this point to look into libraries that do human pose/gait recognition. Some library may be repurposed to recognize the "field gait", assuming the field outer mark is its "gait"-equivalent.


How can the importance sampling ratio be different than zero when the target policy is deterministic?

You're correct, when the target policy $\pi$ is deterministic, the importance sampling ratio will be $\geq 1$ along the trajectory where the behaviour policy $b$ happened to have taken the same actions that $\pi$ would have taken, and turns to $0$ as soon as $b$ makes one "mistake" (selects an action that $\pi$ would not have selected).

Before importance sampling is introduced in the book, I believe the only off-policy method you will have seen is one-step $Q$-learning, which can only propagate observations back along exactly one step. With the importance sampling ratio, you can often do a bit better. You're right, there is a risk that it turns to $0$ rather quickly (especially when $\pi$ and $b$ are very different from each other), at which point it essentially "truncates" your trajectory and ignores all subsequent experience... but that still can be better than one-step, there is a chance that the ratio will remain $1$ for at least a few steps. It will occasionally still only permit $1$-step returns, but also sometimes $2$-step returns, sometimes $3$-step returns, etc., which is often better than only having $1$-step returns.

Whenever the importance sampling ratio is not $0$, it can also give more emphasis to the observations resulting from trajectories that would be common under $\pi$, but are uncommon under $b$. Such trajectories will have a ratio $> 1$. Emphasizing such trajectories more can be beneficial, because they don't get experienced often under $b$, so without the extra emphasis it can be difficult to properly learn what would have happened under $\pi$.

Of course, it is also worth noting that your quote says (emphasis mine):

  The target policy $\pi$ [...] may be deterministic [...]

It says that $\pi$ may be deterministic (and in practice it very often is, because we very often take $\pi$ to be the greedy policy)... but sometimes it won't be. The entire approach using the importance sampling ratio is well-defined also for cases where we choose $\pi$ not to be deterministic. In such situations, we'll often be able to propagate observations over significantly longer trajectories (although there is also a risk of excessive variance and/or numeric instability when $b$ selects actions that are highly unlikely according to $b$, but highly likely according to $\pi$).


In what ways can connectionist A.I. be integrated with GOFAI?

A neural net with even a single hidden layer is capable of Universal function approximation - it can approximate any continuous function 'as closely as you like'.

Hence, one option would be to look for GOFAI applications that would benefit from this property - for example, in state-space search approaches where the utility of a state is not readily defined in advance, and could instead be learned.


Is this referring to the true underlying distribution, or the distribution of our sample?

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution.

"The true underlying distribution" is the closest "distribution" that is explicitly mentioned as such in the part of the text preceding the phrase "this distribution", so that's what it's referring to. For clarity, I've put the two things that are "the same" in bold in the above quote.


In attention models with multiple layers, are weight matrices shared across layers?

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples:
ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, W_i^K, W_i^V \right\}$ across all Transformer layers for large networks either result in the same accuracy or slightly improved accuracy, while significantly reducing model size. Sharing the position-wise FFN layer though hindered performance.
Text-to-Text Transfer Transformer shows that sharing the weights between encoder and decoder layer of the transformer (so e.g. layer 1 encoding weights = layer 1 decoding weights) barely affected accuracy (it dropped by 0.5%), but the model size is halved.
I'm sure there are more papers I have forgotten. Sharing weights is still an active area of research, but for vanilla transformers it is assumed they do not share weights.


Assuming an AI can die, who manages the state?

Following on from your own software verification-based answer to this question, it seems clear that ordinary (i.e. physical), notions of death or imprisonment are not strong enough constraints on an AI (since it's always possible that a state snapshot has been or can be made).

What is therefore needed is some means of moving the AI into a 'mentally constrained' state, so that (as per the 'formal AI death' paper) what it can subsequently do is limited, even if escapes from an AI-box or is re-instantiated.

One might imagine that this could be done via a form of two-level dialogue, in which:

The AI is supplied with percepts intended to further constrain it
("explaining the error of it's ways", if you like).
Its state snapshot is then examined to try and get some indication of whether it is being appropriately persuaded.

In principle, 1. could be done by a human programmer/psychiatrist/philosopher while 2. could be simulated via a 'black box' method such as Monte Carlo Tree Search.

However, is seems likely that this would in general be a monstrously lengthy process that would be better done by a supervisory AI which combined both steps (and which could use more 'whitebox' analysis methods for 2.).

So, to answer the question of "who manages the state", the conclusion seems to be: "another AI" (or at least a program that's highly competent at all of percept generation/pattern recognition/AI simulation).


Can a neural network be trained on a dataset containing only values for true output for a classification problem?

Yes, you can and the answer is One-Class Classification. A well-written resource to understand is this.


Why exactly do neural networks require i.i.d. data?

There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - that it has been sampled fairly from the population that the learning algorithm has been set up to approximate.
The term i.i.d. stands for "independent and identically distributed". If you pick trajectories from RL then the sampling (per individual* record as used to train with) is not independent. Even if you choose the start of the trajectory randomly, you will have made one random choice then your remaining choices are made according to the trajectory - technically they are chosen by the policy and environment dynamics over a single step, which is usually not enough to make the second, third etc steps from a trajectory fully independent random samples from the training population. To make the sampled observations independent, you should make a new random choice, and to be identically distributed that choice has to be made fairly over the whole dataset.
If a subset of samples drawn for a mini-batch (or as consecutive individual samples) are correlated for an online algorithm like gradient descent, it causes a problem. For the algorithm to converge towards a globally optimal solution, it needs errors and the gradients they generate to be unbiased samples of the "true" function gradients across the loss function. Whilst correlated data from a trajectory does not do this at all - it exhibits a strong sampling bias, causing weight updates to move consistently in the wrong direction.
You can demonstrate this effect trying to learn any simple function like $y = x^2$ with $x$ from $-1$ to $+1$. Training a NN with randomly selected $x,y$ pairs results in far better results than training it serially with ~2000 records starting $(-1.000, 1.000) (-0.999, 0.998) (-0.998, 0.996)$ etc

So, why do feed-forward NNs required the i.i.d. assumption but not RNNs?

RNNs do need i.i.d. data, but the "unit" of sampling here is each sequence.

* By "individually" I mean in terms of how they are used in the training process. A "unit" here is the smallest set of records that leads to a complete measure of loss in your optimisation.
It is ok to draw longer trajectories at random if your smallest unit of measurable error is from a trajectory, as in Monte Carlo or TD($\lambda$) for RL, or for RNNs. There can be subtle problems with this - the trajectories still have to be i.i.d. as in they represent fair samples from the assumed population of trajectories.
If you are working at the level of longer trajectories, typically this reduces systemic bias: In RL it reduces bias from initial conditions due to bootstrap process. For RNNs I am less sure what this would mean, but suspect there is some equivalent. Typically, it also increases variance, meaning you may need more training samples. There is a bias/variance trade-off, which is why setting $\lambda$ somewhere between 0 and 1 in TD($\lambda$) is often the optimal choice. Note in RL this is a different source of bias and variance than discussed when considering number of parameters and regularisation in supervised learning.


What introductory books to reinforcement learning do you know, and how do they approach this topic?

In addition to the ones you mentioned, I would add Algorithms of Reinforcement Learning by Csaba Szepesvári. There is a number of professors who use it as a reference in their RL teaching materials (for example this one).
It generally follows the same outline as Sutton & Barto's book (except the part on bandits, it is included in the Chapter on Control). In fact, it may be considered as a condensed version of Sutton & Barto (about 100 pages). In addition, it's freely available online.
I like the author's justification as to why he wrote this book, so I'm just going to quote it:

Why did I write this book? Good question! There exist a good number of
really great books on Reinforcement Learning. So why a new book? I had
selfish reasons: I wanted a short book, which nevertheless contained
the major ideas underlying state-of-the-art RL algorithms (back in
2010), a discussion of their relative strengths and weaknesses, with
hints on what is known (and not known, but would be good to know)
about these algorithms.


Which unsupervised anomaly detection algorithms are there?

If you are OK to use python, thy novelty-detection with sklearn:

https://scikit-learn.org/stable/modules/outlier_detection.html


transform Location data into int which will be used as input to ML model

In order to accurately input location data into a machine learning model it really depends on what your goal is and what type of algorithm you are working with.  If you are working with a strictly numerical algorithm and your data seems to be spread far apart, it might be easier to convert your country-state-city location to a longitude, latitude feature where the exact value is the centroid location of the given city.  This kaggle post has a good writeup of how to run build features for geo-spatial data.

However, if you have only a small number of different locations per identifier (country, city, state) you could just represent them as seperate location classes and subclasses like you would for machine learning detection model.  You could think "vehicle" -> "bike" -> "road-bike" similarly to "country" -> "state" -> "city".  In this case, you would want to look at hierachical methods for machine learning and see some ways they represent their data.  Although I would say this method is more frowned upon for larger datasets, for smaller datasets it might be a better option.


Is there any research on models that provide uncertainty estimation?

Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, Monte Carlo Dropout (MC dropout) or Bayesian neural networks. In theory, these techniques look promising. In practice, I don't think they are the ultimate solution to the problem of uncertainty estimation in deep learning. In fact, e.g. in the case of Bayesian neural networks, they have some disadvantages, such as more parameters to tweak and save.


There may be a conceptual disconnect between the term Semantic Detection and the task of Head Tracking, since sequential recognition of an object in a set of visual samples representing continuous movement isn't technically a semantics problem.  Although a mapping strategy that works with semantic processing may somehow apply to, with appropriate modifications, the mapping of obstructions and tracked objects and their relationship in space-time.

Tracking a head is primarily a surface fitting convergence problem, where the convergence goal and the process that achieved it must be sustainable, especially if the system is required to take actions during the tracking operation.  Distinguishing heads from one another requires a high degree of reliability and accuracy in this convergent continuity.  That is the primary challenge.

Drawing boxes and displaying counts are trivial operations once the head tracking works.

These are some models listed in the question or improvements over them.

Fast LBP Face Detection on low-power SIMD architectures, Olexa Bilaniuk, Ehsan Fazl-Ersi, Robert Laganiere, Christina Xu, Daniel Laroche, Craig Moulder
*A Real-Time AdaBoost Cascade Face Tracker Based on Likelihood Map and Optical Flow, Ranftl, Alonso-Fernandez, Karlsson, Bigun, IET Biometrics, 2017, Vol. 6, no 6, p. 468-477
A Real-Time AdaBoost Cascade Face Tracker Based on Likelihood Map and Optical Flow, IET Biometrics, 6(6): 468-477
A New Approach to Urban Pedestrian Detection for Automatic Braking, Alberto Broggi et. al. IEEE Transactions on Intelligent Transportation Systems, Vol. 10, No. 4, 2009
Mask R-CNN, Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, Facebook AI Research (FAIR)

One seasoned expert spoke to a particular detail in the final question.

  The thought process experts use to choose a model has no equivalent point value in StackExchange reputation.  As experts retire, some may be willing to sell the professional heuristics they acquired over decades of research and development work if they haven't done so already to a book publisher.

Nonetheless, the above links are a good furtherance of the thought process begun in this question along a direction of continued effective thought.  The following steps provide a road map for how to develop approaches in general.

Read the academic materials until you understand.
Determine your selection criteria.
Test the options against that criteria.
Pick the winner.
If something blocks progress with that one, pick the runner up.


Is AlphaFold just making a good estimate of the protein structure?

AlphaFold (version 1 and 2) predicts (so estimates) the 3D shape of the protein from the sequence of amino acids. AlphaFold's performance is measured with the global distance test (GDT), which is a measure of similarity between two protein structures (the prediction and the ground-truth) that ranges from 0 to 100.
There is a short video and a longer one (both by DeepMind) that summarise the issue of protein folding, how it is important, how well AlphaFold approximately solves it (in the competition Critical Assessment of protein Structure Prediction (CASP)), i.e. AlphaFold 2 achieves a median GDT score of 92.4 (and 87 on the hardest proteins), which is a lot higher than AlphaFold 1's GDT score of 58 (which was the highest achieved score at the time), where, according to John Moult (president of CASP), a score around 90 is considered a satisfactory solution to the protein folding problem. You can find more details about AlphaFold 2 in this DeepMind blog post and about AlphaFold 1 in this other blog post or the associated paper published in Nature this year. You can find the code for AlphaFold 1 here, but there are other community/open-source implementations.
Despite the importance of the problem and achievement, there is clearly a lot of hype about this breakthrough (given also that it was achieved by DeepMind). This is also discussed in this video by Lex Fridman.


How to connect AI neural network processor to laptop?

Check of deep learning box on the internet.
You will have to think hard in order to make a good and cost effective one.

But there are a few people who have made one of their own and shared their experiences.

Here are some medium articles:

Building your own Deep Learning dream machine

Building a 270* Teraflops Deep Learning Box for Under $10,000


What does deep learning offer with respect to standard machine learning?

Deep learning allows you to solve complex problems without necessarily being able to specify the important "features" or key input variables for the model in advance.

To give an example, a problem that may be easily tackled without deep learning could be predicting the frequency and claim amounts of insurance vehicle claims,  given historical claim data that may include various attributes of the policy holder and their vehicle.  In this example, the "features" to be specified in the model are the known attributes of policy holder and vehicle.  The model will then attempt to utilise these features to make predictions.

On the other hand,  facial recognition is a problem more suited to deep learning algorithms.  This is because it is difficult to manually identify what combinations of pixels may be important features to include in a conventional machine learning model.  A multi-layered neural network however has the potential to identify/create the important features itself, which may include for example eyes, nose and mouth, and then utilise these features to recognise faces and other objects.


How can I train a neural network if I don't have enough data?

You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (Self-training). With augmentation you grant the system way more robustness so i would really recommend this. For example this github. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.

You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.


What is ergodicity in a Markov Decision Process (MDP)?

In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). However, the unichain assumption does not mean that every policy will eventually visit every state. I believe your confusion arises from the difference between unichain and more constrained ergodic MDPs.
Puterman defines that a MDP is (emphasis in the following mine):

recurrent or ergodic if the state transition matrix corresponding to every deterministic stationary policy consists of a single recurrent
class, and
unichain, if the state transition matrix corresponding to every deterministic stationary policy is unichain, that is, it consists of a
single recurrent class and a possibly empty set of transient states.

Before unpacking this further, let's first recap what recurrent and transient states in a Markov chain are. The following definitions can be found in Puterman, Appendix A.2. For a state $s$, associate two random variables $\nu_s$ and $\tau_s$ that represent the number of visits and the time of the first visit (or first return if the chain starts in $s$) to state $s$. A state $s$ is

recurrent (sometimes also called ergodic), if $P_s(\tau_s < \infty) = 1$, that is, the state is eventually visited (or returned to), and
transient, if $\mathbb{E}[\tau_s] = \infty$ and $P_s(\tau_s < \infty) < 1$.

It is also true that $s$ is recurrent if and only if $\mathbb{E}[\nu_s] = \infty$, i.e., it is visited infinitely often, and $s$ is transient if and only if $\mathbb{E}[\nu_s] < \infty$, i.e., it is visited only finitely often (and thereby never again after some finite time).
So let's now return to the two types of MDPs above. Consider an arbitrary deterministic stationary policy $\pi$ which maps any state $s$ to an action $a = \pi(s)$. If the MDP is ergodic, then the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class (it does not matter in which state $s_0$ the chain starts, the same stationary distribution is reached). There is a single class of recurrent states, i.e., all states are recurrent, therefore any $s$ is visited infinitely often and $\mu_\pi(s) > 0$.
Now, if the MDP is unichain, then once again the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class. But, importantly, there may exist a policy $\pi$ for which the set of transient states in the induced Markov chain is not empty. Because any transient state $s$ will only be visited a finite number of times, in the (infinite horizon) stationary distribution $\mu_\pi(s)=0$!
So indeed, if the MDP is of the stricter ergodic type, every policy will eventually visit every state. This is not true for unichain MDPs however.
A final remark: some authors define a policy as ergodic (e.g., Kearns & Singh, 2002), if the resulting Markov chain over states is ergodic (i.e., has a unique stationary distribution). The unichain MDP is a type of MDP where every policy is ergodic.
References:
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming.
Kearns & Singh. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49, 209–232, 2002


Can variations in microphones used in training set and test set impact the accuracy of speech recognition models?

Yes it can. However, other differences between training and test data with audio could have greater effect:

Identity of the speaker (including effects from gender, age, physical build, local accent, amongst others)
Acoustics of the recording environment (including proximity to the microphone, size of space, presence of hard surfaces, background noise)

If any of these may vary from your training data, then it becomes harder to predict your generalised accuracy during training and early model selection.

One possibility is to ensure your cross-validation set (which you absolutely should have) also separates data out by things that will vary from training to test. So instead of random train/cv split, you split by data that is key for generalisation. This is sometimes called a stratified train/test split.

If your only concern is variation in microphone, then split your train/cv sets by microphone type. You will get a better assessment early on in the model selection process how well the training is generalising, and can focus your search on models that do well despite this expected difference.


What kind of neural network architecture is suitable for variable length block-like time series data?

(this response should be a comment but I don't have yet the reputation to comment).

If I'm understanding your problem correctly you have a variable number of input which have an order and only one output ? It look like the kind of task where you could use recurrent neural network (the most common ones are the LSTM and GRU).

If you use a recurrent neural network you could (if you have the timestamps of your data) cut the hourly interval into smaller time step to help detect pattern.


How to store datasets of lexical connections?

I would think you could use a graph database, perhaps Neo4J or Titan or something of that nature.  Or, if you want a simple file format, you could use one of the many formats that exist for representing graphs.  You can find a list and overview of some of them here.

Another option would be to store them in RDF using a triplestore like Jena.


Do RNNs/LSTMs really need to be sequential?

You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
Can RNNs/LSTMs do that? No.
Can Attention (i.e. Transformers) do it? Yes.
The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.
In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.
Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.


Why do we get a three-dimensional output after a convolutional layer?

If you have a $h_i \times w_i \times d_i$ input, where $h_i, w_i$ and $d_i$ respectively refer to the height, width and depth of the input, then we usually apply $m$ $h_k \times w_k \times d_i$ kernels (or filters) to this input (with the appropriate stride and padding), where $m$ is usually a hyper-parameter. So, after the application of $m$ kernels, you will obtain $m$ $h_o \times w_o \times 1$ so-called feature maps (also known as activation maps), which are usually concatenated along the depth dimension, hence your output will have a depth of $m$ (given that the application of a kernel to the input usually produces a two-dimensional output). For this reason, the output is usually referred to as output volume.

In the context of CNNs, the kernels are learned, so they are not constant (at least, during the learning process, but, after training, they usually remain constant, unless you perform continual lifelong learning). Each kernel will be different from any other kernel, so each kernel will be doing a different convolution with the input (with respect to the other kernels), therefore, each kernel will be responsible for filtering (or detecting) a specific and different (with respect to the other kernels) feature of the input, which can, for example, be the initial image or the output of another convolutional layer.


Can the addition of unnoticeable noise to images be used to create subliminals?

I would say no, at least not now. To achieve that you need to have a model that represents the structure of a human brain at a very detailed level, so that inherits the same flaws.


What approach should I take to model forecasting problem in machine learning?

In general, this type of problem is called a regression problem since the target variable (i.e. travel time) can take any value in a continuous domain. In theory, you can use any regression algorithms (a subset supervised learning techniques) to solve this problem. Some of the most popular ones are linear regression, K-nearest neighbor (regressor), and neural networks.

As you observed already, different algorithms result in (sometimes significantly) different results. Also, the parameter configurations (e.g., number of hidden layers in Neural Networks) can make a big difference. Sometimes, ensembling different models can be helpful, but in general, you should try to avoid overfitting (when your model is more complex than your data such that it memorizes the training set instead of learning it!). That may result in a very good performance on your training set but perform very poorly on your professor's testing set.

What I would do is:

exploring the dataset to see what are the contributing factor in travel time (any correlation between the columns).
cleaning and preprocessing my dataset (duplicates, null values, outliers)
reshaping my dataset if needed (normalizing some columns, merging or splitting columns)
dividing my dataset to training and evaluation subsets (so I train on one part and test on the other part to avoid overfitting)
choosing a simple baseline, applying and measuring the accuracy metrics.
trying to fine tune parameters of my baseline or trying other more advanced techniques.
comparing the results and improving any part of the pipeline when necessary (more/less cleaning, parameter tuning, ensembling).


What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I had a very similar issue as you did with the dimensions. Here's the rundown:

Every node you see inside the LSTM cell has the exact same output dimensions, including the cell state. Otherwise, you'll see with the forget gate and output gate, how could you possible do an element wise multiplication with the cell state? They have to have the same dimensions in order for that to work.

Using an example where n_hiddenunits = 256:

Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256

Now this can obviously be problematic if you want the LSTM to output, say, a one hot vector of size 5. So to do this, a softmax layer is slapped onto the end of the hidden state, to convert it to the correct dimension. So just a standard FFNN with normal weights (no bias', because softmax). Now, also imagining that we input a one hot vector of size 5:

input size: 5
total input size to all gates: 256+5 = 261 (the hidden state and input are appended)
Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256
Final output size: 5

That is the final dimensions of the cell.


How is the state-value function expressed as a product of sums?

A quick review of resolving expectations: If you know that a discrete random variable $X$, drawn from set $\mathcal{X}$ has probability distribution $p(x) = \mathbf{Pr}\{X=x \}$, then
$$\mathbb{E}[X] = \sum_{x \in \mathcal{X}} xp(x)$$
This equation is the core of what is going on when resolving the expectation in your quoted equation.
Resolving the expectation to show how the value function of a state relates to the possible next rewards and future states means summing up all possible rewards and next states. There are two components to the distribution over the single step involved - the policy $\pi(a|s)$, and the state progression $P^a_{ss'}$. As they are both independent probabilities, they need to be multiplied to establish the combined probability of any specific trajectory.
So, looking at only a single trajectory starting from state $s$, the trajectory of selecting action $a$ and ending up in state $s'$ has a probability of:
$$p_{\pi}(a,s'|s) = \pi(a|s) P^a_{ss'}$$
Iterating over all possible trajectories to get the expected value of some function of the end of the trajectory $f(s,a,s')$ looks like this:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\sum_{s'}P_{ss'}^a f(s,a,s')$$
It is important to note that the sums are nested here, not separately resolved then multiplied. This is standard notation, but you could add some brackets to show it:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\left(\sum_{s'}\left(P_{ss'}^a f(s,a,s')\right)\right)$$
In the equation from the book, $f(s,a,s') = R_{ss'}^a + \gamma v_{\pi}(s')$


Is a basic neural network architecture better with small datasets?

I'm not aware of a direct way for finding the best NN architecture for a given task, but the recommended way, as far as I know, is to devise a network that can overfit the training data, and then apply regularization on top of it.
That way, you can be almost sure you're not underfitting/underperforming due to network capacity.


What are standard datasets for fully connected neural networks?

You can use MNIST obviously but I'd also suggest you have a look at UC Irvine's datasets: https://archive.ics.uci.edu/ml/datasets.php


"Regression", is to estimate a continuous value as a function of some other parameters.

There are different forms of regression, such as linear regression and logistic regression, that differ in the assumptions they make on the data.

You could try looking up those for a start.


Are there any online competitions for Reinforcement Learning?

AICrowd has numerous challenges in the domain, with some very interesting challenges running currently. Here is a short list:

https://www.aicrowd.com/challenges?&categories=reinforcement-learning (Complete list with RL tag)
https://www.aicrowd.com/challenges/neurips-2020-procgen-competition
https://www.aicrowd.com/challenges/neurips-2020-flatland-challenge
https://www.aicrowd.com/challenges/neurips-2019-minerl-competition

Hope this helps!


Building ML to finding the closest matching image based on users drawing

There is a good project you can research to find information on this specific task, it is Google quickdraw. It has 50 million drawings across 345 categories and it was used in Google autoDraw. AutoDraw guesses what you’re trying to draw.

You can use quickdraw dataset to train a neural network (Sequence-to-Sequence Variational Autoencoder) in order to get draws from your Sketches. for more info on this read this papper

Classified Sketches into categories is a difficult task because Sketches are a high-level representation that does not always convey enough information to distinguish between different categories.

The the best approach will depend on your specific needs


Is the stride applied both in the horizontal and vertical directions in convolutional neural networks?

Yes, in Keras this is simply implemented by using a tuple for the stride argument of a convolutional layer, with each element of the tuple corresponding to the stride of each dimension.


How to transform inputs and extract useful outputs in a Neural Network?

This is a good question which I wrestled with myself when first trying to code an ANN.

Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described here.

Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.

How to prepare the input data:

The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range.

So if you want to input the square footage of the house into a neuron, you need to normalize the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to z-score-normalize the inputs.

Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. Store the mean and standard deviation---you'll need this information to normalize new square footage values when testing.
Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its z-score.
Do the above for each input variable.

Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just give any hidden layer neurons the outputs of the previous layer directly---they will be able to handle them just fine.

When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the maximum possible house value as its prediction. What should the highest value the network can estimate be? The right approach here simply depends on your application. This is what I did:

Calculate the mean of [the/each] output variable and store it.
Calculate the maximum deviation of the output variable from the mean.
Python: MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])
When the network returns output(s) between -1 and 1, multiply the output by MaxDev and add it to the mean.

Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):

If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)
If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).

Observe that the scheme presented here satisfies these two conditions.

Notice that this scheme would allow your network to only predict house values inside the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.

For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.

If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.

I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.


Why is breadth-first search only optimal when the cost solution is a non-decreasing function?

This is well covered in the corresponding chapters of Russell & Norvig (Ch. 3 & 4). It also depends on the distinction between TREE-SEARCH and GRAPH-SEARCH.

First, note that Breadth-first search also can't handle cost functions that vary between nodes! Breath-first search only cares about the number of moves needed to reach a state, not the total cost of getting there, so if some states are cheaper on a per-move basis, it's not optimal.

Ok, back to the question: basically, the statement you reference applies to Breadth-first search run on a GRAPH-SEARCH problem. In this kind of problem, it is possible to loop back to an earlier state from a later state (like in chess, where you may move your king back and forth).

If the costs for moving between two states are negative, but constant across all the possible moves, then Breadth-first search will not chose to loop between them forever, since it always expands the nearest node to the start by total number of moves, not the cheapest node. Since you could always add one more loop, to make a path shorter, breadth-first search will not find the shortest path (by cost), but will find the shortest path by number of moves instead.


Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Spinning Up by Open Ai.
Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html


Can a program that requires feedback be considered an AI?

There is no "process of AI" as such.  There are many, many different approaches to AI, different ones of which are used in specific applications.

As to whether a purely trial and error approach could be considered AI... I'd offer up a qualified "maybe".  If you do nothing but an exhaustive scan of the solution space, for every trial, then I'd say "No, it's not really any kind of AI".  OTOH, if you're using a knowledge-base of some sort and applying some kind of reasoning (even if it's a heuristic) , and if you have a system that somehow learns from the feedback from the user and gets "smarter" over time, then you're likely working on something that could be considered AI.

All of that said, the exact definition of what is and isn't AI is somewhat fuzzy.   One popular definition is something like "any technology that allows a computer to do something well that currently only humans can do well". So if you're doing something that fits that descriptions, it's possibly an aspect of AI.   And consider again that most people don't really consider "brute force" solutions to be AI.


How to find AI specialists interested in additive Manufacturing

You should probably compile a list of Universities/Instituitions that specialise in Additive Manufacturing, and target graduating students or researchers looking for a new challenge. This may prove more fruitful than targeting the normal channels. You may have to search worldwide, but of course this adds the issue of work visa etc.
There is no easy way to solve this....

Yo can try posting on the Academia forum too.


Is algorithmic bias due to the training dataset used?

As the name implies, algorithmic bias is related with the used algorithm. Due to the way it was programmed or devised, the algorithm will be biased in some of its samples.

From Communications of the ACM:

  [Algorithms] often inadvertently pick up the human biases that are incorporated when the algorithm is programmed, or when humans interact with that algorithm.

A machine learning model can be biased if the wrong data-set is used, of course. That is usually only referred to as bias, and often associated with the bias-variance tradeoff.


Tensorflow: tf.contrib.DNNClassifier(..) or tf.estimator.DNNClassifier(..)

Programming questions are a bit out of scope of this site, but it's a more generic one. For some reason the developers didn't specify it in code, but everything in contrib is experimental. From this discussion:

The distinction between core and contrib is really in core things don't change. Things are backward compatible until release 2.0, and nobody's thinking about that right now.
If you have something in core, it's stable, you should use it. If you have something in contrib, the API may change and depending on your needs you may or may not want to use it.

When a particular class or function is only in contrib, there's basically no choice. But for classes that have "graduated" to the core, contrib version gets deprecated automatically. So you want to use tf.estimator.DNNClassifier.


Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.


Why the non-exploitation of edge labels in current graph convolutions "results in an overly homogeneous view of local graph neighborhoods"?

Consider a two-dimensional convolution layer with 3x3 kernels. The 2d inputs of this layer can be seen as a particular graph with each pixel being a graph node, that is connected to 8 of his neighbors:

The 3x3 kernels of the convolutional layer not only process the information about neighborhood relation between pixels, but also about their relative orientation. For example the [0,0] element of the kernel might represent the weight of the node to NW. And the [1,2] element of the kernel represent the weight the node across the S edge:

Now, if we make a convolution that "doesn't exploit edge labels" then we'll have to forget the labels on the picture above, making us loose directional information:

All we can say now is that the "red" node has those neighbors, but we don't really know how they are oriented relative to it. Since now the sub-graph does not provide any directional information, the learned convolution kernels will be direction-agnostic - in other words, they will be rotation-invariant.


Is an algorithm that is no longer actively learning an AI?

In my view learning is a useful aspect of an AI system, which makes it more resilient towards a changing environment, but it is not a requirement. Early AI systems (Expert systems, theorem solvers) would usually not use learning algorithms, but had a fixed knowledge base.

It also works a lot better with numerical models than with symbolic processing; the proliferation of training data which boosts the most recent AI wave (which is based more on statistical and numerical methods) has made it somewhat necessary for modern systems to have a learning/training component.

However, one needs to distinguish between the initial training, which leads to a production system which then becomes fixed (and will not learn from new input) and those which have a learning component built in as an intrinsic part and will continue to adapt throughout their lifetime.

A cautionary example is Microsoft's "Tay" chatbot. This was still actively learning from conversations and was quickly abused by subjecting it to undesirable input. Switching off the learning component would have been prudent in this case, but it would not have taken the AI out of it.

So, to answer the question: learning is not a required part of an AI system once it is operational. While modern systems generally have some element of learning to build their initial models, older systems were simply hand-coded without any automated learning at all. But one would still regard them as AI system.


Histopathological image vs. natural image

Histopathological images compared to natural images difference so, that in histopathological  images the image needed to be modeled may contain millions of pixels, whereas a recognition from natural image that this is a dog / house / certain person needs significantly less information to be extracted from the picture.

Histopathological images use WSIs (Whole slide image) which contain the part of tissue as a whole. It needs to be split to 256 x 256 pixel patches and those are used for ROI hunting (Region of Interest) and other analysis one by one.

Histopathological images (WSI) are quite rare nowadays and hold privately. More open data would rise the accuracy of machine learning and increase the amount of results that could be found from the data.

Complete source:

https://arxiv.org/pdf/1709.00786v1.pdf


What is the difference between artificial intelligence and cognitive science?

Another impression that I get is that cognitive science is more about trying to find out how the human intelligence or mind works. And that it would use artificial intelligence to make tests or experiments, to test ideas and so forth.

I think that's pretty much it.  I mean, clearly there is some overlap, but I feel like most people who use "cognitive science" are referring more to understanding human intelligence for its own sake.  Artificial Intelligence, OTOH, is more about implementing "intelligence" on a computer, where the techniques used may or may not be influenced by research done under the rubric of cognitive science.


What are some online courses for deep reinforcement learning?

This answer will point the reader to potentially useful resources, but I can't ensure that the courses are good (because I have never followed them).
Free

Reinforcement Learning in the Open AI Gym (a small course that you can find in the YouTube channel suggested in the other answer) by Phil Tabor

The free course Advanced Deep Learning & Reinforcement Learning by DeepMind can be useful. See video 14, which discusses deep RL topics (e.q. DQN).

The free course CS234: Reinforcement Learning Winter 2020 can also be useful. In particular, lesson 6 is about deep reinforcement learning. Here's the course schedule.

Paid

Advanced AI: Deep Reinforcement Learning in Python (at Udemy) seems to be completely dedicated to DRL.

Machine Learning: Beginner Reinforcement Learning in Python  (at Udemy)

Deep Reinforcement Learning 2.0 (at Udemy)

Modern Reinforcement Learning: Deep Q Learning in PyTorch by Phil Tabor

Modern Reinforcement Learning: Actor-Critic Methods by Phil Tabor

In any case, if you are familiar with RL and deep learning topics, I encourage you to directly read the DQN papers (both by DeepMind folks)

Playing Atari with Deep Reinforcement Learning (2013)
Human-level control through deep reinforcement learning (2015)

Of course, deep RL isn't just DQN, but these are two very important papers that you should read.
You should also note that deep RL isn't anything special. It's just RL concepts combined with function approximators and deep learning tricks or deep neural networks. So, if you are really familiar with deep learning and RL (including the usage of neural networks to approximate policies and value functions), you don't need any course to understand deep RL concepts. You can just pick any deep RL paper and you can potentially understand it, although you may require more than 1-2 iterations (but that depends on the person).


Predict frequently purchased items under certain conditions with customer purchasing history data

Let start by the concrete question, and follow talking about the general problem.

a)

The concrete question "find items that are particularly frequently purchased through online stores by paying shipping fee" needs few or none usage of applied AI, just a few of statistics.

The question talks about "item purchased" and "buy method", thus, we have an information database with entries like:

  sale(online,item1).

  sale(shop,item2).

  sale(online,item2).

  sale(online,item2).

  ...

(note records can be repeated)

the percentage of online sales of item "X" is defined as the fraction of sales online of this item, sale(online,X), over the total sales of this item, sale(_,X):

In the previous data examples, p(item1)=1/1, p(item2)=2/3.

High p(X) means items that are preferred for online shopping.

Other probabilities can be defined for similar cases.

b)

About the general case, we are talking about data mining. The are very good packages (open source, ...) for them: weka, IBM DWE, ... . By example, using Weka J48 over a database defined as:

  sale( purchase_identifier, buy method, item )

where "purchase_identifier" must group item that has been purchased in a single buyout (cash ticket). J48 will then provide as answer rules as: item "foo" is usually purchased in online shop when also item "bar" is purchased.


What is the difference between sensitivity analysis and parameter tuning?

I would generally assume that parameter tuning is the process of finding the combination of hyperparameters (e.g., population size, crossover and mutation operators and rates, etc.) that yield the best performance on your problem. When you're thinking about the way that performance varies with parameter choice, this is the "what". What is the best choice of parameters.
Sensitivity analysis is the "how" or "how much". If I change my crossover rate from 0.9 to 1.0, how significant is the change in performance of my algorithm. Is my performance more or less stable across a wide range of choices (good) or is it highly dependent on finding this one little peak in the parameter space and every other choice is much worse (bad).


For which of the following reasons can Hill Climbing get stuck?

This question really looks like a homework problem, in part because it is too vague (what does it mean to 'get stuck' exactly?).

Hill climbing stops when it reaches a local maximum.

Hill climbing is an uninformed search algorithm, so it does not make use of a heuristic.

Hill climbing may or may not stop on a ridge, depending on the implementation. Some common implementations move randomly when equal (but not better) moves are available. Others stop in this situation.

Hill climbing may or may not stop on a plateau, depending on how the implementation handles the case of ties, like with ridges. Whether you think it 'gets stuck' depends a lot on your definition. Implementations that stop when encountering a tie will stop. Implementations that don't stop will eventually exit the plateau through random movements, but it could take infinite time if your movements in the gradient-free region are unlucky.


Emotions can of course be triggered by lots of different things. I think the most rich source could well be socialbots like Mitsuku.com and Zo.ai -- Steve Worswick is the owner of Mitsuku and may be interested in helping you by doing (appropriately filtered) chat log queries. You can get him on Twitter at @Mitsuku.


Extracting algebraic constraints from the input data

I'm assuming the question intends to find the maximally constraining bounding relation from among a large variety of other possible relations.

The bounding relation is likely linked to the chaos idea of attractors$^1$. Some contemporary visualizations of attractors in phase space are visualized as bounds rather than translucent paths or cross-sectional means.

The simple form used as an example, the circle, never actually exist in nature. It is an approximation of a differential equation for an orbit with a eccentricity of zero in three space projected onto a parallel plane. But even if this problem is restricted to $\mathbb{R}^2$, the concepts of attractors may pertain.

If quadratic or cubic approximation is adequate for your purposes then it is a matter of finding the zero terms in the generalization. To find non-polynomial relations, the differential equations are the first things to find, but there may be no equivalent closed form in algebra

[1] http://ijfritz.byethost4.com/Chaos/ch_close.htm?i=1


How does Watson find and evaluate its evidence to the answer?

Watson starts off by searching its massive database of sources for stuff that might be pertinent to the question. Next, it searches through all of the search results and turns them into candidate answers. For example, if one of the search results is an article, Watson might pick the title of the article as a possible answer. After finding all of these candidate answers, it proceeds to iteratively score them to determine which one is best.

The scoring process is very complicated, and involves finding supporting evidence for each answer, and then combining many different scoring algorithms to determine which candidate answer is the best. You can read a more detailed (but still very conceptual) overview here, by the creators of Watson.


Is the definition of machine learning by Mitchell in his book "Machine Learning" valid?

Think of a computer as a Turing Machine--this idea is a model of computation, and all of modern computing is based on the Turing-Church thesis.

Machine and program can be interchangeable—at the end of the day, it's all algorithms, whether hard coded in the form of microchip, or in the form of software.  (Any microchip can be emulated as software.)

Pre-modern computers were mechanical in popular sense. Examples include Babbage's
 difference engine, and mechanical calculators in general.  These led to programmable mechanical calculators and electromechanical computers such as IBM's Harvard Mark I, based on Babbage's notion of an Analytic Engine.

In this context, Machine Learning may also connote software that runs on a machine (hardware) as opposed to human/animal learning, which utilizes a biological medium.

The etymology of "mechanics", and by extension, "machine" is worth looking at.

mechanisma (in the sense of "mechanism")
mechanicus (in the sense of "belonging to mechanics")
mechanema (in the sense of "a trick")

Mechanema is instructive in that math and engineering can be understood as the "trick to doing things". (Machination is not pejorative in this sense, as "noun of action from past participle stem of machinari 'contrive skillfully, to design; to scheme, to plot,'"—one can plot the course of a celestial body.)

Early uses of the mech- root

Aristotle's Μηχανικά ("Mechanical Problems"*)

Euler's Mechanica (Euler was referring to Classical Mechanics, an analytic and predictive science made possible by calculus, which is based on functions.)


How can I detect moving objects in a video by OpenCV without using deep learning techniques?

After a quick scan, it would seem that, in the history of object detection, machine learning has always been at the forefront. Before then, it would just be a heuristic approach.

For a quick answer, here: https://towardsdatascience.com/real-time-object-detection-without-machine-learning-5139b399ee7d

That goes over object detection without using machine learning in OpenCV

That being said, there are some segments of computer vision that are not strictly machine learning. For example, a commonly used algorithm Selective Search for region proposal doesn't use machine learning, and the TF Algorithm for background generation doesn't either.

But when it comes to object detection, the gap between machine learning and all other methods is so sheer that ML is the only method really considered today.


Teaching a NN to manipulate pseudoRNG over a long time scale?

The point of pseudoRNG is to be unmodable and unpredictable, making it hard to train an AI to learn. It would more likely be useful and more efficient to have the equation that the game uses for generation available, so that you can manually make the check, or to just have a list of the loop if the pseudoRNG is based on the time elapsed.


How to turn a ternary constraint into three binary constraints?

Let $D_a$ be the domain for A, and $a_i$ the elements of $D_a$.  Let $D_b$ and $D_c$ work similarly for $B$ and $C$ respectively.

We introduce $D_t = \{t | t = (c_i - b_j, c_i)\}$ for all $c_i$ in $D_c$ and $b_j$ in $D_b$.  We can see that $\{t[0]\}$ (i.e. $\{c_i - b_j, \forall i,j\}$) must be equal to $D_a$.

So we can represent the constraint on $A$ by the relation $R_At = { (a_k, t_l) | a_k = t_l[0]}$ for $a_k$ in $D_a$ and $t_l$ in $D_t$.  That is, $a_k$ must equal the first element in the pair from $t_l$, which means there is a $b_j$ and a $c_i$ consistent with that $a_k$.

The constraint on $B$ is the relation $R_{Bt} = \{(b_j, t_l) | b_j + t_l[0] = t_l[1]\}$.  Since $t_l$[0] = $c_i - b_j$, $t_l$[0] + $b_j$ must equal $c_i$, which is t[1], so if this holds, there is a $b_j$ in $D_b$ that is consistent with $t_l$.

Lastly, $R_{Ct} = \{ (c_i, t_l) | c_i = t_l[1]\}$, which is really just an identity.  So what we've done is solved $A + B = C$ for $A$ and encoded that into $R_{At}$.  From that solution and knowledge of the original $C$ (stored in t[1]) we can re-create $C$ from a valid $B$, which is the $R_{Bt}$ relation.

And lastly $C$ must be in the original set of $C$'s used to build all the $t$'s.  If all three of these constraints hold, then we must have a $C$, a $B$ used to solve for $A$, and an $A$ that matches that solution.

The second part is really just path-consistency.  For every $A$, choose a $B$, and see if there's a possible $C = A + B$.  If so, Add that those values to final domains $D_a$, $D_b$, and $D_c$.  I forget if there's a better performing algorithm than this, but I doubt it since here you can use associativity of addition to avoid checking the opposite order of selecting $A$ and $B$.  In general, you have to consider both orders.


What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

I no longer really use validation that much, but rather only training and testing.  Why? Because I mostly follow Ron Kohavi's (Stanford Univ) approach to CV.  I have done a lot of validation but it seemed to be overkill, essentially causing me to ask why I have this very small-sampled parameter watch on the side from which I am supposed to learn from. You know if an ANN is overfitting when the training error diverges away from testing error -- it's an indication the model is breaking up. See my blog on this.


Backpropagation equation for a variant on the usual Linear Neuron architecture

The forward prop equation is:

$$
Z = (X-A)W - B = XW - AW - B
$$

So the derivatives for $Z$ w.r.t $W$, $A$, $B$ and $X$ should be:

$$
\frac{\partial Z}{\partial W} = X-A  \\
\frac{\partial Z}{\partial A} = - W  \\
\frac{\partial Z}{\partial B} = - 1 \\
\frac{\partial Z}{\partial X} = W
$$

I don't know why he needs the last one though. The first is, like you said, as expected. The other two are wrong, I don't know why he used them in the implementation.


What is the bias-variance trade-off in reinforcement learning?

The bias-variance trade-off that you're referring to has to do with the return estimator. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many sources of randomness, such as stochastic transitions or rewards. Monte Carlo RL algorithms estimate returns by running full trajectories and literally averaging the return achieved for each state. This imposes very few assumptions on the system (in fact, you don't even need the markovian property for these methods), so bias is low. However variance is high since each estimate depends on the literal trajectories that you observe. As such, you'll need many, many trajectories to get a good estimate of the value function.
On the other hand, with TD methods, you estimate returns as $R_t + \gamma V(S_{t+1})$, where $V$ is your estimate of the value function. Using $V$ this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value "piecewise" rather than just via literal trajectories that you happened to witness.


It's a pretty grey area.

I think they are usually talking about the full spectrum of the effort in taking an "AI project" all the way from the drawing board to production. This could easily be anything from a Stock Market Price Prediction model, to deploying a customer service chatbot/Virtual Assistant. I find that people who ask such questions only care about the fact that "AI" is being used to solve a problem and actually is working in the real world.

The effort in the Model would be the standard DataScience/Machine learning process of obtaining data, feature engineering, training and refining your model and eventually deploying it somewhere where it works. It could be in a web browser, it could be in an app, an API endpoint, etc.

For the customer service chatbot it could easily be largely a pure programming project utilising a cloud provider's resource for to conversational capabilities, but as long as it falls under the bracket of "AI" and has been completed and deployed to production/an environment where it works, its end to end.

Disclaimer: Depending on the technical knowledge of who is asking, they might care that you have done some real machine learning work in the development process, and so the example of the bot wouldn't fly.

Hope this helps somehow.


You need to create an active learning loop over the process of the learning. Try to start from a history of tickets and using doc2vec to get the similarity. When you find a bad result in the result of your classifier, then report it and then try to retrain the classifier. Also, you can wait to retrain the model, up to finding the predefined batch0size of the new data which are not in the training set.

Also, to get a better result in the active learning loop, you can testify incoming data by the measuring of the classifier uncertainty over it. If the entropy of the classifier over the data is not in a good situation, you can label the data by the operator (as an oracle) and then up reach to the predefined batch-size, retrain the classifier.

Morevoer, to know better about the active learning process and query strategies follow this link (and other articles in that link like this article).


Predict next event based on previous events and discrete reward values

Your problem does look like it could be a good match to reinforcement learning, or at least the related idea of contextual bandits. Whether or not it would be a good match to the full reinforcement learning algorithm depends on whether any of the data you are processing could be considered part of an environment state, and whether or not that state evolves based on rules that an agent could learn to take advantage of.

Previously, I used only action data to predict the next action using LSTM and GRU. However, how could I use this reward value in this prediction?

There are a few different ways to do this using reinforcement learning theory. The simplest is to build a regression predictor that approximates a sum of future rewards (also known as the return or utility) depending on a proposed action from the current state. Then you could use the value function approximator (formal name for the predictor you just built) to predict results from each possible action and pick the maximising one. It is possible to learn such a value function from a historical dataset using methods such as Q learning.
The subject is too complex to teach from scratch in a single answer here. A good learning resource is Reinforcement Learning: An Introduction by Sutton & Barto, which the authors have made available for free.

However, as the rewards are discrete, I am not sure if RL could do that.

Yes it can. Reinforcement learning just requires that rewards in each situation follow a consistent distribution of values. Always returning discrete values is not a problem, neither is always returning the same value in the same situation. Randomness in the reward value - such as sometimes returning a discrete value and other times not in the same situation - is also OK. You can treat missing values as zero, since you are concerned only with the sum of received rewards, using zero when no value is available has no effect on what will be considered the optimal solution.

Also, is it possible to solve this problem with Inverse RL?

Probably not. Inverse RL is concerned with figuring out the parameters that an existing agent is working with by observing it. For instance you could use it to observe a creature's behaviour and figure out which rewards were more valuable to it. In your case you have the reward values, so you don't need to figure them out.
Caveat: You need to figure out what constitutes state in your environment. If there is some state that can be used for predictions, but the agent's behaviour never changes the state, then you may want to spend some time modelling your problem as a contextual bandit instead. Bandit algorithms are introduced in the same book, Reinforcement Learning: An Introduction, but only as much is needed to teach about the full RL problem - bandit solvers can get far more sophisticated than the book considers.
Note that if the history of agent actions impacts the reward (e.g. it is a matter of timing when to take the right action), then that history is part of the state, and you likely do have a full reinforcement learning problem to solve.


What is a good descriptor for similar objects?

How to develop a program that can sort images by similarity is similar to the problem of how to develop a program that can sort words by how similar they look.

For example:
"theory" is more similar to "teoryyy" than to "abc".

What determines similarity of two words or images are these factors:

how many parts are common to both images
how many parts are new in either image
how many parts are missing in either image
how many parts are zoomed in/out or stretched.
how many parts are displaced

There may be some other rules. By combining these rules you can explain how similarity recognition works. This is at the core of intelligence.


Do algorithms like OpenAI's "think up strategies"?

The key here is think up strategies. If we define this as examining, creating a hypothesis, and testing it as strategizing then yes AI has the ability to strategize. It can examine other users' games, quantifies actions that correlated with victory then test if it gains victory by doing those actions.

Strategy by definition is: a plan of action or policy designed to achieve a major or overall aim.

AI can not classically plan a series of actions designed to achieve a major victory. Instead, it learns the right strategy by testing simulated scenarios, like someone who thinks about the consequences before doing them, but the AI actually has the opportunity to play the game hundreds of times, in order to learn the correct strategy. Similar to Bill Murray in the movie Groundhog Day, learning the ideal day to live. The AI can strategize by experiencing the game over and over until it fine-tunes what an ideal game should be and has seen enough examples of games to not be outwitted by gimmicky strategies.

To summarize, AI can strategize, just in a way fairly different than people.


Which predictive algorithm can be used to predict a number given other numbers?

Impossible to solve until you define an error measurement ( by example $|R-R'|$ or $(R-R')^2$ ) and how this error changes when A, B and C change.
Extreme example: assume $R()$ is random (unrelated to A, B, C values) but static (always same $R(A,B,C)$ for same values of A,B,C). Given some values of A, B, C, you can only answer the value of $R(A,B,C)$ when A,B,C was in the training set. $R(A,B,C)$ is undefined and no predictable when A,B,C was not in the training set.
Moreover, improvements can be done if $R()$ has some properties, by example, if it is possible to state that $R(A,B,C)=R(B,A,C)$ or that $R(A1,B1,C1)=R(A2,B2,C2)$ if $A1+B1+C1=A2+B2+C2$.


When we use the term rationality in AI, it tends to conform to the game theory/decision theory definition of rational agent.

In a solved or tractable game, an agent can have perfect rationality.  If the game is intractable, rationality is necessarily bounded. (Here, "game" can be taken to mean any problem.)

There is also the issue of imperfect information and incomplete information.

Rationality isn't restricted to objectively optimal decisions but includes subjectively optimal decisions, where the the optimality can only be presumed.  (That's why defection is the optimal strategy in 1-shot Prisoner's Dilemma, where the agents don't know the decisionmaking process of the the competitor.)

Rationality here conforms to Russell & Norvig's definition, where it is related to performance in an environment.

What may be rational in one environment may not be rational in a different environment.  Additionally, what may be locally rational for a simple reflex agent will not appear rational from the perspective of an agent with more knowledge, or a learning agent.

Iterated Dilemmas, where there is communication in the form of prior choices, may provide an analogy.  An agent that always defects, even where the competitor has shown willingness to cooperate, may not be regarded as rational because  defecting vs. a cooperative agent does not maximize utility.  A simple reflex agent wouldn't have the capacity to alter its strategy.

However, rationality used in the most general sense might allow that, to the agent making the decision, if the decision is based on achieving an objective, and the decision is reached utilizing the information available to that agent, the decision is may be regarded as rational, regardless of actual optimality.


What are the mathematical prerequisites for an AI researcher?

Good Mathematics Foundation
Begin by ensuring full competency with intermediate algebra and some other foundations of calculus and discrete math, including the terminology and basic concepts within these topics.

Infinite series
Logical proofs
Linear algebra and matrices
Analytic geometry, especially the distinction between local and global extremes (minima and maxima), saddle points, and points of inflection
Set theory
Probability
Statistics

Foundations of Cybernetics
Norbert Wiener, Cybernetics, 1948, MIT Press, contains time series and feedback concepts with clarity and command not seen in subsequent works; it also contains an introduction to information theory beginning with Shannon's log2 formula for a defining the amount of information in a bit. This is important to understand the expansion of the information entropy concept.
Calculus
Find a good calculus book and make sure you have clarity around key theory and application in these categories.

Time series
Infinite series
Convergence — Artificial networks ideally converge to an optimum during learning.
Partial differentials
Jacobian and Hessian matrices
Multivariate math
Boundary regions
Discrete math

Much of that is in Calculus, Strang, MIT, Wellesley-Cambridge Press. Although the PDF is available on the web, it is basic and not particularly deep.  The one in our laboratory's library is Intermediate Calculus, Hurley, Holt Rinehart & Winston, 1980. It is comprehensive and in some ways better laid out than the one I have in my home library, which Princeton uses for sophomores.
Ensure you are comfortable working in spaces beyond ℝ2 (beyond 2D).  For instance, RNNs are often in spaces such as ℝ4 thorugh ℝ7 because of the horizontal, vertical, pixel depth, and movie frame dimensions.
Finite Math
It is unfortunate that no combination of any three books I can think of has all of these.

Directed graphs — Learn this BEFORE trees or circuits (artificial nets) because it is the superset topography of all those configurations
Abstract symbol trees (ASTs)
Advanced set theory
Decision trees
Markov chains
Chaos theory (especially the difference between random and pseudo-random)
Game Theory starting with Von Neumann and Morgenstern's Game Theory, the seminal work in that field
Convergence in discrete systems especially the application of theory to signal saturation in integer, fixed point, or floating-point arithmetic
Statistical means, deviations, correlation, and the more progressive concepts of entropy, relative entropy, and cross-entropy
Curve fitting
Convolution
Probability especially Bayes' Theorem
Algorithmic theory (Gödel's uncertainty theorems and Turing completeness)

Chemistry and Neurology
It is good to recall chemical equilibria from high school chemistry.  Balance plays a key role in more sophisticated AI designs.  Understanding the symbiotic relationship between generative and discriminative models in GANs will help a student further this understanding.
The control functions within biological systems remain a primary source of proofs of concept in artificial intelligence research.  As researchers become more creative in imagining forms of adaptation that do not directly mimic some aspect of biology (still a distance off as of this writing) creativity may play a larger role in AI research objective formulation.
Even so, AI will probably remain a largely interdisciplinary field.


Is there a place where I can read or watch to get an accurate TensorFlow code wise explanation?

The best resource for learning TensorFlow 1.9 and earlier is this course by Stanford. Also additional resources for the entire overview of TensorFlow and its comparisons with NumPy has been made in this video. For hands on models check these videos by Sentdex and also some high level tutorials by Hvass Labs.


The model size (i.e., the number of parameters) is completely independent of the batch size you use during the training.

Whatever the batch size is, your model processes each input within the batch independently. Concretely, batch size affects how many samples your model has to process before it makes an update.

For example, when batch size is 1, it computes the gradients and updates the parameters for each input. When batch size is the whole dataset, it computes the gradients for each input in the dataset independently. It then takes the average of those gradients and uses this average to update the model, so there will be a single update per epoch.

In your case, the model takes an input of size (width=224, height=224, color_channels=3) whatever batch size you use during the training.


Train and Test Accuracy of GRU network not increasing after 2nd epoch

Couple reccomendations:
1) I dont think your overfitting, your test loss is not ever increasing and is staying reasonbly proportional to train loss -- This may indicate that whatever loss your using is not a good indicator of the metric of interest (in this case, it seems you want that to be accuracy, but data is imbalnced so maybe look at avg precision?)
2) Use a lr scheduler, or something like a ReduceLROnPlateau callback to reduce the lr once the loss converges
3) Adding more sources of the underrepresented class is valid but i reccomend just using class weighting. Effectively its the same (as an expectation) and will save you train time.

Good Luck!


Is it worth getting a masters in AI from a top 100 ranked university?

This is a very vague question as well as a really vague statement. Deciding on your career based on "one person who did x said y..." shouldn't be a sole guide to anything in life. Even gathering a multitude of opinions would be useless if your sample is biased on either opinion, good or bad. Personally if I had the rep I would flag the question as it is also a duplicate...your duplicate. Anyways, I 'd like to try and answer  objectively even if I am biased towards a positive attitude by personal and peer experience.

  By worth it I mean will it push your career forward or potentially lead to a good PhD program in the future?

There is nothing that "pushes" your career towards a set goal, other than your own ambition and effort. Receiving the MsC title won't do any good if you don't have the work to show for it. Yes a masters in A.I. is a step in the ladder of pursuing a PhD, but in order to achieve it you need to tick some boxes. There should be a specific sub-domain you target (else I believe a lifetime would not be enough to gain expertise). One should seek out the people in his network that can assist to that, be it professors, supervisors, colleagues or family. Academia does not pay that well , even directors of AI programs do not get as much as in industry thus you should decide which path is more keen to you. But most importantly you need to work hard. Often times a masters opens up mid-level/senior engineer positions, with some experience so it sure does help. Before proceeding to the second part of the question we need to define what "a good PhD" means. Does it mean getting into Stanford? Does it mean finding the right people that have the expertise at whichever university regardless of ranking? You need to decide on that. Then it is easier as there are steps for both. Not definitive ones, but in general you can carve out a plan. For example getting into Stanford would mean that you have excellent grades, a great network for recommendation letters along with published articles. Oh you need to be able to afford it too. The second would suggest that you search tirelessly for universities that have labs with high output, talk to the people in charge and decide which one is best for you.

  A post doc I spoke to about the subject said it's not worth it because the professors at not a top 100 school might not make very good mentors and you might end up wasting your time and money by learning to do things in bad way.

I am sorry to criticize this harshly, but it is among the most preposterous statements I have ever read regarding academia. Let's grab a rankings list. For example the Times one. This would mean that Karlsruhe, Eindhoven, Uppsala, Copenhagen, St Andrews (selection is arbitrary but I know that St Andrews has a great research oriented department) among others do not have a single person that would help you and they might even make you learn things the wrong way...let's be serious please.

It is true that there are bad apples, there exist everywhere. While pursuing a phd you won't attend any lectures though. You will study and do research. That means questioning everything, even your mentors. This is how science advances. Another thing you should consider is what do you expect from the PhD. Do you want to enter the industry or pursue a career in academia? Depending on your answer you should seek different institutions and different people.

I would suggest these sources too:

Ai masters Q#1
ML Learning Reddit

More importantly I would suggest taking time to talk with people from universities. Try tracking down alumni from schools you like and talk to them about their experience, people are always eager to talk about themselves. Then decide what it is that you want to do. A million stack questions want make the decision for you.

DISCLAIMER: This is a positively biased opinion based on personal experience as well as based on the notion that in order to achieve a goal one must have clearly predefined it and work hard towards it.

DISCLAIMER 2: Times just updated their rankings so the previous link is deprecated...not that it matters as most of ranking sites are marketing stunts and unis pay for them shhh


Assuming that you are performing a normal discrete convolution, then you would use the original source image for all calculations, and only replace at the end. As you put it, like this:

  put it in distance image and we don't consider these results when we shift the kernel on source image(It means don't replace the results on source image for next steps calculations)?

The other option, changing in-place during the operation, does not relate to any common use that I know of, but might have some interesting behaviour in studies of cellular automata or similar.


Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.

If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the "normal" transaction or the client will change of bank. According to this post (https://www.quora.com/How-many-transactions-do-typical-banks-process-everyday) just the mastercard alone represent 3.4billion transaction/day, just imagine if 1% of the daily transaction where blocked every day, it would represent 34 million of transactions blocked without any valid reason.

It's different from a lot of classification problem where you want to have balanced dataset, here you try to detect anomalies and, by definition, they are rare so they should be as rare in your dataset.


How is gradient being calculated in Andrej Karpathy's pong code?

logp seen in code is actually logit p which has this story behind:

Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.
The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).

Sigmoid near logp is like follows:

The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.

Source: [1]

In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):

For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).

In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.

Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.
Sources:
[1] https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/


How will morality questions be settled in the domain of self-driving cars?

I don't think these questions will need to be answered.

A self driving car will almost certainly avoid a situation like the ones described well before a human would have and hence would not have to choose.

For example it would slow down as soon as it sees a child close to the road. It will identify and react to the fact that the child starts moving towards the road and act before the situation requires the "drastic" scenarios that we invent.

If for some reason the car has to choose, it can also make the impact/avoidance have the highest chance of not killing the occupant of the vehicle considering how well designed cars are these days and that the speeds involved shouldn't be anything wild like 200mph.

Not to mention a network of cars along with street cameras and sensors would act/work together to resolve as a swarm intelligence so the car ahead or the traffic camera can warn/tell other cars that something they cannot see is a potential hazard. I can go on and on...

In my opinion, the bottom line is a self driving car will not road rage, drive at dangerous speeds in a residential area, get tired and fall asleep, text and drive or drink and drive, etc... I cannot wait.


Are simple animal snares and traps a form of automation? Of computation?

Absolutely these traps and snares are a form of automation.

They take a task--harvesting small animals--which was traditionally done by hunting them, and make the process automatic.  The mechanism requires a human to set up, but its function is automatic.  This is to say that the mechanism operates without human involvement.

Absolutely this is a form of computation.

As DuttaA observed, these machines utilize a simple "IF/THEN" statement.  In the case of the snare:

  IF the hook is displaced from the base, THEN the sapling straightens

These simple machines will also return True or False:

  TRUE: The trap catches an animal
  FALSE: The trap is sprung but empty

The small animals are the input and, potentially, the output, depending on whether the mechanism returns "true".

(The use of "true" has historically included phrases such as "their aim was true" in the sense of shooting an arrow or throwing a spear.)