Get object's orientation or angle after object detection

I think the problem can be phrased (more generally) as a Pose Estimation Problem.  That term might help in obtaining better search results when searching for relevant papers.
One paper that I found on the given topic was this one. Even if it is maybe (for whatever reason) not what you are looking for precisely, it might still contain valuable references to different techniques that might be applicable to your problem at hand.


What is the difference between batches in deep Q learning and supervised learning?

From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)

You are wrong.
The weights are adjusted w.r.t. the average gradient, and this must be calculated using individual loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.

But is this the same in DQNs?

The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}\hat{q}(S_{t+1},a', \theta)$ to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.
In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.
If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the .fit function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of $(\mathbf{x}_i, \mathbf{y}_i)$ examples directly by sampling a minibatch. In RL you get $(\mathbf{s}_i, \mathbf{a}_i, r, \mathbf{s'}_i, done)$ and must construct the $(\mathbf{x}_i, \mathbf{y}_i)$ minibatch from these before passing to your .fit function


How can I draw a conceptual dependency for the statement "Place all ingredients in a bowl and mix thoroughly"?

As it's two clauses ("place in bowl" and "mix"), I would actually use two separate CD structures. The first one is a PTRANS (not ATRANS, as you don't change ownership, but location); the second one is a bit trickier.

Mix can be paraphrased as stir, or move around. You would typically do this with a spoon or similar implement. I would do it as a PROPEL with the instrument spoon and the object the ingredients. PROPEL, according to Schank (1975) is the "application of a physical force to an object".

So the first one is:

(PTRANS
  (ACTOR *you*)
  (OBJECT "all ingredients")
  (TO "bowl"))

And the second is:

(PROPEL
  (ACTOR *you*)
  (OBJECT "ingredients")
  (INSTRUMENT "a spoon")
  (DIRECTION "circular"))

It's debatable if you PROPEL the ingredients with the instrument spoon, or you PROPEL the object spoon with the target/location/direction ingredients.

If you draw them in a graphical representation, you can have both ACTs leading off the shared actor '*you*'; or you could have them as separate graphs linked by 'then' or some other indicator of sequentiality.

Schank, Roger (1975) The primitive ACTs of conceptual dependency, Proc of 1975 workshop on theoretical issues in natural language processing, 34-37


What does it mean when a model "statistically outperforms" another?

Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will always give you a better-performing model.
However, since you can retrain many times to get a distribution of models, you can use a statistical test like the T-Test to say "algorithm A usually produces a better model than algorithm B," which is what they mean by "statistically outperforms."
p-value is usually set by consensus in the field. The higher the p-value, the less confidence you have that there's a statistical difference between the distribution of values being compared. 0.1 might be normal in a field where data is very expensive to collect (like risky, long-term studies of humans), but in machine learning, it's usually easy enough to retrain a model that 0.01 is common, and demonstrates very high confidence. To know more about selecting and interpreting the values, I recommend Wikipedia's page on statistical significance.


Should the range and initial values of weights and biases be adjusted to fit input and output data?

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?

No; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to meaningfully apply neural networks to real-world problems, and fortunately this is not necessary.
The desired situation is for both weights & biases to remain in a relatively small* range around zero, among other reasons because this avoids the exploding & vanishing gradient problems, which is catastrophic for learning; trying to adjust for the scale of our inputs & outputs by scaling accordingly the model weights & biases is the wrong approach, and never followed.
[*How small? Well, see here and here for what a change in just the standard deviation of a zero-mean weight initialization from 1.0 and 0.1 to 0.01 can do to the model performance]
It is a well-established fact by now that neural nets work with normalized inputs, and, depending on the problem, normalized outputs as well; which answers to your objection:

One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.

It depends indeed, but the solution here is to scale the output(s) as well, and not the weights.
Now, it's true that de-scaling the outputs back to their original range in order to be able to meaningfully compare them with the ground truth (and possibly compute more meaningful metrics, like the error on our true output range, and not the scaled one) is AFAIK seldom mentioned in introductory expositions; but this is indeed the correct thing to do (especially for regression problems, where metrics like MSE are scale-sensitive), instead of trying to manually intervene on the weights range. For details, see own answers in How to interpret MSE in Keras Regressor and ANN regression accuracy and loss stuck.

what are the rules of thumb for how to decide about range and initial values of weights and biases?

Range aside, the general rule, as already implied, is to initialize the biases with zeros and the weights with small random values around zero. Nevertheless, the exact details are an area of active research. Currently used initialization schemes that are already integrated into the relevant frameworks (Tensorflow, Keras, Pytorch etc) are the Glorot (or Xavier) and He initializations (for a nice overview, see Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming).
Beyond these routinely-used approaches that have already reached the practitioner's workbench, and moving closer to the front of active theoretical research, the Lottery Ticket Hypothesis (finding "winning" weight initializations that require minimal training) is an ultra-hot topic lately.


Isn't deep fake detection bound to fail?

Not necessarily it depends on the function of the problem space for both the GANs.

A real world example: a batter's reaction time and a pitchers max speed are actual bounded values based on genetics and physics. If the max speed a pitcher can pitch is greater than the max reaction time a human needs to effectively hit against them they will permanently be a better pitcher because the threshold of reaction time.

We don't yet know if a maximum threshold on realistic fake image generation exists or if a threshold on detection exists.

As both reach near perfect accuracy it could be that the amount of nodes needed to detect a nearly perfect generated image from real image is more neurons than atoms in the universe, or conversely the amount of nodes needed to generated a nearly perfect image could reach impossible port potions we won't know until we continue to make better and better networks that close in on the boundary of generation and detection of real vs fake images from a neural network.

Edit:
Let's imagine this problem one adversary edits an image with a colored line of pixels the goal hide the line by editing the image, the student is responsible with finding the line after the adversary changes the image. The problem can become infinitely difficult change all pixels to be the color of the line. The line is impossible to find the adversary always wins, if it finds this solution meaning it is in its reachable problem space based on its hardware capabilities and its learning model.

Deep fake detection is not bound to fail because the limit on the effectiveness of a generative model may have a steeper limit than a discriminator at near optimal performance. I have not seen any paper about this specifically and in fact I believe the discriminator has a more difficult job in most cases I just disagree with deciding at this moment that the detectors are doomed. The combination of creating real images motion and sound perfectly in sink is not a trivial problem, in some scenarios it is basically impossible.


Mathematical calculation behind decision tree classifier with continuous variables

Of course, it depends on what algorithm you use. Typically, a top-down algorithm is used.
You gather all the training data at the root. The base decision is going to be whatever class you have most of. Now, we see if we can do better.
We consider all possible splits. For categorical variables, every value gets its own node. For continuous variables, we can use any possible midpoint between two values (if the values were sorted). For your example, possible splits are Gene-A < 0.5, Gene-A < 17, Gene-B < 1, Gene-B < 3.5, and so on. There is a total of 10 possible splits.
For each of those candidate splits, we measure how much the entropy decreases  (or whatever criterion we selected) and, if this decrease looks significant enough, we introduce this split.
For example. Our entropy in the root node is $-0.4 \log_2 0.4 - 0.6 \log_2 0.6 \approx 0.97$. If we introduce the split Gene-A < 0.5, we get one leaf with entropy $1$ (with 2 data points in it), and one leaf with entropy $0.918$ (with 3 data points). The total decrease of entropy is $0.97 - (\frac25 \times 1 + \frac35 \times 0.918) \approx 0.02$.
For the split Gene-A < 17 we get a decrease of entropy of about $0.3219$.
The best splits for the root are Gene-B < 5.5 and Gene-C < 456. These both reduce the entropy by about $0.42$, which is a substantial improvement.
When you choose a split, you introduce a leaf for the possible outcomes of the test. Here it's just 2 leaves: "yes, the value is smaller than the threshold" or "no, it is not smaller". In every leaf, we collect the training data from the parent that corresponds to this choice. So, if we select Gene-B < 5.5 as our split, the "yes" leaf will contain the first, fourth and fifth data points, and the "no" leaf will contain the other data points.
Then we continue, by repeating the process for each of the leaves. In our example, the "yes" branch can still be split further. A good split would be Gene-C < 288, which results in pure leaves (they have 0 entropy).
When a leaf is "pure enough" (it has very low entropy) or we don't think we have enough data, or the best split for a leaf is not a significant improvement, or we have reached a maximum depth, you stop the process for that leaf. In this leaf you can store the count for all the classes you have in the training data.
If you have to make a prediction for a new data point (from the test set), you start at the root and look at the test (the splitting criterion). For example, for the first test point, we have that Gene-B < 5.5 is false, so we go to the 'no' branch. You continue until you get to a leaf.
In a leaf, you would predict whatever class you have most of. If the user wants, you can also output a probability by giving the proportion.
For the first test point, we go to the "no" branch of the first test, and we end up in a leaf; our prediction would be "Case". For the second test point, we go to the "yes" branch of the first test. Here we test whether 777 < 288, which is false, so we go to the "no" branch, and end up in a leaf. This leaf contains only "Control" cases, so our prediction would be "Control".


Object detection is a regression of the bounding box (rectangle) around the object. In this way, the two ways you suggest are equivalent.

What I suggest you to look at is lane detection for self-driving cars - a well studied problem. It seems that this is a very relevant task (if not the same), so its solution should work for you as well.


What is the difference between meta-learning and zero-shot learning?

First see the definition of meta-learning:

  Meta-learning is a subfield of machine learning where automatic learning algorithms are applied to metadata about machine learning experiments. As of 2017 the term had not found a standard interpretation, however the main goal is to use such metadata to understand how automatic learning can become flexible in solving learning problems, hence to improve the performance of existing learning algorithms or to learn (induce) the learning algorithm itself, hence the alternative term learning to learn.

and zero-shot learning:

  Zero-shot learning is being able to solve a task despite not having received any training examples of that task. For a concrete example, imagine recognizing a category of object in photos without ever having seen a photo of that kind of object before. If you've read a very detailed description of a cat, you might be able to tell what a cat is in a photograph the first time you see it.

As you can see, these are different but meta-learning can be used in zero-shot learning to work better. For example see this article, as an instance.


Drone training, how to train without training data?

It's debatable whether neural networks can find a better solution than PID, if your goal is to simply keep the output around a certain reference point PID should do a perfect job pretty much. If you really want to use "intelligent" control with NN you can look into reinforcement learning.
Few interesting papers that directly adress your problem:

Autonomous helicopter flight via reinforcement learning
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Autonomous Helicopter Control Using Reinforcement Learning Policy Search Methods

You can also find more in the references of these papers.


Unclear definition of a "leaf" and diverging UTC values in the Monte Carlo Tree Search

Q1. When expanding the choices at the leaf node L, do I expand all, a few or just one child?

Expanding all nodes or expanding just one node are both possible. There are different advantages and disadvantages. The obvious disadvantage of immediately expanding them all is that your memory usage will grow more quickly. I suppose that the primary advantage is that you no longer need to keep track separately of "actions that are legal but for which I did not yet create child nodes" and "already-created nodes", which I guess might sometimes lead to better computational efficiency (especially when memory isn't a concern for you, for instance if you do relatively few iterations anyway or if you have a huge amount of memory available).
In the more "modern" variants of MCTS that also use trained policy networks (like all the work inspired by AlphaGo / AlphaGo Zero / AlphaZero), it typically makes most sense to expand all children at once, because the trained network will immediately give you policy values for all children anyway, so then you can prime them all at once with those policy head outputs and also immediately start using all of those values.
In the case where you choose to expand only one child at a time, indeed the terminology of "leaf" nodes becomes confusing / incorrect. I never really like the term "leaf" node in the context of MCTS anyway; they're really not leaf nodes (unless they happen to represent terminal game states), they're just nodes for which did not yet choose to instantiate all the child nodes. It's a bit more verbose, but I prefer referring to them as "nodes that have not yet been fully expanded". That would change the description of the Selection phase to something more like:

Selection: Start from root R. Choose a node L that has not yet been fully expanded (or a terminal node) by iteration of some choice algorithm that determines which child node to choose at each branching.

In case that a few or all child-nodes are chosen during expansion at the leaf, one is faced with the problem that some of those nodes will not be simulated in that MCTS step and subsequently will have a diverging UTC value $w_i/n_i + c \sqrt{\frac{\ln{N_i}}{n_i}}\to \infty$, since $n_i\to 0$.

That's true, but I don't see that as a problem. Suppose we're at a non-fully-expanded node; a node where maybe some legal actions already have corresponding child nodes, or maybe none do, but at least some legal actions still don't have corresponding child nodes. We can view these as having a visit count of $0$, which we can view as leading to a UCB1 value of $\infty$. Since UCT picks nodes (or actions) according to an $\arg\max$ over the UCB1 values, we can simply think of this as UCT always preferring to pick actions that we did not yet expand over actions that we've already expanded and created a child node for. This leads to an implementation where we'll only re-visit a node that we've already visited before if we have already visited each of its possible siblings at least once too.


What's the function that SGD takes to calculate the gradient?

Welcome to AI Stack exchange!

You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing.

So what do we do? Well, conveniently, it is possible to find the gradient of segments of a function, without having to know the entirety of the function itself (it's worth noting that it actually is possible to find the resultant function and with a lot of effort find it's derivative. This proves to be much more work than it's worth though, as we don't need the general derivative that tells us what the gradient is for whatever input we give it, we only need the derivative at the specific input we just fed through the network).

This might be hard to understand, but if you're familiar with the chain rule, it might make a bit more sense. The chain rule essentially allows you to split a function into components, and find the gradient of those specific components. By combining all of that, you end up with some nice gradients at each weight/bias with respect to the loss function. Take the negative of the gradient, and you now have the change required to decrease the loss function.

This is obviously quite hard to understand without an example, so here's the best one I've ever found, that helped me very much.

Also, as a side note, the whole mini batch thing is used to minimise catastrophic forgetting (where the network begins to "unlearn" old inputs). To deal with minibatchs, what you do is take input individually, then find the gradient for all weights and bias' for that specific input and remember the changes you want to make. Do that for all inputs in the minibatch, then finally add all the changes together to get the resultant best change  for each weight and bias. Only then do you update the weights and bias'.

Let me know if you have any further questions


Can programs like AlphaGo be said to be means of dealing with computational intractability?

I think that the technique AlphaGo used to solve the computational intractability problem of the search space are not new, it uses the Monte Carlo Tree Search. The real innovation in AlphaGo was to figure out how to compute the evaluation function of a move, that was the really tricky part. For this they used combinations Deep and Reinforcement learning techniques.


Two Short Answers
The short answer from a theoretical perspective is that ...

A Dynamic Computational Graph is a mutable system represented as a directed graph of data flow between operations.  It can be visualized as shapes containing text connected by arrows, whereby the vertices (shapes) represent operations on the data flowing along the edges (arrows).

Note that such a graph defines dependencies in the data flow but not necessarily the temporal order of the application of operations, which can become ambiguous in the retention of state in vertices or cycles in the graph without an additional mechanism to specify temporal precedence.
The short answer from an applications development perspective is that ...

A Dynamic Computational Graph framework is a system of libraries, interfaces, and components that provide a flexible, programmatic, run time interface that facilitates the construction and modification of systems by connecting a finite but perhaps extensible set of operations.

The PyTorch Framework
PyTorch is the integration of the Torch framework with the Python language and data structuring. Torch competes with Theano, TensorFlow, and other dynamic computational system construction frameworks.

———   Additional Approaches to Understanding   ———

Arbitrary Computational Structures of Arbitrary Discrete Tensors
One of the components that can be used to construct a computational system is an element designed to be interconnected to create neural networks.  The availability of these supports the construction of deep learning and back-propagating neural networks.  A wide variety of other systems involving the assembly of components that work with potentially multidimensional data in arbitrarily defined computational structures can also be constructed.
The data can be scalar values, such as floating-point numbers, integers, or strings, or orthogonal aggregations of these, such as vectors, matrices, cubes, or hyper-cubes.  The operations on the generalization of these data forms are discrete tensors and the structures created from the assembly of tensor operations into working systems are data flows.
Points of Reference for Understanding the Dynamic Computation Concept
Dynamic Computational Graphs are not a particularly new concept, even though the term is relatively new.  The interest in DCGs among computer scientists is not as new as the term Data Scientist.  Nonetheless, the question correctly states that there are few well-written resources available (other than code examples) from which one can learn the overall concept surrounding their emergence and use.
One possible point of reference for beginning to understand DCGs is the Command design pattern which is one of the many design patterns popularized by the proponents of object-oriented design.  The Command design pattern considers operations as computation units the details of which are hidden from the command objects that trigger them.  The Command design pattern is often used in conjunction with the Interpreter design pattern.
In the case of DCGs, the Composite and Facade design patterns are also involved to facilitate the definition of plug-and-play discrete tensor operations that can be assembled together in patterns to form systems.
This particular combination of design patterns to form systems is actually a software abstraction that largely resembles the radical idea that led to the emergence of the Von Neumann architecture, central to most computers today.  Von Neumann's contribution to the emergence of the computer is the idea of permitting arbitrary algorithms containing Boolean logic, arithmetic, and branching to be represented and stored as data -- a program.
Another forerunner of DCGs is expression engines.  Expression engines can be as simple as arithmetic engines and as complex as applications such as Mathematica.  A rules engine is a little like DCGs except that rules engines are declarative and meta-rules for rules engines operate on those declarations.
Programs Manipulating Programs
What these have in common with DCGs is that the flow of data and operations to be applied can be defined at run time.  As with DCGs, some of these software libraries and applications have APIs or other mechanisms to permit operations to be applied to functional details.  It is essentially the idea of a program permitting the manipulation of another program.
Another reference point for understanding this principle at a primitive level is the switch-case statement available in some computer languages.  It is a source code structure whereby the programmer essentially expresses, "We're not sure what must be done, but the value of this variable will tell the real-time execution model what to do from a set of possibilities."
The switch-case statement is an abstraction that extends the idea of deferring the decision as to the direction of computation until run time.  It is the software version of what is done inside the control unit of a contemporary CPU and an extension of the concept of deferring some algorithm details.  A table of functors (function pointers) in C or polymorphism in C++, Java, or Python are other primitive examples.
Dynamic Computation takes the abstraction further.  They defer most if not all of the specification of computations and the relationships between them to run time.   This comprehensive generalization broadens the possibilities of functional modification at run time.
Directed Graph Representation of Computation
That's what the Dynamic Computational model is.  Now for the Graph part.
Once one decides to defer the choice of operations to be performed until run time, a structure is required to hold the operations, their dependency relationships, and perhaps mapping parameters.  Such a representation is more than a syntactic tree (such as a tree representing the hierarchy of source code).  Unlike an assembly language program or machine code, it must be easily and arbitrarily mutable.  It must contain more information than a data flow graph and much more than a memory map.  What must that data structure that specifies the computational structure look like?
Fortunately, any arbitrary, finite, bounded algorithm can be represented as a directed graph of dependencies between specified operations.  In such a graph, the vertices (often represented as nodes of various shapes when displayed) represent operations performed on the data and the edges (often represented as arrows when displayed) are digital representations of information originating resulting from some operation (or system input) and upon which other operations (or system output) depend.
Keep in mind that the directed graph is neither an algorithm (in that a precise sequence of operations is specified) nor a declaration (in that data can be explicitly stored and loops, branches, functions, and modules may be definable and nested).
Most of these Dynamic Computational Graph frameworks and libraries permit the components to do computations on the component input that support machine learning.  Vertices in the directed graph can be simulations of neurons for the construction of a neural net or components that support differential calculus.  These frameworks present possibilities of constructs that can be used for deep learning in a more generalized sense.
In the Context of Computer History
Again, nothing mentioned thus far is new to computer science.  LISP permits computational schematics to be modified by other algorithms.  And generalized input dimensionality and numerocity is built into a number of longstanding plug-and-play interfaces and protocols.  The idea of a framework for learning dates back to the same mid-Twentieth Century period too.
What is new and gaining in popularity is a particular combination of integrated features and the associated set of terminology, an aggregation of existing terminology for each of the features, leading to a wider base for comprehension by those already studying for and working in the software industry.

Contemporary (trendy) flavor of API interfaces
Object orientation
Discrete tensor support
The directed graph abstraction
Interoperability with popular languages and packages that support big data, data mining, machine learning, and statistical analysis
Support for arbitrary and systematic neural network construction
The possibility of dynamic neural network structural adaptation (which facilitates experimentation on neural plasticity)

Many of these frameworks support adaptability to changing input dimensionality (number of dimensions and the range of each).
Similarity to Abstract Symbol Trees in Compilers
A dependency graph of inputs and outputs of operations also appears within abstract symbol trees (AST), which some of the more progressive compilers construct during the interpretation of the source code structure.  The AST is then used to generate assembler instructions or machine instructions in the process of linking with libraries and forming an executable.  The AST is a directed graph that represents the structure of data, operations performed, and the control flow specified by the source code.
The data flow is simply the set of dependencies between operations, which must be inherent in the AST for the AST to be used to create execution instructions in assembler or machine code that precisely follows the algorithm specified in the source code.
Dynamic Computational Graph frameworks, unlike switch-case statements or AST models in compilers, can be manipulated in real-time, optimized, tuned (as in the case of plastic artificial nets), inverted, transformed by tensors, decimated, modified to add or remove entropy, mutated according to a set of rules, or otherwise translated into derivative forms.  They can be stored as files or streams and then retrieved from them.
This is a trivial concept for LISP programmers or those that understand the nature of John von Neumann's recommendation to store operational specifications as data.  In this later sense, a program is a data stream to instruct, through a compiler and operating system, a dynamic computational system implemented in VLSI digital circuitry.
Achieving Adaptable Dimensionality and Numerocity
In the question is the comment that one doesn't, "Need to have data set -- that all the instances within it have the same, a fixed number of inputs." That statement does not promote accurate comprehension.  There are clearer ways to say what is true about input adaptability.
The interface between a DCG and other components of an overall system must be defined, but these interfaces may have dynamic dimensionality or numerocity built into them.  It is a matter of abstraction.
For instance, a discrete tensor object type presents a specific software interface, yet a tensor is a dynamic mathematical concept around which a common interface can be used.  A discrete tensor may be a scalar, a vector, a matrix, a cube, or a hypercube, and the range of dependent variables for each dimension may be variable.
It can be the case that the number of nodes in a layer of the system defined in a Dynamic Computational Graph can be a function of the number of inputs of a particular type, and that too can be a computation deferred to run time.
The framework may be programmed to select layer structure (an extension of the switch-case paradigm again) or calculate parameters defining the structure sizes and depth or activation.  However, these sophisticated features are not what qualifies the framework as a Dynamic Computational Graph framework.
What Qualifies a Framework to Support Dynamic Computational Graphs?
To qualify as a Dynamic Computational Graph framework, the framework must merely support the deferring of the determination of algorithm to run time, therefore opening the door to a plethora of operations on the computational dependencies and data flow at run time.  The basics of the operations deferred must include the specification, manipulation, execution, and storage of the directed graphs that represent systems of operations.
If the specification of the algorithm is NOT deferred until run time but is compiled into the executable designed for a specific operating system with only the traditional flexibility provided by low-level languages such as if-then-else, switch-case, polymorphism, arrays of functors, and variable-length strings, it is considered a static algorithm.
If the operations, the dependencies between them, the data flow, the dimensionality of the data within the flow, and the adaptability of the system to the input numerocity and dimensionality are all variable at run time in a way to create a highly adaptive system, then the algorithm is dynamic in these ways.
Again, LISP programs that operate on LISP programs, rules engines with meta-rule capabilities, expression engines, discrete tensor object libraries, and even relatively simple Command design patterns are all dynamic in some sense, deferring some characteristics to run time.  DCGs are flexible and comprehensive in their capabilities to support arbitrary computational constructs in such a way to create a rich environment for deep learning experimentation and systems implementation.
When to Use Dynamic Computational Graphs
The pros and cons of DCGs are entirely problem-specific.  If you investigate the various dynamic programming concepts above and others that may be closely tied to them in the associated literature, it will become obvious whether you need a Dynamic Computational Graph or not.
In general, if you need to represent an arbitrary and changing model of computation to facilitate the implementation of the deep learning system, mathematical manipulation system, adaptive system, or another flexible and complex software construct that maps to the DCG paradigm well, then a proof of concept using a Dynamic Computational Graph framework is a good first step in defining your software architecture for the problem's solution.
Not all learning software uses DCG's, but they are often a good choice when the systematic and possibly continuous manipulation of an arbitrary computational structure is a run time requirement.


What is the difference between asymmetric and depthwise separable convolution?

They are not the same thing.

asymmetric convolutions work by taking the x and y axes of the image separately. For example performing a convolution with an $(n \times 1)$ kernel before one with a $(1 \times n)$ kernel.

On the other-hand depth-wise separable convolutions separate the spatial and channel components of a 2D convolution. It will first perform the $(n \times n)$ convolution on each channel separately (full kernel shape will be $(n \times n \times 1)$ rather than $(n \times n \times k)$ where $k$ is the number of channels in the previous layer) before doing a $(1 \times 1)$ convolution to learn a relationship between the channels (full kernel size for that being $(1 \times 1 \times k)$)


How to perform binary classification when one class is more predominant than the other?

As described in this post, this problem is known as "unbalanced dataset" problem, which can have different solution approaches. If you use supervised learning, augmentation approaches
could  help. Otherwise, unsupervised approaches need some proper distance measure for outliers detection.


Teaching a neural network to play a card game with two phases

I've played this game. If I remember correctly, a successful strategy (for winning as many tricks you estimated to win) involves continually evaluating how well you're doing. If you're doing very poorly relative to your initial estimation, that's valuable information to have.

I think you should have one network that outputs--at every time step--both an estimate of future tricks won and an action of which card to play. This has the additional advantage of giving you more estimation experience with which to train. After all rounds, you'll know how far off each of your estimations were throughout the game. This should speed up learning on that part of the game.


Why is an average of all returns used to update the value in the first-visit MC control?

Why don't they just update the value with a weight for the value from previous episodes $\alpha$ and a weight $1- \alpha$ for the new episode return as it is done in TD-Learning?

In my opinion, this is a mistake in the book. I went back and checked that this is still the same in the finished second edition, and it is still there.

Keeping all returns and taking averages works fine in a prediction scenario with a fixed policy, and it is somewhat simpler and more intuitive to explain as "this is the mean value" (efficiency can come later, after comprehension). However, the pseudo-code is incorrect for a control scenario where older returns (that you still refer in the list) will not reflect the current policy, so will be biased.

In practice, no-one really uses the MC Control algorithms as written. Nearly always a learning rate parameter, $\alpha$ is used to update estimates.

  What is the difference?

In a prediction scenario, where you want to evaluate a fixed policy, it would be slightly more efficient to use a count of samples for each $s,a$ pair and variable $\alpha = \frac{1}{N(s,a)}$ which is mathematically identical to keeping a list and taking the average as needed.

In a control scenario using a list of old return values is not only memory inefficient, it is also sample inefficient as you would need samples from each improved policy to outnumber the older samples in order to remove sampling bias due to using returns from initial worse policies. A fixed, or slowly decaying $\alpha$ is a simple and efficient way of "forgetting" the old return values as they become less relevant to the task.

Another way to achieve similar forgetting may be to have a maximum size for each $Returns(s,a)$ list. However, that is not mentioned in the MC Control pseudocode.

Perhaps I am missing something, as the book has been through significant review process. However, it may be that this detail is overlooked because the basic MC Control scenarios are not as interesting as all the extensions and combinations with TD, where you will find the book does use a learning rate $\alpha$ - including when making comparison charts of MC vs TD approaches etc. The explicit list of returns and averaging over them for control only occurs in a couple of places in the text in this pseudocode, and is not mentioned again.


Can I find a mapping that minimizes the maximum distance ratio of certain vectors?

An interesting question. I would start by finding n nearest neighbors of each data point, then calculate their center of mass c and the point's distance d to its nth nearest neighbor. The smaller the d is the larger the density is around a given point. You could then iteratively step every point away from their c in an inverse proportion to the distance d with a suitable step size. This would spread out the clusters.

But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping R^n -> R^n by using an other neural network and apply it to new samples?

This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.

A more rigorous approach might be a variational autoencoder, you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.


Is the mean-squared error always convex in the context of neural networks?

Answer in short:  MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is here.


Does a better discriminator in GANs mean better sample generation by the generator?

If you start to train a GAN in a way in which the discriminator becomes very powerful than the generator, then the training of generator will not be very much successful. In the same way, if the discriminator is too low in comparison with generator, it would let it produce any thing. So, either way, it will affect the training of the entire system.
So, if you see here, both generator and discriminator and competing against each other and on the other side they are dependent on each other for efficient training.
But I've read in a research paper that adding noise to discriminator is good for the overall stability. And sometimes people also change the labelling system as well. For instance, the value for YES is 1 and people change it to 0.9. This actually keeps the discriminator from being over confident.


Should the biases be zero or randomly initialised?

From the stanford CNN class (http://cs231n.github.io/neural-networks-2/):

  Blockquote
  Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.


What is the current artificial general intelligence technology valuation?

I will try to give some sense to this question.

Artificial general intelligence (AGI) is the hypothetical[1] intelligence of a machine that has the capacity to understand or learn any intellectual task that a human being can. It is a primary goal of some artificial intelligence research and a common topic in science fiction and futures studies. AGI can also be referred to as strong AI,[2][3][4] full AI,[5] or general intelligent action.[6] Some academic sources reserve the term "strong AI" for machines that can experience consciousness.

These are the first sentences on AGI on wikipedia (link), and the softest limit there is

[learn] any intellectual task that a human being can.

Even taking only this, it would mean that any AGI has infinite economic value. As soon as there is something that can learn any human task and has the speed of current GPUs/CPUs it could potentially immediatly replace every human in every task. There are certainly enough computers with CPUs and GPUs out there.
This question is still a little flawed because you not only have to constrain the definition of AGI but also how it would actually be implemented.


How is weighted average computed in Deep Q networks

Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.
We have
$$
Q' = Q + \alpha(R-Q)
$$
This can be rewritten as
$$
Q' = (1-\alpha)Q + \alpha R
$$
When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get
$$
Q' = \frac{n-1}{n}Q + \frac{1}{n}R
$$
This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.


Where can I find the proof of the universal approximation theorem?

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used). Note that these proofs tell you that neural networks can approximate any continuous function, but they do not tell you exactly how you need to train your neural network so that it approximates your desired function. Moreover, most papers on the topic are quite technical and mathematical, so, if you do not have a solid knowledge of approximation theory and related fields, they may be difficult to read and understand. Nonetheless, below there are some links to some possibly useful articles and papers.
The article A visual proof that neural nets can compute any function (by Michael Nielsen) should give you some intuition behind the universality of neural networks, so this is probably the first article you should read.
Then you should probably read the paper Approximation by Superpositions of a Sigmoidal Function (1989), by G. Cybenko, who proves that multi-layer perceptrons (i.e. feed-forward neural networks with at least one hidden layer) can approximate any continuous function. However, he assumes that the neural network uses sigmoid activations functions, which, nowadays, have been replaced in many scenarios by ReLU activation functions. Other works (e.g. [1, 2]) showed that you don't necessarily need sigmoid activation functions, but only certain classes of activation functions do not make neural networks universal.
The universality property (i.e. the ability to approximate any continuous function) has also been proved in the case of convolutional neural networks. For example, see Universality of Deep Convolutional Neural Networks (2020), by Ding-Xuan Zhou, which shows that convolutional neural networks can approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough.
See also page 632 of Recurrent Neural Networks Are Universal Approximators (2006), by Schäfer et al., which shows that recurrent neural networks are universal function approximators. See also On the computational power of neural nets (1992, COLT) by Siegelmann and Sontag. This answer could also be useful.


Are Rationality and Intelligence distinct?

From Norvig and Russel definitions of rationality:

  Thinking Rationally - The Greek philosopher Aristotle was one of the first to attempt to codify “right thinking,” that
  is, irrefutable reasoning processes. His syllogisms provided patterns for argument structures
  that always yielded correct conclusions when given correct premises—for example, “Socrates
  is a man; all men are mortal; therefore, Socrates is mortal.” These laws of thought were
  supposed to govern the operation of the mind; their study initiated the field called logic.
  Acting Rationally - An agent is just something that acts. Of course,
  all computer programs do something, but computer agents are expected to do more: operate
  autonomously, perceive their environment, persist over a prolonged time period, adapt to
  change, and create and pursue goals.
  A rational agent is one that acts so as to achieve the
  best outcome or, when there is uncertainty, the best expected outcome.
  In the “laws of thought” approach to AI, the emphasis was on correct inferences. Making
  correct inferences is sometimes part of being a rational agent, because one way to act
  rationally is to reason logically to the conclusion that a given action will achieve one’s goals
  and then to act on that conclusion. On the other hand, correct inference is not all of rationality;
  in some situations, there is no provably correct thing to do, but something must still be
  done. There are also ways of acting rationally that cannot be said to involve inference. For
  example, recoiling from a hot stove is a reflex action that is usually more successful than a
  slower action taken after careful deliberation.

Clearly and also intuitively, rationality is well defined.

Intelligence as seen form mathematical and computational approach:

Intelligence can be the ability for an agent to make rational or irrational decisions, on a varying time frame and also choose the level of rationality (strictly in a computational sense). For example, I have exams and I want to watch TV, on a time frame of a week/month the rational decision would be to study so that I can enjoy the fruits of my labor which will be much more than the instantaneous pleasure of TV (also I can watch reruns). But for a time frame of an hour watching TV is definitely the most rewarding thing. So intelligence can be defined as the capability in deciding the length of time frame to be rational (what we call visionaries those who can see rewards far in the future).

Also as Game Theory or economics suggest, we can have different definitions for rationality depending on our needs. Thus, watching TV to gain knowledge might be more important to someone than studying, so effectively he has a different rationality function (arbitrary made-up term) to satisfy. Thus, intelligence can be deciding our rationality function, based on our needs and external experiences (learning in a nutshell). Also we may decide to minimize our rationality cost function or leave it in an intermediate state (which can be thought of as the minima for a different rationality cost function, thus only rationality functions are the true variable and not the intelligent decision to minimize it or not).

Lets take an example of bees (I am not sure whether this is the correct interpretation though): Bees can hardly be called intelligent (no foresight), but they are rational. They perform the task assigned to them with efficiency and toil (even though this does not reward the bee itself, it rewards the genes carried by the bees and ensures it survival through the queen - can be thought of as evolutionary coded intelligence). Bees perform these jobs in an apparently rational way, which has been decided by 1000's of years of evolution. Though bees individually are of hardly any importance, together they create a truly intelligent community - taking smart decisions and actions albeit their farsightedness is only within a  smaller time frame compared to humans. Thus in common terms, it can be thought that rationality always almost leads to intelligence, but the same cannot be said for vice versa (as for intelligence you now have a choice for rationality function and you can choose not to satisfy it or choose an irrational function).

An important consequence of bees not being intelligent is that they are always performing rational actions as hard coded in their genes, which is causing the entire colony to behave in an intelligent and a very optimized energy efficient way (but there maybe better strategies, we can never know whether they follow the best strategy unless we account for all the variables).

TL;DR : Intelligence can be thought of the ability of an agent to choose the amount of rationality the agent wishes to satisfy. Using mathematics we can always find one or more completely rational methods of solving a problem with variables being time and the environment. But intelligent beings can add more variables like their experience, needs and motivation. Intelligent beings to some extent can single-handedly manipulate the external environment to suit their needs.

From psychological viewpoint:

Here are a few definitions of different types of intelligence and learning - Quite good and concise.

IQ - In science, the term intelligence typically refers to what we could call academic or cognitive intelligence. In their book on intelligence, professors Resing and Drenth (2007)* answer the question 'What is intelligence?' using the following definition: "The whole of cognitive or intellectual abilities required to obtain knowledge, and to use that knowledge in a good way to solve problems that have a well described goal and structure."

Intelligence - Wikipedia - Actually has some good definitions.

Intelligence Quotient - Wikipedia

Emotional Intelligence - Emotional intelligence is the ability to identify and manage your own emotions and the emotions of others. It is generally said to include three skills: emotional awareness; the ability to harness emotions and apply them to tasks like thinking and problem solving; and the ability to manage emotions, which includes regulating your own emotions and cheering up or calming down other people.

Emotional Intelligence - PsychCentral

Emotional Intelligence - Wikipedia

Hope this is of some insight!


Assessing the Question

  Based on the x,y values of each detection, I am trying to find the direction of the bicycles and if the bicycle crosses the whole frame to do a UP/DOWN count.

It appears from the question that there is an interest in both counting bicycles and determining the direction of travel of each. from within the frames of a video stream or file. We can assume that each bicycle has at least one rider. It also appears that it is not a problem involving a fixed optical system positioned to point at a path that is tangential to the optical path, which would make the problem much easier, locking the approximate distance of the bicycle wheels to the optical system to near constant.

The use of the SSD mobile-net model seems reasonable as a starting point for developing expertise.

Starting With ML Design Basics

Let's consider the purpose of CNN and RNN designs.

The purpose of a convolutional network is to deal equally with regions in a multi-dimensional array of values in discrete samples of \R^n during an adaptive (learning) process.
The purpose of a recurrent network is to adapt to (learn) potentially complex temporal (time-wise) trends in potentially complex nonlinear systems.

Understand that the SSD class of algorithms do not do what natural visual systems do. They do not zoom in and out on independent objects within the network seamlessly. They cannot note that a base ball player is running to first base and a ball is coming from the catcher at the same time, requiring independent conceptual zoom operations within the neural network. This cannot be done with a zoom lens. That is why the Director of Photography is such a key role in movie making. The visual data must contribute well to the story telling, using lighting, camera orientation, panning, zooming, and depth of focus.

Although one can create several bicycle concept classes to cover various bicycle sizes, orientations relative to the optics, and distances away, there are limitations to this approach, which can be diminished with circuit parallelism in hardware. Multi-threading and serial evaluation can, depending on resources and patience factors, increase training time beyond what is practical. The challenge is to create seamlessness between low level concept classes of a bicycle in a frame as the bicycle angle and distance changes relative to the optical path.

Deeper into Details

The, "heavy distortion on the objects," could be a show-stopper if the root cause of the distortion mentioned is poor resolution in the time, horizontal, or vertical dimensions. The most significant and consistent image-oriented feature of a bicycle is two ellipses (not always circles) in close horizontal proximity and even closer vertical proximity — the two wheels. The wheels need to be recognizable.

Two general categories of networks were mentioned in the question, CNNs and RNNs, which, in general are the two most relevant overall categories of components in a visual system that recognizes motion. We have some nomenclature in the question, which begins the mathematical theory behind the design of the training of the networks and the real time requirements on those network components once the network components are trained.

  ... each ... frame ... has multiple detections of D1, D2, D3, ..., Dn

$$ D_i \land i \in {1, 2, ..., n} \\
\Downarrow \\
D_{xywh} $$

The above nomenclature presumably refers to a post-learning detection of concept classes $C_a, C_b, C_d$, where there is a many-to-one relationship between the above numeric indices for detections and these letter indices for the concepts of a bicycle to be recognized. Each concept class $C$ might correspond to a particular recognizable bicycle feature set given a particular range of distances to the optics and orientation of the wheels relative to the direction of light rays between the wheels and the camera. The designer, considering this correspondence cannot dismiss the turning of the front wheel. We cannot assume that the eccentricity of the visual representations of the two wheels will be the same, since the bicycle may be turning. Even in this more complex case, the ellipses are likely the two most differentiating features of bicycles in common scenes.

This may be a good time to point out that tricycle recognition may require the recognition of an entirely distinct set of concept classes.

Also notice that, if the optics (camera) is at a drastically different altitude than the wheels of the bicycles, such as images from a drone or a camera on a tall pole, the problem is a different one. This is the intensely effective quality of natural vision systems. Over millions of years, the kind of training that recognizes a bicycle from a drone video stream having only been trained to recognize a bicycle from ground level has emerged. Nature's ability to apply cognitive abilities to visual sequence recognition to use in trajectory prediction is not yet realized in software and hardware and the main problem in automated vehicle piloting and driving.

Two Different Output Requirements

There are two somewhat distinct problems that must be considered in analysis of the project requirements. The output could be either of these two, depending on whether counting bicycles is the primary goal or whether metrics of travel is its own independent objective.

Unit vector $\vec{r}$, presumably in $\mathbb{R}^2$ as a pixel vector, essentially a normalized vector of the first derivative of pixel position with respect to time. The center of the bike, in this case, would be based on the features of the bike in the field of view.
Unit vector $\vec{r}$, presumably in $\mathbb{R}^2$ as a geocoordinate unit vector, essentially a normalized vector of the first derivative of position with respect to time. The center of the bike, in this case, would be based on the features of the bike in geo-space without the altitude coordinate.

Approaches to AI Design

The common, but inefficient, artificial network approach is to locate the bicycle in each frame with a CNN and then use one of the progressive RNN types (either a GRU or a b-LSTM network) to recognize motion trends. One of the largest drawbacks is that you may have many concept classes that represent adjacent size-distance-orientation concepts (kernel based recognition models) of a bicycle to train into the CNN. If the bike is traveling toward or away from the optics at some angle, then the disappearance of the bike from $D_a$ and its appearance in $D_b$ needs to be construed as the contiguous motion of one bike. This is not an easy challenge but is heavily covered in the literature.

It is recommended to use web searches designed to search scholarly articles, not dummies guides, which are not reliable. There many academic publications that can be found with the search term, "Image recognition changing distance orientation." Looking at old articles from the 90s will provide a good historical context. Looking at new ones from the last three years will provide a survey of the current state of research.

Other Questions Within the Primary Question

The original types of recurrent networks are essentially for historical context. The dominant in-field recurrent network successes are often of the LSTM, b-LSTM, or GRU types.

The language (Scala, Python, Java, C, C++) is not particularly relevant if you are delegating the training to a GPU (which always runs C/C++ code),
so it may be unwise to consider reliability concerns as a primary driver for programming language selection.

Regarding, "How AI manages to count the objects," AI doesn't — not at the current state of technology. There is no one approach or algorithm across AI technology that dominates over all other approaches for all domains, into which bicycles can be plugged in and it works.

Currently, the AI engineer designs how the objects will be counted based on the characteristics of the objects to be counted, the meta-features of the incoming stream or data set, and the specifics of the recognition challenge. This is again because the wider capabilities of natural vision systems using the more sophisticated neural nets in animals and people has not yet been invented.

Final Recommendations Regarding System Design

The division between the use of kernels, in the CNN context, and the use of one of the recurrent network types is critical. If the engineer tries to delegate too much to the kernel, the above issue of bicycle distance to the optics and turning corners is exacerbated, because kernel operations do not lend themselves well to orientation and distance complexity. However, the CNN approach is excellent for the most upstream operations, such as edge detection and primitive object detection.

Let the recurrent network (of the more advanced types mentioned above) detect the bicycles as their distance and orientation in relation to the optical path changes, unless you have a sizable GPU farm that will perform CNN operations covering many distance and orientation ranges in parallel. Even if you do or have the patience of a saint, it may be best to delegate total bicycle recognition to the recurrent network anyway, since this is likely closer to the way natural systems do it and the modeling of bicycle travel between distance and orientation categories can be made more naturally seamless.

Recapping the comment above in this context, the problem complexity would be much lower if the bicycles were on a path, could not turn corners, and must travel either right to left or left to right within a speed range governed by the flow of traffic.

Response to Comments

Regarding Edit 1,

  Edit 1: The area of view under the camera is fixed. And we expect the bicycles to move from one entry side. Lets assume that the view and entry/exit is like shown in this video,

YouTube.com reports, "This video does not exist." That the camera is fixed, had been assumed in the writing of this answer because no camera trajectory described in the question. Had the expectation that bicycles will move from one entry side of the frame been included in the question, the answer would have addressed that case, but there was no hint of that requirement prior to Edit 1. Nonetheless, much of the content in this answer to the more general case still applies.

Regarding practicality, let's differentiate practical from prefabricated. The problem originally described before Edit 1 has no prefabricated solution into which one can plug bicycle data and tweak a few parameters to achieve success. In fact, the general interest on the web in seeking prefabricated solutions is usually met with the practical reality that such plug and play cases in machine learning are rare. Most of the time an approach that involves design and experimentation is usually the case.

Those that have hired and managed human beings know that even the hoped-for high level AI of the future, although possibly quite practical, may be as prefabricated as idealized. For instance, hiring an EE does not mean that the electrical engineering department will immediately see a practical improvement in throughput. Management, training, and workflow design will still be prerequisites to employee effectiveness. To have a practical and reliable bicycle counting, some comprehension of concepts to guide initial experimentation, design, and tuning of the training and use scenarios will likely be necessary.

If the author of the question has one of those very rare cases where bicycles travel in exactly one direction and in series, with no foot traffic, trikes, walking pedestrians, or pets, then AI is not necessary at all. A simple LED and photo-transistor with a passive low pass filter and a digital input to a counter circuit will perform a reasonable count. However, if two bicycles might pass in parallel, then we are back with the camera, the need for concept classes, and challenges much like those discussed in detail above.

Regarding steps to a solution, if approaching from the side of a technology to investigate, this answer includes a recommendable sequence, although there is usually quite a bit of overlap and occasional backtracking in actual practice. If this is not a learning exercises and the bicycle problem is one that is actually needed to operate in the field, then the above answer is the correct background to comprehend what it is the machine is required to perform. Following that comprehension would be the investigation of various designs and algorithms, beginning with a search of scholarly articles using the key phrases in the original answer.


Assigning probability to output of a ReLU network

I would just skip the sigmoid function step.

Consider these two scenarios with three choices with the associated values:
0.1,10,100 or 1,100,1000
Given that these scenarios are equivalent in their relativ values, your probabilities should be assigned in the same way. But if you throw in a sigmoid function, the difference between 10 and 100 will be bigger than between 100 and 1000.

Just normalise the values, so that they sum to 1. In that case the choice with a value ten times higher than the value of another choice will be picked ten times as often. Makes sense to me.


Compressing text using AI by sending only prediction rank of next word

If you have a fixed predictor, then yes. If the predictor is not fixed but deterministic, the feasibility depends on the effort needed to update the predictor, and ensuring that messages include a time stamp to ensure the correct version of the predictor is used for compression and inflation.
You get a really nice property if the prediction order is in order of probability that the word occurs, and if you use fewer bits for lower numbers. You would get something that is pretty close to Shannon coding, which is not optimal but is still valid.


Is policy learning and online system identification the same?

From the book Reinforcement Learning, An Introduction (R. Sutton, A. Barto):

  The term system identification is used in adaptive control for what we
  call model-learning (e.g., Goodwin and Sin, 1984; Ljung and S
  ̈oderstrom, 1983; Young, 1984).

Model-learning refers to the act of learning the model (environment). Reinforcement Learning can be divided into two types:

Model-based - first we build a model of an environment and then do the control.
Model-free - we do not try to model the behaviour of the environment.

Policy learning is the act of learning optimal policy. You can do it in two ways:

On-policy learning - learn about the policy $π$ by sampling from the same policy.
Off-policy learning - learn about the policy $π$ from the experience sampled from some other policy (e.g. watching different agent playing a game).


I'm going to assume that by free will, you mean something like the philosophical concept of libertarian free will, which is defended by philosophers like Robert Kane. In Libertarian Free Will, individuals have some capability to make choices about their actions. The classic way to argue this is by assuming some kind of spirit-stuff (e.g. a soul) that exists outside the material world, and that this spirit-stuff constitutes the consciousness of a person. Kane tries some mental gymnastics to avoid this, but then concedes something like it in a footnote. I'm not aware of any serious work that doesn't make some kind of non-physical assumption to justify this view. If someone can point at one, I'll update the answer.

By determinism, I'm going to assume you mean the usual notion of philosophical determinism: since people's decisions depend on what happened in the past, and where they are in the present, they don't really have a choice in any meaningful sense. Philosopher's like Dennett adopt a slightly softer view (compatibilism, essentially: you don't get to make big choices, but you do get to make small ones). Appeals to Quantum Mechanics are common to justify that view. In this context, free action means something more like "did something we couldn't predict exactly". An example might be: you are pre-destined to put a can of campbell's brand tomato soup in your shopping cart, but "make a choice" about exactly which of the dozens of cans you will put in. Since small choices can have large impacts (maybe that can will give you food poisoning, and the others wouldn't), this can make all sorts of things impossible to predict exactly.

I think most AI researchers don't worry too much about these issues, but Turing actually addresses them in his paper right at the start of the field, Computing Machinary and Intelligence. The deterministic/compatibilist view point is introduced as Lady Lovelace's objection: Computers only know how to do what we tell them to, so they can't be called intelligence.

Turing's counterargument uses two prongs. First, Turing notes that computers can probably be made to learn (he was right!). If they can learn, they can do things that we don't expect, just like children do. Second, Turing notes that computers already do all sorts of things we don't expect: they have bugs. Anytime a computer exhibits a bug, it did something that was unexpected. Since we cannot generally rule out bugs in programs, computers will always do surprising things. Therefore, computers satisfy the deterministic notion of free will.

Turing also addresses the libertarian notion of free will, which is part of what he calls the "Theological Objection". The objection is that intelligence requires some kind of divine spark (like free will). Turing argues that we can't detect sparks like this right now (he actually thought we would be able to one day, and spent a lot of time looking at supernatural phenomena too). However, there's no reason to suppose that computers with the right programs won't be endowed with them. A divine creator could decide that anytime you build something brain-like, it gets a spark. If we build a program that's brain-like, maybe it gets a spark too. In the absence of some way to detect souls, it seems like we ought to just agree to treat things that seem intelligent as though they had these souls, since otherwise we don't have a really clear way to decide who is and isn't intelligent. The only remaining way is to say "only things made of human meat have souls and are intelligent". While a lot of people do actually say things like this (e.g. animals have no souls), this is a pretty arbitrary view, and I think there's no hope arguing against it. Turing suggests the "Anthropic Principal": we shouldn't assume that we're special, because the earth isn't in a special place in the galaxy, or in the universe, and we have pretty compelling evidence that we're an evolved version of other animals around us, but some groups (e.g. biblical literalists) find this unconvincing.


What are criteria for ML model to be satisfactory for commercial use?

The answer is "when it works well enough to perform the task that you have set it".
It is a good idea to set your performance criteria in advance so that you can clearly identify the goal that you are trying to achieve and also so that you will know if the model is likely to be successful or not.


I wanted to know how the performance of my net would be compared to the same in Tensor Flow. Not to specific but just a rough aproximation.

This is very hard to answer in specific terms because benchmarking is very hard and is often wrong.

The main point of TensorFlow as I see it is to make it easier for you to use a GPU and further allows you to use a large supply of programs written in Python/JavaScript that still give C++ level performance.

  How fast is TensorFlow compared to self written neural nets?

This is answering the general question of using TensorFlow/PyTorch vs a custom solution, rather than your specific question of how much of a speed up you'd get.

There was a relatively recent MIT paper, Differentiable Programming for Image Processing and Deep Learning in Halide trying to discuss the performance vs flexibility vs time spent coding a solution of 3 different languages.

Specifically they compared a solution in their language Halide vs PyTorch vs CUDA.

  Consider the following example. A recent neural network-based
  image processing approximation algorithm was built around a new
  “bilateral slicing” layer based on the bilateral grid [Chen et al. 2007; Gharbi et al. 2017]. At the time it was published, neither PyTorch
  nor TensorFlow was even capable of practically expressing this
  computation. As a result, the authors had to define an entirely
  new operator, written by hand in about 100 lines of CUDA for the
  forward pass and 200 lines more for its manually-derived gradient
  (Fig. 2, right). This was a sizeable programming task which took
  significant time and expertise. While new operations—added in just
  the last six months before the submission of this paper—now make
  it possible to implement this operation in 42 lines of PyTorch, this
  yields less than 1/3rd the performance on small inputs and runs
  out of memory on realistically-sized images (Fig. 2, middle). The
  challenge of efficiently deriving and computing gradients for custom
  nodes remains a serious obstacle to deep learning.

So in general you'll probably get faster performance with TensorFlow/PyTorch than a custom C++ implementation, but for specific cases if you have CUDA knowledge on top of C++ then you will be able to write more performant programs.


How to handle class imbalancing when the actual data are that way

You can use stratified cross validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same and therefore the test data will always be "realistic".
However, when training a model (using only the training data, of course), the imbalance may have a negative impact. Therefore, have a look at some of the imbalanced learning techniques that are out there to remedy this situation. For example you could try these:

random undersampling: discard random examples from the majority classes until the ratios of class frequencies are close to 1
random oversampling: make random duplicates of minority class examples until the ratios of class frequencies are close to 1
SMOTE: like random oversampling, except that synthetic examples are created instead of random duplicates
balanced bagging: performs random undersampling, but does so multiple times to create an ensemble of models trained on balanced subsets of the training data

etc
You should also take care about the metrics you use to assess predictive performance on the test data. Accuracy could be misleading here, so you may instead find metrics like sensitivity and specificity (calculated for each class individually) more informative.


How does reinforcement learning handle measured disturbances?

Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.

Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $\theta$ randomly initialized and defined as follows.

$$ (p, v) = f_{\theta}(s) $$

The brewing process parameters comprise state $s$. The vector of control change probabilities $p$ have components $p_a = \Pr(a|s)$ for each action $a$. A scalar $v$ estimates the value of the expected outcome $o$ from control state $s$, where $v \approx \mathbb{E} [o|s].

Rather than characterize the variations in brewing stock as disturbance $D$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $\vec{b}$ and feed a normalized signal $\mathcal{N}(b)$ into additional inputs to the deep network. This way, sensitivity to variations in $\vec{b}$ is learned as a critical factor in brewing policy.

In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.

Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.

A focus group
A group of independent testers
Repurchase statistics at a selected set of coffee shops
Feedback controls at a set of offices that provide employees with coffee

For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $\vec{b}$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.

Each search during the training process is guided by MCTS control changes, $a_t ∼ \pi_t$. The final state at the end of training is $s_T$, rated using a fixed scale of taster feedback. Selection of each state $s$ can be based on a control operation search advantage based on three characteristics.

Low past experience (a low count of previous visits in the search)
High probability $p$
High average of values $\{v, \; ...\}$ over the leaf states of brewing trials that selected $a$ from $s$ according to the converging neural network parameters $f_{\theta}$

The output of the search is $\vec{\pi}$ representing a probability distribution over control changes. The deep network parameters $\theta$ converge via gradient descent using a loss function $\ell$.

$$ \ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2 $$

Loss is thus an aggregation of two optimality criteria.

Mean square estimated disparity between the predicted outcome $v_t$t and the taste outcome $o$
Cross-entropy of policy vector $p_t$ and search probabilities ${\pi}_t$
Hyper-parameter $c$ weighted L2 regularization

The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.


What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

Sigmoid is used for binary cases and softmax is its  generalized version for multiple classes. But, essentially what they do is over exaggerate the distances between the various values.
If you have values on a unit sphere, apply sigmoid or softmax on those values would lead to the points going to the poles of the sphere.


I had recently used a slightly unorthodox method to process such images, which involved using RNNs.

Assume the image dimensions to be (16000, 120, 16) = (width, height, channels), as in the question.

Apply a 2D convolution (or multiple such convolutions) of shape(1, k, c),
such that the output of the convolutions becomes (16000, 1, c). So if you only use a single convolutional layer, k=120.

Then, squeeze the extra dimension, to get the shape (16000, c).

The problem has now been transformed back into a sequence problem! You can use RNN variants for further processing.


In the paper that you cite, Inverse Reward Design (2017), the authors actually define what they mean by "proxy reward function".

We formalize this in a probabilistic model that relates the proxy (designed) reward to the true reward

So, the proxy reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word "proxy" is thus consistent with the general usage of the word in computer science, i.e. a "proxy reward function" is a reward function that is used instead of the intended (optimal) reward function.


How should I approach the game "Achtung, Die Kurve" ("Curve Fever") using AI?

Start slowly.
Don't jump straight into Deep Learning, arguably the most complex class in Reinforcement Learning techniques. First work with simpler algorithms, like the original Q-Learning. Define what are good inputs and outputs for your game, and start tuning some hyper-parameters (like future rewards discount factor).
From there, go for Deep Learning. Check other implementations (like DQN, Atari n-step Q-Learning and A3C), and adapt their code to yours, rather than starting from scratch.


How does visual cortex share convolution weight

In the human brain every common pattern is recognised by a multitude of pattern recognisers (be they neurons or microcolumns). That's pretty obvious because neurons die all the time, but we don't wake up and have forgotten how to recognise the letter 'A'. In fact we have to take out rather big chunks of the neocortex until functionality degrades, which is why Alzheimer's patients show symptoms only when the brain is already visibly messed up.

Translational invariance within a level of the hierarchy has to be learned. Basically you need filters for the same edge all over your V1. An object moving across your field of vision only becomes invariant in it's representation on a higher level. Unfortunately we don't magically learn a representation that we can easily turn and twist, our ability to recognise objects from an unusual angle degrades with how uncommon the angle is.

A nice thought experiment to illustrate the point: Imagine a cube sitting in front of you. Now pull one of the corners of the cube upwards until the cube is dangling in front of you, one corner pointing straight to the ceiling, the opposite corner pointing to the ground. Now indicate with your finger where the rest of the corner are.

If you are anything like me, this is really difficult. I think the first time I did't even realise I had to point out six corners!

Of course this is 3D and we might still have inbuilt 2D invariance, but it also turns out that faces that are turned upside down have to be processed much higher into the cortex to be recognised as faces and so on ...

Concerning the learning of a lot of different feature vectors across the field of vision: This might be prevented by the fact that the neocortex learns sequences of input by predicting (via depolarisation) the next pattern. So you might have the situation that a higher level tells the lower level that there is an object moving from left to right and the lower level will predict that the edge it is detecting at point y will reoccur at point y+x.

This setup differs from the training of NNs in two ways: The data is already translational and the translation is predicted, which facilitates learning the same features. Basically two pattern recogniser close to each other get pretty much the same pattern one after the other whenever something moves in a certain direction and the second occurrence is predicted, which means it will be more likely detected, which means it will be learned. (I don't want to dole out a lecture about the cortex, but the prediction is a big deal because it allows neurons to fire quicker which means they beat other neurons before they are laterally inhibited and only if they actually fire will they learn.)

As a disclaimer I want to add that this is just my current understanding of the issue and I'm pretty sure the actual explanation is more complicated. For example I've read that one of the layers of the cortex is important for translational invariance and this layer only exists in the levels close to the sensory input. It is conjectured that this layer (L4) doesn't do the sequence prediction stuff, so maybe having the same kind of input is enough or learning different feature vectors is not actually a problem. There is also the complicating issue that there is poorly understood interplay between the different layers and different levels of the cortex. I would recommend to ask a neuroscientist, except I don't think they figured this out yet.


Is there an open-source implementation for graph convolution networks for weighted graphs?

A Comprehensive Survey on Graph Neural Networks (2019) presents a list of ConvGNN's. All of the following accept weighted graphs, and three accept those with edge weights as well:

And below is a series of open source implementations of many of the above:


Several important researchers distinguish between bandit problems and the general reinforcement learning problem.

The book Reinforcement learning: an introduction by Sutton and Barto describes bandit problems as a special case of the general RL problem.

  The first chapter of this part of the book describes solution methods for the special case of the reinforcement learning problem in which there is only a single state, called bandit problems. The second chapter describes the general problem formulation that we treat throughout the rest of the book — finite Markov decision processes — and its main ideas including Bellman equations and value functions.

This means that you can represent your bandit problem as an MDP with a single state and possibly multiple actions.

In section 1.1.2 of the book Bandit Algorithms (2020), Szepesvari and Lattimore describe the differences between bandits and reinforcement learning

  One of the distinguishing features of all bandit problems studied in this book is that the learner never needs to plan for the future. More precisely, we will invariably make the assumption that the learner's available choices and rewards tomorrow are not affected by their decisions today. Problems that do require this kind of long-term planning fall into the realm of reinforcement learning

This definition is different than the one by Sutton and Barto. In this case, only bandit problems where the learner doesn't need to plan for the future are considered.

In any case, bandit problems and RL problems have a lot of similarities. For example, both attempt to deal with the exploration-exploitation trade-off and, in both cases, the underlying problem can be formulated as a Markov decision process.


What is Precision@K for link prediction in graph embedding meaning?

I understand the confusion and I wanted to refer to this (older post) because the metric really is unclear in the context of the SDNE paper.
Perhaps I can try to explain it for future readers, in hopes that this makes sense. All this is my own interpretation, of course.
SDNE is an autoencoder setup that outputs both node embeddings ($y_i$ vector for focal node $i$) and a reconstruction of the ties of $i$ denoted by $\hat{x}_i$ with the original being $x_i$. Note that $y_i$ is the input to the decoder component, and thus the reconstruction is a function of the embedding.
In SDNE, $x_i$ are the inputs and the "labels", hence autoencoder.
Now, the notion of precision comes from information retrieval. However, for networks, the problem setting differs. We do not retrieve documents repeatedly, instead we literally predict an entire adjacency vector (especially if one uses transformer layers and such). For that reason, the "ranking" part needs to be reformulated to make any sense.
Let's take a naive view and see what would make substantive sense.
In the context of a reconstructed network, precision should mean the following:
"What percentage of reconstructed ties are in the real network?"
Whereas recall would mean.
"What percentage of real ties in the network are found in the reconstruction?"
So we have our reconstruction $\hat{x_i}$ and our ground truth vector $x_i$ - typically rows of the adjacency matrix of the network $\hat{X}$ and $X$ respectively. Let's denote these networks as $\hat{X}$ and $X$ as well as there won't be any confusion (in the paper the authors distinguish the network $G$, its adjacency matrix $S$ and, finally, the inputs and outputs $X$ as subset of $S$.)
The vectors denote ties between $i$ and $j$. With some abuse of notation, we could write for unweighted networks $(i,j) \in X \Leftrightarrow x_{i,j}=1$
Precision would be:
$$\frac{|(i,j) \in \hat{X} \cap (i,j) \in X|}{|(i,j) \in \hat{X}|}
\Leftrightarrow \frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j}=1\}|}{|\{j| \hat{x}_{i,j}=1\}|}$$
and recall would have the denominator with $x_{i,j}=1$ instead of $\hat{x}_{i,j}=1$.
The only difference to the precision@k metric in the paper comes from the ranking. As mentioned above, it is not immediately apparent from the paper how a reconstruction would yield probabilities that we can use for a rank- especially if ties are binary.
However, SDNE does not predict binary ties, even if these appear in the original graph. Instead, it applies a sigmoid function and thus gets some value that is proportional to the likelihood of a tie between two nodes. Long story short, each element in $\hat{x}_i$ is akin to a probabilistic prediction across possible neighbors.
To get the $index(j)$ we can thus rank the values of $\hat{x}_i$ from highest to lowest.
Let the top $k$ of $\hat{x}_i$ be above some cutoff value $t_i(k)$.
We can write precision@k as
$$\frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j} \geq t_i(k)\}|}{|\{j|\hat{x}_{i,j} \geq t_i(k)\}|}=\frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j} \geq t_i(k)\}|}{k}$$
If our network were weighted, we could do a similar ranking for $x_i$. In any case, this solves the first issue.
Now, the main problem comes from the description of $AP(i)$.
Both in the paper, and in the previous answer given, there is an obvious mistake: Where precision@k takes an integer $k$ as parameter, we are to sum over $j$ in $AP(i)$. That is, we are told in the other answer (and in the paper) that
$$AP(i) = \frac{\sum_{j \in S_i} \text{precision@}j(i)}{|S_i|}$$
with $S_i=\{j|x_{i,j}=1\}$
This of course makes no sense. $j$ comes from the node set. Nodes could be numbers, but could also be things like $v_i = $"Apple" and $v_j=$"potato". Obviously, the measure precision@"apple"$(i)$ can not be derived from the above definition.
So, we need to find an interpretation that works. Note first, that the denominator is the number of neighbors of $i$ in the network. Thus, the above sum should maximally yield $|S_i|$.
Furthermore, the authors want to sum over neighbors $j$, employing some sort of precision measure for each. Consequently, whatever is summed up, should sum up to 1 for each $j \in S_i$.
Let's consider an embedding that predicts everything perfectly.
Note that then precision@k is $1$ for every $k$. That leaves us with the conclusion that the measure must be
$$\frac{1}{|S_i|}\sum_{j \in S_i} \frac{\sum_k \text{precision@}k(i,j)}{|k|}$$
where $\text{precision@}k(i,j)$ denotes some node wise measure of precision. In any case, the measure collapses to
$$\frac{\sum_k \text{precision@}k(i)}{|k|}$$
for each $k$ where $\text{precision@}k(i)>0$.


How many types of variational auto-encoders are there?

There are many variations of the original VAE (proposed in the 2013 paper Auto-Encoding Variational Bayes), with different purposes (such as the generation of discrete data or graphs). Of course, I cannot enumerate all of them, so here I will list only the ones I am currently aware of.

Conditional variational auto-encoder (CVAE) (2015, NeurIPS)
Adversarial autoencoders (AAE) (2015, ICLR)
Ladder variational autoencoder (LVAE) (2016,  NeurIPS)
VAE/GAN (2016, PMLR)
Vector quantised variational autoencoder (VQ-VAE) (2017, NeurIPS)
Denoising variational autoencoder (DVAE) (2017, AAAI)
Grammar variational autoencoder (GVAE) (2017, ICML)
Syntax-directed variational autoencoder (SD-VAE) (2017, ICLR)
PixelVAE (2017)
GraphVAE (2018, ICANN)
Constrained graph variational autoencoder (CGVAE) (2018, NeurIPS)
Character VAE (CVAE) (2018, ACS central science)
Regularized VAE (RVAE) (2018, NeurIPS)
Junction tree variational autoencoder (JT-VAE) (2018, PMLR)
InfoVAE/MMD-VAE (2019, AAAI)
Molecular hypergraph grammar variational autoencoder (MHG-VAE) (2019, ICML)
DAG variational autoencoder (D-VAE) (2019, NeurIPS)
Denoising adversarial autoencoders (iDAAE and DAAE) (2019, IEEE Transactions on Neural Networks and Learning Systems)
MACRo-mIcro Disentangled Variational Auto-Encoder (MacridVAE) (2019, NeurIPS)

VAEs have been used for drug design (see e.g. JT-VAE) and in reinforcement learning (see e.g. world models). I don't know if they have been used for self-driving cars, but it's possible.


How can I cluster this data frame with several features and observations?

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering algorithms. sklearn provides functions that implement k-means (and an example), hierarchical clustering algorithms, and other clustering algorithms.

To assess the quality of the produced clusters, you could use the silhouette method (sklearn provides a function that can be used to compute the silhouette score).

Regarding your specific data frame, note that it contains repetitions, so you may want to remove them before starting the clustering procedure. Also, the IDs are not unique, but you probably don't need the IDs for clustering.


Have you tried using a different learning rate?
Yours seems to be quite high, you could try 1e-4 or something in that region.
Coming from the computer vision side, you could also try to improve image readability with prior grayscaling or blurring.
Here are some repos on GTSRB:
https://github.com/Junth/Traffic-Sign-Classification-using-ConvNets
https://github.com/surmenok/GTSRB


How is the depth of a CNN layer determined?

The 96 is the amount of filter maps (also: filter kernels). It is a fundamental of convolutional neural networks. The exact number is not arbritary, although there is no equation or exact rule of restricting the number.

If you have a CNN one single convolution operation would be pointless: since it used for the whole image information it can generalize, but only to specific (meaning: finite amount of) features. Easy example: if a 7x7 filter in the first layer concentrates on round shapes, it can not generalize on let's say red cubes at the same time.

Therefore you convolutional layers have several filter kernels, i.e. several weights where each is used for a convolution. The result of each of these convolutions is one filter map, i.e. the image information convolved by a kernel.

Typically you look at your filter kernels and your problem's domain to figure out what an appropriate number of filter kernels could be. You also have to keep in mind that too few kernels could possibly lose information and overfit to specific patterns, while too many kernels could possibly underfit. Especially when you have far more parameters (primarily the weights of your network) than training data your network will normally perform bad and you have to reduce its size.

The filter kernels should not be confused with batch size. The batch size is the amount of samples (here: images) you train in parallel. Each training step of your network does not only consist of feeding a single image, but feeding batch size number of images, usually combined with batch normalization steps between the layers. Hence, this has nothing to do the amount of filter kernels / convolutions of your network.


Machine learning to find drivers of an event with presence-only data (no absence)

Yes.

Since you have only one type of data, cluster analysis may be a good choice.

You can also try '1-class learning' approaches, although I have found these to be unreliable in the past.

An example of a cluster analysis algorithm in R is kmeans. There are many others. These approaches will reveal points that typify large portions of the dataset. By examining 'typical' cases, and how they differ, you can spot potential causal factors to test experimentally.

An example of a 1-class learning algorithm is a one-class svm. Most svm libraries will accept data of a single class and do the right thing with it. Here's an example with R's e1071 package.