How to change this RNN text classification code to become text generation code?

I found out how to switch it (the code) to do text generation task, use 3D input (X) and 3D labels (Y) as in the source code below:

Source code:

import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0       t1       t2
british  gray     is  cat
0        1        2   (3)  <=x
1        2        3        <=y
white    samoyed  is  dog
4        5        2   (6)  <=x
5        2        6        <=y
'''
Bsize = 2;
Times = 3;
Max_X = 5;
Max_Y = 6;

X = [[[0],[1],[2]], [[4],[5],[2]]];
Y = [[[1],[2],[3]], [[5],[2],[6]]];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  for J in range(len(Y[I])):
    Y[I][J][0] /= Max_Y;

#model
Input    = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,Times,1]);

#single LSTM layer
'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Input);
'''

#multi LSTM layers
#'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
],
return_sequences=True);
Hidden2 = Layers(Input);
#'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden2,Weight3) + Bias3); #sequence of 2d * 2d

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed   = {Input:X, Expected:Y};
Epochs = 10000;

for I in range(Epochs): #number of feeds, 1 feed = 1 batch
  if I%(Epochs/10)==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed).tolist();
print("\nEval:");
for I in range(len(Results)):
  for J in range(len(Results[I])):
    for K in range(len(Results[I][J])):
      Results[I][J][K] = round(Results[I][J][K]*Max_Y);
#end for i
print(Results);

print("\nDone.");
#eof


In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

I think the "No Free Lunch" theorem applies here, or something like it.
Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:
The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.
This is only an educated guess.
As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.


Are there human predictions of when a computer would have been better than a human at Go?

Go predictions were included in the paper:

  The experts are far from infallible. They predicted that AI would be better than humans at Go by about 2027. (This was in 2015, remember.) SOURCE: Experts Predict When Artificial Intelligence Will Exceed Human Performance (MIT Tech Review)


GAN model predictions before training is predictable

I took a look at your model. It seems you have incorrect architecture. The Conv2D layers in your D should have following params: (n_filters, kernel=3, padding='same'), where n_filters is the number of filters and it usually should be doubled as per DCGAN architecture. You can also use strides but since your images are small it won't make any sense.
The D network also should not include UpSampling2D layer. This layer can be used in G network instead of Conv2DTranspose.
Your images should be normalized to the range of [-1, 1] and the activation function of G should be tanh.
You also included m.add(keras.layers.Reshape((8, 8, 3))) to your G network which is the final size of your data. Since you have the final size in the first layers you don't have to include upsampling Conv2DTranspose layers.
Your final model should look like
def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same',
                                     input_shape=[8, 8, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(16, kernel=3, strides=1, padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(8*8*16, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((8, 8, 16)))

    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same'))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2D(1, kernel=3, strides=1, padding='same', activation='tanh'))

    return model

Note that I did not test the model, so you should adapt it to the shape of your data.
I encourage you to follow this tutorial.
Update regarding overfitting
GANs do not have overfitting problem in the classical sense. Instead, vanilla GAN have other problems like vanishing gradient. It happens when the D becomes overconfident regarding fake samples. In that case, D stops providing useful information to the G, since its error becomes 0. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:

Another problem is the mode collapse. In that case, the G tricks the D by producing only one type of samples which looks realistic but G won't represent the real data distribution. Therefore, the generated samples will be homogeneous.
For more information, see Improved Techniques for Training GANs and Wasserstein GAN.


Should I just use exploitation after I have trained the Q agent?

Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as

$$
\pi(s) = \operatorname{argmax}_{a^*}Q(s, a), \; \forall s \in \mathcal{S}
$$
where $Q(s, a)$ is your estimated value function and $\pi$ the policy greedily derived from it.

This means that you would just exploit your current knowledge of the return. This is probably a good thing to do if you believe your value function is optimal and the dynamics of the environment don't change.

Of course, if your policy is not optimal, you may not want to always execute the greedy action. In that case, you could still perform some form of exploration (e.g. with the $\epsilon$-greedy policy).

Moreover, if the dynamics (e.g. the reward function) of your environment change over time, you could continually train your RL agent. If you are interested in continual RL, the paper Continual Reinforcement Learning with Complex Synapses (2018) could potentially be useful.


How to explain peak in training history of a convolutional neural network?

I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.


What is the most effective way to build a classifier?

Your approach would definitely work. I would recommend training a variety of classifiers and comparing their performance using multiclass roc analysis. Also, think about other useful features in addition to the ones you mentioned (e.g. pos tag). Feature engineering is one of the most important factors in building good predictive models. Another thing to keep in mind is that the classes could be highly imbalanced which might influence your model's performance.


From a purely dataset-development perspective I would just label the classes with numbers starting from 0 (ID of the first person) to N (ID of the last person.

During training however, what you do with those classes will vary depending on the type of architecture you are training.  For example, if you are building a neural-network classifier you could simply have multiple output nodes.  When running backprop with a label for the ith person you could put a 1 on the ith output layer and -1 on the rest of them.  This is similar to this answer as well

Whatever you end up doing, try not to label your data to the benefit of the algorithm you're trying to run unless you really know what you're doing.  You don't want to accidentally lose information or make your training data less interpretable and harder to deal with in the long run.


Applying Eligibility Traces to Q-Learning algorithm does not improve results (And might not function well)


Is there a central focus on the communication methods between AI and humans?

This is one of the main research areas of my lab which researches intelligent prosthetics which also give sensory feedback such as touch and kinaesthesia (the feeling of a limb moving in space) to the user.  We use reinforcement learning to bridge the gap in control and have preliminary work in communicating to the user predictions made by the artificial agent.


Is music/sound similarity comparison feasible on neural networks?

Yes, it is possible, even if the best approach could be different from  neural networks. Anyway, you should extract some significant features from the audio (energy, onsets, root frequencies, and other). Usually, more features than those really needed are extracted and afterwards the most sigificant are selected through some algorithm (e.g. PCA). In this way you will obtain an array of features (say between 10 and 100 features) with which you can train your NNs.

Note that NNs do not tell you why two audio are similar but only if they are or not. This is a big disadvantage. Instead, algorithms based on grey-box modeling such as rule or case based algorithm (maybe using fuzzy logic) could be more useful, provided that you have a deeper knowledge of the problem.

References and deepening sources: SMC Lab from University of Padua education material


If you can remember everything and there's no randomisation in your outcome like chess, there is absolutely no reason not to do that. Anybody who can remember all the possible board configurations in chess, by definition plays perfect chess. A perfect player would never lose.

Unfortunately, most practical problems can't be solved by brute-force, and that includes chess. All chess players learn general chess principles. Nobody could memorize all possible opening variations.

If you can somehow memorize every possibility, why not? But can you?


If an image contains two distinct objects, should I create a copy of this image with distinct labels for each copy?

You should use both classes together. Let's say you use the method you proposed. Then they will be contradicting each other as one teaches the network to recognize people, not objects and the other teaches the network to recognizes objects not person. There is no need for seperation of the two classes, unless you are making two seperate classifier. Hope I can help you.


Since a neural network does iteratively learn its own weights I assume you mean the structure of the neural network - the number of layers and nodes per layer.

If what I said above was your question, then yes, it most definitely is being explored.
Even when a neural network is allowed to learn its own structure it still needs to be suited to a specific problem.
Google wrote a blog where they used machine learning to explore the architecture of a neural network calling it the AutoML project where the blog claimed "To make this process of designing machine learning models much more accessible, we’ve been exploring ways to automate the design of machine learning models."
It was found that AutoML can design small neural networks that perform on par with neural networks designed by human experts, but these results were constrained to small academic datasets like CIFAR-10, and Penn Treebank.

This study was further extended to more challenging problems and datasets in another blog where they proposed NASNet

Both these blogs were written in 2017 and I'm not very sure how much progress has been made on this since, but I hope these blogs helped!


Yes, you can train a NN to detect only one type of object like a table. However, you probably will not want to train such a NN from scratch by showing some examples of tables and non-tables. You will need to use transfer learning on a model already trained on several image classes and teach it to also recognize your new class. This transfer learning requires a smaller set of desired images. You may need to give it some negative examples also. You should explore transfer learning with mobilenet, inception, and other pre-trained Tensorflow models if you are willing to use Python and Tensorflow


What is a good convergence criterion for Q-learning in a stochastic environment?

To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and convergence never happens.
For sticking with theoretical guarantees, the learning rate decay process should generally follow the rule that $\sum_t \alpha_t = \infty$ but $\sum_t \alpha_t^2 \ne \infty$ - an example of a learning rate schedule that does this is $\alpha_t = \frac{1}{t}$, although in practice that specific choice could lead to very slow convergence.
Choosing a good starting $\alpha$ and a good decay schedule will depend on the problem, and you may want to base it on experience with similar problems. However, it is not that common to need to gurantee convergence of action values in value-based reinforcement learning. In control problems you often care more about have finding an optimal policy than about perfectly accurate action values. Further to that, many interesting control problems are too complex to solve perfectly in tabular form, so you expect some approximation. It seems relatively common just to pick a learning rate for the problem and stick with it.
If you make your learning rate lower, the Q table will converge to more stable Q values, but possibly at the expense of taking longer to converge on the optimal policy.


How will we recognize a conscious machine?

There are two main subjects you need to look at to understand the problem:

The Turing Test

  The Turing test, developed by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. wiki

See also: Turing Test (Stanford Philosophical Dictionary)

There is a linguistic element, which is that intelligence can be interpreted in many legitimate ways.  Some maintain that Artificial Intelligence has not yet been achieved; other feel a simple automated switch constitutes the most basic form of AI.

Bear in mind that "intelligence" is distinct from consciousness in the sense of "self awareness", but the generalized Turing Test can also be understood as a gauge of the appearance of consciousness.

This leads to Searle's Chinese Room Argument.  I highly recommend reading the Stanford Philosophy link, but the wiki gives a simpler synopsis:

  The Chinese room argument holds that a program cannot give a computer a "mind", "understanding" or "consciousness", regardless of how intelligently or human-like the program may make the computer behave.

The real problem may be, how does one know an algorithm is truly conscious and not merely simulating consciousness?

Philip K. Dick approaches from the opposite direction in Electric Sheep, where one of the conclusions is that "life is life" whether organic or artificial.  This might be said to lead to the "Duck Test" for consciousness: "If it looks like a duck, and quacks like a duck, then it's probably a duck."  (Dick's philosophy was heavily influenced by Christianity, and his view on artificial life may be though of as radically humanist.)

(A parallel philosophical argument might be "how is the appearance of free will distinct from the actuality of free will?"  Free will has been fruitlessly argued about for millennia, but whether the universe is actually deterministic or not, we perceive ourselves to have free will.  If the universe turns out not to be strictly deterministic, it wouldn't functionally change anything.  To make matters more fuzzy, true randomness is nature is only found at the quantum level and then only within certain models. Quantum uncertainty has been proposed as a basis for free will, but we don't know.  Possibly uncertainty is a condition of individuality, which must be subjective in relation to a system or other subjectivities.  Nevertheless, how do we know we are not "robots" acting out a pre-determined sequence of actions, which we perceive as decisions?)

I personally very much like the idea of recursion as function of self awareness.

  At the phenomenal level, consciousness can be described as a singular, unified field of recursive self-awareness, consistently coherent in a particular way; that of a subject located both spatially and temporally in an egocentrically-extended domain, such that conscious self-awareness is explicitly characterized by I-ness, now-ness and here-ness. SOURCE: Peters, Frederic Consciousness as Recursive, Spatiotemporal Self-Location

I'd also posit that being able to read and interpret your own code is a form of self-awareness.

This brings me to the idea that understanding (Chinese Room) is a red herring, and what we're really talking about is interpretation.

We are trapped in subjectivity, humans and automata, and perfect certainty is only achievable in a very limited set of circumstances, such as solved games. (There is also the idea of completeness of a system vs. consistency.)

When we talk about understanding in the sense of anything abstract, by which I means semantics as opposed to syntactics, meaning vs. form, one can say that understanding is a function of interpretation, regardless of the accuracy of the interpretation.

--------------------------------------

Fun Speculation:

I've learned never to underestimate the insight of artists, and the best speculative fiction authors are narrative philosophers in the tradition of Plato.  Dick related memory to identity, regardless of whether the memories are real or artificial.  This might be thought of as the narrative conception of the self--I am a product of my experience.  The real "me" is not merely my body, but "the story of me"--the subjective history that led to this moment of me-ness.

I think it is a not unreasonable assumption that artificial consciousness may arise out of understanding of narrative.  Words are just symbols, and there are all kinds of semantic issues, but actions have a concrete aspect.  Game theory studies actions in the form of choices, not only in the sense of equilibria, but also as a form of communication.   (See: iterated dilemma)  The choices agents make in iterated dilemmas constitute a narrative history than can be analyzed and understood mathematically, and these analyses are used for decision making.

It seems to me that the idea of consciousness can be related to decision making.  If you're just a transistor, that consciousness is quite limited, more akin to a cell than a complex organism. It may come down to whether one considers intelligence and consciousness to be spectrums as opposed to thresholds.

If you believe consciousness to be a spectrum, then limited artificial consciousness has already been achieved. Consciousness analogous to human-level self-awareness is still in the future.

See Also:
Definitions of the Self (wiki); Self Knowledge (wiki)


Why isn't my Neural Network based calculator working?

A neural network is not good at selecting a function based on those 3 input parameters, because of the way a neuron is setup.

What you should do is either make a neural network for each operation, or use different input neurons for each operation. E.g. 2 input neurons for the addition operation, 2 for the multiplication, and 2 for the minus. 6 inputs in total of which 4 will always be 0.

This will make it easier for the neural network to calculate the result.


Why do we normalize data in a deep neural network?

I answered a similar question earlier and here is a piece of my answer that i think covers your question:

Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) but lately has been tied to the optimization process (How Does Batch Normalization Help Optimization?).

This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized or how it will effect your network, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given your initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: Fixup Initialization).


Which algorithms, between ant colony or classical routing algorithms, have a better time complexity for the shortest path problem?

No. In general, you can't find a tight bound for evolutionary algorithms, and it is one of the main difference of these algorithms with the classical algorithms.

You should notice that it does not mean you can't find when the evolutionary algorithms are finished! But, you can't find a tight bound for the algorithms time complexity to reach to the optimal solution or how much that solution is near to the optimal solution (in contrast to the approximation algorithms).


Can you explain me this CNN architecture?

While it would certainly help if the link to the paper could also be posted, I will give it a shot based on what I understand from this picture.

1) For any convolutional layer, there are few important things to configure, namely, the kernel (or, filter) size, number of kernels, stride. Padding is also important but it is generally defined to be zero unless mentioned otherwise. Let us consider the picture block-by-block.

The first block contains 3 convolutional layers: (i) 2 conv layers with 96 filters each and the size of each filter is $ 3 \times 3$ (and stride $=1$ by default since it is not mentioned), and (ii) another conv layer with same configurations as above but with stride $=2$.

The second block is pretty much the same as the previous except the number of filters is increased to 192 for each layer that is defined.

The only considerable change in the third block is the introduction of $ 1 \times 1$ convolutional filters instead of $3 \times 3$.

And finally, a global average pooling layer is used (instead of a fully connected layer).

2) As for your analysis, it is exactly the case in fully connected layers, wherein the number of units in the input layer must match the vectorized dimensions of the input data. But, in the case of CNN, we give the images directly as the input to the network. The whole idea of a CNN is to understand the spatial structure of the data by analyzing patches of the image at a time (which is what the filter size defines). This PyTorch tutorial should give an idea as to how exactly the input is given to CNN.


In practice, are perceptrons typically implemented as objects?

Unless one performs an exhaustive search, it's difficult to answer your question.
However, in the widely used libraries, such as TensorFlow, PyTorch and sklearn, most abstractions (like neural networks and layers) are implemented as classes (see this, this and this examples, respectively), as the main programming language supported by these libraries is Python, which an object-oriented language (but note that Python also supports other programming paradigms, such as functional programming).
I don't know the statistics, but, from my experience, I would say that OOP (which tends to be intuitive to humans for obvious reasons) is the mostly widely used programming paradigm, as opposed to the (pure) functional paradigm.
However, in general, the programming paradigm used to implement a certain concept probably depends on the language that you want to use. For example, in Haskell, a purely functional programming language, you will probably implement a perceptron as a sequence of functions (see this example). Another example is NumPy, which, although the primary interface is written in Python, under the hood, is primarily implemented in C, a non-OOP language (see e.g. this example, where you see many functions, but no class).
This should also partially answer your other question. In some cases, you will implement a concept using the programming language and paradigm that improves the efficiency of your implementation (e.g. the case of NumPy).


Extending FaceNet’s triplet loss to object recognition

Check this page out, it describes how to develop apply triplet loss to a network: https://towardsdatascience.com/image-similarity-using-triplet-loss-3744c0f67973


What is the Bellman Equation actually telling?

For a Markov Decision Process $(\mathcal{S}, \mathcal{A}, P, R)$ (here $P(s, s') = \mathbb{P}(S_{t+1} = s' | S_t = s, A_t = a))$;, let us define the value of being in a certain state. That is,
$$v_\pi(s) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[\sum_{i=0}^\infty \gamma^{i+t}r(s_{t+i}, a_{i+t}) | S_t =s\right].$$
That is, the value of being in state $s$ at time $t$ is equal to the expected value of the discounted sum of future rewards, where the expectation is taken with respect to $\pi(\cdot | s)$ the action policy and the environment dynamics $P$.
We will now define
$$\sum_{i=0}^\infty \gamma^{i+t}r(s_{t+i}, a_{i+t}) = G_t$$
Now, we can rewrite this as
$$\mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[G_t | S_t =s\right] =  \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma G_{t+1} | S_t =s\right].$$
The RHS is now in a nicer format and we can express it as
$$\mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[G_t | S_t =s\right] = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma v_\pi(s') | S_t =s\right],$$
where $s'$ is the state that we transition into at time $t+1$. Note that $s'$ is a random variable and we are taking the expectation over this according to the action policy and the environment dynamics -- this is because the state we transition into depends first on the action we select and then the environment dynamics $P$.
You can see that we now have
$$v_\pi(s) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma v_\pi(s') | S_t =s\right].$$
This is the Bellman equation (at least a form of it) and it expresses a recursive relationship between the values of states. That is, the value of being in state $s$ is equal to the expected immediate reward from being in this state plus the value of being in the state that we transition into. This relationship is useful in Reinforcement Learning as many algorithms use this equation to form update rules to approximate the value/state-action value function of the MDP, such as the  SARSA algorithm, so it is more than just a philosophy, it is the driving force behind many of the RL algorithms.
Now, when I say at least a form of it, that is because in RL it is also common to see a Bellman equation for the state action value function
$$q_\pi(s, a) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ G_t | S_t = s, A_t = a\right] = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + v_\pi(s') | S_t = s, A_t = a\right];$$
noting that the value function $v$ is the expectation of $q$ over the action space, i.e. we marginalise out the action we condition on -- thus there is the recursive relationship.
As pointed out in the comments for this question, it is also worth noting that Bellman equations originated in Dynamic Programming, which exist to solve planning problems such as the knapsack problem.


What is the AI discipline where an algorithm learns from an initial training set, but then refines its learning as it uses that training?

I believe this can best be done with reinforcement learning via Deep Q Learning. That's where I would start. Steps are:

Initialize a Q table.
Choose an action.
Perform the action.
Measure the reward.
Update the Q.

A neural net will approximate the Q function. See: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0

Also consider policy gradients, actor critic, and inverse reinforcement learning.


How to choose a suitable threshold value for shi-tomasi corner detection algorithm?

The threshold is typically chosen empirically, so there is no exact answer.
It's dependent on how many corners you wish to select, and how strict you want the detection, which could depend on the use case, the dataset, and the block size of the algorithm. If you’re not sure what to choose for the threshold, I would suggest using a scheme relative to your images to deduce the best threshold, based on what performs best for your use case.
For example, let the criteria for selection, or scoring function, be $R$, which in the Shi-Tomasi case is $R = min(\lambda_1, \lambda_2)$. We could choose some value $q \in (0,1)$, so that the threshold becomes
$t = q\max_p{R}$,
where the max $R$ is calculated over all points $p$, an approach similar to OpenCV Good Features to Track.
You could examine the corners detected and adjust $q$ based on how stringently you wanted to select corners. If you don't want to waste the time manually fine-tuning the threshold, you could check out some automated approaches in the literature, such as Automated thresholding for low-complexity corner detection


In OCR, how should I deal with the warped text on the sides of oval objects?

There are many papers on this but the following is a good start:

How to unwrap wine labels programmatically.  The author includes source code in Python.

You mentioned you do not want to do a panoramic view but that has more than one meaning.  If I assume you mean you do not want to rotate the can while taking multiple photos, or you don't want to take multiple photos from different angles, you could try a pericentric lens.  This would require some image processing to do the unwrapping.  More resolution is needed as the wrapping is much more severe.  The advantage though is that you will have a single image of the full cylindrical surface and won't miss any features or text.


How to decide whether a problem needs to be solved algorithmically or with machine learning techniques?

There are two different problems described in the linked question and your question: optimization and learning.

Optimization

If you are asking about optimization (the second linked question: Search minimum value with learning machine algorithm) you can have 3 different approaches:

analytical approach
numerial methods
metaheuristics

As you suggest, it is usually better to try them from the first to the last one. It is common that the first approach is unfeasible for optimizing for target function, but very often you can use either mathematical optimization for some specific classes of problems (e.g. linear/quadratic programming) or iterative methods (e.g. conjugate gradient method). Only after considering this approaches it makes sense for the third class of approaches, genetic algorithms being a notable example, which is often classified as an AI approach.

Learning

If you are asking about learning, then the first linked question (Ideas on how to make a neural net learn how to split sequence into sub sequences) seems to be intended as an example. However it doesn't make clear what the problem is, as the target function seems to be obvious, so no learning is needed.

In this case it also makes sense to first try to pin down the problem mathematically and resort to machine learning if it is impossible and if you have the data (input/output examples).


If the accuracy of my current model is low ($50 \%$) and we want to minimize time in collecting more data, should we try other models?

To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set.

Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m  training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully used, then you need more training data.


Dataset containing images of varying dimensions and orientations

Almost always people will resize all their images to the same size before sending them to the CNN.  Unless you're up for a real challenge this is probably what you should do.

That said, it is possible to build a single CNN that takes input of images as varying dimensions.  There are a number of ways you might try to do this, and I'm not aware of any published science analyzing these different choices.  The key is that the set of learned parameters needs to be shared between the different inputs sizes.  While convolutions can be applied at different images sizes, ultimately they always get converted to a single vector to make predictions with, and the size of that vector will depend on the geometries of the inputs, convolutions and pooling layers.  You'd probably want to dynamically change the pooling layers based on the input geometry and leave the convolutions the same, since the convolutional layers have parameters and pooling usually doesn't.  So on bigger images you pool more aggressively.

Practically you'd want to group together similarly (identically) sized images together into minibatches for efficient processing.  This is common for LSTM type models.  This technique is commonly called "bucketing".  See for example http://mxnet.io/how_to/bucketing.html for a description of how to do this efficiently.


Why do we use the word "kernel" in the expression "Gaussian kernel"?

The usage of the word "kernel" in the context of support vector machines probably comes from its usage in the context of integral transforms.

See the article Kernel of an integral operator, and the questions What is the difference between a kernel and a function? and Why is the kernel of an integral transform called kernel?.

The word "kernel" has been used in many other contexts, such as in computer vision, to refer to a certain function with a special purpose. See e.g. the paper that introduced SIFT.


Does backpropagation update weights one layer at a time?

The answer is implied in the term "backpropagation". All gradients are calculated at the same time. That is, the error from your loss function is propagated backwards from the output and through your whole network. This propagation results in an error associated with each weight in the network, which determines how you change each weight in order to minimize your loss function.

  or do we allow $V$ to update along with $W$?

Yes. This saves time, since many of the results of intermediate computations used to update $V$ can be reused in the update of $W$.

See http://neuralnetworksanddeeplearning.com for a detailed description of the backpropagation algorithm.


Move blocks to create a designed surface

Essentially, you could do something like have the robot randomly make moves (moving around and moving blocks) for some number of steps. Repeat this a bunch of times, and record the 'score' at the end (how close you are to a perfect result grid). Tell your algorithm to act more like the best scoring runs (Optimize a loss function), and start it over. Hopefully, you'll eventually get a robot that manages the task - the whole 'optimal path' thing will come around by the computer telling itself to learn from the lowest cost examples.

Remember, you're letting the machine do the hard thinking about the best way to do this or that. All you have to do is give it the framework to learn.


How do we derive the expression for average reward setting in continuing tasks?

We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition dynamics of the MDP.
This means that, eventually, $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$ for some large $t$. Therefore, as we take the average of our expected values of the rewards received for an infinitely long period of time, this will have converged due to what I just mentioned of $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$. To see why the two are equal, recall that the reward received is dependent on the current state and the action taken -- to better emphasise this I will briefly denote the reward at time step $t+1$ as $R(S_t, A_t)$. If we are in the steady state distribution, that is, the state distribution is now fixed, and our actions are still taken according to our policy, then the expected value of $R(S_t, A_t)$ will be the same for all future $t$ since neither the policy nor the state distribution are changing (recall that the average rewards are a way of evaluating a policy in the average-reward setting so for sure this does not change).
A way to think of this is that since we know that, eventually, $\mathbb{E}[R_t]$ will equal $\mathbb{E}[R_{t+1}]$, and so if we keep have an infinite number of these, the average of them will of course converge to the same value. Imagine if I gave you the sequence 1, 2, 3, 4, 4, 4, 4, ........, 4 and asked you to take the average - if we had an infinite amount of 4's then the average would of course be 4.


How is the number of parameters reduced in the group convolution?

The number of parameters is filters*input_channels*output_channels

Groups are formed among input and output channels.

So instead of input_channels*output_channels with two groups you get (input_channels/2)*(output_channels/2) + (input_channels/2)*(output_channels/2)


How are the "Link Strength true", "Link Strength blind" and "Mutual Information" calculated in this report on Bayesian networks?

The $MI$ is easy enough.
For $MI(X \to Z)$, we get $U([0.5, 0.5]) - 0.5 U([0.9, 0.1]) - 0.5 U([0.1, 0.9])$. The $[0.5, 0.5]$ comes from $0.5 [0.9, 0.1] + 0.5[0.1, 0.9]$ -- this is how to calculate $P(Z)$ from $P(Z|X)$ and $P(X)$.
For $MI(X \to Y)$, we need to marginalize out $Z$. This takes a bit of work, but is not too hard. I manually verified all of the reported values in the figure, and they are all correct.
The $\mathrm{LS}{\mathrm{true}}$ values are a bit trickier, but still feasible. I managed to get the same values. It is important when calculating $\mathrm{LS}{\mathrm{true}}(X \to Z)$ that $Y$ is not a parent, so it does not come into it -- which is why the MI gives exactly the same value.
The "blind" version took me a bit more effort to implement, but again, I am getting the same values as reported in the table.
I used the "simple formula" at the top of page 5.


Machine learning to predict 8*8 matrix values using three independent matrices

In principle, you can use a fully connected neural network with reshaping for this kind of problem. The questions you should ask yourself are:

Baselines: What are the simplest algorithms to approach the problem? How good would a human be?
What do I know? Are there any properties of the 8x8 matrix that will always be true? For example, it seems as if the values from left to right strictly increase. Same for top to bottom. This can be used!
Are the outputs independent? (e.g. having the (1,1) entry of the matrix, do I know something about the $(i, j)$ entry of it?

Then, of course, there are more specific things I could imagine. If you were not clear about (2), you might have (wrongly) used softmax/tanh/sigmoid in the last layer. You might simply have too little training data for neural networks. Your neural network implementation might be broken.


Why is the target called "target" in Monte Carlo and TD learning if it is not the true target?

It is our "current" target. We assume that the value we get now is at least a closer approximation to the "true" target.
We're not so much moving towards a wrong value as we are moving away from a more wrong value.
Of course, it is all base on random trials, so saying anything definite (such as: "we are guaranteed to improve at each step") is hard to show without working probabilistically. The expectation of the error of the value function (as compared to the true value function) will decrease, that is all we can say.


How can AI be used to design UI Interfaces?

If you see the use case, on higher level it seems to generate some visual output - the design but when seen at lower level, this design is output of some code.

One way we can do it is to train a neural network that learns to generate code which can be seen as some form of organized text. So now can be treated as a text generation problem on which you can find a lot of literature.

pix2code is one implementation that uses this idea and even expands it. In this paper the authors took it to another level, they also used the visual part - the GUI and built an architecture that takes both the code and the GUI as input and learns to generate code. So eventually it would be capable of generating code when a simple GUI was given.

Also there are some implementations where even the network could produce code even when a rough sketch was given.


Is there any Python library available for manifold learning using diffusion map?

The first place I would have directed you would be Sklearn and pydiffmap. I found this paper specifically about the problem you are doing using python the reference a package called  megaman  It seems like an active  Github . I suggest not just looking at manifold learning papers but leaning towards a search toward non linear embedding or non linear dimensionality reduction as well. The paper you give doesn't reference local linear embedding but seems to be similar conceptually.

I found this   kernel pca document  which is very straight forward and I think will help you going forward as well. I believe the RBF Kernel PCA part may be of use to you at least as a way of benchmarking how well your diffusion map is functioning.


Could a large number of interconnected tiny turing-complete computer chips be patterned across a wafer to simulate a programmable neural network?

It's been done (essentially). This guy at the following link has used a series of FPGAs to emulate hundreds of 8080s, using them to train a neural network to play Gameboy games.
https://towardsdatascience.com/a-gameboy-supercomputer-33a6955a79a4

IBM's True North being used in Darpa's SyNAPSE program is also very close to what you suggest. https://en.wikipedia.org/wiki/TrueNorth

Also of interest may be SpiNNaker and Intel Loihi.


Programming a bandit to optimize donations

It does not matter to the bandit algorithm that rewards are quantised or fractional, or that they can vary. This is true for pretty much all bandit optimisation algorithms.

So just treat the $0.80 donation as a real valued reward of 0.8, that occurs on a single timestep.

Treating a single reward on a single timestep as if it were multiple rewards across multiple timesteps might cause problems, depending on which algorithm you were using, even if the reward averaged to the same value. For instance it may skew the maths away from theoretical assumptions used in upper confidence bound action selection or gradient-based solutions.

It occurs to me that your $1 experiment is supposed to be some kind of proxy for the real deployment, where donations will be sparse and may vary more. Whilst this may help a little, essentially you are just setting some kind of prior based on results from a test. The real bandit algorithm starts once the system is deployed into production, and could well have very different results when the goal is to collect real and larger donations.


How to find the category of a technical text on a surface-semantic-level

Problem Statement

Find the category of technical text on a surface-semantic-level.
The requirement is to classify the input text of paragraphs into their respective categories.

The categories given as predefined are as follows.

Overview
Data Architecture
Technical Details
Applications
etc

Some document types that would be added in place of 'etc' might include these.

Requirements
High Level Design
System Architecture
Testing Plan
Deployment Plan
Disaster Recovery Plan

Semantic Structure of Technical Documents
It is correct that the terminology between a requirements document and data architecture will likely have much in common.  More precisely, the distributions of linguistic elements of all documents common to a given system or project are likely to contain domain commonalities.  A system that pilots a drone will have the linguistic elements "drone(s)", "hover(ing|s)", "target(ing|s)", "flight plan(s)" in similar distributions throughout its documentations.
Indicators
It is likely that these five distinguishing characteristics can be exploited in categorization.

Sentence semantics
Inserted diagrammatic and pictorial conventions
Header conventions
Linguistic elements that appear commonly in only one type of technical document
Elements and structure from copying previous documents and modifying them or the use of boiler plates and templates in specific departments

Rather than focus entirely on text recognition and abandoning the other items in the list above would be unwise.  Diagrams, such as network diagrams and UML diagrams, may be quite easy to discern using deep convolutional approaches and would clearly identify the category to which the hosting document belongs.  That is also true of test case tables.
Recognizing that the proportions between the five indicators above and sections of indicators are variables in the model upon which training is applied will produce the best results.  For instance, the final paragraph may be more telling than the rest of the body.
Also, be aware that one can couple the proportional appearance of language elements within all text documents on record with the language elements identified in the example and then in later use of the trained model.  The training is likely to progress faster and produce more accurate and reliable results if features include an indication that the linguistic element "test case(s)" appearing in the input text is significantly less prevalent in the domain of technical documentation than the linguistic element "-ing" to indicate continuous tenses of verbs.
Avoid Static Grammars
Language parsers based on fixed language rules (grammars) have not had much success compared to association based semantic mapping, and linguistics has moved away from those static models for similar reasons.  Avoid grammar based parsing.
Existing Work
For the textual categorization, the below academic publications are some of the recent work that have already gained some notoriety.

Semantic clustering and convolutional neural network for short text categorization, P Wang, J Xu, B Xu, C Liu, H Zhang, F Wang… - Proceedings of the 53rd, 2015
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification, D Tang, B Qin, T Liu - Proceedings of the 2015 conference on empirical, 2015
Learning Semantic Representations of Users and Products for Document Level Sentiment Classification*, D Tang, B Qin, T Liu - Proceedings of the 53rd Annual Meeting of the, 2015
Effective use of word order for text categorization with convolutional neural networks, R Johnson, T Zhang - arXiv preprint arXiv:1412.1058, 2014 - arxiv.org
Jumping NLP curves: A review of natural language processing research, E Cambria, B White - IEEE Computational intelligence, 2014
Recurrent Convolutional Neural Networks for Text Classification, S Lai, L Xu, K Liu, J Zhao - AAAI, 2015

Faulty Approaches
The approach in the comment of comparing the semantics of sentences will only determine one aspect of information redundancy between two sentences.
The number of comparisons are also a consideration.  For $\chi$ documents each containing $\sigma$ sentences would require $\chi \, (\chi - 1) \, \sigma \, (\sigma - 1)$ comparisons.  For $\chi = 10,000$ documents containing $\sigma = 1,000$ sentences, we have 99,890,010,000,000 sentence comparisons, the totality of which provides no particularly useful information about the category of any of the documents.
The documents must be related to a concept class, not each other.
Visualizations of the semantics aren't particularly useful unless you are looking for something and the visualization is designed to present that which is sought.
A Better Plan

Determine the number of example training documents, $t$, needed to be categorized by experts to create a sufficiently large training data set. (Consider using the PAC learning framework designed for this purpose.)
Draw that number from the full set of documents, using an appropriate random or highly pseudo-random method, herein referred to as method $\mathbb{D}$. That is the training example set.
Draw that number from the full set of documents again, using method $\mathbb{D}$. That is the test example set.
Have experts label (categorize) both example sets according to technical document type.  Use the same experts for both training and testing, and have them alternate periodically between the two sets so that their learning, fatigue, or boredom curves have a minimal effect on their categorizing of the two sets.
Profile each of the category-labeled documents in terms of the above five indicators.  Note that to use the images embedded in the document as part of the profiling, which could drastically improve system reliability, the images must be run through a separate diagram categorizing network trained separately on images to find diagrams characteristic of one of the technical document types.  That may sound like much work, but consider that categorizing drawing types is a well developed science and the labeling is only of the diagram types that are representative of particular document types.
Train an appropriately designed artificial network to categorize the documents, using the profiling results as example inputs and the labels from the categorizing from the experts.
Test using the test set
Based on the results of the test, decide whether to use the current training or re-execute a previous step using information gained from the first training and train again
Run the trained network on the full document set
Pull a sample from the result using method $\mathbb{D}$ to validate effective completion of the run


LASER creates multilingual contextualized word embeddings, what you do with them is up to you. You can use this as a feature extraction and add whatever you want to the end of the network.
I believe the implementation by facebook does not let you change the weights of the LASER model itself, they are froozen to the best of my knowledge.
So, yes, you can use LASER for downstream tasks, but you cannot change the actual weights of the LASER model.


How to use RNN With Attention Mechanism on Non Textual Data?

The article referenced doesn't highlight the most common terms, which is why you may not be finding what you want. We've found that not all the designs have public implementations in every language. Some we've had to translate from Python or Java to C++ to get the speed we needed for embedded applications. Attention mechanisms are not one thing either. There are attention mechanisms in the cells, before the layers, after the layers, as oversight mechanisms, and in push-pull balancing arrangements. The common cell using attention ideas use gates in the cell and the article only briefly mentions it. That's the long-short term memory cell, which is a type of RNN. Look for LSTM and B-LSTM for the first examples, which are gated. The GRU is also a type of RNN cell with gating. This paper is recent and proposes two mechanisms: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16126/16099. If you search for DiSAN there are a few implementations. The newest kind, which has promise is not really an RNN strictly. That's the recursive network, which can have attention mechanisms but can build learned graph structures instead of tensors. That may also be what you want.


How can I detect thin objects (like pens and pencils) without a bounding box but only 2 endpoints and the orientation?

Recent work achieves a similar task: Object recognition together with the bounding box (e.g. YOLO---there are quite a few on Github too). The bounding box is not enough in your case, but it is an interesting pattern: Recognition plus some form of measurement. Such architectures could be good candidate to start with, and repurpose for stick orientation.

The problem could also leverage the current results in gait recognition. In fact, this looks closer to the problem at hand than object recognition. An example is this model based on multiview (many pictures input) recognition, with a demonstration on Github. Gait recognition is also popular these days, and many inspiring papers and OSS implementations are available.

The above presents two approaches your problem could benefit from, as a "combination". My gut feeling is that tilt and orientations may be easier than direction (i.e. where is the tip?).

The question calls for training a model. An alternative approach, perhaps to start with and get more insight, could be to go with "standard" computer vision algorithms, such as the Hough Transform. This transforms allows to find lines in an image. The mathematics are at reach, and it may work well enough for a quick demo. Also, your handle name suggests "embedded mobile" engineer, and a simple Hough Transform could be cheap on mobile.


Convolutional Neural Network: does each filter in each convolution layer create a new image?

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.

About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.


Proof of uniqueness of value function for MDPs with undiscounted rewards

In the case where the reward is undiscounted, there is no guarantee of convergence as the iteration procedure is not a strict contraction.

Unfortunately I can't find the math mode on the ai stackexchange so my answer can't be very precise.

But an easy example is the following: Take a 'running' reward R of 0 to make things simpler, and a MDP with two states a and b. Take a transition matrix with 0's on the diagonal and 1 off diagonal. You will see that the algorithm will always flip the values of V(a) and V(b), and hence no convergence.


Solution to exercise 3.22 in the RL book by Sutton and Barto

Your answer is correct but I am not sure exactly on how you arrived at it, as e.g. in the last case you don't know that $v_{L,n}(S_0) = v_{R,n}(S_0)$.

I will show for case B when $\gamma = 0.9$ as case A is trivial and hopefully you can apply what I've done in case B to case C so that you get exact answers.

Now, as you stated $v(s) = \mathbb{E}[R_{t+1} + \gamma v(S_{t+1}) | S_t = s]$. Assuming that $\gamma = 0.9$ we can calculate the values for each state under the policy of taking the left action. Note that because we are looking for deterministic policies and the environment is deterministic then a lot of the expectations can be disregarded as nothing random is happening.

'Left Policy'

\begin{align}v(s_0) &= 1 + 0.9 \times v(s_L) \\ v(s_L) &= 0 + 0.9 \times v(s_0) \\ v(s_R) &= 2 + 0.9 \times v(s_0)
\end{align}
We can solve this set of linear equations to get $v(s_0) = \frac{100}{19}, v(s_L) = \frac{90}{19}, v(s_R) = \frac{128}{19}\;.$

'Right Policy'

\begin{align}v(s_0) &= 0 + 0.9 \times v(s_R) \\ v(s_L) &= 0 + 0.9 \times v(s_0) \\ v(s_R) &= 2 + 0.9 \times v(s_0)
\end{align}
We can again solve these to obtain $v(s_0) = \frac{180}{19}, v(s_L) = \frac{162}{19}, v(s_R) = \frac{200}{19}\;.$

As we can see, for each of the states the value function is larger for all of the states under the policy 'go right', thus this is the optimal policy for the case of $\gamma = 0.9$.

It is important to note that if we take the 'left' action in state $s_0$ then our policy would never take us to state $s_R$, and the same for the right action and state $s_L$, however due to the definition of an optimal policy requiring $v_{\pi ^*}(s) \geq v_{\pi}(s)\; \forall s \in \mathcal{S}$ then we must evaluate the value function for all states, even ones that would not be visited under a policy you are evaluating. This means that the state for $s_R$ will change whether we go right or left, because the value of this state depends on the value of $s_0$, which clearly changes depending on whether we go right or left.


How to detect patterns in salary distribution if we are suspecting malicious distribution based on employee's region?

A simple initial approach would be to separate it by position and check for each:

Use linear regression: $\hat{salary} = \sum_i \alpha_i * \hat{region}_i + \sum_k \beta_k * \mathbf{1}[\hat{gender}=k]$ and now you have an intuitive measure by looking at $\alpha$'s and $\beta$'s.

2 issues that may arise with this method:

This assumes though that a linear model is a good fit, which it very well might not be
The amount of data points may be small and this will give you point estimates without any sense of confidence.

2 possible respective solutions:

Use a stronger model with less intuitive parameters (like a neural network) and then use common visualization techniques to determine weight of each input (this could be as simple as changing them and looking at the outcome or something more complex that utilizes the gradient), but even more complex models can still suffer from the second problem, which is why id reccommend option 2
Use a Bayesian model, you can make something hierarchical something as simple as doing the above linear regression with a non informative prior and eventually you'd be able to build confidence intervals and get a better sense of the associations (note if youre looking for what to write this in, I recommend JAGS for its ease of use)


Why is the derivative of the activation functions in neural networks important?

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation:

Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very steep, and a small change in input may result in a drastic change in its output; conversely, a low absolute value means little change, so not steep at all, with the extreme case that the function is constant when the derivative is zero.

Training a neural network essentially amounts to an optimization problem where one wants to minimize a certain value, in this case the error produced by the network on the given training examples. Backpropagation learning can be viewed as a case of gradient descent (the inverse of hill climbing).

If for a moment we assume that your input is only 2-dimensional (just for illustration, the mathematics of course also work for higher dimensions), you could imagine the error function as a landscape with hills, mountains, valleys, ridges etc. You are standing at a high point and want to get down as far as possible. Gradient descent means that, in discrete steps, you always walk down in the direction that has the steepest slope downwards from where you are currently standing, until you eventually reach a (local) minimum.

In order to determine where that steepest slope is, you need the derivative of the activation function. Basically, you want to sort out how much each unit in your network contributes to an error, and adjust in the direction that contributes the most.

Edit: Regarding constant values for a derivative, in the landscape metaphor it would mean that the gradient is the same no matter where you are, so you'll always go in the same direction and never reach an optimum. However, multi-layer networks with linear activation function are kind of besides the point anyhow when you consider that each cell computes a linear combination of its inputs, which then is again a linear function, so the output of the last layer will ultimately be a linear function of the inputs at the first layer. That is to say, anything you can do with a multi-layer net with linear activation functions, you could also achieve with just a single layer.


What are the state-of-the-art approaches for continual learning with neural networks?

What I understand from your questions is that you are trying to avoid catastrophic forgetting while applying online learning.

This problem should be addressed by implementing methods that reduce catastrophic forgetting for different tasks. At first glance it might seem that they don't apply because it's data that change and not a particular task but changing data result in a change of the task. Say your goal is to classify different breeds of dogs. Your online data-set morphs into excluding "Great Danes". Your neural network after enough epochs would forget about "Great Danes". The task is still serving its purpose by classifying different breeds but the task still changed. It changed from recognizing "Great Danes" as a dog breed to not recognizing "Great Danes" as a dog breed. The weights changed to exclude them but the methods I linked tries and keep weights intact even though it was not intended for the purpose of online learning. Just set the hyper parameters to include these techniques to low as I believe data won't have an instant change but would change over time, and you should be fine.

The most obvious technique being storing information as you train. This is called pseudo-rehearsal. With this at least you would be able to use stochastic gradient decent but you need memory and resources as the data set grows.

Then there was an attempt to reduce impacts of weights on old tasks to keep some relevancy to them. Structural Regularization.

Later these guys implemented HAT which seems to keep some weights static while others adapt to new tasks.


Are there any other machine learning models apart from Reinforcement Learning and Q Learning to play video games?

As I see it, it all comes down to game theory, which can be said to form the foundation of successful decision making, and is particularly useful in a context, such as computing, where all parameters can be defined.  (Where it runs into trouble is with the aggregate complexity of the parameters per the combinatorial explosion, although Machine Learning has recently been validated as a method of managing intractability specifically in the context of games.)
You might want to check out Playing Games with Genetic Algorithms and Evolutionary game theory.


Is there a way to make my neural network discard inputs with bad results from learning?

I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the Huber loss(in Pytorch and TensorFlow). Examples that are far from the expected value won't produce big gradients (:


Understanding the On-policy state distribution for episodic tasks with $\gamma \in (0,1)$

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor.
The first is probably what you're familiar with: the $i^{th}$ step receives a reward of $\gamma^{i}r_i$ instead of just $r_i$.
The second, equivalent interpretation, is that at every step, we have a $1-\gamma$ chance of immediately ending the episode. Assuming we don't end the episode, we receive the full $r_i$ reward. This is what Sutton and Barto mean by termination. Note that under the second interpretation, we have a $\gamma^i$ probability of reaching the $i^{th}$ step, hence the connection to the first interpretation.
Your definition of $\eta_{\gamma}(s)$ looks correct to me. Alternatively, we can write it in Sutton's, Barto's notation like
$$\eta_{\gamma}(s) = h(s) + \sum_{\bar s}\eta(\bar s)\sum_{a}\gamma \pi(a | \bar s) p(s | \bar s, a)$$


Should I use my redundant feature as an auxiliary output or as another input feature?

For extra input that does not matter, you should not input it to the network.

  Feature selection, the process of finding and selecting the most
  useful features in a dataset, is a crucial step of the machine
  learning pipeline. Unnecessary features decrease training speed,
  decrease model interpretability, and, most importantly, decrease
  generalization performance on the test set.

Source: A Feature Selection Tool for Machine Learning in Python

As the source says, unnecessary features decreases accuracy and training speed. Moreover, they have no mapping to the labels, so they won't end up being used. They are unnecessary and adding them will only cause you trouble. Hope this helps you and have a nice day!


What is meant by "the number of examples is reduced", and why is this the case?

I think what he means is that, while distribution $P(y  \mid x,s)$ has three variables, $P(y  \mid x)$ has two variables. The number of parameters required to describe a distribution (or samples to approximate it) grows exponentially with the number of variables (for more info see for instance Ian goodfellow, .. deep learning").


What's the advantage of log_softmax over softmax?

The short answer is yes, log_softmax + nll_loss will work better.

I don’t know the implementation details under the hood in PyTorch, but see the screenshot below from the documentation:


What should the output of a neural network that needs to classify in an unsupervised fashion XOR data be?

How to fix the network above to auto-classify XOR data, in unsupervised manner?

This cannot be done, except accidentally.
Unsupervised learning cannot replace or emulate supervised learning.
As a thought experiment, consider why you would expect the network to discover XOR, when simply considering outputs rounded to binary, you could equally find AND, OR, NAND, NOR or any of the 16 possible mapping functions from input to output. All of the possible maps are equally valid functions, and there is no reason why a discovered function mapping should become any one of them by preference.
Unsupervised learning approaches typically find patterns that optimise some measure across the dataset without using labelled data. Clustering is a classic example, and auto-encoding is sometimes considered unsupervised because there is no separate label (although the term self-supervised is also used, because there is still technically a label used in training, it happens to equal the input).
You cannot use auto-encoding approaches here anyway, because XOR needs to map $\{0,1\} \times \{0,1\} \rightarrow \{0,1\}$
You could potentially use a loss function based on how close to a 0 or 1 any output is. That should cause the network to converge to one of the 16 possible binary functions, based on random initialisation. For example, you could use $y(1-y)$ as the loss.


Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.


Are self-driving cars using single frame or multiple frame to make decision?

It varies quite substantially between different self-driving paradigms(rather obviously) but for the most part the vast majority of implementations are using a variety of different reference frames in order to make predictions.

For example, Tesla's Autopilot is being fed many different camera feeds as well as radar and ultrasonic signals that are processed in a variety of temporal contexts.

While, for the most part, all of these programs are very tight-lipped, we can make a variety of assumptions based on the information available and educated assumptions.

As with many large, complex ML/AI systems, there is a large amount of compartmentalization where many different connectionist(or sometimes classic) models are combined(à la youtube recommendation system). Tesla is likely utilizing recurrent and convolutional networks where particular modules(combinations of models) are deciding on specific contexts(temporal or signal-based). These outputs are then most likely fed into an actor network which makes real time decisions.


Is explainable AI more feasible through symbolic AI or soft computing?

XAI is relevant to "black box" AI (machine learning methods where the decision making rationale is not apparent, only the structure of the system that led to that decision.)

Symbolic AI, GOFAI, and Expert Systems are both explainable and understood, in that the the decision-making process is designed by humans.  (Symbolic AI involves human-readable representations of problems.)

To directly answer, XAI is not only feasible in the latter cases, it is a prerequisite. The difficulty is in making black box decision-making explainable.


How should I design the LSTM architecture for multivariate time series forecasting problems?

I have not come across a model like this yet. BUT

If you have not tried smaller models I'd recommend trying that first.

Justification: this lets you use learning curves to diagnose what to do next.

Also, you might try starting with a GRU (the LSTM overhead may not be needed).

One Idea for a starting model

Observe that turnout does not affect weather. So the weather forecast model will be independent of the turnout model but not vice versa.

(Note: I'm using RNN to denote which ever recurrent architecture you choose)

Formulation for simultaneous prediction:

Weather

$\text{RNN}_w$ is your weather prediction model

$\hat w_t = \text{RNN}_w(\hat w_{t-1})$ be the predicted weather at time $t$

Turnout: The idea here is that people will or will not go to the beach for various reasons (overcrowded, too hot or stormy etc). So in the beach population prediction task we use all these as features to predict the population at the next time-step. This reduces the problem to any one of the classical models already developed.

$\text{RNN}_p$ is your turnout (beach population) prediction model

$\hat p_t$ be the predicted turnout at time $t$

$\hat c_t=[\hat p_t, \hat w_t]$ is the concatenation at time t

$\hat p_{t+1} = \text{RNN}_p(\hat c_t)$

Formulation in the presence of weather forecast:

Simply replace $\hat w_t$ with the true weather forecasts $\hat f_t$.

A final warning

Unless you think your RNN is better at weather prediction than the forecasting systems I would not recommend using $\text{RNN}_w$ in a production application.

I hope this helps.


Reinforcement Learning with long term rewards and fixed states and actions

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with it. Its enough to define a reward at the end where you, for example, give a reward of 1 if sentence is satisfactory or -1 if it isn't. Regular tabular Q-learning would easily solve the toy problem that you gave as an example.


Should Monte Carlo tree search be able to consistently beat me in the connect four game?

You should not let the tree grow for only two seconds rather you should use the simulation number equal to 1000 or something like that. I use the simulation number equal to 10000 for making a single move in the tictactoe game and it was working fine for me. Also, after the agent has chosen the move you do not have to start the statistics(N = visit count, V = expected reward, U = UCT score) from the beginning, you can use the current statistics and replace the root node with the chosen node.


Is there any value given to each chess piece in AlphaZero?

is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself?

No, there are no such explicit values assigned to pieces, no manually-constructed evaluation functions. The paper states that "no domain knowledge" is given to the algorithm other than the game's rules (necessary to run simulations / run search algorithms like MCTS).

  I read that the algorithm uses Monte Carlo Tree Search, but what are the key improvements to prior chess algorithms already using MCTS?

The key improvements are in the way that Deep Learning (deep neural networks), Reinforcement Learning, and self-play is combined with MCTS. This is quite similar to the methods previously used by AlphaGo Zero in the game of Go as well. There (likely) have been combinations of Deep Learning + MCTS before (e.g. using a learned Neural Network to bias rollouts in MCTS), but the specific way in which they are combined in AlphaZero is critical (specifically, using the "search behaviour" of MCTS as one of the training signals for the Neural Network). It probably also helped that we're talking about Google here, who could afford to use thousands of Tensor Processing Units (TPUs) for training.

  Is there a hope for being able to run it an average computer? They said it required 9 hours learning (starting with nearly 0 knowledge except rules (and maybe value for piece?)), and 24 millions of games. Is it something doable in maybe 1 month with a average computer?

Based on the information in the paper, I highly doubt this. As mentioned above, training was done with thousands of TPUs (specifically; 5,000 first-generation TPUs and 64 second-generation TPUs) in parallel. One month is only about a factor 80 times greater than the 9 hours training time reported in the paper, whereas all of those TPUs is much more than 80 average computers.

After training, it likely could run on a fairly average computer (or maybe high-end computer) to play though. But it'll need much more power to train first.


Is there a way of deriving a loss function given the neural network and training data?

I don't think there's a way of doing what you want, at least, I've never seen such a thing (and, currently, I am not seeing how this could be done in the general case).

The same neural network model but with different (or same) weights could have been trained with the same loss function or not. For example, although it may not be a good idea, you can train a neural network for classification with the mean squared error, as opposed to the typical cross-entropy. Moreover, even if you know the loss function that the neural network is trained with, the training data alone may not lead to the same set of weights because the actual weights depend on different (possibly stochastic) factors, such as if (or how) you shuffle the data or the batch size.


How to add contextual follow up like Google Assistant

You would need to keep track of the current topic, and references. So, for example, a query When is the next train from London to Birmingham? would result in

topic = TRAIN_TRAVEL
start-loc = London
destination-loc = Birmingham

A follow-up And what about Bristol? would then replace destination-loc with "Bristol" and you would be able to build a new query from that.

The key issues here are the set of topics your assistant will be able to handle, and the relevant object slots. You might also want to clear the topic variable after the next input or so to avoid having it still hanging around even though the user no longer talks about that particular topic.

UPDATE: Just to add how that would work from a technical point of view. The input And what about Bristol? would not be recognised as any known intent, as it is too unspecific. As a fallback your Assistant then takes the last topic, and interprets the input in the light of that context. So here we assume that the intent is 'train travel', as that is the last thing the user spoke about. This will of course not always work, but should sort out the majority of cases.


Single-Shot Learning for Object Re-Identification

I have put my initial idea to a test and used a small pretrained CNN (MobileNet) to compute features for reference images and stored the feature vectors in a "database". Query images go through the exact same network and the resulting feature vector is used for nearest neighbor retrieval in the DB.
from glob import glob

import torch
from PIL import Image
from numpy.linalg import norm
from torchvision import transforms
from torchvision.models import mobilenet_v2

model = mobilenet_v2(pretrained=True)
model.eval()

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# Generate DB
db = {}
dp_paths = glob('db/*.jpg')
for path in dp_paths:
    image = preprocess(Image.open(path)).unsqueeze(0)
    with torch.no_grad():
        output = model(image)
    db[output] = path

# Query
image = preprocess(Image.open('queries/box.jpg')).unsqueeze(0)
with torch.no_grad():
    query = model(image)

# Nearest Neighbor (poor man's version)
min_distance = float('inf')
candidate = None
for k, v in db.items():
    distance = norm(k.numpy() - query.numpy())
    if distance < min_distance:
        min_distance = distance
        candidate = v

print(candidate, min_distance)

At least with my 5 test reference images and several query images it worked without a single failed "classification". However, I am not sure if it will stand up to a larger test...


What is the proof that the branch and bound algorithm always finds optimal path in a graph?

Branch and Bound is similar to an exhaustive search, except it incorporates a method for computing lower bounds on branches. If the lower bound on a given branch is greater than the upper bound on the problem (i.e. the current best solution encountered), that branch can be discarded since it will never produce an optimal solution.

Hence, since you explore all options except those you know will produce values less optimal than your current best value, you are guaranteed to encounter the global optimum.

Note this is a generic algorithm, and you will need to reference a specific implementation if you want proof of why it satisfies these criteria.


This is probably not going to work well as a way to make money. People with far larger budgets, and far more training, are already milking out any money to be made this way. This is probably their day job, and they are good at it.

That said, here are some ideas:

You do not need or want to use a convolutional network for this. Convolutional networks are useful when you want to detect a pattern invariant to translation in complex inputs. The only translation you have is that the teams could appear in either order. Just input them in both possible orders if this is a concern.
You want to find similar games. You do not need or (generally) want to use a neural network to do this. As a starting point, normalize the features and compute the nearest neighbors of an input point directly. Algorithms for doing this are fast and will return all similar points. You can even (with small modifications) output the degree of similarity as a numeric value.
If you want to predict who will win, you will probably have much better luck doing that directly. Build a feed-forward neural network that predicts which team won from the features you gathered. This is a routine classification task, and the resulting model will probably work better than finding similar games and then trying to manually determine what to do.
If you care about scores instead of just who won, use a feed forward network for regression instead of classification. The task is almost identical.


Does net with ReLU not learn when output < 0?

There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is known as the dead ReLU problem. In practice, it is rarely a problem but it can be avoided with smooth rectifiers like ELU and Swish. Another interesting ideas is CRelu, which concatenates both positive and negative parts of the pre-activation, resulting in twice as many outputs, half of which always receive a non-zero gradient.


What models is Google's quick draw using?

Google's Quick Draw became five things.

A pilot application
A market test
An experiment in massively data intensive reinforcement learning 1
A data segue into handwriting recognition 1
A data collection device for other branches of research (not necessarily an intentional purpose during conception) 2

Google's team on the project wished to delve into concept abstraction by tapping the minds of the authors of one billion drawings, attracting the public to the point of data acquisition using a contest between a massively scaled and parallelized game engine and the hundreds of millions of drawing authors.

Concept Model

The concept model (distinct from the algorithm and architecture) is a time series of motor actions. Due to the event model in computer peripheral devices and the coupling with the event model of the Netscape DOM, AJAX RESTful transactions can be stored into a huge data set as pen-up, pen-down, and pen-move operations. This is very much like some of the first photo-plotter commands pioneered by Gerber Scientific Instruments in the 1980s.

Learning Convergence Approach

The concept model is embodied within and trained using a generative recurrent network called sketch-rnn. 3

References

[1] Introducing the Kaggle “Quick, Draw!” Doodle Recognition Challenge,

[2] The Dead-Serious Strategy Behind Google’s Silly AI Experiments, by Katharine Schwab, 12/1/2017, FastCompany.com

[3] A Neural Representation of Sketch Drawings, David Ha, Douglas Eck, 2018


How is the reward in reinforcement learning different from the label in supervised learning problems?

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.
In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use $G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}$ for return here (where $G_t$ and $R_t$ are random variables), there are other definitions, but the argument that follows doesn't change much for them.
You can use the oracle to reduce the RL training process to SL, creating a policy function $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ learned from a dataset that the oracle output. This clearly relates SL with RL, but how do $x_i, y_i$ from SL relate to $s_t, a_t$ from RL in terms of reward values?
The states can relate directly (as input):
$$x_i \equiv s_t$$
The action from the policy function is more indirect, if you want to see how reward is involved:
$$y_i \equiv \pi^*(s_t) = \text{argmax}_a \mathbb{E}_{A \sim \pi^*}[\sum_{k=1}^{\infty} \gamma^k R_{t+k+1} | S_t=s_t, A_t=a]$$
Note the oracle is represented by the optimal policy function $\pi^*(s_t)$, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what $A \sim \pi^*$ is representing).
In practice the optimal policy function is unknown when starting RL, so the learning process cannot be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the $\text{argmax}$ over actions of expected sums of future reward.
Another way to view the difference:

In SL, the signal from the label is an instruction - "associate these two values". Data is supplied to the learning process by some other independent process, and can be learned from directly

In RL, the signal from the reward is a consequence - "this is the value, in context, of what you just did", and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task


How sigmoid funtion helps us in reducing error in neural networks?

There are several nice things about using the logistic function and a few drawbacks as well.  I'm not going to discuss all of them, but here is a small synopsis.
Nice Properties

For networks which calculate probabilities, it brings the input features into a nice 0 to 1 range.
Since the logistic function maps inputs to the range (0,1), it is commonly used as an activation function in neural networks which require some sort of classification.  For example, if we had a convolution neural network (CNN) that classified if an image given to it was a dog or not, it could have an output neuron that returned 0 if the image was not a dog or 1 if it was.  When the CNN is given an image with inputs of varying dimensions and ranges, it's more intuitive to deal with the input features in the realm of 0 to 1 instead of 0 to 255 or whatever input format you're dealing with.
It is smooth and differentiable.
When performing back propagation, it's much easier to deal with differential functions.  Otherwise you'd have to approximate the gradient which may lead to unfavorable results.
A Vanishing Issue
At extreme values, the gradient vanishes.
The OP was also hinting that at very positive and very negative values of X, derivative of the logistic function could become close to 0.  For large input values this may hinder learning for extreme values.  If you think of gradient decent, the jump that gradient descent makes is only as strong as its magnitude of the gradient that it requires.  Analytically speaking, this also hurts back propagation because learning is calculated using the chain rule, which ends up multiplying several gradients together.


Is the research by Stanford University students who use logistic regression to predict sexual orientation from facial images really scientific?

One way to criticize the study could be to attack the data on which the study is based on. An image on a social network is not "neutral" (those are not ID photo) and certainly not images from a dating website (from which the data of the study come from).

For example as a homosexual / heterosexual person you will perhaps put forward different attributes on your photo (facial hair / glasses, type of clothes) to attract gender specific people.

Those are parameters are not directly linked with the "facial profile" of the person but they will influence the black-box model during training, so you will end up thinking you can detect sexual orientation of a person with only his face characteristics but in reality your black-box algorithm have only detect a totally different characteristic (glasses for example) that are link with a specific sexual orientation.


This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$
Code Sample for $f_t$:
import numpy as np
import math

def sigmoid(values):
    sigmoid_applied = []
    for value in values:
        result = 1 / (1 + math.pow(math.e, -value))
        sigmoid_applied.append(result)
    return np.array(sigmoid_applied)

w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])

f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication

Note - There is also a bias term which I haven't included here for simplicity
If you understood $f_t$, you can do the same for other states also.
If you feel I am wrong anywhere in this post, then please do consider adding a comment


Are there any pretrained models for human recognition from all angles?

There were no pre-trained models for human that were taken from all angles (which is a somewhat extreme requirement) or specifically from above.  This condition may change due to the high value of such to the surveillance industry.

There are already several projects for which there is human specific pattern recognition and there have emerged some data sets to support this more specific field of objectives, some of which are based on moving rather than stationary images.  This is for two reasons.

Multiple frames has the potential (should computing resource utilization be overcome) to produce higher accuracy and reliability in human recognition partly because of the identification of joints in movement and partly because there is simply more information available on each object that could be a human.
Trajectory of pedestrians is important for both law enforcement and security and for avoiding pedestrians in vehicle automation.

These are some of the projects and download sites that are related to the goal of the question, but unfortunately do not seem to provide data sets of humans from above or pre-trained networks from that visual perspective.

https://software.intel.com/en-us/openvino-toolkit/documentation/pretrained-models
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/pretrained-cascade-image-classification
https://github.com/matterport/Mask_RCNN
https://drive.google.com/drive/folders/1zvl89AgFAApbH0At-gMuZSeQB_LpNP-M
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
https://modelzoo.co/model/human-pose-estimation-with-tensorflow


Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.
Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.
The target actor network $\mu'$ is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) $y = r + \gamma Q(s', \mu'(s'))$ where $s'$ is the state we transitioned to from $s$ when we took an action $a$.
Now, $a$ was sampled as part of the tuple $(s, a, r, s')$ from our replay buffer, meaning that $a$ will have been chosen according to some past version of our policy, plus some exploration noise $\mathcal{N}$. The way this links to the behaviour policy $\beta$ is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy $\beta$.
The target network $\mu'$ is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy $\beta$, at least not in any useful way for you to think about it.


Why does the BERT NSP head linear layer have two outputs?

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs:

They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but there seems to be some confusion as to whether the output of the sigmoid function should be used as a probability.

Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with tf.nn.softmax_cross_entropy_with_logits. Specifically,

-tf.reduce_sum(one_hot_labels * log_probs, axis=-1)

where one_hot_labels and log_probs are $\mathbb{R}^{N \times 2}$, is much easier to read than
-tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))

where binary_labels and probs are $\mathbb{R}^N$.


How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

One of the best ways to learn is use reference of others.

Have a look at Peter Sadowski - Notes on Backpropagation (Page 3):

There is also a great blog post by Eli Bendersky - The Softmax Function and Its Derivative.


What's the difference between model-free and model-based reinforcement learning?

In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a neural network or other statistical learning model to predict values, or even to predict next state (although the latter may be used as part of a model-based algorithm and be called a "model" regardless of whether the algorithm is model-based or model-free).

Instead, the term refers strictly as to whether, whilst during learning or acting, the agent uses predictions of the environment response. The agent can use a single prediction from the model of next reward and next state (a sample), or it can ask the model for the expected next reward, or the full distribution of next states and next rewards. These predictions can be provided entirely outside of the learning agent - e.g. by computer code that understands the rules of a dice or board game. Or they can be learned by the agent, in which case they will be approximate.

Just because there is a model of the environment implemented, does not mean that a RL agent is "model-based". To qualify as "model-based", the learning algorithms have to explicitly reference the model:

Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning, Actor-Critic are "model free" RL algorithms. They rely on real samples from the environment and never use generated predictions of next state and next reward to alter behaviour (although they might sample from experience memory, which is close to being a model).
The archetypical model-based algorithms are Dynamic Programming (Policy Iteration and Value Iteration) - these all use the model's predictions or distributions of next state and reward in order to calculate optimal actions. Specifically in Dynamic Programming, the model must provide state transition probabilities, and expected reward from any state, action pair. Note this is rarely a learned model.
Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions. In order to pick the best action, it needs to query a model that predicts what will happen on each action, and implement a policy like $\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + v(s'))$ where $p(s',r|s,a)$ is the probability of receiving reward $r$ and next state $s'$ when taking action $a$ in state $s$. That function $p(s',r|s,a)$ is essentially the model.

The RL literature differentiates between "model" as a model of the environment for "model-based" and "model-free" learning, and use of statistical learners, such as neural networks.

In RL, neural networks are often employed to learn and generalise value functions, such as the Q value which predicts total return (sum of discounted rewards) given a state and action pair. Such a trained neural network is often called a "model" in e.g. supervised learning. However, in RL literature, you will see the term "function approximator" used for such a network to avoid ambiguity.

  It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based.

I think here you are using the general understanding of the word "model" to include any structure that makes useful predictions. That would apply to e.g. table of Q values in SARSA.

However, as explained above, that's not how the term is used in RL. So although your understanding that RL builds useful internal representations is correct, you are not technically correct that this can be used to re-frame between "model-free" as "model-based", because those terms have a very specific meaning in RL.

  In that case, when would model-free learners be appropriate?

Generally with current state of art in RL, if you don't have an accurate model provided as part of the problem definition, then model-free approaches are often superior.

There is lots of interest in agents that build predictive models of the environment, and doing so as a "side effect" (whilst still being a model-free algorithm) can still be useful - it may regularise a neural network or help discover key predictive features that can also be used in policy or value networks. However, model-based agents that learn their own models for planning have a problem that inaccuracy in these models can cause instability (the inaccuracies multiply the further into the future the agent looks). Some promising inroads are being made using imagination-based agents and/or mechanisms for deciding when and how much to trust the learned model during planning.

Right now (in 2018), if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as DQN or A3C. That may change as the field is moving fast and new more complex architectures could well be the norm in a few years.