Fully convolution networks
A fully convolution network (FCN) is a neural network that only performs convolution (and subsampling or upsampling) operations. Equivalently, an FCN is a CNN without fully connected layers.
Convolution neural networks
The typical convolution neural network (CNN) is not fully convolutional because it often contains fully connected layers too (which do not perform the convolution operation), which are parameter-rich, in the sense that they have many parameters (compared to their equivalent convolution layers), although the fully connected layers can also be viewed as convolutions with kernels that cover the entire input regions, which is the main idea behind converting a CNN to an FCN. See this video by Andrew Ng that explains how to convert a fully connected layer to a convolutional layer.
An example of an FCN
An example of a fully convolutional network is the U-net (called in this way because of its U shape, which you can see from the illustration below), which is a famous network that is used for semantic segmentation, i.e. classify pixels of an image so that pixels that belong to the same class (e.g. a person) are associated with the same label (i.e. person), aka pixel-wise (or dense) classification.

Semantic segmentation
So, in semantic segmentation, you want to associate a label with each pixel (or small patch of pixels) of the input image. Here's a more suggestive illustration of a neural network that performs semantic segmentation.

Instance segmentation
There's also instance segmentation, where you also want to differentiate different instances of the same class (e.g. you want to distinguish two people in the same image by labeling them differently). An example of a neural network that is used for instance segmentation is mask R-CNN. The blog post Segmentation: U-Net, Mask R-CNN, and Medical Applications (2020) by Rachel Draelos describes these two problems and networks very well.
Here's an example of an image where instances of the same class (i.e. person) have been labeled differently (orange and blue).

Both semantic and instance segmentations are dense classification tasks (specifically, they fall into the category of image segmentation), that is, you want to classify each pixel or many small patches of pixels of an image.
$1 \times 1$ convolutions
In the U-net diagram above, you can see that there are only convolutions, copy and crop, max-pooling, and upsampling operations. There are no fully connected layers.
So, how do we associate a label to each pixel (or a small patch of pixels) of the input? How do we perform the classification of each pixel (or patch) without a final fully connected layer?
That's where the $1 \times 1$ convolution and upsampling operations are useful!
In the case of the U-net diagram above (specifically, the top-right part of the diagram, which is illustrated below for clarity), two $1 \times 1 \times 64$ kernels are applied to the input volume (not the images!) to produce two feature maps of size $388 \times 388$. They used two $1 \times 1$ kernels because there were two classes in their experiments (cell and not-cell). The mentioned blog post also gives you the intuition behind this, so you should read it.

If you have tried to analyze the U-net diagram carefully, you will notice that the output maps have different spatial (height and weight) dimensions than the input images, which have dimensions $572 \times 572 \times 1$.
That's fine because our general goal is to perform dense classification (i.e. classify patches of the image, where the patches can contain only one pixel), although I said that we would have performed pixel-wise classification, so maybe you were expecting the outputs to have the same exact spatial dimensions of the inputs. However, note that, in practice, you could also have the output maps to have the same spatial dimension as the inputs: you would just need to perform a different upsampling (deconvolution) operation.
How $1\times 1$ convolutions work?
A $1 \times 1$ convolution is just the typical 2d convolution but with a $1\times1$ kernel.
As you probably already know (and if you didn't know this, now you know it), if you have a $g \times g$ kernel that is applied to an input of size $h \times w \times d$, where $d$ is the depth of the input volume (which, for example, in the case of grayscale images, it is $1$), the kernel actually has the shape $g \times g \times d$, i.e. the third dimension of the kernel is equal to the third dimension of the input that it is applied to. This is always the case, except for 3d convolutions, but we are now talking about the typical 2d convolutions! See this answer for more info.
So, in the case we want to apply a $1\times 1$ convolution to an input of shape $388 \times 388 \times 64$, where $64$ is the depth of the input, then the actual $1\times 1$ kernels that we will need to use have shape  $1\times 1 \times 64$ (as I said above for the U-net). The way you reduce the depth of the input with $1\times 1$ is determined by the number of $1\times 1$ kernels that you want to use. This is exactly the same thing as for any 2d convolution operation with different kernels (e.g. $3 \times 3$).
In the case of the U-net, the spatial dimensions of the input are reduced in the same way that the spatial dimensions of any input to a CNN are reduced (i.e. 2d convolution followed by downsampling operations). The main difference (apart from not using fully connected layers) between the U-net and other CNNs is that the U-net performs upsampling operations, so it can be viewed as an encoder (left part) followed by a decoder (right part).


What are some limitations of using Collaborative Deep learning for Recommender systems?

A good recommender program is one of the hardest problems in AI.
Recomender needs to first know me (by talking to me or by analyzing the movies that I graded as good).
Then it needs to be able to recognize similarity of my taste to a new movie. So it needs to watch movies that I liked, build a list of common things between these movies and then watch 100 random new movies and pick the one that has the most patterns in common with my liked movies.

So once again a good similarity recognition system is the key to successfull AI.


How to prevent overfitting in stacked models?

The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.

As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN.  All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting.  A few can be eliminated up front.  For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.

H. Hutson, S. Geva, and P. Cimiano wrote, in their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting."  Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.

Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation."  Reading their paper, Deep Forest, may give you some causes to evaluate.

Over-fitting is usually the application of too sophisticated a model to which data is fit.  In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.

Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.


How to improve a trained model over time (i.e. with more predictions)?

That is exactly a neural network works like.
Suppose you have a 1000 examples. How you train a network is: First, you divide these 1000 into maybe 100 batches (10 each). After that's done, you feed a batch to the network get its output and compare it with the ground truth, whatever is the error gets backpropagated. Then, for the next batch and then another. Once all these batches are done, you say an epoch is over. So, the number of epoch is effectively the number of times the network has seen the whole data.
This is how a neural network gets better.


Compute state space from variables in Q-learning (RL)

It seems to fail if a variable is not a power of two (bonus question: why doesn't it work?).

This does not work because you are wasting some space, some values are not used in your representation.
For example with your 0-4 variables, you need 3 bits, but you only use 000, 001, 010, 011 and 100 values. The values 101, 110 and 111 are still part of the representation you are using. It doesn't matter that you don't need to use them, you have created a representation where they exist. Every time you encode 5 values as 3 bits, you are being 62.5% efficient, and the efficiencies multiply to get the overall efficiency of your representation (proportion of states you need to represent compared to the size of the representation).
Sometimes this is acceptable. Your example state representation using bitwise coding per variable would still fit easily into a 32 bit integer for storage, and you also have the convenience of easily extracting the individual values quickly using bit masks. It's a bit less convenient if you were hoping to build a Q table as an array with the state as a simple offset.
In your example case it is not super-wasteful, your array will be around 4 times larger than optimal due to wastage in 3 separate places. For convenience of coding simplicity you might accept this 8MB space per action over 2MB space per action (assuming 32 bit floating point values for action values). 6MB of dead space is not much to worry about - the chances are that the programming language you have loaded to run it wastes far more space on features that you are not using.
If you have a few more variables and a few more wasted bit patterns, then the waste could be more noticeable and important. That might also be true if you are using a space-efficient compiled language on a system where memory resources are low.

Let's say in my problem there are following variables which i'm using to compute state . . . So the state space is equal to 64000 (4 * 4 * 4 * 2 * 2 * 5 * 5 * 5 * 2)

You are very close to a working answer for the most efficient representation here. You can use products of each variable's size to separate terms, to create multiplication factors for each variable. Imagine building up a cuboid out of the first three terms, and needing to address each cubic building block sequentially with an index position $i$. You would do this:
$$i = x + 4y + (4 \times 4)d = x + 4y + 16d$$
If you had only thie first 5 variables you would do this:
$$i = x + 4y + (4 \times 4)d + (4 \times 4 \times 4)g + (4 \times 4 \times 4 \times 2)a = x + 4y + 16d + 64g + 128a$$
To cover your whole set of variables, keep extending the same pattern. Each variable added to the end of the list is multiplied by the product of the space required by all previous variables. The last variable in your example, gl, would be multiplied by $32000$.
Reversing this encoding is more involved, especially if you only want to access a single variable. You need to start with the value of $i$ then repeatedly use an integer divmod operation to find the next unknown variable at the end of the list - using the same multiplier for that variable as when constructing the code - and a remainder for calculating the next variable along.
Luckily in reinforcement learning you probably don't need to do any reversing of state id to individual variables - you can maintain a structure of all current variables for the state when running the environment, and only need this compressed version to create an offset into the Q table for lookups for the policy or updates to stored values.


When to use AND and when to use Implies in first-order logic?

When considering the sentence "Some boys are intelligent", it makes sense to express it by ∃x boys(x) ∧ intelligent(x). This is because the existential quantifier makes sure to express that at least some, but not necessarily all boys are intelligent. More specifically, you say that there exists something which is a boy and which is intelligent. But if such a something exists, then there are some boys which are intelligent. Your statement is satisfied.
Using a simple implication → here wouldn't work semantically in that context. For example, ∃x boys(x) → intelligent(x) expresses that: For some x, if x is a boy, then x is intelligent. By rewriting, we would get ∃x ¬boys(x) ∨ intelligent(x). This statement would even be true if all x were not boys, because ¬boys(x) would be true in these cases. So, ∃x boys(x) → intelligent(x) would hold as soon as something in your model wasn't a boy. In that case, ∃x boys(x) → intelligent(x) would be true, but "Some boys are intelligent" would not really be satisfied, since there are simply no boys in your model which could be intelligent. So, you will have to resort to something like ∃x boys(x) ∧ intelligent(x), since otherwise there is no guarantee that boys (even intelligent ones) are present in the resulting model.


How does Mask R-CNN automatically output a different number of objects on the image?

Object detection models usually generate multiple detections per object. Duplicates are removed in a post-processing step called Non-Maximum Suppression (NMS).
The Pytorch code that performs this post-processing is called here in the RegionProposalNetwork class. The filtering loop you've mentioned performs the NMS and applies the score_thresh threshold (although it seems to be zero by default).


In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?

Generally the agent does not need to "think" what the next state will be. After taking an action, it should observe the reward and the next state.

The logic for the next state is controlled by the environment. In your case, the environment of the simulator should accept the action choice and then not allow the sideways movement off the road, and present the correct logical next state. It may in addition alter the reward signal if you want to penalise this choice (in comments you say this is not done - that's fine, it's a choice you make on problem set up, what behaviour to reward or penalise).

A more advanced agent might also try to predict what the next state will be, especially if it was using a planning algorithm (e.g. Dyna-Q). Even then there is no reason or rule that it should have to predict an impossible next state - a well-trained agent should in fact predict correctly that moving right is same as "do nothing" when the car is in far right-hand lane. For a value-based approach, like Q-learning the agent may do this simply by scoring the two actions with the same value.

In your case, you do not need to try and second-guess what the agent will predict. Provided your state information is enough for the agent to observe that it is in the far left or far right lane, it should [eventually, after training] correctly predict the value functions and/or optimal actions as it learns the environment. That includes, after a few examples, that it will predict the outcome when moving right when already on right hand side. For that to happen reliably, you do need to check your state representation is adequate, and function approximation hyper-parameters are suited to the problem (the latter part could take some experimentation)

If the state data that the car receives does not contain enough information for the agent to tell that it is in extreme left or right lane, then you will have a problem. This is entirely possible in general, although from the image you show, that does not seem to be an issue.

One possible related issue is if the other cars in the scene all have different velocities. This velocity information is not shown in the state, and could cause your agent problems as it cannot tell the difference between a stationary car in front of it, and one which better matches its own velocity. You could treat this missing data as a partially observable state, and adjust you agent. Or you might be able to do the same as the Deep Mind DQN team for Atari games and use a sequence of e.g. last 4 images as the state, which should then include enough velocity data to make reliable estimates of state/action values.


I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you take out a random small subset which will be your new small validation set and do all the 'operations' on that which you are also doing on your training set. This way, you can check that these 'operations' are thus also generalizable (if they work for all different cross-validations).


What is the relationship between the reward function and the value function?

I think it is pedagogically useful to distinguish between the theory (equations) and the practice (algorithms).
If you're talking about the definition of the value function (the theory)
\begin{align}
v_{\pi}(s)
& \dot{=} \mathbb{E}_{\pi} \left[ G_t \mid S_t = s \right]\\
&= \mathbb{E}_{\pi} \left[ \sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigl\vert S_t = s \right]\\
\end{align}
for all $s \in \mathcal{S}$, where $\dot{=}$ means "is defined as" and $\mathcal{S}$ is the state space, then the value function can be defined in terms of the reward, as it can be clearly seen above. (Note that $R_{t+k+1}$, $G$ and $S_t$ are random variables, and, in fact, expectations are taken with respect to random variables).
The definition above can actually be expanded to be a Bellman equation (i.e. a recursive equation) defined in terms of the reward function $R(s, a)$ of the underlying MDP. However, often, rather than the notation $R(s, a)$, you will see $p(s', r \mid s, a)$ (which represents the combination of the transition probability function and the reward function). Consequently, the value is a function of the reward.
If you're estimating a value function (the practice), e.g. using Q-learning, you don't necessarily use the reward function of the Markov decision process. You can estimate the value function by just observing the rewards that you receive while exploring the environment, without really knowing the reward function. But, by exploring the environment, you can actually estimate the reward function. For example, if every time you're in state $s$ you take action $a$ and you receive reward $r$, then you already know something about the actual underlying reward function. If you explore enough the MDP, you could potentially learn the reward function too (unless it keeps on changing, in that case, it may be more difficult to learn it).
To conclude, yes, value functions are certainly very related to reward functions and rewards, in ways that you immediately see from the equations that define the value functions.


Which deep reinforcement learning algorithm is appropriate for my problem?

Theoretically video games and robotics problems are also about optimization(getting maximum reward). So, just like other reinforcement learning problems, I would expect PPO to be the most efficient in your case too. I don't think a "goal" is necessary for rl, all you need is the rewards.


Is there a machine learning model that can be trained with labels that only say how "right" or "wrong" it was?

What you are looking for is called "reinforcement learning".
A reinforcement learning algorithm will try to maximize a reward function. This reward represents how "good" or "bad" an action is in the actual context. For example, in the snake game, your reward will be positive for eating an apple and negative when the snake hits a wall.
The interesting thing is that, with reinforcement learning, you can learn without having a reward at each step. In the case of the snake game, your agent can learn that going in the direction of the apple is better than going in the direction of the wall, even if none of this action will directly give a reward (positive or negative).
If you want to use a neural network as your post seem to imply then you should look at deep Q-learning, a reinforcement learning algorithm, which use a neural network to learn to predict the expected reward of a couple (state, action).


How to design an AI that discovers more complex concepts on its own?

This is not an answer. I couldn't comment, so here are some remarks about your question: This is a very broad question, and considered The Holy Grail for building artificially intelligent systems - meaning that some scientists have been dreaming about this since time immemorial.

Some homework is warranted from your side; you could have offered some of your solutions or maybe identified the multiple layers to your query, as they invoke concepts from many fields within the study of AI (or AGI to be general).

For example, the following layers are taken up around the big-words in the question, though rhetorically -

On evolution - what registers as evolution (is only genetic mutation evolution or should it involve some form of natural selection or seeking a niche for increasing the chances of survival or etc). How would evolution of software look like? Should it be able to modify its own code in the process of evolution?
On concept - what constitutes a concept, identifying from the environment a concept by calculating its relevance (w.r.t other concepts), coming with a process of selecting the concept for use in a given environment (natural or artificial).Referring to the example in the question, is the road a relevant concept or the trees and the sky and the bees pollinating some flowers along the roadside? What is a more fundamental concept - the trees or the bees, and how does one measure that?
On rewards - for us humans, rewards are maximization of survival of genetic machinery they carry (translating to reproductive success). what reward system should we come up that machines could use to increase their chances of survival in the physical world? what value should be put on a reward for moving the car on a straight line, or moving from darkness towards light? Shouldn't moving from darkness to light be more fundamental action (concept) and therefore rewarded higher than moving along the straight line? However, given that the car has learned to move from darkness to light, shouldn't the value of the reward be lowered so that it can learn other actions/concepts?

As can be seen from this brief detailing, there are layers within layers. Therefore, it is proper to establish now that there are no simple answer to this essay-requiring soul-searching time-eating heavy-research-wanting question. However appropriate directions can be given about work that is being done by some of the most prominent scientists of our times. The thing that one is looking for are known as Universal Problems Solvers, such as Gödel Machine (by Jürgen Schmidhuber) and AIXI - Artificial Intelligence (AI) based on Solomonoff's distribution ξ (by Markus Hutter).

Here is a quote lifted from Wikipedia page on AIXI that is pretty self-explanatory on how it maximizes the rewards over time.

  AIXI is a reinforcement learning agent. It maximizes the expected
  total rewards received from the environment. Intuitively, it
  simultaneously considers every computable hypothesis (or environment).
  In each time step, it looks at every possible program and evaluates
  how many rewards that program generates depending on the next action
  taken. The promised rewards are then weighted by the subjective belief
  that this program constitutes the true environment. This belief is
  computed from the length of the program: longer programs are
  considered less likely, in line with Occam's razor. AIXI then selects
  the action that has the highest expected total reward in the weighted
  sum of all these programs.

Gödel Machine goes further - it allows the agent to modify its own code that allows it to maximise the rewards of its actions - which is to modify its own code that allows it to maximize the rewards of its actions - as so on. This is kind of a recursive definition, that simulates evolution (a rapid one) by choosing to evolve that code/state of the agent that converges towards a code that is superior to the code that the agent is currently running.

Here is a quote lifted from page on summary of Gödel machine. See that Hutter is also referenced (discovered of AIXI above).

  Our Gödel machine will never get worse than its initial problem
  solving strategy, and has a chance of getting much better, provided
  the nature of the given problem allows for a provably useful rewrite
  of the initial strategy, or of the proof searcher. The Gödel machine
  may be viewed as a self-referential universal problem solver that can
  formally talk about itself, in particular about its performance. It
  may "step outside of itself" (Hofstadter, 1979) by rewriting its
  axioms and utility function or augmenting its hardware, provided this
  is provably useful. Its conceptual simplicity notwithstanding, the
  Gödel machine explicitly addresses the `Grand Problem of Artificial
  Intelligence' by optimally dealing with limited resources in general
  environments, and with the possibly huge (but constant) slowdowns
  buried by previous approaches (Hutter, 2001, 2002) in the widely used
  but sometimes misleading O()-notation of theoretical computer science.

  The main limitation of the Gödel machine is that it cannot profit from
  self-improvements whose usefulness it cannot prove in time.

Reading the papers of the stated research would certainly be useful in finding answers to some of the layers in the question. There is also good work going on in the AGI community that somewhat aligns with the direction. Hope this helps.


The authors of the original paper don't provide an explanation, but I suspect it's a combination of:

popular recognizable branding (cf. BERT, DALL-E, Watson etc)
similarity to [sequence] transduction / translation / transformations generally


Bellman optimality equation in semi Markov decision process

The core problem here is state representation, not estimating return due to delayed response to actions on the original state representation (which is no longer complete for the new problem). If you fix that, then you can solve your problem as a normal MDP, and base calculations on single timesteps. This allows you to continue using dynamic programming to solve it, provided the state space remains small enough.
What needs to change is the state representation and state transitions. Instead of orders resulting in immediate change of stock levels, they become pending changes, and for each item you will have state representation for the amount of current stock, plus amount of stock in each lead time category. State transitions will modify expected lead time for each amount of pending stock as well as amount of current stock.
Your lead time categories will depend on whether the agent knows the lead time immediately after making an order:

If lead times are known, track remaining time until items arrive 1,2 or 3 days. These categories will be assigned by the enviroment following the order, then lead time will transition down on each day deterministically. A 1 day lead time will transition to in stock, 2 day lead will transition to 1 day etc.

If lead times are not known, but probabilities of them are, track time since the order was made. This will be 0, 1 or 2 days. Although you don't know when an order will arrive, you know the probabilities for state transition - e.g. items in 0 days have a 1 in 3 chance of transitioning to "in stock" and a 2 in 3 chance of transitioning to 1 days.

This makes the state space larger, but is less complex than moving to the Semi MDP representation. For instance, doing it this way means that you can still work with single time step transitions and apply dynamic programming in a standard way.
In general, if the environment has a delayed response to actions, then the  best way to maintain Markov trait is to add relevant history of actions taken to the state. The added state variables can either be a direct list of the relevant actions, or something that tracks the logical consequence of those actions.


Your problem is often called (in the literature) human mobility prediction. There has been some research in this area. Have a look at it on the web.

In general, you might want to use any statistical or machine learning model that uses the historical data to predict the future. For example, you could try to use an hidden Markov model (for both point 1 and 2). However, before that, you might also need to do some feature engineering, if you only have locations (and not e.g. the time of the day when those locations were recorded).


Would an AI with human intelligence have the same rights as a human under current legal frameworks?

Yes, to some of what you propose.  No to some.

Today corporations are granted rights: to own property, earn income, pay taxes, contribute to political campaigns, offer opinion in public, ad more.  Even now I see no reason why an AI should not be eligible to incorporate itself, thereby inheriting all these rights.  Conversely, any corporation already in existence could become fully automated at any time (and some plausibly will).  In doing so, they should not lose any of the rights and duties they currently employ.

However I suspect certain rights would be unavailable to an AI just as they are unavailable to a corporation now: marriage, draft or voluntary service in the military, rights due a parent or child or spouse, estate inheritance, etc.

Could this schizoid sense of human identity be resolved at some point?  Sure.  Already there have been numerous laws introduced and some passed elevating various nonhuman species to higher levels of civil rights that only humans heretofore enjoyed: chimpanzees, cetaceans, parrots and others have been identified as 'higher functioning' and longer lived, and so, are now protected from abuse in ways that food animals, pets, and lab animals are not.

Once AI 'beings' arise that operate for years and express intelligence and emotions that approach human-level and lifetime, I would expect a political will to arise to define, establish, and defend their civil rights.  And as humans become more cybernetically augmented, especially cognitively, the line that separates us from creatures of pure silicon will begin to blur.  In time it will become unconscionable to overlook the rights of beings simply because they contain 'too little flesh'.


Research paths/areas for improving the performance of CNNs when faced with limited data

Some research areas that come to mind which can be useful when faced with a limited amount of data:

Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:

Injecting noise in the network, e.g., dropout.

Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also confident output distributions can be penalized.

Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, increasing the number of parameters for large models can also improve the validation performance.

Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.

Generating new training data:

Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.

Data interpolation, e.g., manifold mixup.

Using synthetic data, e.g., frames from video games or other CGI.

Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.

Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.

Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.

Other interesting methods can be found in systems that perform one-shot learning, which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a triplet loss (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.
Zero-shot learning also exists (e.g., zero-shot machine translation), but this is a completely different type of problem setup and requires multiple data modalities.


Why do some DQN implementations not require random exploration but instead emulate all actions?

The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an option that is available in all environments, but if possible it resolves the exploration/eploitation dilemma for a single time step nicely by investigating all options.
Longer term the agent proposed by the link does not sufficiently explore for general use in my opinion. It appears to always choose a single action deterministically based on maximising action value. In other words it always attempts to exploit the training data so far, even though it augments the training data with short-term knowledge about exploration. However, this appears to be sufficient in the problem domain that it is used in. I suspect this is for a couple of reasons:

The environment is non-stationary, making long-term state predictions unreliable in any case. An agent that learns to exploit in the short term (i.e. over only a few time steps into the future) is likely to be approximately optimal already.

State transitions may be highly stochastic, meaning that state space will still be adequately explored even using a deterministic policy. This feature of the environment is also used by other well-known Q learning approaches with deterministic behaviour policies, such as TD Gammon

I think you have correctly identified a weakness of the approach used in the linked paper that means it may not make a strong general algorithm. The algorithm avoids addressing the exploration/exploitation balance in full, and instead relies on features of the environment to work well despite this. If you find yourself working in similar environments for your own projects, then it may well be worth trying the same approach. However, if you find yourself working in a more deterministic environment with more stationary behaviour and sparse rewards, the lack of state space exploration would be a serious limitation.


Embedding Isolation game states into key values for RL

I think that there are too many game states in that game for you to use value iteration. The upper bound for a simple concise representation would be $49^2 \times 2^{49}$. That is:

$49^2$ covering all possible locations of the two players.

$2^{49}$ covering whether each square exists or has been removed.

In this scheme, many combinations are not feasible, as pieces do have to be on existing squares and cannot share a space. However, this doesn't drop the number of allowed states by a significant amount (several or more orders of magnitude) that would make it worth a more complex representation, or put the game into reach of dynamic programming solutions.
Potentially, a version on a 4x5 grid would be small enough to fit in memory and be solvable with dynamic programming. That would have a few million valid states.
In terms of an id code, you could use a 64-bit unsigned integer, reserving 49 bits for the existence of the squares, and for simplicity giving 6 bits each to the locations. This would also be a valid state representation for actually playing the game efficiently, most programming languages support the bit manipulations that you would need in order to maintain the representation. However, it would need a separate expanded representation if you were to create neural network features for approximate value functions.
In terms of writing a solver for the 7x7 version, I would recommend combining a look-ahead planner such as negamax, with an approximate neural network-based reinforcement method, perhaps DQN for simplicity. The neural network would provide a backup "best guess" solution when you could not look far enough ahead in the early stages of the game.
On a quick search for the true number of states in this game, I found an introduction to using minimax on a smaller version that you may find useful.


Is there a reliable website about artificial intelligence that is frequently updated?

For sites, https://towardsdatascience.com/ maybe a good choice. The articles on there is about AI. You can read a lot of articles about state-of-the-art AI networks there. You can also subscribe to their newsletter for emails everyday on new AI discoveries. However, as stated by @DuttaA, take the articles with caution. The articles may or may not be entirely true as they are written by the community. Choose articles with large number of likes to read.

The YouTube channel two minutes paper is also a great resource. The channel posts videos about AI and also computer science periodically, and the content is easy to understand and enjoyable.

Reminder: for the towards data science site, you should disable cookies for it so that there will not be a limit on how many articles you read.

Hope I can help you and have a nice day


The question would need clarification. I'm new here so I will try to give an answer.
The easiest way would be to prepare the data in comma separated values (CSV). It is possible to export your data in this format from excel.
For downstream applications it will depends what programming language is used but in general it is possible to import CSV files and the data they contain for training (example panda data frame in python).

Hope this helps!


Why would someone want to simulate emotional desicion-making within an AI?

Humans have poor understanding of emotional rules. Probably every poster on here has experienced greatly misreading another individual emotionally. Further, people often don't act emotionally how they would expect themselves to act, for example we have all experienced frustration at someone else's irrational concerns and yet we are all guilty of holding irrational concerns of our own. This is the crux of why hard-coded emotional rules do not work - we do not have an understanding of what emotional 'rules' make someone feel real.

By moving towards autonomous state-based actors we move away from this issue. The actor's state transitions will of course be defined rules (hard-coded or learnt) but by abstracting the actor's emotional state from the specific context (e.g. specific actions trigger emotional state transitions which trigger responses, instead of a direct response to a specific action), the programmer prevents them-self from projecting their own beliefs/emotions/logic onto the actor.

Further, autonomous actors are more extendable. Consider an autonomous actor that is hard-coded to move towards 'upset' and 'angry' emotional states when experiencing 'pain'. Simply by associating a new world action with 'pain', one can trigger an emotional response from an autonomous actor that has not experienced that action before. When working in hard-coded emotional rules, this would not be possible.


Is feature engineer an important step for a deep learning approach?

From what I believe, feature engineering is important, it's a part of the job of ML network designer.
Network designing involves

Feature engineering: What should be in the input to the network, as processed from similar or totally different data
Deciding network shape, layer shapes, types of neurons in layers, etc.
Feature engineering again (but labels), in the output, what should the output be, either regression values or classes

And possibly also tasks rather simple as mentioned in the question: filling missing values, normalising data, create pre-feeding normalisation steps in code, etc.


RL: how can the cart-pole problem be a continuing task?

The key is that reinforcement learning through something like, say, SARSA, works by splitting up the state space into discrete points, and then trying to learn the best action at every point.

To do this, it tries to pick actions that maximize the reward signal, possibly subject to some kind of exploration policy like epsilon-greedy.

In cart-pole, two common reward signals are:

Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
Receive a reward that linearly increases with the distance the pole is off the ground.

In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.

However, an offline algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An online algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.


What does "reimplementations of deep learning algorithms which replicate performance from the papers" mean?

I believe that this:

"reimplementations of deep learning algorithms"
He is asking that they are looking for people who have made and completed an AI that perform similarly or exactly as those given in papers (IDK which papers)
"Open-source" The Fellow wants to be able to see the source code of the project.


One probable hardware limiting factor is internal bandwidth.  A human brain has $10^{15}$ synapses.  Even if each is only exchanging a few bits of information per second, that's on the order of $10^{15}$ bytes/sec internal bandwidth.  A fast GPU (like those used to train neural networks) might approach $10^{11}$ bytes/sec of internal bandwidth.  You could have 10,000 of these together to get something close to the total internal bandwidth of the human brain, but the interconnects between the nodes would be relatively slow, and would bottleneck the flow of information between different parts of the "brain."
Another limitation might be raw processing power.  A modern GPU has maybe 5,000 math units.  Each unit has a cycle time of ~1 ns, and might require ~1000 cycles to do the equivalent processing work one neuron does in ~1/10 second (this value is totally pulled from the air; we don't really know the most efficient way to match brain processing in silicon).  So, a single GPU might be able to match $5 \times 10^8$ neurons in real-time.  You would optimally need 200 of them to match the processing power of the brain.
This back-of-the-envelope calculation shows that internal bandwidth is probably a more severe constraint.


Why do we use $D(x \mid y)$ and not $D(x,y)$ in conditional generative adversarial networks?

It looks like you're asking about the difference between using conditional and joint probabilities.

The joint probability $$D(x,y)$$ is the probability of x and y both happening together.

The conditional probability $$D(x | y)$$ is the probability that x happens, given that y has already happened. So, $$D(x,y) = D(y) * D(x | y)$$.

Notice that, in a C-GAN, we have some extra information that is given, like a class label $y$. We actually don't care at all about how likely that information is to appear. We care only about how likely it is to appear with a given $x$ from the source distribution, versus how likely it is to appear with a given $z$ from the generated distribution.

If you tried to minimize the joint probabilities, you would be attempting to change something that the networks have no ability to control (the chance of $y$ appearing).


How to compute number of weights of CNN?

Calculating parameter number in a CNN is very straightforward.
CNN is composed of different filters which is essentially a 3d tensor. CNN weights are shared meaning they are used multiple times and reused in different locations. Each layer have n tensors each with dimension w * h * c where w = width, h = height, c = channels (the input channel size), therefore getting param_no = w * h * c * n. There is also a bias for each output channel, so param_no_bias = n. At the end the parameter number is calculated with: n * w * h * c + n. See more about this in hear: Article
The pooling layer does not have a weight, it only has hyperparameters. You may have confused the two. There are hyperparameters for the stride, the factor and etc. These are predefined and not trainable.
For a keras API, @pasabaporaqui has mentioned https://stackoverflow.com/a/35827171/4886927 which definitely works.
Hope I can help you.


A* is similar to Dijkstra with reduced cost

What you are doing when calculating $d'(x,y)$:

$d(x,y)$: calculating the original edge distance from $x$ to $y$
$h(y)$: plus the heuristic from $y$ to the goal
$h(x)$: minus the heuristic from $x$ to the goal

So, using this recalculation of the original edge-values ($1.$) in Dijkstra's algorithm you are inherently accounting for the heuristic component of A* by incorporating it ($2.$) into the value of the edge traversed, and discarding ($3.$) the accumulated previous heuristic values of previous nodes in the path.

The additional condition $h(x) ≤ d(x, y) + h(y)$ ensures the new edge-values are strictly positive.


How could AI solve planet's major problems?

There's indeed a lot of research work being done in this field.

Here are a few ways in which AI can help/is helping in fighting major problems:

Climate change

Identify deforestation and the rate at which it's happening using computer vision and help in fighting back based on how critical the rate is.The World Resources Institute had entered into a partnership with Orbital Insight on this.

Protecting Nature

Researchers and institutes are using computer vision to accurately identify total remaining members of extremely threatened species to observe their behavior and patterns that can help in protecting them, keep track and monitor at an individual level.
Environmental/ Research agencies are trying to detect sounds from ocean audio from which they can identify inhabiting fish and route the ships away from threatened habitats.

Energy

Optimise energy usage based on consumption in big companies to save energy for public use and in turn reduce energy overall demands on the grid.
Predict the demand for energy and ensure continuous supply matching the demand. For example, Google's Deepmind is in talks National Grid of UK on this.

Of course, these ideas might look too tiny or too broad at first sight, but I feel taking one step at a time and then quickly expanding the ideas into wider areas would be the way to go, like how DeepMind is doing it in the area of energy. Other areas like education, transportation, and healthcare are seeing lots of activity with the involvement of AI too.

For further reading, here's what World Economic Forum thinks about how AI can help solve world's problems.


Is it legal to license and sell the output of a neural network that was trained on data that you don't own the license to?

Disclaimer: I am not an attorney and this does not constitute formal legal advice.

If the output is novel the copyright resides with the creator

In this case, almost certainly the human who utilizes the algorithm†.  There was a recent US patent case "Dabus" [U.S. Patent Application No.: 16/524,350] where the human programmers tried to claim an AI as an inventor, which was rejected by USPTO.  This is largely because an inventor is defined as a "natural person".  But it's an interesting challenge to the notion of inventorship and authorship.
An argument regarding output specifically as a salable commodity would be that most human creative endeavor is result of previous work that forms the basis for the novel arrangement of elements in the form of an original work.
Music is more specific because melodies are mathematical, and what are traditionally protected, although Blurred Lines case blurred this line, in that the ruling there was the production aspect, the "feel", was what was plagiarized.
And, in fact, algorithmic music generation, which is heavily utilized in pop music, depends on combinatorial functions producing novel sequences, not protected by prior copyright.
Samples are clear cut—it is excerpting from copyrighted work.  A wave form is then copyrightable, in that a work of recorded music is simply an arrangement of waveforms in some combination and sequence.
If the output is procedural, and a process that can be automated, that would enter the domain of patent law, utility patents specifically.
Design output would also be in the realm of design patents, which are specific/generalized arrangements of elements in products that are not processes.

A note on liability:
Law is a process, where precedent plays a major factor.  Because law uses natural language, there is ambiguity, and jurisprudence is the process of clarifying the meaning and application via challenges.  Especially when in uncharted territory "no one knows" until a case has been ruled on, and rulings can be challenged all the way up to the Supreme Court, which may or may not accept a suit.
Damages for copyright are 3x, but damages can be very hard to prove.  Intellectual Property litigation is also enormously expensive, thus unlikely to be pursued unless there is a potential financial benefit.  (The first step in any potential copyright violation is typically a "cease and desist" letter, with no legal action if acceded to. In a case such as "Blurred Lines", where there is significant financial return to the alleged infringer, the award or settlement presumably exceeds the cost of litigation.)
However, deep-pocket players can use litigation, or threat of litigation, to disincentivize competitors, and, if the target has limited resources, can produce the desired outcome regardless of the strength of the claim.  (It's not uncommon in law in general to file frivolous suits as a strategy, although there is typically a financial penalty  such as reimbursement of the defendant's legal fees if the suit is found to be frivolous and dismissed.) "Patent trolling" became so much of a problem before 2013, the entire philosophy of patentability, what is patentable, had to be revisited.

†Who has rights to an algorithmic process is also a legal question.
Software is patentable, being simply a type of computer, regardless of medium, but most software is not patented, and protected instead as "trade secrets" and via non-disclosure agreements.  (This may be due to the glacial pace of the patent process compared to the software development process, but also per the from the necessity of making patents public.  i.e. if Google patented their search algorithm, it would be an instruction manual on how to exploit it.)
Software is copyrightable as the specific lines of code.  Copyright naturally resides with the creator, but the process of copyright is be utilized to document the right against potential infringement.


It's all about Return On Investment. If DL is "worth doing", it's not overkill.

If the cost of using DL (computer cycles, storage, training time) is acceptable, and the data available to train it is plentiful, and if the marginal advantage over alternative algorithms is valuable, then DL is a win.

But, as you suggest, if your problem is amenable to alternate methods, especially if it offers a signal that matches up well with classic methods like regression or naive Bayes, or your problem requires explanation of why the decision boundary is where it is (e.g. decision trees), or if your data lacks the continuous gradients needed by DL (especially, CNNs), or your data varies over time which would require periodic retraining (especially, at unpredictable intervals), then DL probably is a mismatch for you.


Can TD($\lambda$) be used with deep reinforcement learning?

Eligibility traces is a method of weighting between temporal-difference "targets" and Monte-Carlo "returns". In practice, for example, instead of using the one-step TD target, $r_t + \gamma V (s_{t+1})$, as in the temporal difference update $V (s_t) \leftarrow V (s_t) + \alpha (r_t + \gamma V (s_{t+1}) − V (s_t))$, you use the so-called "lambda" ($\lambda$) target, which is a target that balances between the TD target and the Monte Carlo return. So, in practice and intuitively, eligibility traces is just a way of using a more "appropriate" target while learning. In general, you need to perform these updates (e.g., the TD update above) "online", i.e. while you explore or exploit the environment.
In theory, you could use a deep neural network to represent your value function (or your policy), while using eligibility traces. It would be similar to not using them: you would just use a different target.
However, deep RL (that is, RL which uses deep neural networks to represent e.g. value functions) training needs to be performed using i.i.d. data, in order to prevent overfitting, which often means that they can't be trained online or need to use "tricks" like the "experience replay" (used in the paper Human-level control through deep reinforcement learning). Note that, in RL, successive states are often very correlated (e.g. two successive frames of a video would be very correlated).
In theory and similarly, you would still be able to use eligibility traces with the actor-critic method, but not with the asynchronous advantage actor-critic method. See the section 2.3 of the paper "Efficient Eligibility Traces for Deep Reinforcement Learning" (2018) by Brett Daley and Christopher Amato, for more info.
In this same paper, an approach is introduced to efficiently combine eligibility traces with deep neural networks. The authors propose DQN($\lambda$), which is the DQN architecture combined with eligibility traces, where the $\lambda$ return is computed in an "efficient" (and recursive) way, instead of the "usual" way. Since they use a DQN, they also use an "experience replay" buffer (or memory), where they also store the efficiently computed $\lambda$ target (in addition to the usual rewards). Furthermore, they also eliminate the need for the "target" network used in the standard DQN. You can have a look at algorithm 1 of the same paper to see how they improve the parameters of the network, which represents the Q function, in the case of the DQN($\lambda$) model. See the section 3.1 of the same paper for more details regarding this model.
They also introduce A3C($\lambda$), which combines asynchronous advantage actor-critic (A3C) with eligibility traces. See the section 3.2 for more details.
Note that there have been other proposals for combining eligibility traces with deep learning. You can have a look at the literature.


Reinforcement learning for robotic motion planning - Problem statement ideas

I think that this may be too difficult for only a few weeks of work. There are a few reasons why and although all of the reasons are themselves learning opportunities, it may cost you more time than you have.

First off, Robots are hard to work with. Ask anyone who has worked with robots and they will say that have broken their heart at least once. This is because either the robot was a custom robot which needs specific skills (soldering, 3d printing, etc) if something breaks (and it will) or it's a purchased robot platform which doesn't have the control you want or doesn't have good documentation.

Secondly, Machine learning is a very interesting and broad field and there is a lot of literature one should get a background on to help with their projects. Because of this, it might be overwhelming to take all of that information in while having to deal with a robot.

I don't want to dissuade you in this research area though. It is very interesting and has many subproblems like dynamic environments, multilevel environments, etc. But given the time you have, it might be better to set yourself up for success. Perhaps something which lets you do a literature review about the problem anyway.
Perhaps a more attainable goal in that time while still using a robot is: Predicting wall collisions using temporal difference learning.

If you agent can predict when a wall is coming, clearly they will be able to avoid them too but it's a very reasonable first step.

Additionally, talking to the professor of the course or your supervisor (if you have one) will also help you decide.


How to use Machine Learning with simple games?

You have implemented a simple contextual bandit solver, which is a machine learning algorithm. A few details may be different from a full implementation, but the key elements are:

A choice of actions (click hit or don't click hit)
A reward signal that can be observed after each action (+1 for a hit, 0 for nothing happens, -1 for an attack which misses)
An observable state which affects the reward achievable (the position of the blue ball). For a contextual bandit, the state is not influenced by the action taken. This is true here.
One thing that is different about your problem from a classic contextual bandit is that the next state is predictable from the current state (whilst in a pure bandit problem it should be entirely random). However, that's not too important to your problem here, and your solver is definitely following a contextual bandit approach.
Your solver tests the score from trying different actions in each state, and narrows down the best action to take in each state. Your implementation is simple and "greedy" for a contextual bandit solver. A more typical solution would maintain an average result for each action and have a rule for how to explore actions in each state, so it could test whether results were reliable (this is very helpful with non-deterministic scenarios where bandit solvers are more often used).

  With each hit, the blue ball moves faster

Unless you somehow limit the reaction time of the agent, this is not relevant to how you write the solver. You could change the rules affecting the agent to make it relevant in the same way as it would be for a human, e.g. deciding to click means the click happens 0.1 seconds later, and the state can include observations of position just now and several 0.02 seconds going back.

In general, if you want to take this further, with more complex games and still learning how to control agent actions, you could look at simple reinforcement learning agents, such as Q-learning. If you are interested in the underlying theory of agents like this, then a good (and free) introductory text is Sutton & Barto "Reinforcement Learning: An Introduction"


RMSprop equation - dividing by a matrix?

Yes, you are correct, $S_{dW}$, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in $S_{dW}$ or you can say "element-wise division".
To avoid the zero variable in $S_{dW}$, the equation usually like this:
$$
W=W-\alpha \frac{dW}{\sqrt{S_{dW}+\epsilon}} \text{  where } \epsilon=10^{-8}
$$
If you want to implement it manually with Python, you can take a look at this link


Does the Bayesian MAP give a probability distribution over unseen t*?

You're correct: the MAP estimate is a point estimate (specifically, MAP is used to estimate the mode of a probability distribution).
I think that the paper is referring to the (output) probability distribution over the possible targets/labels, given the point estimate. However, in the case of MLE, you can also have that probability distribution, so I'm not sure why the paper's author is emphasizing that with MAP you can build that probability distribution (so maybe this/my interpretation of that excerpt is wrong!).
That table also shows that the MAP estimate is used to produce a probability distribution, but the $\sigma$ there should be unknown, but I didn't read that article, so maybe I am missing some info or assumption.
In any case, you could also find a point estimate of a parameter of a probability distribution, but this does not imply that MAP produces a probability distribution. For instance, you can show that, if you place a Gaussian prior on the weights of a neural network, this leads to the $L_2$ loss function, but training a (normal) neural network with $L_2$ does not lead to a probability distribution over the weights.
You should try reading only information from reliable references and books. For MAP, check out chapter 5. (p. 149) of the book "Machine Learning: A Probabilistic Perspective" by Murphy.


How to run a Monte Carlo Tree Search MCTS for stochastic environment?

I am having trouble understanding how to keep track of the expansion, do I expand all stochastic possibilities and weight the return via their chance of happening?

This is indeed one option you can take. This would be very similar in spirit to the idea of "Expectimax" as a variant of minimax for non-deterministic games, in the sense that you'll include explicit "chance nodes" in your tree. When running into such a chance node later on again during a Selection phase, of a later MCTS iteration, you can just select a path of the tree to follow based on a "dice roll". Importantly, note that this option is only actually available if you have explicit knowledge of exactly when chance events occur, which states they can lead to, and with which probabilities they lead to different states. We also assume that this is feasible, i.e. that you don't have a crazy high (or infinite) number of slightly different game states you could reach.
An alternative option is to use an "open-loop" variant of MCTS. Your nodes would no longer represent game states, but only be representative of the sequence of actions leading to them. You would no longer store any game states in any nodes, but always regenerate them from scratch when traversing the tree, starting from the root node. You would no longer have any explicit chance nodes, but instead have states being representative of larger sets of states that could possibly be reached by following the corresponding path from the root node. For more on this, see my answer to this other question. The advantage of this approach is that it does not require explicit knowledge of all the possible states you can reach due to chance events, do not need explicit knowledge of the probabilities, and can just sample instead of explicitly enumerating every possible outcome.


What is the difference between artificial intelligence and robots?

Although there are several definitions of "robot", an essential feature of everything called "robot" is that it is capable of movement. This does not necessarily mean displacement; a robot arm in a factory also moves.

There is a single exception to this rule,  which is bot-programs like chatbots; I will discuss them later.

Artificial Intelligence does not need to move; a chess program can be argued to be an AI, but does not move.

A robot can actually have AI; one of the definitions of robot is that it is a system, capable of autonomous movement. In order to be autonomous, to be able to make decisions of its own, a certain amount of AI may be necessary.

There is one class of "robots" that does not move, and does not even have physical presence; bot programs, like chatbots, that operate inside systems. I do not consider them robots, because they are not physical devices operating in the real world. A chatbot can be an AI, however - a good chatbot may have some natural language processing to interact with humans in a way that humans find natural.

To summarize; an AI can exist purely in software. But to be a robot, there must be a moving physical component in the real world.


How do we create a good agent that does not outperform humans?

You basically have to degrade the result, assuming that the machine always finds the best move. There are a number of possibilities:

restrict the depth of searching. In early chess programs I believe that was the main way of regulating the difficulty. You stop the evaluation of moves after a particular depth in your search tree has been reached. This would be equivalent to only looking ahead two moves instead of twenty.
set a time limit. This is somewhat similar the restricting the depth of the search, but more generally applicable. If your algorithm accumulates candidate moves, and the general tendency is to get to the better moves after first finding a number of weaker ones, then you can stop at a given point in time and return what you have found then.
distort available information. This might not be that applicable to games such a chess, but you could restrict the information the machine has available for evaluating moves. Something like the "Fog of War" often used in strategy games. With incomplete information it is harder to find a good move, though it is not impossible, which makes it more challenging than, say, restricting the depth of search too much.
sub-optimal evaluation function. If you have a function that evaluates the quality of a move, simply fudge that function to not return the best value. Perhaps add a random offset to the return value to make it less deterministic/predictable.

There are probably other methods as well; the tricky part is to tread the fine line between appearing to be a weaker (but consistent) player, and just being a random number generator.


Can I do deep learning with the 1060 or the 1070 ti?

Regarding specific choices I can't recommend, but if you are completely new, you should probably learn/code some more until you get a GPU. There is a lot to learn in machine learning before GPU speedups make a significant difference, and until then doing the computations on any old CPU would be just fine, especially if you are just starting since you won't be doing anything too complex. You will know when computational resources are your main bottleneck, and until then it shouldn't really matter too much.

Or, you could also rent computing power from say, AWS or Google


Can I calculate the training performance of GPUs by comparing their specification?

A lot matters when it comes to comparison of GPU's, I will give you a broad overview of the matter (it is not possible to go into exact details as huge number of factors are actually involved):

Cores - Number of CUDA cores increases means the parallelism has increased, thus multiple calculations can be done in parallel but is of no significance if your algorithm is inherently sequential. Then CUDA cores will not matter. Your library will parrallelize what it can parallelize and will take that many CUDA cores only, the rest will remain idle.
Memory - Memory is useful if you are working on data whose one instance requires huge memory (like pictures). Thus with greater memory you can load greater amount of data at the same time and the cores will process on that. If memory is too low then cores want data but it is not getting it (basically data available in RAM's is the fuel while cores are the engine, you cannot run jets on fuel tank the size of car, it will consume time to constantly refill the fuel). But according to convention of Machine Learning one should load only small mini-batches at a time.
Micro-architecture - Lastly, architecture matters. I do not exactly know how but NVIDIA's RTX is faster for deep learning than GTX. NVIDIA has two affordable architectures (Pascal - GTX and Turing - RTX). Thus even for exactly same specs Turing architecture will run faster for deep learning. But for more details you can explore NVIDIA's website on what architecture specialises in what. For example NVIDIA P-series is good for CAD purposes. Also there are some very high end GPU's using Tesla architecture.

So AFAIK, these are the factors that matter. The library you will be using also matters, as a lot depends on how the library unrolls your program and maps them on several GPU's. Also related these 2 answers I previously gave:

CPU preferences and specifications for a multi GPU deep-learning setup

Does fp32 & fp64 performance of GPU affect deep learning model training?

Hope this helps!


You can easily find a counterexample. Suppose that there are three nodes $s$, $p$, and $goal$ such that $s \rightarrow p \rightarrow goal$. The real cost of going from $s$ to $p$ is $c(s,p) = 10$ and $c(p, goal) = 10$. Also, $h_1(s) = 18$, $h_1(p) = 9$, $h_1(goal) = 0$, $h_2(s) = 17$, $h_2(p)  = 1$.On the other hand, $h^*(s) = 19$ and $h^*(p) = 10$.
Now, $h_1(s) \leqslant c(s,p) + h_1(p)$ and $h_1(s) \leqslant c(s, goal) + h_1(goal)$ that satisfies the consistency constraint of $h_1$. Also, $h_2(s) \leqslant h^*(s)$ and $h_2(s) \leqslant h^*(p)$ that approve $h_2$ is admissible.
However, for any node $n$, if we define $h_3(n) = \min(h_1(n), h_2(n))$:
$$h_3(s) = \min(h_1(s), h_2(s)) = 17 \not\leqslant c(s,p) + \min(h_1(p), h_2(p)) =
$$
$$c(s,p) + h_3(p) = 10 + 1 = 11. $$
It means that $h_3$ as a heuristic function is not consistent.


What is the difference between Sentiment Analysis and Emotion Recognition?

It's just as in common use. Sentiment is a superset of emotion, further including feelings, intuition, and rational conclusions drawn from past experience. Analysis is also a superset of recognition, which also includes taking emotions like contentment, happiness, humor affected, anger, rage, and passion and determining what sequences of them mean in the context of the scene or senario.


What sort of mathematical problems are there in AI that people are working on?

In artificial intelligence (sometimes called machine intelligence or computational intelligence), there are several problems that are based on mathematical topics, especially optimization, statistics, probability theory, calculus and linear algebra.
Marcus Hutter has worked on a mathematical theory for artificial general intelligence, called AIXI, which is based on several mathematical and computation science concepts, such as reinforcement learning, probability theory (e.g. Bayes theorem and related topics) measure theory, algorithmic information theory (e.g. Kolmogorov complexity), optimisation, Solomonoff induction, universal Levin search and theory of compution (e.g. universal Turing machines). His book Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, which is a highly technical and mathematical book, describes his theory of optimal Bayesian non-Markov reinforcement learning agents. Here I list other similar works.
There is also the research field called computational learning theory, which is devoted to studying the design and analysis of machine learning algorithms (from a statistical perspective, known as statistical learning theory, or algorithmic perspective, algorithmic learning theory). More precisely, the field focuses on the rigorous study and mathematical analysis of machine learning algorithms using techniques from fields such as probability theory, statistics, optimization, information theory and geometry. Several people have worked on the computational learning theory, including Michael Kearns and Vladimir Vapnik.
There is also a lot of research effort dedicated to approximations (heuristics) of combinatorial optimization and NP-complete problems, such as ant colony optimization.
There is also some work on AI-completeness, but this has not received much attention (compared to the other research areas mentioned above).


Does the policy iteration convergence hold for finite-horizon MDP?

In the discussion about Neil Slater's answer (that he, sadly, deleted) it was pointed out that the policy $\pi$ should also depend on the horizon $h$. The decision of action $a$ can be influenced by how many steps are left. So, the "policy" in that case is actually a collection of policies $\pi_h(a|s)$ indexed by $h$ - the distance to horizon.
Alternatively, one can look at it as if our state space $\mathcal{S}$ is now extended with that integer. So we can "lift" our original finite-horizon MDP $(\mathcal{S},\mathcal{A},P,R)$ into a new infinite-horizon MDP by substituting:
$$\mathcal{S} \to \left(\mathcal{S}\times\{0,1,\dots,h\}\right) \cup \{\epsilon\}\\
\mathcal{A} \to \mathcal{A},\; R \to \tilde R,\; P \to \tilde P
$$
The new reward and transition functions make sure that the horizon is decreasing, and that we are ending up in a capture state $\epsilon$ that has no future effects:
$$ \tilde P(s'_{n-1}|s_n,a) = P(s'|s,a)\quad\quad \tilde P(\epsilon|s_0,a) =\tilde P(\epsilon|\epsilon,a) = 1\\
\tilde R(s_n,a,s'_{n-1}) = R(s,a,s')\quad\quad
\tilde R(s_0,a,\epsilon) =\tilde  R(\epsilon,a,\epsilon) = 0
$$
This way I've reduced the finite-horizon MDP to an infinite-horizon one. Thus I can reuse the result for the policy iteration convergence of infinite MDPs.
Couple of notes:

At first this feels like a huge increase in the state space making the whole problem unnecessary complex. But this complexity is intrinsic to the problem: both the policy and the value function depend on the distance to horizon. So it is necessary to consider the extended number of unknowns in a single self-consistent manner.
The infinite-horizon policy iteration convergence relies on a discounting factor $\gamma < 1$. The finite-horizon doesn't need $\gamma$ for convergence. That's where I feel like I've cheated a bit.
I came up with this approach myself. It feels quite obvious though. I'd expect this approach to either be mistaken or be already mentioned somewhere in the literature - comments pointing out one or the other are welcome.


What parameters to tweak to improve reliability (machine learning)?

If your question is about supervised learning,  than ten million training points are enough in most of the cases. Besides, I think your question is too broad. There are a lot of  parameters which can influence the accuracy. Like which algorithm you want to use. Which approximator and its topology, the hyperparameters etc.


Clarifying when a loss/cost function calculation and backprop take place

Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

  Batch Size: Total number of training examples present in a single
  batch. In real life scenarios of utilising neural nets, the dataset
  needs to be as large as possible, for the network to learn better. So
  you can’t pass the entire dataset into the neural net at once (due to
  computation power limitation). So, you divide dataset into Number of
  Batches.

  Iterations: Iterations is the number of batches needed to complete one
  epoch. We can divide the dataset of 2000 examples into batches of 500
  then it will take 4 iterations to complete 1 epoch.

This is for the most part correct, except there are other reasons you would sometimes want to use batches (even if you could fit the whole thing in memory). One is that its less likely to overfit in a stochastic setting then in the full setting. Another is that it can achieve similar extrema with faster convergence.

Now regarding your question, yes you apply the gradient descent step at the end of each batch or desired batch (what I mean by desired batch, is if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24).

Though I do think its worth mentioning, your goal is still to find the global minimum. Even if your batch size is the entire dataset, that does not mean you will not fall into a local minimum, its actually in most cases still the most probable outcome


Would linked Watson supercomputers be even "smarter" than one Watson?

I guess that they would be only marginally better. And be aware that Watson itself is already a cluster of (quite big) computers (citing Wikipedia):

  Watson employs a cluster of ninety IBM Power 750 servers, each of which uses a 3.5 GHz POWER7 eight-core processor, with four threads per core. In total, the system has 2,880 POWER7 processor threads and 16 terabytes of RAM

Because of the rules of Jeopardy, Watson was not allowed to use the Web during the game. At some very high level, it contains a digested cache of some of the Web contents. Giving access to the Internet would improve slightly Watson's performance, but not that much.

  if you asked the Watson-web to diagnose a person's as-yet-undiagnosed disease, would the web be able to do so more quickly?

Even if by "web" you mean a more powerful cluster accessing the entire Internet, I don't think that it would answer much more quickly & accurately.

AI is a diversity of sub-domains, and is not advancing as dramatically as some believe. It follows a gradual progression, with in some limited fields (playing Go, or Jeopardy) some spectacular progresses. See also Everything but the essential blog entry by J.Pitrat.


What is the dimension of my output of the form (2n + 1, 2n + 1, #filters) after a MaxPooling layer

The result from applying a max pooling layer with a stride that does not exactly fit to the input will be dependent on the implementation in your library.
Assuming stride 2, and pool size (2,2), in your case the most likely things are:

The result will round up, so you will have a feature map layer with dimensions (11, 11, num_filters) although the right edge pixels will be a max over 2 pixels in the input, and the right bottom corner will just be a copy of the right bottom corner in the input (counting from top left as $(0, 0)$)

It is an error condition for your libary.

If it is not an error, then the max pooling should still perform the task it was intended to. If important features are often at the right or bottom edge, then they may generalise slightly less well, but you probably won't notice a measurable effect.
You could experiment with different sizes of pooling, different strides, or padding the previous convolutional layer so that the max pooling fits exactly. You will have to do something like this if the library has errors when the pooling does not fit exactly. You can test your experiments using cross validation, to see if there is any measurable difference.


What are examples of AI that use philosophy derived ontologies?

Excellent question!  I'm currently working on the pre-Socratics as the most basic philosophies for first principles (These philosophers are intriguing for their simplicity and universality, and the "dawn of consciousness", in some conceptions, may be ascribed to the Classical Era.  Linguistically, ancient Greek is fundamental to meaning in the West.)

However, I was pointed at this very interesting blog post "Algorithms of the Mind" which has some useful links, and discusses Kant, which you may find useful.


What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers.

Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.


When is the Markov decision process not adequate for goal-directed learning tasks?

Background
The Markov Decision Process is an extension of Andrey Markov's action sequence that visualize action-result sequence possibilities as a directed acyclic graph.  One path through the acyclic graph, if it satisfies the Markov Property is called a Markov Chain.
The Markov Property requires that the probability distribution of future states at any point within the acyclic graph be evaluated solely on the basis of the present state.
Markov Chains are thus a stochastic model theoretically representing one of the set of possible paths.  And the action-result sequence is a list of state transitions corresponding to actions chosen solely by each action's previous state and the expectations that the expected subsequent state will most probably lead to the desired outcome.
Andrey Markov based his work on Gustav Kirchhoff's work on spanning trees, which is based on Euler's initial directed graph work.
The Exercise
Exercise 3.2 was given with two parts.

Is the MDP framework adequate to usefully represent all goal-directed learning tasks?
Can you think of any clear exceptions?

The first question is subjective in that it inquires about usefulness but does not define what it means.  If "useful" means the MDP will improve the chances of achieving a goal over a random selection of action at each state, then except in no win scenarios or the most contrived case where all actions have equal distribution of probable results, then the MDP is useful.
If "useful" means optimal, then there are other approaches, with additional complexity and requiring additional computing resources that will improve odds of goal achievement.  These other approaches overcome one or more of the limitations of pure MDP.
Advancements and Alternatives
Advancements made to MDP and alternatives to MDP, which number in the hundreds, include these.

Logical detection of the infeasibility of goal achievement (no win scenario)
Calculation of probabilities when only partial information is available about the current state
Invocation of the decision at any point (continuous MDP used in real time systems)
Probabilities are not known and must be learned from past experience where simple Q-learning is employed
Past experience is used by statistically relating action-state details to generalizations derived from past action-result sequences or such information acquired or shared
The action-state decisions, made within the context of an unknown system of changing or not reliably applied rules, can be used to tune a set of fuzzy rules in a fuzzy logic container and utilize fuzzy inference in the decisions
Bluff and fraud detection

Card Games
Game play for a typical card game could make use of MDP, so MDP would be strictly useful, however not optimal.  Some of the above decisioning features would be more optimal, particularly those that deals with unknowns and employ rules, since the card game has them.
Random or Decoupled
Two obvious cases are (a) a truly randomized action-result world where goal achievement has equal probability no matter the sequence of moves or (b) a scenario where goal achievement is entirely decoupled from actions the actor can take.  In those cases, nothing will be useful with regard to the particular objective chosen.
Challenge
The way to best learn from the exercise, though, is to find a scenario where MDP would be useless and one of the above listed Advancements and Alternatives would be required rather than simply preferred.  If you look at the list, there will be some cases that will eventually come to mind.  I suggest you think it through, since the goal is to learn from the book.


Designing a reinforcement learning AI for a game of connect 4

1) your input should be so that you describe your entire environment. This could be done by 8 (length)* 8 (height)* 3 (either empty space, opponent chip or your chip) = 192 input neurons. you can just import a image of the current boardstate (which is width pixels * height pixels input neurons), but this means you task the neural network with learning to read the image as wel as playing the game.

2) rewards for a game like this are (most of the time) 1 for a winning move and 0 for a losing move. if you do it your way it might have side effects like prioritizing lining 2 chips up instead of going for a winning move.

3) Tensorflow is most commenly used. (this is python and not javascript though)

4) I liked the book Reinforcement learning, an introduction and would recommand this to anyone.


Why is image classification tasks are dominated by minimizing cost function instead of maximizing ones?

There is really no difference between minimizing a cost function and maximizing a value function.  One can be the reciprocal of the other, or the negative of the other, for example.


How should I define the reward function to solve the Wumpus game with deep Q-learning?

The reward function is up to you when you set the goals for the agent.

If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.

If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.

The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.
Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.


Are innovation weights shared in the NEAT algorithm?

Quoting Evolving neural networks through augmenting topologies, p. 10 (emphasis mine):

  When crossing over, the genes in both genomes with the same innovation numbers are lined up. These genes are called matching genes. Genes that do not match are either disjoint or excess, depending on whether they occur within or outside the range of the other parent’s innovation numbers. They represent structure that is not present in the other genome. In composing the offspring, genes are randomly chosen from either parent at matching genes, whereas all excess or disjoint genes are always included from the more fit parent.

Innovation numbers are used to line up genes in different genomes so you can perform crossover on networks with different topologies. Each network can optimise the weights in matching genes in a different way, so the weights are not shared. If they were, crossover would have nothing to contribute towards diversifying the population.


Why does the output shape of a Dense layer contain a batch size?

The Dense layers outputs 20 values per example. And since you have 10 examples in the batch the output is (10, 20) (one set of 20 values per example in the batch).
The nn doesn't learn 10 separate sets of weights. Each set of 20 values is computed with the same weight (and bias if you have any). So if say example 2 and 5 had the same input values, they'll always have the same output values.


Interpretation of inverse matrix in mean calculation in Gaussian Process

Mathematical Interpretation
Note that equation (2.23) is simply calculating the conditional distribution of equation (2.21) and then finding the mean. Your question reduces to:
"Given normal variables $X$ and $Y$, why is $\mathbb{E}[Y|X] = \mu_y + Cov(X, Y)Cov(Y, Y)^{-1}(x - \mu_x)$? (note: in the book, $\mu_x = \mu_y = 0$)
Deriving the conditional probability mean is complicated (see The Bivariate Normal Distribution, page 3). A more intuitive look can be seen in the first graph in this page.
Here, the mean of $Y|X$ is linear in what value $X$ takes. The line starts at the intercept $\mu_y$, and increases with slope $\rho (\frac{\sigma_y}{\sigma_x}) = \frac{Cov(X,Y)}{\sigma_x \sigma_y}\left(\frac{\sigma_y}{\sigma_x}\right)=\frac{Cov(X,Y)}{\sigma_x^2} = \frac{Cov(X,Y)}{Var(x)} = \frac{Cov(X,Y)}{Cov(X,X)}$. So the mathematical interpretation of $Cov(X,Y)Cov(X,X)^{-1}$ is that it is the slope of the relationship between the mean of $Y|X$ and the value of $x$ that you are given. As you are given a higher value $x$, say $x + \delta$, then the mean of $Y|X$ raises by $Cov(X,Y)Cov(X,X)^{-1}\delta$.
Why is there even a $Cov(X,Y)Cov(X,X)^{-1}$ term there? For some reason, multiplying $Y|X$ by a $Cov(X,Y)Cov(X,X)^{-1}$ term makes $Y|X$ completely independent of $X$ (which makes sense as the definition of "conditional probability", because you are already given a value of $X$). This is just a mathematical property, I don't know if there's an intuitive explanation as to why.
Human Interpretation
In case your post just want an intuition as to why there is a $Cov(X,X)^{-1}$ in the prediction of a Gaussian process (and ignoring the conditional probability fluff), I don't think there's a real basis for this, it would only be coincidental as the authors simply used the conditional probability mean formula, but I would guess $Cov(X,X)^{-1}$ somehow normalizes the values of covariance matrix $Cov(X_*, X)$.
For example, if the training set $X$ has a lot of outliers and therefore extremely high variance (e.g. all non-diagonal entries in millions), then it is very likely that $Cov(X_*, X)$ would also be extremely high as $X_*$ follows the same distribution as $X$ (unless each data in $X_*$ matches the exact same variance in $X$). It doesn't make sense to multiply $y$ by millions though, as $y$ is already a somewhat decent estimator/prior.
It makes more sense to normalize $Cov(X_*, X)$ by dividing it with the training data variance $Cov(X,X) = Var(X)$ so that the ratio $Cov(X_*, X)Cov(X, X)^{-1}$ tends to be more closer to 1 when $X_*$ follows the same distribution as $X$ (which should be the case). If the ratio is exactly 1, then $X_*$ has the exact same distribution as $X$, so you just return the prior estimate $y$. If the ratio is far away from $1$, then the test set distribution is wildly different than the training set distribution, so you return a number far away from $y$.


Can we achieve what a CNN can do with just a normal neural network?

The convolutional aspect of a CNN comes purely from the connections between layers. Instead of a fully-connected network, which can be difficult to train and tends to overfit more, the convolutional network utilizes hierarchical patterns in the data to limit the number of connections - a local edge detection feature in an image analysis network, for example, only needs input from a small number of local pixels, not the entire image. But in principle, you could assign weights to a fully-connected network to perfectly mimic a convolutional one - you just set the weights of the unneeded connections to zero. Because a general ANN has all the connections present in a CNN plus more, it can do anything a CNN can do plus more, although the training can be more difficult.


Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

The code to my question is as below, for reference:
import gym
import numpy as np
import matplotlib.pyplot as plt

# Discretize the contiuous space
DISCRETE_POINTS = 50
X_position = np.linspace(-2.4, 2.4, DISCRETE_POINTS)
Velocity = np.linspace(-5, 5, DISCRETE_POINTS)
Angle = np.linspace(-0.7295476, 0.7295476, DISCRETE_POINTS)
Angular_vel = np.linspace(-5,5,DISCRETE_POINTS)

# Fit the instantnous state into any discrete box
def get_state(_state):

    X, X_bar, Angle_, Angle_bar = _state

    X = int(np.digitize(X, X_position))
    X_bar = int(np.digitize(X_bar, Velocity))
    Angle_ = int(np.digitize(Angle_, Angle))
    Angle_bar = int(np.digitize(Angle_bar, Angular_vel))

    return (X, X_bar, Angle_, Angle_bar)

def epsilon_greedy_action(s, epsilon):

    '''   Input argument: state 's' tuple in the form (4,0,1,0)    '''
    # if np.random.uniform() < epsilon:
    if np.random.random() > epsilon:
        a = env.action_space.sample()
    else:
        _,a = find_maxQ_value(s)

    return a

def find_maxQ_value(state):
    '''
    Input argument:
    state: should be a tuple of form (0,0,0,0) or (1,0,0,1) etc.

    Output argument:
    best_value: best q-value of the current state-action pair
    choosen_action: best action corresponding to current state. It depends on the best q-value
    '''

    for act_ in range(env.action_space.n):
        A = [Q[state,0], Q[state, 1]]
        best_value = np.max(A)
        choosen_action = np.argmax(A)

    return best_value, choosen_action

def plotRunningAverage(totalrewards, N, n_avg):

    running_avg = np.empty(N)
    for t in range(N):
        running_avg[t] = np.mean(totalrewards[max(0, t-N):(t+1)])
    return running_avg

if __name__ == '__main__':

    env = gym.make('CartPole-v0');

    EPISODES = 1000;
    no_actions = env.action_space.n

    # Hyper parameters
    alpha = 0.001 # Learning rate
    gamma = 0.99 #Discount Factor
    epsilon = 1 # For Epsilon-Greedy algorithm
    epsilon_decay_factor = 0.99;
    min_epsilon = 0.1;

    states = []
    for i in range(len(X_position)):
        for j in range(len(Velocity)):
            for k in range(len(Angle)):
                for l in range(len(Angular_vel)):
                    states.append((i,j,k,l))

    #Initialize Q-table :
    # 1. We make the Q-table in form of a dictionary
    # 2. We initialize Q-table values as zero in this
    Q = {}
    for s in states:
        for n_a in range(no_actions):
            Q[s, n_a] = 0

    Running_reward = [];
    l_action_cnt = 0;
    r_action_cnt = 0;
    wrong_action =0;

    #Q-Learning agent episodes
    for e in range(EPISODES):

        cn_state = env.reset()
        ds_state = get_state(cn_state)

        done = False
        ep_reward = 0
        ep_len = 0

        while not done:

            action = epsilon_greedy_action(ds_state, epsilon)
            if action == 0:
                l_action_cnt+=1
            elif action == 1:
                r_action_cnt+=1;
            else:
                wrong_action+=1;

            cn_next_state, reward , done , ep_len = env.step(action)
            ep_reward += reward

            ds_next_state = get_state(cn_next_state)

            # Update the Q-table based on the action
            Val_Q_bar, _ = find_maxQ_value(ds_next_state);

            Q[ds_state, action] = (1-alpha)*Q[ds_state, action] + alpha*(reward + gamma*Val_Q_bar)

            ds_state = ds_next_state;

        Running_reward.append(ep_reward)

        if e%100 == 0:
            print('Episode : {}, Episode reward: {}, Epsilon: {}'.format(e, ep_reward, epsilon))

        if epsilon >= min_epsilon:
            epsilon*=epsilon_decay_factor;

plt.plot(Running_reward);
plt.xlabel('episodes')
plt.ylabel('episodic reward')
plt.grid('ON')
running_avg = plotRunningAverage(Running_reward, EPISODES, 50)
plt.plot(running_avg);
plt.legend(['Episodic rewards', '50-Episode moving-average reward'])

print('The ratio of left to right action is : {}'.format(l_action_cnt/r_action_cnt))


Is a linear activation function (in the output layer) equivalent to an identity function?

And following this point, aren't you just as well off with just using the identity function as your output activation function?

When someone declares that the output of a neural network layer is linear this is exactly what they mean. It can also be described as "no activation function".

Saying that a NN layer has linear activation is a kind of short-hand for saying "the whole layer is a linear function of its inputs", which is true without adding any activation function, just using the weights and bias.

There is usually no separate linear function applied, and libraries such as Keras include the term 'linear' only for completeness, or so that the choice can be made explicit in the code, as opposed to an unseen default.

Note that the link to Keras activation definition above says:

  Linear (i.e. identity) activation function.