When the AI singularity takes over, what will there be left for us to do?

Given all your assumptions about AI turn out to be true, we would have some kind of utopia, where no one has to work, and there is plenty of everything. Fair enough. Your other assumptions is about human nature, and that is where I'd challenge your conclusion: Just because there are computers better than humans at some task, that does not automatically take all enjoyment from doing it.

I have three arguments in favor of my stance.

There already are computers better than humans at checkers, chess, backgammon, starcraft, mastermind, go,.. and many more. Yet these games still get played, even if no human can hope to ever be as good as a computer.
There will be domains where the evaluation of quality is so vague or personal, that the notion of being "better" is useless in any objective sense. I am thinking mainly about art. Playing into argument one, photo cameras are already better "painters" of reality than humans can ever hope to be, yet people still paint. And their paintings gets appreciated, even the photo-realistic kind.
I'd say that the whole mindset of "if someone is better than me at x it is not worth doing x" will have outlived its lifetime very shortly. Society didn't always spin that way I feel, it is largely due to the influence of the North American way of life to always strive to be #1 and everything below that being trash. Globalization already puts this way of thinking at risk, with many young people being disillusioned or even depressed because, flippantly put, "whatever you do, there is always an Asian kid who does it 10 times better". We don't have to wait for AI to outshine us, the rest of the world already does. As a consequence, we need to adapt our way of approaching that fact, stop seeing it as diminishing to our worth, and move on.

As a closing note, I also see it as a problem to the general population that within a short time we essentially have to change a very substantial part of our world view, AI being better than us, us not having to work anymore etc. All the economical revolutions in the past, the neolithic, the industrial, and most recently, the digital, had a longer transitional period where people could grow accustomed to the new world. And even with that transition it was hard enough for many people. Yet, most dealt with it, and later generations can hardly imagine a world where the new change doesn't exist yet, and I personally don't see why the next revolution should be any different.


How to build a commercial image captioning system?

Thinking Commercially

A commercial solution will need to be able to ascertain, continuously verify, and utilize the best of options for captioning learning models.

Each fairly successful image captioning learning models could be placed in an adapter to provide a common training, optimizing, testing, evaluation, and usage interface.  The hot swappable container-addon mega pattern used in peripheral device installation, RAID, J2EE containers, browsers, and other containment sub-systems can be applied.

System Description

The system acceptance criteria being as follows.

A new model could be added without stopping or starting the system if tauted as successful by others
Any model can be deleted without starting or stopping the system if it has not been performing well
Each possible state of each model added can be given some percentage of the total system resources
Various training processes and hyper parameter tuning processes can be applied to any of those model-state combinations
A-B testing can be done on any model-state combination
Stats can be displayed on any model-state combination any model overall or any state across all added models
The interface for each model requires that its associated model produces an array of caption-reliability pairs with zero ore more pairs
The reliability measure that is included in the pair with the caption is an indication of the model's assessment of appropriateness to the image
The most appropriate caption is picked from the proposals of the various models

States can be idle or any of these.

Training
Testing
Evaluation
Hyper parameter tuning
In use

For example, for the two learning models usable in captioning systems suggested in this question, NIC and Neural Talk 2 we could have a system resource allocation like this:

30% NIC Training
5% NIC Hyper parameter tuning
45% NT2 In use
15% NT2 Evaluation
5% NT2 Hyper parameter tuning

Samples may be pulled from a pool of samples that have been vetted.  That pool may be augmented by real images passing through the system, filtered in accordance with security criteria to avoid external attempts at control.

In the assignment of resources, the sample pool selection criteria must be specified.  If the system is already at 100%, the model-state combinations from which the resources shall be drawn must also be specified.

Handling Multiple Output Options

Since there may be more than one model in use and each model may have zero, one, or multiple caption suggestions for each image, each with a reliability measure, the outputs must be analyzed to provide the best choice to associate with the image being analyzed.  Additional system criteria must cover this process scenario.  For any given image, final evaluation must follow the following general guidelines.

If multiple models produce similar or exactly the same captions, they most be weighted higher in final evaluation.
If the model's reliability is proven in actual use based on feedback from end users, the model's output must be weighed higher accordingly
Re-entrant models (such as reinforcement learning network models) must have access to the end user feedback for additional learning even in in-use state
Clear winners are chosen
Close races are disambiguated through trained functionality
Exact ties are broken through pseudo random index generation

Another artificial network may be placed at the output and appropriate encoding and normalization may be applied before training so that a properly trained network, converged using a quantification of the above additional criteria, can select the best caption from the options for each image.

Phased Development Approach

Phase one of such a system would likely require manual handling of model-state allocation.  Phase two would be semi-automation.  The location of new models would still require expert attention.  Perhaps further in the future a hunt for new models could be automated too.


Does replacing 3x3 filters with 3x1 and 1x3 filters improve the performance?

If the filter is separable, that is, the NxM kernel can be mathematically equal to the convolution of a Nx1 filter and a 1xM filter, there are a very important increase in performance.
Using separable convolution, the network can made an optimal usage of the shared memory and of the parallelism in memory access. See this excellent article for details. These improvements are bigger for bigger kernels.
Also the training is improved, starting by the simple fact that a NxM filter has a number of parameters proportional to N*M but the related separable one has N+M.


When we create artificial life and artificial intelligence will we require it to obey human laws?

For those times when AI does interact with humans, I believe that AI would be held at LEAST to the same standards humans are. The problem comes in when we ask "who is really to blame". If a self-driving car cuts you off in traffic and causes you to wreck, you can't take the AI in the car to court. Do you take the company? The programmer? The owner of the car? Some entity will likely be held responsible, the question is just which one.

As for future human-like AI, I believe my answer still remains true. Having a human level AI changes the meaning of the word "entity". If a human-like AI breaks a law, it may be because it was programmed to do so. I don't think our current legal system is ready for such cases, but it have to evolve in the future.


When should I use feature learning as opposed to feature engineering?

manual feature engineering started becoming obsolete

That is wrong.

  Any suggestion on when to use manual feature engineering, feature learning or a combination of the two?

Deep learning is awesome for natural signals like images, audio or large amounts of unstructured text (e.g. arbitrary crawled websites)

There are some basic steps that make almost always sense. For example, for image recognition, it is the normalization from pixel color features in the range $\{0, 1, \dots, 255 \}$ to features in $[0, 1]$.

And then there are a lot of use cases where you have structured, CSV-like data. For those, every single case I had so far was improved by feature engineering. Most of the time you actually don't have a choice here as well: how would you feed in a date time object into a neural network?

Now, where does feature engineering make sense in image recognition?

Resource limitations: neural networks often take a lot of resources. For example, viola-jones face detection is very resource friendly in contrast. But there are developments like the mobile net to relax that issue.
Prior knowledge: for more special image data like CT scans you might observe super simple operations that almost give you the target. This might make it simpler to interpret the results at the end and, again, it helps to keep resource usage low.

One other field where I suspect that manual feature engineering could help is adversarial attacks. But for this I'm not 100% sure.


Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that

  Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.

I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.

Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.


What needs to be done to make a fair algorithm?

The paper Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges argues that ensuring fairness is not a trivial task and that the current statistical formalizations of fairness lead to a long list of criteria that are each flawed (or even harmful) in different contexts, that is, there are trade-offs between the proposed formalizations. Therefore, fairness constraints in algorithms have to be specific to the domains to which the algorithms are applied. To achieve that, there is a need for collaboration with domain experts and explainable artificial intelligence.

The major obstacle towards fair machine learning algorithms is the presence of algorithmic bias, which can be subdivided into the following main categories:

the bias in the data, and
the inductive bias (the implicit or explicit assumptions behind the algorithm or model) or, in general, any bias introduced during the development of the algorithm or model (for example, a certain choice of a subset of features can change the outcome of the model).

The bias in the data can be due to different factors, such as a biased choice of the collected (or labelled) data or measurement errors (which can make the data not representative of the population). Causal inference can be used to understand the causal relationships in the data, thus it can be used to find the source of bias in the data. To avoid unfairness due to bias in the data, there is a need to analyze and understand the data, so that to improve its quality (for example, by increasing the diversity of the data). However, the bias in the data is not always easily reducible, given that certain outcomes of an experiment may rarely occur or are hard to produce in practice, so unbiased data may not always be easily collectible.

In the paper The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables (2017), the authors address the selective labels problem (that is, the judgments of decision-makers determine which instances are labeled in the data, which can thus introduce bias in the data) and develop an approach called contraction, which can be used to compare the performance of predictive models and human decision-makers (even in the presence of unobservables). There are also works that are based on Bayesian or causal inference (for example, risk-adjusted regression)

The sample bias (the data is not representative of the overall population due to a systematic intentional or unintentional error in data collection, a measurement error, which can also be due to social prejudices) is also a form of bias in the data. In the paper Residual unfairness in fair machine learning from prejudiced data (2018), the authors address this problem in the context of and of police stop-and-frisk (where a biased police behavior leads to over-proportional stopping of a racial minority group). Nathan Kallus and Angela Zhou show that adjusting the classifier for fairness does not solve the sample bias problem.

In the paper Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges, the author argues that any attempt to reduce the bias introduced during the development of the algorithms or models, if it does not take into account the specific social and moral context where they are supposed to be applied, can still lead to algorithmic bias. Furthermore, algorithms should undergo frequent reevaluations, given that, for example, the underlying population or the application context may change.


Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?

Setting too high a learning rate will extend the time to get a good result.

In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.


How can I model this problem of delivering assets by choosing a route with reinforcement learning?

I do not see how you came to choose prices as actions. Normally, actions are something like go left, go right, jump, stay etc.
Analogously, I would say that in your case the actions are visiting a certain location, whereas locations are what you referred to as states. I'd go for something like that:
locations = {location1=(c1,cr1), location2=(c1,cr2), ...}
Actions = {Visit location1, Visit location2, ...}
States = {--set of all the possible paths the model can generate until the budget is possibly exhausted--}

The reward function could then be a combination of both the acceptance (vs. rejection) of a route/path by the user and the inverse of the cost associated with the suggested path (because you want the model to favor cheap paths in order to keep your own business costs low).
How you balance these two terms is up to exploration.
For rapid prototyping, check out stable baselines, which offers a bunch of highly optimized RL algorithms.


Time leap (To achieve advance technology) using AI by simulating virtual world at a faster rate

This is a fun and instructive question, so I'm going to attempt an adequate answer.

Q: How many decades are we far from achieving a virtual world just like our real world?

Nobody knows, although it's popular to use 2050, a number they came up with, I believe, based on some form of Moore's Law.  But they're just guessing, and nobody knows if they type of singularity proposed is even possible.  We still don't understand how the human brain works, or if strong Artificial General Intelligence is even possible.  (I personally think it definitely is, but am skeptical of the optimistic predictions in terms of when.  There is a wide spectrum of opinions on the matter.)

Time Leap

It seems to me that in modeling, for instance, the entire planet in this manner, the simulation is going to use all kinds of abstraction (as opposed to modeling every atom and sub-atomic state.) For that reason, I'd think any simulation would only approximate reality, and not be 100% accurate predictively. My guess is you'd have to run a very high number of such simulations, then statistically determine which outcome would be most likely based on a set of outcomes, but the more you narrowed that to a single result, the lower the accuracy.

I've been thinking about Conway's Game of Life in relation to Laplace's Demon lately. Perhaps Game of Life would provide some insight. I think part of the problem there is that there is still a hard limit to the speed of information in the system (which can be analogous to the hard limit of c in the "real" world.)  Game of Life is a deterministic with a single node for each ply of the (predictive) game tree. Predicting the outcome of a novel, non-trivial starting configuration is indistinguishable from running the simulation.  But the only entity that has perfect information in regards to the entire system is the system itself. It's possible that an intelligent construct within Game of Life could model outcomes of local phenomena, but perfect information on the entire system is an impossibility, and as with quantum phenomena, trading of information actually affects the system. (See glider guns.)

@k.c. posted a very useful link on the Limits of Computation, which is probably a much less convoluted approach to the question;)

But the problem you propose is deeper because we still don't even know if there is true randomness is nature (quantum phenomenon) or if indeterminacy, and the probabilities that arise out of it, are merely functions of the limits of observations, which is well defined in the mathematics, but also has a philosophical component.

Quantum phenomena don't directly affect the macro world, but can bleed into it in areas such as formation of crystalline structures.

So until we can solve the randomness question, and find some way around the observational problems at the quantum level, I don't think we could model the type of systems you're asking about with perfect accuracy.

Further Reading:

I strongly advise you take a look at the Quantum Thief trilogy by Hannu Rajaneimi.  I always recommend these books for interesting questions such as this, not simply because Rananeimi deeply explores the type of post-Singularity world you're thinking about, but because he's a mathematical physicist, and his grasp of the mathematics involved, including quantum theory, is far superior to mine and almost certainly other authors of speculative fiction, whether literary or under the moniker of "futurists".

His take is that even with all of the computing power you envision, at a level of technology where matter and information are interchangeable, perfectly accurate prediction of the future is not possible.

But it is this next part you may find interesting:

Rajaneimi inverts your concept of the "Time Leap" by instead talking about the "Deep Time"

The Deep Time refers to what a consciousness can experience at peak processor speeds.  For instance, thousands years can be experienced in mere moments of organic human time.

I'm not going to give a spoiler on what happens at the end of the trilogy, but it involves the topology of black holes.


In the graph search version of A*, can I stop the search the first time I encounter the goal node?

No, the optimal path is found when you pop the goal state. If you stop the search when you first add the goal state then the final path may not be optimal.


When does AlphaZero play suboptimal moves?

During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node of the MCTS search tree). Instead, for the purpose of generating a more diverse set of experience, it samples moves proportionally to the visit counts. This means that in any given situation encountered during self-play, the move considered to be "optimal" will still have the largest probability of getting picked, but other moves will also have smaller probabilities of getting picked. In theory, it might even sometimes pick the move that it expects to be the worst one (very rarely)!

If I recall correctly, they only did what I describe above for the first 30 moves of any game, and afterwards move on to greedy play. This still results in a very diverse set of 30-move-starts for every game it experiences though. I'm not 100% sure if I remember this detail correctly though, maybe they only did this earlier (in AlphaGo Zero for example), and no longer do it in AlphaZero. Would have to check the paper to make sure.

Additionally, whenever they start a new search process, they perturb the prior probabilities assigned by the learned policy network to all the moves available in the root node. This is done in a non-deterministic way using Dirichlet noise. I think this is not really explicitly mentioned in the AlphaZero paper, but it is in the Supplementary Materials (and also in the AlphaGo Zero paper?). Anyway, this also means that if precisely the same game state is encountered twice in two different games of self-play, the search behaviour may be slightly different due to the introduced stochasticity, and hence it may come to a different conclusion as to what move is "optimal".


How to handle varying types and length of inputs in a neural network?

In short:
1/2 ->  ANNs don't have problems with "different types" of data as long as they are represented using real numbers: the inputs for your ANN represent lengths and are easy to understand and process.
2/2 ->  The variable number of inputs is a little bit more tricky. In general it is not a problem either. The net will compensate the absence of some inputs from time to time (when food is too far away?). You don't necessarily have to use RNNs for your case.

In full:
I think that the inputs of your net could be improved.

When building a new ANN you always have to consider few aspects:

Neural Nets are just stateless functions that map inputs to outputs.
There is no magic here: with good inputs you can get good outputs.
Think about how you would write an algorithm to solve that problem. Which would the best set of inputs be? Can some computation be done somewhere else? (using other ANNs, if you want to create ANN chains, or traditional algorithms).
Preprocessing can do a big difference!
Think about the space separation that your inputs require/cause.

All these points are related one to each other and they're only about the inputs you're providing to your net.

Now, keeping these points in mind, you can check again your inputs to see if you can improve something:

Objective 1: eat food
To be sure the snake will get toward the food we definitely need to pass the ANN the position of the food. But you're sure that a long list of coordinates is the best option? Maybe the coordinate of the nearest food can be enough.
If you were THAT snake what would you  like to receive? Or what would you receive if you were a real snake? Imagine your snake has a sensor that determines the best place to get food. It'll be like a compass, outputting only one value, from 0 to 365 (or choose whatever range you like). This compass will point at the closest bigger cluster of food or at the closest food.
Remember: 1 value is always better than N and you should try to avoid a variable number of inputs, it's always a smell that something doesn't go well.

Objective 2: don't die
The distance from the centre works well. It's a simple threshold that the ANN will consider when moving, that's effective and easy to achieve. Why don't you also add another compass that will always tell you the position of the centre instead of the coordinates of the snake? In this way you've used 2 values related to each other instead of 3, more difficult to decipher for an ANN.

The time is not really necessary. You're inserting a value that isn't related to any other, or the environment, or the food. Any possible behaviour is not influenced by its value (if the ANN knew that it has only 15 seconds left, it wouldn't change its behaviour. The same if it knew that it's "alive" since 1 minute or 4000).


Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\lambda}$ returns for the critic (value function estimator), but also for the actor's policy gradients.

Note that you do not explicitly see the $G_t^{\lambda}$ returns mentioned in the pseudocode. They are being used implicitly through eligibility traces, which allow for an efficient online implementation (the "backward view").

I do indeed have the impression that such uses are fairly rare in recent research though. I haven't personally played around with policy gradient methods to tell from personal experience why that would be. My guess would be that it is because policy gradient methods are almost always combined with Deep Neural Networks, and variance is already a big enough problem in training these things without starting to involve long-trajectory returns.

If you use large $\lambda$ with $\lambda$-returns, you get low bias, but high variance. For $\lambda = 1$, you basically get REINFORCE again, which isn't really used much in practice, and has very high variance. For $\lambda = 0$, you just get one-step returns again. Higher values for $\lambda$ (such as $\lambda = 0.8$) tend to work very well in my experience with tabular methods or linear function approximation, but I suspect the variance may simply be too much when using DNNs.

Note that it is quite popular to use $n$-step returns with a fixed, generally fairly small, $n$ in Deep RL approaches. For instance, I believe the original A3C paper used $5$-step returns, and Rainbow uses $3$-step returns. These often work better in practice than $1$-step returns, but still have reasonably low variance due to using small $n$.


Why can't neural networks learn functions outside of the specified domains?

The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way extends to us as well. The model is our brain, the data is our experience and sensory input, and the learning procedure is there but unknown (for now $<$insert evil laugh$>$).

Your model's inability to understand that it is a sin function is normally by construction. First, though lets start with the fact that if someone showed me $sin(x)$ on $[-1,1]$, a sinusoid is the last thing id think of:

 It looks almost linear. So for sake of argument i'm going to continue on the assumption you meant an entire period ($[-\pi, \pi])$.

Now, given this most people who ever got past pre-calculus would assume sinusoid, as youd expect, but let me show you how unstable that idea is. Lets use a full period, shifted by $\frac{\pi}{2}$

Now i don't know about you but my spidey-sense would make me think of a Gaussian before anything. This is the same sinusoid along an entire period but just the slight difference changes an entire perspective.

Now lets talk about some of the mathematics. Given the uncountable $(x,sin(x))$ pairs between $[-1, 1]$, we can extract the function $sin(x)$ with certainty (assuming infintely differentiable) by simply taking the taylor series along any of the points in that domain. More so, we only need an infinitiscemally small continuous subset of the function to achieve that result, but as a person, would you be able to tell from $[-.0001, .0001]$? (i don't think so!), because by default we quantize, and what were gonna give to the computer is a quantization as well. We are giving it a finite countable set of points and expect it to to generalize to an uncountable set. This is silly, since there exists an infinite set of functions that can fit those exact points (concept of overfitting steps in here).

So if its so unfeasible for a computer to correctly solve the function, why can us humans extrapolate it so well from the $[-\pi, \pi]$ domain? The answer is our preset biases. Growing up we have dealt so much with periodicity, specifically sins and cosines, that graphed period triggered something in our brains and we just know what it is. But this isnt always a benefit, i could draw you a low-ordered polynomial on that domain that would make you think that exact same thought making you wrong.

So going back to my initial point, the problems lies with the models creator and not the model. Like i explained just now, its unfeasible without preset biases to learn the function we want, so what should we do? give up? NO, its our goal as the model architect to figure out what we want and how we can best achieve it! so if we want it to accomplish this task, lets try our best in modeling these biases. A quick example of how you could do that, is to force the function to fit the fourier coefficients, thatll probably solve this quite quickly (ofcourse this limits this models ability to solve for other common functions)

Takeaway: It's not the model thats struggling, its us using preset biases that the model does not have access to. The solution to this is be clever and figure out how to have it either start with these preset biases or learn them.


How does the NEAT speciation algorithm work?

The first equation deals with distance. Delta, or distance, is the measure of how compatible two genomes are with each other. c1, c2 and c3 are parameters you set to dictate the importance of E,  D and W. Note that if you change cc1, c2 or c3, you will most likely also have to change dt, which is the distance threshold, or the maximum distance apart 2 genomes can be before they are separated into different species. E represents the total number of excess genes. D represents the total number of disjoint genes in both genomes, W represents the total weight difference between genes that match, and finally, N represents the number of connections / genes the genome with the larger number of connections has. For example, take the following 2 genomes:

[[1,.25][2,.55],[4,.78],[6,.2]]
and
[[1,.15][3,.92],[5,.37]]

Where the 0 index represents innovation number and the 1 index represents weight value. E  would be 1, since there is 1 excess gene, gene 6. D would be 4, since connections 2 and 4 are not in genome 2, and connections 3 and 5 are not in genome 1. W would be .10, since only connection 1 is shared between the two genomes.

The second formula is a bit more complicated. From my understanding, correct me if I'm wrong, this is a formula for adjusting fitness. f′i is the adjusted fitness, which will replace the original fitness, fi. For every genome j in the entire population, yes, entire population and not just every genome in its specie, it will calculate the distance between j and i, i being the genome of fitness fi. Then it will sum up all the distance    values, and divide the original fitness fi by the total distance sum, and set f'i to that. Next,

  Every species is
  assigned a potentially different number of offspring in proportion to the sum of adjusted fitnesses f`i of its member organisms.

This assigning of number of species offspring is used so that one specie can't take over the entire population, which is the whole point of speciation in the first place, so in conclusion, these two formulas are vital to the function and efficiency of the NEAT algorithm.


Can we apply transfer learning between any two different CNN architectures?

No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.
In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that this Google search seems to yield some results.
You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.


There are a number of open source implementations of the C4.5 algorithm invented by Ross Quinlan:

Java: https://github.com/scottjulian/C4.5
Python: https://github.com/dpkravi/DecisionTreeClassifier
C++: https://github.com/zgyao/DecisionTree_C4.5
JavaScript: https://github.com/miguelmota/C4.5


Darknet is "native" framework, so basically, you don't need to implement anything, all code for yolov3 is available at their github repo, you just need to figure it out, play with it. Keras, in my opinion, is not flexible enough to easily implement yolo. If you want to implement yolo from scratch I would probably go with PyTorch it has a dynamic graph + more integration with python. Regarding speed, it's hard to say objectively which framework is faster, especially comparing with darknet, since probably nobody did a comparative analysis. The easiest solution to speed up yolo3 is obviously buying more powerful GPU :D.

Also, in my understanding what they did in yolov3 is that they intentionally sacrificed speed in order to be able to detect smaller objects, so if you don't care too much about small grouped up objects go with yolov2 it is very fast and has a pretty decent mAP.


In the context of importance sampling ratio, how is the equation $\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$ derived?

It has nothing to do with Bayes rule, like you said, $G_t$ is the return we get by following the policy $b$ , so the value of the state we get that return in is equal to the expected value of that return. On page 59 of the book, equation 3.14 derives the value of the state as expected value of the return but in this case we are following the behaviour policy $b$ so you would replace $\pi (a \mid s)$ with $b (a \mid s)$ and $v_\pi (s')$ with $v_b (s')$ and you would get the value of the state following the policy $b$, that is $v_b (s)$.
We are actually interested in the value of the state when we follow target policy $\pi$ , but we got the value by following behaviour policy, so we will multiply that expression with importance sampling ratio. If we do that $b (a \mid s)$ terms will cancel out and you are only left with term $\pi (a \mid s)$ from importance sampling ratio so we will get the value by following the target policy $\pi$ that is $v_\pi (s)$


What is the goal of weight initialization in neural networks?

Is it trying to make sure there is no symmetry in the gradients?

The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry.

Is it trying to allow the gradients to be large so it can quickly converge?

The time it takes to converge, is I think a property of the optimizer and not of the weights initialization. Of course, the manner in which we initialize our weights matters but I think Optimization Algorithms contribute more towards convergence

What are the properties in an initialization that makes the network learn faster?

Glorot and Bengio believed that Xavier weight initialization would maintain the variance of activations and back-propagated gradients all the way up or down the layers of a network. Incidentally, when they trained deeper networks that used ReLUs, it was found that a 30-layer CNN using Xavier initialization stalled completely and didn’t learn at all. Thus, it depends on the particular problem at hand.


Should intelligent AI be granted the same rights as humans?

I'll attempt to analyze a couple of different perspectives.
1. It is artificial
Synonyms: insincere, feigned, false.
There is the idea that any "intelligence" created by humanity is not actually intelligent and, by definition, it is not possible. If you look at the structure of the human brain and compare it to anything humans have created thus far, none of the computers come close to the power of the brain. Sure they can hold data, or recognize images, but they cannot do everything the human brain can do as fast as the brain can do it with as little space as the brain occupies.
Hypothetically if a computer could do that, how do we determine its intelligence? The word artificial defines that the intelligence is not sincere or real. This means that even if humanity creates something that appears intelligent, it has simply become more complex. It is a better fake, but it is still fake. Any money not printed by the government is by definition counterfeit. Even if someone finds a way to make an exact duplicate, that doesn't mean that the money is legal tender.
2. Misuse of power
If an AI is given rights and chooses to exercise those rights in a way that agrees with its creator's views, possibly through loyalty to its creator, or through hidden motives, then anyone with the capabilities to create such an AI would become extremely powerful by advancing their own beliefs through the creation of more AIs. This might also lead to the ruthless parallelization that you mentioned, but with (even more) selfish goals in mind.
If this were not the case, and an AI could be created to be neutral with free will and uncontrollable by humans, then perhaps an AI could be given rights. But I do not believe this would ever be the case. With great power comes great responsibility. Even with free will, a true AI would most likely end up serving humanity, because humans have control of the plugs and the electricity, the Internet, the software, and the hardware. The social implications of this for the AI are not promising. It's not even just the ongoing control of these resources that is the issue. Whoever creates the software and hardware for the AI would have special knowledge. If fine adjustments were made, specific individuals would undoubtedly hold sole control of the AI, as adjustments could be made to the code in such a way that the AI behaves the same except under specific circumstances, and then when something goes wrong (assuming the AI has its own rights), then the AI would be blamed rather than the programmers who were responsible.
3. Anthropocentrism
In order for humanity to get away from anthropocentrism, we would have to become less selfish when it comes to humans, first. Until we can solve every existing social problem within humanity, there is no reason to believe that we could cease thinking of humanity as more important than created machines. After all, supposing there were an almighty God that created humanity, wouldn't the humans always be beneath God, never to be equals? We can't fully understand our own biology. If an AI were created, would it be able to understand its makings in the same way its creators would? Being the creator would give humanity a sense of megalomania. I do not think that we would relinquish our dominion over our own technological creations. That is as unlikely to happen as the wealthiest of humanity willingly giving the entirety of their money, power, and assets to the poorest of humanity. Greed prevents it.
4. Post-human fundamentalism
Humans worship technology with their attention, their time, and their culture. Some movies show technologically advanced robots suppressing mankind to the point of near-extinction. If this were the case and humanity were in danger of being surpassed by its technology, humanity would not stand idly and watch its extinction at the hands of its creation. Though people may believe superior technology could be created, in the event we reached such a point humanity would fight to prove the opposite, as our survival instincts would take over.
5. A balance?
Personally, I do not think the technology itself it actually possible, though people may be deceived into thinking such an accomplishment has been achieved. If the technology were completed, I still think that anthropocentrism will always lead, because if humanity is the creator, humanity will do its best to ensure it retains control of all technological resources, not simply due to fear of being made obsolete, but also because absolute power corrupts absolutely. Humanity does not have a good historical record when it comes to morality. There is always a poor class of people. If wealth were distributed equally, some people would become lazy. There is always injustice somewhere in the world, and until we can fix it (I think we cannot), then we will never be able to handle the creation of true AI. I hope and think that it will never be created.


How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.

There are 2 solutions to this problem:
The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and here is a paper which describes this approach with 3 actor-critic agents working together to move a robotic arm.

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.

This is a more general answer but should help you and others on their related problems.


How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you are increasing the number of discernible areas. As you increase the number of lines you can use to break up the input space, the lines can be placed to approximate any curve (sinusoidal) As the number of neurons approaches infinity, the accuracy of categorizing different inputs across this curve increases.

Interestingly enough, if one graphed "Number of Neurons (x) vs Accuracy (y)", it would look sinusoidal.


How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.?

There are some works that do this comparison. Briefly, it's been observed that the performance of models trained via FL drops as data distributions between participating agents differ. When data is IID-like though, performance is comparable to centralized training. Some works that I'm aware of are as follows:

Overcoming Forgetting in Federated Learning on Non-IID Data
Improving Accuracy of Federated Learning in Non-IID Settings
Federated Learning with Non-IID Data

There are probably many more around. It's an active area of research.


What Constitutes Messages in Junction Tree Algorithm?

I think an example could make you understand better.
Suppose you want to calculate $P(X|S)$, you now need to put evidence on $S$(so you have to change all tables where $S$ appears making probability zero where $S$ does not appear).
At this point, you can proceed with the collect and distribute method in order to propagate the evidence throughout the graph, in order to maintain the global consistency property. Now you can choose any $cluster\space V$ that contains $X$, marginalize with respect to $S$, and the job is done.


Is it illegal to use pictures of public figures to train a neural network?

If you use any pictures you find online, you can use them as you wish: as long as you don't (re)publish them under your name. Also, if you really want to play safe, never upload them at all. Download them, use them, disregard them.

People can't proof you used certain pictures to train your network by looking at your network data.

And the 'big' companies out there, like Google, train their networks by using public photos.

And follow this chart:

As you are using it for a personal/educational project, it shouldn't be a problem.


How could we define passion in artificial intelligence?

An elementary approach to 'passion' would be to pre-assign different areas for the program to be 'passionate' about and associate different numeric 'drive strengths' with each (perhaps adaptively). Mechanisms of this sort were studied in Toby Tyrell's widely cited PhD thesis on 'Action Selection in Animals'

More recently, some more sophisticated AI architectures have been developed under the heading of 'Intrinsic Motivation'.

Here  is a link to a paper on the subject by Pierre-Yves Oudeyer, a leading expert in the field of Developmental Robotics.

With regard to the question "would this cause the machine to do a better job?", that would very much depend on how open-ended the architecture is:

It's clearly easier if, rather than having to spell everything out in detail to a machine, we can simply specify a problem at a high-level and let its own motivations cause it to explore promising avenues.

Conversely, if motivations are too open ended, it may well spend all its time doing the equivalent of 'doodling on its paper' (Hofstadter).

Hence, like people, the quality of the output will be a function of its internal dispositions and could be measured in the same way for a given task (e.g. quantitatively for scientific activities, qualatatively for the arts).


How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update?

The Bellman equation in RL is usually defined $$v_\pi(s) = \sum_a \pi(a|s) \sum_{s', r} p(s', r|s, a)\left[r + v_\pi(s')\right] = \mathbb{E}_{s' \sim p, a \sim \pi}\left[r(s, a) + v_\pi(s')\right] \; .$$
The way you have written it is correct, but I just thought I would point this out. Regardless, your intuition is correct in that it expresses a recursive relationship such that the value of your current state $s$ is equal to the sum of the expected reward from this state plus the expected value of the state you transition into.
You do, in fact, implement the Q-learning update in Deep Q-Learning. The loss function that you minimise in DQN is
$$ L(\theta) = \mathbb{E}_{(s,a,r,s')\sim U(D)}\left[\left( r + \gamma \max_{a'}Q(s', a'; \theta^-) - Q(s, a; \theta)\right)^2 \right]\;$$
where $U(D)$ denotes uniformly at random from replay buffer $D$ and $\theta$ are your network parameters (the network parameterises the Q-function), and $\theta^-$ are a previous iteration of the parameters that are updated every $c$ episodes to help with convergence of the network.
As you can see, the loss function is minimising the 'Bellman error' error from your equation 2. Lets think about why this is.
The TD update you provide is gradually shifting the Q value for $(s, a)$ towards $r + \max_a Q(s', a)$ - this is what we want after all since it eventually converges to the optimal Q-function.
Now lets think about the Deep Q-learning case. We want our network to approximate $Q(s, a)$ and so if we train the network, using the MSE loss, with $r + \max_a Q(s', a)$ as our target then our network will gradually be shifted towards predicting $r + \max_aQ(s', a)$ (which again would give us optimal Q-values for state-action pairs), just like with the TD update.
This is assuming that you know how training of neural networks works so if you don't then I would recommend asking/searching for a relevant question that explains this.


What information should be cached in experience replay for actor-critic?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $Q(s)$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.

Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).


What kind of education is required for researchers in AI?

As a full-time AI researcher myself, I'd say that a PhD in machine learning would certainly be one useful option.

However, in order make much-needed progress, AI needs avoid falling into the trap of thinking that currently fashionable methods are any kind of 'silver bullet'. There's some danger that a PhD that heads straight into (say) some sub-sub-sub area of DL would end up imposing too much bias on the student's subsequent perspective.

AI research is an essentially multi-disciplinary activity. Other possible backgrounds therefore include:

Mathematics or physics (to first degree or PhD level). A strong background in either of these never did anyone any harm. People who are competent in these fields tend to be able to turn their abilities to new domains relatively easily.
Software Engineering. One of the things that AI needs are integrative architectures for knowledge engineering. Here's why. I believe that one of the reasons that we haven't yet managed to do OCR at the level of a 5 year old is that we've yet to accept that we have to 'build a sledgehammer to crack a nut'. Software architects are used to managing large-scale complexity, so they may be able to help.
Cognitive Science, Psychology, Cognitive Linguistics. The reasons here are obvious.

Above all, I personally think that a good AI researcher should be creative, inquisitive and prepared to question received wisdom, all of which are more important in practice than specifics of their background.


In theory, if one could build a computing device that matched or exceeded the cognitive capabilities of a sentient being, it should be possible.

(Singlarity adherents believe we will one day be able to transfer the human mind into an artificial computing platform, and it logically follows that one could "hack" such a mind, or build from the ground up, to create a truly Artificial Intelligence.)

But this may be like fusion power, where the old adage is that it is "always 20 years away."


Which learning algorithms are suitable for data leakage detection and prevention?

This seems to fall broadly into the regime of a classification problem as you want to classify an outgoing communication as "contains proprietary information" or "does not contain proprietary information".  As such, any classification approach could be applied.  Neural Networks certainly seem like a valid approach, but you might also get good mileage out of Random Forests, Support Vector Machines, a Naive Bayes classifier, etc.

GA's are more aimed towards optimization than classification, so I wouldn't say that a GA, in and of itself, would map cleanly to solving this kind of problem. If GA's had applicability here, I think it would be more likely to be in terms of training a model rooted in one of the other techniques.


Train a competitive layer on nonnormalized vectors using LVQ technique

We have 2 classes , 1 subclass for each class

\begin{equation}
    W^2=\begin{vmatrix}
1 & 0\\
0 & 1\\
\end{vmatrix}
\end{equation}

$p_1$:

\begin{equation}
    \alpha^1=compet(n^1)=compet\begin{vmatrix}
||w_1-p_1||\\
||w_2-p_1||\\
\end{vmatrix} = compet\begin{vmatrix}
||\begin{vmatrix}
0 & 1\\
\end{vmatrix}^T-\begin{vmatrix}
1 & 1\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
1 & 0\\
\end{vmatrix}^T-\begin{vmatrix}
1 & 1\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
1\\
1\\
\end{vmatrix}) = \begin{vmatrix}
1\\
0\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1= \begin{vmatrix}
1 & 0\\
0 & 1\\
\end{vmatrix}\begin{vmatrix}
1\\
0\\
\end{vmatrix}=\begin{vmatrix}
1\\
0\\
\end{vmatrix}
\end{equation}

\begin{equation}
    W_1(1) = W_1(0) + \alpha\cdot(p_1-W_1(0))=\begin{vmatrix}
0\\
1\\
\end{vmatrix}+0.5\cdot(\begin{vmatrix}
1\\
1\\
\end{vmatrix}-\begin{vmatrix}
0\\
1\\
\end{vmatrix})=\begin{vmatrix}
0\\
1\\
\end{vmatrix}+\begin{vmatrix}
0.5\\
0\\
\end{vmatrix}=\begin{vmatrix}
0.5\\
1\\
\end{vmatrix}
\end{equation}

$p_2$ :

\begin{equation}
    \alpha^1=compet(n^1)=compet\begin{vmatrix}
||w_1-p_2||\\
||w_2-p_2||\\
\end{vmatrix} = compet\begin{vmatrix}
||\begin{vmatrix}
0.5 & 1\\
\end{vmatrix}^T-\begin{vmatrix}
1 & 2\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
1 & 0\\
\end{vmatrix}^T-\begin{vmatrix}
1 & 2\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
1.8027756377\\
2.8284271247\\
\end{vmatrix}) = \begin{vmatrix}
0\\
1\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1= \begin{vmatrix}
1 & 0\\
0 & 1\\
\end{vmatrix}\begin{vmatrix}
0\\
1\\
\end{vmatrix}=\begin{vmatrix}
0\\
1\\
\end{vmatrix}
\end{equation}

wrong class

\begin{equation}
    W_2(1) = W_2(0) - \alpha\cdot(p_2-W_2(0))=\begin{vmatrix}
1\\
0\\
\end{vmatrix}-0.5\cdot(\begin{vmatrix}
-2\\
2\\
\end{vmatrix}=\begin{vmatrix}
2\\
-1\\
\end{vmatrix}
\end{equation}

$p_3$ :

\begin{equation}
    \alpha^1= compet\begin{vmatrix}
||\begin{vmatrix}
0.5 & 1\\
\end{vmatrix}^T-\begin{vmatrix}
-2 & 2\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
2 & -1\
\end{vmatrix}^T-\begin{vmatrix}
-2 & 2\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
70\\
5\\
\end{vmatrix}) = \begin{vmatrix}
1\\
0\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1\begin{vmatrix}
1\\
0\\
\end{vmatrix}
\end{equation}

wrong class

\begin{equation}
    W_1(2) = W_1(1) - \alpha\cdot(p_3-W_1(1))=\begin{vmatrix}
0.5\\
0\\
\end{vmatrix}-0.5\cdot(\begin{vmatrix}
-2.5\\
1\\
\end{vmatrix}=\begin{vmatrix}
1.75\\
0.5\\
\end{vmatrix}
\end{equation}

$p_2$ :

\begin{equation}
    \alpha^1=compet(n^1)= compet\begin{vmatrix}
||\begin{vmatrix}
1.75 & 0.5\\
\end{vmatrix}^T-\begin{vmatrix}
-1 & 2\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
2 & -1\
\end{vmatrix}^T-\begin{vmatrix}
-1 & 2\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
3.13\\
4.24\\
\end{vmatrix}) = \begin{vmatrix}
1\\
0\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1\begin{vmatrix}
1\\
0\\
\end{vmatrix}
\end{equation}

\begin{equation}
    W_1(3) = W_1(2) - \alpha\cdot(p_2-W_1(2))=\begin{vmatrix}
1.75\\
0.5\\
\end{vmatrix}+0.5\cdot(\begin{vmatrix}
-2.75\\
1.5\\
\end{vmatrix}=\begin{vmatrix}
0.375\\
1.25\\
\end{vmatrix}
\end{equation}

$p_3$ :

\begin{equation}
    \alpha^1=compet(n^1)= compet\begin{vmatrix}
||\begin{vmatrix}
0.375 & 1.25\\
\end{vmatrix}^T-\begin{vmatrix}
-2 & 2\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
2 & -1\
\end{vmatrix}^T-\begin{vmatrix}
-2 & 2\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
2.95\\
5\\
\end{vmatrix}) = \begin{vmatrix}
1\\
0\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1\begin{vmatrix}
1\\
0\\
\end{vmatrix}
\end{equation}

wrong class

\begin{equation}
    W_1(4) = W_1(1) - \alpha\cdot(p_3-W_1(3))=\begin{vmatrix}
0.375\\
1.25\\
\end{vmatrix}-0.5\cdot(\begin{vmatrix}
-2.375\\
0.75\\
\end{vmatrix}=\begin{vmatrix}
1.5625\
0.875\\
\end{vmatrix}
\end{equation}

p_1 :

\begin{equation}
    \alpha^1=compet(n^1)= compet\begin{vmatrix}
||\begin{vmatrix}
1.5625 & 0.875\\
\end{vmatrix}^T-\begin{vmatrix}
1 & 1\\
\end{vmatrix}^T||\\
||\begin{vmatrix}
2 & -1\
\end{vmatrix}^T-\begin{vmatrix}
1 & 1\\
\end{vmatrix}^T||\\
\end{vmatrix}=
compet(\begin{vmatrix}
0.57\\
2.23\\
\end{vmatrix}) = \begin{vmatrix}
1\\
0\\
\end{vmatrix})
\end{equation}

\begin{equation}
   \alpha^2=W^2\cdot\alpha^1\begin{vmatrix}
1\\
0\\
\end{vmatrix}
\end{equation}

\begin{equation}
    W_1(5) = W_1(4) - \alpha\cdot(p_1-W_1(4))=\begin{vmatrix}
1.5625\\
0.875\\
\end{vmatrix}+(\begin{vmatrix}
-0.28125\\
0.0625\\
\end{vmatrix}=\begin{vmatrix}
1.28125\
0.9375\\
\end{vmatrix}
\end{equation}


Intuitively understanding translational invariance in CNNs

The translational invariance comes from the fact that the kernels are applied everywhere. Basically the kernels are "translated", i.e. shifted, and that means the feature they detect are detected everywhere.

Pooling may help with that in some cases, because in the end you need a dog at the right side to activate the same neuron as a dog on the left, so there needs to be funnelling of the information, but it is not strictly necessary. You could also imagine a CNN with a dog kernel in a late layer, which just recognises dogs everywhere, with no pooling necessary.

Fully connected networks are wasteful in that they don't share weights. To get the necessary translational invariance you therefore need to learn pretty much the same weights at all positions. But they do preserve the spatial relationship between pixels.

Pooling reduces the dimensionality, but it also forgets the precise position of a feature. This might help with generalisation, but I'm not sure whether that's a big factor.


How can I combine the readings of multiple lidars into 1 point cloud?

A high-level solution is to use the NVIDIA DriveWorks SDK.  NVIDIA's Developer Blog has a good post, "Point Cloud Processing with NVIDIA DriveWorks SDK".

The NVIDIA DriveWorks SDK contains a collection of CUDA-based low level point cloud processing modules optimized for NVIDIA DRIVE AGX platforms. The DriveWorks Point Cloud Processing modules include common algorithms that any AV developer working with point cloud representations would need, such as accumulation and registration.

I believe you don't need the AGX hardware for the problem you posted.


About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

Rosenblatt was probably discussing a specific architecture, for which there are many.  However, for general purpose feed-forward back-propagation ANNs used for function aproximation and classification analysis, you can use whatever activation functions you want on the input-side, hidden layers, and output-side.  Examples are identity, logistic, tanh, exponential, Hermite, Laguerre, RBFN, ReLu, softmax, etc.  "Automatically," likely refers to the iterative learning process, which tends to be similar to gradient descent, during which partial derivatives of prediction error w.r.t to coefficients decrease.


What is the relation between self-taught learning and transfer learning?

After a lot of searches, I think self-taught learning is a Transfer learning category, I think when Self-taught learning paper published (2007), there isn't any good survey on transfer learning, and as seen by high citation of the paper of Pan, his paper (which published in 2009) describes transfer learning in a clear way that did not exist before it.
Also, it is reasonable to consider Self-Taught learning a transfer learning category because he actually transfers the knowledge learned from unlabelled data to use it in the supervised task that we want to do (there is no need for the unlabeled data (that used for training) to follow the same class labels or generative distribution as the labelled data that will be used for the supervised task).
If someone could find something wrong with my answer or there is something missing, then please tell me.


Is the final model scaling done on the full training set?

The short answer is yes.
When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more data leads (generally) to better models" is valid to the scaling to the same extent it's valid for the model itself.


What is the best way to find the similarities between two text documents?

I would suggest to convert the documents into TF-IDF(use Gensim) vectors and then compare them using various similarity calculating techniques like cosine similarity.
You should read this amazing article for the same. I once used it while working on my project.
https://medium.com/@adriensieg/text-similarities-da019229c894


How do biological neurons weights get initialized?

In short

I mentioned in another post, how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of central nervous system development and synaptogenesis.

A bit more detail (with the most relevant parts italicized below)

The neocortex (one of its columns, more specifically) is a region of the brain that somewhat resembles an ANN. It has a laminar structure with layers that receive and send axons from other brain regions. Those layers can be viewed as "input" and "output" layers of an ANN (axons "send" signals, dendrites "receive"). Other layers are intermediate-processing layers and can be viewed as the ANN "hidden" layers.

When building an ANN, the programmer can set the number of layers and the number of units in each layer. In the neocortex, the number of layers and layer cell counts are determined mostly by genes (however, see: Human echolocation for an example of post-birth brain plasticity). Chemical cues guide the positions of the cell bodies and create the laminar structure. They also seem to guide long term axonal connections between distant brain regions. The cells then sprout dendrites in certain characteristic "tree-like" patterns (see: NeuroMorpho.org for examples). The dendrites will then form synapses with axons or other cell bodies they encounter along the way, generally based on the encountered cell type.

This last phase is probably the most analogous to the idea of random weight initialization in ANNs. Based on where the cell is positioned and its type, the encountered other neurons will be somewhat random and so will the connections to them. These connections are probably not going to be very strong initially but will have room to get stronger during learning (probably analogous to initial random weights between 0 and ~0.1, with 1 being the strongest possible connection). Furthermore, most cells are either inhibitory or excitatory (analogous to negative and positive weights).

Keep in mind this randomization process has a heavy spatial component in real brains. The neurons are small and so they will make these connections to nearby neurons that are 10-200 microns away. The long-distance connections between brain regions are mostly "programmed-in" via genes. In most ANNs, there is generally no distance-based aspect to the initialization of connection weights (although convolutional ANNs implicitly perform something like distance-based wiring by using the sliding window).

There is also the synaptic pruning phenomenon, which might be analogous to creating many low weight connections in an ANN initially (birth), training it for some number of epochs (adolescence), and then removing most low-weight connections (consolidation in adulthood).


How would a quantum computer potentially facilitate artificial consciousness, assuming it is possible?

Well, this is quite deep question. I would recommend you to have a look on the UK physicist Roger Penrose: https://en.wikipedia.org/wiki/Roger_Penrose
He has some books on the theme of consciousness and quantum mechanics. There are however critics of his ideas. One of them is that the quantum effects are too small to direct any considerable influence on the brain.


Why do Bellman equations indirectly create a policy?

Policy gradient methods directly learn parameters of a policy function, which is a mapping from states to actions. For example, $p(s, a)$ can denote a function which takes a state $s$ and an action $a$ as input, and returns a probability of taking action $a$ in state $s$ as output (equivalently it could just take $s$ as input, and output a vector or a distribution of probabilities for all actions). Such a function will often have some parameters (for example, a vector of weights if it's a linear function of the state, or matrices of weights in the case of a neural network). Policy gradient methods directly modify the function by modifying (learning) the parameters of that function.

The Bellman equation is typically used to learn a function that computes different kinds of values, such as $Q(s, a)$ values, which can intuitively be understood to denote all the future rewards that we expect to get if we take action $a$ in state $s$ (and follow a certain policy, typically "the optimal policy" afterwards until the end of time). Such a $Q(s, a)$ function that we learn is not a directly learned policy, because it is not a direct mapping from current states to actions, it doesn't directly tell us which action to take.

It can still be used indirectly to create a policy. For example, once such a $Q(s, a)$ function is learned, the greedy policy can be easily implemented by looping over all possible actions, and taking the action that gives the greatest $Q(s, a)$ output. This is indirect because the learned function does not directly give you the actions, but only provides information which you can convert into actions with some extra processing.


How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

The Ornstein Ulhenebck Process is defined as (in the continuous setting) :
$$dX_t = -\beta(X_t - \alpha)dt + \sigma dW_t$$
The analogue for this process in the discrete time case which I assume will be applicable in the RL case will be:
$$X_{t+1} = X_t -\beta(X_t - \alpha) + \sigma \{W_{t+1}-W_t\}=$$
$$X_{t+1} = (1 -\beta)X_t - \alpha + \sigma \{W_{t+1}-W_t\}$$
In the RL seting the terms in the equation probably means:

$X_t$ will stand for a state in RL i.e. the state is the number $\in \mathbb R$ where the particle moves to at time $t$.
$\beta$ and $\alpha$ are just constants which decide certain movement characteristics of the particle. Check here for graphs plotted for various $\beta$.
$W_t$ is a Weiner process which starts at $W_0 = 0$ and then adds independent increments of $\mathcal N(\mu,\sigma)$ as $W_{t+1} = W_t+\mathcal N(\mu,\sigma)$ which is basically a radom walk. More generally we use $\mathcal N(0,1)$. This is formulated as $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$. This is because of the fact, $W_t$ can be written recursively as $W_t = \mathcal N(0,1)+W_{t-1} = \mathcal N(0,1) + \mathcal N(0,1) + ...W_s$ and since the samplings are independent at each step the mean get added as $\mu_t+\mu_{t-1}...$ and the variances as $\sigma_t^2 + \sigma_{t-1}^2...$. SInce her the means and variances are $0$ and $1$ respectively, the final mean $\mu = 0$ and variance $\sigma^2 = (t-s)$. And hence, byproperties of Gaussian random variables you can write (it is easy to show this via variable tranformation) $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$.  Here, is the formulation of standard Weiner process.
$\sigma$ will be the weighting factor of the Weiner process, which just  means the amount of noise being added to the process.

Another useful resource on discrete Ornstein Ulhenbeck process, much less generalized.
I think now you can extend this to whatever scenario you are intereted in RL setting.


Why is probability that at least one hypothesis out of $k$ being consistent with $m$ training examples $k(1- \epsilon)^m$?

Let $A$ and $B$ be two events. In general, the probability that either $A$ or $B$ occurs is defined as

$$
P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)
$$

If $A$ and $B$ are disjoint, i.e. they cannot happen at the same time, then $P(A \text{ and } B) = 0$, so the above formula becomes

$$
P(A \text{ or } B) = P(A) + P(B)
$$

If the probability of one arbitrary hypothesis being consistent with $m$ training examples is $(1-\epsilon)^m$, then, given the rule above and assuming that only one hypothesis is consistent with $m$ training examples, the probability of one or more (i.e. at least one) of the hypotheses being consistent with training examples is the sum of the probabilities, i.e. $k (1-\epsilon)^m$.

This probability can be bigger than one if more than one hypothesis is consistent with $m$ training examples. In that case, you have to take into account the probability that both hypotheses are consistent.

See e.g. notes General Probability, I: Rules of probability for more details about the union rule and other rules of probability.


How to interpret this learning curve of my neural network?

It looks like your network is overfitting, because the training loss carries on decreasing to zero even though validation loss levels off, and then starts to increase again.
I would guess that your network is essentially "memorising" the training examples because you're getting a near zero loss in training.
You could try:

applying some form of regularisation
reducing the number of neurons or layers
collecting more training data to see if that helps.

Ideally you want to minimise validation loss, so epoch 20 was about the best point in your graph. Look into early stopping: this is an effective way to avoid overfitting too much.
When the learning rate is too high, you often see the loss 'jumping around' quite rapidly, and when the learning rate is too low, the graph barely changes at all. You could try fixing the learning rate closer to what it is around epoch 10 and see if this is "the sweet spot"; it's mostly a game of experimenting to see which works best.


Which areas of applied math are relevant to AI?

EE Math Refresh

AI is an interdisciplinary field.  You can begin by ensuring you are fresh in the mathematics you've already taken. You may already have all the books from your BS and MS.

Infinite series
Logical proofs
Linear algebra and matrices
Analytic geometry, especially the distinction between local and global extremes (minima and maxima), saddle points, and points of inflection
Set theory
Probability, especially Bayes' Theorem and the common probability distributions
Statistics, especially variance and correlation coefficient
Time series
Infinite series
Convergence — Central to AI batch processing
Partial differentials
Jacobian and Hessian matrices
Multivariate math — Spaces beyond ℝ3 are common in AI.
Boundary regions
Discrete math
Criteria for system stability — Central to real time intelligent control
Bernhard Riemann's manifolds is important for more advanced AI theory

An EE would already understand circuit paths (Gustav Kirchhoff's work, signal transmission, feedback theory, and control theory. To see the connection between AI and EE, Norbert Wiener's seminal work, Cybernetics, 1948, MIT Press, will be a helpful read and give you a clearer picture that both fields have common roots.

Finite Math

Since finite math curricula differ between schools of engineering, this list can be used as a checklist for AI.

Predicate logic and rules engines
Graphs (as in vertices connected by edges) and algorithms for them — pioneered by Leonhard Euler
Game theory — John von Neumann and Oskar Morgenstern's Game Theory (published in 1944 but still fresh and pertinent)
Decision trees
Markov chains and the Markov property — Andrey Markov's work
Information theory — Claude Shannon's extensions of the thermodynamic concepts of entropy, especially uniqueness, redundancy, relative entropy, and cross entropy — A necessity to more fully understand feature abstraction and auto-encoding
Chaos theory, especially auto-correlation, for analysis of chaotic system behavior in phase space — Chaos Theory Tamed, Garnett P. Williams, 1997, provides an excellent overview
Difference between random and pseudo random number generation
Curve fitting and gradient descent, especially the Levenberg–Marquardt algorithm
Stateless versus stateful algorithms, AI system components, and services
Algorithmic theory, especially tail recursion and algorithms for generalization, abstraction, and object recognition
Gödel's uncertainty theorems
Turing completeness and how it does not overturn yet overcomes Gödel's uncertainty
Turing's Imitation Game as a test for conversational intelligence
Other test criteria for intelligence outside the conversation context (i.e. piloting, driving, business intelligence, et cetera)
Topology — pioneered by Henri Poincaré
Computational linguistics — pioneered by Richard Hook Richens — the foundation of much NLP (natural language processing)

Machine Learning

Since machine learning is trending and will probably remain strong even as other AI technologies catch up, there are a few

MLPs (multilayer perceptrons)
Convolution and the use of convolution layers and pooling layers in deep convolution networks
LSTM (long-short term memory) networks, their predecessor RNNs (recurrent neural networks), and some of the newer competitors in the stateful network space
Attention based networks and other gating strategies (largely found online using academic searches for papers)

Soft Sciences

Because most of what is occurring, has occurred, and will occur in AI for the foreseeable future is the replication of human brain capabilities, the EE can point her or his analytical skills at biology (especially neurology), genetics as it may apply to the emergence and control of human intelligence (bioinformatics), psychology (especially cognitive science), and sociology (such as.

Because of the emergence of GANs and other devices that work on the basis of balance and equilibrium, it is useful to recall chemical equilibria and biological stasis.  These concepts apply directly to AI and are likely to play a more major role as the gap between brain network research and computing narrows further.

Automated Vehicles

The obvious emerging AI technology that is nicely aligned with the EE background is the movement of road and air vehicles from human driving and piloting through the various phases of automation.  The goal that most corporations see as the ideal is to progress forward until the driver's seats and cockpits are no longer installed in cars and planes because human interaction other than to provide the transportation destination poses a public safety issue.

Obviously, computer vision, object recognition, sound recognition, and collision avoidance technique are essentials and highly dependent upon rigorous mathematical treatment.


Which marketing-related classification challenges is a feed forward neural network suited to solve?

Has a data set been chosen? There's a wide variety of data out there (see this, this, or this).  Analytical possibilities are largely determined by the available data, so choosing your data would probably help to clarify your objectives.


Variable Number of Inputs to Neural Networks

We can generalize both problem and solution by removing the specifics of housing.

Representing Forward Propagation

We have a function $f$ we wish to obtain via the training of an artificial network that produces a scalar result $s$, the sole dependent variable and the generalization of market price.

$s = f(s_1, s_2, ..., s_k, c_1, c_2, ..., c_v)$

The independent scalar variables $s_1$ through $s_k$ are the generalization of a constant number $k$ of property features from the tax authority, assessor's office, or inspection document.  The question calls this structured data, however it is questionable whether $k$ is truly effectively constant.  In normal practice, some of $s_i$ will be unassigned.  Since the question overlooks the additional complexity of missing scalars, so will this answer.

The independent cube variables $c_1$ through $c_v$ are the generalizations of a variable number $v$ of property images from Google photography, real estate agents, buyers, sellers, and other potential sources.  The dimensions of each cube are horizontal and vertical positions and pixel structure element number.

It is unlikely that the resolution values for each cube are uniform between samples in real life, which the question did not mention, so this answer will overlook that complexity and focus on the variability of $v$, the quantity of cubes representing images for a given example.  Since $v$ cannot meaningfully be either negative or infinite, we can assume $v$ to be a non-negative integer.

Terminology

The observational unit should not be considered the house or the property but an image of it, which may be a member of a camera location and orientation category relative to the elements of the property.  Each image capture is an observation, distinct in both Bergsonian and clock time.  Each item under evaluation is an example from the sample of all items, in this case, all properties in the region for which prediction is attempted.

Design of a Solution

Each of the $v$ cubes demonstrate zero or more additional features of the item being evaluated, real properties in the question's specific case.

It may be reasonable to assume that such features may either positively or negative affect the example's label corresponding to the result $s$, but not both positively and negatively affect it.  If that is the case, we can aggregate the features across the set of cubes for each example under the reasonable assumption that there are no points of inflection.  Such may be reasonable because, for instance, a feature regarding the uniformity of paint coverage, lawn care, or roofing material may have no inflection point.  Such aspects of the property cannot be too uniform.  That makes the substitution straightforward.

A reasonably versatile way to generalize the aggregation of directionally consistent function is using a substitution, which may be what the question meant by feed in the embeddings of the image to your final model.

$s = f = f'(s_1, s_2, ..., s_k, v, h\big(v, \sum_{j = 1}^v g(c_j)\big)$

Note the elements of this substitution.

$h$ is a vector function that normalizes the distribution of each feature found in the feature vector before using it as a set of inputs to $f'$.
$g$ is a generalization of the cube, with the input (independent variable) being a cube representing the image and with the output (dependent variable) being a vector of features extracted.
$v$, the number of cubes (visual observations) is fed along with extracted features into the function $f'$, which can be realized through the convergence of a multilayer perceptron.

If the images are grouped in terms of the location of the camera, then this principle can be applied iteratively, where $o$ is the number of distinct categories of camera orientation.  In this case, the pairs $(v_z, h_z)$ represent the cube quantity and feature aggregations for image camera location category $z$.

$s = f = f'(s_1, s_2, ..., s_k, v_1, h_1, v_2, h_2, ... v_o, h_o)$

Given sensible models $h$ and $g$, training for prediction is straightforward.

Image feature extraction can be realized through ConvNet approaches such as OverFeat1, AlexNet2, CaffeNet3, GoogLeNet4, VGG 65 or PatreoNet6.  Tuning such models produces $g$.

The nature of function $h$ may be homogeneous or heterogeneous across dimensions.  Each component of the feature vector arising from extraction can have applied to it a function such as any of these or others, where $q$ is the feature index, $p_{qi}$ is learning parameter i.

$h_q(x) = x$
$h_q(x) = \large{x^{p_{q1} + p_{q2} v}}$
$h_q(x) = \log (x + p_{q1} v + p_{q2})$
$h_q(x) = \large{\epsilon^x}$
Others

Scalar function 1 is best when the designer wishes the convergence during the training of $f'$ in such a way that normalization is accomplished in the net.  It is a good choice for features where its frequency of occurrence and magnitude are across the entire set of images for a given example item is roughly proportional to the resulting value of that item.

Function 2 presents flexibility in normalization curvature with respect to the number of cubes.  Function 3 presents attenuation of the frequency of feature occurrence.  Function 4 presents compounding of feature effect with recurrence in the images of the same example.

The key is then the selection of how to deal with the substitution in the training in terms of procedure and wiring of corrective signaling.  Procedurally, there are three options.

Train both the ConvNet function $g$ and the multilayer perceptron function $f'$ together, extending the applicable principles of back-propagation and gradient descent.
Extract features first, tuning the ConvNet corresponding to $g$ prior to training the network corresponding to $f'$.  The advantage of this approach is manual control over and interim evaluation of feature extraction.
Use something similar to the mini-batch approach to find a balance between the above two extremes.

———

Footnotes

[1] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229v4

[2] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Neural Information Processing Systems, 2012, pp. 1106–1114

[3] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093

[4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, arXiv preprint arXiv:1409.4842

[5] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556

[6] K. Nogueira,  W. O. Miranda,  J. A. Dos Santos,  Improving spatial feature representation from aerial scenes by using convolutional networks, in:  Graphics, Patterns and Images (SIBGRAPI), 2015 28th SIBGRAPI Conference on, IEEE, 2015, pp. 289–296.


How is per-decision importance sampling derived in Sutton & Barto's book?

As mentioned in the comments your assumption about independence is wrong. Here's why. To prove independence we need to show the following holds:
$$P(X=x, Y=y) = P(X=x)P(Y=y)$$
in the case of RL this becomes:
$$P(X=a, X=a') = P(X=a)P(Y=a')$$
The left hand side has the value:
$$P(X=a, Y=a') = b(A_t = a| S_t = s) p(s'|a,s) b(A_{t+1} = a'|, S_{t+1} = s')$$
while the right hand side has the value:
$$P(X=a)P(Y=a') = b(A_t = a| S_t = s)b(A_{t+1} = a'| S_{t+1} = s')$$
And hence not independent.
Now let use look at why the following expression holds:

Eq.5.14:   $\mathbb{E}[\rho_{t:T-1}R_{t+k}] = \mathbb{E}[\rho_{t:t+k-1}R_{t+k}]$

I will not derive the exact expressions, but I hope you can form the reasoning I provide. By the rules of probability we know that sum of joint probability is equal to 1 i.e.:
$$\sum_{X_1..X_n} P(X_1=a_1, X_2=a_2,...X_n = a_n) = 1$$
I have alredy showed above, the trajectory is not independent. So $R_{t+k}$ will depend on the trajectory $S_{t:t+k-1}$ where $S_{t:t+k-1}$ is a particular trajectory. At the end of this trajectory we get a reward $R_{t+k}$ and thus $R_{t+k}$ is exclusively a function of $S_{t:t+k-1}$ i.e. $R_{t+k} = f(S_{t:t+k-1})$. The trajectory after this $S_{t+k:T-1}$ irrelevant since it will always sum up of to 1. i.e once you have reached a particular state at time step $t+k-1$ you are now conditioning based on that $P(S_{t+k:T-1}|S_{t:t+k-1})$ and taking the expected value over all trajectories possible from thereon i.e. $\sum_{S_{t+k:T-1}} P(S_{t+k:T-1}|S_{t:t+k-1})$ which is 1 by probability rules. Thus, what you are really doing is:
$$P(S_{t:t+k-1})R_{t+k}(\sum_{S_{t+k:T-1}} P(S_{t+k:T-1}|S_{t:t+k-1}))$$
and hence the remaining trajectory has no contribution.
Another way of thinking this is you are taking weighted trajectories till time step $t+k-1$ weighted by rewards $R_{t+k}$ and hence you cannot sum up to 1. The rest of the trajectory after $t+k-1$ will sum up to 1.
I hope this qualitative description suffices. You can do the maths, but you must be careful with the notations and the assumptions you make.
Also all the equations are correct, I hope you can indirectly see it from my reasoning.


How to use mixed data for image segmentation?

You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.


Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

No, mainly because these are all stochastic approximations and may not represent the true values.

Almost nothing good can be said about NN approximations to value and Q functions(at least according to a professor I have had).


How can certain information about the goal be given to the RL learning algorithm?

I see three main ways to do this, which one makes more sense will depend on your application.

One is to append that information to the state/observations like you mentioned. While this information is static for a particular episode, it will be different across episodes and the policy should learn to condition the actions it chooses based on this information.

Another would be to leave goal information out entirely and force the agent to learn a policy that works when the goal is unknown. This will likely be more difficult to learn and you may end up with a policy that moves to the average of all goals or explores and tries them all.

A third option and probably the most natural is to have some context cue that the agent can observe (e.g. often in lab experiments with rats, a cue is placed on a wall that lets the rat know which way to go to get a reward). This is similar to the first method, except that the cue has to be observed rather than given directly. For the Mountain Car example, this could be an extra signal that the agent only sees in a particular location, such as the bottom of the valley or when it moves close to a particular side.


How to make chatbot using NLP like Dialogflow?

You are right that you will have to implement the above steps before building any chat bot, this is because your computer doesn't understand text like you and I do. Therefore, text has to be pre-processed(converted) into a format that the computer can work with Tokenization, Parts of Speech Tagging and Named Entity Recognition are such pre-processing techniques.

Rule based means when you create rules for deciding how should your chat bot behave in a given situation.

You can have a look at this blog which teaches how to deploy a chat bot using DialogFlow on Slack - https://www.analyticsvidhya.com/blog/2018/03/how-to-build-an-intelligent-chatbot-for-slack-using-dialogflow-api/


How should the output layer of an LSTM be when the output are word embeddings?

Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/

After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.


Importance sampling is typically used when the distribution of interest is difficult to sample from - e.g. it could be computationally expensive to draw samples from the distribution - or when the distribution is only known up to a multiplicative constant, such as in Bayesian statistics where it is intractable to calculate the marginal likelihood; that is
$$p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)} \propto p(x|\theta)p(\theta)$$
where $p(x)$ is our marginal likelihood that may be intractable and so we can't calculate the full posterior and so other methods must be used to generate samples from this distribution. When I say intractable, note that
$$p(x) = \int_{\Theta} p(x|\theta)p(\theta) d\theta$$
and so intractable here means that either a) the integral has no analytical solution or b) a numerical method for computing this integral may be too expensive to run.
In the instance of your die example, you are correct that you could calculate the theoretical expectation of the bias dice analytically and this would probably be a relatively simple calculation. However, to motivate why importance sampling may be be useful in this scenario, consider calculating the expectation using Monte Carlo methods. It would be much simpler to uniformly sample a random integer from 1-6 and calculate the importance sampling ratio $x \frac{g(x)}{f(x)}$ than it would be to draw samples from the bias dice, not least because most programming languages have built in methods to randomly sample integers.
As your question is tagged as reinforcement learning I will add why it is useful in the RL domain. One reason is that it may be our policy of interest is expensive to sample from, so instead we can just generate actions from some other simple policy whilst still learning about the policy of interest. Second, we could be interested in a policy that is deterministic (greedy) but still be able to explore, so we can have an off-policy distribution that explores much more frequently.
NB: it may not be clear how you can use importance sampling if the distribution is only known up to a constant so see this answer for an explanation.


block of worlds with position aware goal stack planning

Your best bet would be to formulate the problem in PDDL, which should be fairly easy, and then use a standard planner to generate a plan from that description.
In PDDL you describe the properties and the possible actions, the start state and the goal state, and the planner will then take this to produce a sequence of actions that leads from the start state to the goal state. There is a planner available on-line that you can use.


Using sigmoid in LSTM network for multi-step forecasting

Yes, due the input, output being constrained between zero and one that would be the only viable activation function.


Are there any visual elements commonly associated with AI?

A characteristic visual element for neural network might be the node diagram, which shows (schematically) the nodes in the layers of a neural network and the connections between the nodes, both within and between the layers, that get weighted as the network is trained.

If you do a Google search on "neural network node diagram", you will see what I mean.

You might also think of any sort of video or photo with computer-identified features highlighted, as in a Google image search for "machine vision feature detection".

If you are looking for something more concrete, you can think of racks of GPU cards, or a spinning LIDAR scanner.


I don't think this classify as an NLP problem, there is almost no semantic analysis needed, it is more like a classification problem using categorical features.

NLTK is surely valuable if you want to perform some text 'cleaning' or preprocessing before encoding the variables. The only NLP application that I think you could apply here is some sentiment analysis on the comments to extract extra features (like a number expressing the negativeness or positiveness of each comment). Nevertheless you might want to do that using some pre-trained models cause your dataset is pretty small.


How are small scale features represented in an Inverse Graphics Network (autoencoder)?

Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never knows. However we can run PCA (Principal Component Analysis) to see which feature is the most "important" of all, aka which feature affects the most in the output image. Then, you can try adjusting the value to manually see and guess what teh value do, but you will never know what exactly it does as it is an arbitrary feature, not specifically set by the network. One value may mean multiple things, or just things we human don't understand. See this amazing video about this for details:

https://youtu.be/4VAkrUNLKSo

This video explains what PCA does and also an example of the features generated by the network.

For small scales features, they may simply be ignored as they don't contribute much to the loss or accuracy, or maybe they are represented by a big dot or something else until the last few layers. With just 80 features one cannot fully represent a face with such details, and with the resolution the networks like these are trained on, small features like these probably won't be shown in the image.


Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I found the answer to my question, went back to the Python script and in the command that fits the model i.e.

 LR = LogisticRegression (C=0.1, solver = "sag",max_iter=1000).fit (X_train, y_train)

The parameter C was set to 0.001 which is a very small value (meaning lambda is very high as C=1/lambda) (C is the regularization strength and smaller values indicate stronger regularization). more on that matter can be found here and here


How to start learning about artificial intelligence?

Read:

'Artificial Intelligence - A modern approach' by Russell and Norvig.

'Fluid Concepts and Creative Analogies' by Douglas Hofstadter.

'Machine Learning and Pattern Recognition' by Bishop

'The Emotion Machine' by Marvin Minsky


Traveling salesman problem variant: which algorithm to choose?

If you have lots of rules then you should be able to come up with lots of heuristics for A*. If I were you I would try A* first, and come up with as many heuristics as I could. You can also use Deep Q learning. I don't think you want to just throw coordinates into a Deep Q network. You want a simpler representation. I would probably give the nodes symbolic identifiers like sparse matrices, integers or sigmoids, and then craft the loss function to reflect the actual cost of moving from node to node and ascribe a high cost to illegal moves.


How to make meaningful sentences from a set of words?

How do you define "meaningful"? Generally, you would start from concepts and meanings, and then realise them in syntactic structures using lexical items (words). You seem to want to start in the middle somehow.

For turning a semantic representation into a valid sentence, you would use a generator; these are often based on grammars. Examples exist which take a grammar, fill in random words, and create a syntactically well-formed sentence; often they will, however, be rather non-sensical or meaningless. Have a look at this site which describes the Syntax Construction Kit. The author, Mark Rosenfelder, links to a number of toy programs which do exactly that. Just substitute his lexicon with the list of words created by your LDA process. See for example this generator based on generative grammar.


How to "combine" two images for CNN input (classification task)?

You can combine the image output using concatenation. Please refer to this paper:

http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf

You can have a look at the Figure 2. And if you are using caffe, there is a layer called Concat layer. You can use it for your purpose.

I am not fully clear about what you want to do. But like you said, if you want to pass the image values from the first layer to some layers. Try reading about skip architectures.

If you want to use this network as real/fake finder, you can take the difference between two images and convert it to classification problem.

Hope it helps.


What is a Markov chain and how can it be used in creating artificial intelligence?

A Markov model includes the probability of transitioning to each state considering the current state. "Each state" may be just one point - whether it rained on specific day, for instance - or it might look like multiple things - like a pair of words. You've probably seen automatically generated weird text that almost makes sense, like Garkov (the output of a Markov model based on the Garfield comic strips). That Coding Horror article also mentions the applications of Markov techniques to Google's PageRank.

Markov models are really only powerful when they have a lot of input to work with. If a machine looked through a lot of English text, it would get a pretty good idea of what words generally come after other words. Or after looking through someone's location history, it could figure out where that person is likely to go next from a certain place. Constantly updating the "input corpus" as more data is received would let the machine tune the probabilities of all the state transitions.

Genetic algorithms are fairly different things. They create functions by shuffling around parts of functions and seeing how good each function is at a certain task. A child algorithm will depend on its parents, but Markov models are interested mostly in predicting what thing will come next in a sequence, not creating a new chunk of code. You might be able to use a Markov model to spit out a candidate function, though, depending on how simple the "alphabet" is. You could even then give more weight to the transitions in successful algorithms.