Which challenges in information security can be solved better using neural networks?

The most natural place where artificial networks can be used in information security is in attack detection.

The security team leaders of more than one web hosting company told me the same story.  Their teams' daily challenges are to defend against the attacks mounted continuously by several overseas teams against the IT security of their hosting infrastructure.  Attack teams specialize in one target hosting company, and there are often many attack teams on one defensive security team.

The hosting company security teams try to preemptively defend the assets of their companies from attacks.  These teams also have to restore their hosting system with minimal impact on hosted sites whenever an attack achieves its goal, which may be data corruption, data theft, the insertion of malware, or some other manipulation.

The primary mechanism for preemptive defense is detection of an attacker and an ascertaining the attack's intended target within the hosting infrastructure's defenses, after which known blocks to these attacks are employed, such as IP or MAC filters, modification of access rights, upgrades that plug known security holes, or other measures.

The progress of attacks occur across some set of changing fronts, as in a war or a game of chess.  Information security may soon become largely a game between two sets of computer systems.  One trying attacks, and the other trying to detect and block them.

For the defending team, high speed, real time detection of the attack is key.  A properly designed deep learning solution, configured and integrated into a defensive system, is more likely to block an attack than a person watching a dashboard of log file stats and system metrics.  A deep network could also be trained to discover the subsystem or specific defense against which the attack is intended.

Once detection occurs, the application of patches, exclusion filters, and denial criteria may best be accomplished through a rules based system.


How to measure sample efficiency of a reinforcement learning algorithm?

First of all, let's recall some definitions.
A sample in the context of (deep) RL is a tuple $(s_t, a_t, r_t, s_{t+1})$ representing the information associated with a single interaction with the environment.
As for sample efficiency, it is defined as follows [1]:

Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance.

So, how you measure it is closely related to the way it is defined.
For example, one way to do it would be as shown in the figure below:

On the y-axis you have the performance of your RL algorithm (e.g. in terms of average return over episodes as done in [2], or mean total episode reward across different environment runs as done in [3])

On the x-axis you have the number of samples that you took.

The dashed line corresponds to your performance baseline (e.g. the performance at which a certain game or any other RL environment is considered solved).

So, you can measure sample efficiency at the intersection, where you get the number of samples needed to reach the performance baseline. So, an algorithm that requires less samples would be more sample-efficient.

Another way to do it would be the other way around, i.e. the RL agent is provided with a limited budget for the number of samples it could take. As a result, you can measure sample efficiency my measuring the area under the curve, as illustrated below. So, that would be how much performance you got just by using those samples in the budget. An algorithm that achieves a higher performance than another with the same amount of samples would then be more sample efficient.

I am not aware if there exist RL libraries that would provide you with this measure out-of-the-box. However, if you're using Python for example, I believe that using libraries like scipy or scikit-learn along with matplotlib could do the job.
NB: Image credits go to the following presentation: DLRLSS 2019 - Sample Efficient RL - Harm Van Seijen


What order should I learn about Neural Networks?

I think, once you are covered with the common stuff, you can probably go on and study all kinds of neural network variants.

The common stuff:

a) An undergraduate level Linear Algebra course -- covering matrix calculus. You might find this useful.

b) An undergraduate level study in statistical inference. Concepts from this topic will come up most of the time and you might have hard time getting around even though you understand the rest of the math. I would recommend this.

c) A starter book on neural networks. Ex- Neural networks by Raul Rojas.

After all these are covered you will certainly be ready for learning the variants of neural networks with ease. For LSTM I would recommend Alex Graves.


Can a basic CNN (Conv2D, MaxPooling2D, UpSampling2D) find a good approximation of a product of its input channels?

I think U-Net is already a quite complex network that probably should (as of my experience) be able to approximate this multiplication. However this would still be an approximation that might not be accurate and maybe only solves a function that looks like a multiplication for the range of the shown input samples defined by your dataset (therefore potentially overfitting on your training dataset).

So in general if you know that your target function does have this multiplication then you should definetly excplicitly enforce it in your network. If you know so much about the wanted function it's always better to build a good fitting architecture than using a general neural network architecture. That will ease the optimization and should generalize much better.

However, it's hard or impossible to tell you for sure how much depth or complexity you need to solve this task within your wanted accuracy. Eventually you should just try it out.


Best approach for 2D Grid Image Segmentation

This is a really cool problem. You already have a working model here are a few different ways of going forward with the project.

Grouping text based on locality. "no segmentation"  Text region extraction in a document image based on the Delaunay tessellation
Segmentation  Multiscale Edge-Based Text Extraction from Complex Images
Training a map of the image then extracting text form the map.  Multi Oriented Text Detection

Training the model to detect boxes followed by extracting the text from each box seems like a very smart direction to move in. This paper talks about the former.  automatic image segmentation and edge detection  Good luck!


What was the first machine that was able to carry on a conversation?

In 1986, the first PC therapist program was written by Joseph Weintraub. This program won the first Loebner Prize in 1991, and then again in 1992, 1993 and 1995.

In 1981 or 1982, Jabberwacky was founded, which is the foundation of the current Cleverbot. Jabberwacky  appeared on the internet in 1997, reaching the third place for the Loebner Prize in 2003, the second place in 2004, and won in 2005 and 2006. In 2008, Cleverbot was launched as an variant of Jabberwacky.

I'm not sure these are really the earliest, but that also depends on what you want earliest (programming started, first conversation,  first decent conversation, etc.). Also, it depends on what you call a "real response".


Why do neural networks based AI players still require extensive search techniques?

It is not very accurate to say that AI players requires extensive search while humans don't. Rather, it is a question of degree.

AIs do a lot more calculating, because that's what computers are good at.

Human intuition is much more powerful than a neural network can currently hope to match, because it is much more integrated into a world of knowledge about the game, it uses orders of magnitudes more neurons and it is not a static thing that just provides a move.

But if a human player stops calculating ahead his playing strength will drop very significantly. This can be seen by assessing the performance in games with short time controls: The less time you have the less calculation is happening, your intuition however is fast.

If you take out the search component of AlphaGo it would still play quite strongly, probably at a low dan level. Of course that is also far below its strength.

So, search is always an important component of playing strength, just more so for machines.


Do we need the transition probability function when calculating the importance sampling ratio?

There is one thing I don't particularly understand. Why do we need the state-transition probability function when calculating the importance sampling ratio for off-policy prediction?

It is not needed for calculation. It must be included in the theory, to compare the correct probability of each trajectory (on-policy vs off-policy). However, the state transition probabilities are then also shown to cancel out exactly, so there is no requirement to know what the values are.

State transition probabilities are irrelevant to probability ratios between identical trajectories where the policy varies but the environment does not. Which is the case for off-policy learning.

This is all explained on the relevant pages of the book, but replicating here for completeness. If your trajectory is from steps $m$ to $n$ and called $\tau = (s_m, a_m, r_{m+1}, s_{m+1}, a_{m+1}, r_{m+2} . . . r_{n}, s_n)$, then the probability of seeing that trajectory under two different policies $\pi(a|s)$ and $b(a|s)$ is:

$$p(\tau|\pi) = (\prod_{i=m}^{n-1} \pi(a_i|s_i))(\prod_{j=m+1}^{n} p(s_j,
r_j|s_{j-1},a_{j-1}))$$

$$p(\tau|b) = (\prod_{i=m}^{n-1} b(a_i|s_i))(\prod_{j=m+1}^{n} p(s_j, r_j|s_{j-1},a_{j-1}))$$

You can clearly see that the second product in both cases is the same, and cancels out in the ratio:

$$\frac{p(\tau|\pi)}{p(\tau|b)} =\frac{\prod_{i=m}^{n-1} \pi(a_i|s_i)}{\prod_{i=m}^{n-1} b(a_i|s_i)}$$

So it doesn't matter what $p(s', r|s, a)$ actually is, just that it exists and is not allowed to change in theory between the on-policy and off-policy cases. Or in other words, that the environment is a consistent MDP.


How should dropout change with network depth?

We can think of the dropout as of averaging over many networks. Not as a way to mute or denoise your data. Therefore the rule of thumb is the dropout should be applied to higher layers in your network. It is usually applied to fully connected layers (if at all!). It is a common practice to keep 0.5-0.75. neurons active. Personally I prefer L2 much more.


What roles knowledge bases play now and will play in the future?

First of all, I would like to point out the main differences between knowledge base and (Deep) machine learning, specially when the main focus is on "AI" not "Data Science":

NNs are like a black box; Even if they learn a dataset and gain the power of generalization over the problem domain, you'd never know how they are working. if you scrutinize the details of the developed model, all you see are digits, weights, poor and strong connections and transform functions. the "feature extraction" step before the training phase literally tells you: "hey human, enough with your complicated world, let's start zeros and ones". In the case of DL, it is worse! we do not even see what the selected and effective features are. I'm not a DL expert but as much as I know, DL's black box is darker! But knowledge bases are written in a human-friendly language. after a knowledge accumulation phase, you could see all the connections between the entities, and more important, you could interpret those connections. if you cut a wire in a knowledge base, your model will lose just a bit of its power, and you know what exactly it will lose; for example disconnecting the "Pluto" node from the "solar system" node, will tell your model what deGrasse Tyson told us. but in a ML model, this might turn it into a pure useless one: what happens if you manipulate the connection between the neuron number 14 and 47 in a NN model used to predict which planets belong to the solar system?!
ML models are merely an inscription of the data. They do not have the power of inference, and they don't give you one. knowledge base is on the other hand capable of inference from the prior knowledge as you indicated in your question. It is shown that DL models that have been trained with say image classification data, could also be applied to voice detection problem. But this doesn't mean DL models could apply its prior knowledge in the domain of images to the domain of voices.
You need kilos of data for traditional ML algorithms and tons of data for DL ones. but a single instance of a dataset will create a meaningful knowledge base for you.

There are two main research topics in NLP: machine translation and question answering. Practically it has been shown that DL works significantly with machine translation problems but acts kind of stupid in question answering challenge, specially when the domain of topics covered in the human-machine conversation is broad. Knowledge bases are no good choice for machine translation but are probably the key to a noble question answering machine. Since what matters in machine translation is only the translated version of a text (and I don't care how on earth has the machine done that as far as it is true) but in question answering problem, I don't need a parrot who repeats the same information I gave it to him, but an intelligent creature who gives me "apple is eatable" after I tell him "apple is a fruit" and "all fruits are eatable". ML models are used to elicit underneath patterns from the dataset (translation) while knowledge bases are used to extend the domain of knowns. ML models nitpick, KBs explore!


Given enough graphical data, could you train an AI to plot a polynomial graph based on the input conditions?

Yes this is possible, using any machine learning approach that supports regression. You have two main approaches:

Input $h$ the height of the drop, multiple outputs, one per time offset that you want to plot. Each individual output calculates the predicted force at a specific offset time.
Inputs $h$ the height of the drop and $t$ a time offset, one output. The single output calculates the predicted force due to given height and at given time.

The main thing to bear in mind is that statistical learning techniques typically do not generate physics-like models. Test inputs close to training examples should generate reasonable graphs that interpolate between those from training data. Test inputs far away from the training examples (e.g. you train on data of drops from 60m to 140m, but use an input of 10m or 200m) will likely generate wildly incorrect outputs. The main exception to this is if your ML model includes some good guesses at the underlying physics model, in which case it is possible that a regression algorithm will tune the parameters of that model plus filter out terms that should not be part of the model, resulting in a system that extrapolates much better. That is very unlikely happen by chance, it requires up-front design.


Is it possible to have a variable-length latent vector in an autoencoder?

If you use RNNs, then I think the solution is to use padding (zero padding) with max sequence length (that is the max number of words in a text) in order to tell your model to skip the zeros when possible. In that way, your model will try to learn a good representation of your input with fixed size. If you do not know this dimension, a solution may be to grid search this hyperparameter.
If you still want to exploit the dimensionality difference, maybe you can train different models with fixed size dimension of the representation dependently of the dimension of the input. That is, for example, use one for small, one for medium and one for large dimensions, but this should surely require to have a large and quite balanced initial dataset.
Another idea could be to use the autoencoder with a fixed latent dimension. Then, do effective clustering on your samples using their latent representation, considering that similar representations should have similar dimensionality requirements (?). After that, you could train your initial dataset on k models, the same number as the clusters. That is, there should be k different latent spaces. The goal is to match each instance to the correct model. At first, you should train them all with each instance, but as the training progresses, you should maybe go with binary search for each instance in order it to find the correct model, assuming that there is total order in the measuring of the dimensionality requirements. Of course, this is just an idea, I don't know if it is going to be really helpful at all.


How can I change observation states' values in OpenAI gym's cartpole environment?

I don't recommend changing the rules of the environment.
What you could do:
Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go into and then let your agent use the bucket number as the observation.
e.g. Say I do have a continuous state space with one variable in range $[-\infty,\infty]$
The buckets can be as follows:
0). x < -1000
1). -1000 $\le$ x $<$ -500
2). -500 $\le$ x $<$ -100
3). -100 $\le$ x $<$ -50
4). -50 $\le$ x $<$ 0
5). 0 $\le$ x $<$ 50
6). 50 $\le$ x $<$ 100
7). 100 $\le$ x $<$ 500
8). 500 $\le$ x $<$ 1000
9). x > 1000
Therefore in this example scenario there are 9 buckets. Hence, the observations can be in range [0, 9] discretely.


What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a specific state. A value function is not a "return function", it is an "expected return function" and that is an important difference.

A return is a measured value (or a random variable, when discussed in the abstract) representing the actual (discounted) sum of rewards seen following a specific state or state/action pair.

Typically there is no need to express an individual return as a "return function", although you may find many formulae in RL for sampling or estimating specific return values in order to calculate targets or errors for the value function.

  A return (or cumulative reward) function effectively adds up the rewards from the current state to the goal state. This implies it's model-based.

If you have a simple MDP, already accurately modelled, where you can calculate expected return directly from that model, then, yes, in theory, that would be a value function. However, this could be more computationally intensive to resolve than dynamic programming (e.g. Policy Iteration or Value Iteration), and in many cases you don't have any such model, but can still apply RL approaches to learn a value function from experience.


In general, the expression "temporal feature" might refer to any feature that is associated with or changes over time.
However, in the context of signal processing, a temporal feature might refer to any feature of the data before being transformed to the Fourier, frequency or spectral domain, using the Fourier transform. In this context, the domain of the untransformed data is often called "time domain" (as opposed to the "frequency" or "spectral" domain, which is the domain of the transformed data), even though it might not be strictly associated with or defined as a function of time. For example, in image processing, an image can be interpreted as a 2D signal. The domain of an image can be referred to as the "time domain", even though it is usually and more correctly referred to as the "spatial domain" (given that an image can be thought of as a function from a pixel, which is defined by two numbers $x$ and $y$, to a value, e.g. a grayscale value). You can transform this image, using the Fourier transform, to the spectral domain. In that case, the domain of the result of the transformation can be referred to as the "spectral domain".
In the paper Learning Temporal Features Using a Deep Neural Network and its Application to Music Genre Classification, the authors define spectral and temporal features

Extracting features from audio that are relevant to the task at hand is a very important step in many music information retrieval (MIR) applications, and the choice of features has a huge impact on the performance. For the past decades, numerous features have been introduced and successfully applied to many different kinds of MIR systems. These audio features can be broadly categorized into two groups: 1) spectral and 2) temporal features.
Spectral features (SFs) represent the spectral characteristics of music in a relatively short period of time. In a musical sense, it can be said to reveal the timbre or tonal characteristics of music. Some of popular SFs include: spectral centroid, spectral spread, spectral flux, spectral flatness measure, mel-frequency cepstral coefficients (MFCCs) and chroma.
On the other hand, temporal features (TFs) describe the relatively long-term dynamics of a music signal over time such as temporal transition or rhythmic characteristics. These include zero-crossing rate (ZCR), temporal envelope, tempo histogram, and so on.
The two groups are not mutually exclusive, however, and many MIR applications use a combination of many different features.

These definitions are given in the context of signal processing (as mentioned above).
To conclude, the meaning of "temporal feature" might change depending on the context. Hence, you should interpret it given the context, but it is almost always associated with time (in some way).


You are referring to 'proactive AI' as opposed to 'reactive AI' like Alexa, Cortana, Siri, Bixby, Google Assistant, and others.  There hasn't been much progress in this area of AI.  Google's recent demonstration of Duplex addresses this to some extent.  Some chatbots are proactive. Genesys provides such capability.  Check out their video

Azure's bot service has a page on how to implement proactivity and there is another video that walks through the whole process: Learn to build Proactive Bot in 30 Minutes.


What is the difference between a stationary and a non-stationary policy?

A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy), or a conditional density, $\pi(A \mid S)$ (a stochastic policy). A non-stationary policy is a policy that is not stationary. More precisely, $\pi_i$ may not be equal to $\pi_j$, for $i \neq j \geq 0$, where $i$ and $j$ are thus two different time steps.
There are problems where a stationary optimal policy is guaranteed to exist. For example, in the case of a stochastic (there is a probability density that models the dynamics of the environment, that is, the transition function and the reward function) and discrete-time Markov decision process (MDP) with finite numbers of states and actions, and bounded rewards, where the objective is the long-run average reward, a stationary optimal policy exists. The proof of this fact is in the book Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994), by Martin L. Puterman, which apparently is not freely available on the web.


Suitable reward function for trading buy and sell orders

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function.
$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$
where $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals a buy and sell, respectively, and zero otherwise; $sc$ represents the selling cost and $bc$ the buying cost. ROI is the difference between final bank balance and starting bank balance after trading.
You are correct, that the machine learning algorithm will then be influenced by spikes just before a sell.
Nicholls et al showed that using the average profit or area under the trade resulted in better performing trading rules. This approach was used by Schoreels et al. This approach focuses on being in the market to capitalize on profit. It does not penalize the trading rule when it is in the market and the market is going down. The accumulated asset value (AAV) is defined as:
$AAV = \frac{\sum_{i=1}^N [(Price_s - sc) - (Price_b + bc)]}{N}$
where $i$ is a buy and sell trading event, $N$ is the number of buy and sell events, $s$ the day the sale took place, and $b$ is the day the purchase took place.
Nicholls MSc thesis [available April 2019] showed that the fitness function used by Allen and Karjalainen is the preferred fitness function when evolving trading rules for the JSE using evolutionary programs.
Allen and Karjalainen used a fitness function based on the compounded excess returns over the buy-and-hold (buy the first day, sell the last day) strategy. The excess return is given by:
$\Delta r = r - r_{bh}$
where the continuously compounded return of the trading rule is computed as
$r = \sum_{t=1}^T r_i I_b(t) + \sum_{t=1}^T r_f I_s(t) + n\log\left(\frac{1-c}{1+c'}\right)$
and the return for the buy-and-hold strategy is calculated as
$r_{bh} = \sum_{t=1}^T r_t + \log\left(\frac{1-c}{1+c'}\right)$
In the above,
$r_i = \log P_t - \log P_{t-1}$
and $P$ is the daily close price for a given day $t$, $c$ denotes the one-way transaction cost; $r_f$ is the risk free cost when the trader is not trading, $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals buy and sell, respectively, and zero otherwise; $n$ denotes the number of trades and $r_{bh}$ represents the returns of a buy-and-hold, while $r$ represents the returns of the trader.
A fixed trading cost of $c = 0.25\%$ of the transaction was defined but this could be anything like a STATE fee + Brocker fee + Tax, and might even be 2 different values, one for buying and one for selling. Which was the approach used by Nicholls. The continuously compounded return function rewards an individual when the share value is dropping and the individual is out of the market. The continuously compounded return function penalises the individual when the market is rising and the individual is out of the market.
I would recommend that you use the compounded excess return over the buy and hold strategy as your reward function.


How does the network know which objects to track in the paper "Label-Free Supervision of Neural Networks with Physics and Domain Knowledge"?

They are not tracking anything, instead they are trying to find an object which satisfies free fall equation. Gravity acts the same regardless of object's properties - at least in vacuum.

"In this paper, we model prior knowledge on the structure
of the outputs by providing a weighted constraint function
g used to penalize “structures” that are not
consistent with our prior knowledge."

There is a restriction to the possible parabola that is given in the last equation on page 2. They are training the Network to learn time dependence of that equation, which is the same for all objects.


How can we get a differentiable neural network to count things?

Estimating from an observation is a function, but "really counting" is a process. Feed-forward neural networks can learn arbitrary functions from training examples, but they cannot represent (and therefore cannot learn) processes. They can attempt to estimate the results of completing a process as a function, but that is not the same thing as actually performing the process.
To learn arbitrary processes from examples requires a model with some concept of state and evolution over time in addition to any functions that are required. Recurrent neural networks (RNNs) are suitable models for those kinds of learning problems, but so are other AI learning constructs.
This rabbit hole goes deep, and for any model you could build it is possible to ask:

Is the system really performing a counting task similar to how a human might attempt the same task?

Has the sytem really learned to count, or has it been constructed by the developer so that some parts of the process are inevitable?

Has the system learned to count in general within any sub-component, or are the things that it can count tightly coupled across components?

There are other questions you could ask too, depending on your goals for producing such a model.
It is worth noting that for small quantities and certain patterns, that human "counting" does more closely resemble the estimation process, or perhaps is more akin to an NLP problem. For example, consider how you "count" the pips on a die - although you can use a counting process to confirm what you see, typically you do not do so. Instead, some part of your brain is supplying a very accurate answer quickly and unconsciously:

More generally, counting in humans likely consists of multiple related strategies for converting observations into symbolic and/or sensory representations of quantity. Some may be instinct, some are learned conscious behaviours and some appear to be in-between, perhaps originally learned but turned into subconscious skill through repetition.
Ignoring whether or not you are implementing your AI project for counting items in an image wholly within a neural network for now, I think you need the following components:

A state that represents and tracks the start, progress so far and end of the counting process with respect to the input being processed.

An accumulator that represents quantity counted so far.

A detector that can trigger a counting event when it observes something that needs to be counted.

A strategy or planner for processing input so that detection events are separated and only triggered once for each valid event.

The last component can vary a lot, and humans will use a range of different strategies for counting depending on the difficulty of the task. For instance, you might use working memory (a limited resource in humans) to track a few key points in an image to help segment an image into smaller sections as you work across it. Or you might make visual marks on an image to track each object that had already been counted. For a human, counting strategies can be mixed and matched, and switched between sometimes even during the same counting task.
All of these components could be represented in learnable parts of an AI process, but they do not have to be. When you suggest in your question that

I don't want to use a detector, then count the number of bounding boxes.

then you are most likely saying that you don't want to use a fixed hard-coded strategy. You want to somehow create a neural network that can discover at least one strategy for processing the input.
The trouble you will face is that giving a large RNN model (e.g. LSTM) a dataset of examples with input images and output correct counts will likely be too much of a challenge. Discovery of a robust object counting system from scratch will be too hard.
There are a few things that may help you construct something that "really" learns to count. Here are a couple of ideas:

Curriculum learning, where you start training the neural network with some hand-holding examples, and slowly ramp up the complexity. This is analogous to how we teach children to count.

Designed-in modelling for processing strategy. For instance, you could add a virtual fovea and artificial visual saccades and require that the neural network output a sequence of locations within the image that it picks out to run the detector against. This will be a constraint that allows certain types of human-like counting strategy to work, and could simplify the problem to the point that the network has a chance to learn it.

A paper that uses a "fovea" model for object detection: Object detection through search with a foveated visual system


Should I remove the text overlaying some images in the dataset before training the CNN?

Removing the overlayed text might increase accuracy, but you'd need to train a different model to do this, and that is an entirely different task as it is no longer classification, but generation. There are easier ways to augment your data and probably get similar benefits to your accuracy. However, if you would still like to do this, there is a lot of examples you can find by simple searching "Watermark removal machine learning" in google. Here's an example I found.
Overall, a CNN will be able to look past the overlayed text without issue, and perform classification like it would without the overlayed text. There is the possibility that it actually learns relationships between the overlayed text and the expected output, but that depends on the data, and is likely a harder task then simply identifying features.
The only issue you might run into is if the real data this model will be used on is different to the data provided, as in the real word images do not contain overlayed text describing what the image is.


Java - A good place to begin if over all goal is ML and Ai?

If goal is to develop ML algorithms then focus on Maths concepts linear algebra, probability and statistics. Try out CS problem solving basic data structure and algorithms.
Python has good ML libraries but if you know java then you can pick python easily.


Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://openai.com/blog/better-language-models/


What is the reasoning behind the number of filters in the convolution layer?

From mathematical point of view you are correct as are your calculations. To catch all the patterns you need that many filters, but this is where a whole idea of a training comes in. Main objective of the training in the CNNs is to find just a few good patterns from billions possible ones.

So the direct answer to your question is: The standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns, yes but this is assuming no training taking place. If you conducted training on given data with given model, then 64 to 1024 filters could already extract a lot of useful patterns, perhaps more than needed.


Can we use the recursive least squares as a learning algorithm to an ADALINE?

RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section "8.6 Approximate Second-Order Methods" of the following book available online:
https://www.deeplearningbook.org/contents/optimization.html
Deep Learning
An MIT Press book
Ian Goodfellow and Yoshua Bengio and Aaron Courville


Are the ideas in the paper "Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour" novel?

Poel's paper on Translating Values into Design Requirements articulates a framework for mapping abstract values and norms into concrete design constraints that an engineer could work with. The example used in the paper is mapping beliefs about animal welfare to design constraints on chicken coups.

The newer paper by Tubella et al. on Governance by Glass-Box builds on Poel's idea (and in fact, cite's Poel several times). It basically suggests that we should use Poel's design process, but that in something like an AI system, we also need to use an "observation phase" to validate the system, because unlike the problem of engineering a chicken coup, an AI system may appear to have met design constraints, but routinely violate them in production.

So, you're right that the Tubella et al. paper is essentially proposing the combination of Poel's framework for translating values into design constraints with the old idea of black-box testing, but this combination itself appears to be a new, if modest, contribution.


How important is learning to learn for the development of AGI?

Learning to learn (also known as meta-learning) is very important for the development of artificial general intelligence (AGI), given that one of the desirable and fundamental properties of an AGI is the adaptability to different environments and the ability to continually learn, and meta-learning can be used to achieve that.

Meta-learning is thus related to multi-task, continual (or lifelong) and transfer learning. There are several approaches to meta-learning, such as MAML, but there is usually a meta-learner and a learner.

In the paper Building Machines That Learn and Think Like People (2016), Marcin Andrychowicz et al. argue that truly human-like learning and thinking
machines (or AGI) should

build causal models of the world that support explanation and understanding
ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned
harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations

Another person that thinks that meta-learning is important for the development of AGI is Ben Goertzel. See e.g. his article From Here to Human-Level Artificial General Intelligence in Four (Not All That) Simple Steps.


How do LSTM or GRU gates learn to specialize in their desired tasks?

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer.

Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (say, like a full stop). This forget gate, while it might not force a forget, has the ability to force one, and will likely learn to.

The input gate ultimately adds to the cell state. This gate doesn't have direct influence over the cell state (it can't make it 0, like the forget gate can), but it can add to it and influence it that way. So it is an input gate.

The output gate is used to interpret the hidden state, and get it ready to be combined with the cell state for a final output at that time step.

While these gates all use sigmoid functions, are all initialised randomly and have the same dimensionality, what their output is used in, and the order they're computed in gives them a certain role to play. Initially, they won't conform to this role, but logically as they learn, they likely will.


Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.

  From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning

On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result in the identifiability issue. That is the network might simply learn $V(s)=0$, $A(s,a)=Q(s,a)$ for every state.

To tackle with this issue, we should force an additional constraint on the advantage estimate. We can simply use the equation below as mentioned in the paper, that is, normalize the advantages across actions before combining with the state value.

$$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\frac{1}{|A|}\sum\limits_{a'}A(s,a';\theta,\alpha))$$


What is the difference between imitation learning and classification done by experts?

Imitation learning is supervised learning applied to the RL setting.

In any general RL algorithm (such as Q-learning), the learning is done on the basis of the reward function. However, consider a scenario where you have available the optimal policy in the form of a table, mapping each state to each action. In this scenario you will not care about the rewards received - rather, you'd simply do a table lookup to decide the optimal action.

This scenario is impractical in most settings because the table for the optimal policy will be too big. However, if you have enough entries from the table, you can use a general function approximator such as a neural network to find the optimal action. Again, you do not need to look at the rewards, but only at the state $\rightarrow$ action mappings. I do not know imitation learning in detail beyond this, but I suspect in the case of discrete actions (such as in Chess, Go), it would be trained with a cross-entropy objective as is typical of classification tasks.

I suspect that the reason this has a different name in the RL setting is because this is different than how the conventional RL algorithms work. Also, much of RL thinking is inspired from everyday interaction / biology. Think of how we learn how to drive, or play sports such as soccer. Typically there is a coach who tells you what actions to take under different conditions, and you imitate those actions.


How to solve the problem of too big activations when using genetic algorithms to train neural networks?

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones.

Your weights should be both positive and negative.

In addition, once trained, they tend to follow a certain size distribution. It helps if you start with values within that range. This is often called Xavier or Glorot initialisation. Basically, if your number of inputs to a layer is n_inputs and number of outputs is n_outputs, you will have n_inputs * n_outputs weights, and if you initialise them with a random number generator, then you should use a multiplier something like this:

w = (rand() - 0.5) * sqrt(6/(n_inputs + n_outputs))

Note that is an addition inside the square root, you don't take the total number of weights.

As you are using a GA, you may want to use some form of clipping or normalisation to prevent mutations from drifting too far away from these useful weights. I'm not entirely sure what would be best, as usually I would use back propagation to train a large network. Potentially a maxnorm regularisation would help - choose a value (maybe even make it adjustable via mutation) that the norm of weight matrix in each layer should not exceed (separately for each layer, or one overall value, up to you), and if any learning step creates a network with too high a norm, scale it down. The L2 norm, sqrt(sum_of_squared_weights) is usual choice for maxnorm regularisation, and this works well with backpropagation-based supervised learning.


The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.
For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.


How to understand marginal loglikelihood objective function as loss function (explanation of an article)?

I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.
The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.
You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: https://www.deeplearningbook.org/
or here : https://en.wikipedia.org/wiki/Loss_functions_for_classification
I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in A Practical Bayesian Framework for Backpropagation networks by McKay 1992.
It is explained very well in a video series by Hinton on youtube here:
https://www.youtube.com/watch?v=YcwZFNd3UvI
Hopefully these references help you I wish I could explain in greater detail.


Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Depends on perspective.
On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works such as

Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning
rate. Artificial Intelligence, 136(2):215 – 250, 2002.

Michael Bowling. Convergence and no-regret in multiagent learning. In Proceedings of
the 17th International Conference on Neural Information Processing Systems, NIPS’04,
pages 209–216, Cambridge, MA, USA, 2004. MIT Press.

M. D. Awheda and H. M. Schwartz. Exponential moving average q-learning algorithm. In
2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning
(ADPRL), pages 31–38, April 2013.

Sherief Abdallah and Victor Lesser. A multiagent reinforcement learning algorithm with
non-linear dynamics. Journal of Artificial Intelligence Research, 33:521–549, 2008.

However, you can also claim that you simply have a single agent learning on a non-stationary environment (the environment contains both the game rules and the opponent), and you simply learn on that basis. From this perspective, there is no multi-agent learning at all.


How are afterstate value functions mathematically defined?

Based on this and this resources, let me give an answer to my own question, but, essentially, I will just rewrite the contents of the first resource here, for reproducibility, with some minor changes to the notation (to be consistent with Sutton & Barto's book, 2nd edition). Note that I am not fully sure if this formulation is universal (i.e. maybe there are other ways of formulating it), but the contents of the first resource seem to be consistent with the contents in the second resource.
Setup
Let's assume that we have an infinite-horizon MDP
$$\mathcal{M} = (\mathcal{S}, \mathcal{Y}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma),$$
where

$\mathcal{S}$ is the set of states
$\mathcal{Y} \subseteq \mathcal{S}$ is the set of afterstates (aka post-decision states or "end of period" states [1], which can also be written as after-states)
$\mathcal{A}$ is the set of actions
$\mathcal{T}$ is the transition function
$\mathcal{R}$ is the reward function
$\gamma$ is a discount factor

Let

$y \in \mathcal{Y}$ be an afterstate
$f: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{Y}$ be a deterministic function (from state-action pairs to afterstates), so we have $f(s, a) = y$

The transition function $\mathcal{T}$ for $\mathcal{M}$ is defined as
\begin{align}
\mathcal{T}(s, a, s^{\prime})
&\doteq P ( s^{\prime} \mid f(s, a)) \\
&= P ( s^{\prime} \mid y)
\end{align}
A transition is composed of 2 steps

a deterministic step, where we apply the deterministic function $f(s, a) = y$, which depends on an action $a$ taken in the state $s$, followed by
a stochastic step, where we apply the probability distribution $P (s^{\prime} \mid y)$, which does not depend on the action $a$ anymore, but only on $y$

So, I have denoted afterstates with a different letter, $y$, because afterstates are reached with a deterministic function $f$, while other states, $s$ or $s'$, are reached with $P$.
After having taken the action $a$ in the state $s$, we get a reward (i.e. we get a reward in step 1), but we do not get a reward after the stochastic step (given that no action is taken).
So, we can define the reward function $\mathcal{R}$ for this MDP as follows
$$
\mathcal{R} (s, a, s^{\prime} ) \doteq \mathcal{R}(s, a)
$$
The situation is illustrated by the following diagram

So, here, $P$ is the stochastic transition function (i.e. a probability distribution) as used above. Note that, here, $r_t$ is a specific realization of $R_t$ (the random variable) in the formulas below.
State value function
Let's recall the definition of the state value function $v_\pi(s)$ for a given policy $\pi$ (as defined in Sutton & Barto, section 3.5)
\begin{align}
v_{\pi}(s)
&\doteq
\mathbb{E}_{\pi}\left[G_{t} \mid S_{t}=s\right] \\
&=
\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k} R_{t+k+1} \mid S_{t}=s\right],
\end{align}
for all $s \in \mathcal{S}$ and
\begin{align}
G_{t}
&\doteq
\sum_{k=0}^{\infty} \gamma^{k} R_{t+k+1} \\
&=
R_{t+1} + \gamma R_{t+2} + \gamma^{2} R_{t+3}+ \cdots \\
&=
\mathcal{R}(s_t, a_t) + \gamma \mathcal{R}(s_{t+1}, a_{t+1})+\gamma^{2} \mathcal{R}(s_{t+2}, a_{t+2}) +\cdots,
\end{align}
where $\pi(s_t) = a_t$ and $\mathcal{R}(s_t, a_t) = R_{t+1}$, for $t=0, 1, 2, \dots$. (So, note that $\mathcal{R} \neq R_t$: the first is the reward function, while the second is a random variable that represents the reward received after having taken action $a_t$ in step $s_t$)
The optimal state value function is defined as
$$
v_{*}(s) \doteq \max _{\pi} v_{\pi}(s)
$$
Afterstate value function
Similarly, we will define the afterstate value function, but we will use the letter $w$ just to differentiate it from $v$ and $q$.
\begin{align}
w_{\pi}\left(y\right)
&\doteq
\mathbb{E}_{\pi}\left[G_{t+1} \mid Y_{t}=y\right]
\\
&=
\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k} R_{t+k+2} \mid Y_{t}=y\right]
\\
&=
\mathbb{E}_{\pi}\left[
R_{t+2} + \gamma R_{t+3}+\gamma^{2} R_{t+4} + \cdots
 \mid Y_{t}=y\right]
\\
&=
\mathbb{E}_{\pi}\left[
\mathcal{R}(s_{t+1}, a_{t+1})+\gamma \mathcal{R}(s_{t+2}, a_{t+2}) + \gamma^{2} \mathcal{R}(s_{t+3}, a_{t+3}) + \cdots
 \mid Y_{t}=y\right] ,
\end{align}
where $\mathcal{R}(s_{t+1}, a_{t+1}) = R_{t+2}$, for all $t$.
In other words, the value of an afterstate $y$ (at time step $t$, i.e. given $Y_t = y$) is defined as the expectation of the return starting from the state that you ended up in after the afterstate $y$.
This seems reasonable to me and is similar to my proposal for the definition of the afterstate value function in the question, although I was not considering any deterministic functions in a potential formulation, and I was also not thinking of afterstates as intermediate states, reached by a deterministic step, between the usual states.
Similarly to the optimal state value function, we also define the optimal afterstate value function
$$
w_{*}(y) \doteq \max _{\pi} w_{\pi}(y)
$$
Afterstate value function defined in terms of the state value function
We can define the afterstate value function in terms
$$
w_{*}(y)
=
\sum_{s^{\prime}} P (s^{\prime} \mid y ) v_{*} ( s^{\prime} )
$$
In other words, $w_{*}(y)$ is defined as an expectation over the value of next possible states $s'$ from the afterstate $y$.
This seems to be correct and consistent with the above definitions.
More equations
In this and this resources, the state value function is also defined in terms of afterstate value function as follows
$$v_{*}(s)=\max_{a}\left(\mathcal{R}(s, a)+\gamma w_{*}(f(s, a))\right)$$
The Bellman equation for afterstate value function (from which an update rule can be derived) is given by
$$
w_{*}(y)
=
\sum_{s^{\prime}}
P(s^{\prime} \mid y )
\max_{a} (
\mathcal{R} (s^{\prime}, a)
+
\gamma
w_{*}(f ( s^{\prime}, a ))),
$$
which is really similar to the Bellman equation for the state value function.
Finally, we can also express the state-action value function in terms of the afterstate value function
$$
q_\pi(s_t, a_t)
=
\mathcal{R}\left(s_{t}, a_{t}\right)+\gamma w_{\pi}\left(f\left(s_{t}, a_{t}\right)\right)
$$
Given that this answer is already quite long, see the resource for more details (including an algorithm based on the afterstate Bellman equation).
Implementation
If you are the type of person that understands the concepts by looking at the code, then this Github project, which implements a Monte Carlo method that uses afterstates to play tic-tac-toe, may be useful. Afterstates are useful in tic-tac-toe because it is a 2-player game, where two agents take actions in turn, so we can estimate the action that you should take deterministically (as if it was the $f$ above) before the other agent takes an action (probabilistically), at least, this is my current interpretation of the usefulness of afterstates in this game (and similar games/problems).


Extract features with CNN and pass as sequence to RNN

The approach that you don't train the whole net, but just the latter part of it (all starting with lstm in our case), can actually work. The idea is that the inception was already pretrained a very large dataset (imagenet for instance). And it's capable of extracting some useful information from it. Actually there are different domains of images in the imagenet and the inception net needed to capture a vast variety of input information to classify images well. The idea is that the pretrained inception is already capable to extract almost everything what could possibly be useful (unless your images aren't something completely different from imagenet, but that a rare case). Then you adapt the lstm layers and the fully connected layers to correctly process that information. Maybe you aren't going to get the perfect score with this approach and maybe it's better to train the whole large net including the inception part on the new data to lower the distributional shift and that's what people usually do in fact, but it takes more time to train and if you don't have enough data you won't be able to achieve results that are significantly better than those with a frozen CNN part.


Can autoencoders be used for supervised learning?

One such paper I know of and which I implemented is Semi-Supervised Learning using Ladder Networks . I quote here their description of the model:

  Our approach follows Valpola (2015), who proposed a Ladder network where the auxiliary task is
  to denoise representations at every level of the model.  The model structure is an autoencoder with
  skip connections from the encoder to decoder and the learning task is similar to that in denoising
  autoencoders but applied to every layer, not just the inputs. The skip connections relieve the pressure
  to represent details in the higher layers of the model because,  through the skip connections, the decoder can recover any details discarded by the encoder.

For further explanations on the architecture check Deconstructing the Ladder Network Architecture by Yoshua Bengio.


Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Yes, you can also automate the choice of certain hyperparameters of the evolutionary algorithm. In this context, this process is called self-adaptation. There are different ways of performing self-adaptation (depending on the hyper-parameter that needs to self-adapt). See e.g. the chapter Self-Adaptation in Evolutionary Algorithms (by Silja Meyer-Nieberg and Hans-Georg Beyer) of the book Parameter Setting in Evolutionary Algorithms (2007).


Is there a system that automatically designs a set of rules from the examples?

This is an active research topic.
Consider reading

J.Pitrat's Artificial Beings, the conscience of a conscious machine book

This Understanding machine learning book

old Lenat's papers on Eurisko.

and recent papers such as Learning Structured Embeddings of Knowledge Bases and several others in Artificial Intelligence Journal

about the MILEPOST GCC and Ctuning projects

this paper on Reconciling deep learning with symbolic artificial intelligence: representing objects and relations

My hope is to get funded so that RefPerSys could (in a few years) be able to deduce some rules and metarules.


What is the intuition behind TD($\lambda$)?

TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.
More precisely, TD($\lambda$) is temporal-difference learning with a $\lambda$-return, which is defined as an average of all $n$-step returns, for all $n$, where an $n$-step return is the target used to update the estimate of the value function that contains $n$ future rewards (plus an estimate of the value function of the state $n$ steps in the future). For example, TD(0) (e.g. Q-learning is usually presented as a TD(0) method) uses a $1$-step return, that is, it uses one future reward (plus an estimate of the value of the next state) to compute the target. The letter $\lambda$ actually refers to a parameter used in this context to weigh the combination of TD and MC methods. There are actually two different perspectives of TD($\lambda$), the forward view and the backward view (eligibility traces).
The blog post Reinforcement Learning: Eligibility Traces and TD(lambda) gives a quite intuitive overview of TD($\lambda$), and, for more details, read the related chapter of the book Reinforcement Learning: An Introduction.


How do I identify the number and type of objects in the same picture?

Your problem might involve a combination of object recognition (or object classification), which is the problem of determining which objects are in an image, and object detection (which could also be called object localization), which is the task of locating a specific object in the image.

A naive approach to solve your problem would then be to first locate the position of all possible objects in the image (object detection), then, for each these positions, perform object recognition (or classification). Then you would count the number of detected and classified objects for each possible class.

There are several models that can be used to solve your problem.

The YOLO (You Only Look Once) model (introduced in the paper You Only Look Once: Unified, Real-Time Object Detection) (2016) can be used to detect multiple objects in an image. See the official YOLO implementation, which is part of the Darknet framework. Have also a look at this article, which explains how to use the official implementation of YOLO.

You could also use the SSD model (introduced in the paper Single Shot MultiBox Detector). There is also R-CNN (Regions with CNN features), Fast R-CNN or Faster R-CNN, but these should perform worse than both YOLO and SSD.

So, even if you do not care about the locations, YOLO (and other mentioned methods) can give them to you, in addition to the number and type of objects in the image.


Deep Q-Learning poor convergence on Stochastic Environment

The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The inputs contain all information that is necessary.

However, the problem certainly becomes much easier (probably too easy) if the initial problem is always the same. The DQN algorithm will be very likely to "pick up" on this trend and "exploit" it. The inputs may be sufficient to learn a more general Minesweeper strategy, but if it is consistently faced with the same "level" every single time, it will be much easier for the DQN algorithm to just memorize exactly where the mines are and play perfectly based on that memory, rather than any actual strategy. Due to the way learning is implemented (based on gradient descent), the algorithm will generally tend to converge to such an easily-reachable "memorization" strategy rather than something that is actually "smart".

For that reason, I do think training will be much slower in the case where the layout is randomized. I'm not just thinking along the lines of e.g. twice as slow here, but would expect multiple orders of magnitude more experience required for successful learning. That's just my educated guess though, never tried training DQN for minesweeper specifically so can't tell for sure. It might also require more elaborate hyperparameter tuning, and maybe require a different network architecture (probably require a larger network).


Learning Rate Decay and Exploration Rate Decay

Welcome to AI.SE !
First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay : they play at the same scale (the number of successives batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specially at the same number.

For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, an you don't want the LR to disappear to brutally.
However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if you ER is initially low), and also to the "learning speed" of your model : if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper here, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.


How can we create an AI to develop mobile apps?

We don't know how to do that yet. The problem is one of scale:

Despite many years of research into program synthesis via heuristic methods, it's still not possible to automatically create programs (e.g. via Genetic Programming (GP), Grammatical Evolution (GE) or Learning Classifier Systems (LCS)) that are thousands of lines long, whether that's for mobile or any other application area.

Contrary to popular belief, alternative formal methods approaches can indeed be used to create sizeable programs, but the kind of interaction that a mobile app would typically require is not easily specified in this way.

The scale at which heuristic approaches are currently viable is closer to the scale of expressions (e.g. single program statements) than entire programs. An intermediate approach is therefore to provide a program template and let GP etc generate the missing parts of the template.

This paper describes how to combine Machine Learning with the 'Template Method' Design Pattern in order to create larger programs than would otherwise be possible, giving the specific example of a 'hyper-quicksort'.


Cold start collaborative filtering with NLP

From what I understood you will not have any cold start problem because you basically process the user preferences description against movies descriptions to get recommendations. So you don't use other users feedback at any time of the process which is not collaborative filtering.

  Here is instead the approach I would suggest in your case to get movies recommendations for each user:

Compute the similarity between the user description and each movie description. This can be done using the Universal Sentence Encoder. It's an 2018 paper by Google which represents any sentence as a vector of 215 values (i.e embeddings).
The semantic similarity between 2 sentences is computed using the dot product of their embeddings. Fortunately, the implementation was integrated to Tensorflow Hub and easily be used (see my answer here for details).
Choose the highest similarity values and recommend the corresponding movies to the user. Notice that you can still use this approach along with a collaborative filtering one.


Bag-of-Words (BoW) model in image detection

Here is an illustration of the entire process without any equation so you can get the big picture.

Features are extracted from the image. Let's take the example of very common features like SIFT. For many key points (or even each pixel) of the image, a 128-dimensional SIFT feature is computed. If processing numerous images the number of features becomes very big.

A way of having a more compact representation of the set of images is to use the Bag-of-Words (or Bag-of-Visual-Words) technique. The idea is to find k words (i.e., k SIFT features) from which every single image will be represented. We call this set of k words, the dictionary.

Then, each SIFT feature will be assigned to the nearest word (i.e., nearest SIFT feature with respect to the Euclidean distance for instance) of the dictionary. You can see this as a dictionary in which the words "go", "going", and "gone" would all be represented by the word "go").

In the end, each image is represented by only k values (counts for the number of words/features assigned to each word of the dictionary). This is an histogram, and you can normalize it to get a single vector of proportions representing the image.


Is it possible to use k-nearest neighbour for classification with more than two attributes?

The number of features is not important to use K-NN algotihm. You have to decide distance measure to detect neighbors. I share with you some links that you can check to see which kinds of distance measures that you can use. Just decide the meause and use your feature vectors in the measure.
https://www.kdnuggets.com/2020/11/most-popular-distance-metrics-knn.html
https://medium.com/@luigi.fiori.lf0303/distance-metrics-and-k-nearest-neighbor-knn-1b840969c0f4


Is there a way of pre-determining whether a CNN model will perform better than another?

I would like you to try with following changes in the model.

Introduce batch normalization layer in the model.
Try with the batch size of 32 - 64
Use different architecture, Like VGG, Resnet, etc..

There is no full proof answer to this question but you can get best by trying some know strategies.


What is the most general definition of "intelligence"?

I'm going to preface this answer by noting that persons much smarter than myself have treated this subject in some detail.  That said, as far as I can discern:

When we talk about intelligence we're referring to problem solving strength in relation to a problem, relative to the strength of other intelligences.

This is a somewhat game-theoretic conception, related to rationality and the concept of the rational agent.  Regarding intelligence in this manner may be unavoidable. Specifically, we could define intelligence as the ability to understand a problem or solution or abstract concepts, but we can't validate that understanding without testing it.  (For instance, I might believe I grasp a mathematical technique, but the only way to determine if that belief is real or illusory is to utilize that technique and evaluate the results.)
The reason games like Chess and Go have been used as milestones, aside from longstanding human interest in the games, is that they provide models with simple, fully definable parameters, and, in the case of Go at least, have complexity akin to nature, by which I mean unsolvable/intractable. (Compare to strength at Tic-Tac-Toe, which is trivially solved.)
However, we should consider a point made in this concise answer to a question involving the Turing Test:

"...is [intelligence] defined purely by behaviour in an environment, or by the mechanisms that arrive at that behaviour?"

This is important because Google just gave control over data center cooling to an AI.  Here it is clearly the mechanism itself that demonstrates utility, but if we call that mechanism intelligent, for intelligence to have meaning, we still have to contend with "intelligent how?"  (In what way is it intelligent?) If we want to know "how intelligent?" (its degree of utility) we still have to evaluate its performance in relation to the performance of other mechanisms.
(In the case of the automata controlling the air conditioning at Google, we can say that it is more intelligent than the prior control system, and by how much.)
Because we're starting to talk about more "generalized intelligence", defined here as mechanisms that can be applied to a set of problems, (I include minimax as a form of "axiomatic intelligence" and machine learning as a form "adaptive intelligence"), it may be worthwhile to expand and clarify the definition:

Intelligence is the problem solving strength of a mechanism in relation to a problem or a set of problems, relative to the strength of other mechanisms.

or, if we wanted to be pithy:

Intelligence is as intelligence does (and how well.)


What is the difference between neural networks and other ways of curve fitting?

In some sense, you're right that a neural net is just another tool to fit data. However, it's quite the tool! There's this universal approximation theorem saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.

The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a couple of matrix products (the $\hat{\beta}=(X^TX)^{-1}X^Ty$).


Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

This is very difficult to tell with the information provided, but the phenomenon  is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations:

Data from the training set could be identical or leaking in to the validation set.
Using a high dropout rate can cause this as well as other generalization techniques.
The training set may contain outliers, contributing to higher loss and difficulty in learning.

Also, note that the there is not so much an issue with the training loss bouncing around as long as it decreases over time. You may want to try different optimizers as well to see if that changes this behavior.


Human Height estimation using person detection techniques

Eh, "I won't be using camera calibration." .. Not sure, what you mean.

1st

At first, imagine a sheet of paper laying there on the floor. And first to try to transform the paper sheet (i.e. with some text on it), the picture of the sheet angled by paralaxe to the straight view: Just a transformation by a matrix. It would be highly valuable to have there some floor of square tiles. And once you would estimate/set size of the tiles on the floor, you can then transform the paper sheet.

Then, also some vertical calibration will be needed: Wherever within the scene. A wall of tiles of known shape & size would be perfect. But not really necessary: Any known/premeasured chair would be enough! ..rather two such chairs, in an opposite nooks, for numerical stability.

So yes, that could be treated as "camera calibration".

2nd

Then you can advance to the next level of the image processing:

Once you see feet of the person, on which tile of the floor the person stays,
and with the known vertical direction,

by the hid/covered known background, you can decide their height, pretty precisely, yes.

The "known background"

Just two images should be enough, of the background place: empty non-occupied, and hid/covered. The clear-empty room can be taken either onence, or periodically, or even just a partial image should be enough: just to detect shape of the person-figure.

3rd

And the third level: Really high. The person could stay in some deformed position. By interpolation of the known human skeleton to the shape of the person-figure on the screen, you can assume the curves of the skeleton of the person, so your estimated/measured total height of the person would be immune to the non-strightened angles in joints of their limbs/backbone. So even if the person knows about your observing, the AI would not get fooled. But you would need/find more refencial points on their body, to interconnect the 2D shape to the internal 3D model.

quantity of snaps in general

More snaps of the person, as they walk through the room, would be helpful: Any model, set up by any single image, should be verifiable/applicable to each of the rest of the images of the same person. Again: stability and "back testing", verifications, robustness.

You are not limited to just a single image, you can have as many as your camera is able to make. So use them, to get not only the measured size, but also to level up the "probability" of the measured result, the accuracy.

On the other hand, you can also flip the coin, and to try to guess the maximum info from the minimum snaps of the scene. But it does not seem, this would be your case. Still: Processing 24 fps recording 20 seconds long.. That definitely requires some power/time to process.

Summary

The 3D models are necessary: Of the surrounding space, of the outer shapes of the person-figure, of the internal structure of the measured body.


Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

Is something like this already out there? What is it called?

There are three algorithms that have similarities to yours, but are not identical:

NEAT, which is an evolutionary method that mutates neural networks and compares their fitness across populations of variations.
The Cross Entropy Method, which trains a single, initially randomly behaving, neural network based on the outcomes of its best performances.
REINFORCE is a policy-gradient method that trains a neural network after completing games based on the results of every game. It is very similar to your idea, except that it includes the maths/logic to sample a full gradient of all NN parameters in a single go. It does not require making a test change to a single parameter, but can calculate the full error gradient at once. Usually, however, to reduce noise and effects of correlation within a game, multiple gradients are collected from different games and averaged.

The main similarity in each case is that you are performing a policy search, and training based on selecting outcomes that are closer to your goal after trials. A policy-based approach means that your NN represents the choice of action directly. A policy is a function that converts knowledge of the state of an environment into an action.

  If nothing like this is out there, could you give me any tips to optimize this method or point out any key points I should keep in minds while carrying this out?

Your suggested method is not exactly the same as any of the suggested algorithms, and may have a problem with performance due to this:

  The untrained player starts off with a randomized algorithm, then (for back-propagation) at the end of each session it modifies one of the parameters by a bit in the neural network, and a new session is started with the slightly modified NN. If this session ends with more points than the previous session, it keeps the changes and makes more changes towards that direction, otherwise, the changes are redone, or possibly reversed. I would visualise this as gradient descent similar to that of Supervised learning.

This single parameter change search is likely to be slow, probably unusably so for any reasonably-sized neural network. Effectively you would be performing a single parameter step for a very noisy gradient descent algorithm, per full play of the game. All of NEAT, CEM and REINFORCE are likely to perform much better than this (and none of them are state-of-the-art).

  is there any way to make the learning faster while still keeping the basic idea in mind?

Instead of sampling the total reward at the end of each game and generating a single parameter's gradient, use a deep reinforcement learning method that works with full gradients.

You might find REINFORCE interesting, as it logically the idea closest to yours, and it leads to some state-of the art methods such as A3C, DDPG, PPO . . . the main problem with all of these being that the maths is hard to understand, even though the implementations can be simple you may be left scratching your head as to why they even work. However, if you are happy with just seeing the policy gradient theorem as some maths which completes your idea, then it is there for the taking.

Otherwise, if your goal is to implement something you understand the internals of fully, I would suggest that you search for a tutorial on deep Q learning, maybe something that implements DQN, as the maths and intuition behind it are more accessible.

  How exactly is this different from deep Q-learning (if it is)?

Both your suggested method and Q learning can probably be considered "deep reinforcement learning".

The main differences are:

Q learning is a value-based method, and yours is a policy-based method
Q learning is off-policy (it learns best overall actions even whilst exploring other possibilities) whilst your approach is on-policy (the actions it learns to take are the same as the actions it takes during exploration)
Deep Q learning updates all the parameters of the NN on each time step, whilst you will be updating a single parameter per episode. That makes Q learning far more sample efficient than your suggested approach.


Implementing SARSA for a 2-stage Markov Decision Process

In this game you can view end of an episode two ways:

There is an implied, terminal, fourth state $s_4$ representing the end of the game.

You could view the process as a continuous repeating one, where no matter what the choice is made in $s_2$ or $s_3$, the following state is $s_1$.

The first, terminating, view is a simpler and entirely natural view since nothing that the agent does in one episode can influence the next. It will result in a Q table that predicts future rewards within a single episode for the current agent (as opposed to discounted view over multiple episodes).
You are over-complicating things for yourself by ignoring that a zero reward is still a reward (of $0$). There is no need to remove $R$ from your initial update rule. In many environments there are rewards collected before the end of an episode.
In addition, to complete the standard episodic view, you can note that $Q(s_4, \cdot) = 0$ always by definition, hence so does $\text{max}_{a'}[Q(s_4, a'] = 0$. It is common here though to have a branch based on detecting a terminal state, and use a different update rule:
$$Q_{t+1}(S_3,a_{23}) = Q_t(S_3,a_{23}) + \alpha\left(R - Q_t(s_3,a_{23}) \right)$$
In brief, most implementations of TD algorithms do this:

Always assume a reward on each time step, which can be set to $0$

Special case for end of episode with a simplified update rule, to avoid needing to store, look up or calculate the $0$ value associated with terminal states

When implementing the environment, it is common to have a step function that always returns reward, next state and whether or not it is terminal e.g.
reward, next_state, done = env.step(action)

Details may vary around this. If you are working with an environment that does not have such a function (many will not have an inherent reward), then it is common to implement a similar function as a convenient wrapper to the environment so that the agent code does not have to include calculations of what the reward should be or whether the state is terminal.


What are AI use cases for communication service providers?

AI could hold the key in automating and optimizing networks. On the subscriber side, ML and AI will assist telecom operators in profiling the subscribers. This will be achieved by analyzing network activity, conversion rate of offers and data usage trends.

Below are a few use cases and how they will transform the telecommunication sector. (Source H2o.ai blog https://www.h2o.ai/telecom/ )

Old generation telecom technologies.

Reactive Maintenance
Network optimization with human intervention
Centralized intelligence
Security attack repair
Backlogged customer tickets

Future generation AI based telecom technologies.

Predictive Maintenance
Self-optimizing network
Optimal network quality
Intelligence at the edge
Security attack prediction
Improved customer experience through customer service chat bots.
Speech and voice services for customer which allows users to explore media content by spoken word rather than remote control.
Predictive maintenance which is the ability to fix problems with telecom hardware such as cell towers, power lines e.t.c before they happen by detecting signals that usually lead to failure.


Can a sentence have different parse trees?

But, is that correct? Is it possible a sentence has more than one valid parse tree (e.g. constituency-based)?

The fact that a single sequence of words can be parsed in different ways depending on context (or "grounding") is a common basis of miscommunication, misunderstanding, innuendo and jokes.
One classic NLP-related "joke" (around longer than modern AI and NLP) is:

Time flies like an arrow.
Fruit flies like a banana.

There are actually several valid parse trees for even these simple sentences. Which ones come "naturally" will depend on context - anecdotally I only half got the joke when I was younger, because I did not know there were such things as fruit flies, so I was partly confused by literal (but still validly parsed, and somewhat funny) meaning that all fruit can fly about as well as a banana does.
Analysing these kinds of ambiguous sentences leads to the grounding problem - the fact that without some referent for symbols, a grammar is devoid of meaning, even if you know the rules and can construct valid sequences. For instance, the above joke works partly because the nature of time, when referred in a particular way (singular noun, not as a possession or property of another object), leads to a well-known metaphorical reading of the first sentence.
A statistical ML parser could get both sentences correct through training on many relevant examples (or trivially by including the examples themselves with correct parse trees). This has not solved the grounding problem, but may be of practical use for any machine required to handle natural language input and map it to some task.
I did check a while ago though, and most Parts Of Speech taggers in Pythons NLTK get both sentences wrong - I suspect because resolving sentences like those above and AI "getting language jokes" is not a high priority compared to more practical uses for chatbots/summarisers, etc.


The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust its actions to get the most points and therefore the most reward. This is called reinforcement learing. The reward motivates the AI to adapt its actions, so to speak.

In a more technical term, the AI wants to maximize utility, which depends on the implemented utility function. In the case of DQN, this would be maximizing the score in the game.

The human brain functions in a similar fashion, although a little more complex and often not as straight forward. We as humans usually try to adjust our actions to produce a high output of dopamine and serotonin. This is in a way similar to the reward used to control AIs during reinforcement learning. The human brain learns which actions produce the most amount of those substances and finds strategies to maximize the output. This is, of course, a simplification of this complex process, but you get the picture.

When you talk about motivation, please don't mix it up with consciousness or qualia. Those are not required for motivation at all. If you want to discuss consciousness and qualia in AI, that's a totally different ball game.

A child isn't curious for the sake of curiosity. It gets positive reinforcement when exploring because the utility function of the child's brain rewards exploration by releasing rewarding neurotransmitters. So the mechanism is the same. Applying this to AI means defining a utility function that rewards new experiences. There is no inner drive without some kind of reinforcing reward.


Why am I getting the logarithm of the probability bigger than zero when using Neural Spline Flows?

I have always seen the negative odd-logs ration when doing similar work. As it is now, I don't think you are constraining the splines appropriately Link to reading


Is a multi-layer Kohonen network possible?

Kohonen networks by definition are single layer FCNN's, but what differentiates them from others is their unsupervised training procedure.

This procedure is a function of the input, the weights and some hyperparameters. This means if you have a multilayer network you could train only the final layers weights using this procedure. Think of it this way, let $f$ be the final layer, and $g$ be the composition of all other layers, such that the whole network $N(x) = f \circ g(x)$. Using kohonens training procedure, you could learn $f$'s weights where the input is $g(x)$ rather than x. But then how would you learn $g$'s weights?

So you could do this by iterative learning. let $N$ be a 2 layer network: $N = f_2 \circ f_1$. first learn $f_1$ using the single layer procedure, and now you could learn $f_2$ in the same manner, except the input features its trying to cluster would be $f_1(x)$ rather than $x$.

There does exist cons to this procedure though: Using a kohonen network at each layer will try to cluster it to the best of its abilities, not so that its final composition will be best (a common problem in many optimization procedures that arent end-to-end), but that its current representation is. This may lead to non-optimal results, not achieving that deep representation that you're looking for that you could achieve in autoencoders or other deeper unsupervised models.


What do the words "coarse" and "fine" mean in the context of computer vision?

tl;dr

  What does that mean in the context of this paper?

With "coarse segmentation" the author means a segmentation that doesn't have much detail. "Fine segmentation", on the other hand, refers to a segmentation with a high level of detail.

  But also more importantly [what does that mean in the context of] general computer vision?

The most common use in CV is to describe how general or specific a class in is classification. A "coarse class" is a very broad one, while a "fine class" is a very specific one.

Intended use

What the author refers to is the level of detail of the resulting segmentation.

A coarse segmentation would mean that we have large blobs covering each class without much detail. On the other hand, a fine segmentation would have a much higher level of detail which can even go down to pixel level (i.e. pixel-by-pixel correct segmentation).

To make this clear look at the following two examples. As we go from the left to right, the segmentation maps go from coarse to fine:

Note that in the rightmost images the result an almost pixel-perfect (i.e. fine details) segmentation map, while the ones on the left don't have much detail and can be considered coarse.

Alternative use

Because this isn't an established terminology, some of the times coarse and fine can refer to the nature of the classes in a classification task. Take the top image for example; the label for a coarse classification task could be a tree. For a fine classification task we would have labels like oak tree, pine tree, etc.

The most prominent example of this is the cifar dataset which has two versions: a coarse one that has 10 classes and a fine one that has 100 classes, which are all subclasses of the coarse classes. For example a coarse class is fish, while the fine ones are aquarium fish, flatfish, ray, shark, trout, etc.

For semantic segmentation an example could be the following: you want to make a street segmentation model. This a coarse segmentation would mean that it just splits the image into road, vehicle, etc. A fine segmentation, on the other hand, could also detect the type of vehicle, e.g. truck, car, etc.


Who first coined the term Artificial Intelligence?

John McCarthy (1927 - 2011) was an American computer scientist. A pioneer in the foundations of artificial intelligence research, he coined the term "artificial intelligence". He was one of the creators of the (original) Lisp programming language, which was quite involved in early AI research in the 1960s and 1970s.

He coined the term in 1955, and organized the first Artificial Intelligence conference in 1956, while working as a math teacher at Dartmouth. He founded the AI labs at MIT and Stanford.

He's responsible for developing several other important concepts in today's mainstream computer science. Namely, he developed garbage collection (used by a Lisp interpreter) and designed the first time-sharing systems.

On a side note, McCarthy predicted that creating a truly intelligent machine would require 1.8 Einsteins and one-tenth the resources of the Manhattan Project


How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof.
It is possible for the functions to resolve to the same value in a specific MDP, if, for instance, you use $R(s, a, s')$ and the value returned only depends on $s$, then $R(s, a, s') = R(s)$. This is not true in general, but as the reward functions are often under your control, it can be the case quite often.
For instance, in scenarios where the agent's goal is to reach some pre-defined state, as in the grid world example from the video, then there is no difference between $R(s, a, s')$ or $R(s)$. Given that is the case, for those example problems you may as well use $R(s)$, as it simplifies the expressions that you need to calculate for algorithms like Q-learning.
I think the lecturer did not mean "equivalent" in the mathematical sense, but in the sense that future lectures will use one of the functions, and a lot of what you will learn is going to be much the same as if you had used a different reward function.

Finally, when should we use one representation over the other and why are there three representations?

Typically, I don't use any of those representations by default. I tend to use Sutton & Barto's $p(s', r|s, a)$ notation for combined state transitions and rewards. That expression returns probability of transitioning to state $s'$ and receiving reward $r$ when starting in state $s$ and taking action $a$. For discrete actions, you can re-write the expectation of the different functions $R$ in terms of this function as follows:
$$\mathbb{E}[R(s)] = \sum_{a \in \mathcal{A}(s)}\sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)\qquad*$$
$$\mathbb{E}[R(s,a)] = \sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)$$
$$\mathbb{E}[R(s,a, s')] = \sum_{r \in {R}}r\frac{p(s', r|s, a)}{p(s'|s, a)}$$
I think this is one way to see how the functions in the video are closely related.
Which one would you use? Depends on what you are doing. If you want to simplify an equation or code, then use the simplest version of the reward function that fits with the reward scheme you set up for the goals of the problem. For instance, if there is one goal state to exit a maze, and an episode ends as soon as this happens, then you don't care how you got to that state or what the previous state was, and can use $R(s)$
In practice, what happens if you use a different reward function is that you need to pay attention to where it appears in things like the Bellman equation for theoretical treatments. When you get to implement model-free methods like Q-learning, $R(s)$ or its variants don't really appear except in the theory.

* This is not technically correct in all cases. The assumption I have made is that $R(s)$ is a reward granted at the point of leaving state $s$, and is independent of how the state is left and where the agent ends up next.
If this was a fixed reward for entering state $s$, regardless of how, then it could be written around $R(s')$ as follows:
$$\mathbb{E}[R(s')] = \sum_{s \in \mathcal{S}}\sum_{a \in \mathcal{A}(s)}\sum_{r \in {R}}rp(s', r|s, a)$$
i.e. by summing all the rewards that end up at $s'$


How do you implement NEAT by taking into account the loops?

NEAT does not employ a feed forward concept and also it does not take any special action to avoid loops.
Network is evaluated in non-recursive model. The only non-deterministic loop, the evaluation has is the loop for activating all the outputs. Pseudo code is something like this,
Until all the outputs are active
    for all non-sensor nodes
        activate node
        sum the input
    for all non-sensor and active nodes
        calculate the output

NOTE1: You can use a defensive mechanism (like a counter) to avoid a infinite loop
NOTE2: When summing the input, outputs of nodes which were at least evaluated/calculated once are considered, otherwise their outputs are assumed to be zero.
This is the note from the author of NEAT about identifying loops,

Note that checking networks for loops in general in not necessary and therefore I stopped writing this function
bool Network::integrity()


Why are Variational autoencoder's output is blurred while GANs output is crisp and has sharp edges?

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output.

If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get a high reconstruction quality, e.g. Spatial-Z-VAE


In which scenario would you want to have two adjacent pooling layers?

Assuming you're not referring to any particular type of pooling operation, it's possible that you could have, for example, a mean pool followed by a max or min pool. What this could do is combine the idea of reducing the dimensionality of your data from a holistic perspective with the mean pool, and then choosing the best of your averages with your max pool.


How to know if a real-time classifier is achivable in a low-power emdedded system?

Regarding your first point, it depends on what neural network you would like to use, the sensor temporal resolution, and the capabilities of the embedded system. You can figure out the number of operations required for a forward pass of your network, then when combined with the internal clock of the embedded system, you can calculate the approximate time it would take for one classification event in real time.
A good explanation is given here
What is the computational complexity of the forward pass of a convolutional neural network?
If you have the computational complexity of your network, and the number of CPU cycles per seconds, you can roughly approximate the time.


On what basis is MATLAB "inflexible" to perform ML/AI research on it?

(This question is could be considered off-topic or opinion-based, but I will answer it by providing facts that could hinder the adoption of MATLAB by AI researchers).
There are 2 main reasons why MATLAB may not be the "best" programming language/environment for research (in AI and other areas too)

It's closed source (i.e. it's more difficult to extend it; the most similar open-source alternative to MATLAB would be Octave)
It's owned by a company (i.e., it's proprietary, so, for you to use MATLAB, you need a license, which could be given to you by your university, which is my case, or you need to buy it, but not everyone can afford this).

In my view, these are clear obstacles for people that want to share their research and make sure that people can reproduce their results or build new ideas on top of them. However, in some cases, using proprietary software for research may be the best (or even only) option (see e.g. this for more info). In fact, I have also seen MATLAB being used for research in artificial intelligence (here is an example), but the choice of MATLAB might have been due to the familiarity of the authors with the language/environment.


How to implement an "unknown" class in multi-class classification with neural networks?

The usual way to implement this would be to add the new class with data examples.

Some things you need to address:

Sourcing new data for your "other" class.
Ensuring the amount and variation of data in "other" class examples matches how the predictor will be used.

Code examples for this are not necessary, as you would just use the same network design as you already have and just add another output. This is a data and model definition problem.

Logically you have another option: As well as outputting the predicted class, you predict separately whether there is any detectable object at all as a true/false value. This still requires the additional data, but is for example how the YOLO algorithm works for object detection. Object detection has a specific meaning - it involves finding the co-ordinates and class of possibly multiple objects in an image. This goes beyond the wording of your question, but is a typical end goal if you are asking this kind of question.

YOLO predicts the presence of an object separately from the class of object. The additional data for YOLO training comes from segmenting the source images, so many parts of the target image are background with no objects. In that case the additional data you require is due to more detailed labelling within each image example.

YOLO is quite complicated architecture, so you might want to look at this example using Keras on a Github project for more details, if object detection is your goal.


How can I create an embedding layer to convert words to a vector space from scratch?

Word2vec embedding are trained by simple auto-encoder model that takes a word and tries to predict one word form the window of surrounding words.

You could define it like this:
num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts next words. And it outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

It is trained both ways from beginning to end of sentence and reverse. Loss is a regular categorical_crossentropy Detailed explenation can be found here [http://jalammar.github.io/illustrated-word2vec/]