What should the target be when the neural network outputs multiple Q values in deep Q-learning?

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_2 + \gamma max_aQ(s',a',\theta) , Q(s_1,a_3),...$ ?

Other than what looks like a couple of small typos, then yes.

This is an implementation issue for DQN, where you have decided to create a function that outputs multiple Q functions at once. There is nothing about this in Q learning theory, so you need to figure out what will generate the correct error (and therefore gradients) for an update step.

You don't know the TD targets for actions that were not taken, and cannot make any update for them, so the gradients for these actions must be zero. One way to achieve that is to feed back the network's own output for those actions. This is common practice because you can use built-in functions from neural network libraries to handle minibatches*.

There are some details worth clarifying:

You have substituted the third entry in the array with the calculated TD target because the action from experience replay is $a_2$. In general you substitute for the action taken. Looks like you have this correct.
You have $r_1$ in your experience replay table, but put $r_2$ in your TD target formula. Looks like a typo. Another typo is that you maximise over $a$ but reference $a'$. Also, you reference $s'$ but don't define it anywhere. Fixing these issues gives $r_1 + \gamma \text{max}_{a'}Q(s_2,a',\theta)$
For the TD target it is often worth using a dedicated target network that every N steps is copied from the learning network. It helps with stability. This can be noted as a "frozen copy" of $\theta$ noted $\theta^-$, and the neural network approximate Q function often noted $\hat{q}$ giving formula of $r_1 + \gamma \text{max}_{a'}\hat{q}(s_2,a',\theta^-)$ for your example.

* If you want you can also calculate the gradient more directly from the single action that was taken, and back propagate from there, knowing that all the other outputs will have a gradient componnet of zero. That requires implementing at least some of the back propagation yourself.


Validation Loss Fluctuates then Decrease alongside Validation Accuracy Increases

This question has been asked a year ago when I faced this problem I searched and it hasn't any answers on it, I tried different ways and finally, data augmentation helped me.
I used data augmentation and a very small learning rate. If the fluctuations are big, the batch size should be increased, and the learning rate should be decreased. After all, using more epochs helps you to have an almost smooth plot.


How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?

What you want to do is called multi-task learning. Here's what you do:

Create a second Input.
Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
Put a dense layer of 2 after this.
Put softmax with units = number of classes.
Add CE loss calculated with this one and the ground labels to the loss of U-Net.

This is in regards to TensorFlow. The same can be done in PyTorch easily.


In a face database containing multiple images per subject, how do we determine the face image which is most suited for face recognition?

This should not matter that much as long as you do enough preprocessing?
But one could use e.g. the Facenet architecture to extract the embedding vector for each face of the same subject.
Then you could compute the covariance matrix and perform PCA on it. This would give you the most significant features with which you can decide which face to take.
Alternatively, you could do something in the direction of the eigenfaces: https://en.wikipedia.org/wiki/Eigenface.


How to handle images that don’t pertain to image classifier at all?

Yes, a category "no person" or "random image" would make sense. Binary classification is only helpful if you know that your input always belongs to one or the other category, for example by pre-filtering the inputs.


Why do we have to dot product in the Low-rank Bilinear Pooling?

The paper (which I was not familiar with) proposes a new pooling method, which they name "Low-rank Bi-linear Pooling". Because it is a pooling method, its output should be a single value over some input spatial region (the goal of pooling is to down-sample an input representation). They do not want to use the resulting vector from the element-wise multiplication, they want to get a single scalar. Notice their initial formula:
$$
f_{i}=\sum_{j=1}^{N} \sum_{k=1}^{M} w_{i j k} x_{j} y_{k}+b_{i}=\mathbf{x}^{T} \mathbf{W}_{i} \mathbf{y}+b_{i}\label{1}\tag{1}
$$


I read the second chapter in that book for my project on recommendation systems. It is an awesome book and it starts by addressing concepts from the basics( At least chapter 2 was like that). Book explains concepts with python. You will need a bit of knowledge on python basics . The book transits slowly from beginner to an intermediate level. Hope this answers your question. I would recommend it.


Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4).

Equation (3.4) says (leaving out the middle part):

$$p(s' \vert s, a) \doteq \sum_{r \in \mathcal{R}} p(s', r \vert s, a).$$

If you plug this into Equation (3.6) to substitute the denominator, you can't forget about that sum, you have to include the complete sum in the denominator. Because we have two different sums summing over all rewards in $\mathcal{R}$, I'll change the symbol used in the second sum to $r'$ rather than $r$. This yields:

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{\sum_{r' \in \mathcal{R}} p(s', r' \vert s, a)}.$$

The numerator and denominator are different and will not cancel out.

Intuitively, the numerator is just a single probability; the probability of observing a specific next state $s'$ and a reward $r$ given a specific current state $s$ and action $a$.

The denominator is a sum of many such probabilities, for all possible rewards $r'$ rather than just one specific reward $r$. Because this sum "covers" all possible events for the rewards, it essentially represents simply the probability of observing state $s'$  given state $s$ and action $a$, paired with any arbitrary reward. That can more simply be denoted as $p(s' \vert s, a)$.... which makes sense because that's the very thing we started with :D

Equation (3.6) is as follows (again leaving the middle part out):

$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

In normal English, the left-hand side says "what reward do we expect to get (on average) if we transition from state $s$ to state $s'$ by executing action $a$?"

Such an $\color{red}{\text{expectation of a quantity }r}$ can always be computed by multiplying $\color{blue}{\text{the possible values that }r\text{ can take}}$ by $\color{orange}{\text{the probability of each particular value occurring}}$, and summing up those multiplications. In math, this looks like:

$$\color{red}{r(s, a, s')} = \sum_{\color{blue}{r \in \mathcal{R}}} \color{blue}{r} \times \color{orange}{p(r \vert s, a, s')}.$$

Now, there is a rule in probability that says (see Rule of Multiplication on https://stattrek.com/probability/probability-rules.aspx):

$$p(A, B) = p(A) \times p(B \vert A),$$

so, if we take $A = s'$, $B = r$, and $s, a$ as additional givens for all probabilities, we get:

$$p(s', r \vert s, a) = p(s' \vert s, a) \times p(r \vert s, a, s').$$

This can be rewritten (dividing both sides of the equation by $p(s' \vert s, a)$ and swapping the left-hand and right-hand sides) to:

$$p(r \vert s, a, s') = \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$

Plugging this in for the orange term in the coloured equation above (which itself is hopefully fairly easy to understand intuitively) yields Equation (3.6).


How is it possible to teach a neural network to perform addition?

Welcome to AI.SE @bilanush.

Here's an example approach that might make things clearer. There are other ways to train a neural network to do this however.

In your earlier example with an image, you probably noticed that the network receives the image as a series of values, representing each pixel in the image. The network then learns which of a series of output neurons should be active in response to a given set of pixel values. Those output neurons, when read in an appropriate way, correspond to the correct label for the image. The difference between the set of outputs that should have been active, and the set that were active, forms the basis of the error signal that allows the network to learn.

You've probably heard that computers represent numbers with binary digits. So you could think of the number 16 as being: 00010000 in "8-bit binary". In 16-bit binary, this number would be 0000000000010000, and so on.

So one way of viewing your problem is a function mapping binary inputs to binary outputs (very similar to labelling a black-and-white image). For instance, the input 00010000 (16) should produce the output 00010001 (17). The input 00100011 (35) should produce the output 00100100 (36), and so on.

As before, you will have a set of output neurons. In this case, it should be wide as the set of input neurons. As before, the error signal is the difference between the expected inputs and outputs.

As to the question of how they can learn this function "without plus", in fact the individual neurons in a network perform just two operations: addition of their inputs, and a non-linear transformation of the sum. It has been proven that these are sufficient to learn any function from inputs to outputs, as long as the network contains 3 layers or more, and the as long as the middle layer is wide enough, but here it should be easy to see how addition might emerge.


What are the most challenging tasks aiming to achieve the lowest error rate?

Yes. Here are some of the most prominent ones and their respective state-of-the-art errors:

CIFAR-10: ~3.5% error
CIFAR-100: ~24% error
STL-10: ~26% error
SVHN: ~1.7% error
ImageNet tasks: the best 2012 classification task solution got 15% top-5 error, better results are currently available

You can check an updated list of solutions here. Also, a more comprehensive list of modern datasets can be found here.


How can I implement 2D CNN filter with channelwise-bound kernel weights?

Assuming you want HxWx1 kernel to perform convolution on hxwxc images.
Here's sample code which uses single channel kernel to operate on multichannel feature: maps
import torch
import torch.nn as nn
import torch.nn.functional as F

class model(nn.Module):
    def __init__(self, in_ch=4):
        super().__init__()
        self.in_ch  = in_ch
        # single channel kernel initialization
        self.kernel = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=1, stride=1)

    def forward(self, x):
        bs, ch, w, h = x.shape
        x   = x.view((bs, ch, 1, w, h))
        out = self.kernel(x[:, 0])
        # reusing of same kernel
        for i in range(1, self.in_ch):
            out = torch.cat((out, self.kernel(x[:, i])), 1)
        return out

net = model(4)
print(net)
inp = torch.randn((10, 4, 100, 100))
out = net(inp)
print(out.shape)

(The main hack is in the forward function)


Why studying machine learning is an opportunity in today's world?

What sort of opportunity it is depends on how much you want to focus on it.

If you want to be a regular programmer, you might take the time to learn a high level interface for some machine learning tools, such as Tensorflow or Keras. There will be plenty of things you don't know how to do (even within those tools), but you may be able to apply predesigned model architectures to problems. The models won't be as good as one designed specifically for the problem, but it's one more tool in your toolbox, and it's possible you'd be able to get some useful results occasionally without devoting a huge amount of time to mastering the techniques.

But if you want to really focus on machine learning, at the research level, you can potentially tackle problems that existing techniques haven't been able to solve. This is where most of the big projects that you've probably heard of will be happening: self-driving cars, AlphaGo, etc. What you can expect here is a lot of hard work. You will need to develop a fairly deep understanding of the mathematics involved so you can visualize (to some extent) what is happening in the potentially high dimensional spaces involved, identify potential failure modes, and identify models that won't fall into them. It involves a lot of trial and error, failed attempts and gradual improvement before you're able to develop a model that beats the stuff already out there.

It's very rewarding work if you enjoy it. There are well-paying positions in the field, but that's just a bonus if you already enjoy the work, and it isn't enough of a bonus if you don't. In my opinion, going into this for just the money would be a mistake. There are almost definitely other jobs that pay just as well but don't take anywhere near the time investment to become (and stay) competent at them. But if you really want to work in this field for its own sake, and also want to make sure you don't starve while you're doing it, it's absolutely worth it.


Algorithms can be greedy. What are some other algorithmic vices?

Algorithms can be racist, sexist, and otherwise bigoted. When we feed them data produced by systems that are biased against groups of people, the algorithm will learn to behave that way. We're used to garbage in garbage out, now we have to worry about racism in racism out.

See:
Facial Recognition Is Accurate, if You’re a White Guy
Rise of the Racist Robots – how AI is Learning all our Worst Impulses


What are some implications of Gödel's theorems on AI research?

Definitely there are a lot of implications for AI, including:

Inference with first-order-logic is semi-decidable. This is a big disappointment for all the folks that wanted to use logic as a primary AI tool.
Basic equivalence of two first-order logic statements is undecidable, which has implications for knowledge-based systems and databases. For example, optimisation of database queries is an undecidable problem because of this.
Equivalence of two context-free grammars is undecidable, which is a problem for formal linguistic approach toward language processing
When doing planning in AI, just finding a feasible plan is undecidable for some planning languages that are needed in practice.
When doing automatic program generation - we are faced with a bunch of decidability results, since any reasonable programming language is as powerful as a Turing machine.
Finally, all non-trivial questions about an expressive computing paradigm, such as Perti nets or cellular automata, are undecidable.


Is there an AI model with "certainty" built in?

You make a valid point, vanilla neural networks cannot give you more than a point estimate of class confidence. If one wanted to actually gain an idea of variance, you need a framework that allows such a mechanism.

A popular methodology to this is Bayesian modeling. In other words given some data, $\Omega$, you want to create some form of descriminative model $p(y|x;\theta)$ where $\theta$ denotes the parameters that are random variables. This is unline NN's where they also learn some descriminative model $p(y|x;\theta)$ where $\theta$ are fixed. This difference is key to your goal, because if $\theta$ is fixed, $Var(Y|X) = 0$ since any time you put in the same $X$ into the network, itll always come out the same. On the other hand a bayesian model will have a variance, $Var(Y|X) = E[(Y - E[Y|X])^2 | X]$ and $(Y - E[Y|X])^2 | X$  is no longer gauranteed to be 0 and can be empirically measured through some MC method (or analytically based on your model).

Note that your idea is on the right track, that the more data points you have, the more confident your estimates will be, but the only issue is that just considering $\frac{P}{(P+N)}$ in a neural network will be just a heuristic and is difficult to quantify its association to the variance.

I think this may fit your fancy: high level overview of bayesian network blog post. This is a high level overview of using a neural netowrk paradigm but including uncertainty in the weights. Theres tons of tricks out there to make it so you can still even train these with gradient descent and such.


How to add more than 1 agent in one generation with Q Learning

The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).

GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the best agent is the output of the algorithm*.

Whilst you can run muliple agents to collect more data easily enough in RL, they typically won't perform better or worse other than by random chance. The best performance is not indicative of the agent's overall performance in the way that GA is, because there is only one agent. It would not be as meaningful to pick out the best ones for display. Instead, after a certain number of training episodes, you should take the agent so far, stop it running exploratory moves (set $\epsilon = 0$ if you are using $\epsilon$-greedy exploration). Then render the behaviour of that agent.

If you want to compare RL versus GA for learning efficiency, then one arguably fair comparison would be to render the agent each time it has trained on a number of episodes in RL equal to the population size used in the GA version. If you are only showing results after every 100 generations of GA, then multiply number of training episodes by 100 to compare using the same amount of data input.

You can also, separately to this concern, run RL with multiple exploring agents at once. If you want to have parallel training with multiple environments running in RL, then you will need a distributed environment. Each instance of the environment would run one agent, and collect training data. You have a rough choice between:

Feeding everything observed into one central experience replay memory and a single training loop routine samples from the whole memory and updates the agent. This should work OK for Q-learning.
Calculate update gradients on each distributed environment, collate them centrally and update the agent on a mean gradient update step before sending out the updated agent to all the distributed systems. This is the approach typically used by A3C and A2C which benefit from having multiple agents running at once.

In both cases, the latest parameters of the agent (the neural network weights) need to be regularly copied out to each environment so that each instance can work as much as possible with a current policy.

Setting up a distributed learning environment for RL is more work than for GA, because you need to move a lot more data between agents to complete learning, whilst for GA you only need to measure fitness. However, you should find that Q learning can be a lot more efficient (in terms of number of simulations required) than a GA-based approach for many control problems.

* As an aside, this can be a weakness of GA if you are running in an environment with random choices or events - the GA can select as it's "best" result at any stage something that is less optimal but that was lucky. On average over many generations this effect should be removed, because the same agent won't be luck every generation, but it does mean in some environments that you will get an over-estimate of performance unless your fitness assessment is very thorough (or perhaps run separately every so many generations, to double-check). Similar concerns occur in RL too, but do not affect which agent you select for assessment or display, since there is only ever one set of agent parameters at any time.


If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

Is this a sign that the algorithm diverged?

It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.

Other common causes of algorithm failing with a fixed action choice include:

Neural network inputs not scaled before use.
Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)
State representation too far from Markov property assumption
A bug in code (almost anywhere, unfortunately)
Catastrophic forgetting due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.


Are there profitable hedge funds using AI?

High-frequency trading is where you see it being used, essentially, decision-making algorithms analyzing and making transactions in microseconds.  It accounts for a significant percentage of market activity, and has been considered a source of greater market volatility (See: Flash Crashes).

You can bet that hedge funds are evaluating every form of AI for trading and predicting market trends in general, but, unlike academics (and a segment of the tech sector) I doubt they consider it in their interests to publish methods which competitors could then utilize.

It's difficult to source information on exactly what is being looked into and utilized in the financial sector because there is a lot of marketing noise (exaggerated claims,
 unreliable sources) but I did find these articles:

Will AI-Powered Hedge Funds Outsmart the Market? (Tech Review; 2/4/2016)
"AI Hedge Fund Is Said to Liquidate After Less Than Two Years (Bloomberg; 9/7/2018)"

My gut tells me that Machine Learning methods will surpass humans in these kinds of decisions before too long (educated guess), and only increase in utility as the dataset these algorithms draw from grows, because it's ultimately a statistical problem.

AI is related to Game Theory, the study of economic decision-making, in the context of utility.  Game Theory might be said to have had its greatest success in computing in general, via minimax, but has traditionally been much harder to apply in real world economics.  Minimax can be utilized in machine learning, and it's actually hard to think of any economic decision-making that wouldn't utilize it in some form.


How do I use a taxonomy and the support vector machine for question classification?

This is not an answer (I don't have enough reputation to comment). I did something close to this in my master's thesis and think it is close to what you are interested in.

In it, I had developed a framework for extracting metadata from web-based educational content. This metadata was used for classifying the the educaitonal content for many different attributes, which could then be used for faster and more efficient search and discovery of educational content.

The educational resources (containing the content) could be anything like text or PDFs of assignments, homework, assignments, online books, exam questions, courses etc (which many colleges host online). To identify what kind of educational resource it is, I would parse the text and look for keywords and formatting styles (preprocessing included constructing 2-grams and 3-grams, POS tagging, using small specific parsers for NER, dates and other text entities one encounters in educational content).

For some part I used Wordnet (also available under python-nltk) to obtain relationships between different entities and also to find closeness between them. DBpedia was also used. However, for the most part I had to identify the most commonly occuring terms and build a taxonomy by hand. (It took a lot of time!). I obtained a lot of candidates for keywords by looking at openly available taxonomies.

For extracting domain specific taxonomy/ontology, one needs manually to build it. Ontology generation from text is an active area of research and building domain-specific ontology has been tried for many years. One example of such taxonomy (here thesaurus) is agrovoc where domain experts have contributed to the knowledge by identifying agricultural entities manually.

There are a lot of places where domain specific vocabulary is available; maybe you can use that. In some aspects it is close to supervised machine learning, where one has some nice data and correspondingly nice output. However, on my part, there wasn't much learning in it - more like template matching.

Hope this helps.


Is it possible to use a trained neural network to predict a feature, given other features and output?

From your question, it appears that you would like to use other features in your data to predict one of the features. I am not sure I understood your question clearly, but anyways, either, you would be using the feature you want to predict as the output of the network.

Also, if you want to use the output of the network and other features to predict the new output, I think possibly you are trying to use a Recurrent Neural Network based approach, in which the past output is taken into account for future predictions. My personal experience with RNN's is that they are really good at learning such dependencies, as the RNN cells consider the present input, as well as the past output in modelling such tasks. If your problem is a sequence to sequence prediction task, I would definitely suggest trying RNN's (especially LSTMs, ideal for learning long term temporal dependencies) but ultimately, there is No Free Lunch. So try different approaches and see which one works for you best. Good Luck!


What causes a model to require a low learning rate?

Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.

where alpha is learning rate

If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function.

hence result in a higher loss.

Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.

There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors

size of training data: as the size of the training data increases training time for model increases. If you want to go with less training model time you can choose a higher learning rate but may result in bad performance.
Optimizer(gradient descent) will be slow down whenever the gradient is small then it is better to go with a higher learning rate.

PS. It is always better to go with different rounds of gradient descent


Convolution layer neurons when extracting multiple feature maps

Each feature map (or kernel) is independent of each other. If you had $3$ of these filters, your output shape would be $(28, 28, 3)$ (given the appropriate amount of padding and stride) with a total of $75*3=225$ trainable weights.


When we have multiple traces, do we average over traces or the total number of times we have visited that state?

For every visit MC you create a list for each state. Every time you enter a state you calculate the returns for the episode and append these returns to a list. Once you have done this for all the episode you want to average over you simply calculate the value of a state to be the average of this list of returns for the state.
First visit MC is almost the same except that you only append the returns to the state returns list for the first time you visit the state in an episode.
Your workings are correct, you average over the number of times you have visited the state (in every visit MC). So, in your example, you would get the value of HOUSE to be 0, as you stated. If you were doing first visit MC then the returns for episode 1 would be 2 and the returns for episode 2 would be -2, which you would then average over the two first visits to again give you 0 (this would not always be the case that they are equal after 2 episodes but in the limit both methods do converge to the true state value function).


Why do neural networks trained on identical datasets and with identical hyper-parameters have different performances?

This should simply be due to the fact that network weight initialization is random.


Features Map convolutional neural network

Since the kernel size is limited. A feature can encompass more than 3 pixels or say 5 pixels, for that we cannot find it in 1 layer. So by adding 2 layers and having an activation function that is not like a step function which you would prefer, we extend the feature to 5 or 9 pixels. And with n layers, a feature covering 2n+1, or 4n+1 can be seen.

PS. You have some spelling errors. Take your time to frame well. Welcome!


Is it possible to train a neural network to learn something via video footage?

Short answer: No.

Long answer: A neural network is a function that maps input data (e.g. a picture) to output data (e.g. probability that the picture contains a dog). What you propose does not seem like the sort of task this tool is suited for, though I'm not willing to claim that it can't be done.

If you are interested in computer-generated artwork, I encourage you to learn about Generative Adversarial Networks and style transfer.


Tweaking a CNN for large number of input channels

As far as I know, more than 3 channel is perfectly fine, since, 3 channels are what we use for images and that's enough since we can only see this many colors, but I don't see why more than that wouldn't work

Your 2nd question is like asking whether or not you will be good at a sport... Just try it

For your 3rd question, I've never seen any language AI using CNN instead they all use RNN, not sure if that's what you meant though


Can neural network take decision about its own weights (update of weights)?

Establishing Names and Terms
The mathematician's name is Kurt Gödel, an Austrian-American, and the machine concept named after him is the Gödel machine.  The idea of meta-learning (learning how to learn) can be extended recursively to learning how to learn how to learn how to learn and so on.
Taking a decision is when the decision is external to the system and the system either copes or executes that decision, whereas making a decision is when the decision is internal to the system and the system executes it, reports it, or delegates it.
Both of these have been tried in the laboratory or in real systems and various levels of recursion have achieved success, but not indefinite recursion.
A Few Points from AI History
Creating a learn how to learn recursion was the object of several early LISP efforts at MIT, and meta-learning is in active research in multiple universities and corporations as of this writing.  The first successful application of a single level of recursion were the expert systems that could acquire knowledge using meta-rules, which were rules about creating new rules.
Current Artificial Networks Regarding Meta-learning
It is correct that back propagation is not meta-learning.  It is a single level of learning in the above recursive learning concept.  Artificial networks have been applied to learn to adjust the hyper-parameters of another network, but these strategies require large quantities of data, and they don't cognate.
They don't know how to adjust hyper-parameters.  They don't build models of learning and then use those models to learn.  The adjustment mechanism is the application of a learned function, not a conception of learning.
Requirements for Meta-learning
To establish a meta-learning paradigm, we must consider the elements of learning.

Existence of function abstraction
Ability to guess a parameterized function
Ability to mutate that function
Ability to tune the parameters
A target functional behavior
A way to detect whether the function is approaching the target behavior
A strategy for deciding when to mutate and when to tune
A way of performing experiments in isolation
A way to recurse in a way that incorporates ALL of the above

Once this has been achieved, it may be described as self-improvement and may qualify as what most theorists think of as a Gödel machine.
Specifics in the Question

Can [an artificial] network [make] decision about its own weights (update of weights) during training phase or during the phase of parallel training and inference?

Yes.

[Can] one [level] of [a] hierarchical [artificial network] [make] decision[s] about weights of other [levels]?

Yes, but in a limited way as of the time of this writing.

I am very keen to understand about self-awareness ...

Self-awareness is the ability of the system to analyze and either utilize the result of that analysis, report it, or delegate based on the result IN COMBINATION WITH the ability to use itself as the object of analysis.
This can be as simple as a program that parses Java and produces statistics using its own code as the code to parse.  It can be as complex as an anthropologist studying humanity.  It can be as personal as a human looking in the mirror and wondering about the qualities of person they are.  It can be as deep as someone wondering their purpose in their current life.
In all cases, there are those two qualities.

The ability to perform a type of analysis that could be applied toward a class of objects of which itself is a member
The selection of itself for analysis

There is one other aspect of awareness beyond these two that are not required but usually associated with awareness: Some way of directing attention of the analysis capability to itself regularly.  Here are a few examples from human experience.

Practice of daily meditation
Reading a book on purpose
Seeing a cognitive behavioral therapist
Writing an autobiography
The first, fourth, and tenth Step of anonymous programs
Watching a candid movie of one's self
A family member that likes to step back and evaluate the family
Religious accountability groups

Self learning is like self-awareness except that learning replaces awareness.  Obviously, self-learning is dependent on some degree of self-awareness.  We now have a more activity based list because of the additional element of adjustment or inception of action.

Decide, on the basis of personal abilities and interests, after evaluating options for living, to relocate
Realizing that self cannot remember commitments to meetings to buy a technology device that notifies the user before such commitments and using the calendar app rigorously
Finding that repeated attempts to eat greens rather than junk food is not working and joining a community of those with eating disorders with solutions to that issue
Deciding that childhood views of future were good ones and abandoning current paths for ones that were bolder and will likely lead to fulfillment

Self-improvement is a super-set of self-learning, since the only point of learning is improvement, however there are forms of improvement that don't strictly involve learning.  A system (or person) can execute something already learned that results in an improvement.

Artificial general intelligence (e.g. Gödel machine)

There is no mathematical proof that a Gödel machine would exhibit general artificial intelligence.  More importantly, there is no proof and a considerably body of evidence against the proposition that humans are generally intelligent.
Lastly, there is no proof that general intelligence is achievable in a biological or artificial system.  Gödel's second incompleteness theorem is strong evidence that such ideas may be naive.

[Artificial] networks are usually mentioned as examples of special, single-purpose intelligence, but I can not see the reason for such limitation if [they] essentially [try] to mimic human brains, at least in purpose if not in mechanics.

Artificial networks emerged out of a desire to mimic human brains, but the perceptron design is based on an old view of neuron functionality.  Simulating a single neuron might require an entire rack of CPUs and GPUs.
Furthermore, we are not yet clear on what intelligence is.  Definitions proposed vary widely.  The fields of artificial networks, bioinformatics, ontology of ideas, semantics, and the relationships between stable adaptivity and algorithms are all in their early stages, both theoretically and experimentally.

Maybe this desired [capability] is already effectively achieved [or currently emerging] in the operation of recurrent ANNs as the effect of collective behavior?

No, but RNNs are a tiny step closer to biological neurons in that their layers maintain state beyond the attenuation matrix.
By attenuation matrix is meant the matrix of parameters used in vector-matrix multiplication to control the strength of signals from the activation functions of one layer to the activation functions of the next.  It's common name in machine learning literature is simply, "The parameters."  Learning occurs as they change to converge the network on an optimal state.
RNNs are also capable of being Turing complete, so they can theoretically realize arbitrary algorithms.
However, the Requirements for Meta-learning above are not an expected capability of any single instance of an artificial networks, either of the MLP (multilayer perceptron) type or the RNN type.  Whether balancing two networks in symbiotic arrangement as in GANs, or a comprehensive recursive algorithm, or the simulation of neural and synaptic plasticity in silicon will lead to Gödel machines is unknown.
Let Science be Science
Whether ideas such as general intelligence are possible is unknown.  Whether ideas about singularities in books and mass media are realistic is unknown.  Whether the prophetic warnings of screenwriters about the emergence of artificial entities that become the dominant species on earth is unknown.
Jaques Ellul is perhaps the most scientifically accurate prophetic voice.  In his Techological Sociey, he presents heaps evidence in support of the idea that humans are already serving an autonomous technology and have been since prior to industrialization.
A Swiss philosopher, Francis Schaeffer, once prophecied, "They will say all kinds of things in the name of science that have nothing to do with science."  We should be careful to keep conjecture from reaching the status of scientific fact in technical conversation.  If we have no carefully drawn theory or empirical evidence to support a conjecture, it should be stated as a proposal, not a conclusion.


An "elevator pitch" breakdown of areas of applications for Reinforcement Learning & Neural Networks vs. Genetic Algorithms

Your question suggests a confusion of techniques, representations and problems.

Neural Networks are a representation that can be used to approximate functions. A neural network approximates a function that maps from inputs to outputs by optimizing parameters (weights).
Genetic Algorithms are a technique that can be used to optimize a problem. You might chose to use a GA to optimize the weights in a neural network for instance. Or you might use it to optimize a different representation or approximation of a function.
Reinforcement Learning is a problem. In a Reinforcement learning problem, the agent learns a function mapping states to actions. You can learn this function directly in some problem domains, or by a near-direct approximation (like tile-coding), or with a function appropriator (like a neural network).


What do the numbers in this CNN architecture stand for?

This number refers to the number of kernels (or feature maps) that are convolved with the input. So, for example, in the first convolutional layer, $64$ $3 \times 3$ kernels are convolved with the image.

The ResNet presented in Deep Residual Learning for Image Recognition is used for image classification. Furthermore, note that your diagram already contains a fully connected layer at the end.


Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why wouldn't they work?
Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output is squashed from $[0,1]$. If the entire layer uses a SoftMax function, then the output of all neurons is squashed from $[0,1]$ and their sum equals 1. In other others, they represent a set of probabilities, where you can then use cross-entropy to optimize their values (cross-entropy measures the difference between two probability distributions).
ReLU and ELU are simply other types of functions, whose output is not limited to the range $[0, 1]$. They are differentiable, like other activation functions, and so they can be used in any neural network.


Are there neural networks that accept graphs or trees as inputs?

Yes, there are numerous, coming under the umbrella term Graph Neural Networks (GNN).

The most common input structures accepted by these techniques are the adjacency matrix of the graph (optionally accompanied by its node feature matrix and/or edge feature matrix, if the graph has such information).

A Comprehensive Survey on Graph Neural Networks, Wu et al (2019) divides GNN's into four subgroups:

Recurrent graph neural networks (RecGNN)
Convolutional graph neural networks (ConvGNN)
Graph autoencoders (GAE)
Spatial-temporal graph neural networks (STGNN)

ConvGNN's can themselves be classified by whether they use Spectral methods or Spatial methods, and GAE's by whether they are designed for Network embedding or Graph generation.


Is it possible to use deeplearning with spark (with a distributed databases as HDFS or Cassandra)?

Yes, it is possible to use deep learning architecture with Apache Spark now. Databricks have Spark-deep-learning which is a pipeline based in python and uses tensorflow and keras.

https://github.com/databricks/spark-deep-learning

You can check this. There is also BigDL by Intel analytics.
https://github.com/intel-analytics/BigDL


Is there a continuous conditional variational auto-encoder?

Whether a discrete or continuous class, you can model it the same.
Denote the encoder $q$ and the decoder $p$. Recall the variational autoencoder's goal is to minimize the $KL$ divergence between $q$ and $p$'s posterior. i.e. $\min_{\theta, \phi} \ KL(q(z|x;\theta) || p(z|x; \phi))$ where $\theta$ and $\phi$ parameterize the encoder and decoder respectively. To make this tractable this is generally done by using the Evidence Lower Bound (because it has the same minimum) and parametrizing $q$ with some form of reparametrization trick to make sampling differentiable.
Now your goal is to condition the sampling. In other words you are looking for modeling $p(x|z, c;\phi)$ and in turn will once again require $q(z|x, c; \theta)$. Your goal will now intuitively become once again $\min_{\theta, \phi} \ KL(q(z|x, c;\theta) || p(z|x, c; \phi))$. This is still simply transformed into the ELBO for tractability purposes. In other words your loss becomes $E_q[log \ p(x|z,c)] - KL(q(z|x,c)||p(z|c)$.
Takeaway: Conditioning doesn't change much, just embed your context and inject it both into the encoder and decoder, the fact that its continuous doesn't change anything. For implementation details, normally people just project/normalize and concatenate it somehow to some representation of $x$ in both the decoder/encoder.


Taxi-v3 help. What is meant exactly by convergence of the algo, the highest reward and optimal action for every state?

Your Q learning update expression looks correct. The Total Reward will not be the same at the end of each episode because the starting position of taxi is different in each episode so the number of steps necessary to reach the final destination will be different in each episode. The graph that you posted shows that the algorithm converged after short amount of episodes so 100000 episodes might be too much. Since the environment is simple try manually calculating optimal policy for some starting specific position and then see if the algorithm does the same sequence of actions.


What is the approach to deduce formal rules based on data?

If you don't have non-examples of your pattern and don't have some kind of heuristic guide, unfortunately the answer is that you can't. "All sentences" will always be 100% compatible with your examples, and you'll never be able to collect evidence that disconfirms (or even decreases the likelihood) of that hypothesis. Even if you rule out that hypothesis by fiat, the hypothesis that the only non-accepted sentence is

  The answers for “I have a bunch of books and want to infer topics” and “I have a set of contextless sentences that fit a pattern and another set of contextless sentences that don’t fit that pattern and want to find the pattern” are extremely different.

will be impossible to rule out

With a good enough heuristic guide, you might be able to do something, but that would require knowing statistical facts about examples and non-examples and then sampling from the world IID, and just happening to not get any non-examples. That doesn't seem like the situation you're in. If you really have nothing to quantify the non-examples in any way, there's nothing that can be done.


Is there anything that ensures that convolutional filters don't end up the same?

No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has $n$ features, but 2 of them are the same, then the model effectively has $n-1$ features, which is a less expressive model than that of $n$ features, and therefore usually has a larger loss function.
But even if the weights are different, some of them can be very similar. If you visualize the first layer of your convolution filters, and you have a large number of them (e.g. 100), you will see some of them are learning to detect roughly the same edges (same orientation and placement). These features are so similar, they are effectively redundant in the model and do not add to its predictive power.
There's actually an entire field of research on identifying redundant features and pruning them. Le Cun shows in Optimal Brain Damage that pruning out redundant features not only make the model smaller and faster for inference, but can also even improve the model's accuracy.
Here is a blog post for a high level overview of one of the pruning methods for more info.


Mathematical foundations of the ability to learn

Computational learning theory (CLT or COLT) is devoted to the mathematical and computational analysis of machine learning algorithms, so it is concerned with the learnability (i.e. generalization, bounds, efficiency, etc.) of certain tasks, given a learner (or a learning algorithm), a hypothesis space, data, etc.
CLT can be divided into (at least) two subfields: statistical learning theory (SLT) and algorithmic learning theory (ALT). The most famous CLT frameworks are PAC learning and the VC theory (which extends PAC learning to infinite-dimensional hypothesis spaces).
The following are some good resources you can use to get started with CLT (but you can find more here).

The book Machine Learning (1997) by Mitchell
The book Foundations of Machine Learning (2012) by Mohri et al.
The book Understanding Machine Learning: From Theory to Algorithms (2014) by Shalev-Shwartz et al.
The paper An overview of statistical learning theory (1999) by Vapnik (who is one of the main contributors to SLT)
The paper Introduction to Statistical Learning Theory (2014) by Bousquet et al.

Here's a related question on this site: What sort of mathematical problems are there in AI that people are working on?.


The Project Summarized

The project goal appears to be a common one: Routing correspondence in an efficient manner to maintain good but low cost customer and public relations. A few features of the project were mentioned.

Neural network project
Received some design and project history from predecessor
Classifies messages for telcos
Sends results to support groups at appropriate locales
Uses 2 relu layers, ending with a softmax
Word2Vec embedding
Trained with a clean language file
All special characters and numbers removed

The requirements for current development were indicated. The current work is to develop an artificial network that places incoming messages into one of two categories accurately and reliably.

Moderated — insulting, fraudulent in purpose (spam), trivial routine
Operative — relevant question requiring internal human attention

Research and development is beginning along reasonable lines.

Trained with 300,000 messages
Word2Vec used
40% of classified as moderated
Permuted cycles and epochs
Achieved 90% accuracy
Loss stays near 0.5
In test, operative accuracy 0.9, moderated accuracy max of 0.6

First Obstacle and Feasibility

The first obstacle encountered is that in QA using production environment data, 90% of the messages where left unclassified, 5% of the classifications were accurate, and the remaining 5% were inaccurately classified.

It is correct that the even split of 5% accuracy and 5% inaccuracy indicates that information learned is not yet transferable to the quality assurance test phase using real production environment messages. In information theory phraseology, no bits of usable information were transferred and entropy remained unchanged on this first experiment.

These kinds of disappointments are not uncommon when first approaching the use of AI in an existing business environment, so this initial outcome should not be taken as a sign that the idea won't work. The approach will likely work, especially with foul language, which is not dependent on cultural references, analogies, or other semantic complexity.

Recognizing notices that are for audit purposes only, from a social network accounts or purchase confirmations, can be handled through rules. The rule creation and maintenance can theoretically be automated too, and some proprietary systems exist that do exactly that. Such automation can be learned using the appropriate training data, but real time feedback is usually employed, and those systems are usually model based. That is an option for further down the R&D road.

The scope of the project is probably too small, but that's not a big surprise either. Most projects suffer from early overoptimism. A pertinent quote from Redford's The Melagro Beanfield War illuminates the practical purpose of optimism.

          APPARITION

  I don't know if your friend knows what he's in for.

          AMARANTE

  Nobody would do anything if they knew what they were in for.

Initial Comments

It is not necessary to reduce the number of message categories to two, but there is nothing wrong with starting R&D by refining approach and high level design with the simplest case.

The last layer may be more training efficient if a binary threshold is used for the activation function instead of softmax, since there is only one bit of output needed when there are only two categories. This also forces the network training objective to be the definitive selection of a category, which may benefit the overall rate of R&D progress.

There may be ways of improving outcomes by adding more metrics in the code to beyond just 'accuracy'. Others who work with such details every day may have more domain specific knowledge in this regard.

Culture and Pattern Detection

Insults and curse words are entirely different kinds of things. Foul language is a linguistic symbol or phrase that fits into a broadcasting or publishing category of prohibition. The rules of prohibition are well established in most languages and could be held in a configuration file along with the permutations of each symbol or phrase. In the case of sh*t, related forms include sh*tty, sh*thead, and so on.

It is also useful to distinguish the sub-sets of foul language.

Cursing (expressing the wish for calamity to befall the recipient)
Swearing (considered blasphemy by some)
Exclamations that are considered foul by publishers and broadcasters
Additional items parents don't want their children to hear
Edge cases like crap

The term foul language is a super-set of these.

Distribution Alignment

Learning algorithms and theory are based on probabilistic alignment of feature distributions between training and use. The distribution of training data must closely resembles the distribution found when the trained AI component is later used. If not, the convergence of learning processes on some optimal behavior defined by gain or loss functions may succeed but the execution of that behavior in the business or industry may fail.

Internationalization

Multilingual AI should usually be fully internationalized. Training and use of training with two distinct dialects will almost always perform poorly. That creates a data acquisition challenge.

As stated above, classification and learning depend on the alignment of statistical distributions between data used in training and data processing relying on the use of what was learned. This is also true of human learning, so this requirement will not likely be overcome any time soon.

All these forms of foul language must be programmed flexibly across these cultural dimensions.

Character set
Collation order
Language
Dialect
Other locale related determinants
Education level
Economic strata

Once one of these is included in the model (which will be imperative) then there is no reason why the others cannot be included at little cost, so it is wise to begin with standard dimensions of flexibility. The alternative will likely lead to costly branching complexity to represent specific rules, which could have been made more maintainable by generalizing for international use up front.

Insult Recognition

Insults require comprehension beyond the current state of technology. Cognitive science may change that in the future, but projections are mere conjecture.

Use of a regular expression engine with a fuzzy logic comparator is achievable and may appease the stakeholders of the project, but identifying insults may be infeasible at this time, and the expectations should be set with stakeholders to avoid later surprises. Consider these examples.

The nose on your face looks like a camel.
Kiss the darkest part of my little white. (From Avatar screenplay)

The word combinations in these are not likely to be in some data set you can use for training, so Word2Vec will not help in these types of cases. Additional layers may assist with proper handling of the at least some of the semantic and referential complexity of insults, but only some.

Explicit Answers to Explicit Questions

  Is it possible to accomplish this task with a neural network?

Yes, in combination with excellence in higher level system design and best practices for internationalization.

  Is the structure of this neural network correct for this task?

The initial experiments look like a reasonable beginning toward what would later be correct enough. Do not be discouraged, but don't expect the first pass at something like this to look much like what passes user acceptance testing a year from now. Experts can't pull that rate of R&D progress off, unless they hack and cobble something together from previous work.

  Are 300k messages enough to train the neural network?

Probably not. In fact, 300m messages will not catch all combinations of cultural references, analogies, colloquialisms, variations in dialect, plays on words, and games that spammers play to avoid detection.

What would really help is a feedback mechanism so that production outcomes are driving the training rather than a necessarily limited data set. Canned data sets are usually restricted in the accuracy of their probabilistic representation of social phenomena. None will likely infer dialect and other locale features to better detect insults. A Parisian insult may have nothing in common with a Creole insult.

The feedback mechanism must be based on impressions in some way to become and remain accurate. The impressions must be labelled with all the locale data that is reasonably easy to collect and possibly correlated to the impression.

This implies the use of rules acquisition, fuzzy logic control, reinforcement learning, or the application of naive Bayesian approaches somewhere appropriate within the system architecture.

  Do I need to clean up the data from uppercase, special characters, numbers etc?

Numbers can be relevant. Because of historical events and religious texts, 13 and 666 might be indications of something offensive, respectively. One can also use numbers and punctuation to convey word content. Here are some examples of spam detection resistant click bait.

I've got a 6ex opportunity 4u.
Wanna 69?
Values are rising 50%! We have 9 investment choices 4 you to check out.

The meaning of the term special character is vague and ambiguous. Any character in UTF-8 is legitimate for almost all Internet communications today. HTML5 provides additional entities beginning with an ampersand and ending with a semicolon. (See https://dev.w3.org/html5/html-author/charref.)

Filtering these out is a mistake. Spammers leverage these standards to penetrate spam detection. In this example, the stroke similarities of a capital ell (L) and those of the British pound symbol can be exploited to produce spam detection resistant click bait.

Do you like hot £egs?

Removing special characters that fit within the Internet standards of UTF-8 and HTML entities will likely lead to disaster. It is recommended not to follow that part of the predecessor's design.

Regarding emoticons and other ideograms, these are linguistic elements that may represent in text encoding the volume, pitch, or tone modulation of phonetics, or they may represent face or body language. In many languages ideograms are used in place of words. For a global system running in parallel with the blogsphere, emoticons are part of linguistic expression.

For that reason, they are not significantly different than word roots, prefixes, suffixes, conjugations, or word pairs as linguistic elements which can also express emotion as well as logical reasoning. For the learning algorithm to learn categorization behavior in the presence of ideograms, the ideograms must remain in training features and later in real time processing of those features using the results of training.

Additional Information

Some additional information is covered in this existing post: Spam Detection using Recurrent Neural Networks.

Since spam detection is closely related to fraud detection, the spammer fraudulently acting like a relationship already exists with their recipients, this existing post may be of assistance too: Can we implement GAN (Generative adversarial neural networks) for classication problem like Fraud detecion?

Another resource that may help is this: https://www.tensorflow.org/tutorials/representation/word2vec


Which model is better given their training and validation errors?

I would say that your intuition is correct: the model associated with the first plot is likely to generalise more than the one associated with the second plot.

In both cases, it doesn't seem that your model has overfitted the training data. Overfitting often occurs when the training error keeps decreasing but the validation error starts to increase. In both your plots, both the training and validation errors keep decreasing (even if slowly, after a while).

Underfitting occurs when your model hasn't learned enough even about your training data. The smaller the training and validation error, the more likely your model has not underfitted, but the value of RMSE depends on the range of your inputs. See e.g. What are good RMSE values? for more info.

See also this article Overfitting and Underfitting With Machine Learning Algorithms for a general overview of the concepts of overfitting and underfitting.


Is there a training data capacity limit for AlphaZero (Chess)?

AlphaZero is on-policy*, which partially answers your question.
An on-policy algorithm is not the same as an online policy though, it is not required that updates are made on every step. It is simply required that all data used in the update is taken from the same "current" policy.
In practice, AlphaZero buffers results from games played with the current policy to create a dataset used to update its neural networks. That buffer is then emptied after the data has been used.
From the AlphaZero paper:

At the end of the game, the terminal position $s_T$ is
scored according to the rules of the game to compute the game outcome $z: −1$ for a loss, $0$ for a draw, and $+1$ for a win. The neural network parameters $\theta$ are updated so as to minimise the error between the predicted outcome $v_t$ and the game outcome $z$, and to maximise the similarity of the policy vector $p_t$
to the search probabilities $\pi_t$.

This implies only a single game is buffered in this way before running each update and then discarding the dataset generated in that game. Theoretically the same approach could be used with any number of games for each update step (provided the training system has capacity to store more moves).

* AlphaZero is on-policy because the core algorithm requires using a specific policy and then updating it to match an improved version of the same policy discovered using MCTS for planning during play.
It could be possible to construct an off-policy update mechanism using similar MCTS routine. I am not sure why this is not considered, but suspect it would be due to complexity/efficiency of the algorithm compared to the ease of generating new game data.


What is the difference between return and expected return?

Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and $R_{i}$ is the reward at time step $i$. Here $G_t$ and $R_i$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using the notation used in the book Reinforcement Learning: An Introduction, 2nd edition).
The expected return is defined as
\begin{align}
v^\pi(s)
&= \mathbb{E}\left[G_t \mid S_t = s \right] \\
&= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigm\vert S_t = s \right]
\end{align}
In other words, the value of a state $s$ (associated with a policy $\pi$) is equal to the expectation of the return $G_t$ given that $S_t = s$, so $v^\pi(s)$ is defined as a conditional expectation. Note also that the expected value is usually defined with respect to a random variable, which is the case. Note also that $S_t$ is a random variable, while $s$ is a realization of this random variable.
A policy is not a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution over actions given states. There are also deterministic policies. Have a look at this question What is the difference between a stochastic and a deterministic policy? for more details about the definition of stochastic and deterministic policies.

If an agent is following a policy, the agent wants to find the value at $S_{t}$ by calculating a sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?. Furthermore, note that the expectation of a discrete random variable is defined as a weighted average.


What loss function is appropriate for finding "points of interest" in a array of x,y inputs

You don't have to use machine learning to solve the problem.

Unify the scale of each data input (or each curve), such as normalize to $[0,1]$ (not necessary).
Calculate the slope of each pair of points $\frac{(y2 - y1)}{(x2 - x1)}$.
Set the threshold. Compare the difference between two adjacent slopes, the difference exceeds the threshold are marked as POI.

Isn't that simpler?

If you must solve the problem with CNN, what I can think of is that you first collect (or draw) a bunch of curves, mark the POI in advance, and then feed it to the CNN model.


How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

I'll answer in a couple of stages.
I feel somewhat lost as to what the input for the NN should look like.
Your choices boil down to two options, each with their own multitude of variants:

Vector Representation: Your input is a vector of the same size as your vocabulary where the elements represent the tokens in the input example. The most basic version of this is a bag-of-words (BOW) encoding with a 1 for each word that occurs in the input example and a 0 otherwise. Some other variants are (normalized) word counts or TF-IDF values. With this representation padding will not be necessary as each example will be encoded as a vector of the same size as the vocabulary. However, it suffers from a variety of issues: the input is high-dimensional and very sparse making learning difficult (as you note), it does not encode word order, and the individual word representations have little (TF-IDF) to no (BOW, counts) semantic information. It also limits your NN architecture to a feed-forward network, as more "interesting" architectures such a RNNs, CNNs, and transformers assume a matrix-like input, described below.

Matrix Representation: Here your input representation is a matrix with each row being a vector (i.e. embedding) representation of the token at that index in the input example. How you actually get the pretrained embeddings into the model depends on a number of implementation-specific factors, but this stackoverflow question shows how to load embeddings from gensim into PyTorch. Here padding is necessary because the input examples will have variable numbers of tokens. This stackoverflow answer shows how to add zero padding in PyTorch.
This representation will be significantly better than the vector representation as it is relatively low-dimensional and non-sparse, it maintains word order, and using pretrained word-embeddings means your model will have access to semantic information. In fact, this last point leads to your next question.

Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.
Word embeddings are based on the assumptions of distributional semantics, the core tenet of which is often quoted as "a word is characterized by the company it keeps". That is, the meaning of a word is how it relates to other words. In the context of NLP, models can make better decisions because similar words are treated similarly from the get-go.
For example, say that articles about furry pets get a lot of likes (entirely plausible if you ask me). However, the mentions of furry pets in these articles will be varied, including words like "dog", "cat", "chinchilla", "poodle", "doggo", "good boy", etc. An input representation that treats these mentions as completely distinct (such as BOW) will need to learn individual correlations between each word and the number of likes (that's a lot of learning). A well-trained word embedding, on the other hand, will be able to immediately group these mentions together and learn general correlations between groups of similar words and likes. Fair warning, this is a very imprecise description of why word embeddings work, but I hope it gives you some intuitive understanding.
Finally, since you're doing regression, make sure you choose your objective function accordingly. Mean squared error would be my first try.


How does the target network in double DQNs find the maximum Q value for each action?

Both in DQN and in DDQN, the target network starts as an exact copy of the Q-network, that has the same weights, layers, input and output dimensions, etc., as the Q-network.
The main idea of the DQN agent is that the Q-network predicts the Q-values of actions from a given state and selects the maximum of them and uses the mean squared error (MSE) as its cost/loss function. That is, it performs gradient descent steps on
$$\left(Y_{t}^{\mathrm{DQN}} -Q\left(s_t, a_t;\boldsymbol{\theta}\right)\right)^2,$$
where the target $Y_{t}^{\mathrm{DQN}}$ is defined (in the case of DQN) as
$$
Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right)
$$
$\boldsymbol{\theta}$ are the Q-network weights and $\boldsymbol{\theta^-}$ are the target network weights.
After a usually fixed number of timesteps, the target network updates its weights by copying the weights of the Q-network. So, basically, the target network never performs a feed-forward training phase and, thus, ignores a cost function.
In the case of DDQN, the target is defined as
$$
Y_{t}^{\text {DDQN}} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{-}\right)
$$
This target is used to decouple the selection of the action (i.e. the argmax part) from its evaluation (i.e. the computation of the Q value at the next state with this selected action), as stated the paper that introduced the DDQN)

The max operator in standard Q-learning and DQN, in (2) and (3), uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation


Why does the cost function contain a 2 at the denominator?

To simplify the derivative, probably. Otherwise there will be constant 2 in it.


In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this?

Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of overfitting to the training data. Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.

This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.

To improve on the situation, you need to use some form of regularisation. The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.

Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.

In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into K-fold cross-validation in order to get more accurate scores, and help you choose the best model for generalisation.


Normalization for well known data sets like coco-text and total text data set

Find the largest height and width amongst all the images. Let us call it H and W respectively. It is true that you cannot resize the images, but say if you have an image of height h and width w where h < H, w < W. To the right of the image append W - w number of columns and at the bottom of the image append H - h number of rows having some constant value (0 is okay for grey-scale and B/W images and 0 for each of the channel in case of colour images).

In this way all the images will be of same size. Since you appending at the right and bottom of the image, the annotations will not lose its meaning in the transformed image in terms of the position and content of the text to recognised.

You could also try pixelRNN kind of ideas after you are done with DNN. RNN can handle variable length inputs and in your case it will be sequence of pixels. Here you don't need to append rows and columns to the image.


Is it necessary to clear the replay memory regularly in a DQN when an agent plays against itself?

The purpose of the replay memory in DQN and similar architectures is to ensure that the gradients of the deep net are stable and doesn't diverge. Limiting what memory to keep and how far into the past it reaches is a problem that arises in in practical AI deployments. Throwing it away usually isn't the best decision because the agent's gradients may become unstable. Rather, keeping a window of the last T timesteps/gameplays can keep enough information while ensuring that the data is relevant as the agent improves. There are some additional methods which can improve the results as well:

Only keeping relevant timesteps (relevant depends highly on the application)
Removing timesteps from memory with probability relative to their age (the oldest has the highest probability to be removed)

It should be noted, that a larger replay memory is usually better than a smaller one to ensure stability but although memory is cheap it isn't infinite.


Should I prefer cropped images or realistic images for object detection?

Google recommandation seems to answer this:

The training data should be as close as possible to the data on which
predictions are to be made.
For example, if your use case involves blurry and low-resolution
images (such as from a security camera), your training data should be
composed of blurry, low-resolution images. In general, you should also
consider providing multiple angles, resolutions, and backgrounds for
your training images.

https://cloud.google.com/vision/automl/object-detection/docs/prepare
Would you agree, also in the case of symbols?


What is Imagination Learning and Imagination machines?

The whole idea of Imagination Learning seems to be in its infancy. The author of the response you linked wrote a paper on the subject, but as he notes the paper is more of an outline for "a new overarching challenge for AI." The second link you posted is not directly related to the ideas referenced in the paper on Imagination Learning, other than that both use the idea of imagination in humans as inspiration. In short, there does not seem to be much information on this topic and I suspect it will stay that way until we learn some of the underlying processes that go along with imagination in humans and then maybe Imagination Machines may come to fruition.


Training actor-critic algorithms in games with opponents

The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying to apply such algorithms to settings that are actually multi-agent settings, it is indeed common to encounter the problems you describe. They can all be viewed as your agent "overfitting" to the opponent they're training against:

  Play against random player. I had rather good results, but not as good as expected since most interesting states cannot be reached with a random opponent.

In this case, your agent will "overfit" against random opponents. It will likely become capable of beating them relatively easily, and to some extent this may also still translate to relatively weak non-random agents (like simple heuristic approaches), but it is unlikely to generalise to strong opponents.

  Play against list of specific AIs. Results were excellent against those AIs, but very bad with never seen opponents

Here you pretty much exactly give a description of what I mean when I say "overfitting" against the opponent you're training against.

  Play against itself. Seemed the best to me, but I could not get any convergence due to non-stationary environment.

When training using self-play like this, there is still a danger of overfitting against "itself". This may turn out to work well in some games, but in other games, there are indeed risk of instability / lack of convergence. This can, for example, be due to the agent "rotating" through a circle of strategies that beat each other. Intuitively, you could think of Rock-Paper-Scissors. A randomly-initialised agent may, for example, have a slight tendency to play Rock more often than the others. Self-play will then lead to a strategy that primarily plays Paper, to beat the current "Rock" strategy. Continued self-play training can then figure out that a tendency to play more Scissors will be strong against itself. Etc., these strategies can keep rotating forever.

The standard approach for self-play training in games is the one popularised by AlphaGo Zero and AlphaZero, and also around the same time independently published with results on the game of Hex. That last paper coined the term "Expert Iteration" for the approach, which I like to use. The basic idea of Expert Iteration is to have:

A policy that we're training (parameterised by some function approximator like a Deep Neural Net)
A tree search algorithm, like Monte Carlo tree search (MCTS).

These two components can then iteratively improve each other. The trained policy can be used to guide the search behaviour of MCTS. MCTS can be used to generate a new distribution over actions which, thanks to additional search effort, is typically expected to be a little bit better than then trained policy. That slightly better distribution is then used as a training target for the policy, using a Cross-Entropy loss. This means that the policy is trained to mimic the MCTS search behaviour, and also used to improve that same MCTS search behaviour.

This training procedure has been found to lead to strong results. One of the prevailing hypotheses is that the additional search performed by MCTS can help to stabilise training, and avoid overfitting to the "self" opponent; the search algorithm can actually "reason" about what a strong opponent (not necessarily the "self" agent) could do.

The Expert Iteration training procedure I described above often works well, but it doesn't really answer your question about training Actor-Critic approaches in games with opponents... because it's not really an Actor-Critic algorithm!

In Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates, we (I'm first author on this paper) propose an approach where the policy in self-play is trained using a training target based on the value estimates of MCTS, rather than the visit counts of MCTS (which is what we use in the standard Expert Iteration framework).

I don't know if it necessarily should exactly be called an Actor-Critic algorithm, but you could intuitively view it as an "Actor-Critic" approach where MCTS is the critic. As mentioned right before Section IV, the resulting gradient estimator also turns out to be very similar to that of the "Mean Actor Critic".

If Expert-Iteration-style solutions are not an option (for example because tree search is infeasible), I would suggest taking a look at some of the following Policy-Gradient-based approaches (and possibly later research that cites these papers):

Learning with Opponent-Learning Awareness
Policy Gradient Search: Online Planning and Expert Iteration without Search Trees


How to stay a up-to-date researcher in ML/RL community?

There are some wonderful resources for keeping up to date in the ML community. Here are just a handful that a coworker showed me:

Deep Learning Monitor: this site contains hot and new papers along with tweets that are popularized by the community! You can even checkout RL papers specifically here
arxiv-sanity: this site updates with popular and new papers that make it onto Arxiv
papers with code: this site is wonderful because not only does it link to papers, but it links to their implementation for reproduction or assistance in your own personal projects. They even have a leaderboard and track state of the art (SoTA) on tons of different tasks
DL_twitter loop: You can't forget twitter, given that most researchers use it; this is just a single nice group you may like


How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly.

You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let us forget all about Neural Networks for a bit, and talk about something far simpler.

Linear Regression

In the above figure, we clearly have one independent variable on the x-axis, and one dependent variable on the y-axis. The red line has an intercept of zero, and let's say a slope of 0.5. Therefore,
$$ y = 0.5x + 0 $$

This, right here, is a single perceptron. You take a value of x, lets say 8, pass it through the the node, and get a value as output, 4. Simple!
But what is the model in this case? Is it the output? No. Its the set [0.5, 0] that represents the red line above. The outputs are simply points on that line.

A neural network model is always a set of values - a matrix or a tensor, if you will.

The plots in your question, do not represent outputs. They represent the models. But now that you've possibly understood what a linear model with one indpendent variable looks like, I hope you can appreciate that having 2 independent variables will give us a plane in 3-D space. This is called multiple regression.

This forms the first layer of a neural network with linear activation functions. Assuming $ x_{i} $ and $ x_{j} $ as the two independent variables, the first layer computes
$$ y_{1} = w_{1}x_{i} + w_{2}x_{j} + b_{1} $$

Note that while $ y_{1} $ is the output of the first layer, the set $ [w_{1}, w_{2}, b_{1}] $ is the model of the first layer and can be plotted as a plane in 3D space.
The second layer, again a linear layer, computes
$$ y_{2} = w_{3}y_{1} + b_{2} $$

Substitute $ y_{1} $ in above and what do you get? Another linear model!

$$ y_{2} = w_{3}(w_{1}x_{i} + w_{2}x_{j} + b_{1}) + b_{2} $$

Adding layers to a neural network is only compounding of functions.

Compounding linear functions on linear functions result in linear functions.

Well, then, what was the point of adding a layer? Seems useless, right?

Yes, adding linear layers to a neural network is absolutely useless. But what happens if the activation functions of each perceptron, each layer was not linear? For example the sigmoid or the most widely used today, ReLU.

Compounding non-linear functions on non-linear functions can increase non-linearity.

The ReLU looks like this
$$ y = max(0, x) $$

This is definitely non-linear but not as non-linear as let's say the sine wave. But can we approximate the sine wave by somehow "compounding" multiple, say $ N $ ReLUs?

$$ \sin(x) \approx a + \sum_{N}b*max(0, c + dx)$$

And here the variables $ a, b,c, d $ are the trainable "weights" in neural network terminology.

If you remember the structure of the perceptron, the first operation is often denoted as a summation over all the inputs. This is how non-linearity is approximated in Neural Networks. Now one may ask: So, summing over non-linear functions can approximate any function, right? So a single hidden layer between input layer and output layer (one that sums over all the outputs of the hidden layer units) should be enough? Why do we often see neural network architectures with so many hidden layers? This is one of the most important yet often overlooked aspect of neural networks and deep-learning.

To quote, Dr. Ian J Goodfellow, one of the brightest minds in AI,

  A feedforward network with a single (hidden) layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly.

So, what is the ideal number of hidden layers? There's no magic number! ;-)

For more mathematical rigor on how neural networks approximate non-linear functions, one should learn about the Universal Approximation Theorem. Beginners should check this out.

But why should we care for increased non-linearity? For that I'd direct you to this.

Note that all of the above discussion is with respect to regression. For classification, the non-linear surface learned is regarded as a decision boundary and points above and below the surface are classified into different classes. However, an alternative, and arguably better, way to look at this is that given a dataset that is not linearly seperable, a neural network first transforms the input dataset into a linearly seperable form and then uses a linear decision boundary on it. For more on this, definitely check out Christopher Olah's amazing blog.

Finally, yes all independent variables must be normalized before training a neural network. This is to equalize the scale of different variables. More info here.


How can I perform multivariable regression with neural networks?

Have a look at sklearn's sklearn.neural_network.MLPRegressor class, which uses a multi-layer neural network to do regression. You first need to define the object MLPRegressor, for example, by specifying the value of the parameter hidden_layer_sizes, which determines the number of layers and the number of neurons per layer, then you should call the method fit on this created object and pass to it your data matrix $X \in \mathbb{R}^{n \times m}$, where $n$ is the number of samples and $m$ is the number of features.


Solving a Multi-Armed, "Multi-Bandit" Problem

Although you can frame your problem as a bandit problem or RL, it has other workable interpretations. Critical information from your comments is that:

Total reward is not a simple sum of all the results from 66 different machines. There are interactions between machines.
Total reward is deterministic.

This looks like a problem in combinatorial optimisation. There are many possible techniques you can throw at this. Which ones work best will depend on how nonlinearities and dependence between choices on different machines affect the end results.

Best Case

With deterministic results, if changes between machines were completely isolated, you could search each machine in turn, because you can treat all other 65 components as a constant if you don't change their settings. That would be very simple to code and take $7 \times 66 = 462$ steps to find the optimimum result.

Worst Case

In the worst case, the dependencies are so strong and chaotic that there is essentially no predictable difference between changing a single machine's setting and all of them. Pseudo-random number generators and secure hashing functions have this property, as do many quite simple physical systems with feedback loops.

In the worst case, there will be a "magic setting" with best results, and only a brute force search through all combinations of levers will find it.

In order to apply any more efficient search method, you have to assume that the response to combinations of levers is not quite so chaotic.

How to Search?

It seems likely from your description, that the best search algorithm is going to be somewhere between simple machine-by-machine optimisation and an exhaustive global search. However, it is hard to tell just where on that spectrum it lies.

There are a few different ways to frame it as reinforcement learning. For instance, you could use current switch combination as state, and run 66 switch changes as an "episode".

I would suggest that genetic algorithms are a good match for this search task, assuming there is at least some local-only effect that means combining two good solutions is likley to result in a third good solution. Genetic algorithms don't need calculations for gradients, and fit nicely with discrete combinations. Your genome can simply be the 66 different switch positions, and the fitness rating your black box score for those positions.

Plenty of other combinatorial search algorithms are available. Enough to fill a book or two. One place you could look for inspiration is Clever Algorithms: Nature-Inspired Programming Recipes which is a free PDF.


Precise localization and characterization of rudimentary shapes with neural networks

I think you're describing "object localization and detection" which combines object identification with discovery of its spatial placement in the field of view.  There's been a lot of work on this in the past 5 years using CNNs and variants.  It's very much in demand for pedestrian/obstruction detection, identification, and avoidance in autonomous car systems.

Here's a place to start:  https://github.com/kjw0612/awesome-deep-vision

Especially, take a look at "Faster R-CNN" by Ren et al, 2016.


I've worked on the BRATS dataset and I can verify that this is pretty much standard process. Besides throwing the totally blank images, I also throw away the images in the beginning and ending of the sequence that show the tip of the scull and the base of the neck.

Generally when dealing with MRIs,
I do this with a script (think of is as a preprocessing step) I run on each image that counts the amount of pixels that have a positive intensity (I actually add a small value to this to account for noise). Let's say for images with values 0-255 I count the amount of pixels with an intensity of over 10-15, let's call this $x$. After that I set a threshold (empirically), let's call it $t$ and discard images with $x < t$.

Specifically for BRATS, because you have the labels, you can see which of these have the desired classes and discard most of the rest. If you try to train a network on the dataset as is you face an enormous imbalance ratio. I've had trouble training networks due to this and the most success I got was when I threw away most of the irrelevant images.


Do convolutional neural networks also have recurrent connections?

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN.
RNNs and CNNs are different architectures but they can be used together.

Here is another sentence with the same structure:

  It is clear than dogs, specifically corgis, and cats are very common in online memes.


Positioning of batch normalization layer when converting strided convolution to convolution + blurpool

After finding the paper authors' Github, I saw that, although they only have a MobileNet V2 model implemented, they choose the Subsample-after-ReLu option (the first one in the question).
Although this doesn't fully answer my question, I'll take "the paper authors do it this way" as enough reason to prefer this over the alternative.


Is artificial intelligence really just human intelligence?

This is an old question, going back at least to 1950. It is one of the original objections to AI that Turing considers and attempts to refute in his seminal 1950 paper Computing Machinery and Intelligence.

Turing actually attributes this objection to Lady Lovelace, apparently quoted by another author. In Turing's paper, this is objection #6: Lady Lovelace's Objection, in section 6 of the paper. The objection is concisely stated as

  The Analytical Engine has no pretensions to originate anything. It can
  do whatever we know how to order it to perform.

where "The Analytical Engine" was an early design for an all-mechanical general purpose computer.

Turing offers two replies to this objection. First, he reminds us that computer programs have bugs. That is, they often do things their creators did not intend. This is unsatisfying to many readers, but it does address the objection: programs may act in ways that are unrelated to our intelligence, and in doing so, might display unexpected intelligent behaviors. In this sense, their intelligence would not be an intentional product of human intelligence.

Turing's stronger objection comes from an anticipation that learning would eventually move to the center of AI research (keep in mind again, this is written in 1950, well before any reasonable learning algorithms had been proposed!). Turing uses the example of a robotic child in Section 7 of the paper (Learning Machines) to elaborate on his point. A child is created by its parents, but, endowed with the ability to learn, quickly begins to display behaviors its parents do not anticipate or intend. No one would suggest that a person's intelligence is "really just" the intelligence of their parents, even though their parents created them, and are partially responsible for that intelligence.

Likewise, Turing's proposed robotic child is created by a parent, but, endowed with learning, quickly begins to engage in behaviors the parent does not anticipate or intend. Therefore, machine intelligence need not be reduced to just human intelligence.

I think that if Turing were alive today, he would agree that we are now beginning to move into the era of learning machines he anticipated. Some of our programs now engage in intelligent behaviors that we do not anticipate or understand. For example, self-driving cars now kill or maim people, because they have learned behaviors their creators did not intend or anticipate, perhaps not unlike a reckless teenage driver.


What type of model should I fit to increase accuracy?

after reading some literature in the area I'd recommend the following:

Try using Convolution Neural Networks (CNNs) this paper outlines some really good points on why should you use CNNs.
Try a Combination of the different layers in the same model. Start with some Convolution Layers, then some LSTMs and then a couple of Dense Layers followed by Dropout


Is it possible to use an internal layer's outputs in a loss function?

Yes, you can do that, and it is a standard practice. One famous example is the "Inception" network architecture. To keep inner subnets from "dying out",  several outputs from inner layers are extracted and passed into FC->Softmax. Then all the outputs are averaged in the loss function.
From practical point of view, you won't be able to implement such things with basic Sequential model construction. Most libraries allow you to move beyond it, though, and this distinction is usually well-documented. For example in the tensorflow guide on functional API it states right in the introduction:

The functional API is a way to create models that are more flexible than the tf.keras.Sequential API. The functional API can handle models with non-linear topology, shared layers, and even multiple inputs or outputs.

Hope that'll help you to get started.


What are the criteria for a system to be considered intelligent?

Typically, I think of intelligence in terms of the control of perception. [1] A related, but different, definition of intelligence is the (at least partial) restriction of possible future states. For example, an intelligent Chess player is one whose future rarely includes 'lost at chess to a weaker opponent' states; they're able to make changes that move those states to 'won at chess' states.

These are both broad and continuous definitions of intelligence, where we can talk about differences of degree. A sundial doesn't exert any control over its environment; it passively casts a shadow, and so doesn't have intelligence worth speaking of. A thermostat attached to a heating or cooling system, on the other hand, does exert control over its environment, trying to keep the temperature of its sensor within some preferred range. So a thermostat does have intelligence, but not very much.

Self-driving cars obviously fit those definitions of intelligence.

[1] Control is meant in the context of control theory, a branch of engineering that deals with dynamical systems that perceive some fact about the external world and also have a way by which they change that fact. When perception is explicitly contrasted to observations, it typically refers to an abstract feature of observations (you observe the intensity of light from individual pixels, you perceive the apple that they represent) but here I mean it as a superset that includes observation. The thermostat is a dynamical system that perceives temperature and acts to exert pressure on the temperature it perceives.

(There's a philosophical point here that the thermostat cares directly about its sensor reading, not whatever the temperature "actually" is. I think that's not something that should be included in intelligence, and should deserve a name of its own, because understanding the difference between perception and reality and seeking to make sure one's perceptions are accurate to reality is another thing that seems partially independent of intelligence.)


When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?

You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$.
The $\geq$ inequality is also correct, of course, and it could be that the authors' intention was to show that $\pi_{k+1}$ and $\pi_k$ satisfy the condition for the policy improvement theorem:

Let $\pi$ and $\pi'$ be any pair of deterministic policies such that, for all $s\in\mathcal{S}$,
$$q_\pi(s,\pi'(s))\geq v_\pi(s)\tag{4.7}$$

So they kinda relaxed the equality to show that the condition is satisfied "word-for-word" for $\pi'=\pi_{k+1}$ and $\pi=\pi_k$.
Although, I must say that they don't seem to be consistent in this - for example, they keep the strict equality at the derivation at the bottom of the page 78. So it might as well be that you've discovered a typo in Sutton and Barto.