Why cannot an AI agent adjust the reward function directly?

Why do both approaches prevent the AI agent from changing its reward function at will?

In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the goals of the agent. Sometimes this is obviously something that should not be under the agent's control, if the reward is a real-world quantity that it should maximise - e.g. the amount of money it makes in profit - then it makes no sense that the agent could arbitrarily declare that the quantity is different to the thing that it observed.

Other times, there is some flexibility, an agent that needs to escape a maze in a short time could have -1 reward per time step inside the maze or -0.1 reward per time step, or +1 reward for escaping with a discount factor applied. However, the flexibility can only go so far before it describes a different problem. Changing the -1 per time step to +1 per time step means that the agent's goal switches from escaping to staying in the maze.

In general, multiplying all the rewards in a MDP by some positive constant does not change a reinforcement learning problem. Sometimes it may be worth doing this scaling to make it easier for a specific approach, such as neural networks, to work efficiently. However, this is not something to put directly under the agent's control, but a hyperparameter like the number of hidden layers in the neural network. As a hyperparameter, usually the reward scaling is very flexible and not something worth spending much effort tuning - unlike the architecture of a neural network.

  What happens if we do allow the AI agent to do so?

Unless significant constraints are placed on what is allowed to change, then the agent will get any amount of reward it "wants" by doing anything it "wants", within whatever constraints are placed on changes allowed to the reward function. Typically in RL this would result in an agent that acts more or less randomly whilst getting progressively higher and higher reward on each iteration. Or in other words, an agent that does not attempt to solve any kind of problem.

There are a few special cases where a reward function can be adjusted or learned. One common case is inverse reinforcement learning, where an agent's activities are observed, it is assumed to be solving an MDP-like problem, and you are interested in understanding how it solves it, including what reward function it is using. The reward function must be learned by fitting it to observations of the agent.


How can we teach a neural net to make arbitrary data associations?

In a nutshell : Memorizing is not Learning
So, first let's just remind the classical use of a neural net, in Supervised Learning :

You have a set of $(x_{train}, y_{train}) \in X \times Y$ pairs, and you want to extract a general mapping law from $X$ to $Y$
You use a neural net function $f_{\theta} : x \rightarrow f_{\theta}(x)$, with $\theta$ the weights (parameters) of your net.
You optimise $f_{\theta}$ by minimizing the prediction error, represented by the loss function.

Can this solve your question ? Well, I don't think so. With this scheme, your neural net will learn an appropriate mapping from the set $X$ to the set $Y$, but this mapping is appropriate according to your loss function , not to your $(x_{train}, y_{train})$ pairs.
Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each $(x_{train}, y_{train})$.
So classical Supervised Deep Learning should not memorize $(x_{train}, y_{train})$  pairs.
However, you could memorize using a net with too many parameters :  it's Overfitting !

In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each $(x_{train}, y_{train})$  pair you feed during training.
However, for an input $x$ that it never saw during training, $f_{\theta}(x)$ would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.

But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be Expert Systems, I don't know them enough to explain them, but you may check that if you want.
What about the brain ?
The matter in answering this question is that we don't really know how does the brain work. I highly recommend this article discussing neural networks and the brain.
Some thoughts to start :

The brain has an incredibly huge amount of parameters, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks :  so the brain could be also able to overfit, and thus to memorize by this mean.
Our brain is not a feed forward network at all, we can't delimitate any layer, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.
It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !
A more personal thought : the brain is able to both learn and memorize ("The exception that proves the rule" motto shows that I think), while learning and memorizing are antonyms for neural nets...


Does Monte Carlo Tree Search not work on games without the same initial state?

If the initial state is not always the same, but if your agent is allowed to observe what the initial state is before it has to start running the search algorithm, there's basically no problem; it has all the information it needs when it starts running the tree search. This is how we typically use MCTS (or any other tree searches): we first observe what the current state looks like, and then start running the tree search for this state.
If for whatever reason you already have to start running your tree search without being allowed to observe which initial state has been randomly drawn, you can easily just pretend that you actually do have a "dummy", deterministic initial state which you can observe right before the real initial state, and pretend that the random sampling of an initial state is actually a random event / random transition resulting from a "dummy action". Then you can handle this scenario in exactly the same way that you would any normal game that has non-deterministic transitions (such as games that involve dice): if you have access to explicit knowledge of which stochastic events are possible, and what their probabilities are, you can encode them as chance nodes. If you do not have such explicit knowledge available, you can use an "open-loop" MCTS. See also: https://ai.stackexchange.com/a/13919/1641

Presence of opponents / other players in the game is a different issue. MCTS and other game tree search algorithms were designed specifically for this, they have no problems handling that. In a tree search, you do not just have nodes for your own agent and the states in which it is allowed to act; you also have nodes for any opponents and any states in which they are allowed to act, and the tree search enables you to reason about what actions your opponents are likely to take.


Why do we use a delay when feeding our input data to the echo state network?

It is not a 'delay' but a 'look ahead' as some people call it.  This is how far in the future the model can forecast (or predict).

In reference to the GitHub source, the code uses a look ahead ('delay') of one.  The echo state network learns on the 'inputData' using a future version of itself called 'targetData'.

Here is an analogy.  It is early on Friday, June 5, and I have the last 51 days worth of Amazon stock prices.  I can use the first 50 days of prices, which go up to and include Wednesday, June 3, to train a network to predict Thursday, June 4.  If I have a lot of historical data on Amazon, I can train many 50 day windows.  Each 50 day window would have as its target a stock price one day in the future.

Performance generally degrades the further into the future you try to forecast.  One reason is due to the build up of error over time.  This is true in life.  We usually know fairly accurately what we will do tomorrow, less so next week, and it would be a wild guess to predict what we will be doing in ten years.


debugging perceptron for digital AND circuit

You have several flaws in your code that will lead to unexpected behavior. I identified the following flaws you need to address. Once you have fixed them, you should output your updated weights during each step of training to see what the learning algorithm actually does and if it is going in the right direction. I will not address pure style aspects (like using a simpler case-statement instead of a series of ifs and stuff like that) and focus on the real errors.

Skipping training case 1

Once you are in training case 4, you set j to 1, expecting to use training case 1 next. But later in your code you increase j by one (j++) and go directly to training case 2, skipping the first one. This means you only run through training case 1 during your first pass of the loop.

Assigning j a value instead of comparing it

  if(j=1)

The if statement will always be true, because you do not compare j to 1 but you set j to 1. You basically assign a value and test if the value is true. Correct would be:

  if(j==1)

Ignoring bias during test

During the test steps after training you forget to add the bias that you used during training:

  output = weight1*input1 + weight2*input2

should actually be:

  output = weight1*input1 + weight2*input2 + 2

Otherwise your perceptron behaves differently during training and testing.

Perceptron output must be 1 or 0

Those were all implementation issues. This point actually seems to come from a misunderstanding of perceptrons. A real perceptron can either fire or not, meaning it outputs either 1 or 0, nothing in between. You calculate your output the following way and use the output to compare to the expected output:

  output = weight1*training1 + weight2*training2 + 2;

  if(output != expectedoutput )

Your output here is a float value. Only in edge cases it will result in 1 or 0. What you actually want to do for a real perceptron is:

  if (weight1*training1 + weight2*training2 + 2 >= 0) {
   output = 1
  } else {
   output = 0
  }

  if(output != expectedoutput)

After you fixed those errors and studied the output of your learning algorithms, you should be able to get the perceptron to work. If you have any questions, please leave a comment and I will try to help.

Edit:

I don't have a C compiler here, so I quickly implemented your solution with my suggested fixes in Python 3.6 and the results are very close to the correct solution. The final weights obviously have to be -1 and -1 for an AND Gate with bias 2. The learning process gets very close but never reaches -1 and -1 exactly, which leads to the wrong output in the end. This is a very good showcase to illustrate why you prefer a sigmoid neuron instead of a perceptron in modern neural networks. Here is a relevant question from cross-validated concerning the difference of those two types of neurons.


How to properly optimize shared network between actor and critic?

Just use one class inheriting from nn.Module called e.g. ActorCriticModel.

Then, have two members called self.actor and self.critic and define them to have the desired architecture.Then, in the forward() method return two values, one for the actor output (which is a vector) and one for the critic value (which is a scalar).

This way you can use only one optimizer.


Is the singularity equivalent to the creation of God?

infinite being, eternal, supernatural and existing by itself, necessary cause and ultimate end of all that exists

As the Portuguese dictionary says, the problem of calling God is the connection with spirituality, it is even strange to call a machine of God.

Now, we can put Google as an example. They know what you do, where you are, what you like and what you do not want anyone to know, and he's everywhere listening to you. The only way to not be under Google's vision is to stop using a cell phone, computer and living a boring life so carefully that you would have to have to not be watched.

It's funny the relationship. According to the Bible (I am an atheist), the man departed from God in Babylon (cities).

But can we call Google God? If Google does not have a backup of its core scattered through space and other planets, I think the human can still destroy the world and bring Google together. But following rational ideology, he is closer to being God than anything on earth, including us.


Handle non-existing states in q-learning

You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector.

The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and the value in state $s'$ will, then, be a weighted average of the values in those neighbouring states. Formally, $$Q(s') = \frac{s_{i+1} - s'}{s_{i+1}-s_i}Q(s_i) + \frac{s'-s_i}{s_{i+1}-s_i}Q(s_{i+1}),$$ for $s_i < s'< s_{i+1}$, where $s_i$ and $s_{i+1}$ are the neighbours, $s_i$ is the $i$-th element of DS(i.e. si = DS[i] = Q-table[i,1]), and $Q(s_i)$ is related to the Q-table as Qsi = Q-table[i,2] (asuming array indexing starts from 1).

Restricting the actions would work as follows. For simplicity, assume that the agent chooses the next state directly, i.e. $s'=a$. Then, if you have an array of actions A = [a1, a2, ... , aM], $M \le N$ then each $a_i$ needs to be present in the state array DS (i.e. A is a subset of DS, formally $A \subseteq S$). This may not be desirable but it is an option.


Typically what you probably would want to do is do the training on something other than a Raspberry Pi. I think for what you're wanting to accomplish with having a computer talk back to your parrot, you won't need anything too crazy with a bunch of GPU's - but I don't think you'll want to necessarily do the training on a Pi either.

Here are some questions I have:

What are you going to plat back to the parrot? i.e. are you going to play it back random parrot sounds you found online or sounds that you've recorded? are you going to play it back what it just said? are you going to play it back a modification of what it just said?
Do you want it to respond to the parrot anytime the parrot speaks? Or when the parrot "says" something specific?

I think depending on the answers to those two questions, there are a couple of different paths that you could go down.

As for the hardware of the Raspberry Pi itself, I have never done any speech recognition with it, but I have done image recognition with it via the Movidius Neural Compute Stick which according to this Quora post, may be able to be used to offload some of the processing "relatively easily".

Here are some other links you may find valuable:

Tensorflow speech recognition: https://github.com/pannous/tensorflow-speech-recognition

The YouTube video that the above repo references: https://www.youtube.com/watch?v=u9FPqkuoEJ8

How to control a raspberry pi with your voice: http://www.techradar.com/how-to/how-to-control-the-raspberry-pi-with-your-voice
Voice recognition software setup on Raspberry Pi: https://diyhacking.com/best-voice-recognition-software-for-raspberry-pi/


Why does keras model get bigger after training?

The answer is that it's the size of the Adam optimizer state. When I change the optimizer to SGD (the vanilla optimizer), the size is not big anymore.
As far as I know, the Adam optimizer maintains gradients information of previous training iterations. And the gradient size can be as big as the model size. That's why it causes the file size to be so big.

With this in mind, when you save your model please make sure to set include_optimizer=False if you seem to use an optimizer that maintains big state similar to Adam.

Beware though, it means that you cannot load the model and continue training it again, it should only be used for inference.


Have agents that "dream" been explored in Reinforcement Learning?

Yes, the concept of dreaming or imagining has already been explored in reinforcement learning.
For example, have a look at Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. Hamrick et al., which is a paper that I gave a talk/presentation on 1-2 years ago (though I don't remember well the details anymore).
There is also a blog post about the topic Agents that imagine and plan (2017) by DeepMind, which discusses two more recent papers and also mentions Hamrick's paper.
In 2018, another related and interesting paper was also presented at NIPS, i.e. World Models, by Ha and Schmidhuber.
If you search for "imagination/dreaming in reinforcement learning" on the web, you will find more papers and articles about this interesting topic.


If the normal equation works, why do we need gradient descent?

That normal equation is sometimes called the closed-form solution.
The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources).
This answer gives you more details and an example.


How does non-max suppression work when one or multiple bounding boxes are predicted for the same object?

I might be able to help with the theory, but the coding... it is a non standard API such as Tensorflow or Pytorch (it might be custom code for what I can tell).
The key element here is that the bouding boxes are removed only if they hold a prediciton for the same class that the box that is overlapping with (but with less confidence, that is why it gets removed).
Here is an example, where we have:

Two classes $c \in [c_1, c_2] = [$ "star"$, $ "moon" $]$
Three bounding boxes

The blue bounding boxes holds prediction for the class $c_1$ so their predictions are $p(c_1)_{box1} = 0.8$ and $ p(c_1)_{box2} = 0.9$. On the other hand, the green box holds a prediction for the class $c_2$.
The three boxes are highly overlaping so the overlap between any box $x$ with any box $y$ will be above the IoU threshold: $IoU(box_x, box_y) > 0.5$. So in principle all boxes are suceptible to be removed.
However the NMS only applies for boxes predicting the same class (in the case the blue one). So the NMS algorihtm is: if the boxes are overlapping, $IoU(box_1, box_2) > 0.5$, which is true, remove all non maximal class probability boxes. Said differently, take just the box with highest $p(c_1)$ and remove the rest. So the $box_1$ with class probability $p(c_1) = 0.8$ would be removed.
So what happens with green box? Isn't it overlapping as well? Yes, but consider that the green box is not trying to predict the same object, is trying to predict another object, $c_2$, which happens to be very close to the first object, $c_1$. This way object detectors support detection of different overlapping objects.


What happens when I mix activation functions?

The general answer to the behavior of combining common activation functions is that the laws of calculus must be applied, specifically differential calculus, the results must be obtained through experiment to be sure of the qualities of the assembled function, and the additional complexity is likely to increase computation time. The exception to such increase will be when the computational burden of the combination is small compared to the convergence advantages the combination provides.
This appears to be true of Swish, the name given to the activation function defined as
$$f(x) = x \, \mathbb{S}(\beta x) \; \text{,}$$
where $f()$ is the Swish activation function and $\mathbb{S}$ is the sigmoid function. Note that Swish is not strictly a combination of activation functions. It is formed through the addition of a hyper-parameter $\beta$ inside the sigmoid function and a multiplication of the input to the sigmoid function result.
It does not appear to be developed by Google. The originally anonymously submitted paper (for double blind review as a ICLR 2018 paper), Searching for Activation Functions, was authored by Prajit Ramachandran, Barret Zoph, and Quoc V. Le around 2017. This is their claim.

Our experiments show that the best discovered activation function, ... Swish, ... tends to work better than ReLU on deeper models across a number of challenging datasets.

Any change in activation function to any one layer will, except in the astronomically rare case, impact accuracy, reliability, and computational efficiency. Whether the change is significant cannot be generalized. That's why new ideas are tested against data sets traditionally used to gauge usefulness1.
Combining activation functions to form new activation functions is not common. For instance, AlexNet does not combine them.2. It is, however, very common to use different activation functions in different layers of a single, effective network design.

Footnotes
[1] Whether these traditions create a bias is another question. Those who follow the theory of use case analysis pioneered by Swedish computer scientist Ivar Hjalmar Jacobson or 6 Sigma ideas would say that these tests are unit test, not functional tests against real world use cases, and they have a point.
[2] To correct any misconceptions that may arise from another answer, AlexNet, the name given to the approach outlined in ImageNet Classification with Deep Convolutional Neural Networks (2012) by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton from the University of Toronto, does not involve combining activation functions to form new ones. They write this.

The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels.
...
The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. The internal layers are pure ReLU and the output layer is Softmax.

There are also convolution kernels and pooling layers in the AlexNet approach's series of layers used by them, and the design has entered common use since their winning of the ImageNet competition in 2012. Other approaches have won subsequent competitions.


Should I grey-scale the coloured frames/channels to build the approximation of the state?

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy?

In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and green. This also makes intuitive sense to me since it's like a bit-encoding.

Generally, if you have an environment like this, I would (without any other information) guess that each of the 5 colors have some meaning that may be relevant for your agent. That's just what I would guess though. In theory, it might be possible that white means one thing (e.g. "empty"), and every other colour means the same other thing (e.g. "not empty"). If you had domain knowledge like that, and knew that it is only important whether or not any given pixel is white, you could of course binarise your input.
But in general, if the colours might be important, I'd recommend including them. If you really only have a few distinct colours like that though, I would not recommend encoding them in some format like RGB where values can range from 0 to 1 or 0 to 255. I would recommend having 4 (or 5?) binary channels in your input:

Binary channel containing 1s for pixels that are black, and 0s for all other pixels.
Binary channel containing 1s for pixels that are white, and 0s for all other pixels.
Binary channel containing 1s for pixels that are blue, and 0s for all other pixels.
...
etc.

The reason for this is that deep neural networks often tend to have an easier time learning with binary inputs, and here you can completely binarise your inputs without requiring an excessively high number of channels. If you had hundreds or thousands of different possible colours, this would probably no longer be a good idea.


Can an object's movement (instead of its appearance) be used to classify it?

Machine learning systems work based on input data. The form of that data is irrelevant. It may need some configuring on the technical side of things, but the general ability to learn from the dataset remains the same.
ML systems are built to learn how to interpret data, without you needing to predefine exactly what it should be looking for.

But even then, you can reason that videos are no different from images. A video is essentially nothing but a flipbook of images. If you were to paste every frame of the video in a long sequence, you would have one big picture.
The rest should follow as usual: if a machine learning system can be taught to interpret images, it can evidently also interpret these "frame sequence" images, which means it's able to interpret the dataset that we colloquially call a video.

That being said, since a video contains much more information (a 5 second 30fps video is 150x as much data as a single image with the same resolution), the required learning process increases exponentially.

So yes, it's perfectly possible, but it will require more processing power and training as there are more input variables to account for.

There are ways to reduce this increase in complexity. For example, to check if someone is drunk or not, rather than push the entire video through the learning algorithm, you instead preprocess the video to figure out the person's gait or even just their footsteps (relative location and timing) and only push this processed data through.

This dramatically cuts down on the complexity of the network needed; at the cost of requiring preprocessing which may introduce issues (bad preprocessing = bad learning).


Bellman Expectation Equation leading to results where value iteration would not converge to the optimal policy

What am I missing here?

You are not missing anything mathematically.
Potentially what you are missing is that the discount factor $\gamma$, is part of the problem definition. In reinforcement learning (RL), you do not always solve problems to obtain the highest total sum of rewards. Instead you solve problems to obtain the highest expected return on any action. If you use a discount factor, then it affects the return calculation, which in turn may affect which behaviours are optimal.
If you choose $gamma$ such that an agent would move away from a terminal state to receive the highest expected return, then you have defined the problem such that this behaviour is optimal.
This can be an issue when solving RL problems and you have taken a free choice for reward functions and definition of return (this is quite common, often it is the researcher's problem to set these values up to define the agent's goals). Often you can use a discount factor to make a problem more numerically stable, because it will keep sums of rewards over many time steps within bounds. Also, it can be used as a way to search for faster solutions - i.e. in less time steps to reach episode end - because the agent will prioritise reaching higher values sooner if it can. Your example MDP deliberately puts those two effects in opposition to each other.
Assuming you want the agent to find the terminal state by getting over the larger negative reward just before it: Probably you should not use a discount factor for your MDP. Alternatively, if the reward signal is more of a free choice for that problem, you could offset it (by $+0.1$) and then most values of discount factor will still resolve to reaching the terminal state as optimal behaviour.
There are a few common ways to mitigate or avoid the issue of changing problem definition by selecting $\gamma$:

Use a value of $\gamma$ derived from the problem. In some cases, the value of $\gamma$ is a realistic physical part of a problem. For instance in financial problems, cash now is often preferable to cash later.

Set $\gamma$ to relavtively high value, such as $0.99$ or $0.999$, depending on episode length and relative sizes of rewards. This can work well enough for a wide range of problems. It is a common solution when using DQN. Effectively this approximates the next approach (average rewards) for a low cost.

Use average reward settings. There are separate Bellman equations, TD updates etc based on maximising expected average reward per time step as opposed to maximising expected discounted return.


In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?

The lower bound in MINE is as follows:
$$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$
Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$
Note that in the above equation, the first term is calculated from the joint distribution while the second term from the marginals of $X$ and $Z$. In the implementation of MINE, these statistics are calculated over the data from a minibatch. The marginal distribution is obtained by shuffling the values of Z (or X) along the batch dimension. Hence, in this case, the gradient is as follows.
$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$

As mentioned, the expectation over the marginals is not calculated
over the true marginal distribution (i.e. over the entire dataset)
but from the shuffled samples in the minibatch. Hence, the above gradient $G_B$ is biased.
When we maintain an exponential moving average of
$\mathbb{E}_B[e^{T_{\theta}}]$, we kind of incorporate statistics from outside the current minibatch also (i.e. over the entire dataset). This is an attempt to get an approximation of the true marginal estimate. The denominator term in the gradient allows for such a computationally inexpensive bias reduction trick.


How important will statistical learning be to a conscious AI?

Computational Learning Theory gives us an interesting framework to understand what statistical learning is doing.

The gist of it is, we can model the process of statistical learning as one of formal deduction. The learning itself does not require a random element.

This shouldn't be too surprising. Consider a classic decision tree learner like C4.5 or ID3: the algorithm works through the data deterministically, and no random decisions are made. When asked to make a prediction, the learned model returns the frequency of each possible label within the most similar subpopulation from its training data, with similarity defined according to the algorithm's rules for partitioning data.

There's no reason you can't write a decision tree learner (or even a deep learning algorithm) in Prolog, it just might not be very efficient, or very practical.


The problem was due to the following issues in my implementation:

The offspring generated in the crossover was not mutated (!)
The mutations did not occur with the expected frequencies (too few links and weight mutations)
The sigmoid activation had to be steepened

Another thing that previously caused issues was the network.activate function. Make sure that you wait for the network to stabilize when doing classification tasks, so all signals have time to propagate through the network.


What is the idea called involving an AI that will eventually rule humanity?

If I'm not mistaken you're looking for Roko's Basilisk,

  in which an otherwise benevolent future AI system tortures simulations of those who did not work to bring the system into existence


Where can I find an implementation of the wake-sleep algorithm?

I found the following detailed and well documented Python notebook, which uses only NumPy.


How to build a DQN agent with state and action being arrays?

Let's assume you would like to work with a classic DQN. You need to train the Q-network where inputs are the states and actions. The DQN is a function of Q(states, actions). The network is supposed to predict Q-value. The agent must pick up the action that produces the highest Q by giving the all possible actions to the network, in your case. Let's assume the current state is
[1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0]

You must pick up an item for the user which state is 1? you need to generate the all possible actions, for example,
[1, 2, -1, -1, -1, 3, 4, 5, -1, -1, -1]
[12, 1, -1, -1, -1, 2, 3, 4, -1, -1, -1]
[11, 12, -1, -1, -1, 1, 2, 3, -1, -1, -1]
...

You will need to generate all of these possible actions into the Q-network. Your worst case would be around 479 million possible actions to calculate in DQN.
If you would like to implement DQNs for this problem, I do recommend you to check out REINFORCE -> DQN -> Advantage Actor-Critic (A2C), which is the combination of policy network and DQN. With A2C, you can produce continuous actions with a policy network, action in range a[i]∈(0,1)*12, then probably round the generated actions. Then, the next problem is the policy network may produce an impossible action. So, you might skip to the nearest possible item.
For further reading on continuous action domains, I recommend to check out DDPG, PPO


Your stride will be just an integer number only when doing 1D convolution. When doing 2D convolution you get 2 integers like your example: (1, 1).

When there is more than one dimension to a stride, then each is applied independently in its associated dimension, and all valid combinations that fit in the area or volume should be calculated.

A 2D stride defines a rectangular grid of points where an operation is applied. The first number is distance between grid points along first (x) axis, the second number is distance between grid points along second (y) axis. You can decide to downsample each dimension differently by using different numbers for each dimension.

In TensorFlow is common to see 4D stride parameters, with axes of (example, height, width, channel). Typically you don't want to skip any examples or channels in a normal CNN, even though the framework can do that, so you will see values like (1, 2, 2, 1) used when using stride to downsample just image height and width dimensions between layers.


How do randomly initialized neural networks behave?

The output of randomly initialised neural networks will be random. I am not sure if I understand your question correctly but I would like to try and explain what OpenAI is trying to achieve with their RND Distillation approach.

Any re-enforcement or supervised "learning" algorithm such as a neural network (NN) with back-propagation, genetic algorithm, partial swarm etc requires a fitness function. The fitness function "directs" the algorithm to a solution.

The basic idea is that the algorithm is randomly initialised. The algorithm is run, and then parts of the algorithm are changed. In this case the weights of the NN.

A number of algorithms exist to update the weights, the most common back propagation / gradient decent as used in the paper. To update the weights the output of the NN is compared to the optimum output. Sometimes the optimum is known. Other times it is not. In the case of a game the optimum output is not known. It is assumed that the more points gained the better the NN does. In this case the fitness function or policy is the sum of points gained when playing the game. But gradient descent needs an error (i.e difference between output and expected output), which does not exist when playing a game, because you don't know what the points should be. However the NN could play one round of the game score some points, then run the NN again on the next move to determine a prediction of future points. The difference between the points gained and the prediction is then the error, which gradient descent can use. [Hope this makes sense]

This is fine for games like pong, a NN could play the game pong randomly and score point at each prediction, which will direct gradient descent into updating the weights to maximise points.

However, games such as Montezuma’s Revenge, requires a lot of complex actions before points are gained. If the points stay 0, how does gradient descent determine if the weights should be decreased or increased?

Instead, the authors initialise a second random neural network which remains the same for the duration of training. As mentioned above the output of a randomly initialised neural network is random.

The output of the actual NN, and world state is then passed through the random NN (RND), which will return a random value. The NN is then required to not only determine a game action but also determine the result of the RND.

The policy function is extended from just points to also predict the result of the RND. The greater the prediction error the more bonus points are allocated to the policy function. This encourages gradient descent to change the weights of the NN to explore unseen parts of the game as this increases the prediction error of the RND which increases points. At the same time gradient descent must update the weights of the NN to predict the RND.

I hope this answers your question.

In summary randomly initialised NN include multilayered perceptrons or convolutional networks will behave randomly with respect to their inputs. The RND network is randomly set once at the star of training, and used to basically remind the NN undergoing training that it has done this before do not do this again - i.e. rewards the NN for doing something new (exploration).


You are right, that pseudocode is not correct. In particular, the definition of $H_t^i$ in line $11$ should be changed; all the way on the right-hand side, it should have $3N - 3j$ columns of $0$s, rather than $3j$ columns of $0$s.

With that change, every matrix $H_t^i$ will have the same number of columns:

$$6 + 3j - 3 + 3N - 3j = 3 + 3N,$$

which evaluates to a total of $48$ in your case (because you have $N = 15$ landmarks). That's precisely the correct dimensionality required for matrix multiplication with your $\mu_t$ vector.

The version of the book that you linked to appears to be a fairly old draft. This webpage contains errata for the third edition for the book, in which page 393 corresponds to what was page 310 in your version of the book. The errata for that third edition of the book can be downloaded at the following URL: http://probabilistic-robotics.informatik.uni-freiburg.de/corrections/pg393.pdf

There you'll find the fix that I described above, but also some other fixes (most of them are just notational, adding bars over the $\mu$ vectors, but it looks like a more serious issue was additionally fixed in line 13, where a minus was changed to a plus).


Can maximum likelihood be used as a classifier?

If you read nothing else, maximum likelihood estimate => chance that the data predicted is the data observed. If you have a range of points (2, 3, 4, 5, 71) your MLE is going to favour ~4.5 because of means and standard deviations. MLE speeds up finding good input parameters, usually for a different classifier.

To answer your question:

1) Columbia University have a great example of using MLE classifiers, where everything is broken down into bitesize (or bytesize) chunks. Read this. Seriously.

2) In short, MLE is best used for simple, univariable distributions. It doesn't scale well to big problems but is waaay faster than a Bayesian network for simple tasks like predicting your height based on the heights of your immediate relatives. If you want to get technical, the conditional probability network of the Bayesian model reveals insights faster than the chain multiplication of the more primitive MLE.

Hope it helps!


Can I start with perfect discriminator in GAN?

If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.
You can find more information in this article. (I strongly suggest to read)
Also from the paper " Towards principled methods for training generative adversarial networks":

In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the
cost function on $\theta$ better approximates the JSD), and then do gradient steps on  $\theta$, alternating these
two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates
to the generator get consistently worse. The original GAN paper argued that this issue arose from
saturation, and switched to another similar cost function that doesn’t have this problem. However,
even with this new cost function, updates tend to get worse and optimization gets massively unstable.

Note 1: $\theta$ is parameters of generator
Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to $L(D^{ \ast }, g_{\theta})=  2 JSD(P_{r}, P_{g}) - 2log(2)$
Also from the paper of Wasserstein Loss:

The fact that the EM distance is continuous and differentiable a.e. means that
we can (and should) train the critic till optimality. The argument is simple, the more we train the critic, the more reliable gradient of the Wasserstein we get, which is actually useful by the fact that Wasserstein is differentiable almost everywhere. For the JS, as the discriminator gets better the gradients get more reliable but the true gradient is 0 since the JS is locally saturated and we get vanishing gradients ... In Figure 2 we show a proof of concept of this, where we train a GAN discriminator and a WGAN critic till optimality. The discriminator learns very quickly to distinguish between fake and real, and as expected provides no reliable gradient information. The critic, however, can’t saturate, and converges to a linear function that gives remarkably clean gradients everywhere.


Reinforcement learning: How to deal with illegal actions?

In my project I also had the problem that the action space is not the same for every state of the environment. I do not like the approach to penalty forbidden actions with a high negative reward since it feels a bit like cheating. However it might work, I just haven't tried it.

The approach I used, which you could apply as well, is to integrate an additional function into your action space. This function would map an action to a specific amount of kW. Thereby, depending on the current state, the function maps the action to the amount of kW to charge or discharge your battery with. This has the advantage that you do not have to deal with illegal actions.

This could be applied as follows:
Instead of defining for every action the amount to charge / discharge your battery with, you create a set of functions that defines the respective amount. Here an example with five actions:

Action: Discharge the battery entirely
Action. Discharge the battery so that half of its capacity remains,
otherwise do nothing
Action: do nothing
Action: charge the batter to half its capacity, otherwise do nothing
Action: charge the battery to its maximum capacity


Are there any strategies that would help me visualize the 'behavior space' and make a novelty function?

I think that the best approach is to "switch point of view" from the general, objective-oriented, Genetic Algorithm's behaviours.

Usually GAs rely on individualism: the best survives. To do this you have to define what 'best' means and this is done through a fitness function that measures something objective, independent from the individual (i.e. a score, classical example).

If you want to measure novelty, collectivism becomes more important, as the whole population (present and past) must be considered. You have to think about measuring how different the individual X did from 'the rest' (i.e. location never reached before).

Another interesting point is that you should think in a more multidimensional way: a score is uni-dimensional, everyone wants to get higher, that is easy. A location in space is more difficult to achieve and that is why novelty becomes interesting.

To sum up: stay multidimensional, always treat the population as a multi-organism rather than focusing on the single individual. I hope this helps :)


Can a data compression function be used to make predictions?

The way some (not all) compression algorithms work is that they encode frequent events in a short code, and rarer events with a longer code. Overall you save more space by encoding the common elements than you need to expend coding the rare ones. One example of this is a Huffman code, which uses a variable length encoding based on the frequency of the items.

You can use a compression algorithm for prediction if if encodes more than one event at a time. For example, word pairs rather than individual words. Each word pair will have a code, and the common word pairs (eg of the) will have shorter codes that the ones which are less common (eg of three). For prediction, select all the word pairs that start with your known sequence (eg of). Now select from that list the pair with the shortest code (which is more common), so in this example of would more likely be followed by the rather than three. After than, repeat the process with the next word, so look for pairs that begin with the.

All you need is the compression 'code book' which is produced during the compression process -- it's essentially a model of the data you compressed. This also works for longer sequences than pairs, of course.

If you want to know more about the topic, I can recommend Managing Gigabytes by Witten, Moffat, and Bell. Great book on compression techniques.


I do not know about all competitions, but the Loebner prize is one of the more well known competitions based around the concept of the Turing Test, specifically for chatbots, and judges must ask the same questions of all bots. The questions are only revealed during the competition, it is not possible for bot programmers to prepare for them other than making a generally good chatbot. They judges can then rank bots on how they handle the same set of questions. E.g. from chatbots.org/ai_zone/viewthread/867 here are the 2012 "qualifying" questions that were released following the test:

My name is Bill. What is your name?
How many letters are there in the name Bill?
How many letters are there in my name?
Which is larger, an apple or a watermelon?
How much is 3 + 2?
How much is three plus two?
What is my name?
If John is taller than Mary, who is the shorter?
If it were 3:15 AM now, what time would it be in 60 minutes?
My friend John likes to fish for trout.  What does John like to fish for?
What number comes after seventeen?
What is the name of my friend who fishes for trout?
What whould I use to put a nail into a wall?
What is the 3rd letter in the alphabet?
What time is it now?

According to the WikiPedia description, the style of the questions has changed over time. The above set in my opinion seem to be designed to test very simple grammar, common sense and memory targets.


What is correct update when the some indexes are not available?

What you are referring to as the situation where

  some indexes are not available

is simply the situation where some actions are not available/valid in some state. So, yes, the ${\arg \max }$ will be calculated based only on the available actions in that state. More formally,
$$\underset{a \in \mathcal{A}(s)}{\arg \max } \, Q(s, a)$$

where $Q(s,a)$ has been initialized for all $s \in \mathcal{S}^{+}$ and $a \in \mathcal{A}(s)$ and $\mathcal{A}(s)$ is defined as the set of all actions available in state $s$.
See Sutton and Barto's Intro to RL book, chapter 6 (the part on Q-learning).

In the same book (chapter 3), the authors state that:

  To simplify notation, we sometimes assume the special case in which
  the action set is the same in all states and write it simply as $\mathcal{A}$.

Therefore, in many sources, you may see the $\arg \max$ expressed as $\underset{a \in \mathcal{A}} {\arg \max}$ or even $\underset{a} {\arg \max}$. It's implicit that it is taken with respect to the set of actions $\mathcal{A}(s)$ available in state $s$.


Is input normalization built-in into mammals sensory neurons?

Yes, for many sensory inputs there is indeed something similar to normalization. But its not rally the same as in classical data analytics compared to what eg min/max normalization does or other technics.

Lets look on some examples and considerations:

mammals don't perceive heat or loudness in a linear way. This is because already many sensory receptors have chemical / physical limits. Double decibels will not perceived with double intensity. Inside your ear, the small hammer and abil will brace to protect you. --> its like normalization with logarithmic effects applied.
heat perception is more like a difference integration than a absolute temperature measurement. Its measured via H+ ions flow in mitochondria in the cell (if i recall correctly)
On the neuronal side gradual signals in the dendrites (analog signal) sum up gradually to later form an spike at the axon hill. where in turn a fire frequency is then encoded - the maximum frequency of this serves as a a natural maximum limit. I remember that grasshoppers increase axon fire frequency when objects started covering more ommatidial area on their "eye". The more of their "eyes" are covered by the shadow the more input on the neuron --> higher fire rate.
a lot of sensory input is post processed in higher cerebral areas. Eg. compared to what is expect able and heuristics are applied to compare a signal with former events.
when doing computational data analysis we may want go for accuracy and maximum comparability. Mostly on all data that could be available. --> eg. with respect to properties of a standard normal distribution. Hence we put some effort to be accurate and know the true parameters, remove outliers and so on --> big data comes into play here.
Nature in contrast strives often for efficiency with the means of reaching the minimal required with minimal resources.

Summary:
Compared to normalization in an analytical sense (eg. mean, min-max or other feature normalization techniques), nature is often only interested in the current difference between stimuli. And this only within some relevant range. Other information is not integrated. And normalization with the goal of making measurement points comparable only happens within this range of the mapping function provided by the sensor/neuron/receptor whatever!

So this should also answer your question about, why you are not smelling something in the city after a while any more. However, this for sure happens at higher cerebral regions (it might also be that your smell receptors saturate) its the same principle. Your consciousness just saves energy by not concentrating on something that is anyway not changing.

If you want to read more have a look here: https://en.wikipedia.org/wiki/Weber%E2%80%93Fechner_law


What are the techniques for detecting and preventing overfitting?

Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.).

The more interesting technique is to observe your validation loss with respect to the number of parameters in the network (often controlled by the number of layers/feature maps). If the validation starts dropping with raising your model's complexity, then your optimization might be bad or simply the model remembers all of the training samples and overfits badly.


Understanding the pseudocode of uniform-cost search from the book "Artificial Intelligence: A Modern Approach"

I think this is a problem with missing brackets in pseudocode — clearly the state is only added to the frontier if it hasn't been explored already, so it would be:

if not [contains(frontier, state) OR contains(explored, state)] then

which is equivalent to your interpretation of

if not [contains(frontier, state)] AND not [contains(explored, state)] then

(according to De Morgan's Laws)

Being half-way between a programming language and natural language, this is a case where pseudocode is not quite precise enough.


Do the order of the features ie channel matter for a 1d convolutional network?

Generally, order matters. A (trained) Neural Network (NN) is just a mathematical function trained on taking some given input and producing the corresponding output. So, if you train a certain node on producing large output if (and only if) an animal is present in a picture (for example), but later you give it the numeric evidence for a car being present in the image, it will still produce large output, indicating an animal being present in the image. This is simply because a node doesn't know what it is receiving or supposed to detect. It just follows its standard mathematical procedure.
So, if you train your network on one kind of input data, you must provide the same kind of input data during testing (or at least you cannot simply exchange inputs or outputs of certain nodes without manually correcting for that in the remainder of the network).
In the most simple case, consider whether you could simply swap the inputs to that simple function: $f(x,y) = x^2 + y$. If you swap the inputs, the output will be different. The exactly same applies to NNs.
Addition:
I think this post explains and especially illustrates the intuition nicely.


How can I have the same input and output shape in an auto-encoder?

If you look at Keras' output, there are various steps which lose pixels:
Max pooling on odd sizes will always lose one pixel. Conv2D using 3x3 kernels will also lose 2pixels, although I'm puzzled that it doesn't seem to happen in the downsampling steps.
Intuitively, padding the original images with enough border pixels to compensate for the pixel loss due to the various layers would be the simplest solution. At the moment I can't calculate how much it should be, but I suspect rounding up to a multiple of 4 should take care of the max pooling layers. For denoising, borders could be just copied from the outermost pixels, probably with some sort of low pass filtering to avoid artefacts.


How to get more accuracy of the logistic regression model?

Try Rectification
Improve the features available to your model, Remove some of the NOISE present in the data.

In audio data, a common way to do this is to smooth the data and then rectify it so that the total amount of sound energy over time is more distinguishable.

# Rectify the audio signal
audio_rectified = audio.apply(np.abs)

You can also calculate the absolute value of each time point. This is also called Rectification because you ensure that all time points are positive.

Smooth your data by taking the rolling mean in a window of say 50 samples

audio_rectified_smooth = audio_rectified.rolling(50).mean()

Calculating the envelope of each sound and smoothing it will eliminate much of the noise and you have a cleaner signal.

Calculate Spectrogram

Calculate a spectrogram of sound(i.e combining of windows Fourier transforms). This describes what spectral content (e.g., low and high pitches) are present in the sound over time. there is a lot more information in a spectrogram compared to a raw audio file. By computing the spectral features, you have a much better idea of what's going on.

this is similar to how we calculate rolling mean :

We calculate multiple fourier transforms in a sliding window to see how it changes over time.For each timepoint, we take a window of time around it, calculate a fourier transform for the window, then slide to the next window . The result is a description of the fourier transform as it changes throughout the time-series called a short-time fourier transform or STFT.

Choose a windows size and shape
At a timepoint, calculate the FFT for that window
Slide the window over by one
Aggregate the results

Calculating the STFT

We can calculate the STFT with librosa $\rightarrow$ import librosa as lr
There are several parameters we can tweak (such as window size)
For our purposes, we'll convert into decibels which normalizes the average values of all frequencies.
We can then visualize it with the specshow() function

This how you can calculate the STFT
# Calculating the STFT
# Import the functions we'll use for the STFT
from librosa.core import stft, amplitude_to_db
from librosa.display import specshow

# Calculate our STFT
HOP_LENGTH = 2**4
SIZE_WINDOW = 2**7
audio_spec = stft(audio, hop_length=HOP_LENGTH, n_fft=SIZE_WINDOW)

# Convert into decibels for visualization
spec_db = amplitude_to_db(audio_spec)

# Visualize
specshow(spec_db, sr=sfreq, x_axis='time', y_axis='hz', hop_length=HOP_LENGTH)

Try Spectral feature engineering
you can also perform the Spectral feature engineering on you baby audio data

since each time-series has a different spectral pattern.
We can calculate these spectral patterns by analyzing the spectrogram.
For example, spectral bandwidth and spectrum centroids describe where most of the energy is at each moment in time.

# Calculate the spectral centroid and bandwidth for the spectrogram
bandwidths = lr.feature.spectral_bandwidth(S=spec)[0]
centroids = lr.feature.spectral_centroid(S=spec)[0]

# Display these features on top of the spectrogram
ax = specshow(spec, x_axis='time', y_axis='hz', hop_length=HOP_LENGTH)
ax.plot(times_spec, centroids)
ax.fill_between(times_spec, centroids - bandwidths / 2,centroids + bandwidths / 2, alpha=0.5)

Now you can Combine spectral(spec $\rightarrow$ spectral dataframe) and temporal features in a classifier
centroids_all = []
bandwidths_all = []

for spec in spectrograms:
    bandwidths = lr.feature.spectral_bandwidth(S=lr.db_to_amplitude(spec))
    centroids = lr.feature.spectral_centroid(S=lr.db_to_amplitude(spec))
    # Calculate the mean spectral bandwidth
    bandwidths_all.append(np.mean(bandwidths))
    # Calculate the mean spectral centroid
    centroids_all.append(np.mean(centroids))
# Create our X matrix
X = np.column_stack([means, stds, maxs, tempo_mean, tempo_max, tempo_std, bandwidths_all, centroids_all])

for your logistic regression models

One of the way to improve accuracy is by optimising the prediction probability cutoff scores generated by your logit model.
You can Normalize all your features to the same scale before putting them in a machine learning model.
Look for class imbalance in your data.
You can Optimize on other Metrics also such as Log Loss and F1-Score.
Tune the hyperparameters of your model. In the case of LogisticRegression, parameter $C$ is a hyperparameter.


How do you know if an agent has learnt its environment in reinforcement learning?

This depends on the complexity of the environment being learned, and the purpose for learning it. There is no general answer.

For the simple environments used to teach reinforcement learning (RL), often the optimal solution is obvious, or can be calculated and proven optimal. For instance, any environment that can be solved using policy iteration will have a known optimal policy and optimal value function. The goals of these environments are to teach, or to check correctness of agents - it helps in these cases to have a well known correct answer.

At the next level up in terms of complexity are well-studied environments which can have achievable targets set for learning agents. The goals of these environments include getting useful metrics for learning agents, such as how many episodes it takes a particular implementation to learn it well enough. Defining "well enough" is a matter of experience with existing agents.

Getting more complex still, in general it is not possible to know whether an agent has fully optimised against its environment. The subject area of sequential decision making that includes RL agents can cover scenarios such as driving a car or playing a computer game. We don't know when any agent, whether they are based on RL or some other approach, has fully learning an environment, and instead must construct tests of behaviour - e.g. make an agent simulate driving in a set of scenarios, and expect at least safe behaviour in each of them, essentially a driving test similar to one a person might take. In these environments, often the tests are based on "good enough to use" goals. We can say an agent has learned to drive if it drives more safely than an average human.

In the special case of competitive games, we can score agents against each other or against human players. You might say that an agent has learned its environment if it beats some standard player, but also you can rank agents against each other and declare a particular agent as the current best.

It is possible to mix and match these ideas. The Atari games learning suite has benchmark scores to reach that count as "standard human", and recently agents have been published that beat all of those scores.

  What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?

If you include training episodes, then RL learns mainly by "trial and error". So you should expect an agent to make deliberate mistakes as it tests to see what happens. In some environments these could be critical mistakes leading to failed episodes.

If you ignore the training episodes and are interested only in performance without exploration - e.g. testing every few hundred episodes - then you can expect performance to vary depending on the type of agent and environment. Some agents even exhibit "catastrophic forgetting", which as the name implies causes performance to drop significantly - this can be caused by a successful agent over-fitting to all the recent successful episodes without errors it just experienced, and losing the ability to predict the true lower value of incorrect actions.

Neither failed episodes during training nor catastrophic forgetting are inevitable. It depends on the environment and type of agent.


Keras pattern finding between hash and word

I'm not quite sure it's possible. Hash functions are used to map an input to an output in a way that is not reversible.

Many companies store a hash of your password on their servers so in case of a security breach they haven't given the adversaries a long list of passwords.

As far as it goes for finding the exact hash of a word, it seems infeasible.

Edit: Binary classification refers to the possible output being two possible states. A ten dimensional one-hot vector is not binary.


Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations.
Policy iteration
In policy iteration, a policy lookup table is generated, which can be arbitrary. It usually maps a deterministic policy $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$, but can also be of the form $\pi(a|s): \mathcal{A} \times  \mathcal{S} \rightarrow \mathbb{R} = Pr\{A_t = a |S_t =s\}$. Policy iteration then alternately evaluates then improves that policy, with the improvement always being to act greedily with respect to expected return. Because the policy function can be arbitrary, and also the current value estimates during evaluation might not relate to it directly, the function $\pi(s)$ or $\pi(a|s)$ needs to be shown.
Typically with policy iteration, you will see this update rule:
$$V(s) \leftarrow \sum_{r,s'} p(r,s'|s,\pi(s))(r + \gamma V(s'))$$
The above rule is for evaluating a deterministic policy, and is probably more commonly used. There is no real benefit in policy iteration to working with stochastic policies.
For completeness, the update rule for an arbitrary stochastic policy is:
$$V(s) \leftarrow \sum_a \pi(a|s) \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
Value iteration
In value iteration, the current policy to evaluate is to always take the greedy action with respect to the current evaluations. As such, it does not need to be explicity written, because it can be derived from the value function, and so can the terms in the Bellman equation (specifically the Bellman equation for the optimal value function is used here, which usually does not refer the policy). What you would typically write for the update step is:
$$V(s) \leftarrow \text{max}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
However, you can write this out as if there was a policy table:
$$\pi(s) \leftarrow \text{argmax}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
$$a \leftarrow \pi(s)$$
$$V(s) \leftarrow \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$$
This is not the usual way to implement it though, because of the extra maximum value search required to identify the action. In simple value iteration it does not matter what the interim action choices and policies actually are, and you can always derive them from the value function if you want to know.
Other value-based methods
You will find other algorithms that drive the current policy direct from a value function, and when they are described in pseudo-code they might not have an explicit policy function. It is still there, only the Bellman update is easily calculated directly from the value function, so the policy is not shown in the update step. Descriptions of SARSA and Q-learning are often like that.


The reason for its slowness is quite obvious-- the computer must perform tens of thousands of iterations on each feature map. So, how do we decrease the computational complexity of the algorithm?

In terms of computational complexity / algorithm, there is not a lot to gain; max pooling simply has to go through all the feature maps to find the maximum numbers in each of the sections to be "merged/pooled" by taking the max.

There likely is a lot to gain in terms of implementation though. The current implementation is entirely in pure Python, and pure Python is notoriously slow. Those kinds of loops can be run significantly faster using numpy operations rather than manual Python loops. Such operations tend to be much faster due to:

running optimized C code rather than Python code, and
in some cases, using vectorized operations to perform multiple similar computations at different indices simultaneously, rather than doing them one-by-one

I did not yet try to "translate" your pure python code into python code using numpy. However, some examples of numpy-based implementations can be found in various answers to this question on StackOverflow.

I assume that your choice to manually implement things like max pooling is because you want to learn about implementing it / understand it better. If, instead, your goal is simply to get something running as quickly as possible, it may be a good idea to look into using a framework such as Tensorflow or PyTorch. These come with efficient implementations of many things you'll want for Neural Networks, including Max Pooling.


Your zero-classes just do not exist for the model. There is no information about them in training and test sets. I think the reason for overfitting is that your have very small size of the training set comparing to the number of parameters in your model. If it is easier for a model to remember all training entries - it will just do that. You need to prune your model to force it to make some generalizations about the data.


Would it be ethical to allow an AI to make life-or-death medical decisions?

I will interpret the questions as being about triage. This is particularly important in crisis situations, where a lot of such life-or-death decisions have to be taken.

In the START system there are four different categories:

the deceased, who are beyond help
the injured who could be helped by immediate transportation
the injured with less severe injuries whose transport can be delayed
those with minor injuries not requiring urgent care.

According to the category assigned to a patient, you then decide what to do. Other systems might be more fine-grained, but the principle is the same: effectively the human decision is in the classification, which then guides the assignment of resources. In the above list, the second category would probably be the highest priority for treatment. But once the category has been decided, the course of action has been determined, though the actual treatment options will then be considered for each case.

The ethics are thus in the judgment of the survival chances: if a nurse decides patient X is too far gone to warrant treatment, that's it. So this is the hard part, the life-or-death decision.

There are then two aspects to consider:

The accuracy of the diagnosis
The predicted likelihood of treatment being effective

The diagnosis should be (I'm not a medic!) fairly neutral. There are of course mis-diagnoses, but there is no value judgment involved. There might be some 'leakage' between step 1 and step 2, in that a human being might be influenced by the prospects of treatment of a diagnosis when deciding what the problem is. As in: this is a nice person, I don't want them to die, so I (subconsciously) exclude the diagnosis X which would mean certain death.

In this case a computer system that had sufficient accuracy when making a diagnosis would IMHO actually be more ethical than a human being. Provided, of course, that there is no bias in the diagnosis. Which I think is a practical problem, but in theory could be dealt with.

Once the diagnosis has been determined, the estimation of treatment success is the next decision. This also can be subject to human error, and a computer system with access to statistical information (again, unbiased) of issues and their likelihood of survival could be more independent of other factors which a human being might use, but which are unrelated to the case.

So basically, I would say that a computer system could be more ethical than a human being in such a situation. Because you can defend the way it reached it's decision, and it has been without taking into account factors not related to the problem at hand.

However, it's not always that easy. There are plenty of cases where other factors influence the decision. As long as there are fixed rules, that might not be a problem, though. Some issues would be (in each case the patients would have the same diagnosis and projected survival chance):

One ventilator, but an 8-year-old child and a 85-year-old man
A 30-year-old pregnant woman and the Prime Minister of the country
A one-month-old baby and a 25-year-old student
A homeless person and a billionaire

As a given, the system would have to be agnostic of general features such as gender, race, religion, etc. But how do you take into account age, social status, and a whole host of other factors?

If there is no difference in the situation, the fairest way would be to toss a coin. That would surely upset a lot of people ("How can you justify leaving our president to die in this time of crisis!"), but if you had explicit rules ("If one patient is in a higher tax bracket, they get priority") you might upset even more people. The advantage of having such rules would be to make the bias explicit, and in a way protect a nurse or paramedic from having to decide between a rich senator and a homeless person — whose family is more likely to go after you if you decided against them? And if you have explicit, unambiguous rules, why not use them to guide an AI system?

Every human being has their own preferences, and I am glad that I don't have to make those kinds of decisions — it sounds like a horrific task to me. How can you even sleep in peace after basically having condemned one each of the above cases to death?

That is another factor about the ethics of AI: by relieving humans from having to make such decisions, it would also have a beneficial effect. If the final decision is the same that the human would come to, then it's a win/win situation. But that is probably unlikely, due to a whole range of subconscious biases.

The prime issue to me seems the lack of recourse in the case of "computer said no". When a human makes a decision (like a referee in a game), there will always be an argument if people are unhappy with it. But in this case there could be none. The oracle has spoken, your father will be left to die. Sorry, no other outcome possible. It would probably be the same with a human decision, but it wouldn't feel as 'cold': you can observe that the person making the decision has to struggle with it. And you might understand that it was not an easy choice. With a computer, that element is missing.

Anyway, to summarise: given various caveats,

sufficiently accurate and unbiased diagnosis,
unbiased prediction of treatment outcome,
transparent handling of non-medical factors,

I would say that an AI system would me more ethical, as it has a principled way of reaching a decision which does not disadvantage any group of patients and would always get the same decision for the same patient; furthermore it takes a heavy burden off triage staff who otherwise have to make such decisions.

Which does not mean that I would be happy with my loved-ones' survival being decided by such a system :)


Is it ethical to create a chatbot to answer questions on Stack Overflow?

Yes, it is possible, and has actually been done in the past.

The University of Antwerp created a bot to answer questions (this is the technical report). It focused on the git tag only though (even though it did answer one mysql question).

Its accuracy was pretty good, and the bots in the tests did earn some reputation. So I assume it is possible.

But do note that the last bot in the tests revealed that it was a bot, and thus got banned. So if you reveal that the account you are running the bot on is a bot, there is a high chance that it will get banned.


When should one prefer using Total Variational Divergence over KL divergence in RL

I did not read those two specified linked/cited papers and I am not currently familiar with the total variation distance, but I think I can answer some of your questions, given that I am reasonably familiar with the KL divergence.

When you compute the $D_{KL}$ between two polices, what does that tell you about them

The KL divergence is a measure of "distance" (or divergence, as the name suggests) between two probability distributions (i.e. probability measures) or probability densities. In reinforcement learning, (stochastic) policies are probability distributions. For example, in the case your Markov decision process (MDP) has a discrete set of actions, then your policy can be denoted as $$\pi(a \mid s),$$which is the conditional probability distribution over all possible actions, given a specific state $s$. Hence, the KL divergence is a natural measure of how two policies are similar or different.
There are 4 properties of the KL divergence that you always need to keep in mind

It is asymmetric, i.e., in general, $D_{KL}(q, p) \neq D_{KL}(p, q)$ (where $p$ and $q$ are p.d.s); consequently, the KL divergence cannot be a metric (because metrics are symmetric!)
It is always non-negative
It is zero when $p = q$.
It is unbounded, i.e. it can be arbitrarily large; so, in other words, two probability distributions can be infinitely different, which may not be very intuitive: in fact, in the past, I used the KL divergence and, because of this property, it wasn't always clear how I should interpret the KL divergence (but this may also be due to my not extremely solid understanding of this measure).

and how is it different from what a $D_{TV}$ between the same two policies tells you?

$D_{TV}$ is also a measure of the distance between two probability distributions, but it is bounded, specifically, in the range $[0, 1]$ [1]. This property may be useful in some circumstances (which ones?). In any case, the fact that it lies in the range $[0, 1]$ potentially makes its interpretation more intuitive. More precisely, if you know the maximum and minimum values that a measure can give you, you can have a better idea of the relative difference between probability distributions. For instance, imagine that you have p.d.s $q$, $p$ and $p'$. If you compute $D_{TV}(q, p)$ and $D_{TV}(q, p')$, you can have a sense (in terms of percentage) of how much $p'$ and $p$ differ with respect to $q$.
The choice between $D_{TV}$ and $D_{KL}$ is probably motivated by their specific properties (and it will probably depend on a case by case basis, and I expect the authors of the research papers to motivate the usage of a specific measure/metric). However, keep in mind that there is not always a closed-form solution not even to calculate the KL divergence, so you may need to approximate it (e.g. by sampling: note that the KL divergence is defined as an expectation/integral so you can approximate it with a sampling technique). So, this (computability and/or approximability) may also be a parameter to take into account when choosing one over the other.
By the way, I think that your definition of the total variational divergence is wrong, although the DTV is related to the DKL, specifically, as follows [1]
\begin{align}
D_{TV} \leq \sqrt{\frac{1}{2} D_{KL}}
\end{align}
So the DTV is bounded by the KL divergence. Given that the KL divergence is unbounded (e.g. it can take very big values, such as 600k, this bound should be very loose).
Take a look at the paper On choosing and bounding probability metrics
 (2002, by Alison L. Gibbs and  Francis Edward Su) or this book for information about $D_{TV}$ (and other measures/metrics).


How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.


Modelling gut-feeling/subconscious knowledge of stock market traders

You could perhaps model gut feeling or subconscious bias as a prior in a Bayesian context, and then try to learn from the data how much to modify/moderate the bias in each individual case.

I think there is another issue with the problem you outlined.  We might expect it to be normal to see more losers than winners in the long run: trading is a zero-sum game where the house always takes its cut. The trick to being a successful trading algorithm seems to be to make the losers small (cut them early) and the winners big (let them run).


It is quite common in DQN to instead of having the neural network represent function $f(s,a) = \hat{q}(s,a,\theta)$ directly, it actually represents $f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N_a,\theta)]$ where $N_a$ is the maximum action, and the input the current state. That is what is going on here. It is usually done for a performance gain, since calculating all values at once is faster than individually.

However, in a Q learning update, you cannot adjust this vector of output values for  actions that you did not take. You can do one of two things:

Figure out the gradient due to the one item with a TD error, and propagate that backwards. This involves inserting a known gradient into the normal training update step in a specific place and working from there. This works best if you are implementing your own backpropagation with low-level tools, otherwise it can be a bit fiddly figuring out how to do it in a framework like Keras.
Force the gradients of all other items to be zero by setting the target outputs to be whatever the learning network is currently generating.

If you are using something like Keras, the second approach is the way to go. A concrete example where you have two networks n_learn and n_target that output arrays of Q values might be like this:

For each sample (s, a, r, next_s, done) in your minibatch*

Calculate array of action values from your learning network qvals = n_learn.predict(s)
Calulate TD target for $(s,a)$ e.g. td_target = r + max(n_target.predict(next_s)) (discount factor and how to handle terminal states not shown)
Alter the one array item that you know about from this sample qvals[a] = td_target
Append s to your train_X data and qvals to your train_Y data

Fit the minibatch n_learn.fit(train_X, train_Y)

* It is possible to vectorise these calculations for efficiency. I show it as a for loop as it is simpler to describe that way


Is there a probabilistic version of minimax?

Yes, there is at least one probabilistic version of minimax, which is called expectiminimax. In expectiminimax, in addition to min and max nodes, there are also chance nodes, which perform a weighted sum of the successors, so the probabilities associated with chance nodes must be known. Given that expectiminimax assumes the existence of random events (represented by the chance nodes), the decisions are thus based on expected values.

Section 5.5 of the book Artificial Intelligence: A Modern Approach provides a description of the expectiminimax algorithm, which was introduced by Donald Michie in Game-playing and game-learning automata (1966). The paper Optimal strategy in games with chance nodes (2007) also gives a decent description of the expectiminimax algorithm.


Why can't I reproduce the experiments in the original paper that introduced the Firefly Algorithm?

I wrote some python code to reproduce this paper's purported results. My code very efficiently optimizes simple smooth functions like bowls, but does not come close to reproducing the paper's claimed results on more complex functions, including with the parameters the authors report. I think that, since both @Jairo and I were unable to reproduce the results from the information in the paper, independently, it is likely that something is wrong with the paper.

It may be possible to reproduce the paper's claimed behaviors using a library like py_swarm, however the paper is using a Firefly Algorithm (already a rarer form of a PSO), and is using a home-brew variant of that algorithm, so my guess is that rolling your own is required, as we both did.

Additionally, I find the paper's claims fairly unlikely. For example, in section 4.1, they claim to converge on the global minimum of a function with many local minima in just 10 iterations. It seems to me like most of the time the fireflies will quickly converge on the bottom of one of the gullies in this function, and get stuck there. This is also what I observe in my replications. I suspect the authors may have cherry-picked their results from the best runs without reporting this, or have omitted some key detail from the paper.

Here is my replication code, in case someone else wants to try to reproduce this:

from math import sin, pi, exp, sqrt
from random import random
from copy import deepcopy

def michalewiz_objective(x):
    result = 0
    for i, x_i in enumerate(x):
        result -= sin(x_i)*(sin(i*(x_i**2)/pi))**20
    return -result

def bowl_objective(x):
    return -sum([x_i**2 for x_i in x])

pop_size = 40
max_generations = 10
alpha = 0.2
gamma = 1
beta_0 = 1
d = 2
I = michalewiz_objective
#I = bowl_objective

def move(firefly, other_firefly):
    radius = sqrt(sum([(firefly[i] - other_firefly[i])**2 for i in range(0, len(firefly))]))
    for i, value in enumerate(firefly):
        firefly[i] += beta_0*exp(-gamma*radius**2)*(other_firefly[i] - firefly[i])
        firefly[i] += alpha * (random() - 0.5)

# Using 4*random() to match Figure 3's apparent spread.
fireflies = [[4*random() for i in range(0, d)] for j in range(0, pop_size)]

for generation in range(0, max_generations):
    new_fireflies = [deepcopy(firefly) for firefly in fireflies]
    for index, firefly in enumerate(fireflies):
        for other_firefly in fireflies:
            if I(other_firefly) > I(firefly):
                move(new_fireflies[index], other_firefly)
    fireflies = new_fireflies
    best = max([I(f) for f in fireflies])
    mean = [sum([f[0] for f in fireflies]), sum([f[1] for f in fireflies])]
    print(best)
    print(mean)


What are the differences in testing between traditional software and artificial intelligence?

Testing machine learning programs is quite different than testing traditional software.
The main reason why this is the case is quite simple, if you're familiar with machine learning.
ML programs are not just if statements and loops, but they are composed of models, which can even be black-box models, such as neural networks (i.e. it's difficult to interpret the function that they compute). These models are trained to approximate some unknown function, which is only partially described by some given data, which can also contain spurious/noisy information.
For this reason, traditional testing techniques, such as statement coverage, are insufficient to fully "test" ML programs: e.g. even if all statements of your ML program are covered, your ML program can still fail, e.g. it may predict the wrong class for some previously unseen input. So, testing ML programs requires not only traditional testing techniques, but also other approaches that attempt to address the (un-)desirable behaviour (such as the generalization) of the models.
Currently, I am also doing research on this specific topic, so I can say that there are already several people working on this topic, but the field is still quite immature and there are still many unsolved problems. It's also important to note that the general "testing problem", even for traditional software, is not yet solved.


Train a model using a multi-column text-filled excel sheet

You will need to convert that to something that a neural network can understand.

Movie Name

Is useless. At least you want to judge a movie by its name.

Description

You will need to perform a Tokenization. Grab the x's more common words and convert them on an array.
I recommend you to see these videos from TensorFlow.
https://www.youtube.com/watch?v=fNxaJsNG3-s&t=18s
There you can find the Google Colab Links for the job.

Director

There are two options. A One Hot Array or an integer for every director.
One Hot Array may be better if you don't have an order of similarity of the directors. But will increase the size of the inputs.
If you have an order of similarity of the directors an integer for each director will work fine.

Rating

Nothing to do here. Is ready to go.
You can perform this work directly on excel, but tensorflow has great tools for it.
It’s hard to run the model after you have trained it on excel.
If you are to comfortable on excel or you can install new software on your computer, I’ve made a backpropagation algorithm that runs on excel and give you a formula to paste in a module.
https://github.com/TorrensJoaquin/Multivariant-Nonlinear-Regression-in-VBA-Small-Neural-Network


Detect data in tables of roughly the same structure

Assuming all of the tables will be oriented in similar ways (label and value running horizontally) and that all writing will be printed rather than handwritten, one solution method would be to use an image segmentation method such as edge detection to segregate these horizontal (label, value) pairs and then use a library like Tesseract for OCR.
There are many types of image segmentation methods that may all have value, but if my assumption holds regarding the neat, structured nature of the tables, then I think simple edge detection methods could be sufficient.


Has reinforcement learning been used to prove mathematical theorems?

Artificial Intelligence for Theorem Proving is an active research area as witnessed by the existence of the AITP conference and of many publications on the topic. Some papers are mentioned in this thread: https://coq.discourse.group/t/machine-learning-and-hammers-for-coq/303. I haven't read these papers myself, so I cannot point you to a paper using reinforcement learning specifically, but given the important activity in this domain, I would be very surprised if it hadn't been attempted.


You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.


LSTM text classifier shows unexpected cyclical pattern in loss

It turns out that the zig-zag pattern is an inherent effect of using a word embedding layer. I don't fully understand the phenomenon, but I believe it has a strong correlation with the embeddings acting as a sort of memory slots, which can change relatively quickly, and the LSTM generating a summary of the sequence, so that the model can remember past combinations.
I found this plot of a training loss curve of word2vec and it exhibits the same per-epoch pattern.

Edit
After conducting several experiments, I've isolated the causes. It seems that this is an indirect effect of having a large model capacity. In my case, I had too large word embeddings (size 1024) and too many classes (2002), which also increases model capacity, so the model was doing an almost per-sample learning. Reducing both resulted in a smooth-as-silk learning curve and a better generalisation.


What other kind of AIs exist apart from goal-driven?

Whether all AI are goal-driven depends on how far you're willing to extend the definition of 'goal'.

AIs are often defined through their primary tasks. Face recognition is a process; a face recognition AI is a limited, special-purpose AI. Paperclip collection is a process, the Paperclip Maximizer is a general, goal-driven AI.

There will always be some sort of goal, but it may in certain cases become so nebulous it's really a stretch to continue calling it a goal. For example,  unsupervised learning (as suggested by Aiden Grossman) - the goal you set is "learn", with very little qualifiers. The underlying AI may develop in surprising and unexpected ways; it may even modify own goals.

Similarly, another "aimless" AI type is a simulator of biological intelligence - say, replicating neural connections of an insect in software, making it run and observing the behaviors. The AI itself is not given a specific goal - it's a tool/subject of study, not something directly producing results. Still, the goals are exactly the same as goals of the original insect: feed, multiply, stay safe; unattainable due to hardware limitations but still pursued.

Therefore, whether all AIs are goal-driven is arguable. The only AI that is not actually driven by any goal whatsoever is one that's switched off. But whether you consider all the possible nebulous, abstract or accidental goals that drive the AI to call it actually goal-driven, that's up to debate.


How do we choose the activation function for each hidden node?

To know the form of your non-linear function,  firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions,  weight decay, momentum and dropout etc.


Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

My understanding from your question is that you have the following data generated from a random policy:
$$[s_0, s_1, s_2 . . . s_n]$$
That is, the state observed at each time step.
You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do not have any observations other than a single list of states.
You wish to obtain an estimate for $Q(s_j, a_j)$, given a specific state/action pair that has not been observed before. This is not possible. You do not have any data from your observations regarding rewards, nor the actions taken by the random agent. At minimum, you would need data about rewards and actions that were observed. Observed rewards are needed to calculate returns, and value functions are expected returns. Observed actions are needed to assign returns to the correct $(s,a)$ pair when calculating estimates.
In addition, if you want to make a meaningful estimate for an as-yet unseen state/action pair, then you would need to be using some form of function approximation for the action value function Q. Otherwise your estimate will be whatever default estimate you had assigned at the beginning of learning.
What inferences can you make given the data you have? Very roughly:
$$p(s_{k+1} | s_k, \pi ) \gt 0, \forall k \in [0,n-1]$$
i.e. the state sequence that was observed gives you proof of non-zero transition probabilities between those states under the given policy. This is not enough to learn a value function, but might have other uses.
With repeated observations - multiple lists of observed states - you could infer what those probabilities were, but would not be able to separate the effect of action choice from state transition rules (because the action was not observed). If you added reward observations then you could learn a state value function.


How can I identify bigrams and trigrams that represent concepts?

One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. Here, you can find more information about it.


Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep function only slows down the program here.

The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.


Is there any situation in which breadth-first search is preferable over A*?

The only general situation that comes to my mind where BFS could be preferred over A* is when your graph is unweighted and the heuristic function is $h(n) = 0, \forall n \in V$. However, in that case, A* (which is equivalent to UCS) behaves like BFS (except for the goal test: see section 3.4.2 of this book), i.e. it will first expand nodes at level $l$, then at level $l+1$, etc., this is because nodes at level $l+1$ are farther away from the initial node than nodes at level $l$, i.e. $f(n') = g(n') > f(n) = g(n)$, for all $n' \in V_{l+1}$ and $n \in V_{l}$, where $g(n)$ is the cost of the shortest path from the initial node to $n$ and $V_{l}$ is the subset of nodes of the search space that belong to level/layer $l$ of the search tree.
So, if you think that your goal is close to the initial node in terms of levels/layers and each node has a small branching factor (i.e. not many children), then BFS may be a good idea (but you could also use A*).
BFS can also be used as a sub-routine in other algorithms, such as the Ford–Fulkerson algorithm. So, in these cases, BFS may also be preferable.


Using tensor networks as machine learning models

I do not feel comfortable proclaiming that I would be a machine learning expert.
But I want to point out that there is indeed interest in applying tensor networks in machine learning settings.
Let me highlight three particular settings in the following.
They can be used to find the governing equations behind dynamical systems analogous to the SINDy algorithm.
The reasoning is that the dynamics of your system may not be sparse but may indeed have a low-rank structure. (See this paper for a comparison of sparse and low-rank structures and this paper for a discussion about why low-rank matrices so often appear in big data.)
The resulting modification of SINDy is the appropriately named MANDy algorithm.
But if your dynamical law exhibits additional structure (despite being low-rank) then this structure is not used. (Requiring you to collect unnecessary samples.)
This work tackles this issue by allowing you to specify the structure that you expect your law to exhibit.
Another area of interest is the application of tensor networks as a means of compression.
Tensor networks can be used to approximate multivariate functions.
Either directly (see this article or this article for further details) or by representing their high-dimensional coefficient tensor (see e.g. here).
The last paper also highlights where such functions may occur: when solving stochastic or parametric PDEs.
"What does this have to do with machine learning?" you may rightfully ask.
These PDEs are hard to solve and it is often easier to minimize a residual than to apply a Galerkin method. The results can be shown to be equivalent with high probability. (See this paper for a proof of this statement and an application to some stochastic PDEs or this paper for an application to an optimal control problem.)
Moreover, there is the problem of tensor completion - a generalization of matrix completion to higher dimensions. (This and this are references for algorithms.)
These methods find application in data science where they can be used to decompose or denoise data. (Unfortunately, I don't have a reference with examples but I remember a presentation where these methods where used on EEG signals.)
Additionally, I recently found this interesting symposium covering even more potential applications of tensor networks in machine learning:
https://itsatcuny.org/calendar/quantum-inspired-machine-learning