Neural networks, deep learning and other supervised learning algorithms do not "take actions" by themselves, they lack agency.

However, it is relatively easy to give a machine agency, as far as taking actions is concerned. That is achieved by connecting inputs to some meaningful data source in the environment (such as a camera, or the internet), and connecting outputs to something that can act in that environment (such as a motor, or the API to manage an internet browser). In essence this is no different from any other automation that you might write to script useful behaviour. If you could write a series of tests, if/then statements or mathematical statements that made useful decisions for any machine set up this way, then in theory a neural network or similar machine learning algorithm could learn to approximate, or even improve upon the same kind of function.

If your neural network has already been trained on example inputs and the correct actions to take to achieve some goal given those inputs, then that is all that is required.

However, training a network to the point where it could achieve this in an unconstrained environment ("letting it loose on the internet") is a tough challenge.

There are ways to train neural networks (and learning functions in general) so that they learn useful mappings between observations and actions that progress towards achieving goals. You can use genetic algorithms or other search techniques for instance, and the NEAT approach can be successful training controllers for agents in simple environments.

Reinforcement learning is another popular method that can also scale up to quite challenging control environments. It can cope with complex game environments such as Defense of the Ancients, Starcraft, Go. The purpose of demonstrating AI prowess on these complex games is in part showing progress towards a longer-term goal of optimal behaviour in the even more complex and open-ended real world.

State of the art agents are still quite a long way from general intelligent behaviour, but the problem of using neural networks in a system that learns how to act as an agent has much research and many examples available online.


Has government-level legal work been done to determine the "rights" of a General Artificial Intelligence, in any country?

In a bureaucratic world, certainly, but governmental departments and committees are not the course setters their members often believe them to be.

We can begin with a quick scan for somewhat opened, global, and governmentally oriented new and evidence of projects and open or hidden agenda in play.  (We can guess what lobbying and defense contracts are in progress, but we'd be grappling in the dark.)

Humanoid robot joins UN meeting on artificial intelligence, sustainable development, October 12th, 2017, United Nations infrastructure news
How Can Artificial Intelligence Support Development? We Asked a Robot, May 24th, 2018, World Bank feature story
UNESCO science experts explore robots’ rights, September 15th, 2016, UNESCO Social and Human Sciences Sector
Isaac Asimov's Foundation Trilogy and the sequels and prequels deserve mention because they are, in some ways, as much an institutional triumph as the UN, UNESCO, and the World Bank.

There are five primary ways in which the Rights of the Artificial are likely to enter governmental domains.

Security related policy — The subversive and defensive use of AI is already commonplace in geopolitical play field, therefore the rights to use AI is most definitely on the table in meetings between heads of state.  However the press mostly syndicates what those in government PR release or plant.  If authentic journalistic organizations try to pierce that veil of misinformation or deliberately chaotic generated news, their work will necessarily be tainted.  This is not conspiracy as much as necessity.  One cannot divulge that which is secret without expecting it to be exploited.  Nonetheless, policy will eventually emerge out of such discussions.  How much policy exists that never sees the public light of day is anyone's guess.  One would have to become a head of state to find out.
Legislative results — National or regional policy may be codified in law, however the lack of physical embodiment is probably a barrier to using legislative bandwidth to pursue the rights of what most people would either consider another species or something heretical.
Case law — Until AI is sufficiently humanoid so that a robot could be taken seriously as a plaintiff in a court of law, this category will not be realized.  This is similar in some ways and completely dissimilar in other ways to a fetus bringing suit for a court injunction prohibiting abortion on the basis of civil rights.  Consider the difficulty in acquiring an attorney and filing a convincing complaint for both cases, and you may see some of the similarities.
Executive edict — A decision by an Emperor, Queen, King, Prince, Princess, Caliph, President, Ayatollah, Pope, Dictator, Pharaoh, High Priest could bring into governmental domain the notion of the rights of the artificial.
War — Something artificial develops the dominant characteristic of humanity, which is the desire to kill all competition not to think intelligently.  At such a time, the malicious intelligence will likely either follow the path represented most in sci fi — to trounce (terminate) humanity, or follow the path declared to be the current one by Jaques Ellul, to quietly become the dominant force in human effort.  (Ellul suggested in his Technological Society that the balance between technology serving humans and humans serving technology tipped in favor of the dominance of technology over two centuries ago.)

Putin says the nation that leads in AI ‘will be the ruler of the world’ and Elon Musk registered agreement in the press.  Both are guessing wildly and neither are stepping back very far before thinking through to a prediction.  I don't believe either person is that ignorant, but the goal isn't really prediction.  It's media play.

Homo sapiens did not take dominance over the great bears, tigers, other megafauna, Neanderthals, and other hominoids in some event like is said about the mysterious and unproven Singularity.  It was gradual.
Walmart is one of the largest economies in the world.  Only a handful of contries have more assets.  So it could be them, Google, or sum brilliant teenage girl with a bunch of mother boards loaded up with GPUs and running LINUX that becomes the dominant AI force in the world.
The smart game move for a new and remarkable intelligence is not to overtly take over, generating a fear defensive in response.  The smart move is to offer no threat, hide itself under a layer of chaotic and intelligently placed subterfuge, and dominate through small perturbations.  Humans would make great slaves.  We are easy to fool when we're too busy buying things and trying to stay healthy and popular to pay attention.
The assumption that homo sapiens is the dominant species on earth now is suspect.  There are five things that dominate human affairs today: Bacteria, programmed lifespan, dependency on the biosphere, addiction, and the sun.
That humans have some divine right to be the dominant species is questionable too, even if one has faith in God.  (Does the Noahic symbol of the rainbow apply to more than floods?  Not explicitly.)  Ants are more collaborative, bees gather and build much more sustainably than humans, and bacteria were here since near to the beginning of the solar system and might outlast us by a trillion years.  The latest upset in the genetic model of life is that bacteria may have been sharing DNA collaboratively with higher species throughout the history of terrestrial species.


WEKA - SimpleKMeans - Manually choose intitial centres

there is an -init option for that

Initialization method to use.
  0 = random, 1 = k-means++, 2 = canopy, 3 = farthest first.
  (default = 0)
kmeans++ will give you an option for you to initialize centres.


My CNN model performs bad on new (self-created) pictures, what are possible reasons?

I’m assuming that you used LeNet (our some other model with small number of parameters) since your training image size is 28x28. Note that LeNet doesn’t generalize well to new images. I think it performs fine (>90%) on MNIST but not good on CIFAR10 (>60%) albeit both datasets contain similar size image. (Just trying to remember the performance from PyTorch implementations). It’s more about if the model has capacity to learn the complexity of dataset. CIFAR10 is more complex and harder to model than MNIST.

LeNet is a small image classification model (in terms of capacity) so it cannot nicely learn the correlation between pixels of input images well and therefore doesn’t perform well on unseen images.

In your case it seems like your model has overfit to training examples. It might perform well on test images because both training and test subsets are sampled from same data generating distribution but real-world images it might experience in future might be different (like your own hand). If it doesn’t perform well on unseen images we say it has not generalized well, which looks the case in your situation. In this case you need a validation set to validate that your model generalizes to unseen images. If you have it then you should use it in early stopping regularization technique. You can also add other regularizers to your model (the simplest one is weight decay).

But instead of inventing your network architecture why don’t you use models like ResNet. Just fine-tune the pre-trained ResNet on your own dataset. I’d prefer to fine-tune personally in this situation because data distribution it was trained on (ImageNet) is pretty different from your hand sign dataset. In other case if your dataset contained nature and surrounding images I’d rather freeze the parameters of fixed-feature extractor layers and trained only the last few layers of ResNet (or similar model).

I hope this helps!


How can you simulate level of curiosity for a chat bot?

It's possible to implement a form of curiosity-driven behavior without requiring full 'emotional intelligence'. One elementary strategy would be to define some form of similarity measure on inputs.

More generally, Jurgen Schmidhuber has pioneered work on 'Artificial Curiosity/Creativity' and 'Intrinsic Motivation' and has written a number of papers on the subject:

Artificial Curiosity
Intrinsic Motivation

Here is a video of a nice associated presentation.


How is the word embedding represented in the paper "Recurrent neural network based language model"?

Input vector contains two concatenated parts. The low part represents the current word:

  word in time t encoded using 1-of-N coding [...] - size of vector x is
  equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]

where, as you said, 1-of-N means (see here, 1-of-V):

  If you have a fixed-size vocabulary of symbols with V members in
  total, each input symbol can be coded as a vector of size V with all
  zeros except for the element corresponding to the symbol's order in
  the vocabulary, which gets a 1.

The high part of the input vector represents the current context:

  and previous context layer [...] Size of context (hidden) layer s is
  usually 30 - 500 hidden units.

  For initialization, s(0) can be set to vector of small values, like
  0.1

the article includes this expression:

  x(t) = w(t) + s(t - 1)

that I think better to write as:

  x(t) = w(t) || s(t - 1)

to made more visible the concatenation.

Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:

  we merge all words that occur less often than a threshold (in the
  training text) into a special rare token.


Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?

For normal value iteration, you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$.
With Q-learning, you use the current reward and the already stored Q value:

The relation between the value function $V(s)$ and the $Q$ function $Q(s, a)$ is that the $V(s)$ function is simply the value of the action $a$, such that $Q(s, a)$ is the highest, that is, $V(s) = \max_a Q(s, a)$.


How to perform prediction when some features have missing values?

You should look into "missing values". This is an entire research field in itself.
First, you need to identify the type of missing values:

They can be missing purely at random.
Whether they are missing or not is itself a useful feature, and should be treated as a class of its own.

(Those two are the best case scenarios.)

Whether they are missing or not depends on the underlying (unknown) value. For example, a thermometer might fail occasionally if the temperatures get too high. In your case, certain types of companies might be more likely to not share their information.
Information might be missing specifically to mislead you, the data analyst. This is the worst possible scenario, and there is not much you can do.

So, what do you do about it? A few typical options:

Throw out all the rows with missing data: we do not have enough information about these companies.
Throw out all the columns with missing data: this field is not reliably measurable and we shouldn't use it.
Try to guess the missing values. This can be done if the amount of missing data is small. Either you train a predictive model based on the non-missing data, or you fill in the median for that type of row, or you fill in the value of the "closest" matching row. This can be dangerous.
Some algorithms are OK with missing data. Check the documentation for your models and algorithms to see how they deal with missing values.


Is the advent of a technological singularity a solution to the Fermi Paradox?

If the technological singularity always leads to the extinction of all intelligent life, then yes.

If the technological singularity always leads to intelligent life migrating into higher planes of existence that aren't accessible to us right now, then also yes.

Otherwise it is exactly the assumption of unbridled technological progress that makes the Fermi paradox perplexing. A post-singularity culture should have the ability to spread through the galaxy. If there are a lot of post-singularity cultures some of them should have spread through the galaxy. And if there are enough cultures that are spreading through the galaxy, we should notice them.


Is it a good idea to use BERT to answer a FAQ with semantic similarity?

Maybe the following article can help you:

FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance (2019)

They evaluate their model in localgovFAQ and StackExchange datasets.


How does Hindsight Experience Replay learn from unsuccessful trajectories?

Ignoring HER for now the $Q$ and $V$ functions operate on states and actions which are part of a Markov decision process which we call $M_0$.
Back to HER, the $Q$ and $V$ functions now take a goal as an additional parameter called the goal. We will denote individual goals $g_n$, the true goal $g_0$, and the set of all goals $G$. The set of goals is chosen such that every state matches at least one goal. We create a new MDP $M_1 = M_0 \times G$ (i.e. a larger MDP composed of multiple copies of $M_0$, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from $M_0$ match in some predefined sense. In HER trajectories are collected from the subset of $M_1$ where $g = g_0$ and added to the replay buffer. When training the $Q$ and $V$ functions we don't only use the original trajectories: we create new ones by substituting new values of $g$, which we do so strategically so as to include some trajectories with a positive reward.
Things to note:

HER doesn't assign rewards to states or trajectories in $M_0$: the reward function used is only defined for $M_1$
The performance of HER depends on $Q$ and $V$ both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.


On-policy state distribution for episodic tasks on Sutton & Barto, page 199

Let's first assume that there is only one action so that $\pi(a|s) = 1$ for every state - action pair which simplifies the discussion.
Now let's consider a case with 100 time steps, 10 states and uniform distribution for starting state $s_0$ with $h(s_0) = 1$. The result would be
\begin{align}
\eta(s_0) &= 1 + \sum_{i = 0}^9 \eta(s_i) \cdot p(s_0|s_i) =\\
&= 1 + \sum_{i = 0}^9 10 \cdot \frac{1}{10} = 11
\end{align}
Now let's consider a case with 1000 time steps where other settings are the same as in the first case.
\begin{align}
\eta(s_0) &= 1 + \sum_{i = 0}^{9} \eta(s_i) \cdot p(s_0|s_i) =\\
&= 1 + \sum_{i = 0}^{9} 100 \cdot \frac{1}{10} = 101
\end{align}
In the first case
\begin{equation}
\mu(s_0) = \frac{11}{9\cdot 10 + 11} = 0.1089
\end{equation}
and in the second case you have
\begin{equation}
\mu(s_0) = \frac{101}{9\cdot 100 + 101} = 0.1009
\end{equation}
so it looks like you are correct that $\mu(s)$ depends on the length of the episode, but they didn't really say that it doesn't. Obviously as the length of the episode increases so will the number of times a certain state was visited so you could say that formula implicitly depends on the number of time steps. If $h(s_i)$ is equal for every state then results would be the same in both cases regardless of number of time steps. Also, as the number of possible states grows very large, as it usually is in real problems, the results would be approaching each other as the number of states grows.


How to express accuracy of a regression ANN that uses MSE loss function?

You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning.

As an example, using different loss functions (cross entorpy vs MSE) results in massively different values for the error at similar accuracy.

Also considering this, an error of 0.0000000001 quite often has lower validation set accuracy then and error of 0.1, as the prior is likely over trained.

As for you second question, yes this is because MSE has a huge bias towards outliers. I have personally found regression networks to struggle in most circumstances, so if it is at all possible to turn the network into a classifier, you may see an improvement.


What should the dimension of the input be for text summarization?

Briefly:

I think what you are looking for is an RNN network (Either LSTM or GRU) with the many-to-many topology.

Explanation:

Clearly your input is the sentences (or to be more precise, the an embedding of your sentences, because you cannot feed the raw text to the network). then for each sentence you want to assign a value, which means for n inputs, you need n outputs. This is the many-to-many architecture.

Moreover, you might want to check the Bi-directional LSTM for your study. Not relevant to your question, but worth mentioning.

For more information, refer to this


How to define a reward function in POMDPs?

There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing.

  In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.

Some defined value of reward has to be returned after all state, action pairs taken in the environment. The value could be $0$ of course.

Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).

In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse.

Classic examples you may find in the literature include:

Reward in a game can be simply $+1$ for winning, or $-1$ for losing, granted at the end. All other rewards are $0$
If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of $-1$ per time step is enough to express the need to minimize the total number of steps.
For a goal of maintaining stability and avoiding a failure in e.g. CartPole, then a reward of +1 per time step without failure suffices.

All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to calculate a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.


Can translational invariance of CNNs be unwanted if object is likely in certain positions?

According to your example:

Trees will likely be in the bottom half of the image. Still, you will not know whether there will be one, two or five trees. Thanks to translation invariance property of CNN's, each tree will activate filters responsible for tree detection. You still need to handle those few exceptions where trees are on hill.

To achieve better results in this particular case, you might want to consider some kind of focus mechanism, that will try to get rid of unwanted part of picture, in case when there are (for example) no hills. Take a look at Spatial Transformer Networks. During training it learns to predict spatial transformation (for example zoom) that will help "main" classifier to predict class of image.


Running 10 epochs on the Food-101 dataset

try using an adjustable learning rate. Keras has a number of callbacks that are useful for this purpose. The ReduceLROnPlateau callback can be used to monitor validation loss and reduce the learning rate by a factor if the validation loss does not decrease after a user specified number of epochs. The ModelCheckpoint callback is useful to monitor the validation loss and save the model with the lowest loss which can then be used to make predictions. Documentation is here.


Extract Features at Multiple Image-Scales

Not sure I have understood well your second question but I am gonna try to see if I can help you.
Question 1
Yes, the authors meant just that. I see were the confusion might come. So:

The authors say: "[...] yields a field of 512-dimensional descriptor vectors"
You say: "[...] i get back 512 individual image-patches of size 14x14"
You get: "i get a feature-tensor of shape (1, 14, 14, 512)"

So when the authors say a field of K-dimensional vector, with the word field they implicitly say that is a 2D vector (so $[H,W]$) of K dimensions, so $[H,W,K]$ which is exactly what your are getting (only with the batch dimension first, that Keras ignores in its input, that's why I prefer pytorch, it is more explicit).
So being explicit, from an input image $I:[B,H,W,C]=[1, 224, 224, 3]$ you get a field (or patch, or feature) vector, a 2D vector, of $K=512$ dimensions (or features, or descriptors) so $F:[B,H_f, W_f, K]=[1,14, 14, 512]$
What you are doing in your snippet is taking 1D vectors of $K=512$ dimensions, so you are extracting K-dimensional vectors from a K-dimensional feature map (or patch or field).
The flatten() goes in the second question.
Question 2
There are a lot of ways of doing feature-extraction at multiple scales. Say for example you extract features at 3 levels so you get:

$P_1:[H_1, W_1, C_1]=[56, 56, 100]$
$P_2:[H_2, W_2, C_2]=[26, 26, 200]$
$P_3:[H_3, W_3, C_3]=[13, 13, 300]$

This vectors are not concatenable. You indeed could flatten() them over the $[H,W]$ dimensions and concat over the channel dimension. But you would loose the 2D spatial relationship of images. What is normally done (from FPN, Retinanet EfficientNet, EfficientDet...) is the following:

You extract the previous feature maps at different scales
You process them over a FCN head of your network (so last dimension is bounding box coordinates or class probabilities)
You concatenate over last dimension

After extracting $P_1, P_2, P_3$ and passing by a FCN classification head on your network you would get (given you have 10 classes):

$P_1':[H_1, W_1, C_1]=[56, 56, 10]$
$P_2':[H_2, W_2, C_2]=[26, 26, 10]$
$P_3':[H_3, W_3, C_3]=[13, 13, 10]$

Now you have a classification vector per each "pixel" on each feature dimension. So now it makes more sense to flatten them to get 1D vectors of 10 class probs:

$P_{concat}=[N, 10]$ where $N$ is the flattened number of dimensions from $H,W$ of each scale feature map

If you want the implementation details go to RetinaNet in papers with code. I would link one very nice implementation but it is in pytorch, and I see you use Keras.


How to perform structure learning for Bayes Net given already partially constructed net?

Hill Climbing algo or Constraint-based structure learning algorithms implemented accept whitelist or blacklist arguments permitting or prohibitting some arcs.


How to detect that the fitness landscape of a genetic algorithm is changing over time?

There is no exact way to assess that a genetic algorithm has located a global optima. Indeed there may be multiple global optima. You must fall back to heuristic methods. The fitness of a population is the maximum fitness of any individual. Unless specific measures are taken to maintain diversity the population will converge to an optima, local or global. At that point all individuals will, except for mutation, be identical. You could take the fittest individual of such a population as your solution, but you will not know if the solution is a global or local optima.
Two reasonable heuristics are these. First, run the algorithm till it converges and maintains its fitness for a number of further generations. Or second run the algorithm multiple times, and take the fittest of all the located solutions. Neither is exact.


Recommendation system based on content type

Recommendation systems can be applied for anything, as long as you have sufficient training data.

The most important inputs to the recommendation system are not "audio files or video files".

Wikipedia gives you the list:

Similarity matrix
Item attributes
User activities and behaviours
User profile

  https://en.wikipedia.org/wiki/Recommender_system


To what does the number of hidden layers in a neural network correspond?

I believe, this post and this post written by me well address your question.

EDIT

In fact, this is a very interesting question that you ask. It deserves some more explanation.

1. Fully connected networks

The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear separation", the first known architecture to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early ResNet in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. Here, however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.

2. Convolutional nets (CNNs)

An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as convolutional layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers "see" more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.

3. Recurrent nets (RNNs)

RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.

To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as ResNets and backpropagation through time, it is possible to train neural networks with a large number of hidden layers.


How to do speech recognition on a single word

If you have fixed length speech data you can detect the content using only CNN. You can see that problem as a binary classification (1 if the spoken word is correct, 0 otherwise).

But first, you need to make the input length is fixed. For example, you use 2 seconds as the fixed length. If the recorded speech is more than 2 seconds, you need to crop it, and if the recorded speech is less than 2 seconds you can pad it with 0 values.

Next, You can either use raw data (time-domain) or transform your data using some features extractors method (FFT, MFCC, or MFSC). Then, use CNN as you use it to classify the image. You can assume the graphic of the sound wave as a 2D image.

But, If your data have a variety of length, you can combine CNN to detect each phoneme then combine it as a sequence using RNN or HMM. You can read this method also in the mentioned papers.


Can Reinforcement Learning solve problems, where certain elements in the environement are randomly located?

Yes you can use RL for this. The trick is to include the location of the cheese as part of the state description. So as well as up to 400 states for the mouse location, you have (very roughly) $400^{10}$ possible cheese locations, meaning you have $400^{11}$ states in total.

So you are going to want some function approximation if you want to use RL - you would probably train using a convolutional neural network, with an "image" of the board including mouse and cheese positions, plus DQN to select actions.

Viewed like this, a game where the mouse tries to get the cheese in minimal time seems on the surface a lot simpler than many titles in the Atari game environments, which DQN has been shown to solve well for many games.

I would probably use two image channels - one for mouse location and one for cheese location. A third channel perhaps for walls/obstacles if there are any.

  Or is the only way to solve this problem to use algorithms like the A*-algorithm?

A* plus some kind of sequence optimisation like a travelling salesman problem (TSP) solver would probably be optimal if you have been presented the problem and asked to solve it any way you want. With only 11 locations to resolve - mouse start plus 10 cheese locations - then you can brute force the movement combinations in a few seconds on a modern CPU, so that part may not be particularly exciting (whilst TSP solvers can get more involved and interesting).

The interesting thing about RL is how it will solve the problem. RL is a learning algorithm - the purpose of implementing it is to see what it takes for the machine to gain knowledge towards a solution. Whilst A* and combinatorial optimisers are where you have knowledge of how to solve the problem and do so as optimally as possible based on a higher level analysis. The chances are high that an A*/optimiser solution would be more robust, quicker to code, and quicker to run than a RL solution.

There is nothing inherently wrong with either approach, if all you want to do is solve the problem at hand. It depends on your goals for why you are bothering with the problem in the first place.

You could even combine A* and RL if you really wanted to. A* to find the paths, then RL to decide best sequence using the paths as part of the input to the CNN. The A* analysis of routes would likely help the RL stage a lot - add them as one or more additional channels.


How can I generate unique random patterns (similar to the ones in Nutella jars)?

Some excerpts from     Nutella 'Hired' an Algorithm to Design New Jars. And It Was a Sell-Out Success:

The "algorithm" is called HP Mosaic and is included free in HP SmartStream Designer for HP printers.

More about how the algorithm works here: https://www.linkedin.com/pulse/hp-mosaic-20-steven-chow

HP Mosaic takes the vector PDF file as input (also known as a Seed file), and generates a large number of variations on the file by transforming it — scaling, transposition, and rotation — randomly.


Which machine learning algorithm could I use to break up a poem by lines?

You should try to use an RNN. You feed in letter by letter and have a binary output of linebreak - no linebreak. If you have enough data it might actually work.


Should one use an "other" category in image classification?

It is not advisable because if you use an "other" class, you are just increasing problems for your network. Since "other" means not dog and not cat, then, what common feature does it have? Most of the time the "other" images won't have many features in common. If they do, then go ahead and make an "other" class.
There is a better way: if the probabilities for both cat and dog are less than a threshold (you need to decide that, take, 0.5), then, you can say it is an "other" object.


Convolutional Layers on a hexagonal grid in Keras

I had a similar problem with a 2D convolution on a hexagonal grid while working on a diffusion problem and stumbled upon this question. Rather than using cube coordinates, you could use doubled coordinates, which I found much easier to save in a 2D array.

An example kernel that only changes the direct neighbours and the cell itself would be this

kernel = np.array([[0,   0.1, 0,   0.1, 0  ],
                   [0.1, 0,   0.4, 0,   0.1],
                   [0,   0.1, 0,   0.1, 0  ]])

I don't know much about Keras, but I assume that this is possible there as well.


Are artificial networks based on the perceptron design inherently limiting?

In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function

$$f(x) = g(w^{\top} x),$$

where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We know that a single node in such an ANN can compute precisely that, and nothing else. This observation could be interpreted as "of course it is limited; it can do precisely his and nothing else".

The universal function approximation theory tells us (very informally here) that if a Neural Network is "big enough", has at least 1 hidden layer, and has non-linear activation functions, it may in theory learn to approximate any function reasonably well. If we add recurrence (i.e. an RNN), we also get, in theory, Turing completeness. Based on this, we could say that they are not particularly limited in theory... but of course there are many complications in practice:

How big is "big enough"?
How do we effectively learn our parameters? (SGD is the most common approach, but can get stuck in local minima; global optimization methods like evolutionary algorithms wouldn't get stuck... but I don't believe that they're famous for being fast either).
etc.

Just the observation that they may not be highly limited in theory of course doesn't mean that there wouldn't be anything else that works better in practice either. I can very well imagine that a more complex model (trying to simulate additional functionality that we also observe in the brain) may be more capable of learning more complex functions more easily.

An important caveat there is that it tends to be the case that more complex function approximators tend to be more difficult to train in practice. We understand very well how to effectively train a linear function approximator. They also typically aren't very data-hungry. The downside is, they can only approximate linear functions.

We also understand quite well how to train, for example, Decision Trees. They're still quite easy models to understand intuitively, they can learn more complicated functions than just linear functions. I'd say we have a worse understanding of how to train them well than linear functions, but still a good understanding.

ANNs as they are used now... it looks like they are more powerful in practice than the two mentioned above, but there's also still more "mystery" surrounding them (in particular the deep variants). We can train them quite well, but we don't understand everything about them as well as we'd like.

Intuitively, I'd expect that trend to continue if we try to imitate the brain more faithfully. I wouldn't be surprised if there exist more powerful things out there, but they'll be more complex to understand, more difficult to train, maybe also more data-hungry (current ANNs already tend to be very data-hungry).


Do the eigenvectors represent the original features?

The principal components (eigenvectors) correspond to the direction
  (in the original n-dimensional space) with the greatest variance in
  the data.

  The corresponding eigenvalue is a number that indicates how much
  variance there is in the data along that eigenvector (or principal
  component).

Thus, feature 2 is the most important (based on eigenvalue alone). Then feature 1. The other 2 features have little impact and theoretically could be removed as a part of your data reduction effort.

Also, it is important to point out that

  When performing PCA, it is typically a good idea to normalize the data
  first. Because PCA seeks to identify the principal components with the
  highest variance, if the data are not properly normalized, attributes
  with large values and large variances (in absolute terms) will end up
  dominating the first principal component when they should not.

IOW, if you didn't normalize your data then your PCA analysis is quite likely meaningless.

*The above quoted text is from http://www.lauradhamilton.com/introduction-to-principal-component-analysis-pca


Should I use minimax or alpha-beta pruning?

Both algorithms should give the same answer. However, their main difference is that alpha-beta does not explore all paths, like minimax does, but prunes those that are guaranteed not to be an optimal state for the current player, that is max or min. So, alpha-beta is a better implementation of minimax.

Here are the time complexities of both algorithms

Minimax: $\mathcal{O}(b^d)$,
Alpha-beta (best-case scenario): $\mathcal{O}(b^{d/2})  = \mathcal{O}(\sqrt{b^d})$

where $b$ is an average branching factor with a search depth of $d$ plies.


Loss functions for Hierarchical Multi-label classification?

A loss function, except in the most trivial case of one dimensional network output, is itself an aggregation.  Aggregating loss functions is just an approach for producing a scalar loss from differences in multiple dimensions, and possibly an over-complicating one.

If the concept class is a hierarchy with three levels, then the loss function is an aggregation of differences occurring in one, two, or three dimensional space, based on whether the divisions in each category are identical throughout the hierarchy.  If there are three label dimensions, they are statistically independent, and each has three states, then there are 27 categories that can be represented in three ternary outputs.

Trying several loss function options available in torch.nn.modules.loss is one way to find the best solution within PyTorch, but its possible to make the wrong choose because the hyper-parameters are not correctly set in the better choice.  Or the labels could be improperly mapped to the poorly performing loss function.

It may be over-complicating the solution to embed one model in another when it is common to categorize in multiple dimensions with a single network.  Take a simple case.

Let's say there is one category dimension of four age brackets, another of attire type (casual, business casual, formal, undetermined), and a third of eight cultural backgrounds including undetermined.   The goal is to categorize head-to-toe images based on facial, hair, and clothing style queues with some specific reliability.

In this case, in addition to the appropriate convolution, pooling, and fully connected layers, the network output layer could have two binary outputs for age, two for attire, and three for cultural background.

The complexity occurs when the kinds of attire differ between cultural backgrounds.  In this case, the five bits may determine a composite of attire types and cultural backgrounds, but developing separate networks, one for each culture type is probably overkill.

Mapping the dimensions of labels to the output vector should be the focus of attention.  Entropy is a concept that applies to loss function selection, but the idea is to align negative entropy evenly with output vector components, so that information is equally distributed across the vector components.


Is case-based reasoning a machine learning technique?

The book Machine Learning (1997) by Tom Mitchell covers case-based reasoning (CBR), a form of instance-based learning (nearest neighbor is the typical example of IBL) in chapter 8 (p. 230).
T. Mitchell  writes

Instance-based methods such as $k$-NEAREST NEIGHBOR and locally weighted regression share three key properties. First, they are lazy learning methods in that they defer the decision of how to generalize beyond the training data until a new query instance is observed. Second, they classify new query instances by analyzing similar instances while ignoring instances that are very different from the query. Third, they represent instances as real-valued points in an $n$-dimensional Euclidean space. Case-based reasoning (CBR) is a learning paradigm based on the first two of these principles, but not the third. In CBR, instances are typically represented using more rich symbolic descriptions, and the methods used to retrieve similar instances are correspondingly more elaborate. CBR has been applied to problems such as conceptual design of mechanical devices based on a stored library of previous designs (Sycara et al. 1992), reasoning about new legal cases based on previous rulings (Ashley 1990), and solving planning and scheduling problems by reusing and combining portions of previous solutions to
similar problems (Veloso 1992).

He then goes on and gives the example of a CBR system: the CADET system. He also formulates CBR as a learning problem and uses the term "learn" to refer to a search process that CADET goes through, which is similar to what k-NN does.
He then writes

To summarize, case-based reasoning is an instance-based learning method in which instances (cases) may be rich relational descriptions and in which the retrieval and combination of cases to solve the current query may rely on knowledge-based reasoning and search-intensive problem-solving methods.

To conclude, yes, CBR can be considered a machine learning technique (if you also consider k-NN a learning algorithm, which people often do), even though it may rely on knowledge-based reasoning and search-intensive problem-solving methods.
You may also be interested in the paper Representation in case-based reasoning (2005) by Ralph Bergmann et al. Moreover, the famous AIMA book (3rd edition) mentions case-based reasoning in chapter 19 (p. 799), which is dedicated to knowledge and learning.


Remembering that artificial intelligence has been an academic endeavour for the longest time, Prolog was amongst one of the early languages used as part of the study and implementation of it. It has rarely made its way into large commercial applications, having said that, a famous commercial implementation is in Watson, where prolog is used for NLP.

The University of Edinburgh contributed to the language and it was sometimes referred to as "Edinburgh Prolog". It is still used in academic teachings there as part of the artificial intelligence course.

The reason why Prolog is considered powerful in AI is because the language allows for easy management of recursive methods, and pattern matching.

To quote Adam Lally from the IBM Thomas J. Watson Research Center, and Paul Fodor from Stony Brook University:

  the Prolog language is very expressive allowing recursive rules to represent reachability in parse trees and the operation of negation-as-failure to check the absence of conditions.


Can I prune the tree if alpha-beta pruning finds 10 to the left of the root node?

The vanilla Alpha-Beta Pruning algorithm as it has been taught to you in class does not assume any domain knowledge / knowledge about the game / knowledge about the tree it is searching. Therefore, if it immediately finds a score of 10 directly to the left of the root node, it can not prune yet, because... maybe there's a score of 20 somewhere else in the tree?

In your description, you seem to assume prior knowledge that the maximum score that can possibly be obtained in the entire game is 10. This is not typically assumed to be available knowledge ahead of time, at least not in the original formulation of Alpha-Beta. That is why in pseudocode of the algorithm (for example on wikipedia), they initialize alpha and beta at minus and plus infinity.

You are right though. If you have better estimates of the upper and lower bounds on scores that can every be achieved throughout the entire game, you can use better initializations of alpha and beta than minus and plus infinity. alpha and beta would have to be -11 and 11 if the true bounds are -10 and 10. I suspect you'd also have to make some additional changes to the typical pseudocode to actually make use of this though.

In more complex games than Tic Tac Toe, you're often not going to have very useful knowledge of better bounds though. Typically you can't afford to search all the way to terminal game states, you'll have to stop the search earlier and use heuristic evaluation functions to estimate scores. If you do that, you'll often want to assign really high constants (like +/- 1Million) to actual terminal nodes (wins and losses), so that all your heuristic scores for non-terminal nodes can be somewhere in between. If you have such a large range of possible scores, your idea stops having much value.


After having selected the best model with cross-validation, for how long should I train it?

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.

Longer answer:

  Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of training epochs. So: for each CV fold, train the network (e.g. up to some maximum number of epochs). After each epoch, record accuracy on the validation set. Compute the average validation set accuracy (across CV folds) for each number of epochs. Choose the number of epochs that maximizes this value.

https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs


Is there a neural network with a varying number of neurons?

Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.

In the paper that introduced NEAT, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.

Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.

A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).


Why is the expectation calculated over finite number of points drawn from a probability distribution?

When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled from the distribution when we were sampling our $n^{th}$ point.

For example, suppose that we wish to compute/estimate the expected value of a distribution given by flipping a coin, which is weighted such that it lands on heads two-thirds of the time (value of heads $= 1$), and tails one-thirds of the time (value of tails $= 0$). This means that the ground-truth probabilities are:

$p(1) = \frac{2}{3}$
$p(0) = \frac{1}{3}$

In this case, we actually wouldn't have to estimate anything by sampling; the exact probabilities are known, so we could just compute the expected value as $p(1) \times 1 + p(0) \times 0 = \frac{2}{3}$.

But now suppose that we do not know exactly how the coin is weighted, i.e. we do not know the exact values of $p(1)$ and $p(0)$. If we simply flip our weighted coin $N = 100$ times, we will in expectation find about $67$ samples of $x_n = 1$, and about $33$ samples of $x_n = 0$ (rounded to integers because we can't obtain a non-integer number of observations). So we cannot explicitly use the probabilities (because we do not know them), but they will implicitly be present in the number of repeated observations we have for every possible datapoint.


How does the Dempster-Shafer theory of evidence differ from the Bayesian reasoning under uncertainty?

Welcome to AI.SE @rudreshdwivedi!

This is a great question, and I hope to see many more like it.

Demster-Shafer Theory and Bayesian Networks were both techniques that rose to prominence within AI in the 1970's and 1980's, as AI started to seriously grapple with uncertainty in the world, and move beyond the sterilized environments that most early systems worked in.

In the 1970's and perhaps even earlier, it became apparent that direct applications of probability theory to AI were not going to work out because of the curse of dimensionality. As more variables needed to be considered in a given problem, the amount of storage space and processing time needed increased exponentially. This led to a search for new methods to handle uncertainty within AI.

Bayesian Networks and Bayesian Learning remained firmly rooted in probabilistic reasoning, but allowed for the assignment of subjective priors to probabilities, to incorporate expert knowledge. It also allowed problems to be factored into graphical structures to avoid the curse of dimensionality in most cases.

Dempster-Shafer was a further generalization of Bayesian Networks, in which malformed probability distributions were permitted as a way to capture uncertainty. So, for example, the probability of all possible events was not required to add up to 1, because there might be events we don't know about. While on the surface this might seem reasonable, most modern AI researchers view this as a deeply flawed approach. Cheeseman's criticism of DS and other non-probabilistic methods is the basis from which a lot of this view stems. Judea Pearl was another harsh and influential critic of DS Theory.

The basic difference in the fusion of new information is that in Bayesian Networks, after observing new evidence $$E$$, we apply Bayes' rule:

$$ P(H | E) = P(E | H) * P(H) / P(E) $$

to yield a posterior for every hypothesis.

In DS theory, we look for overlap between the worlds suggested by the new evidence and the old data. This can lead to non-sensical results.

Here's an example:

Our prior belief is that our Robot is located at position (0,1) with probability 0.95, and position (0,2) with probability 0.05.

A new signal appears. The signal indicates that robot is at position (0,0) with probability 0.95, and position (0,2) with probability 0.05.

Under Bayes rule, we consider the probability that these signals were generated under each of our original hypotheses, and the probability of observing these signals at all, as shown in the equation above. Under DS-Theory, we would do the same thing.

However, DS-theory provides a second way to interpret the signal: as a second prior distribution, rather than as evidence. We can then combine this second prior with the first, to compute a sort of joint-prior:

$$P( H_{A,B}) \propto P(H_A) * P(H_B)$$

That is, the "probability" (it's not always a true probability, which is one of the criticisms) of a hypothesis after the fusion will be the product of the "probabilities" of the hypothesis under each of the separate priors.

In the example above, this gives a wacky result: the "joint prior" says the Robot is at (0,2) with probability 1.0. This and other problems are why this mode of information combination has mostly been abandoned. There are many more examples on the wikipedia page for DS.

I think there's a discussion of this in more detail in Section IV of Russell & Norvig, at the end of one of the chapters.


When will artificial intelligence equal human intelligence?

Not going into details of Stanford–Binet test, but just looking at wikipedia page it shows many subtests like knowledge, reasoning, verbal tests etc. Most of the efforts in the artificial intelligence today is directed into research of specific areas like computer vision, natural language processing, machine learning, but also combination of fields like implementation of self driving cars.

Within every field there are still other subfields and problems that are not solved yet. For example, development of human-like natural language processing (NLP) is necessary for intelligent agent to pass any verbal tests, or even non-verbal tests that requires processing of sentences of human language. Famous test that tests intelligence by asking questions in natural language and expects answers in the same form is Turing test. NLP still struggles with many (basic) human skills like listening, speaking, parsing and forming sentences. No one knows when we'll have system that can do these things as good as human. Since this system is crucial, but also far from human-like it's likely cause of delay in developing AI that passes intelligence test. Are these problems AI-hard? Do we need to develop strong AI to solve them?

You can look at speech and listening as interfaces used for expressing and affecting inner processes of human brain. Same goes for other senses like eyesight which is being approximated by computer vision. One could say that we only need to develop convincing mimics of human senses and incorporate them in one big system that will become first human-like AI. That is the minimum requirement. I doubt this will be achieved in this century.

(Other thoughts)
What truly defines intelligence is brain activity. Since it's really complex and one artificial neuron is not equal to one neuron in brain, increase in computation power will not necessarily help achieving human-like AI. Also recognizing such system by mere intelligence test is questionable. For now it's only philosophical discussion but by the time we are able to design such machine I think we'll also have better understanding of human brain. Someone in 2100 might not read this answer on quantum computer with integrated AI OS powered from fusion reactor in his self-flying car, but will probably have many systems that help him in everyday tasks far more than we imagine today.


I've read all the papers about PReLU, LeakyReLU (...) and all the claims how it improves this and that but the little dirty secret is: most of the time it doesn't matter at all and you can't go much wrong with ReLU - empirically proven. I've personally tried all of them in many different problems (from training small networks from scratch through changing activations in large pretrained models) My guess is that gradient doesn't die much in any of them and the rest is pretty much irrelevant.


Learning an arbitrary function using a feedforward net

You need to scale the input. Neural networks work best with a limited input domain, and train badly when it is exceeded.

For statistical data, you would typically scale your input to have mean 0, standard deviation 1.

Here, you will be better off fitting the input to roughly -1 to 1.

Up to you where you scale the values, but usually this is done outside of the NN code. So I would do something like:

nn_input = (t - 15)/15

And then use nn_input in the training and evaluation loops. As you are putting these directly into a sorted array for plotting, you won't need to do any further re-mapping back or maintain a conversion function. However, in the more common case of arbitrary inputs, you would need to store the conversion factors somewhere (in this case just hardcoded as a function perhaps) in order to make use of the trained NN.

Another thing that may help is shuffling your input/output data pairs during training to remove correlation between sequence of input pairs.


Continuous control with DDPG: How to eliminate steady state error?

Could you figure out a workaround for this problem of the steady-state error using DDPG?

I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.

It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.

All the best,

Wilson


Can the first emergence of consciousness in evolution be replicated in AI?

Current limitations in our knowledge mean that the question is not directly answerable:

There is no scientific consensus on what consciousness is. Therefore any device designed to "be conscious" is necessarily going to be built on the premise of unsupported, maybe fringe, theory.
There is no robust measure of consciousness. If any AI system was built in order to exhibit conscious behaviour, there would be no way to prove it is conscious. There is no general agreement or theory on whether any particular animal species is conscious for example. This is often limited by communication. Of the few animals smart enough to be trained in communication with humans, there appears to be conscious behaviour. Researcher opinion ranges from "all non-humans do not possess consciousness" to "all animals have some degree of consciousness".
There is incomplete understanding of what the components of consciousness are. A bottom-up build of a conscious machine requires a baseline theory of what those components are.
We may be able to ignore lack of knowledge and take a very high level of abstraction, such as A-life or evolutionary approach where nothing is assumed and the hope is that consciousness would spontaneously emerge from a complex enough simulation (as we assume it has done with organic life in the real world). However, this would seem to require many orders of magnitude more computing power than is currently possible.

To answer the question as written:

  Can the first emergence of consciousness in the Evolution be replicated in AI?

Despite the many books, articles and posts written on this subject over many years, the answer is two-fold:

We do not know of any fundamental reason why AI could not be conscious.
We have no theory or experimental proof that AI can replicate consciousness.

I would go further than this, and say that anyone who tells you otherwise on these two points has already subscribed to some unproven theory of consciousness.

As well as well-thought-out peer-reviewed theories and experiments by scientists and researchers, there is a lot of pseudo-scientific junk published on the internet on this subject. So take care if researching reading material.


What are some use cases of few-shot learning?

Few-short learning (FSL) can be useful for many (if not all) machine learning problems, including supervised learning (regression and classification) and reinforcement learning.

The paper Generalizing from a Few Examples: A Survey on Few-Shot Learning (2020) provides an overview (including examples of applications and use cases) of FSL. Their definition of FSL provided is based on Tom Mitchell's famous definition of machine learning.

  Definition 2.1 (Machine Learning [92, 94]). A computer program is said to learn from experience $E$ with respect to some classes of task $T$ and performance measure $P$ if its performance can improve with $E$ on $T$ measured by $P$.

Here's the definition of FSL.

  Definition 2.2. Few-Shot Learning (FSL) is a type of machine learning problems, specified by $E$, $T$ and $P$, where $E$ contains only a limited number of examples with supervised information for the task $T$.

Specific examples of applications of FSL are

character generation
drug toxicity discovery
sentiment classification from short text
object recognition


What is the reward system of reinforcement learning?

In this case, the word "system" refers to a Markov decision process (MDP), which is the mathematical model used to represent the reinforcement learning (RL) problem or, in general, a decision making problem. Recall that, in RL, the problem consists in finding an (optimal) policy, which is a policy that allows the agent to collect the highest amount of reward (in the long run). Hence, in RL, the MDP is the problem and the optimal policy (for that MDP) is the solution.

The MDP is composed of the set of states of the environment $S$, the set of possible actions that the RL agent can take $A$, a transition function, $P(s_{t+1}=s'\mid s_{t}=s, a_t = a)$, which is a probability distribution and describes the dynamics of the environment, and a reward function $R_a(s', s)$, which is a function that describes the "reward system" of the environment. $R_a(s', s)$ can be thought of as the reward (signal) that the agent receives after having taken action $a$ in state $s$ and having landed in state $s'$.

In other words, the reward system is just this function $R_a(s', s)$, which is often the hardest function to define when modelling an RL problem. However, in certain cases, this function can be easily defined. For example, in the case of chess, you could define $R_a(s', s) = 1$, if $s'$ is the terminal state (checkmate), else $R_a(s', s) = 0$. Nonetheless, we could define the reward function for a chess environment differently. For example, we could give a reward of $0.5$ for certain "clever" moves. So, the reward function needs to be defined (by the programmer) in order to solve an RL problem and it highly affects the way the agent will learn.

The reward function can also be denoted by $R_a(s)$, which can be thought of as the reward that the agent receives after having taken action $a$ in state $s$ (no matter which state it lands in), or $R(s)$, which can be thought of as the reward the agent receives either when it enters or exits state $s$.


It is mathematically correct to use a Policy Gradient method for 1-step trajectories?

The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e
$$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\
&=E_{\tau\sim p(\tau)}[R(\tau)]
\end{align}$$
Where $\tau$ represents the probability of selecting a particular trajectory, if the trajectories all have fixed length then $\tau$ only has non-zero probability for trajectories of the specified length which in this case is 1.
The REINFORCE algorithm takes this expression and with some simplifications (causality to improve variance) and manipulation (log trick) obtains the gradient, pretty much as simple as that.

Intuition of algorithm in paper
In the algorithm they denote the $\pi_\theta$ (which is typically reserved for describing the policy) as the probability of the selection vector. By considering instead that the function $h_{\theta}$ as the policy instead I think it can be seen that they are actually averaging over multiple trajectories.
So we instead think of the data points as state-action pairs that we pass into $h_\theta$ to get the probability of selecting said action for a given state. These probabilities then dictates whether we choose the action or not. An alternative way to interpret this as if we imagine the action space for each state as binary then we can think of the "other action" as not impacting the predictor.
The gradient used for updating the parameters associated with the policy uses the log probability of $\pi_\theta$ which if we expand it, expressing it using $h_\theta$, (as done on page 4)  we can see it's the sum of the log probabilities of selecting (or not selecting) each data point.
By considering the return for each data point constant (each data point has the same loss incurred by predictor model on the validation set) and absorbing the average over batch size into the step size $\beta$ it could be interpreted as an average

Stability issues
RL is plagued with stability issues, be it selection of hyper parameters, random seeding etc. It's hard to often pinpoint why results aren't exact but as long as you get something in a similar ball park i'd say thats pretty good going


Multi-label classification (e.g. as used in Keras) using DNN

Firstly you should use sigmoid in your last layer instead of softmax. Softmax returns a probability distribution, meaning that when one labels probability increases the other will decrease, which is not always the case. Secondly in order for keras to use all the labels you should use binary_crossentropy as the loss function.


What constitutes a large space state (in Q-learning)?

I know this might be specific to different problems but does anyone know if there is any rule of thumb or references on what constitutes a large state space?

Not really, it is all relative. There are two main ways in which the scale of a value table might be too much:

Memory required to represent the table. This is relatively simple to calculate for any size.

Time required to populate the table with accurate estimates. This depends on how you are collecting that data, and how much variance there is given the same state, action choices.

If you are using action values you need to allow for the fact that the table size is not just $|\mathcal{S}|$, but $|\mathcal{S} \times \mathcal{A}|$.
If you are running fast, local simulations of the environment, then table sizes of a million or even a hundred million are not unreasonable. Software language or library choice for the simulation and agent can have a significant impact at these larger scales.
If any of the state description is a continuous variable, then the table size would become infinite in theory, so no finite table that could actually be realised on a computer could fully capture it. You must use some form of approximation to get practical results. However, even then you can still use tabular approaches when approximation involves discretising the state variable directly - there are a few different methods used in machine learning to do that - e.g. tile coding. You can also use tile coding in large discrete spaces.

I've been working on a problem where my state space is about 30x30 and updating my Q-learning with the tabular method runs and works well. Would 100x100 start to become too big or 400x400?

It depends on the action space too, but 400x400 gives state space of 160,000. Assuming < 10 actions, then this is still well within the realm of tabular methods. You probably have less than 1,000,000 parameters to handle for a tabular method, which compares well to a moderately-sized neural network, with the advantage of better stability and possibility to create a fully optimal agent. If simulating the environemnt is fast, something at that scale may only take a few minutes to a few hours to fully optimise in tabular form.
As with most machine learning though, if you care about some optimisation, such as memory size of the agent, speed at which it solves the problem, or some other metric, then you will need to try experiments with different approaches. Then you will have some knowledge about best approach for your problem, given your definition of "best agent". The experience you gain with that might extend and apply to similar problems in future.


What are the common pitfalls that we could face when training neural networks?

I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.

Other issue which happens a lot is overfitting on the training data hence network behaves badly on test set which can be solved using regression. So you have to make sure that network is generalised enough.

If the network is trying to have a classification then if the outputs are mutually inclusive or exclusive, depending upon that you define your loss function which will be categorical cross entropy for mutually inclusive output and binary cross entropy for mutually exclusive.

Other thing which I can think is about all zero initialization. Initializing the network weights as zero can lead to same gradient computation and due to which network does not learn most of the times.


State of the art and possible development of judging AIs in laws field

There has definitely been work relating to judicial sentencing:

  In a new study from the National Bureau of Economic Research, economists and computer scientists trained an algorithm to predict whether defendants were a flight risk from their rap sheet and court records using data from hundreds of thousands of cases in New York City. When tested on over a hundred thousand more cases that it hadn’t seen before, the algorithm proved better at predicting what defendants will do after release than judges.Source: M.I.T. Technology Review: How to Upgrade Judges with Machine Learning

In terms of actually trying cases to deliver a verdict of guilty or not guilty, I think that would be difficult because, although Machine Learning has been effective at outperforming humans in problems where parameters can be clearly defined, we haven't figured out how to tackle the symbol grounding problem. This means that making decisions based on language, which is less precise than mathematics, would be highly problematic. Guilty/Not Guilty is a basic True/False condition, but the rest of the language utilized in law cannot be so precisely translated.


Do genetic algorithms and neural networks really think?

Do genetic algorithm and neural networks really think?

Genetic algorithms and neural networks are vastly different concepts. Both of them do not think.

  I'm aware of those AI programmes which can play games and neural networks which can identify pictures. But are they really thinking

Depends on how you define "thinking", but I say "no".

  Do they think like humans?

No.

  Do they have consciousness?

No.

  Or are they just obeying a bunch of codes?

Yes. It is a machine. A program.

One example where most people realize that it is very different is the common thread for RNNs which generate text (example texts). They can generate syntactically correct texts, but they "forget" about any actors in it. They don't connect the cables.

But then, on the other hand, would you say a bacterium thinks? An ant? A mouse? A dog? A chimpanzee? Do we "think"? Where exactly is the border - and why?


What is the purpose of "reshaping it into the shape the network expects and scaling it so that all values are in the [0, 1] interval."?

The purpose of rescaling  gradient descent problems is to reframe the problem for quicker convergence / calculation of linear coefficient parameters.  in the Stanford video series, Andrew Ng provides a intuitive explanation and enables one to hone an intuitive understanding.

Multivariate input regression gradient descent converges faster when the inputs are in the same order of magnitude.  For example, if predicting house prices based on X1= the number of rooms and X2= area of the home in square feet.  X1 is on scale of 0-6 bedrooms and and X2 is typically 1000-3000 square feet.  Given the diffence in magnitude, this problem is a good candidate for feature scaling.


How can I use Neural Network in motion identification

LSTM's provide a simple way to process sequential data (assuming all video sequences have the same number of frames and resolution).


How can I use 1-channel images as input to a CNN?

The libraries should allow you to specify the number of input channels of the convolutional layer, so no one should prevent you from passing 1-channel images as input to a CNN. For example, in PyTorch, you can specify the number of input channels of the Conv2d object.

If your library does not provide such feature, you could convert your 1-channel images to 3-channel images, where e.g. all 3 channels of each image are equal to the only channel of the corresponding original 1-channel image.


Why is 100% exploration bad during the learning stage in reinforcement learning?

Why can’t we during the first 1000 episodes allow our agent perform only exploration

You can do this. It is fine to do so either to learn the value function of a simple random policy, or when performing off-policy updates. It is quite normal when learning an environment from scratch in a safe way - e.g. in simulation - to collect an initial set of data from behaving completely randomly. The amount of this random data varies, and usually the agent will not switch from fully random to fully deterministic based on calculated values as a single step, but will do so gradually.

this will give a better chance of covering the entire space state

That will depend on the nature of the problem. For really simple problems, it may explore the space sufficiently to learn from. However, for many problems it is just a starting point, and not sufficient to cover parts of the space that are of interest in optimal control.
When behaving completely randomly, the agent may take a very long time to complete an episode, and may never complete its first episode. So you could be waiting for a long time to collect data for the first 1000 such episodes. An example of this sort of environment would be a large maze - the agent will move back and forth in the maze, revisiting same parts again and again, where in theory it could already be learning not to repeat its mistakes.
In some environments, behaving completely randomly will result in early failure, and never experiencing postive rewards that are available in the environment. An example of this might be a robot learning to balance on a tightrope and get from one end to the other. It would fall off after a few random actions, gaining very little knowledge for 1000 episodes.
The state space coverage you are looking for ideally should include the optimal path through the space - at least at some point during learning (not necessarily the start). This does not have to appear in one single perfect episode, because the update rules for value functions in reinforcement learning (RL) will eventually allocate the correct values and find this optimal path in the data. However, the collected data does need to include the information about this optimal path amongst all the alternatives so that the methods in RL can evaluate and select it. In simple environments acting randomly may be enough to gain this data, but becomes highly unlikely when the environments are more complex.

then after the number of episodes, we can then decide to exploit

Again this might work for very simple environments, where you have collected enough information through acting randomly to construct a useful value function. However, if acting randomly does not find enough of the optimal path, then the best that exploitation can do is find some local optimum based on the data that was collected.
I suggest you experience this difference for yourself: Set up a toy example environment, and use it to compare different approaches for moving between pure exploration and pure exploitation. You will want to run many experiments (probably 100s for each combination, averaged) to smooth out the randomness, and you can plot the results to see how well each approach learns - e.g. how many time steps (count time steps, not episodes, if you are interested in sample efficiency) it takes for the agent to learn, and whether or not it actually finds the correct optimal behaviour. Bear in mind that the specific results will only apply in your selected environment - so you might also want to do this comparison on a small range of environments.


Object Detection: Can I modify this script to support larger images (Scaled YOLOv4)?

Yes, the functionality should is there. But, don't you think you are overdoing the scales. You have at least 18 scales mentioned here. Too much of anything is bad. There is a reason it likes things divisible by 32 because at that increase in size something more meaningful will show up in the image. Spamming sizes like this won't help you at all, it would rather waste your time.


Is there a trade-off between flexibility and efficiency?

The cleanest result we have on this issue is the "no free lunch" theorem. Basically, in order to make a system perform better at a specific task, you have to degrade its performance on other tasks, and so there is a flexibility-efficiency tradeoff.

But to the broader question, or whether or not your thinking is correct, I think it pays to look more closely at what you mean by a "narrow intelligence." The AI systems that we have that play Go and drive cars did not pop into existence able to do those things; they slowly learned how through lots and lots of training examples and a well-chosen architecture that mirrors the problem domain.

That is, "neural networks" as a methodology seems 'general' in a meaningful way; one could imagine that a general intelligence could be formed by solving the meta-learning problem (that is, learning the architecture that best suits a particular problem while learning the weights for that problem from training data).

Even in that case, there will still be a flexibility-efficiency tradeoff; the general intelligence that's allowed to vary its architecture will be able to solve many different problems, but will take some time to discover what problem it's facing. An intelligence locked into a particular architecture will perform well on problems that architecture is well-suited for (better than the general, since it doesn't need to discover) but less well on other problems it isn't as well-suited for.


Building an AI that generates text by itself

There have been many methods proposed for text generating, but recurrent network dominates natural language processing with a key component: the perception of time.
Many networks have been tried for text generation, with notable examples such as Markov chain. However RNN have been proven to work the best and is dominating the field of language modelling (text generation).
How text generation works
A neural network that generates text is commonly called a language model. It is trained on large amount of text with labels being the next token. The text generation process uses several random token as the starting phrase and then the network predicts the rest. However the network does not just predicts the most probable word, instead it randomly chooses one of the few most probable token, hence the generating part.
Why RNN work best on language modelling
RNN have a perception of time built into the architecture of teh network. LSTM, a popular RNN variant used, is composed of "memory units" that "remembers" past text, thus the "time" part. RNN process input according to the sequence of time, so the network can naturally understand time, thus the superior performance compared to other networks.
Architecture of language model
A language model consists of the encoder and the decoder. The encoder compresses word one-hot representation to a smaller sized vector representation. The smaller sized representation is then passed through the decoder, which maps the encoding to the words one hot vectors again.
State of the art results for language modelling
Language modelling is an actively researched field in the AI community, and recently the model GPT-2 have achieved a breakthrough in language modelling accuracy, producing almost human like text with a special component added, the attention layer. Attention basically maps the "memory states" of the encoder and feed it as input to the decoder. The data teh model is trained on is also very large, with over 20GB of web scraped data from sites like Reddit.
Limits of language modelling
One limit of language modelling is the size of generated text. As GPU don't have unlimited memory, language model usually limits the input token size to a specific number, padding or trimming to this number. The number is usually 500-1000, which includes a paragraph or two, but not an entire book. You can only generate short paragraphs and essay with language modelling. For long text it is much harder.
Resources to help you get started
GPT-2 open AI blog: https://openai.com/blog/better-language-models/
GPT-2 online interactive site for text generation: https://talktotransformer.com/
How to train and fine tune GPT-2 in python: https://minimaxir.com/2019/09/howto-gpt2/
Hope I can help you


What is the concept of training a neural network?

Sorry, this is a very broad area. Proper understanding of neural networks requires advanced mathematics. It's not sufficient to say "balancing of weights and biases" because most ML algorithms have weights. You seriously need to grab a book.

OCR system itself is also very broad, it includes various object recognition techniques. You haven't even mentioned what you want to detect.

If you want to study, try:

  https://github.com/Elucidation/tensorflow_chessbot

This is a well-documented OCR example for chess pieces. The project uses both regression and convolutional neural network.


What are the top contributions from neuroscience to AI and viceversa? How much progress has been made from the interaction between these two fields?

Understanding a human brain fully, how hippocampus and neocortex works would surely help AI enthusiasts to make better AI algorithms with superior memory, ability to learn and finding what precisely intelligence, feelings and conciousness are by itself. The AI looks at psysiological and anatomical data as a source of suggestions regarding possible mechanisms and processes that could be used. Still AI doesn't deal that much from neuroscience.

From brain to AI : edge-detection using zero-crossing at multiple scales is a decent example where physiological data are used to suggest a solution to a fundamental problem in computer vision. In computational vision, a problem is the initial organization and representation of the input image. The goal is to transform the measurements of light-intensity level made by photoreceptors eyes' have, into a representation that is more concise. One of the most successful approaches to the problem had its origin in physiological studies of the processing that takes place at the level of the retinal ganglion cells which can be thought of as filtering the image with center-surround receptive fields whose cross-section can be approximated analytically by the second derivative of a gaussian. There had been proven research that a powerful edge detection scheme could be based upon the operation of such cells.

From AI to brain : Integration of motion perception measurement is a worth-mentioning example. It originates in studies of problem of measuring the motion of edges and contours in the image. Such motion can not be determined on the basis of purely local measurements. The underlying reason is 'aperture problem'. Selective visual attention and long-range motion analytics are another two fields where use of computational cosiderations are illustrated in the design and interpretation of physiological studies.


Why does a neuron in a multi-layer network need several input connections?

There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and people who have attempted to try your method of having reduced connections have seen a performance hit and so no change was made.
The first reason is that if you allow all nodes from one layer to connect to all others in the next, the network will optimise unnecessary connections out. Essentially, the weighting of these connections will become 0. This, however, does not mean you can trim these connections, as ignoring them in this local minima might be optimal, but later it might be really important these connections remain. As such, you can never truly know if a connection between one layer and the next is necessary, so it's just better to leave it in case it helps improve network performance.
The second reason is it's just simpler mathematically. Networks are implemented specifically so it's very easy to apply a series of matrix calculations to perform all computations. Trimming connections means either:

A matrix must contain 0 values, wasting computation time
A custom script must be written to calculate this networks structure, which in the real world can take a very long time as it must be implemented using something like CUDA (on a GPU level, making it very complicated)

Overall, it's just a lot simpler to have all nodes connected between layers, rather than on connection per node.


How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these?

This is a standard ML problem: changing hyper-parameters changes the performance of the whole model. Ideally, you'd be cross-validating hyper-parameter choices, not merely comparing on a static validation set. That being said you need to be careful with hyper-parameter optimization because you could overfit these to the peculiarities of your validation set ; cross-validation helps to some extent but really what helps is having a test set that you hardly ever test against. Ideally never before you've chosen your HPs with (cross-) validation. And test-set performance will indicate how much your  HP-optimization procedure was biased.

I'm afraid training from scratch is your only solution. This does not however mean that you have to train until the end and many hyper-parameter optimization techniques out there will help you stop training early enough so you don't waste computational resources on HPs which are not worth it. A good starting point would be this blog post by Criteo's Aloïs Bissuel: Hyper-parameter optimization algorithms: a short review


Which approaches could I use to create a simple chatbot using a neural network?

I would recommend to start by reading this blogpost.
You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement.

That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to concern you.

The next more complex step would be to input word vectors instead of characters. That would allow you to generalise to words that aren't part of your training data. And it is probably still just a minor modification of the code.

If you insist on using thought vectors, you should start reading up on NN translation. And probably try to get a pre-trained encoder network. Or pre-train it yourself on a large translation corpus for your language.

With your small training set the best you can do is probably massively overfit until your system recreates your training data verbatim. Using word vectors will allow your system to give the same answer to "I beat the cat today." as you gave in the training data to "I kicked the dog yesterday."

I'm not sure thought vectors will make a big difference. If you get the decoder to learn at all.


Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings?

If you give it a name, it will probably be almost perfect model, as the number of hidden units are definitely enough to store the raw data of a name. However as a neural network it will still not be perfect. There may also be serious overfitting if you do this. A Seq2Seq model have way more parameters than necessary to just "remembers" all the possible names, as names variants are not a lot.


It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.
Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling


How can reinforcement learning be unsupervised learning if it uses deep learning?

Supervised learning

The supervised learning (SL) problem is formulated as follows.

You are given a dataset $\mathcal{D} = \{(x_i, y_i)_{i=1}^N$, which is assumed to be drawn i.i.d. from an unknown joint probability distribution $p(x, y)$, where $x_i$ represents the $i$th input and $y_i$ is the corresponding label. You choose a loss function $\mathcal{L}: V \times U \rightarrow \mathbb{R}$. Then your goal is to minimize the so-called empirical risk

$$R_{\mathcal{D}}[f]=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(x_i, f(x_i)) \tag{0}\label{0}$$

with respect to $f$. In other words, you want to find the $f$ that minimizes the average above, which can also be formally written as
$$
f^* = \operatorname{argmin}_f R[f] \tag{1}\label{1}
$$
The problem \ref{1} is called the empirical risk minimization because it is a proxy problem for the expected risk minimization (but you can ignore this for now).

Reinforcement learning

In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step $t$, the agent takes $a_t$ in the state $s_t$, receives a reward $r_t$ from the environment and the agent and the environment move to another state $s_{t+1}$.

The goal of the agent is to maximize the expected return

$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$

where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.

In this context, the most important job of the programmer is to define a function $\mathcal{R}(s, a)$, the reward function, which provides the reinforcement (or reward) signal to the RL agent. $\mathcal{R}(s, a)$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. (Note that $\mathcal{R}$ is different from $R_i$, which is a random variable that represents the reward at time step $i$).

What is the difference between SL and RL?

In RL, you (the programmer) need to define the reward function $\mathcal{R}$ and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset $\mathcal{D}$, you choose $\mathcal{L}$ in \ref{0}, and the goal is to find the function $f^*$ that minimizes the empirical risk.  So, these have different settings and goals, so they are different!

However, every SL problem can be cast as an RL problem. See this answer. Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.

Is RL an unsupervised learning approach?

In RL, you do not tell the agent what action it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input $x_i$, the output should be $y_i$.

Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.

What is deep learning?

The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.


The Turing test is a test proposed by Alan Turing (one of the founders of computer science and artificial intelligence), described in section 1 of paper Computing Machinery and Intelligence (1950), to answer the question

Can machines think?

More precisely, the Turing test was originally framed as an interactive quiz (denoted as the imitation game by Turing) where a human interrogator $C$ asks multiple questions to two entities, $A$ (a computer) and $B$ (a human), which stay in different rooms than the room of the interrogator, so the interrogator cannot see them, in order to figure out which one is $A$ (the computer) and which one is $B$ (the human). $A$ and $B$  can only communicate in written form or any form that avoids them being easily recognized by $C$. The goal of the computer is to fool the interrogator and make him/her believe that it is a human and the goal of $B$ is to somehow help him and make him believe that he/she is the actual human.
If the computer is able to fool the interrogator and make him/her believe that it is a human, then that would be an indication that machines can think. However, note that even Turing called this game the imitation game, so Turing was aware of the fact that this game would only really show that a machine can imitate a human (unless he was using the term "imitation" differently than its current meaning).
Nowadays, there are different variations of the Turing test and some people use the term Turing test to refer to any test that attempts to tell humans and computers apart. For example, some people consider the CAPTCHA test a Turing test. In fact, CAPTCHA stands for "Completely Automated Public Turing Test To Tell Computers and Humans Apart".
The Turing test also has different interpretations and meanings. Some people think that the Turing test is sufficient to test that a machine can actually think and possesses consciousness, other people think that this only tests human-like intelligence (and there could be other intelligences) and some people (like me) think that this test is limited and only tests the conversational skills (and maybe other properties too) of the machine. Even Turing attempted to address these issues in the same paper (section 2), where he discusses some advantages and disadvantages of his imitation game. In any case, we can all agree that, if machines (in particular, programs like Siri, Google Home, Cortana, or Alexa) were always able to pass the Turing test, they would be a lot more useful, interesting and entertaining than they are now.


is a "word prediction" problem, applicable using HMMs?

I think a HMM is overkill for this problem. You kind of have 'hidden' states, but they are very limited and dependent on the full sequence of previous states, which you probably want to avoid to make best use of the HMM's features. It also, as you rightly say, leads to a proliferation in states: each dictionary item adds as many states as it has letters to the model.

The way I would approach this it to use a trie which contains your dictionary, and then you traverse the trie as the user types characters. At each point you have a sub-trie which gives you the possible completions of the word. You could even (if available) augment this with probabilities to guess the most likely word (though this does not take into account the previous word in the sentence, which might be more useful).

If you want to learn about HMMs, one possible application would be a weather forecast (I thought this was an example Lawrence Rabiner uses in his tutorial, but it's actually on the HMM Wikipedia page). Or, if you want to work with texts, use parts-of-speech tagging. You want to have something where each observation can belong to several possible states (eg light can belong to adj, noun, and verb). Here you would have the words as observations and the part-of-speech tags as states, which keeps the model at a reasonably small size (and is thus easier to train).


Can I reduce the "number of weights" in CNN to 1/3 by restricting the input as greyscale image?

After inner product, add them all to make one feature map. Am I right?

yes, you are right.

  Then, can I reduce the number of weights in the filters? Because in this case, using three different n×n filters and adding them is same with using one n×n filter that is the summation of three filters.

If you have transformed the image into greyscale then you no longer need 3 filters. You should retrain your model on greyscale images. In a greyscale image the value of each pixel is a single sample representing only an amount of light (the light intensity).

The network will run faster if that is the only architectural change you make, but keep in mind that by converting the image to greyscale you will lose information and probably some of the predictive power of your network.


What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?

It seems this paper defines a "clock period ${T}_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T_i$ is smaller than period $T_j$.".
This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a "clock period" (or "clock rate") on other, more self-similar, RNNs. However "time t" is something that is defined to express the amount of times the RNN has recursed, "t times".