Non-Convex Loss Function in Deep Learning Is a Big Deal?

Optimization

In optimization, the loss function (sometimes called the error function) is a function that aggregates the disparity between actual and ideal behavioral states in multiple dimensions and over a sequence of input cases.  In re-entrant (reinforced) learning, a feedback scalar or vector acts as a corrective signal that can replace or further aggregate with the loss function and additionally impact the training back-propagation.

Convergence

In any of these architectures, including systems without NN components but attempt to adapt by converging on an optimal static or moving target behavior, one generality can be made.  As the goal state is approached, the probability of successful convergence increases if the size of each incremental estimation decreases.  The estimation is nothing more than an informed guess.

Decreasing the learning rate as the detected convergence value improves is one strategy being used experimentally if not in production, but that strategy has drawbacks when used alone.  Involving the loss function in slowing down as the destination approaches is a best practice.

Biology Analogy

A biological system of a similar nature is the human subjective experience of pain.  As the pain level goes down, the human brain cares less about the pain, therefore the steps taken to reduce it decrease and eventually vanish.  Evolution has proven such to be advantageous for the same reason.

The Mathematics Involved

Maximizing the probability of an unknown function to be learned adequately to perform well is done in NNs by convergence.  The term gradient descent is often used to describe the iterative process intended to converge on some ideal characterized by labeled data, a concept built into the network, or some fitness signal.  The likelihood of converging on at least a local minima (which may or may not be the global minima) is much higher if the slope decreases as the minima is approached in successive approximation scenarios.  This is when d2E / dt2 is positive.

The geometric idea of convexity is correlated to the calculus concept of the second derivative of a line or surface with respect to time or some other measure of forward progress.  In successive approximation, the independent variable that measures forward progress could be time, computing cycles, the index of the training sample, iteration number, or some aggregation of these.  The second derivative is the rate of the rate of change.

Convergence is more likely if the rate of change decreases as the disparity between what is perceived as optimal behavior and what is the actual current behavior.  In other terms, the relationship between risk in making the next adjustment to circuit (NN) behavior should approach zero as proximity to the optimal behavior approaches zero.  (If one senses they have worked there way to close proximity of their desired state, it makes no sense to make wild guesses.)

An Easy to Visualize Analogy

If you drop a rubber ball into a rigid cone, it will take time to reach thermodynamic rest, at the bottom.  A lossless ball (considered impossible) will bounce forever.  A paraboloid (parabolic in two dimensions like a solar reflector) will produce faster convergence with the same rubber ball because the ball drops in energy (the sum of kinetic and potential energy) with each bounce.  The trajectory does not overshoot the bottom nearly as much or as frequently.  This analogy is not perfect, but it provides a visual image without a diagram.

If you aggregate your disparity between your target trained behavior and the current in-training behavior in a way where the second derivative is negative (concave loss function with respect to distance) on either side of the targeted ideal, convergence is much less likely.  In the analogy, the rubber ball is likely to bounce out of the flared cone altogether.  A lossless ball will always bounce out eventually.

A more provincial analogy is that it would be like trying to catch a baseball with the back of a baseball glove.

Concave, Convex, or Zero Second Derivative

Whether continuous or discrete, convex functions converge much more frequently and usually with less time and computing resource than concave ones.  A second derivative of zero is in the middle.  The word linear is actually incorrect for this zero acceleration case.  The correct term is first degree polynomial.

Sum of squares over the dimensions of the domain (inputs) and over the sequence of input cases will perform well in many cases.  If you were to sum the square roots of the absolute value of error instead, your NN will rarely converge at all.

Executive Summary

The following three things are more likely to be favorable if you find a way to aggregate your disparity between ideal and actual current behaviors in a way where the second derivative is greater than zero.

Reliability of eventual convergence
Speed of convergence
Response time in the case of re-entrant (reinforced) learning
Savings of computing cycles
Reduction in the complexity of introspection
Conservation of computational memory (RAM or SDD)
Conservation of space needed for persistance and archiving
Reduction of project cost to the business


When should the iterative deepening search and the depth-limited search be used?

If your search tree's branching factor is finite but its depth is unbounded, then DFS isn't "complete". What that means is that it may not ever find the goal node even if it exists. You would use these techniques precisely when this situation arises.

To understand this, let's say you're at the node A in the following tree:

     A
    / \
   /   \
  B     C
 / \   / \
.   . .   .

You choose to explore subtree rooted at B. Now if there are no goal nodes in that subtree, but the goal nodes were in the subtree rooted at C, then we're stuck. Because  we will keep searching there because we don't know beforehand that there are no goal nodes there, next node we explore could always be a goal node. But this essentially means that we'll never finish searching the subtree rooted at B in finite time, and never backtrack to explore the subtree rooted at C.

So to overcome this you would want to iterative deepening depth-first search or depth-limited search.

In depth-limited search you set a limit on the depth you'll explore. Let's say you choose $d$ as the depth limit because you know that the solution lies within that depth limit.

And let's say the branching factor for your tree is $b$. Going back to our example, you have finite elements in the subtree rooted at B (at most $b^d$, can you reason why?), so we'll be able to look at all of them and eventually backtrack and start exploring subtree rooted at C, and we'll be able to find our answer.

Now if you don't know the depth $d$ which goal nodes lie at, then you'll use iterative deepening DFS, which essentially does depth limited search but keeps increasing the value of $d$ on each iteration, so you're guaranteed to find a goal node, if it exists.


Which meta-learning approach selection methodology should I use for similarity learning of an image?

The only principled approach is to see if the problem you are working has been worked on before. If yes, then, use what they used for your problem. It is the alternative to go through all the models one-by-one.
Like, in your case, it bears a semblance with face models, so, Siamese network with ArcFace loss for getting your embeddings is a good bet for you.


Why is gradient ascent necessary when training Actor Critic agents?

It might seem to give the same update direction but would it converge to desirable policy parameters?

Actor-Critic is proposed alongside the policy gradient theorem in Sutton 1999. It is shown to maximize the state-value function. If you are able to show that the technique of yours is, in fact, maximizing some desirable objective function, you could propose it with some soundness as well.


Can I use neural networks for a problem (in description)?

To answer the titular question first: Yes, of course you can. Whether a neural network can give you better results than a simpler model, however, depends on:

How complex/non-linear the relationship between your input and output variables is;
Whether the neural network you have specified is able to learn this relationship efficiently;
How much training data you have.

With the information you have provided in the question, there's really no telling how it will perform against a simpler model, so I would advice trying both.

As for how your output varies over time and location, just include both in your model. In case of a linear regression model, accounting for spatiotemporal autocorrelation could be done with a mixed model using an appropriate covariance structure. The challenge for a neural network would be how to specify one that can learn this type of relationship (for starters, read up on RNNs for longitudinal data).


The long-short term memory (LSTM) is a type of recurrent neural network, which is only suited for sequence modelling, that is, to keep track of statistical dependencies between elements of a sequence.

The LSTM prediction capabilities are limited to the training data that is used to train it, the inductive bias (in the case of LSTMs, the inductive bias particularly refers to the fact that elements of a sequence are dependent on each other) and the available computation resources. However, storytelling often assumes the existence of a common-sense knowledge between the storyteller and the listener (but LSTM completely ignores this) and requires the true understanding of language, which is believed to be an AI-complete problem (in simple words, it is a very complex task, which probably cannot be exactly solved with statistical models).

Furthermore, even though LSTM partially addresses the vanishing gradient problem (and they were specifically created to partially solve this issue), they can still suffer from the exploding gradient problem. Moreover, although LSTMs and, in general, neural networks are universal function approximators, in practice, functions may not be continuous, which is an assumption made in the universal approximation theorems.


What make a CNN suitable for image classification or for semantic segmentation?

Disclaimer: This question is very broad, my answer is admittedly partial and is intended to just give an idea of what's out there and how to find out more.

  How do you "say" a network: "classify me these images" or "do semantic segmentation"?

You're mixing two very different problems there. Although there are SO many variations of problems people are applying CNNs to, for this example we can focus on the "classification of something in the image" subset ad identify 4 key tasks:

Image Classification answers the question "What is this image about" (the answer is an image where each pixel is assigned to one of the given classes)
Semantic Segmentation answer he question "What areas of this image are part of a Cat?" (e.g.,
Object Detection answers the question "Where are the objects in the image AND what objects are they?" (e.g. of answer: "Cat in bounding box at x,y,w,h [10,20,50,60]")
Semantic Segmentation answers the question "Where are the individual objects in this image AND what class are they AND give me the pixels that belong to each object". You may guess from the number of ANDs there, this is the hardest of the four. The output here would be a set of class, bounding_box, mask tuples where the mask is typically defined in relation to the returned bounding box.

So, how do we build networks capable of solving one problem or the other?
We build architectures towards one specific problem, exploiting reusable parts where possible. For example, typically classification and object detection are based on a deep "backbone" that extracts highly complex features from the image, that finally are used by a classifier layer interprets to make a prediction (for image classification) or a box prediction head to predict where objects lie in the image (very big simplification, look up object detection architectures and how they work for the proper description!).

  What do I have to do to use a CNN for classification or for semantic segmentation?

In principle you can't just take a network built for classification and just "ask" it to do semantic segmentation (think of it as trying to use a screwdriver as scissors... it just was not built for that!). You need changes in the architecture, which necessarily imply new training, at the very least for the new parts that were added.


How many hidden layers are needed for this training data set

You need to perform Hyperparameter Tuning to identify -

Number of hidden layers.
Number of neurons in each of the hidden layers.
Dropout
The activation function you use in each of your hidden layers.

There parameters are only related to how you build your model. There are others that relate to training like batch size, number of epochs and so on. Your model's performance ultimately depends on how well you tune your hyperparameters.

Also note that hyperparameter tuning is a trial and error task because it depends on several factors that may not be obvious to us. With experience, experts do build certain thumb rules about what may be the right choice, but there is no way to generalize it. "Some say the number of hidden layers is roughly the number of separation times" - is just another thumb rule. You simply need to find out what best suits your scenario.


Is learning possible without random thoughts and actions?

Learning is possible without random thoughts and actions. Knowledge can be encapsulated in predetermined forms and passed through predetermined knowledge transfer mechanisms. Much of civilization is based on these predeterminations. Without them, humanity would be thrown back possibly 120,000 years.

However, initial discovery requires trials and review of their outcome. Purely deterministic identifications of trials is necessarily systematic and the system used may interplay with the phenomena under study in such a way as to miss important cases. Furthermore, when the complexity of the phenomenon is high, the number of trials is often too numerous to check entirely. In this second scenario, random selection of trials is wise for a similar reason illustrated by this simple example.

The phenomenon has one behavior for even numbers and another for odd. The system for determining trials is to check every factor of 100 to cover the range from 1 to 10,000 in 100 trials. The odd behavior will be inadvertently overlooked.

  Intelligence begins once the thoughts/actions are logical rather than purely random based.

For the above reasons, intelligence begins with logic only in determining the domain of trials but, when one of the above two cases apply, is often quickly followed by lack of logic in the selection from that domain. Once models have formed as a result of these initial discovery activities, logical inference is useful again, to combine them in various ways. Through this process, engineering, business, resource planning, and other intelligence related disciplines have improved the living conditions for the species, albeit inconsistently.

So there are two limitations of logic.

Piercingly deep searches require some temporary dismissal of logic.
Logic applied by large populations produces inconsistently logical results.

It is naive to assume that there were logical faults that produces the inconsistency in this second limitation. There is no logical proof that logic necessarily improves conditions logically across the population that uses it. There may be qualities of the goal sets of many individuals that thwart the vision of logic producing peace and prosperity for all. That belief is not new and no person or political group has been able to make it work.

  Intelligence needs intelligence to coexist and a sharing communication network for validation/rejection. ... I believe that we must keep the human intelligence in a parental role for long enough time until at least the AI had fully assimilated our values.

That assumes that those values are best. Some might want to agree on those values before letting the AI assimilate them, which, given the insanity evident in human history, could lead to a war. If we give the AI the below four objectives, we could probably live without the remainder of human values.

Survival of the biosphere
Human freedom of expression
Evenly distributed, high prosperity across the human population
Encouragement of the social rule, "Be with others how you would want them to be with you," between humans, between AI robots, and between AI robots and humans.

This statement dismisses the human record.

  The actual danger is to leave the artificial intelligence parenting another AI and loose control of it. ... the purpose should always be to help humans achieve mastery of the environment while ensuring our collective preservation. ... AI should not be left unsupervised as we would not give guns to kids, do we? ... To resume it all AI needs an environment and supervision where to learn and grow. The environment can vary but the supervision must stay in place.

It is impossible to supervise all adults and it will later become impossible to supervise all AI in data centers and robots. There are already more processes running in computers than there are people in the world. This is why risk management must be applied proactively, during the research and development of the AI. Some of the questions on this site address that very challenge. If it is not done proactively, it will become untenable and control will already have been passed for a random future regarding our species.

  Only develop artificial intelligence that is limited by our own beliefs and values rather than searching for something greater than us.

There is no universal set of beliefs among humans. There will inescapably be a wide range in AI developments with a wide range of intended purposes. The key is to encourage researchers to think beyond their obsession with getting programs to do amusing and impressive things. AI research must continue to keep one foot in technology but another foot firmly placed in the practical elements of ethics, social science, environmental science, economics, and risk management.


Which technologies should humanity develop before developing an AI with general superhuman intelligence?

I think most answers that come to mind are going to be connected to the control problem in some way---e.g. sandboxing frameworks, AI behavior monitoring protocols, etc.

However, if you aren't interested in those for now, that pretty much leaves two possible needs such technology could satisfy: the technical challenges of AI itself, and the need to optimize the social impact on AI.

The technologies satisfying the first need are relatively "obvious" once you have the AI implementation figured out---depending on what that is, there will be certain hardware and software requirements (massively parallel processing, neural network frameworks, etc). Most of these technologies seem likely to arise "naturally" from the status quo---if we just keep developing our hardware and software as we are now, we will eventually get there.

Coming up with technologies satisfying the second need is a challenge, and far more interesting in my opinion. We need to start a discussion about this, so you have a great question here. I don't know that I can contribute much, but maybe I can provide some starter ideas:

More robust encryption schemes, so that the massive amount of information handled by AI is secure even in the face of AI-capable adversaries. We'd want these encryption schemes to be information-theoretic secure, so that it is mathematically impossible to recover the plaintext, rather than just computationally difficult, since an AI may be able to improve it's own computational efficiency with unprecedented ease. Quantum cryptography may help.
Improved robotics, which would require improved battery technology, to multiply by a great factor the ability of AI to manipulate the physical world and assist with physical tasks (obviously requires a fantastic solution to the control problem).
If you consider it separate from the control problem (and I think there is a useful distinction here), lots of AI-monitoring technology will be necessary. Think of technologies acting as regulators for newly automated tasks---e.g. traffic monitoring algorithms to watch for undesirable behavior among automated vehicles.

Answering this question is very tough, particularly because you are asking for technologies which we'd have to develop before we have AI. Of course, once we have general AI we can instruct it to develop any of these technologies for us, supposing the control problem is adequately solved. Nevertheless it seems wise to solve develop improved cryptography immediately since even with weak AI (or a weak control problem solution) the need is urgent.


What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I don't think there's a "standard way" of expressing the forward pass: you use the transpose when you need to use it, and this depends on how you define the weights and inputs matrices, and on the architecture of your neural network. For example, in a fully connected feedforward neural network, you know that every neuron in the previous layer is connected to every neuron in the current layer, so, as long as this is satisfied when you multiply the matrices, it does not matter whether you use transposes or not, and I don't think that, in computational terms, it makes any difference if you use transposes or not. (By the way, if you are writing something, I suggest that you always specify the dimensions of your matrices and your conventions).
Of course, if you want to use a library like TensorFlow, you will probably need to follow the conventions of the library, but this is another story.


Do you know any examples of geometric deep learning used in industry?

Recursive artificial networks are a type aligned well with GDL, and it is an increasingly popular one. Using non-orthogonal geometries is natural when using directed graphs as semantic association, data flow, or composition networks.

This more free-form approach enters into industry slowly because most programming languages use orthogonal structures designed around multidimensional arrays and nested loops to iterate through them, made popular in the FORTRAN era. It is common that a student learns to loop through an array before learning to call a function or test and branch. Computer science is, in some ways, entrenched in the orthogonal structure.

To call the two categories Euclidean and non-euclidean is inaccurate. The initial trend in machine learning followed a Cartesian paradigm, one locked into 90° angles in data structures. The free-form graph is non-cartesian but still just as Euclidean, as Euclid worked with angles other than 90° frequently.

Apple and Google are likely candidates for this seepage of recursive networks into industry data centers, for NLP applications. Language associations are free-form even though these structures are serialized for speech and written text.

Cognition is also non-cartesian. Graph based optimization (pre-GDL) has been part of cognitive research since the 1970s, especially in the LISP space at MIT and CMU and the U.S. Naval Research facilities for strategic analysis, Optimal strategies for a class of constrained sequential problems by JB Kadane, HA Simon, The Annals of Statistics, 1977. Algorithm 2 is a convergence strategy, but without layers of cells, yet the searching concepts of gradient are already present in the maximal and minimal operations in the search in steps c and f.

The aeronautics and auto industry is a likely candidates for seepage into the embedded computing for automated driving and flight. Most of this is either company confidential or classified.

This answer to the question about topological sophistication provides some background why non-euclidean space may be a more natural way to map associations, flow, and composition.

These are some practical applications.

Convolutional networks on graphs for learning molecular fingerprints by DK Duvenaud, D Maclaurin, J Iparraguirre, Advances in neural, 2015
CD-aware recursive neural networks for jet physics by Gilles Louppe, Kyunghyun Cho, Cyril Becot, Kyle Cranmer, 2019
Stability Properties of Graph Neural Networks by Fernando Gama, Joan Bruna, Alejandro Ribeiro
Isospectralization, or how to hear shape, style, and correspondence by Luca Cosmo, Mikhail Panine, Arianna Rampini, Maks Ovsjanikov, Michael M. Bronstein, Emanuele Rodolà
Fake News Detection on Social Media using Geometric Deep Learning, by Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, Michael M. Bronstein

We've used recursive networks to align complex models with data acquired during laboratory experimentation and hope to publish on that approach as part of another publication. It is effective, since the relationships between features in the real world form squares and cubes only by chance, so it is not very common. Research into materials and energy cannot be confined by Cartesian conceptions, even if the graphs published are on Cartesian coordinate axes, so that others with traditional analytic geometry backgrounds can understand the graphs quickly.


Can Google's patented ML algorithms be used commercially?

Can you use them commercially?

Yes.

Is Google able to sue you any time they want?

Yes.

Will they do that...

Probably not.

Google isn't a known patent bully, I would give them the benefit of the doubt in this kind of situation and say, unless you start really giving them real trouble, they wouldn't do anything. Some companies/people know an idea can really be used for good and patent it to protect its use not to inhibit its use. By patenting the idea and setting a precedent of not suing they are effectively allowing everyone to benefit. Maybe in the future Google cloud, Azure and Amazon web services will lose some money in a legal battle, but I doubt you personally will be hit with a lawsuit.


Reinforcement Learning in Commercial Strategy Games

Investigating reinforcement as a way of producing more interesting behavior than behavior trees for AI based commercial strategy game development is a good idea. The simple test game given can be described briefly.

Human pitted against AI
Each player gets a fleet of ships
Each ship begins with a positive health level
One ship can be ordered to fire one shot at one ship per turn
Each hit produces a decrease in ship health
Ships with non-positive health are removed
Player with no remaining ships are removed
The remaining player wins

The question is correct in that game states must be represented as tensors. Ideally, they should be terse representations, with minimal redundancy. The actions must be represented as tensors too. The goal to train a network to play a game is generally feasible, but the simple game may not provide sufficient complexity to lead to learning. It is probably too simple. What is there to learn? In terms of the expectation regarding the result of the game, what advantage is there to firing on one ship over another?

It may be useful to understand that although machine learning can involve networks that can be represented as directed graphs, it is not the graphs of the network cells that are saved or loaded to persist learned behavior. The networks are parameterized, and those parameters are the representation of what is learned. Where this is most important is that, provided the form of the representation is standardized, the learning hardware and software environment need not match that of the environment that later leverages what is learned. The comment, "Build a custom runtime that can run the net [using the previously learned parameters]," is the correct method if this approach is taken.

In reinforcement learning, the representation is less like a graph and more like a structure involving a flexible behavioral policy that is affected by a changing distribution of probabilities. The expectations of future game states resulting from actions are part of the evaluation that must occur before each move. One theoretical approach to automated game play, for which a few varieties exist, is Q-learning. There are others using Markov trees and recurrent networks.

If the approach is to learn something in the lab that can be used in an xBox or Playstation, then supervised learning is indicated, and a high level language may assist in rapid prototyping. The result of training would be saved in language independent form, therefore readable by the portable c/c++. If the approach is to continuously change, then the reinforcement technology must be running on those platforms and most therefore be contained in statically or dynamically linked portable c/c++.

Both of these development approaches can be used in conjunction. For this basic investigative work and probably the first year or two of R&D, the use of Python for rapid prototyping would not require the use of Pybind. For the most common operations, the interoperability between Python and the mature C code ported from FORTRAN for tensor processing years ago is already present. The Python should never run on the game platform. It is not compiled and runs orders of magnitude less efficiently than C/C++ or Java. Determining whether the C++ API of tensorflow ports well to the desired target game environments is part of the architecture process. The Tensorflow underlying C++ API that Python ML newbies aren't generally concerned with is worth review by those who develop commercial games.

If 99.9% of the time, games are still developed on a Microsoft OS, game developers may be excessively patient. Fedora LINUX is much more productive for C, C++, Java, and Python development for both low level and high level programming tasks.

None of this has addressed the mention of a more creative opponent in the question.

The classic training goal in AI game play is to produce winning behavior, but games don't sell when the human always loses. Setting reward and loss functions to optimize on the basis of victory will not create sustainable sales of commercial game products. What a game developer might more wisely chose as a basis for optimization is engagement. Sacrifice victory in favor of sensation. You want gamers to blog to each other and talk in school and work about some crazy and unexpected thing the game did. Consequently, the goals of the learning must be adjusted in terms of training to maximize game value based on these product market realities.


This task falls within the overlapping fields of information extraction and pattern mining. Information extraction involves automatically extracting instances of specified relations from data. While pattern mining involves using data mining algorithms to discover interesting, unexpected and useful patterns between data in databases (Philippe F).

On your question you have stated that you have experimented with markov models with poor results. A better approach if you prefer working with markov models would be to use hierarchical markov models. Hierarchical markov models have multiple 'levels' of states which can describe input sequences at different levels of granularity. Hierarchical markov models are good at categorizing human behavior at various levels of abstraction i.e. a persons location in a room can be further interpreted to determine more complex information such as what activity the person is performing.

However my recommendation is that you implement random forest classifiers for this problem. Random forests provide excellent classification accuracy with a relatively simple implementation. Additionally random forests provide the ability to inspect trees for tweaking parameters to improve accuracy. You can also use cross validation to evaluate your model and calculate its accuracy. Consider using Python's Scikit-learn library's implementation of random forest classifier for this analysis.

In your json code action package you have declared metrics such as DRAG_START, TRASH_SYMBOL, OPEN and CLOSE. My suggestion is that for your model to be accurate you also need to declare lower level actions such as: time between clicks, change in the direction of mouse motion, screen region hover count, task completion time and time between a click and a succeeding mouse movement.

For further reference I recommend that you look at the papers below which I found useful and relevant to your question.

Hierarchical Hidden Markov Models for Information Extraction https://www.biostat.wisc.edu/~craven/papers/ijcai03.pdf

Detecting Abnormal User Behavior Through Pattern-mining Input Device Analytics https://www.ignacioxd.com/files/bib/Dominguez2015-Concentration.pdf

Hierarchical Hidden Markov Model in Detecting Activities of Daily Living in Wearable Videos for Studies of Dementia   https://arxiv.org/ftp/arxiv/papers/1111/1111.1817.pdf


There is absolutely no doubt that fuzzy logic can contribute to the reliability and accuracy of real time diagnostics of mechanical, thermodynamic, and electro-magnetic devices.  I can state this with assurance because it exists in aeronautics products today.

One of the senior PhDs at the research facility wanted to acquire data via standard techniques and had purchased a brigade of high end Hewlett Packard workstations and instrumentation clusters.  Another senior PhD there wanted to build a fuzzy logic container to provide early warnings of failures, just as implied in the question.  There was some interplay between those two projects in their application in the field.

My direct involvement was to teach the first PhD OO and C++ and design and implement a scientific computing framework to churn the Big Data for the second PhD.  The framework provided a super-set of the union of SciPy and TensorFlow with a little PyTorch added.  That we developed these techniques before Java and Python existed is why Hewlett Packard sent engineers to study what we were doing, with DoD permission, of course.

What you want to do is this:

Become familiar with the open source options in Java, Python, and C.  The general trend is to either use Java-ML or use C wrapped in the above mentioned Py libraries.
Use 16 or 24 bit multi-channel data acquisition with time stamps and latched, concurrent A-to-D conversion at a frequency at least twice1 the 16th harmonic of the vibration base frequencies.
Ensure that you have sufficient system memory to hold what for embedded qualifies as Big Data2.
Ensure that you know how spectra (both amplitude and phase) can be obtained via FFT from channels.
Use LINUX, a single board computer, and either Intel Movidius3 or NVidia CUDA for hardware acceleration.
The fuzzy logic container and any assistive reinforced learning can be on the embedded CPU and leverage the hardware acceleration.
The fuzzy rules can be acquired from the experts to which you have access, as indicated in the question.
Read Chaos Theory Tamed by Williams, 1997 to get familiar with phase relationships between input channels and how to analyze changes in chaotic behavior between the harmonics.

Start with small POCs and generate independent components that can be connected and tested independently with input that came from the output of previously tested components.  Here are some, in rough order of development and signal path.

Data acquisition of multiple samples of multiple channels to a matrix.
FFT of acquired data to an amplitude-phase array in the frequency domain.
Analysis to determine the patterns in metrics that precede failure.
Introduce and train artificial networks (either LSTM or attention type) where indicated to remove redundancy from the information that will be input to the fuzzy logic container.
Write up the rules and place them in the container.
Train the fuzz.
Test the system.

You will need to work with the experts on most of these steps.  If you try to involve them at the rule writing stage and they weren't privy to any of the previous developments, they may be confused.

I'm not a lawyer, but, because of the involvement of failure prediction experts, I think you'll need an internationally binding NDA to ensure the experts don't walk away with the product of your research and move to another country.

These are a few considerations for product reliability you may need to consider even after you have a working system.

Develop a way to adapt the system in real time ... to tracking changes in what indicates near failure based on a centralized database related to the product.
Consider version control and deployment challenges.

I'm sorry that I have no further references.  You can try the IEEE library if you have access.  I have been too busy doing this type of work to publish what we do, but I think most of the important directions are included here already.

FootNotes

[1] Not much can be learned about impending failure from the primary harmonic of vibration.  The first 16 harmonics are sufficient.  Because of Nyquist Criteria considerations, the sample rate must be twice the top harmonic.  This means that if your vibration is 1 KHz, the sample rate must be at least 32 KHz.

[2] Big Data input rates may be necessary because with 8 channels of strain input from Wheatstone bridges of gauges, 8 channels of accelerometer, 8 channels of IR sensing and the above vibration base harmonic, we have (24 channels + 1 time stamp) x 32 KHz x 3 bytes per channel-sample = 24 Mbytes per second.  If your failure detection window holds 10 seconds of vibration data as IEEE 64 bit reals and you send 1 second (overlapping) Hamming windows to the hardware FFT, just for that alone will need 12.8 Gbyte input matrix, 6.4 for the input and 6.4 for the output.

[3] Available for embedded applications from Mouser and other distributors as properly paired single board computers and Movidius mPCIe (mini PCI express) daughter boards from makers like Aaeon.


How to find a cost function for human data

You've hit upon the central conundrum of supervised learning: if you want a machine to learn to do something, you need to know how to explain what that something is.

In the case of music, there are several possible approaches:

Make one set of "bad" songs, and one set of "good" songs. Develop a measure of how similar two songs are (maybe euclidian distance between their discrete Fourier transforms is a good starting place?). Your cost function is then based minimizing the average distance to "good" songs, and maximizing the average distance to "bad" songs. This may not work well though, because good and bad songs might differ only in the occasional misplaced notes.
Move to a reinforcement learning paradigm. Listen to each song proposed by your network. Give is a score based on your subjective enjoyment. Your cost function is based on maximizing this score. This might work well, but again, it might not. Music is tricky.
Use unsupervised approaches. Reward your network just for making something that resembles music (perhaps using the Fourier transform approach above), without labelling good and bad. The advantage is that you don't need to decide what is good or bad, and so you can use a lot more music in your dataset. The drawback is music as a whole might be too diverse to learn easily from examples.
Treat your music as a sequence of notes, and train a generative model to predict future notes on the basis of past notes. You can then generate new music by starting the model with a set of notes and letting it generate new ones for a long time.


Checking Whether Given Logistic Regression Classifier classifies data


How do I identify a monologue or dialogue in a conversation?

There are a few different ways you could frame the problem...

For example, the simplest, yet probably not the most effective, would be to treat it as a supervised classification task.

In this case, you would gather data, split it into 2 classes(binary). With one dataset consisting of dialogues and the other monologues. Obviously, this framing comes with it's own set of problems. Namely, how do you prepare the dataset, or how do you deal with live examples?

Of course, there are other ways you could tackle the problem, perhaps the detection context. It all depends on what you feel is the easiest tenable solution given the variables involved in your specific instance.


What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?

In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!).

algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function
algorithms that directly attempt to find a policy (e.g. REINFORCE and other so-called "policy gradients" algorithms)
algorithms that use a value function to guide the search for an optimal policy (i.e. actor-critic methods)

More specifically, in actor-critic methods, you have a policy $\pi$ (known as the "actor") and a value function $v$ (known as the "critic"). Hence the name "actor-critic". The idea is that you will use this critic $v$ to "criticize" the policy (or actor) $\pi$, i.e. to guide the search for a good policy. This article 6.6 Actor-Critic Methods (from Sutton and Barto's book) explains the concept quite well.

In your specific example, the policy and value function are assumed to be differentiable (otherwise, you wouldn't be able to compute the derivatives anyway!). They are typically neural networks. $\theta_\pi$ are the parameters of the neural network that represent the policy (i.e. a neural network that receives as input a state and produces as output an action or probability distribution over actions). Similarly, $\theta_v$ are the parameters of the neural network that represent the critic. Then $\partial \theta_\pi$ and $\partial \theta_\pi $ will represent an accumulation of the gradients with respect to the actor and critic,  respectively. (The accumulation is probably over the steps from $i=t-1$ to $i = t_{\text{start}}$, but I can't say more because I never implemented actor-critic methods). This should give you an idea, though!


Is it useful to eliminate the less relevant filters from a trained CNN?

NOTE: All the observations and results are from the paper The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

To answer your questions one by one:

Yes there are ways to determine which filters have more impact on the output. Its a very naive way but works very good in practice. Filters with small weights impact output less (according to empirical evidence), which basically means neurons whose weights lie in the switching region i.e ~$0$ in ReLu and ~$-1$ to $1$ (say) have less impact on final output.
Yes, just eliminating these lower weight filters eliminate the unnecessary noise and indecisiveness introduced by these filters and suprisingly makes the model perform better (observed empirically).
The concept a relatively old paradigm but has been a new twist by the simplicity of the method of elimination of unnecessary weights in the aforementioned paper and thus winning it the best paper award in ICLR 2019.

TL;DR: Eliminating unnecessary weights makes the model perform better than the original model.

Also here is the TensorFlow code.


How could artificial intelligence harm us?

tl;dr

There are many valid reasons why people might fear (or better be concerned about) AI, not all involve robots and apocalyptic scenarios.

To better illustrate these concerns, I'll try to split them into three categories.

Conscious AI

This is the type of AI that your question is referring to. A super-intelligent conscious AI that will destroy/enslave humanity. This is mostly brought to us by science-fiction. Some notable Hollywood examples are "The terminator", "The Matrix", "Age of Ultron". The most influential novels were written by Isaac Asimov and are referred to as the "Robot series" (which includes "I, robot", which was also adapted as a movie).

The basic premise under most of these works are that AI will evolve to a point where it becomes conscious and will surpass humans in intelligence. While Hollywood movies mainly focus on the robots and the battle between them and humans, not enough emphasis is given to the actual AI (i.e. the "brain" controlling them). As a side note, because of the narrative, this AI is usually portrayed as supercomputer controlling everything (so that the protagonists have a specific target). Not enough exploration has been made on "ambiguous intelligence" (which I think is more realistic).

In the real world, AI is focused on solving specific tasks! An AI agent that is capable of solving problems from different domains (e.g. understanding speech and processing images and driving and ... - like humans are) is referred to as General Artificial Intelligence and is required for AI being able to "think" and become conscious.

Realistically, we are a loooooooong way from General Artificial Intelligence! That being said there is no evidence on why this can't be achieved in the future. So currently, even if we are still in the infancy of AI, we have no reason to believe that AI won't evolve to a point where it is more intelligent than humans.

Using AI with malicious intent

Even though an AI conquering the world is a long way from happening there are several reasons to be concerned with AI today, that don't involve robots!
The second category I want to focus a bit more on is several malicious uses of today's AI.

I'll focus only on AI applications that are available today. Some examples of AI that can be used for malicious intent:

DeepFake: a technique for imposing someones face on an image a video of another person. This has gained popularity recently with celebrity porn and can be used to generate fake news and hoaxes. Sources: 1, 2, 3
With the use of mass surveillance systems and facial recognition software capable of recognizing millions of faces per second, AI can be used for mass surveillance. Even though when we think of mass surveillance we think of China, many western cities like London, Atlanta and Berlin are among the most-surveilled cities in the world. China has taken things a step further by adopting the social credit system, an evaluation system for civilians which seems to be taken straight out of the pages of George Orwell's 1984.
Influencing people through social media. Aside from recognizing user's tastes with the goal of targeted marketing and add placements (a common practice by many internet companies), AI can be used malisciously to influence people's voting (among other things). Sources: 1, 2, 3.
Hacking.
Military applications, e.g. drone attacks, missile targeting systems.

Adverse effects of AI

This category is pretty subjective, but the development of AI might carry some adverse side-effects. The distinction between this category and the previous is that these effects, while harmful, aren't done intentionally; rather they occur with the development of AI. Some examples are:

Jobs becoming redundant. As AI becomes better, many jobs will be replaced by AI. Unfortunately there are not many things that can be done about this, as most technological developments have this side-effect (e.g. agricultural machinery caused many farmers to lose their jobs, automation replaced many factory workers, computers did the same).
Reinforcing the bias in our data. This is a very interesting category, as AI (and especially Neural Networks) are only as good as the data they are trained on and have a tendency of perpetuating and even enhancing different forms of social biases, already existing in the data. There are many examples of networks exhibiting racist and sexist behavior. Sources: 1, 2, 3, 4.


Why does the discount rate in the REINFORCE algorithm appear twice?

The discount factor does appear twice, and this is correct.
This is because the function you are trying to maximise in REINFORCE for an episodic problem (by taking the gradient) is the expected return from a given (distribution of) start state:
$$J(\theta) = \mathbb{E}_{\pi(\theta)}[G_t|S_t = s_0, t=0]$$
Therefore, during the episode, when you sample the returns $G_1$, $G_2$ etc, these will be less relevant to the problem you are solving, reduced by the discount factor a second time as you noted. At the extreme with an episodic problem and $\gamma = 0$ then REINFORCE will only find an optimal policy for the first action.
In continuing problems, you would use different formulations for $J(\theta)$, and these do not lead to the extra factor of $\gamma^t$.


How many layers and neurons on the layers does a neural network need for for sales prediction?

Every answer you get is just an opinion, it is based on experience.
My answer is a single hidden layer of 5 neurons.

Besides, I recommend you to use a TWEANN system (Topology and Weight Evolving ANN).

Such system applys genetic algorithm to search and optimize an ANN with the optimal topology and weights.

Take a look at NEAT and DXNN.


Should the agent play the game until the end or until the winner is found?

You should probably grant reward at the point that the game is logically won. This will help the agent learn more efficiently, by reducing the number of timesteps over which return values need to be backed up.
Stopping the episode at that point should also be fine, and may add some efficiency too, in that there will be more focused relevant data in the experience replay. It seems like on the surface that there is no benefit to exploring or discovering any policy after the game is won, and from the comments no expectation from you as agent developer that the agent has any kind of behaviour - random actions would be fine.
It is still possible that the agent could learn more from play after a winning state. It would require certain things to be true about the environment and additional work from you as developer.
For example, if the game has an end phase where a certain kind of action is more common and it gains something within the game ("victory points", "gold" or some other numbered token that is part of the game mechanics and could be measured), then additional play where this happened could be of interest. Especially if the moves that gained this measure could also be part of winning moves in the earlier game. To allow the agent to learn this though, it would have to be something that it predicted in addition to winning or losing.
One way to achieve this is to have a secondary learning system as part of the agent, that learns to predict gains (or totals) of this resource. Such a prediction could either be learned separately (but very similarly to the action value) and fed into the q function as an input, or it could be a neural network that shares early layers with the q function (or policy function) but with a different head. Adding this kind of secondary function to the neural network can also have a regularising effect on the network, because the interim features have to be good for two types of prediction.
You definitley do not need to consider such an addition. It could be a lot more work. However, for some games it is possible that it helps. Knowing the game, and understanding whether there is any learning experience to be had as a human player to play on beyond winning or losing, might help you decide whether looking into trying to replicate this additional experience for a bot. Even if it works, the effect may be minimal and not worth the difference it makes. For instance running a more basic learning agent for more episodes may still result in a very good agent for the end game. That only costs you more run time for training, not coding effort.


How can the sum of squared errors have negative gradient if it's defined as the squared of the error?

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?

In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any difference between the predicted and correct results. Minimising the value of that measurement is a goal for the optimiser.

How can then exist any negative gradient at the output?

The gradient of the cost function is different to the cost function. Technically it is the derivative of the cost function with respect to the parameters you can change. Taking the derivative changes the powers in the equation.
If you plot the graph from a single example, you can clearly see the gradient is negative when the prediction is too low, and positive when the prediction is too high:

Note this is the opposite relationship to the one you suggest in the question. That is why it is called gradient descent when we apply corrections. The steps to correct the parameters are in the opposite direction to the gradient.
In terms of algebra (and moving the example index into a subscript so that is doesn't get in the way visually), assuming $o_i$ is your output value, and $t_i$ is the target value, then you want the gradient of your loss function $L$ with respect to $o_i$ or $\frac{\partial L}{\partial o_i}$:
$$\frac{\partial L}{\partial o_i} =  \frac{\partial}{\partial o_i} \frac{1}{2}(t_i - o_i)^2 = \frac{\partial}{\partial o_i} \frac{1}{2}(t_i^2 - 2t_i o_i + o_i^2) = \frac{1}{2}(-2t_i + 2o_i) = o_i - t_i$$
This simple gradient value is negative if your output $o_i$ is lower than the target $t_i$, and positive if $o_i$is greater than $t_i$.
As an aside, if you wondered why the multiplier of $\frac{1}{2}$ was included in the definition of SSE, it is intended by design to end up with this simple result. Without it, everything works, just you would end up with the gradient being $2(o_i - t_i)$


Perceptron learning algorithm: different accuracies for different training methods

In Brief:

re-train your dataset. I believe where you get lower accuracy scores, your model has not converged to the final state. duplicate your dataset multiple times and create a bigger one, then train your model with it.

In Detail:

  number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure)

This is a valid accuracy metric. In fact if the value is acc, then 1-acc is called misclassification error. So your metric is good unless you have some class imbalance, where you need to use other metrics such as Cohen's Kappa score.

  the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes.

I strongly believe both strange results happen because the number of your instances (i.e. examples) is low, or let's say your learning rate is small. As you know Perceptron algorithm corrects the weights of the decision hyper-plane by delta learning rule: it reads each instance, calculates the error (in case of binary classification {-1,0,1}) and updates the weights by c.x.E where:

c is learning constant/rate/step
x is the data instance
E is the error

Thus the weights (or slopes) change a little bit every time. There is no guarantee that your model reaches its final states after all the instances are given to it. Since when you start training with different initial weights, or change the order of instances given to your model, the distance or from or moving speed toward the final state changes. So as I mentioned above, re-train your model and let it converge to the final state OR increase your learning step (learning rate) . I believe all the 4 mentioned-above-accuracies will be the same then; I mean not only you should have the same model for different initial weights, but also for batch and online training.

Please update your post with new findings and question my claims if those were false.


How to make a fair comparison of a convolutional neural network (cNN) vs a mutlilayer perceptron (MLP)?

The best best way to monitor an architectures performance would be comparing the resource utilization, model accuracy, value loss and confusion matrix.

e.g VGG16 consumes less system resources in comparison to inception V3

There is an article that goes in depth.


Was the Malaysian Airliner crash caused by a non-benign artificial intelligence system?

Short answer: despite the incredible advances in AI via Machine Learning (and subfields) AI is nowhere near this kind of autonomous decision-making.

I can't prove a negative, but, the level of autonomy Musk is talking about is still on the horizon.

Doesn't mean Musk is wrong about the hypothetical.  imo I'm glad he keeps bringing these kinds of issues to the fore.

Engineers and mathematicians have a bit more credibility on this subject than laypeople.  If it seems alarmist, there are many in the scientific community who feel it is warranted.

What Musk is describing is an extension of what Asimov felt compelled to warn us about via his "Three Laws of Robotics" back in 1942, when computers sucked.  For an explication of this idea re: Machine Learning, see below.

JUST FOR FUN (SORT OF:)

Author and mathematical physicist Hannu Rajaniemi just published a story in the MIT Tech Review on this very subject.

Unchained: A Story of Love, Loss, and Blockchain

Warning: the story is both wickedly funny, surprisingly moving, and very likely prescient!


What's the difference between RMSE and Euclidean distance, and when to use a custom loss?

For the first question, RMSE and Euclidean distance have no difference, not that i know of.

For the second question, you only need the common loss function for normal tasks.

MSE is a common loss function used in linear regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For logistic regression, use Binary Cross Entropy loss. See this for details:

  Cross-entropy loss, or log loss, measure the performance of a
  classification model whose output is a probability value between 0 and
  1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from
  the statement of your problems itself. In classification you work with
  a very particular set of possible output values thus MSE is badly
  defined.

  To better understand the phenomena it is good to follow and understand
  the relations between

  Cross-entropy

  Logistic regression (binary cross-entropy)

  Linear regression (MSE)

  You will notice that both can be seen as a maximum likelihood
  estimator (MLE), simply with different assumptions about the dependent
  variable.

  When you derive the cost function from the aspect of probability and
  distribution, you can observe that MSE happens when you assume the
  error follows Normal Distribution and cross-entropy when you assume
  binomial distribution. It means that implicitly when you use MSE, you
  are doing regression (estimation) and when you use CE, you are doing
  classification.

Source: https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up

For a custom loss function, there is examples like the triplet loss and cases where you need to optimize two loss at once, then you need a custom loss function. For the case of triplet loss, it is used in one-shot learning tasks like face recognition. There, face images are fed through a CNN to get embeddings about description of the face. In the training phase there is three images per batch. One anchor, one positive (which is the same person as anchor
) and one negative (different person from anchor). The loss function, triplet loss, maximizes the distance between the anchor embedding and the negative embedding. In reverse, it also minimize the distance between the anchor embedding and the positive embedding. The loss function is like this:

As you can see, there is two components here, and they are subtracted to maximize the second component and minimize the first. The alpha value is to make the loss positive as the gradient descent optimizes the loss to be as near to zero as possible.

The case of style transfer also introduces a custom loss function. Here is the loss function:

Style transfer basically alters the input image to be styled like the style image of the system.

The loss function consist of style loss and content loss. The content loss decreases the distance of the output image and the style loss decreases the distance of the output image and the style in an artistic sense. The second loss makes the output image imitate the style of the the style image.

In both scenario a custom loss function is used. In both cases, the custom loss function is used to optimize two losses together, either positively or negatively. A weight can also be introduced to weight the two losses as shown in the second case. However the second case introduce a custom loss function. That leads to the second use of custom loss function. They can be used for specific tasks and optimize a specific goal. The first use for custom loss function can be explored but the second requires careful engineering and research.

So in short, custom loss function are used for either combining two loss or a specific task.

Hope I can help you.


What's a good neural network for this problem?

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than few layers, and a relatively small network.
However, I note that:

The input variables are numerous but not that many, maybe 7-10.

This gives you $3^{10} = 59049$ possible inputs. That's not much in terms of amount of data needed for ML statistical models. Assuming that even the best predictions are still probabilistic, then you may only need a million or so examples to create a reasonably accurate lookup table, not needing a neural network at all.
The strength of a neural network is to be able to generalise well from less examples than that. Of course, this is not perfect, but it would be able to do things such as notice if inputs 1,2 and 3 agree then that is always the most likely answer. If that turns out to be true (and not an accident of having low numbers of samples), then the NN could learn that useful pattern using far less data than a table-based approach.

I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate.

This is true, but does not impact your situation, because there is no control flow or loops involved. You have described a simple function. Whilst you or I might inspect the data and look backwards and forwards across it before coming to a decision, a neural network approximating a function does not need to do that, and in simple cases there is usually no benefit to doing so - a statistical summary of the correct mapping from input to output is more than sufficient and likely the best that can be done.

I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.

As all your inputs represent the same kind of thing, you could implement as a RNN with a single input, -1, 0 or +1, always feeding in the predictions by type in the same order. It might resemble how you are thinking about the problem as a human (at least a better analogy than the direct statistical match in a feed-forward network), especially if you implemented a variant of attention. However, I don't think there would be any benefit to that in improved accuracy, and it would be a significant challenge to build that if you are new to AI.


What's the difference between LSTM and GRU?

On the same problems, sometimes GRU is better, sometimes LSTM.

In short, having more parameters (more "knobs") is not always a good thing. The training process needs to learn those parameters. There is a higher chance of over-fitting, amongst other problems.

The parameters are assigned specific roles inside either GRU or LSTM, so if that role is less important for a specific learning challenge, then it can be wasteful or even counter-productive to have the system attempt to learn values for them.

The only way to find out if LSTM is better than GRU on a problem is a hyperparameter search. Unfortunately, you cannot simply swap one for the other, and test that, because the number of cells that optimises a LSTM solution will be different to the number that optimises a GRU.

  When would one use Long Short-Term Memory (LSTM) over Gated Recurrent Units (GRU)?

When it proves better, experimentally. In some problem domains, this may be established and you can check. However, in other problem domains if either GRU or LSTM works well enough to solve a problem (and the superiority of either LSTM or GRU is not the main point of the work), then it may not so clear.


How Watson can help to make medical diagnoses?

Watson can make its diagnosis based on the patient's data and comparing it to the data from millions of other studies.
For example, having enough genetic data and the right algorithms, its AI computing capability demonstrated the huge potential of data analysis based on which it can be used for everything from diagnosing rare illnesses to prescribing perfect dosages of medicine based on the patient's personal genetic makeup. Of course, there are still plenty of challenges that need to be overcome before it can be used in mainstream medicine.
Recently Watson was able to diagnose a rare form of leukaemia after treatment was proved ineffective. It was fed in with the patient’s genetic data and compared to data from other 20 million oncological studies.
Sources:

Artificial Intelligence Used To Detect Rare Leukemia Type In Japan (2016)
Watson correctly diagnoses woman after doctors were stumped (2016)


Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal


How can I make my network treat rotations of the input equally?

If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.

You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.

You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:

from keras.models import Sequential
from keras.layers import Input, Dense
from keras.models import Model
from keras import optimizers
from keras.utils.np_utils import to_categorical
import helper

x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
y_binary = to_categorical(y_)

model = Sequential()
model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
model.add(Dense(2, activation='softmax'))
adam=optimizers.Adam()
model.compile(optimizer=adam,
              loss='categorical_crossentropy',
              metrics=['acc'])
model.fit(x_, y_binary, epochs=100)


Can someone please help me validate my MDP?

The good news is that:

Your MDP appears valid, with well-defined states, actions. It has state transition and reward functions (which you have implemented as matrices). There is nothing else to add, it's a full MDP.
You could use this MDP to evaluate a policy, using a variety of reinforcement learning (RL) methods suitable for finite discrete MDPS. For instance, Dynamic Programming could be used, or Monte Carlo or SARSA.
You could use this MDP to find an optimal policy for the environment it represents, again using a variety of RL methods, such as Value Iteration, Monte Carlo Control, SARSA or Q-Learning.

The bad news is that:

All policies in the MDP as defined are optimal, with expected returns (total reward summed until end of episode) of $v(S1) = 55, v(S2) = 33.75, v(S3) = 21.25$ - solved using Dynamic Programming in case you are wondering.
The MDP is degenerate because action choice has no impact on either state transition or reward. It is effectively a Markov Reward Process (MRP) because the agent policy has been made irrelevant.
Without discounting, the best result is not going from S1-S4 directly, as you appear to want, but repeatedly looping S1-S3-S2-S1-S3-S2... (this is currently hiiden by action choice being irrelevant).

There are a few ways to fix this, but maybe the simplest is to make the rewards more straightforward (e.g. +0, +10, +20, +30 for S1-S1, S1-S2..., -10, 0, +10, +20 for S2-S1, S2-S2...) and add a discount factor, often labelled $\gamma$, when calculating values. A discount factor makes immediate rewards have higher value to the agent, so it will prefer to get a larger reward all at once and end the episode than loop around before finishing.

This whole "bad news" section should not worry you too much though. Instead it points to a different issue. The key point is here:

  The model i have is a pretty uncertain one so i am guessing the agent must initially be given a lot of experience to make any sense of the environment.

It looks like you have assumed that you need to explicitly build a MDP model of your environment in order to progress with your problem. So you are providing an inaccurate model, and expect that RL works with that, improving it as part of searching for an optimal policy.

There are a few different approaches you could take in order to learn a model. In this case as your number of states and actions are very low, then you could do it like this:

Create a 2D tensor (i.e. just a matrix) to count number of times each state, action pair is visited, initialised with all zeroes, and indexed using S, A
Create a 3D tensor to count number of times each state transition is observed, again initialised with all zeroes, indexed using S, A, S'.
Run a large number of iterations with the real environment, choosing actions at random, and adding +1 to each visited S, A pair in the first tensor, and +1 to each S, A, S' triple in the second tensor.
You now have an approximate transition function based on real experience, without needing an initial guess, or anything particularly clever, you are just taking averages in a table. Divide each count of S, A, S' by the total count of S, A to get conditional transition probability $p(s'|s,a)$. It's not really an established, named RL method, but will do.

However, if your construction of the MDP is just step 1 for running some RL policy optimisation approach, none of that is really necessary. Instead, you can use a model-free approach such as tabular Q learning to learn directly online from interactions with the environment. This is likely to be more efficient than learning the model first or alongside the policy optimisation. You don't need the explicit MDP model at all, and adding one can make things more complex - in your case for no real gain.

You probably do still need to define a reward function in your case as there is no inherent reward in the system. You want the agent to reach state S4 as quickly as possible, so you need to monitor the states observed and add a reward signal that is appropriate for this goal. As above, I suggest you modify your planned reward structure to be simple/linear and add discounting to capture the requirement to "increase" state as fast as possible (here I am assuming that being in S2 is still somehow better than being in S1 - if that's not the case, and reaching S4 is the only real goal, then you could simplify further). That's because if you make the rewards for state progression non-linear - as in your example - the agent may find loops that exploit the shape of the reward function and not work to progress states towards S4 as you want.

Beyond this very simple looking environment, there are use cases for systems that learn transition models alongside optimal policies. Whether or not to use them will depend on other qualities of your environment, such as how cheap/fast it is to get real experience of the environment. Using a learned model can help by doing more optimisation with the same raw data, using it to simulate and plan in between taking real actions. However, if the real environment data is very easy to collect, then there may be no point to that.


What would the commercial application of a conscious AI look like/be?

They may be just for fun. If you had a robot that understood you, could hold a conversation with you about your interests, and even had goals of its own (good or bad), it wouldn't really need to do anything special. People would buy it like it was a toy or game.

Also, they might be usable as programmers, artists, designers, anything creative that a computer can't successfully do on its own.

It really just depends on what you define as 'consciousness'. Does it just understand what it's supposed to do, decide if it wants to, and if so, complete the task? Or does it wonder about religion, politics, moral situations, etc. that even regular humans don't fully understand? If it was pretty much just a human, it wouldn't be any more useful than one. Of course unless it can solve problems super quickly and effectively, then it would just be a really good worker.


NoisyNet DQN with default parameters not exploring

I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.

These lines should do the trick:

self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(out_features))
self.bias_sigma = Parameter(torch.Tensor(out_features))

In case you are not familiar, the Parameter class must be imported from torch:

from torch.nn.parameter import Parameter


Pattern-matching is a classical sub-problem of AI systems. Read about the Rete algorithm, or pattern matching in Prolog. Or about Unification. Or about the Warren Abstract Machine. Or about topological data analysis like in GUDHI.
A typical example is the CLIPSrules expert system shell. Or the match expression construct of Ocaml or in Haskell.
In some ways, pattern matching is related to compilers: the GCC compiler (or the Stalin one, or Bigloo, or SBCL) is somehow doing pattern matching on ASTs. So parser generators (e.g. GNU bison, or Iburg) are related to pattern matching.
Patterns also have a different meaning in computer vision.
A classical paper is of course Derivation of a Pattern-Matching Compiler.
A more interesting approach is described in Jacques Pitrat's last book:
Artificial Beings: the conscience of a conscious machine ISBN-13: 978-1848211018
You could also look into Pitrat's blog here.
PS. Jacques Pitrat passed away in oct. 2019.


Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only through gradient descent. RNN or LSTM however have "time" as a mechanism built into the model. The model loops through the model sequentially and have a real "sense of time" even before the model is trained. The model also have "memory" of previous data points to help the prediction. The architecture is based on the progress of time and the gradient are propagated through time as well. This is a much more intuitive way to process time-series data.

A 1D CNN also will work for the task. An example of CNN in time series data is wavenet, which uses CNN for generating incredibly life like speech using dilated convolution neural network. For whether LSTM or CNN works better, it depends on the data. You should try experimenting with both networks to see which works best.

Suppose you need to classify a video's genre. It is much simpler to watch it in sequence then seeing frames of it playing randomly in front of your eyes. This is why an RNN or an LSTM works better in time series data.


Do all expert trajectories have the same starting state in apprenticeship learning?

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because many of the code implementations on github focus on trajectories starting from the same state.

Hope this helps everyone !


Reinforcement learning objective as conditional expectations

In the YouTube depiction of CS294-112 fall 2017 lecture 3 Reinforcement Learning, Levine, the transition of the finite horizon expected reward to a form where each transition is decoupled from the entire Markov chain of state-action marginals is explained between $t_{video}$ = 44:04 and $t_{video}$ = 45:22.

At t=44:29, the probabilistic expectation where no actions take place is given as a starting point.  It's a distribution already known, the initial state distribution.

The statement, "Inside that expectation ...," at $t_{video}$ = 44:39, begins the description of how to build an expectation expression for the first two vertices of the Markov chain.  This is the beginning of a process where expectations $E$ are factored recursively.  The purpose of the expansions is to decouple the expectation at each vertex from those of other vertices in the Markov chain, clarifying dependencies and exposing the mathematical features of the recursion.

The right-hand factor is the integrated expectation of reward over the achievement of the next state-action marginal, where the action is distributed according to the fixed policy.

How a state and action pair fits into a state-action marginals which can then form a Marcov Chain, provided the policy is fixed and known, was explained at $t_{video}$ = 21:40 in this same lecture.  Each vertex in the chain is taken to be the aggregation of a state with the proposed action to be applied to it, leading to a reward $r(s, a)$.

Returning to $t_{video}$ = 44:39, notice the plus sign without the right-hand addend, waiting for another recursion.  When the recursion is expressed later in the process, the right-hand factor becomes the integrated expectation of reward over the achievement of the finite sequence represented by the remainder of the Markov chain.

Multiplication is implied because we are now conditioning on both probabilities of the initial state and a transition from it.  Notice also that the second vertex in the Marcov Chain is a state-action marginal.

Notice now that rewards are additive because the goal of learning was defined as a sum of products, which is a simplifying assumption.

Also notice that each expectation distributions of reward after the first one can be expressed as a product of single-vertex expectation distributions.  Each factor is a localized expectation along the sequence of vertices in the Markov chain of state-action marginals.  This is the factoring of expectation distributions.

This factoring is the application of Product distribution, which can be proven using the "law" of total expectations.

At $t_{video}$ = 44:51, the pattern recursion is expressed with a single ellipsis.  Recognition of the recursion pattern can be facilitated by re-expressing with a few modifications of mathematical style.

Use of the original variable to represent the value objective
Extending the equality
Using two ellipses instead of one clarify the recursion is a continuation of conditions from right to left as well as continuation of rewards from left to right
Strategically placing line returns

$$\pi_{\theta}(\tau) = \sum_{t=1}^T \operatorname{E}_{(s_t, a_t)\sim p_\theta(\tau)}[r(s_t, a_t)] \\ = E_{s_1 \sim p(s_1)}[E_{a_1 \sim \pi(a_1|s_1)}[r(s_1, a_1) \\ + E_{s_2 \sim p(s_2|s_1, a_1)}[E_{a_2 \sim \pi(a_2|s_2)}[r(s_2, a_2)] \\ + \; ... \\ ...|s_2]|s_1,a_1]|s_1]]$$

Inverting the transformation, recombining the expansion back to its original form may also be illustrative.


Can we implement ML engine using any general purpose Micro controller?

It depends on what you mean by "ML engine" and whether you want to train models on the uC or just make predictions.  IF you're doing something simple (maybe linear regression, logistic regression, etc.) you might be able to get away with doing training on a uC, especially for small amounts of data.  But you're almost certainly not going to be training deep neural networks on one of those things, at least not in any reasonable amount of time.

OTOH, the "making a prediction part" is usually much cheaper computationally, so if you have a pre-trained model and some prediction engine that can use that model, you could possibly use that on a microcontroller.


Is machine learning required for deep learning?

Deep learning is part of machine learning.

You will miss out useful information if you ignore machine learning.
You are ok to start your work in machine learning with deep learning and neural networks. You have to start somewhere and starting with a strong and successful method is resaonable, especially if you need to be able to produce good results quickly.
You will learn essential machine learning stuff while reading about deep learning.
The deep learning tutorials and other learning materials you will be reading may not be telling you that what you are learning also applies to other machine learning methods but you will be learning lot's of stuff that applies more generally. You will be studying some machine learning whether you want to or not.
If you have plenty of time a more broad view will help understanding. Still, there is no need to wait with deep learning to after mastering some other methods.
Broader knowledge helps you to relate and memorise concepts and be more aware of potential issues, especially issues that are rarely discussed in the deep learning community. Such knowledge and experience will be most useful when trying to apply deep learning to new problems or if trying to make substantial changes.


How to train an LSTM to classify based on rare historic event?

How about a Temporal Convolutional Network? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences.
There are also claims that TCNs are comparable to RNNs in performance on common tasks, so there's that.


Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?

It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.
Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.
Proposition. In a single-layer neural network with $n$ hidden neurons using $\mathrm{ReLU}$ activation, with one input and output node, the output is linear outside of the region $[A, B]$ for some $A < B \in \mathbb{R}$. In other words, if $x > B$, $f(x) = \alpha x + \beta$ for some constants $\alpha$ and $\beta$, and if $x < A$, $f(x) = \gamma x + \delta$ for some constants $\gamma$ and $\delta$.
Proof. I can write the neural network as a function $f \colon \mathbb R \to \mathbb R$, defined by
$$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$
Note that each neuron switches from being $0$ to a linear function, or vice versa, when $w_i x + b_i = 0$. Define $r_i = -\frac{b_i}{w_i}$. Then, I can set $B = \max_i r_i$ and $A = \min_i r_i$. If $x > B$, each neuron will either be $0$ or linear, so $f$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $x < A$.
Hence, $f$ is a linear function with constant gradient if $x < A$ or $x > B$.
$\square$
If the result isn't clear, here's an illustration of the idea:

This is a $3$-neuron network, and I've marked the points I denote $r_i$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.


What is the process working on Tensorflow model.fit()?

You can save weights during training by passing checkpoint callback to model.fit() method.
# Instantiate your model here
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...)

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Create a callback that saves the model's weights
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_1,
          train_labels_1,
          epochs=50,
          batch_size=batch_size,
          callbacks=[cp_callback],
          validation_data=(test_images_1, test_labels_1),
          verbose=0)

After finishing training 1st dataset, model weights will be saved in file called model_weights.ckpt. Before starting training next dataset, load the model weights as below
# Create a new model instance
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...)

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Load the previously saved weights
model.load_weights(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_2,
          train_labels_2,
          epochs=50,
          batch_size=batch_size,
          callbacks=[cp_callback],
          validation_data=(test_images_2, test_labels_2),
          verbose=0)

Repeat this for all datasets.


Before GAN, what are the commonly used techniques for image-to-image translation?

Image to Image translation is the task of transferring an image's characteristics from one domain and representing it in another. GANs have provided an end to end method to do this task.
Prior to Gans, these tasks were done individually, by using classic image processing techniques mainly.
Techniques such as image denoising, or finding edges in photos, or using web results to join various images were used.

As mentioned by P Isola et. al., in the CycleGAN paper, Hertzmann et al. in their paper, Image Analogies, have employed a non-parametric texture model on a single input-output training image pair.

In Image Quilting for Texture Synthesis and Transfer, the authors used existing patches of images and stitched them together.

In Data-driven Hallucination of Different Times of Day from a Single Outdoor Photo, here the authors compare the input image with an available dataset of time-lapse videos similar to the input and find the frame at the time of input image and frame for a target time. They use local affine transformation to change the scene of an input image into target image.

Recent works have focused on using a dataset of paired input-output examples to learn the translation function using CNNs for semantic segmentation,
Fully convolutional networks for semantic segmentation


How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips.

Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns?

The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and error.

If you are also interested in feedforward networks, see the question How to choose the number of hidden layers and nodes in a feedforward neural network? at Stats SE. Specifically, this answer was helpful.

  There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below:

  $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$

  $N_i$ = number of input neurons.
  $N_o$ = number of output neurons.
  $N_s$ = number of samples in training data set.
  $\alpha$ = an arbitrary scaling factor usually 2-10.

  Others recommend setting $alpha$ to a value between 5 and 10, but I find a value of 2 will often work without overfitting. You can think of alpha as the effective branching factor or number of nonzero weights for each neuron. Dropout layers will bring the "effective" branching factor way down from the actual mean branching factor for your network.

  As explained by this excellent NN Design text, you want to limit the number of free parameters in your model (i.e. its degree or the number of nonzero weights) to a small portion of the degrees of freedom in your data. The degrees of freedom in your data is the number samples * degrees of freedom (dimensions) in each sample or $N_s * (N_i + N_o)$ (assuming they're all independent). So $\alpha$ is a way to indicate how general you want your model to be, or how much you want to prevent overfitting.

  For an automated procedure you'd start with an alpha of 2 (twice as many degrees of freedom in your training data as your model) and work your way up to 10 if the error (loss) for your training dataset is significantly smaller than for your test dataset.


Why is Common Lisp, Python and Prolog used in artificial intelligence?

If we talk about applied AI, the choice of a programming language for an AI application has the same points to be taken into account that in any other software area: speed of generated code, expressive, reusable, etc.

By example, as training of a neural net is very expensive in CPU, languages as C/C++, that produces very optimized code, are very convenient. Moreover, there are GPUs librarians in C/C++ that allows use of strong parallelism.

A system with some complexity will combine more than one language, in order to use the best of the language in the points where it is need.

But returning to the list of languages that appears in the question. As all them are Turing complete compare them means talk about its paradigm, features, syntax and available compilers/interpreters. Obviously, something that exceeds the possibilities of a simple answer. Just to show some key points about the ones mentioned in the question:

Prolog is a programming paradigm by itself. Its main advantage was that Prolog sentences are independent from remainder ones and near to the mathematical definitions of the concepts. Moreover, it is itself a database. Its drawback are also well known: very slow, lack of librarians for i/o, ... . Very interesting (even mandatory) to known a few examples of algorithms in Prolog, but I doubt nobody is using it nowadays, except in obsolete university courses (when you reach the "!", cut its study).

Lisp is also a zombie. Its functional paradigm has been now included in lots of very more modern languages, combining it with object oriented paradigm: scala, haskell, ocaml/F#, ... . Being functional allows a syntax that made easier to express logic concepts as recursive definition of logic or types, ... . Something very interesting in AI.

In the category of object oriented paradigm and valid for all applications, we have Python ( easy to learn, fast prototyping, slow, ... ) C/C++ (very optimized code), Java, ... . More or less, all them are adopting also functional features in latest standards.

In AI there are a lot of very interesting language features to be also considered: rule based systems, ... . Librarians for them can be found in all main languages.

Finally, some words about AGI (strong AI): you do not need a computer. In best moments, we are at the stage of pencil and paper, remainder ones looking at ceiling.


What is the appropriate way to deal with multiple paths to same state in MCTS?

If the state appears twice in the tree, aren't we wasting a lot of resources thinking about it multiple times?

You're right. Precisely the same problem was also noticed decades before MCTS existed, in the classic minimax-style tree search algorithms (alpha-beta search, etc.) that were used in games before MCTS. The solution is also mostly the same; transposition tables.

In the case of MCTS, the statistics used by the algorithm that are normally associated with nodes (or their incoming edges) may instead be stored in entries of a transposition table. I mean stuff like visit counts and sums (or averages) of backpropagated scores.

A brief description of how it would work, and references to more extensive relevant literature, can be found in subsubsection 5.2.4 of the well-known 2012 Survey paper on MCTS.

This does require that you can efficiently (incrementally) compute hash values for the states you encounter, which may not always be easy (should usually be possible, depends on the details of your problem domain). Use of transpositions in MCTS is also not always guaranteed to actually improve performance. It does come with computational overhead, and in games where transpositions are very rare it may be more efficient to simply ignore them and use the regular tree structure.


How to deal with the terminal state in SARSA in a multi-agent setting?

The SARSA update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma Q(S', A') \right].$$
Very similar, the $Q$-learning update rule looks like:
$$Q(S, A) \gets Q(S, A) + \alpha \left[ R + \gamma \max_{A'} Q(S', A') \right].$$
Both of these update rules are formulated for single-agent Markov Decision Processes. Sometimes you can make them work reasonably ok in multi-agent settings, but it is crucial to remember that these update rules should still always be implemented "from the perspective" of a single learning agent, who is oblivious to the presence of other agents and pretends them to be a part of the environment.
What this means is that the states $S$ and $S'$ that you provide in update rules really must both be states in which the learning agent is allowed to make the next move (with the exception being that $S'$ is permitted to be a terminal game state.
So, suppose that you have three subsequent states $S_1$, $S_2$, and $S_3$, where the learning agent gets to select actions in states $S_1$ and $S_3$, and the opponent gets to select an action in state $S_2$. In the update rule, you should completely ignore $S_2$. This means that you should take $S = S_1$, and $S' = S_3$.
Following the reasoning I described above literally may indeed lead to a tricky situation with rewards from transitioning into terminal states, since technically every episode there will be only one agent that directly causes the transition into a terminal state. This issue (plus also some of my explanation above being repeated) is discussed in the "How to see terminal reward in self-play reinforcement learning?" question on this site.


Why is this ResNet50 misclassifying objects?

Your problem is a classification problem. If you are following the tutorial, it is using the ResNet50 network, which is a convolutional neural network with one fully connected layer at the end. At the end the activation function is softmax. Detailed description of the activation function can be found here: Softmax function explained

Basically, softmax increases the difference between the higher probability and lower probability. It also limits the output between 0 and 1.

Problem Origin

Due to the nature of the softmax function, it always chooses the best one and enlarge the value to be a value near one, even if the range of the output predictions is very small like 0-0.1. Also you training data only have the data of the 10 logos labelled, so if the network see unseen images with no logo it recognizes, it predicts the one with the most similarity. If you want to classify images with no logos in it, you should add an extra class in the training dataset and also the code to train the network to learn to classify images which is not in the 5 class into a separate maybe called unlabeled. Hope I can help you and have a nice day.


Does the substituted variable/constant have to appear in the unified term?

Hey I am currently studying First Order Logic right now and I think I can answer your question. Others please correct me if I am wrong.

For the first case, you can generally substitute variables with constants. Hence, you can make the substitution $\theta \leftarrow \{x_1 \leftarrow A, y_1 \leftarrow B \}$. This is used very commonly when you want to infer some query $\alpha$ from your knowledge base. $\alpha$ usually is in the form $P(A,B)$ as you have mentioned. When you unify you get $\neg L(A) \vee H(A)$ and ur substitution has to stay the same throughout the resolution algorithm. I.e, you cannot substitute $x_1$ for another constant / variable.

Regarding the second case, you cannot substitute a constant with a variable $B$ to another variable.

In general, you can substitute a variable with a constant, or another variable. You can also substitute a variable with a skolem function, Eg. $x_1 \leftarrow G(y)$. However, for skolem functions, you cannot substitute $x \leftarrow F(x)$, in which the variable names are the same.

Hope this helps !