What are the various methods for speeding up neural network for inference?

These are some ways to speed up inference:

Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8.
Reference paper: https://arxiv.org/pdf/1502.02551
Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.
Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, but, the speed is on average 7x faster. Reference paper: https://pjreddie.com/media/files/papers/xnor.pdf

https://software.intel.com/en-us/articles/binary-neural-networks

Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.


What are the known ways to test AI black boxes?

On the applicability of Artificial Intelligence in Black Box Testing (Khanna, 2017) might be a good place to start. It classifies some black box testing techniques based on the AI branch it belongs to.


Can I do state space quantization using a KMeans-like algorithm instead of range buckets?

There is this paper Representation and Reinforcement Learning for
Personalized Glycemic Control in Septic Patients, presented in the Machine Learning for Health Workshop in NIPS 2017. Here is a quote from the paper where the authors describe the clustering approach:

  After we generated the state representation,
  we used the k-means clustering algorithm to categorize millions of patient states into 500 clusters
  such that similar clinical states can collapse into the same cluster.


How to produce documents like factset blackline?

Financial information companies spend a very large amount of effort to do this kind of thing properly, and their models are generally proprietary, so the real answer is not something you'll be able to get on a public site.

Of course, you can achieve a very simple approximation with a literal diff on the texts, and a more sophisticated one by using out-of-the-box NLP or computer vision techniques, but you're probably not going to get close to the performance of factset or other competitors using those tools.


Recommendations or resources for neural network/deep learning for time series application?

I'm currently working with Temporal Convolution Networks (TCNs) for making predictions with time series data (link to article here: https://medium.com/@raushan2807/temporal-convolutional-networks-bfea16e6d7d2).  These types of networks, like other types of convolutional networks for time series, use a dilated convolution operation, which, unlike the standard convolution in these networks, is a causal operation that preserves the causality of the time series input.
I've also seen CNNs used in Temporal Differencing applications, but that may be more useful in the context of video processing rather than time series analysis.


Are there any meaningful books entirely written by an artificial intelligence?

James Ryan has done a lot of 'archaeological' work on this; you can find references to his work on his website.
Story generation has been a dream for a long time (in computing terms), and various genres have been explored, with not that much success. There have been episodes of a Western written by a computer (and actually filmed and acted out by human actors, see summary here), and various books, but the technology is nowhere near good enough to produce something worthwhile reading or watching without heavy editing.
So far it's only good for curiosity value.
There is NaNoGenMo, where since 2013 people work on programs generating novels. But most of them — again — are more interesting for curiosity. They either take an existing work and modify it procedurally, or generate random templated text (eg travel reports through a ficticious, auto-generated landscape). I don't think anyone has met the target of creating a 50k novel by computer yet.
Modern approaches with deep learning generators can produce reams of more or less well-formed text, but that doesn't make a compelling story. It's the meaning that is currently the problem, and as AI researchers keep finding out, it's hard.


What is the role of embeddings in a deep recurrent Q network?

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.
$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation)

According to to section 6.1 of the paper, the input is a tuple represented as $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$. Each of these  terms is described in sec 3 as:

$o_t$ - The observation. The authors assume a POMDP

Two types of actions:

$u_t$ - An environment action selects by all the agents at each time step in exchange for a team reward
$m_t$ - A communication action, observed by other agent but has no direct impact on the reward or env

$a$ - The agent (This being a multi-agent DQN algorithm)

They form the input to the Deep Recurrent Q-Network architecture.
The purpose of the embedding network is to receive a tuple of these inputs and produce a state
embedding $z$. This state embedding is then fed  into a hidden state $h^a_{t-1}$ of the RNN.
Though the authors refer to it as an input network that produces embedding, in practice, they use different embedding functions for each of these inputs. The final task/state embedding $z_t^a$ is expressed as a sum:
$$
z_t^a = \text{MLP}(o_t^a) + \text{MLP}(m_{t-1}) + \text{LookupTable}(u_{t-1}) + \text{LookupTable}(a)
$$
The summed embeddings are all of the same size.
This answer makes an assumption that by "embedding layer" you meant the input embedding network. Since the paper makes no reference to a single embedding layer in the model architecture.


What is the difference between 'prediction' and 'control' problem in the context of Reinforcement Learning?

Prediction is the problem of predicting any feature of the environment. In reinforcement learning, the typical feature is the reward or return, but this doesn't have to be always the case. See Multi-timescale nexting in a reinforcement learning robot (2011) by Joseph Modayil et al.
Control is the problem of estimating a policy. Clearly, the term control is related to control theory. In fact, the term control is often used as a synonym for action. See Is there any difference between a control and an action in reinforcement learning?. Similarly, the term controller is also used as a synonym for agent. For example, in the paper Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. Hamrick et al. the term meta-controller is used to refer to an agent. A controlled system can also refer to the environment.
Section 14.1 of the book Reinforcement learning: an introduction (2nd edition) provides more details about the distinction between prediction and control and how this distinction is related to psychological concepts.


Does the correlation between inputs affect the model performance?

Non-correlation does not imply independence, that is, if two features are not correlated (i.e. zero correlation), it does not mean that they are independent. But (non-zero) correlation implies dependence (see https://stats.stackexchange.com/q/113417/82135 for more details). So, if you have non-zero correlation between two features, it means they are dependent. If they are dependent, then one feature gives you information about the other and vice-versa: in a certain way, one of the two is, at least partially, redundant.

Unnecessary features might not affect the performance (e.g. the accuracy) of a model. However, if you reduce the number of features, the learning process might actually be faster.

You may want to try some dimensionality reduction technique, in order to reduce the number of features.


According to this source, one FLOP is roughly equal to two MAC (multiply accumulate). My guess/understanding would be that the distinction is made because neural nets spend compute overwhelmingly on multiply-accumulate operations, and thus optimizations and statistics over MAC operations would be more significant than FLOPs.


Can DQN announce it has things in its hand in a card game?

The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.

However, if you want to explore tactics surrounding declaration, then I think the following applies:

There should be an initial round of actions where the agent may get to decide whether or not to declare a Ronda, based on the cards it holds. These will be different action choices to playing cards, so you would need to alter your action representation to include those choices. Only allow action choices which are valid, so if it is not valid to declare a Ronda or a Tringa when a player does not have one, then the player does not get to make that choice.
You may want to add a state feature "has a Ronda" and "has a Tringa" for the agent's player, to help with the action decision.
You should also add a state feature for each player according to whether they declared a Ronda or a Tringa.
Rather than have the agent learn to detect a lie, given your comment that all cards are played so it is easy to tell (there are only 8 cards total in play by the end), then I would just assume lies are automatically found out and include that in the game engine. In other words, the penalty is always paid.
The interesting question is then whether withholding the declaration can give a tactical advantage when the round is being played (because your opponent knows less about your hand), and whether that advantage offsets the inevitable penalty. This might not be true in your card game, but could be true in games with similar choices.

  Could a DQN handle this?

A DQN is maybe going to struggle with partial information in this game. The opponent's cards are hidden from the network, but could have a non-random influence on the opponent's choices of action. It is possible that you will need to investigate agents that can solve POMDPs to get the best player.

I don't know for certain though. It depends on how much tactical advantage there is in having concealed cards, or how much that is just luck that plays out much the same whether you know the opponent's cards or not. If there is strategy, and ways to determine/guess what the opponent holds based on your cards and their actions so far, this is more like a POMDP.


In what ways is the term "topology" applied to Artificial Intelligence?

In addition to the ways the term topology is itself used generically to describe the "shape" of various aspects of Machine Learning, the term appears in the field Topological Data Analysis:

  In applied mathematics, topological data analysis (TDA) is an approach to the analysis of datasets using techniques from topology. Extraction of information from datasets that are high-dimensional, incomplete and noisy is generally challenging. TDA provides a general framework to analyze such data in a manner that is insensitive to the particular metric chosen and provides dimensionality reduction and robustness to noise. Beyond this, it inherits functoriality, a fundamental concept of modern mathematics, from its topological nature, which allows it to adapt to new mathematical tools.

Some examples of its use in ML:

The ring of algebraic functions on persistence bar codes, Homology, Homotopy and Applications
Persistence Images: A Stable Vector Representation of Persistent Homology
Topological data analysis and machine learning theory
Topological Kernel Learning: Discrete Morse Theory is used to connect kernel machine learning with topological data analysis.


A3C fails to solve MountainCar-v0 enviroment (implementation by OpenAi gym)

I don't know about your first question, but I got a  basic policy gradient approach with the kinetic energy as reward working on MountainCar-v0.

You can implement it based on this blog and the notebook you find there. It uses an MLP with one hidden layer of size 128 and standard policy gradient learning.

The reward engineering boils down to replacing the reward variable with the kinetic energy $v^2$ (no potential energy and no constant factor, the reward itself is not used). I takes $>1000$ episodes to solve the environment consistently.

I'm afraid the solution is not very satisfactory and I don't have the feeling there is much to learn from it. The solution is originally for the cartpole problem and it stops working for me if I change hyperparameters/optimizer or the specifics of the reward.


Are all breakthroughs in AI and machine learning are due to increase in computational resources?

High-level answer: Increase in resources has been important in AI, and definitely was a factor with Deep Blue, but Machine Learning is a newer method that seems to produces more optimal results with less resources on problems of greater complexity.

Here is an article on AlphaGo's hardware: "Google reveals the mysterious custom hardware that powers AlphaGo"

Also an interesting analysis on Quora: "What hardware does AlphaGo run on? Is it customized hardware for best performance?"

Still pretty powerful systems, but I think the algorithms are as important as the computing resources, because all the hardware in the world won't help in the algorithms are poor.

Matthew Lai, creator of Giraffe Chess, said:

  "In the ensuing two decades [since Deep Blue], both computer hardware and AI research advanced the state-of-art chess-playing computers to the point where even the best humans today have no realistic chance of defeating a modern chess engine running on a smartphone."Source: TechExplore

which suggests that hardware and software are both important parts of the overall equation.


Can bounding boxes further improve the performance of a CNN classifier?

Another way to ask the question is: Does sound get clearer when you remove the background noise?

The obvious answer is yes and in the case of image classification, the answer is also generally yes.

In most cases reducing the noise (irrelevant pixels) will strengthen the signal (activations) the neural network is trying to find.


The paper A Brief Introduction to Statistical Shape Analysis (2002) by M. B. Stegmann and D. D. Gomez provides a definition of a landmark in the context of statistical shape analysis, which I will report below.

Definition 1: Shape is all the geometrical information that remains when location, scale and rotational effects are filtered out from an object.

For example, in the following diagram, the shape of a hand, after several transformations (i.e. translation, scaling and rotation), is illustrated. This definition captures the intuitive notion of the shape of an object. In fact, as expected, after these transformations, all of the objects below correspond to a hand.

Definition 2: A landmark is a point of correspondence on each object that matches between and within populations.

In your picture, all of the faces (apart from the occluded one) have 18 green points. Each of these points roughly corresponds to the same position in the face. For example, there's a green point on tip of the nose of each face. In your picture, the populations refer to the faces.
You can divide landmark into three groups

Anatomical landmarks Points assigned by an expert that corresponds between organisms in some biologically meaningful way.

Mathematical landmarks Points located on an object according to some mathematical or geometrical property, i.e. high curvature or an extremum point.

Pseudo-landmarks Constructed points on an object either on the outline or between landmarks.

The landmarks in your picture could be considered anatomical landmarks, so they might have been assigned by an expert (a human), so this should explain why they do not exactly correspond. Moreover, note that e.g. noses have different shapes across humans, so correspondences are unlikely to be exact in most cases.

What is the size of a landmark?

This is just an educated guess, but a landmark is probably just a point (i.e. a coordinate of a pixel in the image), although in your picture they look bigger, but this is probably to make them more noticeable.

What is the acceptable error of its position?

This is also a guess, but this might depend on the application and the used algorithms, i.e., in certain cases, your algorithms may be very sensitive to the position of the landmarks, in other cases, that might not be a big problem.


What's the value of making the RL agent's output stochastic opposed to deterministic?

My understanding of your question is, you have 2 designs:

A deterministic policy that outputs 2 scalar for x and y respectively.
A value function that outputs the probability of each pixel in the 2D grid.

If you choose the max of softmax on (2.), you'll get the same deterministic policy as (1.), assuming there are some tie-breaking designs. So I don't think choosing the max of softmax is a great policy.

So, the question becomes "deterministic policy vs. value function". Since the value function have more information, we can have special designs on exploration.

In (1.), if you use $\epsilon$-greedy on the discrete output, It'll take a random move with $\epsilon$ probability. When exploring (instead of exploiting), all actions except the best action will have the same probability.

However, in (2.), the probability can act like a value function (the value of $Q(s, a) \forall a$ in a certain state $s$), and you can use something like upper confidence bound (UCB) instead of $\epsilon$-greedy. If you are using neural networks to output this probability distribution, you can add an entropy term in the loss function like in A3C to encourage exploration.


Are there any decentralized examples of AI systems with blockchain technology?

Swarm intelligence is the term for systems where relatively simple agents work together to solve a complicated problem in a decentralized fashion.

In general, distributed computing methods are very important for dealing with problems at scale, and many of them embrace decentralization in a deep way. (Given the reality of hardware failure and the massive size of modern datasets relative to individual nodes, the less work is passed through a central bottleneck, the better.)

While there are people interested in doing computation on the blockchain, it seems to me like it's unlikely to be competitive with computation in dedicated clusters (like AWS).


Can the opponent's turn affect the reward for a DQN agent action?

Does the opponent's turn affect the calculated rewards?

Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.

  As far as I know, the reward should only be the result of the agent's action right?

In a well-defined MDP, the reward should be a stochastic function of the current state and the agent's action. The stochastic part can include any changes due to an opponent player.

If the opponent player is random, or follows a well-defined and fixed policy, then you consider them part of the environment. So this requirement is met technically. The reward does only depend on current state and the agent's action. The actual result may happen on the opponent's turn, but that does not matter.

In a card game where the opponent's cards are hidden and affect their strategy, this may not strictly be the case, because the visible state will not determine the opponent's behaviour. The problem stops being an MDP, and starts being a POMDP. Whether or not that impacts the agent will depend on how much strategy relies on the hidden nature of these cards. In blackjack, there is little impact to not knowing an opponent's cards before they are played out - there is little difference between hidden cards and cards that are randomly determined after the agent plays. So you can get away with pretending it is a normal MDP. In poker, the knowledge of hidden cards is almost everything about the game, so a POMDP or other approach that tracks possible hidden state is required.

Note that learning to defeat a random or expert player is usually not the same as learning to play optimally (unless your expert player is already optimal). For that you may need self-play and an agent which learns both players' policies.


Why is the perceptron criterion function differentiable?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$.
Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights.
Assume $y_i = 1$ and $w x_i < 0$ (that is, an error of type "false negative").
In this case, function $[y_i - \text{sign}(w x_i)]^2 = 4$. Derivative on all interval $w x_i < 0$ is zero, thus, the learning algorithm has no any way to decide if it is better increase or decrease $w$.
In same case, $\max(-y_i(w x_i), 0) = - w x_i$, partial derivative is $-x_i$. The learning algorithm knows that it must increase $w$ value if $x_i>0$, decrease otherwise. This is the real reason this loss function is considered more practical than previous one.
How to solve the problem at $w x_i = 0$ ? simply, if you increase $w$ and the result is an exact $0$, assign to it a very small value, $w=\epsilon$. Similar logic for remainder cases.


How does Wolfram's Image Identification Project work?

The ImageIdentify project uses the highly automated "superfunctions" and as part of Wolfram Language API integration. It relies on a complex collection of meta-algorithms and built-in 'knowledge'. It has a built-in classifier trained from a large dataset using Wolfram Data Framework (WDF). However the main classifier is based on the deep neural networks.

Source: How the Wolfram Language Image Identification Project Works

The algorithm isn't perfect and misidentification are more likely to be caused by 'irrelevant objects repeatedly being in training images for a particular type of object'.

You can read more at Wolfram Language Artificial Intelligence: The Image Identification Project.


How to manually draw a $k$-NN decision boundary with $k=1$ given the dataset and labels?

This is a rather involved task. What to do from a high-level theoretical perspective might be easy to see, but it's difficult putting that into code from scratch.
Doing this in Python using existing libraries in not too complicated, though.
See for example this tutorial or this StackOverflow post.
Edit:
Theoretically, I would first plot (draw) the points from your dataset in a graph and then watch out for "decision points" half-way in between any two near-by points (from the dataset) from distinct classes.
For the next step, keep in mind those decision points. Given close-by data points from classes (e.g.) A and B, imagine a straight line connecting these two points from separate classes. Next, take the point half-way along that imaginary line (i.e. your decision point) and draw a "soft" line (maybe using pencil instead of pen) orthogonal/perpendicular to that imaginary line which intersects the imaginary line in the decision point.
Do that for all combinations of "reasonably" close points from different classes.
Parts of the lines you have just drawn will define the final decision boundary. Next, think of each line as consisting of multiple elements, which are separated from one another by means of the points of intersection with other drawn lines. In other words, split lines into elements wherever they intersect other lines. Now, decide which of these elements to outline as the eventual decision boundary (finally using a pen instead of pencil). This step simply involves human intelligence and is difficult to describe. Whenever a line's element accurately separates (logically speaking) two classes, indicate it using a pen. Otherwise, if it does not contribute to separating two classes, don't indicate it.
After having indicated the final decision boundary using a pen, simply erase the pencil drawings. Now, you should be left with a decision boundary.
I hope this is clear and accurate enough.


Neural network with logical hidden layer - how to train it? Is it policy gradient problem? Chaining NNs?

The chain rule applies here as usual, and the term symbolic gradient is interesting.

How such might apply will depend much on the nature of the policy layer representation and the connections of that layer or layers to other non-policy layers.

A binary threshold layer may feed a logical inference layer.
A sigmoid function might feed a fuzzy logic layer.
The logical layer could be a production system adapted to look like a layer.
The logical layer might be more like a PLD (programmable logic device).
The output of the logical layer might be mapped one to one with a perceptron or an LSTM layer

In the example in the question, we can't, from the information given in the question, assume that discrete means a discrete approximation of a curve, since the term used is discrete/symbolic. Since the nature of logic cannot be generalized in a differentiable closed form, it may be that it has to be probabilistically represented in closed form so a derivative may be obtained, if that is possible in the specific case under study.

Back propagation is a feedback mechanism to provide corrective signaling to the portions of the system that must be corrected. The hope is that the direction of the correction during back propagation leads to convergence on the global minimum of loss without severe delay, which is not always easy to make happen, thus Gaussian noise injection explicitly or indirectly through mini-batching, momentum, multiple initialization states, parallel searches with different hyper-parameters, and other techniques.

One technique used quite a bit in analog systems and entering use in digital systems in recent years is the idea of multiple corrective mechanisms. Control theory for instrumentation and countermeasure aeronautics is thick with this concept. For the case given in this question, it can look like this, with $f$ representing feed forward layer with some activation function, $\ell$ representing some logical inference container, $p$ is the parameter set for the function of the same subscript, $\epsilon$ is the evaluation function (error, loss, value, benefit), $b$ representing the corrective feedback using the back propagation, the Jacobian, and the chain rule.

$$ f_1 \quad \rightarrow \;\quad \ell \quad \rightarrow \quad\ f_2 \quad \rightarrow \quad\ \epsilon \\
 p_1 \leftarrow \, b \leftarrow \ell\, ; \quad\quad\quad \; p_2 \, \leftarrow b \leftarrow \epsilon  \\
\quad\quad\quad\quad\quad \ell \quad\quad \leftarrow \quad\quad\quad\quad\quad \epsilon
 $$

In the last line $\epsilon$ is simply a fact (piece of information) passed into the a rules engine session or a floating point representation of a feedback signal assigned as a probability to a rule in a fuzzy logical container. In the first half of the second line, the logical unit is responsible for providing the evaluation function output.

In Norbert Wiener's Cybernetics, he states the following1 with regard to automating the steering of ships via rudder control, from which cybernetics got its name.

  In the important book of MacColl2, we have an example of a complicated system which can be stabilized by two feedbacks but not by one.

He then procures the differential equations and proves MacColl's conclusion. That conclusion is used in nearly every instrumentation amplifier circuit on a chip sold today, if not all of them. There is no way to get the desired stability without the multiple feedback paths.

Similarly, it may not be possible to stabilize the convergence of $f_1, \ell, and f_2$ via a single mechanism of back-propagation using $\epsilon$ and the Jacobians to descend a gradient.

References

[1] Norbert Wiener's *Cybernetics, 1948, MIT, p 106 in the 1996 printing

[2] LeRoy A. MacColl. Fundamental Theory of Servomechanisms. Van Nostrand, New York, 1945


How does AlphaZero's MCTS work when starting from the root node?

I looked at the Python pseudo-code attached to the Data S1 of the Supplementary Materials of the AlphaZero paper. Here is my findings:

Contrary to the paper, AlphaZero does not store $\{N(s, a), W(S, a), Q(s, a), P(s, a)\}$ statistics for each edge $(s,a)$. Instead, AlphaZero stores $\{N(s), W(S), Q(s), P(s)\}$ statistics for each node $s$.
When a leaf node $S_L$ is expanded, it's visit count, value scores, and action policies are immediately updated in $\{N(s), W(S), Q(s), P(s)\}$, so $N(s)$ is at least $1$. This is why in the paper, the backprop step updates for all time steps $t \le L$ rather than $t < L$. It makes sense to update $s_L$ even though there is no corresponding $a_L$ to pair it with.
Therefore, when a new leaf node is expanded, the value $U(s, a)$ of a child of that leaf node will be nonzero, since $\sqrt{\sum_b N(s,b)}$ is actually computed as $N(s_{parent})$ in the code, which is at least 1.
Oddly enough, I think there might be a bug in the pseudocode, because at the beginning on the first iteration (starting at the root node), $U(s,a) = 0$ for all child nodes of the root node. This is because at the first iteration, $N(s_{root}) = 0$. The value of all child nodes will be $0$, and since the authors chose to break ties according to Python's max function, the algorithm simply chooses the first element it finds in case of a tie.
After the first iteration, $N(s_{root}) > 0$ and so $U(s,a) \neq 0$ and things proceed as normal since the backprop step will have updated the visit count of the root node. So this possible bug/unintuitive behavior only affects the first iteration. It is extremely minor and insignificant, and does not affect the outcome of the MCTS, which is probably why it went unnoticed.


How much can the addition of new features improve the performance?

It depends on the used network as well as the feeding mechanism but let's give an example;

When working with LSTM, giving the time data (as an integer sequence) in addition to the time-series data(coming from features) dramatically increases the performance of the network.

[$X_{0}$,$X_{1}$, ...] $\rightarrow$ [[$X_{0}$,$t_{0}$],$[X_{1}$,$t_{1}$], ...]

If you go and look for the kaggle competition winner's notebooks, they do also create additional features based on the featured data.

Let's assume that the performance is already quite high on the three features so that you can predict those three features with high reliability.
It would only make sense to increase the number of features if you would like to predict additional features!


Is there any resource that describes in detail a naive example-based machine translation algorithm?

I have not found any simple implementation of a naive EBMT system, but I found some articles, papers and books that may be helpful (although I haven't read them, apart from the first and last one), so I will list them below.

The web article Example-based machine translation provides a decent high-level explanation of example-based machine translation.

The paper Example-Based Machine Translation: A New Paradigm (2002) by Chunyu Kit et al. also seems to provide a detailed description of the EBMT approach, so this paper should provide you with details you need to implement an EBMT system.

The paper A framework of a mechanical translation between Japanese and English by analogy principle (1984) by Makoto Nagao introduced the example-based machine translation approach, so it will be at least historically relevant.

Additionally, the paper Example-Based Machine Translation of the Basque Language and the book Recent Advances in Example-Based Machine Translation (2003), which is not apparently freely available online, could also be useful.

Finally, the article Machine Translation. From the Cold War to Deep Learning gives a nice high-level overview of the main machine translation approaches, so that you can understand the differences between EBMT and other approaches (especially, in case you are not able to distinguish between EBMT and other MT, e.g. those that use a parallel corpus, such as the supervised statistical machine translation approaches).


What would be the implications of mistakenly adding bias after the activation function?

Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i \right)+b$, then the partial derivative of $L$ with respect to $b$, in Leibniz's notation, is $\frac{\partial L}{\partial b} = 1$. Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i + b \right)$, then the partial derivative of $L$ with respect to $b$ is $\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \sigma} \frac{\partial \sigma}{\partial b}$.

So, in general, the partial derivatives with respect to $b$ would be different in these two cases, thus, in the gradient descent step, $b$ would be updated differently in both cases. In practice, it may or not affect the performance of the model, depending on the importance of the bias with respect to the given problem. Read this answer to understand the role of the bias.


Most Deep Q-learning implementations I have read are based on Deep Q-Networks (DQN).  In DQN, the q-value network maps an input state to a vector of q-values, one for each action:

$$
Q(s, \mathbf{w}) \to \mathbf{v}
$$

where $s$ is the input state from the environment, $\mathbf{w}$ are the parameters of the neural network, and $\mathbf{v}$ is a vector of q-values, where $v_i$ is the estimated q-value of the ith action. In the Sutton and Barto book, the q-value function is written as $Q(s, a, \mathbf{w})$, which corresponds to the network output for action $a$.

Unlike tabular Q-learning, Deep Q-learning updates the parameters of the the neural network according to the gradients of the loss function with respect to the parameters. DQN uses the loss function

$$
L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
$$

where $\gamma$ is the discount rate, $a$ is the selected action (either greedily or randomly for an $epsilon$-greedy behavior policy), $s'$ is the next state, $a'$ is the argmax action for the next state, and $\mathbf{w^-}$ is an older version of the network weights $\mathbf{w}$ that is used to help stabilize training.

In deep Q-learning, training directly updates parameters, not q-values. Parameters are updated by taking a small step in the direction of the gradient of the loss function

$$
\mathbf{w} \gets \mathbf{w} + \alpha [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})] \nabla_w Q(s, a, \mathbf{w})
$$

where $\alpha$ is the learning rate.

In frameworks like tensorflow or pytorch the derivative is calculated automatically by giving the loss function and model parameters directly to an optimizer class which uses some variation of mini-batch gradient descent. In eagerly executed tensorflow updating the parameters for a mini-batch might look something like

batch = buffer.sample(batch_size)
observations, actions, rewards, next_obervations = batch

with tf.GradientTape() as tape:
    qvalues = model(observations, training=True)
    next_qvalues = target_model(next_obervations)
    # r + max_{a'} Q(s', a') for the batch
    target_qvalues = rewards + gamma * tf.reduce_max(next_qvalues, axis=-1)
    # Q(s, a) for the batch
    selected_qvalues = tf.reduce_sum(tf.one_hot(actions, depth=qvalues.shape[-1]) * qvalues, axis=-1)
    loss = tf.reduce_mean((target_qvalues - selected_qvalues)**2)

grads = tape.gradient(loss, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))

Though I am not familiar with the Encog neural network framework you are using, based on the example Brain.java file from your Github repo and Chapter 5 of the Encog User Manual and the Encog neural network examples on Github it looks like weights are updated as follows:

A training set is constructed from pairs of input and target output.
A Propagation instance, train, is constructed with a network and training set. Different subclasses of Propagation use different loss functions to update the network parameters.
The method train.iterate() is called to run the network on the inputs, calculate the loss between the network outputs and target outputs, and update the weights according to the loss.

For DQN, a training set is constructed from a random sample from the experience replay buffer to help stabilize training. A training set could also be the trajectory of an episode, which is what the tupels argument in the example code of the question appears to be.

The input would be the statefirst member of each element of tupels. Since the network produces a vector of q-values, the target output must also be a vector of q-values.

The target output element for the selected action is $r + \gamma max_{a'} Q(s', a', \mathbf{w^-})$,  In the example code of the question, this is

double qnew = 0;
if(i <= tupels.size()-2){
    qnew = tupels.get(i).rewardafter + discountfactor*qMax(tupels.get(i+1));
} else {
    qnew = tupels.get(i).rewardafter;
}
tupels.get(i).qactions.elements[tupels.get(i).actionTaken] = qnew

The target output elements for actions that were not selected should be $Q(s, b, \mathbf{w})$, where $b$ is one of the non-selected actions.  This should have the effect of ignoring the q-values of non-selected actions by making the network output equal to the target output.

So what are the new Q - Values, assuming a discount factor of 0.99 and the learning rate 0.1?

Assuming you mean target outputs by the new Q - Values, and given the trajectory of actions, (1, 1, 1), and q-value vectors from the question, the concrete target outputs are (0, 0 + 0.99 * 0, -5, 0), (0, 0 + 0.99 * 0, 0, 0), and (0, 1 + 0, 0, 0).


The truth value of an array with more than one element is ambiguous - Loading a model saved in h5 format keras 2.2.4

It turns out that the solution to this problem is in the version of h5py.
If you have h5py == 3.1.* then you need to downgraded it to h5py == 2.10.0
Version of Keras needs to be 2.2.4
Or you can upgrade the version of Keras to 2.2.5


Has there been "prescriptive" attempts at language-learning/comprehension in AI?

Yes.  You're basically looking for "rule based" NLP.  For reference, see:

https://en.wikipedia.org/wiki/Rule-based_system

https://www.quora.com/Why-are-rule-based-methods-becoming-unpopular-in-NLP-Are-rule-based-methods-still-in-use-If-yes-where-should-I-look-for-them

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3756254/

https://nlp.stanford.edu/software/regexner.html

http://u.cs.biu.ac.il/~yogo/courses/sem2014/intro.pdf

etc.