Probabilities, Bayes and the Free-energy principle

Laurent U Perrinet, INT

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane - ANR BalaV1
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini - FACETS-ITN Marie Curie Training

Telluride Neuromorphic Workshop - "Compound Eyes and Event-based Vision" - June 28th, 2017 https://laurentperrinet.github.io/talk/2017-06-30-telluride

Outline

Probabilities and Bayesian inference

Variational Inference and the Free-energy principle

Active inference, EMs & oculomotor delays

Probabilities and Bayesian inference

Lets try this using this notebook (solution)

Instead, let’s focus on one variable like the weather. We know how probable it is that it’s sunny or raining. For both cases, we can look at the conditional probabilities. How likely am I to wear a t-shirt if it’s sunny? How likely am I to wear a coat if it’s raining?

There’s a 25% chance that it’s raining. If it is raining, there’s a 75% chance that I’d wear a coat. So, the probability that it is raining and I’m wearing a coat is 25% times 75% which is approximately 19%. The probability that it’s raining and I’m wearing a coat is the probability that it is raining, times the probability that I’d wear a coat if it is raining. We write this:

p(rain,coat)=p(rain)⋅p(coat | rain)

This is a single case of one of the most fundamental identities of probability theory:

p(x,y)=p(x)⋅p(y|x)

We’re factoring the distribution, breaking it down into the product of two pieces. First we look at the probability that one variable, like the weather, will take on a certain value. Then we look at the probability that another variable, like my clothing, will take on a certain value conditioned on the first variable.

Probabilities and Bayesian inference

(see http://colah.github.io/posts/2015-09-Visual-Information/)

The choice of which variable to start with is arbitrary. We could just as easily start by focusing on my clothing and then look at the weather conditioned on it. This might feel a bit less intuitive, because we understand that there’s a causal relationship of the weather influencing what I wear and not the other way around… but it still works!

Let’s go through an example. If we pick a random day, there’s a 38% chance that I’d be wearing a coat. If we know that I’m wearing a coat, how likely is it that it’s raining? Well, I’m more likely to wear a coat in the rain than in the sun, but rain is kind of rare in California, and so it works out that there’s a 50% chance that it’s raining. And so, the probability that it’s raining and I’m wearing a coat is the probability that I’m wearing a coat (38%), times the probability that it would be raining if I was wearing a coat (50%) which is approximately 19%.

p(rain,coat)=p(coat)⋅p(rain | coat)

This gives us a second way to visualize the exact same probability distribution.

Note that the labels have slightly different meanings than in the previous diagram: t-shirt and coat are now marginal probabilities, the probability of me wearing that clothing without consideration of the weather. On the other hand, there are now two rain and sunny labels, for the probabilities of them conditional on me wearing a t-shirt and me wearing a coat respectively.

(You may have heard of Bayes’ Theorem. If you want, you can think of it as the way to translate between these two different ways of displaying the probability distribution!)

Outline

Probabilities and Bayesian inference

Variational Inference and the Free-energy principle

Active inference, EMs & oculomotor delays

Variational Inference and the Free-energy principle

Friston (2010) Nat Neuro Reviews

Variational Inference and the Free-energy principle

Bogacz (2017) Journal of Mathematical Psychology

I enjoyed reading "A tutorial on the free-energy framework for modelling perception and learning" by Rafal Bogacz, which is freely available here. In particular, the author encourages to replicate the results in the paper. He is himself giving solutions in matlab, so I had to do the same in python all within a notebook...

      A tutorial on the free-energy framework for modelling perception and learning
    http://dx.doi.org/10.1016/j.jmp.2015.11.003

online
complex priors

The architecture of the model performing simple perceptual inference. Circles denote neural “nodes”, arrows denote excitatory connections, while lines ended with circles denote inhibitory connections. Labels above the connections encode their strength, and lack of label indicates the strength of 1. Rectangles indicate the values that need to be transmitted via the connections they label.

Variational Inference and the Free-energy principle

Lets try this using this notebook (solution)

First, let's see the application of Bayes theorem We start by considering in this section a simple perceptual problem in which a value of a single variable has to be inferred from a single observation. To make it more concrete, consider a simple organism that tries to infer the size or diameter of a food item, which we denote by $v$, on the basis of light intensity it observes. Let us assume that our simple animal has only one light sensitive receptor which provides it with a noisy estimate of light intensity, which we denote by $u$. Let g denote a non-linear function relating the average light intensity with the size. Since the amount of light reflected is related to the area of an object, in this example we will consider a simple function of $g(v)=v^2$. Let us further assume that the sensory input is noisy—in particular, when the size of food item is v, the perceived light intensity is normally distributed with mean g(v) , and variance $Σ_u$ (although a normal distribution is not the best choice for a distribution of light intensity, as it includes negative numbers, we will still use it for a simplicity): $$ p(u|v)=f(u;g(v),Σu). $$ In $f(x;μ,Σ) $ denotes the density of a normal distribution with mean μ and variance Σ

Due to the noise present in the observed light intensity, the animal can refine its guess for the size v by combining the sensory stimulus with the prior knowledge on how large the food items usually are, that it had learnt from experience. For simplicity, let us assume that our animal expects this size to be normally distributed with mean $v_p$ and variance $Σ_p$ (subscript p stands for “prior”), which we can write as: $$ p(v)=f(v;vp,Σp). $$

Let us now assume that our animal observed a particular value of light intensity, and attempts to estimate the size of the food item on the basis of this observation. We will first consider an exact solution to this problem, and illustrate why it would be difficult to compute it in a simple neural circuit. Then we will present an approximate solution that can be easily implemented in a simple network of neurons.

Exact solution

To compute how likely different sizes $v$ are given the observed sensory input $u$, we could use Bayes’ theorem: $$ p(v|u)=p(v)p(u|v)p(u). $$

Term $p(u)$ in the denominator of equation is a normalization term, which ensures that the posterior probabilities of all sizes $p(v|u)$ integrate to 1:

$$ p(u)=∫p(v)p(u|v)dv. $$

The integral in the above equation sums over the whole range of possible values of $v$, so it is a definite integral, but for brevity of notation we do not state the limits of integration in this and all other integrals in the paper.

Now combining Eqs. we can compute numerically how likely different sizes are given the sensory observation. For readers who are not familiar with such Bayesian inference we recommend doing the following exercise now.

Exercise 1.

Assume that our animal observed the light intensity $u=2$, the level of noise in its receptor is $Σ_u=1$, and the mean and variance of its prior expectation of size are $v_p=3$ and $Σ_p=1$. Write a computer program that computes the posterior probabilities of sizes from 0.01 to 5, and plots them.

Variational Inference and the Free-energy principle

Lets try this using this notebook (solution)

Finding the most likely feature value

Instead of finding the whole posterior distribution p(v|u) , let us try to find the most likely size of the food item v which maximizes p(v|u) . We will denote this most likely size by ϕ , and its posterior probability density by p(ϕ∣∣u) . It is reasonable to assume that in many cases the brain represents at a given moment of time only most likely values of features. For example in case of binocular rivalry, only one of the two possible interpretations of sensory inputs is represented.

We will look for the value ϕ which maximizes p(ϕ∣∣u) . According to Eq. (4), the posterior probability p(ϕ∣∣u) depends on a ratio of two quantities, but the denominator p(u) does not depend on ϕ . Thus the value of ϕ which maximizes p(ϕ∣∣u) is the same one which maximizes the numerator of Eq. (4). We will denote the logarithm of the numerator by F , as it is related to the negative of free energy (as we will describe in Section 3):

$$ F=lnp(ϕ)+lnp(u∣∣ϕ). $$ the definition of this model, one may derive a learning rule to estimate the best $u$

Let's define $F = \log( p(v |u) )$ after some derivation, we get

$$ rac{\partial F}{\partial v} = rac{v - v_p}{\Sigma_p} + \dot{g}(v) \cdot rac{u - g(v)}{\Sigma_u} $$

In the above equation we used the property of logarithm $ln(ab)=lna+lnb$. We will maximize the logarithm of the numerator of Eq. (4), because it has the same maximum as the numerator itself as ln is a monotonic function, and is easier to compute as the expressions for p(u∣∣ϕ) and p(ϕ) involve exponentiation.

To find the parameter ϕ that describes the most likely size of the food item, we will use a simple gradient ascent: i.e. we will modify ϕ proportionally to the gradient of F , which will turn out to be a very simple operation. It is relatively straightforward to compute F by substituting Eqs. (1), (2) ; (3) into Eq. (6) and then to compute the derivative of F (TRY IT YOURSELF).

equation(7) F=lnf(ϕ;vp,Σp)+lnf(u;g(ϕ),Σu)=ln[12πΣp√exp(−(ϕ−vp)22Σp)]+ln[12πΣu√exp(−(u−g(ϕ))22Σu)]=ln12π√−12lnΣp−(ϕ−vp)22Σp+ln12π√−12lnΣu−(u−g(ϕ))22Σu=12(−lnΣp−(ϕ−vp)2Σp−lnΣu−(u−g(ϕ))2Σu)+C. Turn MathJax off

We incorporated the constant terms in the 2nd line above into a constant C . Now we can compute the derivative of F over ϕ :

equation(8) ∂F∂ϕ=vp−ϕΣp+u−g(ϕ)Σug′(ϕ). Turn MathJax off

In the above equation we used the chain rule to compute the second term, and g′(ϕ) is a derivative of function g evaluated at ϕ , so in our example g′(ϕ)=2ϕ . We can find our best guess ϕ for v simply by changing ϕ in proportion to the gradient:

equation(9) ϕ̇ =∂F∂ϕ. Turn MathJax off

In the above equation ϕ̇ is the rate of change of ϕ with time. Let us note that the update of ϕ is very intuitive. It is driven by two terms in Eq. (8): the first moves it towards the mean of the prior, the second moves it according to the sensory stimulus, and both terms are weighted by the reliabilities of prior and sensory input respectively.

Now please note that the above procedure for finding the approximate distribution of distance to food item is computationally much simpler than the exact method presented at the start of the paper. To gain more appreciation for the simplicity of this computation we recommend doing the following exercise.

Exercise 2.

Write a computer program finding the most likely size of the food item ϕ for the situation described in Exercise 1. Initialize ϕ=vp , and then find its values in the next 5 time units (you can use Euler’s method, i.e. update $ϕ(t+Δt)=ϕ(t)+Δt∂F/∂ϕ$ with t=0.01$ ).

Fig. 2(a) shows a solution to Exercise 2. Please notice that it rapidly converges to the value of $ϕ≈1.6 $, which is also the value that maximizes the exact posterior probability $p(v|u)$ shown in Fig. 1.

Variational Inference and the Free-energy principle

Lets try this using this notebook (solution)

one may also learn the variance

The model parameters can be hence optimized by modifying them proportionally to the gradient of $F$. It is straightforward to find the derivatives of F over $v_p$, $Σ_p$ and $Σ_u$:

$$ rac{∂F}{∂v_p}=rac{ϕ−v_p}{Σ_p} $$ $$ rac{∂F}{∂Σp}=rac 1 2 ( rac {(ϕ−v_p)^2} {Σ^2_p}−rac{1}{Σ_p} ) $$ $$ rac{∂F}{∂Σu}=rac 1 2 ( rac {(u−g(ϕ))^2}{Σ^2_u}−rac{1}{Σ_u}. $$

Simulate learning of variance $Σ_i$ over trials. For simplicity, only simulate the network described by Eqs. (59)– (60), and assume that variables ϕ are constant. On each trial generate input $ϕ_i$ from a normal distribution with mean 5 and variance 2, while set $g_i(ϕ_i+1)=5$ (so that the upper level correctly predicts the mean of $ϕ_i$). Simulate the network for 20 time units, and then update weight $Σ_i$ with learning rate $α=0.01$. Simulate 1000 trials and plot how $Σ_i$ changes across trials.

Outline

Probabilities and Bayesian inference

Variational Inference and the Free-energy principle

Active inference, EMs & oculomotor delays

Great; but now, what if I want to learn the right parameters instead of scanning all of them? A solution lies in the Free-energy princple from Karl Friston, and I will give here an introduction to this technique

in a first study, we have proposed that PERCEPTION (following Helmhotz 1866) is an active process of hypothesis testing by which we seek to confirm our predictive models of the (hidden) world: Active inference: (cite TITLE). In theory, one could find any prior to fit any experimental data, but the beauty of the theory comes from the simplicity of the models chosen to model the data at hand, such as saccades...
... and even better if these models may find a possible correspondance into the neural anatomy and explain some deviation to a control behaviour, such as that we modelled for understanding some aspects of the EMs of schizophrenic patients
Today, I will again focus on the problem of sensorimotor delays in the optimal control of (smooth) eye movements under uncertainty. Specifically, we consider delays in the visuo-oculomotor loop and their implications for active inference and I will present the results presented in the following paper (show TITLE).

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment. It shows the states of the environment and the system in terms of a probabilistic dependency graph, where connections denote directed (causal) dependencies. The quantities are described within the nodes of this graph -- with exemplar forms for their dependencies on other variables.
Hidden (external) and internal states of the agent are separated by action and sensory states. Both action and internal states -- encoding a conditional probability density function over hidden states -- minimise free energy. Note that hidden states in the real world and the form of their dynamics can be different from that assumed by the generative model; (this is why hidden states are in bold. )

as a summary, we have shown that, evolving from other theories of prediction of post-diction, we have shown that perceptual results are well modeled by a simple theory. This theory states that the agent models (predicts) the scene at the present time (in spite of inherent but known delays) - and we called it parodiction from the greek "paros" the present time.
More generally, this novel theory proposes that generic neural computations systematically compensate neural delays to represent the predicted visual scene at the present time, though it is not yet directly experienced. a consequence of this view -that is shared among a large number of physicists- is that time as a continous line is an emerging property of the representation used by these agents and does not need to exist physically (though this rather bold statement is quite infalsifiable) - the world is latent - there is only the arrow of time (= laws of nature)
There is a fundamental limit to this theory: the agent has an uniform access to the whole field of view. With a foveal point, it may fail to track - thus we need action as an eye movement ; but this is another project :-)