Detecting switches in a volatile environment

Should I stay or should I go? Adaption of human observers to the volatility of visual inputs

Laurent Perrinet

CausaL Kick-off, 5/4/2019

Acknowledgements:

Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Jean-Bernard Damasse, Laurent Madelain and Anna Montagnini - ANR REM
Frédéric Chavane - INT

This work was supported by ANR project ANR-18-AAPG–“CAUSAL, Cognitive Architectures of Causal Learning”.

(AUTHOR) Hello, I am Laurent Perrinet from the Institute of Neurosciences of la Timone in Marseille, a joint unit from the CNRS and the AMU
(OBJECTIVE)
Let's me first describe the motivation of this work...

Today I will present some recent results on detecting switches in a sequence and potential role for RL and causality measures

In particular, I will try to give some principles of active inference, framed in a practical example of a dynamic, poissbly volatile environment and how we may use it to anticipate ---- and my main goal in general is to illustrate how this theory may give a creative and efficient tool to do psychophysics.

https://laurentperrinet.github.io/talk/2016-10-13-law/ https://laurentperrinet.github.io/talk/2018-02-01-bcp-invibe-fest/ https://laurentperrinet.github.io/talk/2018-04-05-bcp-talk/ https://laurentperrinet.github.io/talk/2019-01-18-laconeu/

this talk= https://laurentperrinet.github.io/talk/2019-04-05-bbcp-causal-kickoff/

(SHOW TITLE)

Should I stay or should I go? - Eye Movements

Montagnini A, Souto D, and Masson GS (2010) J Vis (VSS Abstracts) 10(7):554,
Montagnini A, Perrinet L, and Masson GS (2015) BICV book chapter

Should I stay or should I go? - Eye Movements

Should I stay or should I go? - Random-length block design

Outline

Should I stay or should I go?

Experimental protocol

The Bayesian Changepoint Detector

Results using the BCP

Application to RL?

Experimental protocol - Random-length block design

full code @ github.com/chloepasturel/AnticipatorySPEM

Experimental protocol - Random-length block design

Experimental protocol

full code @ github.com/chloepasturel/AnticipatorySPEM

First, we overlay the results of the bet result for one of the 12 subjects

We rescaled th value given by the observer so that it fits to 0 (sure it goes left) to 1 (sure it goes right)

We observe a pretty good fit of this trace as a function of trial number

In particular, we see 2 main effects: - results are more variable when the bias is around .5 than when it is high (close to zero or one) - switches were detected quite rapidly but with a certain delay of a few trials. Indeed, note that this plot shows the entire sequence but that observers had only access to some internal representation of the memory of the previous observations. When faced with some new observations, the observer has to constantly adapt his response to either exploit this information by considering that this observation belongs to the same sub-block or to explore a novel hypothesis. This compromise is one of the crucial component that we wish to explore.

Let's now have a look at EMs...

Experimental protocol - Fitting eye movements

full code @ github.com/chloepasturel/AnticipatorySPEM

I show here a typical velocity traces for one subject / 2 trials

x-axis is time in milliseconds aligned on target onset, and we show respectively from left to right the fixation in gray, the GAP in pink (300ms) and the run in light gray.
y-axis is the velocity as computed as the gradient of position. Remark that the eyelink provides with the periods of saccades or blinks that we removed from the signal. it is quite noisy and to complement existing signal processing methods, Chloe implemented a robust
fitting method which allows to extract some key components of the velocity traces: maximum speed, latency, temporal inertia ($ au$) and most interestingly acceleration before motion onset. We cross-validated that this method was givinfg similar results to other classical methods but in a more robust fashion/

While being sensible to recording errors, this allows us to extract the anticipatory component of SPEMs and..

Experimental protocol - Fitting eye movements

full code @ github.com/chloepasturel/AnticipatorySPEM

I show here a typical velocity traces for one subject / 2 trials

x-axis is time in milliseconds aligned on target onset, and we show respectively from left to right the fixation in gray, the GAP in pink (300ms) and the run in light gray.
y-axis is the velocity as computed as the gradient of position. Remark that the eyelink provides with the periods of saccades or blinks that we removed from the signal. it is quite noisy and to complement existing signal processing methods, Chloe implemented a robust
fitting method which allows to extract some key components of the velocity traces: maximum speed, latency, temporal inertia ($ au$) and most interestingly acceleration before motion onset. We cross-validated that this method was givinfg similar results to other classical methods but in a more robust fashion/

While being sensible to recording errors, this allows us to extract the anticipatory component of SPEMs and..

Experimental protocol

full code @ github.com/chloepasturel/AnticipatorySPEM

Experimental protocol

full code @ github.com/chloepasturel/AnticipatorySPEM

Experimental protocol

full code @ github.com/chloepasturel/AnticipatorySPEM

quantitatively, one can now plot the results for all subjects
the x-axis corresponds to the probability that was coded at the second layer and which is unknown to the observer
the y -axis shows either the bet or the
dots represent single responses - the saturation giving the identity of the observer

we notice a quite nice linear correlation (black line) for both experiments; of the order of that found in the classical experiment with fixed blocks and a vartiety of bias values. This is surprising as the blocks are of random length, observer can still adapt to such a volatile environment.

Another visualization, the scatter plot of acceleration as a function of probability bet shows also that there is a correlation between both variables.

This allows to make a first point: it is possible to use more genreal models such as hierarchical generative models.

However, while this results seem encouraging, a more finer analysis may be necessary.

Experimental protocol

full code @ github.com/chloepasturel/AnticipatorySPEM

quantitatively, one can now plot the results for all subjects
the x-axis corresponds to the probability that was coded at the second layer and which is unknown to the observer
the y -axis shows either the bet or the
dots represent single responses - the saturation giving the identity of the observer

Another visualization, the scatter plot of acceleration as a function of probability bet shows also that there is a correlation between both variables.

This allows to make a first point: it is possible to use more genreal models such as hierarchical generative models.

However, while this results seem encouraging, a more finer analysis may be necessary.

Outline

Should I stay or should I go?

Experimental protocol

The Bayesian Changepoint Detector

Results using the BCP

Application to RL?

The Bayesian Changepoint Detector - Random-length block design

full code @ github.com/chloepasturel/AnticipatorySPEM

Let's remember our hierarchical generative model.

At any given trial, we wish to construct an algorithm which

We will introduce a fundamental component of Bayesian models : a latent variable

this new variable will be used to test different hypothesis which will be evaluated to predict future states. it is called latent because it aims at representing a variable that is latent (or hidden) to the observer

in our case, we will assume that the bayesian model knows about the structure of the generative model and we will set it to the current run-length $r$, that is, at any given trial, the hypothesis that the past r observations belong to the same block. of course a wrong choice of a latent variables (let's say the temperture in the experimental room) may give unexpected results, even is the bayesian model is "optimal" - an essential point to understand in bayesian inference

The Bayesian Changepoint Detector

Initialize $P(r_0=0)=1$ and $ν^{(0)}_1 = ν_{prior}$ and $χ^{(0)}_1 = χ_{prior}$

The Bayesian Changepoint Detector

Observe New Datum $x_t$ and Perform Prediction $P (x_{t+1} | x_{1:t}) = P (x_{t+1}|x_{1:t} , r_t) \cdot P (r_t|x_{1:t})$

The Bayesian Changepoint Detector

Evaluate (likelihood) Predictive Probability $π_{1:t} = P(x_t |ν^{(r)}_t,χ^{(r)}_t)$
Calculate Growth Probabilities $P(r_t=r_{t-1}+1, x_{1:t}) = P(r_{t-1}, x_{1:t-1}) \cdot π^{(r)}_t \cdot (1−h))$
Calculate Changepoint Probabilities $P(r_t=0, x_{1:t})= \sum_{r_{t-1}} P(r_{t-1}, x_{1:t-1}) \cdot π^{(r)}_t \cdot h$

The Bayesian Changepoint Detector

Calculate Evidence $P(x_{1:t}) = \sum_{r_{t-1}} P (r_t, x_{1:t})$
Determine Run Length Distribution $P (r_t | x_{1:t}) = P (r_t, x_{1:t})/P (x_{1:t}) $

The Bayesian Changepoint Detector

Update Sufficient Statistics :
$ν^{(r+1)}_{t+1} = ν^{(r)}_{t} +1$, $χ^{(r+1)}_{t+1} = χ^{(r)}_{t} + u(x_t)$
$ν^{(0)}_{t+1} = ν_{prior}$, $χ^{(0)}_{t+1} = χ_{prior}$

Bayesian Changepoint Detector

Initialize

$P(r_0=0)=1$ and
$ν^{(0)}_1 = ν_{prior}$ and $χ^{(0)}_1 = χ_{prior}$

Observe New Datum $x_t$
Evaluate Predictive Probability $π_{1:t} = P(x_t |ν^{(r)}_t,χ^{(r)}_t)$
Calculate Growth Probabilities $P(r_t=r_{t-1}+1, x_{1:t}) = P(r_{t-1}, x_{1:t-1}) \cdot π^{(r)}_t \cdot (1−H(r^{(r)}_{t-1}))$
Calculate Changepoint Probabilities $P(r_t=0, x_{1:t})= \sum_{r_{t-1}} P(r_{t-1}, x_{1:t-1}) \cdot π^{(r)}_t \cdot H(r^{(r)}_{t-1})$
Calculate Evidence $P(x_{1:t}) = \sum_{r_{t-1}} P (r_t, x_{1:t})$
Determine Run Length Distribution $P (r_t | x_{1:t}) = P (r_t, x_{1:t})/P (x_{1:t}) $
Update Sufficient Statistics :

$ν^{(0)}_{t+1} = ν_{prior}$, $χ^{(0)}_{t+1} = χ_{prior}$
$ν^{(r+1)}_{t+1} = ν^{(r)}_{t} +1$, $χ^{(r+1)}_{t+1} = χ^{(r)}_{t} + u(x_t)$

Perform Prediction $P (x_{t+1} | x_{1:t}) = P (x_{t+1}|x_{1:t} , r_t) \cdot P (r_t|x_{1:t})$
go to (2)

The Bayesian Changepoint Detector

full code @ github.com/laurentperrinet/bayesianchangepoint

The Bayesian Changepoint Detector - Full model

full code @ github.com/laurentperrinet/bayesianchangepoint

Let's now see the application of our model to a simple synthetic example before applying it to the experimental protocol that we used in our two experiments

we show two panels, one below which displays the value of the belief for the different run-length, and one above where we will show the resaulting prediction of the next outcome. we obtain for any given sequence different values at the given trial in the form of columns for any possible run-length: the belief, and the sufficient statistics for the beta distirbution which allow to provide with an estimate of the current probability
first, we show the value of probability, low probabilities are blueish while high probabilities. at every trial, the agent evluates the value for the different possible run lengths, generating a column. by showing all columns we generate this image which shows the evaluation along the sequence of trials.
second we show above the sequence of observations that were shown to the agent in a light black line. the read line gives an evaluation of the most probable a posteriori probability as the probability to hte run-length the maximum a posteriori belief on the differettn beliefs about run-lengths. using the estimate of the precision at this

We remark two main observations:

first, beliefs grow at the beginning along a linear ridge, as we begin our model by assuming there was a switch at time 0. Then we observe that at a switch (hidden to the model), the model such that belief is more stronlgly diffused until the probability

-second, we may use this information to read-out the information the most probable probability and the confidence interval as shown by the red dashed lines (.05, and .95)

This is in contrast with a fixed length model, for which - the delay will always be similar - there is no dynamic upDATE OF THE INFERRD probability

as a summary, for any given sequnce, we get an estimate of the probability given by the ideal observer. we will now see how we can apply that to our experiemntal protocol.

The Bayesian Changepoint Detector - Fixed window

full code @ github.com/laurentperrinet/bayesianchangepoint

Let's now see the application of our model to a simple synthetic example before applying it to the experimental protocol that we used in our two experiments

we show two panels, one below which displays the value of the belief for the different run-length, and one above where we will show the resaulting prediction of the next outcome. we obtain for any given sequence different values at the given trial in the form of columns for any possible run-length: the belief, and the sufficient statistics for the beta distirbution which allow to provide with an estimate of the current probability
first, we show the value of probability, low probabilities are blueish while high probabilities. at every trial, the agent evluates the value for the different possible run lengths, generating a column. by showing all columns we generate this image which shows the evaluation along the sequence of trials.
second we show above the sequence of observations that were shown to the agent in a light black line. the read line gives an evaluation of the most probable a posteriori probability as the probability to hte run-length the maximum a posteriori belief on the differettn beliefs about run-lengths. using the estimate of the precision at this

We remark two main observations:

first, beliefs grow at the beginning along a linear ridge, as we begin our model by assuming there was a switch at time 0. Then we observe that at a switch (hidden to the model), the model such that belief is more stronlgly diffused until the probability

-second, we may use this information to read-out the information the most probable probability and the confidence interval as shown by the red dashed lines (.05, and .95)

This is in contrast with a fixed length model, for which - the delay will always be similar - there is no dynamic upDATE OF THE INFERRD probability

as a summary, for any given sequnce, we get an estimate of the probability given by the ideal observer. we will now see how we can apply that to our experiemntal protocol.

Outline

Should I stay or should I go?

Experimental protocol

The Bayesian Changepoint Detector

Results using the BCP

Application to RL?

Results using the BCP - Full model

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - Full model

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - Full model

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - Fixed window

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - Fixed window

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - Fixed window

full code @ github.com/laurentperrinet/bayesianchangepoint

       Let's use our model on the different sequences that were generated in our experiments in the different blocks.

we the same arrangement of panels, we show below the dynamical evolution of beliefs and above the resulting readout from the model

we see that as in the synthetic example above, there is a correct detection of switch after a short delay of a few trials

in particular, from this correct detection, the value of the inferred probability approaches the true one as the number of observations increase in one subblock.

again, we see that a fixed length model gives a similar output but with the two disadvantages described above

Let's now see how this applies to our experimental results by comparing human observers to our bayesian agent.

Results using the BCP - fit with BCP

Results using the BCP

full code @ github.com/laurentperrinet/bayesianchangepoint

Results using the BCP - Fixed window

Results using the BCP - Full model

Results using the BCPinterindividual differences

Outline

Should I stay or should I go?

Experimental protocol

The Bayesian Changepoint Detector

Results using the BCP

Application to RL?

Application to RL? - Full model

full code @ github.com/laurentperrinet/bayesianchangepoint

Application to RL? - With hindsight

full code @ github.com/laurentperrinet/bayesianchangepoint

Should I stay or should I go? Adaption of human observers to the volatility of visual inputs

Laurent Perrinet

CausaL Kick-off, 5/4/2019

Acknowledgements:

Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Jean-Bernard Damasse, Laurent Madelain and Anna Montagnini - ANR REM
Frédéric Chavane - INT

This work was supported by ANR project ANR-18-AAPG–“CAUSAL, Cognitive Architectures of Causal Learning”.

https://laurentperrinet.github.io/talk/2019-04-05-bbcp-causal-kickoff