Modelling the dynamics of cognitive processes

Modelling the dynamics of cognitive processes: from the Bayesian brain to particles

Laurent U Perrinet, INT

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane
Pascal Wallisch and Tony Movshon @ NYU for the MT data
Rick Adams and Karl Friston @ UCL
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini for the psychophysics

CIRM, July 7th, 2016

* (AUTHOR) Hello, I am Laurent Perrinet from the Institute of Neurosciences of la Timone in Marseille, a joint unit from the CNRS and the AMU
* (SHOW TITLE) I am interested in the link between the neural code and the structure of the world. in particular, for vision, I am researching the relation between the functional (in terms of the inferential processes leading to  behaviour) organization (anatomy and activity) of low-level visual areas (V1) and the structures of natural scenes, that is of the images that hit the retina and which are
relevant to visual perception in general.
* (OBJECTIVE) in this  talk, I will be focus in highlighting some key challenges in modelizing visual perception and in particular focus on the role of EDP and probabilities in modelling such processes
* (ACKNO) this endeavour involves different techniques and tools ... From the head on, I wish to thanks people who collaborated  and in particular ..   mostly funded by the ANR bala V1
+ LONDON (Jim Bednar, Friston)


so what is visual perception?

Felice Varini (2012) "vingt-trois disques évidés plus douze moitiés et quatre quarts" Exposition:"DYNAMO" Grand Palais, Paris 2013 Photo: André Morin

it is the set of cognitive processes that allow to make sense from the visual inputs

- illusions visuelles:  Felice Varini "vingt-trois disques évidés plus douze moitiés et quatre quarts" Exposition:
"DYNAMO" Grand Palais, Paris 2013 Photo: André Morin

- power.png = efficacité

- eye movements https://www.youtube.com/watch?v=w0xig8ImwDY

Felice Varini (2012) "vingt-trois disques évidés plus douze moitiés et quatre quarts" Exposition:"DYNAMO" Grand Palais, Paris 2013 Photo: André Morin

it is the set of cognitive processes that allow to make sense from the visual inputs

- illusions visuelles:  Felice Varini "vingt-trois disques évidés plus douze moitiés et quatre quarts" Exposition:
"DYNAMO" Grand Palais, Paris 2013 Photo: André Morin

- power.png = efficacité

- eye movements https://www.youtube.com/watch?v=w0xig8ImwDY

Outline

Eye movements and the Bayesian brain: the aperture problem

Eye movements and particles: solving the aperture problem

Active inference, eye movements & oculomotor delays

From probabilities to particles: Flash-lag effect

Eye movements and the Bayesian brain: the aperture problem

Weiss et al (2002) Nature Neuroscience

Standard Models of Motion detection are the well-known motion energy and Reichhardt models: these local motion detectors take the sequence as an input and give as an output different ``scores'' for a set of velocities.

(proba) in fact these models are equivalent at the functional level to a probabilistic measure. so if it's equivalent, why use probas? first, computation of a likelihood explicitly defines the generative models underlying the definition of motion and thus gives all underlying assumptions of a given model. second, we will see that probabilities are well adapted to manipulate bits of information,
(weiss) I show right the architecture of the model by Weiss02: top, the stimulus (here a rhombus), and in the middle what knowledge as represented in motion space (referenced here by the axis of horizontal and vertical translational velocities). we may extract internal knowledge about the motion in a receptive field and represent it by a subset in motion space, represented here by the "white" colours. In the case of the segment, most information is distributed along the line corresponding to the possible velocities. In the case of a dot or of an edge, motion is unambiguous, information lies on a small concentrated area of motion space.
(merge) information is then assumed to be conditionally independent and probabilities are multiplied to give a global readout. finally, the introduction of a bayesian prior was very successful in describing many behavioural and physiological data ... % however, the question is open concerning as how we may select and merge these measures

Eye movements and the Bayesian brain: the aperture problem

LP and Masson (2007) Journal of Physiology (Paris)

Eye movements and Bayes: motion integration

Outline

Eye movements and the Bayesian brain: the aperture problem

Eye movements and particles: solving the aperture problem

Active inference, eye movements & oculomotor delays

From probabilities to particles: Flash-lag effect

Eye movements and particles: solving the aperture problem

This is what I illustrate here for the slanted line: knowing the information collected on the shaded area, motion may correspond to different translational velocities (red arrows), e.~g.~perpendicular to the line or horizontal such as the true physical motion (blue arrow) but also to all other velocities on the constraint line (dashed line) Detection of motion, as a one-to-many mapping of sensible space (retina) to motion space, is thus an ill-posed problem. Note that the same problems faced by perception are found in other mathematical problems such as optical flow estimation in computer vision. that's certainly the reason why there is such great convergence between both fields. How do we solve it? neurophysiological and behavioural data suggests ...
(pack01) ... that the solution to the aperture problem seems to be dynamical. This data was recorded by Chris Pack and Rick Born in neurones from area MT in macaques: (2.a) the early response of neurones is preferentially tuned to the perpendicular to the line, (2.b) while it shifts to the physical direction (2.c) after approx 60 ms.
(OFR) this is also demonstrated at different levels of the system, and in particular for eye movements. (3) This is data recorded at their lab showing eye tracking to the slanted line a showing the same initial bias as with the neural data. These behavioural results replicate those obtained previously in humans by Guillaume Masson at our lab and show the close causality link from neural to behavioural response.
OBSOLETE (Pack03) as was shown physiologically or behaviorally, the transient response to line or line-ending detector detectors was similar, a difference emerged in their dynamics after few milliseconds. This observations suggest that these different cells have a common excitatory drive but that they integrate contextual information from neighboring neurons differently. It therefore suggests that the response to line-endings is not caused by specialized receptors but rather the consequence of a dynamical integration for which motion-based prediction gives a principled approach. (Figure 7 from Pack, 2003) Timing of V1 End-Stopping (A) The first row shows successive snapshots of the one-bar map for a strongly end-stopped V1 cell, for the preferred bar orientation. The cell initially responds well to a bar placed anywhere in its receptive field, but after $20-30 ms$, it responds only when the endpoints are in the receptive field. The second row shows the time course of the response when the analysis was limited to specific regions of the stimulus range. When the analysis was limited to instances when the bar was centered on the receptive field (green square), the neuron responded transiently (green line). When the analysis was limited to instances when the endpoint of the bar appeared in the receptive field (black square), the cell exhibited a strong maintained discharge (black line). (B) The time course of the response, averaged across the population of 29 neurons for the preferred bar orientation. Dotted lines indicate standard error of the mean.

Eye movements and particles: solving the aperture problem

Pack and Born (2001) Nature

This is what I illustrate here for the slanted line: knowing the information collected on the shaded area, motion may correspond to different translational velocities (red arrows), e.~g.~perpendicular to the line or horizontal such as the true physical motion (blue arrow) but also to all other velocities on the constraint line (dashed line) Detection of motion, as a one-to-many mapping of sensible space (retina) to motion space, is thus an ill-posed problem. Note that the same problems faced by perception are found in other mathematical problems such as optical flow estimation in computer vision. that's certainly the reason why there is such great convergence between both fields. How do we solve it? neurophysiological and behavioural data suggests ...
(pack01) ... that the solution to the aperture problem seems to be dynamical. This data was recorded by Chris Pack and Rick Born in neurones from area MT in macaques: (2.a) the early response of neurones is preferentially tuned to the perpendicular to the line, (2.b) while it shifts to the physical direction (2.c) after approx 60 ms.
(OFR) this is also demonstrated at different levels of the system, and in particular for eye movements. (3) This is data recorded at their lab showing eye tracking to the slanted line a showing the same initial bias as with the neural data. These behavioural results replicate those obtained previously in humans by Guillaume Masson at our lab and show the close causality link from neural to behavioural response.
OBSOLETE (Pack03) as was shown physiologically or behaviorally, the transient response to line or line-ending detector detectors was similar, a difference emerged in their dynamics after few milliseconds. This observations suggest that these different cells have a common excitatory drive but that they integrate contextual information from neighboring neurons differently. It therefore suggests that the response to line-endings is not caused by specialized receptors but rather the consequence of a dynamical integration for which motion-based prediction gives a principled approach. (Figure 7 from Pack, 2003) Timing of V1 End-Stopping (A) The first row shows successive snapshots of the one-bar map for a strongly end-stopped V1 cell, for the preferred bar orientation. The cell initially responds well to a bar placed anywhere in its receptive field, but after $20-30 ms$, it responds only when the endpoints are in the receptive field. The second row shows the time course of the response when the analysis was limited to specific regions of the stimulus range. When the analysis was limited to instances when the bar was centered on the receptive field (green square), the neuron responded transiently (green line). When the analysis was limited to instances when the endpoint of the bar appeared in the receptive field (black square), the cell exhibited a strong maintained discharge (black line). (B) The time course of the response, averaged across the population of 29 neurons for the preferred bar orientation. Dotted lines indicate standard error of the mean.

Eye movements and particles: solving the aperture problem

Pack and Born (2001) Nature

This is what I illustrate here for the slanted line: knowing the information collected on the shaded area, motion may correspond to different translational velocities (red arrows), e.~g.~perpendicular to the line or horizontal such as the true physical motion (blue arrow) but also to all other velocities on the constraint line (dashed line) Detection of motion, as a one-to-many mapping of sensible space (retina) to motion space, is thus an ill-posed problem. Note that the same problems faced by perception are found in other mathematical problems such as optical flow estimation in computer vision. that's certainly the reason why there is such great convergence between both fields. How do we solve it? neurophysiological and behavioural data suggests ...
(pack01) ... that the solution to the aperture problem seems to be dynamical. This data was recorded by Chris Pack and Rick Born in neurones from area MT in macaques: (2.a) the early response of neurones is preferentially tuned to the perpendicular to the line, (2.b) while it shifts to the physical direction (2.c) after approx 60 ms.
(OFR) this is also demonstrated at different levels of the system, and in particular for eye movements. (3) This is data recorded at their lab showing eye tracking to the slanted line a showing the same initial bias as with the neural data. These behavioural results replicate those obtained previously in humans by Guillaume Masson at our lab and show the close causality link from neural to behavioural response.
OBSOLETE (Pack03) as was shown physiologically or behaviorally, the transient response to line or line-ending detector detectors was similar, a difference emerged in their dynamics after few milliseconds. This observations suggest that these different cells have a common excitatory drive but that they integrate contextual information from neighboring neurons differently. It therefore suggests that the response to line-endings is not caused by specialized receptors but rather the consequence of a dynamical integration for which motion-based prediction gives a principled approach. (Figure 7 from Pack, 2003) Timing of V1 End-Stopping (A) The first row shows successive snapshots of the one-bar map for a strongly end-stopped V1 cell, for the preferred bar orientation. The cell initially responds well to a bar placed anywhere in its receptive field, but after $20-30 ms$, it responds only when the endpoints are in the receptive field. The second row shows the time course of the response when the analysis was limited to specific regions of the stimulus range. When the analysis was limited to instances when the bar was centered on the receptive field (green square), the neuron responded transiently (green line). When the analysis was limited to instances when the endpoint of the bar appeared in the receptive field (black square), the cell exhibited a strong maintained discharge (black line). (B) The time course of the response, averaged across the population of 29 neurons for the preferred bar orientation. Dotted lines indicate standard error of the mean.

Motion-based predictive coding

LP and Masson (2012) Neural Computation, https://laurentperrinet.github.io/publication/perrinet-12-pred

We will now define a probabilistic framework for the representation of predictive motion information.

(prediction) First, we will define motion-based prediction which is adapted to smooth trajectories such as are observed in natural environments
(markov) Seeing this prediction as a prior on motion transition in the probabilistic domain, we then show how information may be pooled using a Hidden Markov Model, merging current information and measurement likelihood
(SMC) Differently to neural approximations that were used previously, we use a particle filtering method to implement this functional model. Prediction will now take the form of an anisotropic, context-dependent diffusion of local information.

But first, how do we define prediction?
In computer vision, one can define the prediction of the trajectory of some distinguished feature of the image (e.g., a local extremum of luminance) using its motion. here we define the feature to track as the motion at a given position and define Motion-based prediction by modelling smoothness in the trajectory of motion:
knowing the velocity $V_{t-dt}$ at one position $x_{t-dt}$, one expects that velocity will be transported to a new location, that is $x_{t} pprox x_{t-dt} + V_{t-dt} dt$ and approximately conserved in amplitude and direction $V_t pprox V_{t-dt} $. Nothing new: if we go to infinitesimal $dt$, this equation is the advection term in the navier-stokes equations.
This "approximately" may be included in our generative model by some random variable. different values for the variance of this state noise quantify the strength or precision of the prediction... it shapes the association fields in motion space similarly with what happens in the space domain with association field in the orientation domain of static images.
Note that contrary to classical models, position becomes an unknown variable instead of being a parameter as in OF. note also that it is auto-referent since prediction is based itself on motion %: neural = Series / models Grzywack, Burgi et al = "motion tracking"
then, we may use a classical Hidden Markov Model to recurrently update knowledge. This uses three steps knowing the probabilistic motion information at a current time and for simplicity I will just represent local probabilistic motion representations:
prediction of motion thanks to the generative model as a prior of motion transition and an integral over all possible motions. note that prediction is always diffusive
estimation of the likelihood of the predicted state,
merging both, we updated our knowledge from time $ t-dt$ to $ t$
repeating the operation, we define a dynamical system.
this theory seems easy on the paper, and there is nothing new here in the definition of the dynamical system . extit{BUT} ...... I have bad news: it is extremely hard to represent \& implement on classical machines: if we allow ourself a rather modest discretisation if the probability space (position and velocities), the very high dimensionality of the problem makes it tedious to represent ($2e9$ dimensions in the previously shown POF)...

Motion-based predictive coding

LP and Masson (2012) Neural Computation, https://laurentperrinet.github.io/publication/perrinet-12-pred

We will now define a probabilistic framework for the representation of predictive motion information.

(prediction) First, we will define motion-based prediction which is adapted to smooth trajectories such as are observed in natural environments
(markov) Seeing this prediction as a prior on motion transition in the probabilistic domain, we then show how information may be pooled using a Hidden Markov Model, merging current information and measurement likelihood
(SMC) Differently to neural approximations that were used previously, we use a particle filtering method to implement this functional model. Prediction will now take the form of an anisotropic, context-dependent diffusion of local information.

But first, how do we define prediction?
In computer vision, one can define the prediction of the trajectory of some distinguished feature of the image (e.g., a local extremum of luminance) using its motion. here we define the feature to track as the motion at a given position and define Motion-based prediction by modelling smoothness in the trajectory of motion:
knowing the velocity $V_{t-dt}$ at one position $x_{t-dt}$, one expects that velocity will be transported to a new location, that is $x_{t} pprox x_{t-dt} + V_{t-dt} dt$ and approximately conserved in amplitude and direction $V_t pprox V_{t-dt} $. Nothing new: if we go to infinitesimal $dt$, this equation is the advection term in the navier-stokes equations.
This "approximately" may be included in our generative model by some random variable. different values for the variance of this state noise quantify the strength or precision of the prediction... it shapes the association fields in motion space similarly with what happens in the space domain with association field in the orientation domain of static images.
Note that contrary to classical models, position becomes an unknown variable instead of being a parameter as in OF. note also that it is auto-referent since prediction is based itself on motion %: neural = Series / models Grzywack, Burgi et al = "motion tracking"
then, we may use a classical Hidden Markov Model to recurrently update knowledge. This uses three steps knowing the probabilistic motion information at a current time and for simplicity I will just represent local probabilistic motion representations:
prediction of motion thanks to the generative model as a prior of motion transition and an integral over all possible motions. note that prediction is always diffusive
estimation of the likelihood of the predicted state,
merging both, we updated our knowledge from time $ t-dt$ to $ t$
repeating the operation, we define a dynamical system.
this theory seems easy on the paper, and there is nothing new here in the definition of the dynamical system . extit{BUT} ...... I have bad news: it is extremely hard to represent \& implement on classical machines: if we allow ourself a rather modest discretisation if the probability space (position and velocities), the very high dimensionality of the problem makes it tedious to represent ($2e9$ dimensions in the previously shown POF)...

Particle filtering (Sequential Monte-Carlo)

Isard and Blake (1998) IJCV

... and at each point integration over all possible $x-Vdt$ (thus $2e9^2$ )...2 routes :
we may use Laplace's approximation for probabilities, and this Gaussian solution is a spatial Kalman filter: adapted to AP (one object), but lacks generality,
we may do an approximation, for instance by using the neuromimetic models inspired by Grossberg (Burgi, Watamaniuk) : but bad precision, bad prediction. this is why it has been discontinued in the literature,% but the novelty comes from the variables that are represented
instead, we use here a relatively old algorithm used in CV for tracking the contour of objects (a priori unrelated to tracking): CONDENSATION

put simply, it uses Particle filtering, that is implementing the Hidden Markov Model using a set of samples as "control points" (that's why it is also called SMC (Sequential Monte-Carlo) what is extit{NEW} compared to the previous route is that we consider a virtually continuous space with a controlled complexity (given by the number of particles) note that I do not claim that particles are implemented neurally, just that this algorithm provides a good approximation for the complex dynamical system (but see Maass et al) * I will not describe it here (it's not our point now, it's just a solution a method to an hardware problem and I will skip the details now) and we will rather focus on the results...

Prediction is sufficient to solve the aperture problem

LP and Masson (2012) Neural Computation, https://laurentperrinet.github.io/publication/perrinet-12-pred

we may evaluate an average cost on the tracking bias and precision, for each point in the state space here, we simply use as the cost the precision of the retrieved position x, y and velocity u,v of the dot % respectively on top and bottom plots, for left the horizontal component (x, u) on the left, and vertical component on the right (y, v) % \item averaging over 20 trials, one observes 3 types of behaviour: Tracking, FalseTracking, and NoTracking %there is a basin of parameters for which tracking emerges: its boundary is very steep% this will depend on the input showing that blur may be adapted to best fit natural scenes, %% \item this plot also shows that we reach precisions that show a super-resolution ($1e-2$ in position while we have XXX128XXX width images) % \item these properties define the extit{tracking} behaviour and explain why the system shows emergence of line-ending detectors. it would therefore be more correct to call them "smooth trajectory" detectors... in that basin, the system exhibits other properties like motion extrapolation ---that is that it will exhibit some kind of short-term memory%--- that we will detail later. \item[(Left)] Results show the emergence of different states for different prediction precisions: a smooth regime when prediction is lower (No Tracking - NT), a binary stage of motion tracking when it is set as previously (True Tracking - TT) and finally a regime corresponding to higher precisions with relatively efficient mean detection but high bias (False Tracking - FT). % : different stable tracking states. % TODO: As expected, replicating the results of (Weiss, 02) give a result similar to NT (dashed line). % (both axis are logarithmic)We also observe that edge-detector motion grabbers emerge as a property of the dynamical system. \item[(Right)] We give 3 representative examples of the emerging states at one contrast level ($C=0.1$) with starting (red) and end ending (blue) points and resp. NT, TT and FT by showing inferred trajectories for each trial. \item a second reason for which we solve the aperture problem is the fact that incoherent information is explained away by the competition implemented in the inference ...

Outline

Eye movements and the Bayesian brain: the aperture problem

Eye movements and particles: solving the aperture problem

Active inference, eye movements & oculomotor delays

From probabilities to particles: Flash-lag effect

I will present a work done during a visit in Karl Friston's team in London at the UCL: It considers the problem of sensorimotor delays in the optimal control of (smooth) eye movements under uncertainty. Specifically, we consider delays in the visuo-oculomotor loop and their implications for active inference. Active inference uses a generalisation of Kalman filtering to provide Bayes optimal estimates of hidden states and action in generalized coordinates of motion. Representing hidden states in generalized coordinates provides a simple way of compensating for both sensory and oculomotor delays. The efficacy of this scheme is illustrated using neuronal simulations of pursuit initiation responses, with and without compensation. We then consider an extension of the generative model to simulate smooth pursuit eye movements --- in which the visuo-oculomotor system believes both the target and its centre of gaze are attracted to a (hidden) point moving in the visual field. Finally, the generative model is equipped with a hierarchical structure, so that it can recognise and remember unseen (occluded) trajectories and emit anticipatory responses. These simulations speak to a straightforward and neurobiologically plausible solution to the generic problem of integrating information from different sources with different temporal delays and the particular difficulties encountered when a system --- like the oculomotor system --- tries to control its environment with delayed signals. I will give some predictions on neurophysiological observations.

Problem statement: optimal motor control under axonal delays. The central
nervous system has to contend with axonal delays, both at the sensory and the
motor levels. For instance, in the human visuo-oculomotor system, it takes
approximately $ au_s=50~ms$ for the retinal image to reach the visual areas
implicated in motion detection, and a further $ au_m=40~ms $ to reach the
oculomotor muscles.

* ... As a consequence, for a tennis player trying to intercept a
ball at a speed of $20~m.s^{-1}$, the sensed physical position is $1~m$ behind
the true position (as represented here by $ au_s \cdot v$), while the
position at the moment of emitting the motor command will be $.8~m$ ahead of
its execution ($    au_m \cdot v$).  * Note that while the actual position
of the ball when its image formed on the photoreceptors of the retina hits
visual read is approximately at $45$ degrees of eccentricity (red dotted line),
the player's gaze is directed to the ball at its **present** position (red
line), in anticipatory fashion. Optimal control directs action (future motion
of the eye) to the expected position (red dashed line) of the ball in the
future --- and the racket (black dashed line) to the expected position of the
ball when motor commands reach the periphery (muscles).

Experimental results

The first step was to analyze the raw data:

1. The results for the shortest trajectory shows a typical profile for which
the firing rate increases as the bar reaches the receptive field.

2. The results are different for the medium trajectory and the firing rate
increases before the bar reaches the receptive field.

3. We get a similar behaviour for the longest trajectory and the firing rate
increases well before the bar reaches the receptive field.


As a consequence, there seem to be some "priming" (to not say anticipation)
appears to be present in the response to indivual cells..

4. These results are observed in different cells (such as is shown here),
animals, and modalities (VSDI + LFP)

... can we decode the dynamics at the level of the population?

Active inference, eye movements & oculomotor delays

Friston (2010) Nat Neuro Reviews

Unification des theories computationnelles par la minimisation de l'energie libre (MEL).

Cette figure extraite de {Friston10c} represente la place central du principe de MEL dans l'ensemble des theories computationnelles. En particulier, on peut noter que les principes que nous avons detailles plus haut dans les chapitres precedents (reseaux de neurones heuristiques, principes d'optimisation, codage predictif, ...) peuvent se rapporter a ce langage commun. %

Active inference, eye movements & oculomotor delays

LP, Adams and Friston (2015) Biological Cybernetics, https://laurentperrinet.github.io/publication/perrinet-adams-friston-14

Problem statement: optimal motor control under axonal delays. The central nervous system has to contend with axonal delays, both at the sensory and the motor levels. %

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment.

egin{itemize} \item[\odot] It shows the states of the environment and the system in terms of a probabilistic dependency graph, where connections denote directed dependencies. The quantities are described within the nodes of this graph -- with exemplar forms for their dependencies on other variables (see main text). \item[\odot] Hidden (external) and internal states of the agent are separated by action and sensory states. Both action and internal states -- encoding a conditional probability density function over hidden states -- minimise free energy. Note that hidden states in the real world and the form of their dynamics can be different from that assumed by the generative model; this is why hidden states are in bold. See main text for further details.

Active inference, eye movements & oculomotor delays

LP, Adams and Friston (2015) Biological Cybernetics, https://laurentperrinet.github.io/publication/perrinet-adams-friston-14

Problem statement: optimal motor control under axonal delays. The central nervous system has to contend with axonal delays, both at the sensory and the motor levels. %

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment.

Active inference, eye movements & oculomotor delays

LP, Adams and Friston (2015) Biological Cybernetics, https://laurentperrinet.github.io/publication/perrinet-adams-friston-14

This figure illustrates the effects of sensorimotor delays on pursuit initiation (red lines) in relation to compensated (optimal) active inference -- as shown in the previous figure (blue lines). The left panels show the true (solid lines) and estimated sensory input (dotted lines), while action is shown in the right panels. Under pure sensory delays (top row), one can see clearly the delay in sensory predictions, in relation to the true inputs. The thicker (solid and dotted) red lines correspond respectively to (true and predicted) proprioceptive input, reflecting oculomotor displacement. The middle row shows the equivalent results with pure motor delays and the lower row presents the results with combined sensorimotor delays. Of note here is the failure of optimal control with oscillatory fluctuations in oculomotor trajectories, which become unstable under combined sensorimotor delays.

This figure uses the same format as the previous figure -- the only difference is that the target motion has been rectified so that it is (approximately) hemi-sinusoidal. The thing to note here is that the improved accuracy of the pursuit previously apparent at the onset of the second cycle of motion has now disappeared -- because active inference does not have access to the immediately preceding trajectory. This failure of an anticipatory improvement in tracking is contrary to empirical predictions. \item \odot This figure uses the same format as the previous figure -- the only difference is that the generative model has been equipped with a second hierarchical level that contains hidden states, modelling latent periodic behaviour of the (hidden) causes of target motion. With this addition, the improvement in pursuit accuracy apparent at the onset of the second cycle of motion is reinstated. This is because the model has an internal representation of latent causes of target motion that can be called upon even when these causes are not expressed explicitly in the target trajectory.

Active inference, eye movements & oculomotor delays

LP, Adams and Friston (2015) Biological Cybernetics, https://laurentperrinet.github.io/publication/perrinet-adams-friston-14

Outline

Eye movements and the Bayesian brain: the aperture problem

Eye movements and particles: solving the aperture problem

Active inference, eye movements & oculomotor delays

From probabilities to particles: Flash-lag effect

Khoei, Masson and LP (2016) In review, https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17/

3. Following Nihjawan, we have proposed a
diagonal model which extends a motion-based predictive coding model

Herein, we
propose to extend the previously proposed hypothesis that this effect is caused
by the extrapolation of the stimulus' motion to compensate for the neural
delay. However, we propose the novel hypothesis that it explicitly uses the
precision of sensory and predicted motion. By using numerical simulations, we
show that this theory reconciles some previously proposed theories and in
particular why this effect is not seen if the moving dot abruptly stops at the
moment of the flash. More generally, this novel theory proposes that generic
neural computations systematically compensate neural delays to represent the
predicted visual scene at the present time, though it is not yet directly
experienced.

In the current study, the estimated state vector $z = \{x, y, u, v\}$ is
composed of the 2D position ($x$ and $y$) and velocity ($u$ and $v$) of a
(possibly moving) stimulus.  {\sf (A)}~First, we extend a classical Markov
chain into the Nijhawan's diagonal model in order to take into account the
known neural delay $    au$: At time $t$, information is integrated until time
$t- au$, using a Markov chain and a model of state transitions $p(z_t |
z_{t-\delta t})$ such that one can infer the state until the last accessible
information $p( z_{t-   au} | I_{0:(t-  au)})$. This information can then be
``pushed'' forward in time by predicting its trajectory from $t-    au$ to $t$.
In particular $p( z_{t} | I_{0:(t-  au)})$ can be predicted by the same
internal model by using the state transition at the time scale of the delay,
that is, $p(z_t | z_{t- au})$.  This is virtually equivalent to a motion
extrapolation model but without sensory measurements during the time window
between $t- au$ and $t$.  Note that both predictions in this model are based
on the same model of state transitions.  {\sf (B)}~Ones can write a second,
equivalent ``pull'' mode for the diagonal model. Now, the current state is
directly estimated based on a Markov chain on the sequence of delayed
estimations. While being equivalent to the push-mode described above, such a
direct computation allows to more easily combine information from areas with
different delays.  {\sf (C)}~Such a diagonal delay compensation can be
demonstrated in a two-layered neural network including a source (input) and a
target (predictive) layer~\citep{KaplanKhoei14}.  The source layer receives the
delayed sensory information and encodes both position and velocity
topographically on the different retinotopic maps of each layer. Notice that
here, for simplicity, we show only one 2D map of the motions $(x, v)$. The
integration of coherent information can either be done in the source layer
(push mode) or in the target layer (pull mode).  Crucially, to implement a
delay compensation in this motion-based prediction model, one may simply
connect each source neuron to a predictive neuron corresponding to the
corrected position of stimulus $(x+v\cdot   au, v)$ in the target layer. The
precision of this anisotropic connectivity map can be tuned by the width of
convergence from the source to the target populations. Using such a simple
mapping, we have previously shown that the neuronal population activity can
infer the current position along the trajectory despite the existence of neural
delays~\citep{KaplanKhoei14}.

From probabilities to particles: Flash-lag effect

Khoei, Masson and LP (2016) In review, https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17/

**Histogram of the estimated positions as a function of time for the dMBP model.**
Histograms of the inferred horizontal positions (blueish bottom panel) and
horizontal speed (redish top panel) in the dMBP model are shown as a function
of time.  A darker level corresponds to a higher probability, while a light
color corresponds to an unlikely estimation.  In particular, we focus on three
particular epoch along the trajectory, corresponding to the standard, flash
initiated and terminated cycles.  The timing of the flashes are indicated by
dashed vertical lines. In dark, the physical time and in green the delayed
input knowing $ au=100~ms$.  We show the histograms at the two different
levels of our model. The left-hand column illustrates the source layer that
corresponds to the integration of information and, accordingly to the push
mode. The right-hand illustrates the target layer corresponding to the same
information but after motion extrapolation. The histograms indicate the
projected position from the source layer knowing the neural delay $ au$.
Activity in both layers shows three different phases. First, there is a rapid
build-up of the precision of the target after the first appearance of the
moving dot (at $t=300~ms$).  Consistently with the Frölich effect, the
beginning of the trajectory is seen ahead of its physical position (see text
for details). During the second phase, the moving dot is correctly tracked as
    both its velocity and position are correctly inferred. In the source layer,
    there is no extrapolation and the trajectory follows the delayed trajectory
    of the dot (green dotted line). In the target layer, motion extrapolation
    correctly predicts the position at the present time and the position
    follows the actual physical position of the dot (black dotted line).
    Finally, the third phase corresponds to the motion termination. The moving
    dot disappears and the corresponding activity vanishes in the source layer
    at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot
    position was extrapolated and predicted ahead of the terminal position. At
    $t=900~ms$, while motion information is absent, the position information
    is still transiently consistent and extrapolated using a broad, centered
    prior distribution of speeds. Although it is less precise, this position of
    the dot at flash termination is therefore not perceived as leading the
    flash.

From probabilities to particles: Flash-lag effect

Khoei, Masson and LP (2016) In review, https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17/

**Histogram of the estimated positions as a function of time for the dMBP model.**
Histograms of the inferred horizontal positions (blueish bottom panel) and
horizontal speed (redish top panel) in the dMBP model are shown as a function
of time.  A darker level corresponds to a higher probability, while a light
color corresponds to an unlikely estimation.  In particular, we focus on three
particular epoch along the trajectory, corresponding to the standard, flash
initiated and terminated cycles.  The timing of the flashes are indicated by
dashed vertical lines. In dark, the physical time and in green the delayed
input knowing $ au=100~ms$.  We show the histograms at the two different
levels of our model. The left-hand column illustrates the source layer that
corresponds to the integration of information and, accordingly to the push
mode. The right-hand illustrates the target layer corresponding to the same
information but after motion extrapolation. The histograms indicate the
projected position from the source layer knowing the neural delay $ au$.
Activity in both layers shows three different phases. First, there is a rapid
build-up of the precision of the target after the first appearance of the
moving dot (at $t=300~ms$).  Consistently with the Frölich effect, the
beginning of the trajectory is seen ahead of its physical position (see text
for details). During the second phase, the moving dot is correctly tracked as
    both its velocity and position are correctly inferred. In the source layer,
    there is no extrapolation and the trajectory follows the delayed trajectory
    of the dot (green dotted line). In the target layer, motion extrapolation
    correctly predicts the position at the present time and the position
    follows the actual physical position of the dot (black dotted line).
    Finally, the third phase corresponds to the motion termination. The moving
    dot disappears and the corresponding activity vanishes in the source layer
    at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot
    position was extrapolated and predicted ahead of the terminal position. At
    $t=900~ms$, while motion information is absent, the position information
    is still transiently consistent and extrapolated using a broad, centered
    prior distribution of speeds. Although it is less precise, this position of
    the dot at flash termination is therefore not perceived as leading the
    flash.

Outline

Eye movements and the Bayesian brain: the aperture problem

Eye movements and particles: solving the aperture problem

Active inference, eye movements & oculomotor delays

From probabilities to particles: Flash-lag effect