Back to the present: dealing with delays in biological and neuromorphic systems

Laurent U Perrinet, INT

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane - ANR BalaV1
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini - FACETS-ITN Marie Curie Training

Telluride Neuromorphic Workshop - "Compound Eyes and Event-based Vision" - June 28th, 2017 https://laurentperrinet.github.io/talk/2017-06-30-telluride

(AUTHOR) Hi, I am Laurent Perrinet from (LOGO) the Institute de Neurosciences de la Timone in Marseille, a joint unit from the CNRS and AMU. Using computational models, I am investigating the link between the efficiency of behavioural responses in vision, their underlying neural code and their adaptation to the structure of the world.
(SHOW TITLE - THEME) = In particular, I will use today a visual illusion -the FLE- to provide evidence that we use predictive mechanism to compensate for neural delays and perceive the visual world in real-time (I will try to define today what I mean by that...)
(ACKNO) this endeavour involves different techniques, tools and... persons. From the head on, I wish to thank the people who collaborated to this project and in particular Rick Adams and Karl Friston and the Wellcome Trust Centre for Neuroimaging for providing the tools for a successful visit / but also Mina, Guillaume and Anna for the great oppportunity to link theories with psychophysical data; Wahiba, Giacomo and Frédéric for the challenging task of linking such models to the neural activity in NHP, and finally Jean-Bernard, Sohir and Laurent Madelain for their essential knowledge in adaptation and reinforcement.

LP (2010) Neural Computation, https://laurentperrinet.github.io/publication/perrinet-10-shl / github

LP (2015) "Sparse models" in Biologically Inspired Computer Vision / github

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

We first extracted the edges from images using a scale-space analysis (all OSS
on github) The histogram was computed as a a 4-dimensional function of
distance, (symmetrical) azimuth $\psi$, difference of orientation $ heta$ and
ratio of scale. ... second-order statistics are efficiently computed by using a
the algorithm from Geisler et al. (2001), with a more general edge extraction
algorithm that uses sparse coding to avoid multiple responses to a single
edge.  * ... Collinearity and co-circularity results for natural images
replicated qualitatively the results from Geisler et al. (2001), confirming
that prior information about continuations appeared consistently in natural
images.

Probability distribution function of "chevrons" * (angles)  By computing
measures of the independence of the different variables, we found that the
probability density function of the second-order statistics of edges factorizes
with on one side distance and scale and on the other side the 2 angles.  The
first component proved to be quite similar across both classes and the greater
difference is seen for different angle configuration. As it can be reduced to 2
dimensions, we can plot the full probability as shown here by different
contrast values assigned to all possible chevrons configurations, for all
possible "azimuth" values $\psi$ on the horizontal axis and difference of
orientation $   heta$ on the vertical axis. Such a plot most strikingly shows
the difference between these 2 classes.  one issue now that we can show the 2nd
order statistics is to know if it would be possible to quantify such
difference...

* (colin)  let's first replicate the result from Geisler by showing that
relative to a given edge (segment in the center), what is the Here I show for
each distance and angle the most probable difference of angle, showing that
collinear and parallel edges predominate.  (cocir) a similar pattern is observed the
cocircular plot. it reproduces the results from Geisler on natural images, but
laboratory environment shows a strong bias to colinearity. If we believe
bosking link, obviously, this should have a consequence on antomy. thus we test
whether these AF were different across different image classes.

* ... but before starting, I would like to say one word. in fact, it may not obvious to the general public that problems in artificial intelligence and more generally in understanding the brain is different from one thinks.

* this is perfectly illustrated in this cartoon from the famous XKCD author Randall Munroe: it is easy for an artificial intelligence to geolocalize itself anywhere in the world (or to browse a database of zillions of go games as for alpha go), but whenever it has to something apparently simple ("is there a bird?") computers are clueless

* ... Eye movements are part of this category of hard systems. They can be of different nature: Most simply, saccades are an orienting behavior aiming at catching a specific position in  visual space (or vergence movement in depth). if after fixating at the red fixation dot you try to catch the green dot, then you need to do such saccades, which can be very fast (up to 600 deg/s) When the dot is now moving almong a trajectory, this visual motion can are also responsible for more complex movements such as the reflexive Ocular Following Response or the voluntary Smooth Pursuit EM which aims at stabilizing the visual image on the retina. INterestingly, in this latter case the position of the eye quickly centers on the present, physical position of the dot and we will later see why this is challenging


* so, present computers are a bad analogy to understand the brain and we look here at different solutions found in nature by natural selection - so thanks to the organizers for providing a forum for such endeavour - let's start by looking at anatomy:

The Behavioral receptive field

LP and Masson (2012) Bio Behav Reviews, https://laurentperrinet.github.io/publication/masson-12

Montagnini, LP and Masson (2015) in Biologically Inspired Computer Vision

... we have been busy in the last couple of years to make sense of this repertoire of behaviors, in particular in light with the architecture of the brain areas involved in producing these EMs, for instance for pursuit initiation, ...

the Left-side plot (extracted from this review paper) illustrates in primates a sketch of the hierarchy of neuronal population yielding to ocular following responses from the early visual areas (retina, thalamus, primary visual cortex), finessed by processing in motion specific information in downstream sensory areas (MT for speed or MST for global motion) to the brain stem and the oculomotor muscles which tranform this neural activity in a graded motor output. As such, moving stimuli covering the retina are processed by a cascade of neuronal populations, where information is processed through feedforward, lateral and feedback interactions.
in particular, we took advantage of a Bayesian, probabilistic framework to develop an unified theory, the "behavioral receptive field" which encompasses a large range of psychophysical experiments by modeling behavioral responses from population dynamics. Such cascade implements local, context-dependent extraction of motion information… I will speak more about that on Friday for those in the summer school
today, I would like to focus on a particular problem which will help us unravel the dynamics of decision making: oculomotor delays. Indeed, one challenge for modelling is to understand EMs using AI as a problem of optimal motor control under axonal delays. The central nervous system has to contend with axonal delays, both at the sensory and the motor levels. For instance, in the human visuo-oculomotor system, it takes approximately $ au_s=50~ms$ for the retinal image to reach the visual areas implicated in motion detection, and a further $ au_m=40~ms $ to reach the oculomotor muscles.

* ... As a consequence, for a tennis player ---here (highly trained) Jo-Wilfried Tsonga at Wimbledon--- trying to intercept a passing-shot ball at a (conservative) speed of $20~m.s^{-1}$, the position sensed on the retinal space corresponds to the instant when its image formed on the photoreceptors of the retina and reaches our hypothetical motion perception area behind:

and at this instant, the sensed physical position is lagging behind (as represented here by $ au_s \cdot v 1~m$ ), that is, approximately at $45$ degrees of eccentricity (red dotted line),
while the position at the moment of emitting the motor command will be $.8~m$ ahead of its present physical position ($ au_m \cdot v$).
As a consequence, note that the player's gaze is directed to the ball at its present position (red line), in anticipatory fashion. Optimal control directs action (future motion of the eye) to the expected position (red dashed line) of the ball in the future --- and the racket (black dashed line) to the expected position of the ball when motor commands reach the periphery (muscles). This is obviously an interesting challenge for modelling an optimal control theory.

 * As such, during this talk I will first outline one solution to explain how we may perceive a visual motion while compensating for sensory delays

 * knowing this solution, we will then use it to explain the FLE

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

so let's go back on earth

... but first let now apply this model compensating for the aforementioned visual delays using a well described visual illusion: the flash-lag effect:

a first stimulus moves continuously across the screen along the central horizontal axis. In the FLE, as this moving stimulus reaches the center of the screen, a second stimulus is flashed just above it and in perfect vertical alignment. Despite the fact that the respective horizontal positions of each stimulus are physically identical when the flash occurs, the moving stimulus is most often perceived ahead of the flashed one.

debate for 80 years revived recently

motion extrapolation
differential latency
post-diction

we propose to extend the hypothesis previously proposed by Nihjawan that this effect is caused by the extrapolation of the stimulus' motion to compensate for the neural delay. However, this hypothesis was challenged by other hypothesis that this effect is due to either anatomy (differential latencies) or to the way visual awareness processes the sequence of events (the post-diction from Eagleman)

As a matter of fact, the motion extrapolation hypothesis was challenged because you can notice that the FLE is still present at initiation of the movement but this effect is not seen if the moving dot abruptly stops at the moment of the flash

Sensing with delays

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

Our model is very simple: Following what Nihjawan called a "diagonal model", we have proposed to extend a motion-based predictive coding model by including the known sensory delay in the prediction-estimation loop
More formally, the estimated state vector $z = {x, y, u, v}$ is composed of the 2D position ($x$ and $y$) and velocity ($u$ and $v$) of a (possibly moving) stimulus. As such, we extend Nijhawan's diagonal model into a classical Markov chain in order to take into account the known neural delay $ au$: At time $t$, information is integrated until time $t- au$, using a Markov chain and a model of state transitions $p(z_t |z_{t-\delta t})$ such that one can infer the state until the last accessible information $p(z_{t- au} | I_{0:(t- au)})$. This information can then be ``pushed'' forward in time by predicting its trajectory from $t- au$ to $t$. Interestingly, we use the same model of state transition at the time scale of the delay, that is, $p(z_t | z_{t- au})$.
Importantly, the model for transition is based on a simple model of the transport of visual information knowing the velocity. Indeed, knowing $z(t-\Delta t)$ we may infer the future position of this motion at time $t$ by using Newton's first law of mechanics. The precision of the state transition (as given by the shaded area) tunes the strength of this prediction as a diffusion parameter.
Such a model can be implemented in a two-layered neural network including a source (input) and a target (predictive) layer~\citep{KaplanKhoei14}. The source layer receives the delayed sensory information and encodes both position and velocity topographically on the different retinotopic maps of each layer. Notice that here, for simplicity, we show only one 2D map of the motions $(x, v)$. Crucially, the delay compensation in this motion-based prediction model, is simply implemented by connecting each source neuron to a predictive neuron corresponding to the corrected position of stimulus $(x+v\cdot au, v)$ in the target layer. The precision of this anisotropic connectivity map can be tuned by the width of convergence from the source to the target populations. We will see on Friday that this property (that it is implemented in the connectivity) extends to more complex, multi-layered systems which may generalize to more complex trajectories.

Sensing with delays

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

Our model is very simple: Following what Nihjawan called a "diagonal model", we have proposed to extend a motion-based predictive coding model by including the known sensory delay in the prediction-estimation loop
More formally, the estimated state vector $z = {x, y, u, v}$ is composed of the 2D position ($x$ and $y$) and velocity ($u$ and $v$) of a (possibly moving) stimulus. As such, we extend Nijhawan's diagonal model into a classical Markov chain in order to take into account the known neural delay $ au$: At time $t$, information is integrated until time $t- au$, using a Markov chain and a model of state transitions $p(z_t |z_{t-\delta t})$ such that one can infer the state until the last accessible information $p(z_{t- au} | I_{0:(t- au)})$. This information can then be ``pushed'' forward in time by predicting its trajectory from $t- au$ to $t$. Interestingly, we use the same model of state transition at the time scale of the delay, that is, $p(z_t | z_{t- au})$.
Importantly, the model for transition is based on a simple model of the transport of visual information knowing the velocity. Indeed, knowing $z(t-\Delta t)$ we may infer the future position of this motion at time $t$ by using Newton's first law of mechanics. The precision of the state transition (as given by the shaded area) tunes the strength of this prediction as a diffusion parameter.
Such a model can be implemented in a two-layered neural network including a source (input) and a target (predictive) layer~\citep{KaplanKhoei14}. The source layer receives the delayed sensory information and encodes both position and velocity topographically on the different retinotopic maps of each layer. Notice that here, for simplicity, we show only one 2D map of the motions $(x, v)$. Crucially, the delay compensation in this motion-based prediction model, is simply implemented by connecting each source neuron to a predictive neuron corresponding to the corrected position of stimulus $(x+v\cdot au, v)$ in the target layer. The precision of this anisotropic connectivity map can be tuned by the width of convergence from the source to the target populations. We will see on Friday that this property (that it is implemented in the connectivity) extends to more complex, multi-layered systems which may generalize to more complex trajectories.

Results using PBP

Let me show you now the results of our simulations on the standard flash-lag stimulus by first showing the results of a predictive system with no delay compensation. We here simply show the source layer in the prediction model which tries to track moving patterns but do not compensate for the known delay.
We observe that the moving dot is correctly tracked and the flash correctly detected. I have slowed down the speed of the movie to show that relative to each other, the flash and moving dot appear at the same position at the time of the flash.
The situation is different in the motion-based predictive system : in this slowed down movie, at the time the flash appears, the dot's position was predicted ahead of the position of the flash, similarly to what is reported in psychophysics.
Moreover, we have predicted a similar effect with a large range of parameters of the stimulus, such as speed or contrast. But for the moment, let us focus on the reasons why the model shows a similar behavior

Results using MBP

Let me show you now the results of our simulations on the standard flash-lag stimulus by first showing the results of a predictive system with no delay compensation. We here simply show the source layer in the prediction model which tries to track moving patterns but do not compensate for the known delay.
We observe that the moving dot is correctly tracked and the flash correctly detected. I have slowed down the speed of the movie to show that relative to each other, the flash and moving dot appear at the same position at the time of the flash.
The situation is different in the motion-based predictive system : in this slowed down movie, at the time the flash appears, the dot's position was predicted ahead of the position of the flash, similarly to what is reported in psychophysics.
Moreover, we have predicted a similar effect with a large range of parameters of the stimulus, such as speed or contrast. But for the moment, let us focus on the reasons why the model shows a similar behavior

Sensing with delays

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

Sensing with delays

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

Sensing with delays

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

as a summary, we have shown that, evolving from other theories of prediction of post-diction, we have shown that perceptual results are well modeled by a simple theory. This theory states that the agent models (predicts) the scene at the present time (in spite of inherent but known delays) - and we called it parodiction from the greek "paros" the present time.
More generally, this novel theory proposes that generic neural computations systematically compensate neural delays to represent the predicted visual scene at the present time, though it is not yet directly experienced. a consequence of this view -that is shared among a large number of physicists- is that time as a continous line is an emerging property of the representation used by these agents and does not need to exist physically (though this rather bold statement is quite infalsifiable) - the world is latent - there is only the arrow of time (= laws of nature)
There is a fundamental limit to this theory: the agent has an uniform access to the whole field of view. With a foveal point, it may fail to track - thus we need action as an eye movement ; but this is another project :-)