La modélisation biomorphique de la perception visuelle

Laurent Perrinet

Séminaire organisé dans le cadre du projet Biomorphisme. Approches sensibles et conceptuelles des formes du vivant, 11/10/2018

https://laurentperrinet.github.io/talk/2018-10-11-bio-morphisme/

La modélisation biomorphique de la perception visuelle

Outline

Représentation des images / Trame Élasticité (2016)

Apprentissage non-supervisé / Trame Instabilité (2018)

Dynamique de la perception / Tropique (2013)

Bosking et al. (1997) Journal of Neuroscience

... is the set of long-range lateral connections between neurons, which could
act to facilitate detection of contours matching the association field, and/or
inhibit detection of other contours. To fill this role, the lateral connections
would need to be orientation specific and aligned along contours, * (colin) and
indeed such an arrangement has been found in treeshrew's primary visual
cortex  (Bosking et al., J Neurosci 17:2112-27, 1997)
* (neural) if one looks at  the primary visual area in the occipital lobe of the cortex
using optical imaging as here in the treeshrew by Bosking and colleagues under the
supervision of DF, one could represent the distributed, topographical representation
of orientation selectivity. in (A) and (B) the orientation giving the most response
at each cortical position is represented by hue using the code below from orange for
horizontal to blue for verticals, and typical structures are magnified in (C): stripes
(on the periphery) and pinwheels. You can understand this as a packing of a 3D feature
space on the 2D surface of the cortex.
* (method)  Tree shrew orientation preference maps were obtained using optical imaging.
Additionally, 540 nm light was used to map surface blood vessels used for alignment.
Biocytin was then injected into a specific site in V1 and the animal was sacrificed 16
hours later. Slices of V1 were imaged to locate the biocytin bouton and the surface
blood vessels. The blood vessel information was then used to align the orientation
preference maps with the bouton images giving overlaid information on the underlying
connectivity from the injection site on the animal. The original experiment used a total
of ten cases.
* (lateral) we show here one result of Bosking
which overlay over a map or orientation selectivity the network of lateral connectivity
originating froma group of neurons with similar orientations and position. There is
a structure in this connectivity towards locality (more pronounced for site B) +
connecting iso orientations even on long ranges (A). This type of structure tends
to wire together those neurons that have similar orientations, indicating a prior
to colinearities.
*(colin) ... Overall, a typical assumption that the role of lateral interactions is to
enhance the activity of neurons which are collinear : it is the so-called
**association field** formalized in Field 93, as was for instance modeled neurally in
the work from P. Series or in this version for computer vision
* (physio) is there a match of these structures with the statistics of natural images?
 2) Some authors (Kisvarday, 1997, Chavane and Monier) even say it is weak or
inexistent on a the scale of the area... 1:  Hunt & Goodhill have reinterpreted above data and shown that there is more diversity
than that -
* TRANSITION : my goal here will be to tackle this problem at different levels:

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

We first extracted the edges from images using a scale-space analysis (all OSS
on github) The histogram was computed as a a 4-dimensional function of
distance, (symmetrical) azimuth $\psi$, difference of orientation $ heta$ and
ratio of scale. ... second-order statistics are efficiently computed by using a
the algorithm from Geisler et al. (2001), with a more general edge extraction
algorithm that uses sparse coding %to avoid multiple responses to a single
edge.  * ... Collinearity and co-circularity results for natural images
replicated qualitatively the results from Geisler et al. (2001), confirming
that prior information about continuations appeared consistently in natural
images.

Probability distribution function of "chevrons" * (angles)  By computing
measures of the independence of the different variables, we found that the
probability density function of the second-order statistics of edges factorizes
with on one side distance and scale and on the other side the 2 angles.  The
first component proved to be quite similar across both classes and the greater
difference is seen for different angle configuration. As it can be reduced to 2
dimensions, we can plot the full probability as shown here by different
contrast values assigned to all possible chevrons configurations, for all
possible "azimuth" values $\psi$ on the horizontal axis and difference of
orientation $   heta$ on the vertical axis. Such a plot most strikingly shows
the difference between these 2 classes.  one issue now that we can show the 2nd
order statistics is to know if it would be possible to quantify such
difference...

* (colin)  let's first replicate the result from Geisler by showing that
relative to a given edge (segment in the center), what is the Here I show for
each distance and angle the most probable difference of angle, showing that
collinear and parallel edges predominate.  (cocir) a similar pattern is observed the
cocircular plot. it reproduces the results from Geisler on natural images, but
laboratory environment shows a strong bias to colinearity. If we believe
bosking link, obviously, this should have a consequence on antomy. thus we test
whether these AF were different across different image classes.

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

 We show that this geometry alone is sufficient for categorization, and that
 the pattern of errors made by humans is consistent with this procedure.  * To
 see whether the patterns of errors made by humans are consistent with our
 model, we studied the second-order statistics of the 50 non-animal images that
 human subjects in Serre et al. (2007) most commonly falsely reported as having
 an animal. We call this set of images the false-alarm image dataset. (Left)
 This chevron map plot shows the ratio between the second-order statistics of
 the false-alarm images and the full non-animal natural image dataset, computed
 as in Figure 3 (left). Just as for the images that actually do contain animals
 , the images falsely reported as having
 animals have more co-circular and converging (red chevrons) and fewer
 collinear and orthogonal configurations (blue chevrons). (Right) To quantify
 this similarity, we computed the Kullback-Leibler distance between the
 histogram of each of these images from the false-alarm image dataset, and the
 average histogram of each class. The difference between these two distances
 gives a quantitative measure of how close each image is to the average
 histograms for each class. Consistent with the idea that humans are using edge
 co-occurences to do rapid image categorization, the 50 non-animal images that
 were worst classified are biased toward the animal histogram ($d' = 1.04$),
     while the 550 best classified non-animal images are closer to the
     non-animal histogram.

Représentation des images / Trame Élasticité (2016)

TRAME Elasticité Fondation Vasarely from Etienne Rey.

Outline

Représentation des images / Trame Élasticité (2016)

Apprentissage non-supervisé / Trame Instabilité (2018)

Dynamique de la perception / Tropique (2013)

Apprentissage non-supervisé / Trame Instabilité (2018)

Outline

Représentation des images / Trame Élasticité (2016)

Apprentissage non-supervisé / Trame Instabilité (2018)

Dynamique de la perception / Tropique (2013)

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

Problem statement: optimal motor control under axonal delays. The central
nervous system has to contend with axonal delays, both at the sensory and the
motor levels. For instance, in the human visuo-oculomotor system, it takes
approximately $ au_s=50~ms$ for the retinal image to reach the visual areas
implicated in motion detection, and a further $ au_m=40~ms $ to reach the
oculomotor muscles.

Dynamique de la perception / Tropique (2013)

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

* ... As a consequence, for a tennis player trying to intercept a
ball at a speed of $20~m.s^{-1}$, the sensed physical position is $1~m$ behind
the true position (as represented here by $ au_s \cdotec{V}$), while the
position at the moment of emitting the motor command will be $.8~m$ ahead of
its execution ($    au_m \cdotec{V}$).  * Note that while the actual position
of the ball when its image formed on the photoreceptors of the retina hits
visual read is approximately at $45$ degrees of eccentricity (red dotted line),
the player's gaze is directed to the ball at its **present** position (red
line), in anticipatory fashion. Optimal control directs action (future motion
of the eye) to the expected position (red dashed line) of the ball in the
future --- and the racket (black dashed line) to the expected position of the
ball when motor commands reach the periphery (muscles).

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

so let's go back on earth

... but first let now apply this model compensating for the aforementioned visual delays using a well described visual illusion: the flash-lag effect:

a first stimulus moves continuously across the screen along the central horizontal axis. In the FLE, as this moving stimulus reaches the center of the screen, a second stimulus is flashed just above it and in perfect vertical alignment. Despite the fact that the respective horizontal positions of each stimulus are physically identical when the flash occurs, the moving stimulus is most often perceived ahead of the flashed one.

debate for 80 years revived recently

motion extrapolation
differential latency
post-diction

we propose to extend the hypothesis previously proposed by Nihjawan that this effect is caused by the extrapolation of the stimulus' motion to compensate for the neural delay. However, this hypothesis was challenged by other hypothesis that this effect is due to either anatomy (differential latencies) or to the way visual awareness processes the sequence of events (the post-diction from Eagleman)

As a matter of fact, the motion extrapolation hypothesis was challenged because you can notice that the FLE is still present at initiation of the movement but this effect is not seen if the moving dot abruptly stops at the moment of the flash

Dynamique de la perception / Tropique (2013)

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

Dynamique de la perception / Tropique (2013)

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

Dynamique de la perception / Tropique (2013)

Khoei, Masson and LP (2017) PLoS CB https://laurentperrinet.github.io/publication/khoei-masson-perrinet-17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $ au=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $ au_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

Experimental results

The first step was to analyze the raw data:

1. The results for the shortest trajectory shows a typical profile for which
the firing rate increases as the bar reaches the receptive field.

2. The results are different for the medium trajectory and the firing rate
increases before the bar reaches the receptive field.

3. We get a similar behaviour for the longest trajectory and the firing rate
increases well before the bar reaches the receptive field.


As a consequence, there seem to be some "priming" (to not say anticipation)
appears to be present in the response to indivual cells..

4. These results are observed in different cells (such as is shown here),
animals, and modalities (VSDI + LFP)

... can we decode the dynamics at the level of the population?

Dynamique de la perception / Tropique (2013)

Tropique Experimentation Film2 from Etienne Rey.