From the retina to action:
Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

This work was supported by ANR project "Horizontal-V1" N° ANR-17-CE37-0006".

Churchland, P. S. & Sejnowski, T. J. (1992) The computational brain. Cambridge, MA: MIT Press.

From the retina to action: Levels of modeling

(see this viperlib page)

Bosking et al. (1997) Journal of Neuroscience

... is the set of long-range lateral connections between neurons, which could
act to facilitate detection of contours matching the association field, and/or
inhibit detection of other contours. To fill this role, the lateral connections
would need to be orientation specific and aligned along contours, * (colin) and
indeed such an arrangement has been found in treeshrew's primary visual
cortex  (Bosking et al., J Neurosci 17:2112-27, 1997)
* (neural) if one looks at  the primary visual area in the occipital lobe of the cortex
using optical imaging as here in the treeshrew by Bosking and colleagues under the
supervision of DF, one could represent the distributed, topographical representation
of orientation selectivity. in (A) and (B) the orientation giving the most response
at each cortical position is represented by hue using the code below from orange for
horizontal to blue for verticals, and typical structures are magnified in (C): stripes
(on the periphery) and pinwheels. You can understand this as a packing of a 3D feature
space on the 2D surface of the cortex.
* (method)  Tree shrew orientation preference maps were obtained using optical imaging.
Additionally, 540 nm light was used to map surface blood vessels used for alignment.
Biocytin was then injected into a specific site in V1 and the animal was sacrificed 16
hours later. Slices of V1 were imaged to locate the biocytin bouton and the surface
blood vessels. The blood vessel information was then used to align the orientation
preference maps with the bouton images giving overlaid information on the underlying
connectivity from the injection site on the animal. The original experiment used a total
of ten cases.
* (lateral) we show here one result of Bosking
which overlay over a map or orientation selectivity the network of lateral connectivity
originating froma group of neurons with similar orientations and position. There is
a structure in this connectivity towards locality (more pronounced for site B) +
connecting iso orientations even on long ranges (A). This type of structure tends
to wire together those neurons that have similar orientations, indicating a prior
to colinearities.
*(colin) ... Overall, a typical assumption that the role of lateral interactions is to
enhance the activity of neurons which are collinear : it is the so-called
**association field** formalized in Field 93, as was for instance modeled neurally in
the work from P. Series or in this version for computer vision
* (physio) is there a match of these structures with the statistics of natural images?
 2) Some authors (Kisvarday, 1997, Chavane and Monier) even say it is weak or
inexistent on a the scale of the area... 1:  Hunt & Goodhill have reinterpreted above data and shown that there is more diversity
than that -
* TRANSITION : my goal here will be to tackle this problem at different levels:

LP, Adams and Friston (2015) Biological Cybernetics

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment. It shows the states of the environment and the system in terms of a probabilistic dependency graph, where connections denote directed (causal) dependencies. The quantities are described within the nodes of this graph -- with exemplar forms for their dependencies on other variables.
Hidden (external) and internal states of the agent are separated by action and sensory states. Both action and internal states -- encoding a conditional probability density function over hidden states -- minimise free energy. Note that hidden states in the real world and the form of their dynamics can be different from that assumed by the generative model; (this is why hidden states are in bold. )

LP, Adams and Friston (2015) Biological Cybernetics

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

Ravello, LP, Escobar, Palacios (2019) Scientific Reports

Kremkow, LP, Monier, Alonso, Aertsen, Fregnac, Masson (2016) Push-pull receptive field organization and synaptic depression: Mechanisms for reliably encoding naturalistic stimuli in V1

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

I wish to point here to an essential feature compared to classical feed-forward networks: you can not do the inference in one single shot in general; you need a recurrent / recursive network (see arrow) and this is precisely a possible function for one of the most numerous type of synapses: short-ranges lateral interactions

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

LP (2010) Neural Computation

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

Hodgkin & Huxley (1952) The Journal of physiology

      http://www.math.tamu.edu/~roquesol/Computational_Neuroscience_Summer_I_2018_Session_2-3_Print.pdf

Izhikevich (2003) IEEE Transactions on Neural Network

Brunel (2000) Neurocomputing

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

let's move to the second part : learning

the main goal of Olshasuen was not only sparse coding but the fact that using the sparse code, a simple linear hebbian learning allows to separate independent sources

one can go one step before the cortex and ask the same question in the retina are the same process present ?

Theoretical advances in neural networks modelling have recently been pushed by the field of machine learning. These have proposed that biological neural networks could process information according to certain constraints of efficiency, such as minimizing the surprise of sensory states given noisy or ambiguous sensory inputs. We have developed such a Hierarchical (Deep) Predictive Coding model using unsupervised learning on (static) natural images, and that we wish to extend to (dynamic) sensory flows. Compared to classical Deep Learning models, one advantage of this approach is that the learned synaptic weights are meaningful, that is that they often emerge as to represent explicitly components of the input image in increasing levels of complexity: edges at the first layer, curvatures in the second, the eye or mouth in the third layer when presenting images of faces. Such networks can also be extended to supervised or reinforcement learning by adding efferent layers that would accept such labels.

Boutin, Franciosini, Ruffier, LP (2019) submitted

(from http://cs231n.github.io/convolutional-networks/)

Boutin, Franciosini, Ruffier, LP (2019) submitted

Let's now apply that to natural images of faces

From the retina to action:
Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

This work was supported by ANR project "Horizontal-V1" N° ANR-17-CE37-0006".

The free parameters of this network are the anatomical constraints that we impose to dimension layers. In the particular case of this networks, this is the number of layers, the number of channels (that is the number of overlays of the same map in a given layer), and the size of the kernels implementing the convolutions (in resemblance to receptive fields in biological NNs). Indeed, as in most classical Deep Learning neural networks, these parameters are hand tuned. However, we observed that these superstructural numbers have a great influence on the quality and type of filters that could emerge from the learning phase. Theoretically, this make sense as it imposes different levels of competition within each layer across channels but also by defining a characteristic size in local interactions.

We believe that this theoretical problem is similar to that faced by different the primary visual cortex in different species. Given a similar sensory input, the intra-cortical network imposes different levels of anatomical complexities. We predict that, given realistic orders of magnitudes for different species, we could differentially predict the emergence of different macrostructures in different species. In particular, the emergence of a topographically smooth representation of orientations is present in marmosets while mice have a salt-and-pepper representation. Thanks to this level of understanding, we will use our models to make predictions. In particular, we can generate images which are optimal to evoke activity targeted at a given layer, for instance for the second layer (curvatures) versus the first layer (edges). In practice, such images take the form of textures with different levels of structural complexity. For instance, stimuli targeted at the first layer would correspond to a sub-class of stimuli that we have widely used in previous experiments (Motion Clouds). We predict that more complex classes of stimuli should evoke differential activities in both species.

- Task 5: Bluesky: “Efficient hierarchical representations ” Overall, the role of visual information processing in cortical layers is to represent information robustly to noise and visual transformations (translations zooms rotations). There is thus a pressure on the structure of this processing at different temporal scales to maximize the efficiency of this code. This depends both on the statistics of the sensory input (both in space and time) but also on the repertoire of behavioral tasks that are embodied. We have recently developed a hierarchical unsupervised algorithm for neural networks which allows to generate such representations and that fits well to the processing performed by feed-forward, lateral and feedback connections. In such a model, the message passing between neurons is implemented by propagating waves which progressively refine the representation of sensory data. In particular, tuning the complexity of the representation at different layers allows to see the emergence of different structures. In particular, progressively increasing the complexity in lateral connections shows a phase transition from a salt-and-pepper organization to a more continuous representation of the space x orientation space. This could explain the different architectures observed in different animals. Such models could also make predictions on the adaptation of the network to novel environmental or behavioral contingencies. The goal of this Task is two-sided: first to understand the physiological and behavioral data in the light of predictive coding theories, rooting these results in a quantitative model. The second goal is to generate predictions. In particular, using our models, it will be possible to generate novel stimulus classes that specifically tackle the efficiency of the visual system at its different spatial and dynamic granularities. Lead Laurent/Lyle?

Thanks for your attention!

https://laurentperrinet.github.io/talk/2019-04-03-a-course-on-vision-and-modelization

From the retina to action: Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

From the retina to action: Levels of modeling

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

Outline

From the retina to action: Levels of modeling

Sparse coding in neurophysiology and models

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

From the retina to action: Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

From the retina to action:
Understanding visual processing

From the retina to action:
Understanding visual processing