Categorization of microscopy images using a biologically inspired edge co-occurrences descriptor

Lionel Fillatre, Michel Barlaud (i3S) Laurent U Perrinet (INT)

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane
Pascal Wallisch and Tony Movshon @ NYU for the MT data
Rick Adams and Karl Friston @ UCL
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini for the psychophysics

EUVIP Special Session on BICV, Wednesday, October 26, 2016

(AUTHOR) Hi, I am Laurent Perrinet from (LOGO) the Institute de Neurosciences de la Timone in Marseille, a joint unit from the CNRS and AMU. Using computational models, I am investigating the link between the efficiency of behavioural responses in vision, their underlying neural code and their adaptation to the structure of the world.
(SHOW TITLE - THEME) = In particular, eye movements, that is, motion of the eyeballs targeted at orienting our gaze in the visual field, provide a large repertoire of behaviors at different time scales from fast, reflexive-like saccades to smooth movememnts and even to long-term adaptation.
(SHOW TITLE -OBJECTIVE) today, I will be focusing on the key challenges in modelizing visually guided eye movements and more generally to better understand decision making in biology (the living kingdom). in particular, we will focus on the dynamics of these mechanisms to unravel the sequence of processes which allow the efficient response of motor commands to the environment. As such, my goal here is to show that the solutions proposed by motion perception and EMs provide a generic (and rather unique) framework to finesse models of decision making in the generic theory championned by Karl Friston, active inference (AI)
(ACKNO) this endeavour involves different techniques, tools and... persons. From the head on, I wish to thank the people who collaborated to this project and in particular Rick Adams and Karl Friston and the Wellcome Trust Centre for Neuroimaging for providing the tools for a successful visit / but also Mina, Guillaume and Anna for the great oppportunity to link theories with psychophysical data; Wahiba, Giacomo and Frédéric for the challenging task of linking such models to the neural activity in NHP, and finally Jean-Bernard, Sohir and Laurent Madelain for their essential knowledge in adaptation and reinforcement.

more precisely... what are eye movements and how can they be useful for active inference?

Geisler et al. (2001) Vision Research

... it is defined as the set of rules that combine neighboring visual contour elements to refine more global visual processes. however, it is often defined in different spaces (anatomical, visual, behavioural) and it is still an open question as to know if this has a neural correlate.
A successful method to measure the statics of second order was shown by Geisler et al., 2001, Vision Research on a set of natural images (definition)] in this study, they defined second-order statistics to compare an edge as a function of a central reference edge as a pdf on 3 parameters: the distance $d$ between their centers, the angle $\psi$ between the central edge sand the center of the second and $ heta$ the difference between the orientation of both edges. probability is represented by this colormap and to represent on the 2D of the screen this 3d function, they represent in (B) the most probable difference of orientation at each distance and angle, that is $ rg\max_ heta p( heta | d, \phi, \sigma, \pi_0) $, showing the tendency of having collinear, parallel structures in natural images and (C) the most probable angle for each difference of angle and distance, that is $ rg\max_\phi p( \phi | d, heta, \sigma, \pi_0) $ , showing a prior bias in natural image for cocircular edges.

Contour segmentation

Geisler et al. (2001) Vision Research

(Gestalt) ... using this measure of second-order statistics combined with an iterative grouping rule, they could reproduce diverse behavioral results at a global level, for instance here the link that is reported 💠 from a display of artificial edges (A) to 💠 what is reported as perceived (B).
interestingly, this could be successfully modeled by a principled approach such as the non-accidentalness principle see Morel and Grompone, such that they must be a link between both approaches
This thus gives a link between this local dependence present in natural images and the emergence in neural computations of some global Gestalt-like rules as implemented in the brain. One possible candidate substrate for implementing such an association field in the primary visual cortex (V1) in mammals ...

Bosking et al. (1997) Journal of Neuroscience

... is the set of long-range lateral connections between neurons, which could act to facilitate detection of contours matching the association field, and/or inhibit detection of other contours. To fill this role, the lateral connections would need to be orientation specific and aligned along contours, * (colin) and indeed such an arrangement has been found in treeshrew's primary visual cortex (Bosking et al., J Neurosci 17:2112-27, 1997) (neural) if one looks at the primary visual area in the occipital lobe of the cortex using optical imaging as here in the treeshrew by Bosking and colleagues under the supervision of DF, one could represent the distributed, topographical representation of orientation selectivity. in (A) and (B) the orientation giving the most response at each cortical position is represented by hue using the code below from orange for horizontal to blue for verticals, and typical structures are magnified in (C): stripes (on the periphery) and pinwheels. You can understand this as a packing of a 3D feature space on the 2D surface of the cortex. (method) Tree shrew orientation preference maps were obtained using optical imaging. Additionally, 540 nm light was used to map surface blood vessels used for alignment. Biocytin was then injected into a specific site in V1 and the animal was sacrificed 16 hours later. Slices of V1 were imaged to locate the biocytin bouton and the surface blood vessels. The blood vessel information was then used to align the orientation preference maps with the bouton images giving overlaid information on the underlying connectivity from the injection site on the animal. The original experiment used a total of ten cases. (lateral) we show here one result of Bosking which overlay over a map or orientation selectivity the network of lateral connectivity originating froma group of neurons with similar orientations and position. There is a structure in this connectivity towards locality (more pronounced for site B) + connecting iso orientations even on long ranges (A). This type of structure tends to wire together those neurons that have similar orientations, indicating a prior to colinearities. (colin) ... Overall, a typical assumption that the role of lateral interactions is to enhance the activity of neurons which are collinear : it is the so-called association field formalized in Field 93, as was for instance modeled neurally in the work from P. Series or in this version for computer vision (physio) is there a match of these structures with the statistics of natural images? 2) Some authors (Kisvarday, 1997, Chavane and Monier) even say it is weak or inexistent on a the scale of the area... 1: Hunt & Goodhill have reinterpreted above data and shown that there is more diversity than that - TRANSITION : my goal here will be to tackle this problem at different levels:

Outline

The association field

An edge co-occurrences descriptor

Applications to microscopy imaging

Geisler et al. (2001) Vision Research

I will first talk about the association field as measured by edge statistics in understanding lateral connectivity in the primary visual cortex. Using these results, I will give some predictions on behavioural observations. This was joint work with James Bednar who is an expert in topographical models of cortical areas. let's first come back to the results from Geisler

(stats) we will use the same statistical method in this research, ...
(neural) however the underlying statistics was done using an heuristic on the edge extraction (as in Sigman 01) and I show here the edges extracted by Geisler on a sample image as they were used in this study. that's why we decided to keep the same treatment for the statistics but opted for a different, more principled way to extract edges. I collaborated with James Bednar to validate the results on natural images and extend on a quantitative comparison with laboratory environment

LP (2015) Sparse models in BICV, http://github.com/bicv/SparseEdges

Methods:

- LogGabor

 in order to do that, we first used a linear transformation using a log-gabor representation

(definition) this representation is a good and generic model of edges as defined by their shape, orientation and scale. It matches what is well described for the response of simple cells' response in area V1. we show here on the top left that these filters tile evenly the Fourier space, but also that these correspond to a good model of edges at different orientation, scale and phase compared to other dictionaries like the Daubechies wavelet base Db4 in (e) and the steerable pyramid by Eero Simoncelli, \item[(Fischer) obviously, this dictionary is over-complete, but their correlation is easy to compute and allow for a relative translation-rotation-scale invariance. we proved that this was better adapted to the extraction of edges than gabors (Fischer, 07).

coupled with a sparse coding algorithm.

- SparseEdges https://laurentperrinet.github.io/sciblog/files/2015-05-22-a-hitchhiker-guide-to-matching-pursuit/MPtutorial_rec.mp4

(MP) from this linear representation, we searched for the most sparse representation using a $\ell_0$ norm approach for which Matching Pursuit proved to be a good approximation. I refer to this paper that appeared in IEEE TIP for more details [LP (2003) IEEE TIP, LP (2015) BICV] (BICV)] and that i reviewed in a recent book chapter (Perrinet, 2015 bicv), including a model of complex cells' response (Fischer, 2007). It is generic and efficient.

We then computed the ``association field'' ... ../ms/figure_model_ls.png}\% - SVM github links

LP (2015) Sparse models in BICV, http://github.com/bicv/SparseEdges

Methods:

- LogGabor

 in order to do that, we first used a linear transformation using a log-gabor representation

(definition) this representation is a good and generic model of edges as defined by their shape, orientation and scale. It matches what is well described for the response of simple cells' response in area V1. we show here on the top left that these filters tile evenly the Fourier space, but also that these correspond to a good model of edges at different orientation, scale and phase compared to other dictionaries like the Daubechies wavelet base Db4 in (e) and the steerable pyramid by Eero Simoncelli, \item[(Fischer) obviously, this dictionary is over-complete, but their correlation is easy to compute and allow for a relative translation-rotation-scale invariance. we proved that this was better adapted to the extraction of edges than gabors (Fischer, 07).

coupled with a sparse coding algorithm.

- SparseEdges https://laurentperrinet.github.io/sciblog/files/2015-05-22-a-hitchhiker-guide-to-matching-pursuit/MPtutorial_rec.mp4

(MP) from this linear representation, we searched for the most sparse representation using a $\ell_0$ norm approach for which Matching Pursuit proved to be a good approximation. I refer to this paper that appeared in IEEE TIP for more details [LP (2003) IEEE TIP, LP (2015) BICV] (BICV)] and that i reviewed in a recent book chapter (Perrinet, 2015 bicv), including a model of complex cells' response (Fischer, 2007). It is generic and efficient.

We then computed the ``association field'' ... ../ms/figure_model_ls.png}\% - SVM github links

LP (2015) Sparse models in BICV, http://github.com/bicv/SparseEdges

Methods:

- LogGabor

 in order to do that, we first used a linear transformation using a log-gabor representation

(definition) this representation is a good and generic model of edges as defined by their shape, orientation and scale. It matches what is well described for the response of simple cells' response in area V1. we show here on the top left that these filters tile evenly the Fourier space, but also that these correspond to a good model of edges at different orientation, scale and phase compared to other dictionaries like the Daubechies wavelet base Db4 in (e) and the steerable pyramid by Eero Simoncelli, \item[(Fischer) obviously, this dictionary is over-complete, but their correlation is easy to compute and allow for a relative translation-rotation-scale invariance. we proved that this was better adapted to the extraction of edges than gabors (Fischer, 07).

coupled with a sparse coding algorithm.

- SparseEdges https://laurentperrinet.github.io/sciblog/files/2015-05-22-a-hitchhiker-guide-to-matching-pursuit/MPtutorial_rec.mp4

(MP) from this linear representation, we searched for the most sparse representation using a $\ell_0$ norm approach for which Matching Pursuit proved to be a good approximation. I refer to this paper that appeared in IEEE TIP for more details [LP (2003) IEEE TIP, LP (2015) BICV] (BICV)] and that i reviewed in a recent book chapter (Perrinet, 2015 bicv), including a model of complex cells' response (Fischer, 2007). It is generic and efficient.

We then computed the ``association field'' ... ../ms/figure_model_ls.png}\% - SVM github links

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

We first extracted the edges from images using a scale-space analysis (all OSS on github) The histogram was computed as a a 4-dimensional function of distance, (symmetrical) azimuth $\psi$, difference of orientation $ heta$ and ratio of scale. ... second-order statistics are efficiently computed by using a the algorithm from Geisler et al. (2001), with a more general edge extraction algorithm that uses sparse coding %to avoid multiple responses to a single edge. * ... Collinearity and co-circularity results for natural images replicated qualitatively the results from Geisler et al. (2001), confirming that prior information about continuations appeared consistently in natural images.

Probability distribution function of "chevrons" * (angles) By computing measures of the independence of the different variables, we found that the probability density function of the second-order statistics of edges factorizes with on one side distance and scale and on the other side the 2 angles. The first component proved to be quite similar across both classes and the greater difference is seen for different angle configuration. As it can be reduced to 2 dimensions, we can plot the full probability as shown here by different contrast values assigned to all possible chevrons configurations, for all possible "azimuth" values $\psi$ on the horizontal axis and difference of orientation $ heta$ on the vertical axis. Such a plot most strikingly shows the difference between these 2 classes. one issue now that we can show the 2nd order statistics is to know if it would be possible to quantify such difference...

(colin) let's first replicate the result from Geisler by showing that relative to a given edge (segment in the center), what is the Here I show for each distance and angle the most probable difference of angle, showing that collinear and parallel edges predominate. (cocir) a similar pattern is observed the cocircular plot. it reproduces the results from Geisler on natural images, but laboratory environment shows a strong bias to colinearity. If we believe bosking link, obviously, this should have a consequence on antomy. thus we test whether these AF were different across different image classes.

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

we sehow the Ratio of probability distribution function of chevron maps in artificial and animal images compared to the 'natural' set. As previously, we show the probability of chevron maps for two databases (artificial, animal), but here we compare this density to natural images by plotting the log-ratio of densities. These plots highlight the differences with natural images and shows the deviations in co-linearity for artificial images and in co-circularity for animal images (blue: less likely, red: more likely; the opacity is proportional to the absolute log-ratio). In this probabilistic framework, both high and low values carry information on the category of the image. we see a clear differencein natural and laboratory images * (lab) when using the images from the laboratory environment, one finds a different pattern where the colinearity clearly dominates: this quantitatively shows the difference between the edges's second-order statistics...
However, we find that the largely man-made environment in which these animals were reared has a significantly higher probability of collinear edge elements. and this may have consequences on the pattern of lateral interactions that is learned by the primary visual cortex due to plasticity and therefore on the results of Bosking and therefore on our understanding through models * ... We thus predict that if the lateral connection patterns are due to visual experience, the patterns in wild-raised tree shrews would be very different from those measured by Bosking et al. (1997), with shorter-range correlations and less emphasis on collinear continuations. This prediction can be tested in future experiments on matching groups of animals reared in different environments. $p(d,\sigma)$ should be hard-wired while $p( heta, \psi)$ should be adaptive. cocircularity is weak but existent + anti-cocircularity is surprising= should be present as a connection structure. these results may have some repercussions on edge extraction and we could use this prior knowledge to enhance the detection of edges (particles filtering)

We see also a significant diffference when comparing the chevron map on the database containing images with animals with that containing no animal. To test is this difference issignificative, we asked the following question: Making a judgment about the semantic category of a visual scene, such as whether it contains an animal, is typically assumed to involve high-level associative brain areas. Previous explanations require progressively analyzing the scene hierarchically at increasing levels of abstraction, from edge extraction to mid-level object recognition and then object categorization.

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

we computed the statistics of edge co-occurrences for different categories (natural, man-made, or containing an animal) . These differed strongly, with animal images having more curved configurations. Here we show that the statistics of edge co-occurrences alone are sufficient to perform a rough yet robust (translation, scale, and rotation invariant) scene categorization. This seems a bit crazy (in particular with the current trend for deep learning) especially, knowing that most model (like the one from Serre 07) would rather consider that it is better done in 10 processing layers.
Classification results for different sets of natural images (50\% is chance level) and different classification methods. Note that for SVM, we controlled for different feature choice: first-order statistics (SVM 1), full second-order statistics (SVM 2) or the chevron map only (SVM C). Surprisingly, the results are very similar to the results of Serre 07, and we do not lose much performance by choosing the simple chevron map. First, this has direct application in machine learning as this translation- and rotation-independent classifier performs as well as more complicated networks (that is they have more parameters) - and could be imported to existing architectures

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

We show that this geometry alone is sufficient for categorization, and that the pattern of errors made by humans is consistent with this procedure. * To see whether the patterns of errors made by humans are consistent with our model, we studied the second-order statistics of the 50 non-animal images that human subjects in Serre et al. (2007) most commonly falsely reported as having an animal. We call this set of images the false-alarm image dataset. (Left) This chevron map plot shows the ratio between the second-order statistics of the false-alarm images and the full non-animal natural image dataset, computed as in Figure 3 (left). Just as for the images that actually do contain animals , the images falsely reported as having animals have more co-circular and converging (red chevrons) and fewer collinear and orthogonal configurations (blue chevrons). (Right) To quantify this similarity, we computed the Kullback-Leibler distance between the histogram of each of these images from the false-alarm image dataset, and the average histogram of each class. The difference between these two distances gives a quantitative measure of how close each image is to the average histograms for each class. Consistent with the idea that humans are using edge co-occurences to do rapid image categorization, the 50 non-animal images that were worst classified are biased toward the animal histogram ($d' = 1.04$), while the 550 best classified non-animal images are closer to the non-animal histogram.

Outline

The association field

An edge co-occurrences descriptor

Applications to microscopy imaging

Categorization of microscopy images using a biologically inspired edge co-occurrences descriptor

Lionel Fillatre, Michel Barlaud (i3S) Laurent U Perrinet (INT)

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane
Pascal Wallisch and Tony Movshon @ NYU for the MT data
Rick Adams and Karl Friston @ UCL
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini for the psychophysics