Motion-Perception | Laurent Perrinet

Illusions visuelles

Victor Vasarely (1971) Gare Montparnasse

La perception comme processus émergent

La perception comme processus émergent

François Morrelet (1962) Mönchengladbach, Sphère - trames

La perception comme processus émergent

Etienne Rey, Trames

La perception comme processus émergent

Etienne Rey (2025) Variable Density, série Delaunay

Neurosciences computationnelles

Neurosciences computationnelles

Anatomie du système visuel humain

Système visuel humain : le modèle HMAX

[Serre and Poggio, 2007]

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Modèles hybrides d’IA

Utiliser des modèles d’apprentissage profond pour comprendre le cortex sensoriel [Yamins & DiCarlo, 2016]

Modèles de langage

Transformer: Attention is All You Need [Vaswani et al., 2017]

Les Neurosciences Computationnelles c’est un métier ?

Ingénieur, chercheur, journaliste, …

Neurosciences Computationnelles pour l’IA

Neurosciences Computationnelles pour la biologie

Topo Neurosciences Computationnelles

Laurent Perrinet

Computational Neuroscience Center

[2026-06-18]

Tout public / Me contacter : laurent.perrinet@univ-amu.fr

Working Memory in SNNs

Thu, 16 Apr 2026 14:00:00 +0000

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Cerco seminar

[2026-04-16]

Contact me @ laurent.perrinet@univ-amu.fr

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this talk at the CerCO, I will be speaking about working memory, that is storing patterns with duration of the order of seconds, in spiking neural networks. This is a hard problem as spiking neurons have a limited memory of the order of tens of milliseconds. How can one extend this memory to larger durations? Here, I will be presenting a method for building WM in Spiking Neural Networks by using Heterogeneous Delays.

I’d like to thank Antoine for the invitation and you for listening. These slides are available from my web-site, along with a number of references. The outline of the talk is as follows: first, I’ll describe how one may perform computations using Heterogeneous Delays - and present a toy model example; then, I’ll show real scale example quantifying the performance on synthetic data.

Spiking Neural Networks: Leaky Integrate-and-Fire

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neurobiology

Spiking Neural Networks: neurobiology

Spiking Neural Networks: neurobiology

Spiking Neural Networks: neurobiology

Etienne Rey (2010) Spectre audiographique

Spiking Neural Networks: Leaky Integrate-and-Fire

Review on Precise Spiking Motifs.

Spiking Neural Networks: Heterogeneous Delays

Review on Precise Spiking Motifs.

Heterogeneous Delays Spiking Neural Network: HD-SNN

Spiking Neural Network: Polychronization

Izhikevich (2006)

Spiking Neural Network: Polychronization

Izhikevich (2006)

Spiking Neural Network: Polychronization

LP (2026)

Methods : BPTT (snn Torch) - synthetic target

Methods : Weight initialization

$$ I_j(t) = \sum_{i=1}^{N} \bigl ( \sum_{d=1}^{D} \mathbf{W}_{j, i, d} \cdot s_i(t-d) \bigr ) $$ $$ u_j(t) = \beta \cdot u_j(t-1) \cdot (1 - s_j(t-1)) + I_j(t) $$ $$ s_j(t) = \mathbf{H}[u_j(t) \geq \vartheta] $$ $$ \mathbf{W} \mathbf{C} \approx \mathbf{S} $$ $$ w_{j, i, d} = \frac{1}{N \cdot D \cdot p_A \cdot M} \sum_{\mu=1}^{M} \sum_{t=D+1}^{T} s_{j}^{\mu}(t) \cdot s_i^{\mu}(t-d) $$

Results : recall of target

Results : memory retrieval

Results : recall of target with noise

Results : role of parameters

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Cerco seminar

[2026-04-16]

Contact me @ laurent.perrinet@univ-amu.fr

Working Memory in SNNs

Wed, 15 Apr 2026 09:00:00 +0000

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Austrian Symposium on AI, Robotics and Vision

[2026-04-15]

Contact me @ laurent.perrinet@univ-amu.fr

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this talk at this AIROV workshop on Recent Advances in SNNs, I will be speaking about working memory, that is storing patterns with duration of the order of seconds, in spiking neural networks. This is a hard problem as spiking neurons have a limited memory of the order of tens of milliseconds. How can one extend this memory to larger durations? Here, I will be presenting a method for building WM in Spiking Neural Networks by using Heterogeneous Delays.

I’d like to thank Sander Bohté and Sebastian Otte for the organization of this workshop and you for listening. These slides are available from my web-site, along with a number of references. The outline of the talk is as follows: first, I’ll describe how one may perform computations using Heterogeneous Delays - and present a toy model example; then, I’ll show real scale example quantifying the performance on synthetic data.

Polychronization

Methods : BPTT (snn Torch) - synthetic target

Methods : Weight initialization

Results : recall of target

Results : role of parameters

Results : memory retrieval

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Austrian Symposium on AI, Robotics and Vision

[2026-04-15]

Contact me @ laurent.perrinet@univ-amu.fr

2026-04-11-intelligence-du-regard

Sat, 11 Apr 2026 00:00:00 +0000

L’intelligence du regard

Laurent Perrinet

Forum des Sciences Cognitives 2026

[2026-04-11]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

Bonjour, je me présente : Laurent Perrinet. Je suis très heureux de participer au Forum des Sciences Cognitives et je remercie les organisateurs pour cette invitation. Je suis d’autant plus ravi d’y participer que je ne vais pas parler de mes recherches habituelles, mais plutôt exposer la collaboration que j’ai avec un artiste plasticien à Marseille. Mon objectif : vous convaincre des bénéfices que l’on peut tirer à s’ouvrir au monde artistique pour mieux percer les mystères de la cognition dans toute sa diversité.

Les objectifs de cet exposé seront multiples :

Vous faire découvrir certains artistes contemporains qui questionnent notre rapport aux nombres visuels
Montrer la diversité de la vision à travers les interactions entre art et science
Dévoiler certains mystères de la vision au travers de l’expérience artistique et les applications que cela peut avoir sur notre compréhension de la cognition

Art & Sciences révèlent la diversité de notre vision

Etienne Rey

Art & Sciences révèlent la diversité de notre vision

Notre collaboration a commencé quand il m’a invité à présenter mon travail sur la perception visuelle au vernissage de cette œuvre qui représente une visualisation spatio-temporelle du spectre audiographique du son d’une cloche. Il est composé de multiples plaques semi-transparentes et dichroïques, c’est-à-dire ayant la capacité de présenter différentes couleurs selon l’angle de vue. Ce volume sculptural donne, de façon furtive, toute la profondeur de cette expérience sensorielle.

C’est là que se révèle « L’irraisonnable efficacité de la vision » — je reprends les mots de Wigner à propos de la capacité des mathématiques à sonder le monde. Car nous nous retrouvons devant un constat similaire : comment est-il possible avec aussi peu de moyens d’obtenir une perception si vivante du monde qui nous entoure ?

“L’irraisonnable efficacité de la vision”

Nage de la raie, 1894 [Étienne-Jules Marey]

L’enjeu est d’abord scientifique : comprendre le mouvement. Il a d’ailleurs donné son nom à l’ISM, l’Institute for Scientific Motion.

Et cette capacité n’est pas nouvelle.

“L’irraisonnable efficacité de la vision”

Panneau Des Lions [Grotte chauvet, -30 kA]

“L’irraisonnable efficacité de la vision”

Comment la vision a évolué… [LP, 2024, The Conversation]

Illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Illusions visuelles

Illusions visuelles

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Victor Vasarely (1971) Gare Montparnasse

La perception comme processus émergent

La perception comme processus émergent

François Morrelet (1962) Mönchengladbach, Sphère - trames

La perception comme processus émergent

Etienne Rey, Trames

C’est dans ce cadre que nous avons expérimenté avec Etienne Rey sur des trames, qui crée ces interférences. Émergence de nouvelles formes — hexagones comme utilisés à l’Alhambra — espaces tridimensionnels.

Je reviendrai sur le fait que vous pouvez transformer l’image en bougeant les yeux.

La perception comme processus émergent

Etienne Rey (2025) Variable Density, série Delaunay

La vibration des apparences

Paul Cézanne, Montagne Sainte-Victoire, 1904

La vibration des apparences

Merleau-Ponty, Sens et non-sens

Etienne Rey (2019) Horizon faille - Densité flou - Sans gravité - une poétique de l’air à Ardenome Avignon https://www.enrevenantdelexpo.com

La vibration des apparences

Etienne rey (2025) Polaire

La vibration des apparences


def retino_grid(cr, N_rho=34, N_phi=233, N_H, N_V, offset,
 size_mag, ecc_max, alpha, c1, c2, power):
 # https://laurentperrinet.github.io/sciblog/posts/2020-04-16-creating-an-hexagonal-grid.html
 phi_v, rho_v = np.meshgrid(np.linspace(0, 2*np.pi, N_phi, endpoint=False),
 np.linspace(0, ecc_max, N_rho+1, endpoint=True)[1:], sparse=False, indexing='xy')
 phi_v[::2, :] += np.pi/N_phi

 offsets, colors = [-offset, offset], [c1, c2]
 for offset_, color in zip(offsets, colors):
 # convert to cartesian coordinates
 X, Y = (rho_v * np.sin(phi_v) + offset_+1)/2, (rho_v * np.cos(phi_v)+1)/2
 R = size_mag * rho_v**power / N_rho

 for x, y, r in zip(X.ravel(), Y.ravel(), R.ravel()):
 circle(cr, x, y, r)
 cr.set_source_rgba(*hue_to_rgba(color, alpha))
 cr.fill()
 return cr
c_blue, dc = 240, 60
opts = dict(N_rho=N_rho, N_phi=N_phi, N_H=N_H, N_V=N_V,
 offset=0.07, size_mag=0.3, ecc_max=0.8, alpha=0.80,
 c1=c_blue-dc, c2=c_blue+dc)
@disp
def draw(cr, N_H=N_H, N_V=N_V): cr = retino_grid(cr, **opts)

À quoi sert la vision ?

Ilya Repin (1884) An Unexpected Visitor

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor How long?

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor - Age?

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

E Rey et LP (2026) Formes & perception

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

Illusions visuelles

Etienne Rey, Spectre audiographique

Art & Sciences révèlent la vision en action

Art & Sciences révèlent la vision en action

Carlos Cruz-Diez (2013) Chromosaturation

Felice Varini.

Art & Sciences révèlent la vision en action : Tropique

Etienne Rey, Tropique

Art & Sciences révèlent la vision en action : Tropique

Etienne Rey, TRAME ÉLASTICITÉ

Art & Sciences révèlent la vision en action : TRAME ÉLASTICITÉ

Art & Sciences révèlent la vision en action

Ede Rancz, Role of neuromodulators in active perception

Art & Sciences révèlent la vision en action

Ede Rancz, Role of neuromodulators in active perception

L’intelligence du regard

Laurent Perrinet

Forum des Sciences Cognitives 2026

[2026-04-11]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

Diapositives supplémentaires

Victor Vasarely

Victor Vasarely (1962) Mönchengladbach, Sphère - trames

Victor Vasarely

Victor Vasarely (1977)Outdoor Vasarely artwork at the church of Pálos in Pécs.

Victor Vasarely

Victor Vasarely (1962) Supernovae

Victor Vasarely

Victor Vasarely (1976) Fondation Vasarely, Aix-en-Provence

Etienne Rey, Cristal n2

La vibration des apparences

2026-03-05-ue-natural-cognition

Thu, 05 Mar 2026 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2026-03-05] Master 1 Neuroscience, UE Natural Cognition, Artificial Cognition

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

[Serre and Poggio, 2007]

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Hybrid IA models

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Principles of vision?

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Principles of Vision

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2026-03-05] Master 1 Neuroscience, UE Natural Cognition, Artificial Cognition

Contact me @ laurent.perrinet@univ-amu.fr

Artificial neural networks and machine learning applied to the understanding of biological vision

Thu, 05 Mar 2026 00:00:00 +0000

Practical work: https://github.com/laurentperrinet/2026-03_UE-neurosciences-computationnelles/

📖 See the full publication:

Laurent U Perrinet (2026). Working Memory with Polychronous Chains. arXiv preprint arXiv:2604.14096.

Cite URL Preprint Code

2026-02-10-biomplus

Tue, 10 Feb 2026 00:00:00 +0000

Recréer des réseaux neuronaux pour améliorer la compréhension de notre cerveau

Laurent Perrinet

Webinaire Biome+ [Biomimétisme & Neurosciences]

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

Attention in Vision Transformers and in Natural Vision

Saccade selection method. Matthis Dallain with the EDGE Team @ LEAT Laboratory

Learning where to look

Spiking Neural Networks

From frame-based to event-based cameras.

Spiking Neural Networks

The HD-SNN neural network.

Recréer des réseaux neuronaux pour améliorer la compréhension de notre cerveau

Laurent Perrinet

Webinaire Biome+ [Biomimétisme & Neurosciences]

[2026-02-03]

2026-02-03-ai-and-neuroscience-day

Tue, 03 Feb 2026 00:00:00 +0000

Neuroscience & AI: Energy-efficient visual processing algorithms

Laurent Perrinet

Journée Neurosciences et IA / IA et Neurosciences de NeuroMarseille

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

Spiking Neural Networks

From frame-based to event-based cameras.

Spiking Neural Networks

The HD-SNN neural network.

Attention in Vision Transformers and in Natural Vision

Saccade selection method. Matthis Dallain with the EDGE Team @ LEAT Laboratory

Learning where to look

Neuroscience & AI: Energy-efficient visual processing algorithms

Laurent Perrinet

Journée Neurosciences et IA / IA et Neurosciences de NeuroMarseille

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

2026-01-29-emergences

Thu, 29 Jan 2026 00:00:00 +0000

Neurosciences and sparsity

Laurent Perrinet

Séminaire à l’atelier “IA embarquée” du PEPR IA

[2026-01-29]

Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations in computer vision

[Lennie, 2003, The Cost of Cortical Computation]

Neurosciences and sparsity: a survey

Go to wooclap.com
Enter the code HLEQUP
Or directly follow https://app.wooclap.com/HLEQUP?from=instruction-slide

Neurosciences and sparsity: a survey

Neurosciences and sparsity

Neurosciences and sparsity

[Brunel, 2001]

Neurosciences and sparsity

Neurosciences and sparsity

Neurosciences and sparsity

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ & = \frac{1}{2\sigma_n^2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 - \sum_{i=1}^{K} \log Pr( a[i] ) \end{aligned} $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations and learning

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Neurosciences and sparsity

Laurent Perrinet

Séminaire à l’atelier “IA embarquée” du PEPR IA

[2026-01-29]

Contact me @ laurent.perrinet@univ-amu.fr

Ede Rancz

Sat, 03 Jan 2026 09:00:00 +0000

Ede Rancz is a Research Director at the Mediterranean Institute of Neurobiology, Marseille, France. He is interested in the neural circuits underlying visual perception and decision-making. He uses a combination of in vivo and in vitro electrophysiology, optogenetics, and computational modeling to study the function of the visual cortex. He is also interested in the development of new tools and methods for studying neural circuits.

*CENTURI call “Neuromodulatory control of predictive processing in visual cortical circuits” (co-direction with Laurent Perrinet) (POSITION HAS BEEN FILLED)

Using a cloded-loop virtual reality setup, can we trace the role of neuromodulators in active perception in terms of predictive processing?

Thesis director of: Hilde Langengen-Teigen, Mediterranean Institute of Neurobiology, Marseille. This PhD position was made possible thanks to a 3-year contract from AIX-MARSEILLE University awarded by the Turing Centre for Living Systems PhD call (CENTURI).

Population decoding of visual motion direction

Thu, 01 Jan 2026 00:00:00 +0000

🧠 Excited to share our latest research led by Alexandre Lainé and presented this summer at AREADNE 2026!

“Population decoding of visual motion direction”

Our work explores how populations of neurons in the primary visual cortex (V1) of marmoset monkeys encode visual motion direction, with a particular focus on understanding how uncertainty influences this neural decoding process. Key highlights:

Advanced population-level analysis of V1 neural responses to motion stimuli
Novel insights into how the brain handles uncertainty in visual motion processing
Marmoset model providing crucial translational insights for visual neuroscience

This research contributes to our fundamental understanding of how the visual system processes motion information at the earliest stages of cortical processing, with important implications for both basic neuroscience and potential clinical applications. Thank you to the AREADNE organizing committee for hosting such an inspiring conference!

Link to publication: https://laurentperrinet.github.io/publication/laine-26-areadne/

For deeper insights into uncertainty processing mechanisms in the visual cortex, see our Nature Communications Biology study:
Hugo Ladret, Nelson Cortes, Lamyae Ikan, Frédéric Chavane, Christian Casanova, Laurent U Perrinet (2023). Cortical recurrence supports resilience to sensory variance in the primary visual cortex. Nature Communications Biology.
PDF Cite DOI Dataset Code URL bioRxiv HAL

2025-10-16-flash-lag-effect

Thu, 16 Oct 2025 00:00:00 +0000

Mislocalization by Design
The Flash-Lag Effect as Prediction

Laurent Perrinet, CNRS/AMU, Marseille, France

[2025-10-16] Suresh Krishna’s lab meeting

Contact me @ laurent.perrinet@univ-amu.fr

Mislocalization by Design: The Flash-Lag Effect as Prediction »

Why do we sometimes misjudge where visual objects are? This talk explores how predictive processing may cause systematic perceptual mislocalizations. Indeed, the early visual system doesn’t passively process information—it actively predicts the world, compensating for neural delays by extrapolating motion trajectories. Using a Bayesian computational model, I show how this predictive mechanism explains the flash-lag effect: moving objects appear ahead of flashed ones because the brain forecasts their current position while the unpredictable flash cannot be anticipated. This framework reveals that mislocalization isn’t a bug but a feature of efficient visual coding. I’ll discuss how these principles illuminate both biological vision and artificial visual system design, demonstrating that what we perceive as “now” is actually the brain’s best prediction of the present.

Timing in the visual pathways

Ultra-rapid visual processing (see review).

Compensating visual delays (Perrinet, Adams & Friston 2014).

Compensating visual delays (Perrinet Adams & Friston, 2014).

Suppressive travelling waves (Chemla et al, 2019).

Predictive processing

Motion-based prediction (Perrinet et al, 2012).

Flash-lag effect

Flash-lag effect (Khoei et al, 2017).

Diagonal Markov model (Khoei et al, 2017).

Flash-lag effect: MBP (Khoei et al, 2017).

Flash-lag effect (Khoei et al, 2017).

Space-time probability distributions (Khoei et al, 2017).

Motion reversal (Khoei et al, 2017).

Motion reversal (smoothed) (Khoei et al, 2017).

Space-time probability distributions (Khoei et al, 2017).

Limit cycles (Khoei et al, 2017).

Mislocalization by Design
The Flash-Lag Effect as Prediction

Laurent Perrinet, CNRS/AMU, Marseille, France

[2025-10-16] Suresh Krishna’s lab meeting

Contact me @ laurent.perrinet@univ-amu.fr

Mislocalization by Design: The Flash-Lag Effect as Prediction »

Mislocalization by Design: The Flash-Lag Effect as Prediction

Thu, 16 Oct 2025 00:00:00 +0000

DynTex: A Real-Time Generative Model of Dynamic Naturalistic Luminance Textures

Mon, 01 Sep 2025 00:00:00 +0000

🚀 Excited to share our new paper:

“DynTex: A real-time generative model of dynamic naturalistic luminance textures”

…now published in Journal of Vision!

🔹 Why it matters: Dynamic textures (e.g., fire, water, foliage) are everywhere, but modeling them in real-time has been a challenge. DynTex bridges this gap with a biologically inspired, efficient approach.

🔹 Key innovation: A generative model that captures the spatiotemporal statistics of natural scenes while running in real-time.

🔹 Applications: Computer vision, neuroscience, VR/AR, and more.📖

Read it here: https://doi.org/10.1167/jov.25.11.2

More on: https://laurentperrinet.github.io/publication/meso-25/

#DynamicTextures #ComputationalNeuroscience #ComputerVision #GenerativeModels #OpenScience

linkedin, mastodon, bluesky
The Motion Clouds stimuli were originally presented in the following paper (page links to other sresources)
Paula Sanz Leon, Ivo Vanzetta, Guillaume S Masson, Laurent U Perrinet (2012). Motion Clouds: Model-based stimulus synthesis of natural-like random textures for the study of motion perception. Journal of Neurophysiology.
PDF Cite DOI HAL URL arXiv Supp
examples of use: https://laurentperrinet.github.io/sciblog/categories/motionclouds.html

Artificial neural networks and machine learning applied to the understanding of biological vision

Mon, 26 May 2025 14:00:00 +0000

Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research.

2025-05-26-master-m-4-nc

Mon, 26 May 2025 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2025-05-26] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

[Serre and Poggio, 2007]

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Hybrid IA models

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2025-05-26] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

2025-03-11-phd-program-sparse-representations

Tue, 11 Mar 2025 00:00:00 +0000

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2025-03-11]

Code / Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations?

Paysage catalan (Le Chasseur)

to rephrase the expression “The Unreasonable Effectiveness of Mathematics” by Wigner, the “Unreasonable efficiency of vision” is playfully illustrated in this painting from Joan Miró, which allows us to depict this Catalan landscape with the a few strokes where our imagination will fill the gaps and signify the landscape, allowing us to imagine the hunter, the sardine or the plane.

the whole is the sum of a few parts

Sparse coding is a technique used in signal processing and machine learning to represent data in a more concise and efficient manner. It aims to find a sparse representation of the data, which means representing the data with only a small number of non-zero coefficients or activations. In sparse coding, a set of basis functions or atoms is typically defined, and the goal is to find a linear combination of these atoms that best represents the input data. The coefficients of this linear combination are often constrained to be sparse, meaning that only a few of them are allowed to be non-zero.

Sparse representations in computer vision

Sparse representations in computer vision

Sparse representations in computer vision

[LP and Bednar, 2015]

Sparse representations in computer vision

[LP, 2021]

Sparse representations in neuromorphic engineering

Ultimately, we get a list of events for each pixel that can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence, and polarities. Note that since events are generated over time, they are naturally sorted by their time of occurrence. These events are then transmitted in real time to the output bus, often via a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve that connects our retina to the brain. In fact, the output of the retina consists of a million ganglion cells that emit action potentials, which are the only source of information transmitted by the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Sparse representations in neuroscience

[Brunel, 2001]

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations?

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations in a nutshell

Neural implementation = gradient descent

LASSO = least absolute shrinkage and selection operator

Orthogonal Matching Pursuit (OMP): OMP is an iterative algorithm used for sparse signal recovery. It starts with an initial sparse solution and iteratively selects the most correlated dictionary atoms with the residual signal. OMP aims to minimize the L2 norm of the residual while maintaining sparsity. It has a greedy nature and can provide a near-optimal sparse solution.

Basis Pursuit (BP): Basis Pursuit is an optimization problem that seeks the sparsest solution to an underdetermined linear system of equations. It involves minimizing the L1 norm of the coefficient vector subject to a linear constraint. BP can be solved using linear programming techniques or convex optimization algorithms.

Iterative Soft Thresholding Algorithm (ISTA): ISTA is an iterative optimization algorithm commonly used in sparse coding. It alternates between a gradient descent step and a soft thresholding step. The gradient descent step minimizes the data fidelity term, and the soft thresholding step enforces sparsity by setting small coefficients to zero. ISTA converges to a sparse solution and can be used for dictionary learning.

FISTA (Fast Iterative Shrinkage-Thresholding Algorithm): FISTA is an accelerated version of ISTA that improves convergence speed. It incorporates momentum into the optimization process and achieves faster convergence rates.

ADMM (Alternating Direction Method of Multipliers): ADMM is an optimization technique that decomposes the original problem into smaller subproblems and solves them iteratively. It is often used for convex optimization problems with L1 regularization. ADMM has been applied to solve sparse coding problems efficiently.

Matching pursuit algorithm

Init : Residual $R = I$, sparse vector $a$ such that $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- compute $c[i] = \sum_{x, y} (R[x, y] - a[i] \cdot \phi[i, x, y])^2$
- Match: $i^\ast = \arg \min_i c[i]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} ( I[x, y] \cdot \phi[i, x, y])$
- Assign : $a[i^\ast] = \frac{\sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]}{\sum_{x, y} \phi[i^\ast, x, y] \cdot \phi[i^\ast, x, y]}$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, and normalize $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$
- Pursuit : $R[x, y] \leftarrow R[x, y] - a[i^\ast] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
compute $c[i] = \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
compute $X[i, j] = \sum_{x, y} \phi[i, x, y] \cdot \phi[j, x, y]$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i c[i]$
- Assign : $a[i^\ast] = c[i^\ast]$
- Pursuit : $c[i] \leftarrow c[i] - a[i^\ast] \cdot X[i, i^\ast] $

[LP (2004)]

Matching pursuit algorithm

Matching pursuit algorithm

Hebbian learning (once the sparse code is known):

$$ \phi_{i}[x, y] \leftarrow \phi_{i}[x, y] + \eta \cdot a[i] \cdot (I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi_{i}[x, y] ) $$

Matching pursuit algorithm

Sparse representations in a nutshell

Convolutional Sparse Coding

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolution: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolution: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolution: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolution: Mathematics

Convolution: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolution: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Convolutional Sparse Coding

Convolutional Sparse Coding

Convolutional Sparse Coding

[LP, 2015]

Code @ SparseEdges

Convolutional Sparse Coding

[Ladret et al, 2024]

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2025-03-11]

Code / Contact me @ laurent.perrinet@univ-amu.fr

2025-02-14-supaero

Fri, 14 Feb 2025 00:00:00 +0000

Qu'est-ce que les Neurosciences peuvent apporter à l'Intelligence Artificielle ?
[2025-02-14] Airbus Helicopters Laurent Perrinet ― https://laurentperrinet.github.io

Mais que fait un neuroscientifique à Airbus Helictopters?

Je suis moi-même un passionné d’aéronautique et de spatial, ce qui m’a amené à suivre l’école d’aéronautique SUPAERO. Puis vers l’imagerie satellitaire, qui dépendait déjà de l’IA sous la forme des réseaux de neurones. C’est à partir de là, grâce à la rencontre avec mon professeur de mathématiques Manuel Samuelides, que j’ai découvert les neurosciences computationnelles et les pouvoirs qu’elles peuvent offrir pour mieux comprendre le cerveau et pour créer de nouveaux systèmes d’intelligence artificielle. Voici un

Le jury était consistué (de gauche à droite) de Jeanny Hérault (Rapporteur), Michel Imbert (Président), Yves Burnod (Rapporteur, absent de la photo), Manuel Samuelides (Directeur de thèse) et Simon Thorpe (Co-directeur de thèse).

Depuis ce temps là, je développe des réseaux de neurones concus comme des algorithmes / processus d’optimisation numérique, que j’applique pour le traitement automatisé des images. une optique nouvelle n’est pas simplement d’utiliser l’inspiration neuro-mimétique mais de faire des aller retours avec l’expérimentation

mais d’abord quid AI ?

L’intelligence artificielle est-elle “intelligente” ?

L’intelligence artificielle (IA) est-elle “intelligente” ?

L’IA est une science multi-disciplinaire qui vise à créer des machines capables d’exécuter des tâches intelligentes, similaires à celles effectuées par l’être humain.
années 1950-1970 : approches logiques et symboliques
années 1980-2010 : machine learning (apprentissage automatique)
années 2010-2020 : deep learning
années 2020-… : la révolution des transformers

L’intelligence artificielle est-elle “intelligente” ?

Sommet de l’IA de 2025

Enjeux de l’IA embarquée : latence de réponse

Visual latencies [Grimaldi et al, 2022]

Tout d’abord, les systèmes sensoriels biologiques sont composés de séquences de traitement qui possèdent des délais de traitement. Je décris ici la chaîne de traitement d’une image visuelle, ici pour un enfant jouant à un jeu et devant cliquer sur le bon bouton, et qui illustre les différentes latences du traitement de l’information de la vision à l’action.

Si les délais dans un système embarqué sont plus rapides, il reste que les informations dans les différentes étapes de traitement peuvent être décalées et nécessitent un traitement adapté afin de répondre de la façon la plus immédiate possible. Je pense notamment à la détection d’objets en mouvement très rapide dans le cadre spatial.

Tout d’abord, la plupart de ces systèmes traitent des données statiques. Ils ignorent notamment l’aspect dynamique, comme la nécessité de pouvoir répondre à tout moment ou de compenser les délais de traitement. Dans un premier temps, je présenterai un nouveau type de caméra, inspirée du fonctionnement de la rétine et du codage neural par potentiels d’actions ou « spikes ». Ces caméras permettent de capturer l’information sous forme d’événements et nécessitent d’adapter les algorithmes de traitement de l’information, qui sont plus proches de ceux utilisés par le cerveau.

Enjeux de l’IA embarquée : budget énergétique

“L’irraisonnable efficacité de la vision”

Comment la vision a évolué… [Perrinet, 2024]

Illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Illusions visuelles

Illusions visuelles

Illusions visuelles

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Neurosciences computationnelles de la vision

Anatomie du système visuel humain

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Modèles hybrides d’IA

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

From frame-based to event-based cameras.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Les caméras événementielles présentent plusieurs propriétés qui les rendent remarquables. Tout d’abord, la précision temporelle des événements est de l’ordre de la microseconde, ce qui permet d’atteindre une cadence théorique de l’ordre du million d’images par seconde. On peut la comparer à celle d’une caméra classique, qui est de l’ordre de la centaine d’images par seconde, ou à celle d’une caméra à grande vitesse, qui peut atteindre 10 000 images par seconde. Il est difficile d’estimer la fréquence d’échantillonnage de la perception humaine, car si 25 images par seconde sont souvent suffisantes pour visionner un film, il a été démontré que l’œil humain peut distinguer des détails temporels jusqu’à la milliseconde.

Une autre caractéristique importante de ces caméras est leur capacité à détecter une très large gamme de luminosité, dépassant de loin celle des caméras conventionnelles à 120 dB (un facteur d’un million, comparé au facteur de un sur mille de l’œil humain entre la pleine lune et le soleil),

Il convient de noter que la « résolution spatiale » de ces caméras est souvent relativement modeste, de l’ordre du mégapixel. Cependant, il ne s’agit pas d’une limitation technique, mais plutôt d’une conséquence des applications technologiques dans lesquelles ces caméras sont couramment utilisées.

Par rapport aux caméras classiques, qui consomment plusieurs watts, les caméras événementielles consomment très peu d’énergie électrique, de l’ordre de 10 milliwatts, soit une consommation équivalente à celle de l’œil humain. https://en.wikipedia.org/wiki/Event_camera#Functional_description

Levier #1: Réseaux de neurones impulsionnels (SNNs)

[Grimaldi et al, 2023, Precise Spiking Motifs]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Loihi 2

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Ces caméras ne présentent que des avantages, mais alors, comment traiter cette nouvelle représentation des données ? En effet, les neurosciences montrent que les neurones ne manipulent pas des données continues (comme ceux du deep learning), mais communiquent exactement de la même manière en échangeant de brèves impulsions prototypiques, les potentiels d’action (spikes).
Notre solution : une architecture similaire au deep learning, mais chaque neurone (brique élémentaire) est un modèle simplifié de neurone biologique impulsionnel. Cependant, nous nous retrouvons avec un problème par rapport à l’établissement que nous avons réussi à résoudre théoriquement. Un avantage supplémentaire est que ce genre de calcul est actuellement développé sur des puces embarquées (comme les pixels de la caméra évanementielle).
notre architecture fonctionne ainsi directement sur cette même représentation. Un autre avantage : le « always on computing ».

Quels résultats ? Peut-on les évaluer avant d’avoir ces puces ?

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Time-to-Contact maps [Nunes et al, 2023]

2 MINUTES

Nos simulations montrent ainsi une très grande efficacité (ici pour catégoriser un type de flux optique, ce qui peut guider la navigation).
un aspect innovant de notre technologie réside dans notre capacité à utiliser autant de neurones, mais moins de connexions. Nous avons par ailleurs montré que l’efficacité restait acceptable. Par rapport à une technologie classique (en orange) qui montre une baisse rapide, nos résultats montrent une bonne efficacité avec une demi-valeur critique donnée pour un gain de 700x (noter l’axe log). C’est ce qu’on appelle le « frugal computing » et nous œuvrons maintenant à son implémentation dans un PEPR IA.
c’est une étape importante, mais on peut aller plus loin, et je vais vous présenter un deuxième levier : éviter de tout traiter pour ne traiter que ce qui est nécessaire.

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

[Kremkow et al, 2018]

Levier #2: Vision active / Active Vision

---
## Levier #2: Vision active / *Active Vision*




[Jérémie et al, 2024]

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

From: https://pressbooks.umn.edu/sensationandperception/chapter/columns-and-hypercolumns-in-v1/

Qu'est-ce que les Neurosciences peuvent apporter à l'Intelligence Artificielle ?
[2025-02-14] Airbus Helicopters Laurent Perrinet ― https://laurentperrinet.github.io

2025-02-11-neuromath

Tue, 11 Feb 2025 00:00:00 +0000

[2025-02-11] When Cortical Neurons Talk Sideways: Beyond Feedforward Visual Processing

Laurent Perrinet - https://laurentperrinet.github.io Séminaire Neuromathématiques, Collège de France

Hi, thanks for the introduction! I am Laurent Perrinet, a researcher in computational neuroscience and currently a research director at CNRS at the Institute of Neuroscience of la Timone in Marseille. Thank you for inviting me to participate in this “NeuroMathematics” seminar at the intersection of mathematics and neuroscience.

As an engineer by training, I could have pursued a career in aeronautics rather than becoming a neuroscientist. It is thanks to my mathematics professor Manuel Samuelides that I discovered the beauty of neural networks at the end of my engineering studies. This developped a curiosity, and thanks to him, I was also able to study in a mastere of cognitive sciences (now called CogMaster) in 1998. This is where I particularly want to acknowledge Jean Petitot - for his course I discovered how natural image statistics could link to principles in the central nervous system. This was a vivid revelation, and I’m grateful for his guidance in my academic path. Today’s seminar represents a return to these roots, as I’ll present my research progress since my mastere thesis on this very topic.

Today, I will address our current knowledge about horizontal connectivity rules in V1. Why is this important? As a matter of fact, one main function of sensory systems, such as the pivotal role of the primary visual cortex for vision, is to bind together the different visual features to help ultimately build a global perception.

Paysage catalan (Le Chasseur), Joan Miró (1924)

Trames (Etienne Rey)

Contour detection and the Association Field

[Field et al, 1993]

Contour detection and the Association Field

[Field et al, 1993]

Contour detection and the Association Field

[Field et al, 1993]

Natural Images : Edges are on a common circle

[Sigman et al, 2001]

A significant contribution to understanding the association field came from studying edge co-occurrences in natural images by Sigman et al. (2001). They quantified the probability density function of edge co-occurrences based on their relative positions and orientations. The figure demonstrates this by showing the spatial distribution patterns for edges relative to a reference edge at different orientations. For iso-oriented edges (a), the co-occurrence pattern shows clear structure. As the relative orientation increases through 22.5° (b), 45° (c), 67.5° (d), to 90° (e), distinct spatial patterns emerge.

A key finding was that for any given relative orientation between edges, the angle of maximal interaction occurs at the bisector between the orientations. This suggests that co-occurring edges tend to lie on a common circle - a property known as cocircularity. Panel (f) illustrates this geometrical principle: given two edges at angles w (red, 20°) and c (blue, 40°), the cocircularity solutions (green lines at 30° and 120°) represent the possible orientations of connecting circular arcs. This mathematical relationship provides insights into how the visual system might leverage statistical regularities in natural scenes for contour integration. We will go back into the details of this a bit further in the talk.

This association field concept provided a compelling framework for understanding how the visual system may implement contour integration through neural connectivity patterns. but before going there we should go back to the basic anatomy of the visual cortex.

Contour detection and the Association Field

[Field et al, 1993]

Dynamics of vision

Human Visual system (Grimaldi et al 2022)

Anatomy of the Human Visual system

Human Visual system (Grimaldi et al 2022)

Thalamic, short- & long-range lateral, inter-areal

A key feature of primary visual cortex is its layered organization, which is shared across cortical areas. The main thalamic input arrives in layer 4, which connects to a dense network of vertical connections across layers. These columns can then communicate via horizontal connections within layers.

Hubel and Wiesel also proposed the ice-cube model that every point in the visual field produces a response in a 2 mm x 2 mm area of the cortex. Such an area can contain two complete groups of ocular dominance columns, 16 blobs and interblobs that may contain more than two times all of the orientations possible across 180 degrees. This region of the cortex, which Hubel and Wiesel called a hypercolumn (or, more generally, a cortical module) seems both necessary and sufficient for analyzing the image of a point in visual space. Because the cortex is a continuous cellular layer and because it is very hard to establish the boundaries of these modules physically, their existence from a functional standpoint is still the subject of debate. https://thebrain.mcgill.ca/flash/a/a_02/a_02_cl/a_02_cl_vis/a_02_cl_vis.html

Figure 9.2. Hypercolumn Diagram. Ocular dominance columns are segregated into left and right eye inputs. Orientation columns are neurons that get excited at different orientations and a cluster of these is called a pinwheel. Blobs are color selective and for every pinwheel there is a blob. (Credit: McGill: The Brain from Top to Bottom, Figure of hypercolumns, Copyleft https://copyleft.org/, https://thebrain.mcgill.ca/flash/a/a_02/a_02_cl/a_02_cl_vis/a_02_cl_vis.html. No modifications.)

Thalamic, short- & long-range lateral, inter-areal

[Markov et al 2011]

Anatomy of the Primary Visual Cortex

[Kaschube et al (2010)]

V1 is central to these pathways and shows distinctive anatomical and functional properties along with a complex topographical organization.

This figure from Kaschube et al. (2010) illustrates the organization of orientation preference maps in primary visual cortex (V1). Individual V1 neurons exhibit selective responses to oriented visual stimuli (as denoted by varying hues Colors code preferred ORs as indicated by the bars in (C)), with their spatial arrangement following highly structured patterns across the cortical surface. Panel B shows Synthetic orientation-maps of equal column spacing Λ but widely different pinwheel densities ρ. Left to right: solutions of different models: (13–16).. (C) High (blue frame) and low (orange frame) pinwheel density regions in tree shrew visual cortex. (D to F), Optically recorded orientation-maps in tree shrew (D), galago (E), and ferret (F) visual cortex. Regions shown in (C) are marked in (D). White arrows in (F) mark selected pinwheel centers. Framed regions in (C) and (F) are magnified. In many mammals including cats, monkeys and ferrets, orientation preference is organized in a quasi-periodic manner, forming what are known as orientation preference maps. These maps show remarkable consistency in their geometric properties across species, particularly in the spatial organization of pinwheel centers where orientation preferences converge.

However, this organization shows important species-specific variations. Most notably, while primates and carnivores display orderly orientation maps with smooth transitions between preferred orientations, rodents lack such maps and instead show a “salt-and-pepper” arrangement where neighboring neurons have seemingly random orientation preferences. This organizational diversity raises interesting questions about the computational advantages of these different architectures and their relationship to visual processing requirements and behavioral needs across species.

Horizontal connectivity links different hypercolumns

[Bosking et al, 1997]

Contour detection and the Association Field

[Field et al, 1993]

The like-to-like hypothesis

[Field et al, 2013]

The resemblance between what was shown by Bosking and the structure of the association field that we saw above is such that it is tempting to align both and state that the function of horizontal connections is to bind neurons with a selectivity to similar orientations* over long distances. This like-to-like hypothesis has been influential in understanding horizontal connectivity patterns.

However, we should be cautious about overstating these relationships. While horizontal connections show some orientation specificity, recent evidence indicates the connectivity patterns are more complex and heterogeneous than initially proposed. The functional role of this diverse connectivity remains an active area of investigation.

During the remainder of this talk, I will try to shed light on our current knowledege on horizontal connectivities.

Supplementary: the HMAX model

[Serre and Poggio, 2007]

Supplementary: Convolutional Neural Nets (CNN)

[Chavane, LP and Rankin, 2022]

Supplementary: Orientation selectivity in V1

[Hubel & Wiesel, 1962]

Supplementary: Orientation selectivity in V1

[Hubel & Wiesel, 1962]

Supplementary: Marr’s three levels of analysis

Marr, 1982

cut in different levels: Marr (+ Poggio)
arbitrary, but useful division of labor= computational / algorithm / hardware
here:
anatomy
algorithm / model
function

First: What is the anatomy of horizontal connections?

<!--

[Marr, 1982]]

-->

Challenging the like-to-like hypothesis

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

This figure illustrates different hypothetical connectivity rules for horizontal connections in V1. The target neuron (large circle on left) has a specific orientation preference indicated by its color. Following the classical like-to-like hypothesis (shown in panel A), this neuron would preferentially connect to other neurons with matching orientation preference (similar colors) across multiple hypercolumns, as indicated by the vertical red arrows. The radial spread of connections spans approximately three hypercolumns, consistent with anatomical observations. Each hypercolumn contains a complete set of orientation preferences, represented by the different colored neurons.

This first schematic (noted A) represents one of the like-to-like connectivity rules, where horizontal connections strictly follow orientation similarity.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Let’s first shows some functional evidence.

This video shows voltage-sensitive dye imaging (VSDI) data from cat primary visual cortex (area 17) in response to a local oriented grating stimulus. The visualization reveals two key aspects:

The broader activation pattern shown by overall fluorescence changes (gray)
The more restricted orientation-selective response pattern (colored regions)

Two contours are overlaid: a red line marking the boundary of significant activation, and a white line delineating regions with statistically significant orientation selectivity. The orientation selectivity is encoded by color hue.

The bottom plots quantify the spatiotemporal dynamics by showing:

Left: The total activated cortical area over time
Right: The extent of orientation-selective regions over time

Together, these measurements demonstrate how orientation-selective signals propagate laterally beyond the classical feedforward input zone through horizontal connections, while maintaining some degree of feature selectivity.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

This figure shows spatial and temporal dynamics of orientation selectivity in cat V1 analyzed from voltage-sensitive dye imaging data. Panel A displays a cortical orientation map averaged over the final 145ms of the response, where hue indicates preferred orientation and brightness shows orientation tuning strength. The dotted red line delineates the expected retinotopic boundary of feedforward input based on Albus (2004).

The inset quantitatively compares the spatial extent of:

Total cortical activation (grey contour)
Orientation-selective activation (black contour)
Theoretical feedforward input boundary (red contour)

This data demonstrates that orientation-selective responses propagate laterally beyond the classical feedforward input zone through horizontal connections, while maintaining some degree of feature selectivity. The systematic comparison between total activation and selective activation provides direct evidence for how horizontal connectivity shapes the spatiotemporal dynamics of orientation processing in V1.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel C displays intracellular recordings of subthreshold responses visualized as a visuotopic orientation polar map. The color hue represents preferred orientation while brightness indicates the strength of orientation tuning in the membrane potential. White contours outline regions showing statistically significant responses based on both amplitude and orientation selectivity criteria. The middle plots show averaged subthreshold responses to four different oriented stimuli (color-coded) at specific recording locations (marked by circle, triangle and square symbols), with scale bars indicating 50 ms and 1 mV. On the right, normalized orientation tuning curves are shown, computed by integrating responses within a fixed temporal window (shaded region in middle panel). The black circle marks the spontaneous activity level for the depolarizing integral measurement.

These shows a direct functional evidence for a diversity of tuning profile in th horizontal connectivity.

Challenging the like-to-like hypothesis

[Voges and LP, 2012]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Figure 4 illustrates a neural field model that bridges anatomical structure with functional observations in V1, as developed by Rankin and Chavane (2017).

Panel A depicts radial connectivity profiles with Gaussian-decaying inhibition and distance-dependent excitation that peaks periodically at multiples of distance L. The Ring Width (RW) parameter controls the spread of these excitatory peaks.

Panel B shows how local orientation preference maps influence lateral connectivity patterns under different orientation bias (BR) values in the recurrent connections.

Panel C quantifies the orientation tuning that emerges from these connectivity patterns. While orientations are uniformly represented globally, the local excitatory component shows strong bias around -60°. As BR increases above 0.5, the lateral connection orientation bias strengthens, reaching values around k=1 (consistent with Buzás et al. 2006).

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel F maps the normalized selective area (relative to the feedforward footprint) across Ring Width (RW) and orientation bias (BR) parameters. White contours delineate anatomically plausible ranges where k values fall between 0.7-1.2, consistent with experimental measurements. The green region indicates parameter combinations that additionally satisfy constraints on both orientation preference and the observed radial decay of selectivity.

The neural field model effectively connects anatomical connectivity patterns with functional observations of orientation selectivity propagation in V1. The resulting connectivity structure exhibits similarities with “association field” patterns, suggesting potential optimization for encoding natural image statistics. This framework provides a quantitative basis for investigating computational principles underlying horizontal connectivity in visual cortex.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Modelling the Association field

[Field et al, 2013]

Edge co-occurences in natural images

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel A illustrates the groundbreaking approach developed by Geisler et al. (2001) for analyzing edge statistics in natural images. The method involves:

Detecting oriented edge elements in natural images (shown as red segments)
For each edge pair, measuring:
- Their relative orientation difference (𝜃)
- The relative position angle (𝜙)
- Center-to-center distance (d)
- Azimuth difference (φ)
- Co-circularity parameter ψ = φ - θ/2

This quantitative analysis reveals two key distributions:

A strong bias for parallel edge arrangements, evident in the orientation difference histogram
A marked preference for co-circular alignments, shown in the relative position histogram

These statistics vary significantly across image databases. For example, images containing animals exhibit enhanced co-circularity compared to general natural scenes. This suggests that rather than implementing a single fixed association field, the visual system may need to handle diverse statistical regularities present in natural inputs.

The next section will examine how these statistical regularities inform computational models of the association field.

Edge co-occurences in natural images

[Geisler, 2001]

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

The probability distribution function p(ψ,θ) represents the distribution of the different geometrical arrangements of edges’ angles, which we call a “chevron map”. We show here the histogram for non-animal natural images, illustrating the preference for co-linear edge configurations. For each chevron configuration, deeper and deeper red circles indicate configurations that are more and more likely with respect to a uniform prior, with an average maximum of about 3 times more likely, and deeper and deeper blue circles indicate configurations less likely than a flat prior (with a minimum of about 0.8 times as likely). Conveniently, this “chevron map” shows in one graph that non-animal natural images have on average a preference for co-linear and parallel edges, (the horizontal middle axis) and orthogonal angles (the top and bottom rows), along with a slight preference for co-circular configurations (for ψ =0 and ψ = ± π/2, just above and below the central row).

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

This figure shows classification performance across image categories using different statistical features. We used an SVM classifier with three feature sets: first-order orientation statistics (FO), the reduced 2D “chevron map” (CM), and full 4D second-order statistics (SO). The classification accuracy (F1 score) was tested for distinguishing between image categories. Results show strong performance in separating man-made from natural images, as expected. More notably, the classifier achieved ~80% accuracy in discriminating animal vs non-animal natural images, matching human performance levels reported by Serre et al. This suggests that relatively simple edge co-occurrence statistics contain sufficient information for basic image categorization tasks, without requiring higher-level semantic processing.

We also found that our model made the same errors as humans do: if an image without an animal contains more co-circular edges, it is more likely to be falsely categorized as containing an animal.

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

Can we explain the diversity ?

[Bosking et al, 1997]

Indeed, this diversity is revealed in the anatomical data: V1 horizontal connectivity exhibits more complexity than suggested by the classical like-to-like hypothesis. While orientation-specific connections exist, they coexist with non-selective connections that link neurons irrespective of their tuning preferences. This diversity likely serves multiple computational functions:

Specific connections could support contour integration and feature binding
Non-selective connections may enable broad contextual modulation
Mixed connectivity patterns could help maintain network stability while preserving functional specificity

This anatomical heterogeneity aligns with V1’s role in both specialized feature detection and broader contextual processing. Understanding how these distinct connectivity patterns interact remains an active area of research in visual neuroscience.

Predictive processing

Predictive processing

To bridge the gap between anatomical observations and functional requirements of visual processing, We added two key ingredients in the sparse deep predictive coding (SDPC) model :

Sparse connectivity patterns:
- Enforcing regularization of the activity map using L1 penalty
- Activity computed via recurrent local connectivity
- Similar to biological observations
Feedback from efferent layers:
- Predicts activity of afferent layer
- Only residual prediction error is processed
- Defines long-range inter-areal connectivity
- Specific influence demonstrated in Neural Computation paper

By defining a cost on minimizing the prediction error in each layer, everything stays derivable, such that we can use a classical gradient descent. These additions should allow us to better understand how feedback shapes visual processing in biological neural networks.

Predictive processing

Predictive processing

More specifically in the context of our focus today, we can look at the co-occurence

llustration of the procedure to generate interaction map. In this illustrative example we consider a V1 representation with only 4 feature maps (represented in the upper-left box). Step 1 is to extract a neighborhood (of size 3x3 in the illustration only) around the most strongly activated neuron (represented with a red square in the illustration) for a given central preferred orientation (denoted ✓ c ). Step 2 is to normalize the neural activity in the extracted neighborhood using the marginal activity (see Eq.8). Step 3 is to compute the resulting orientation and activity at every position of the neighborhood using a circular mean (see Eq. 11 and Eq. 12 respectively). To keep a concise figure we have illustrated the computation of the central edge of the interaction map only. For simplification, the illustration shows only 1 neighborhood extraction whereas the interaction maps shown in the paper are computed by averaging neighborhoods centered on the 10 most strongly activated neurons

Predictive processing

Predictive processing

Predictive processing with pooling

Challenging the like-to-like hypothesis

[Bosking et al, 1997]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

To conclude, our review of horizontal connectivity in V1 reveals patterns more complex than initially theorized. The classical like-to-like hypothesis, while valuable, doesn’t fully capture the diversity of observed connectivity patterns.

Mathematical modeling has proven essential in bridging theory and biology. Our predictive processing framework shows how simple computational principles can explain the emergence of these complex connectivity patterns. The model demonstrates how feedback influences lateral interactions and reproduces key experimental observations.

However, important questions remain unanswered. We need to better understand how precise timing information is encoded in these circuits, how temporal dynamics shape processing, and whether similar principles apply across other cortical areas.

These fundamental questions will guide future experimental and theoretical work as we continue to unravel the computational principles of cortical processing.

[2025-02-11] When Cortical Neurons Talk Sideways: Beyond Feedforward Visual Processing

Laurent Perrinet - https://laurentperrinet.github.io Séminaire Neuromathématiques, Collège de France

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Dynamics of vision: Neural modeling

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

One advantage of this network is that it is differentiable, enabling us to apply classical machine learning methods, notably supervised learning. We then see the emergence of different convolution kernels, and here I represent a subset of its kernels for different directions, as denoted by the red arrows on the left of the graph. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Population decoding of visual motion direction in V1 marmoset monkey : effects of uncertainty

Wed, 01 Jan 2025 00:00:00 +0000

🧠 Excited to share our latest research led by Alexandre Lainé and presented this summer at CNS2025 in beautiful Firenze, Italy!

“Population decoding of visual motion direction in V1 marmoset monkey: effects of uncertainty”

Advanced population-level analysis of V1 neural responses to motion stimuli
Novel insights into how the brain handles uncertainty in visual motion processing
Marmoset model providing crucial translational insights for visual neuroscience

This research contributes to our fundamental understanding of how the visual system processes motion information at the earliest stages of cortical processing, with important implications for both basic neuroscience and potential clinical applications. Thank you to the CNS organizing committee for hosting such an inspiring conference in the stunning venue of Palazzo dei Congressi in Villa Vittoria! 🇮🇹

#ComputationalNeuroscience #VisualNeuroscience #MotionProcessing #CNS2025 #Neuroscience #Research #MarmosetModel #V1 #PopulationDecoding

Link to publication: https://laurentperrinet.github.io/publication/laine-25-cns/

For deeper insights into uncertainty processing mechanisms in the visual cortex, see our Nature Communications Biology study:
Hugo Ladret, Nelson Cortes, Lamyae Ikan, Frédéric Chavane, Christian Casanova, Laurent U Perrinet (2023). Cortical recurrence supports resilience to sensory variance in the primary visual cortex. Nature Communications Biology.
PDF Cite DOI Dataset Code URL bioRxiv HAL
see a follow-up in:
Alexandre Lainé, Nicholas J. Priebe, Guillaume S. Masson, Laurent U Perrinet (2026). Population decoding of visual motion direction. Proceedings of AREADNE.
PDF Cite Poster LinkedIn
https://neuromatch.social/@laurentperrinet/115050564011598328
https://www.linkedin.com/posts/laurent-perrinet-1857b9_population-decoding-of-visual-motion-direction-activity-7363238280143745026-zPkg
https://bsky.app/profile/laurentperrinet.bsky.social/post/3lwowjtpbw22a

An open-source vision-science tool for the auto-regressive generation of dynamic stochastic textures Motion Clouds

Tue, 27 Aug 2024 00:00:00 +0000

Soutenance de thèse Antoine Grimaldi

Wed, 15 May 2024 13:00:00 +0000

“Neural computations with precise spiking motifs for dynamic vision” Soutenance de thèse Antoine Grimaldi

Date : Jeudi 16 mai à 15h (CEST)
https://neuromatch.social/@laurentperrinet/112454990080998095

Jury

Président : Martin Vinck
Rapporteure : Barbara Webb
Examinateur : Dan Goodman
Examinateur : Andrea Alamia
Examinatrice : Sonja Grün
Examinatrice: Sophie Denève
Directeur de thèse : Laurent Perrinet
Codirecteur de thèse : Jean Martinet

Abstract

Our brains are extremely efficient at solving highly complex visual tasks. In a few hundred milliseconds, we are able to recognise different objects invariant to various characteristics, such as their size or orientation. Recently, artificial neural networks have made great strides in solving the tasks faced by biological systems. They draw on knowledge from neuroscience to form biologically realistic learning architectures that could provide us with interesting insights into how the human brain works. But these architectures still face a number of challenges: the models are not always interpretable, they do not necessarily seem to use the same strategies as their biological equivalents and they are very energy-intensive. We believe that one of the reasons why the visual system is so efficient is that it uses short pulses to represent information: the action potentials, or spikes, emitted by neurons.

Using a neuromorphic approach, the aim of this thesis project is to develop visual information processing models using representations based on spikes, binary events described only by their time and origin. We have chosen to use a dynamic signal, cap- tured by an event-based camera, which transcribes a visual scene using only events, or spikes. We solve visual cognitive tasks using the temporal code formed by precise sequences of events that we call spiking motifs. A large body of experimental evidence suggests that the temporal code carried by these patterns is a strategy used by the brain to encode visual information. We will see that the use of these patterns makes it possible to develop local and biologically realistic learning methods while dynamically and asynchronously processing the events characterising a visual scene. We show that these algorithms can solve an object recognition task and a motion estimation task ultra-fast and efficiently. We also observe the emergence of an organisation of recep- tive fields similar to that of biological systems, suggesting that a similar strategy may be employed by the brain. In the final part of this work, we will detail the development of a new algorithm for detecting this type of activity in recordings of real neurons.

Résumé

Notre cerveau est extrêmement efficace pour résoudre des tâches visuelles très complexes. En quelques centaines de millisecondes, nous sommes capables de reconnaître différents objets de manière invariante à diverses caractéristiques, telles que leur taille ou leur orientation. Récemment, les réseaux neuronaux artificiels ont fait de grands progrès dans la résolution des tâches auxquelles sont confrontés les systèmes biologiques. Ils s’appuient sur les connaissances des neurosciences pour former des architectures d’apprentissage biologiquement réalistes qui pourraient nous fournir des informations intéressantes sur le fonctionnement du cerveau humain. Mais ces architectures sont encore confrontées à un certain nombre de défis : les modèles ne sont pas toujours interprétables, ils ne semblent pas nécessairement utiliser les mêmes stratégies que leurs équivalents biologiques et ils sont très gourmands en énergie. Nous pensons qu’une des raisons de la grande efficacité du système visuel est qu’il utilise des impulsions courtes pour représenter l’information : les potentiels d’action émis par les neurones. En utilisant une approche neuromorphique, l’objectif de ce projet de thèse est de développer des modèles de traitement de l’information visuelle utilisant des représentations basées sur ces impulsions, événements binaires décrits uniquement par leur temps et leur origine. Nous avons choisi d’utiliser un signal dynamique, capturé par une caméra événementielle, qui transcrit une scène visuelle en utilisant uniquement des événements, ou impulsions. Nous résolvons des tâches cognitives visuelles en utilisant le code temporel formé par des séquences précises d’événements que nous appelons motifs d’impulsions. De nombreuses preuves expérimentales suggèrent que le code temporel porté par ces motifs serait une stratégie d’encodage de l’information visuelle utilisée par le cerveau. Nous verrons que l’utilisation de ces motifs permet de développer des méthodes d’apprentissage locales et biologiquement réalistes tout en traitant de manière dynamique et asynchrone les événements caractérisant une scène visuelle. Nous montrons que ces algorithmes permettent de résoudre une tâche de reconnaissance d’objet et une tâche d’estimation de mouvement de manière ultra-rapide et efficace. Nous observons également l’émergence d’une organisation des champs récepteurs similaire à celle des systèmes biologiques, ce qui suggère qu’une stratégie similaire peut être employée par le cerveau. Dans la dernière partie de ce travail, nous détaillerons le développement d’un nouvel algorithme pour détecter ce type d’activité dans des enregistrements de neurones réels.

Artificial neural networks and machine learning applied to the understanding of biological vision

Mon, 13 May 2024 14:00:00 +0000

2024-05-13-master-m-4-nc

Mon, 13 May 2024 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2024-05-13]Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2024-05-13] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

2024-04-10-ue-neurosciences-computationnelles

Wed, 10 Apr 2024 00:00:00 +0000

Artificial neural networks applied to the understanding of biological vision

Laurent Perrinet

[2024-04-10] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

appear bent

effect of context -> 3D

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks applied to the understanding of biological vision

Laurent Perrinet

[2024-04-10] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

2024-04-17-phd-program-sparse-representations

Wed, 10 Apr 2024 00:00:00 +0000

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2024-04-17]

Code / Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations?

Sparse representations in computer vision

Sparse representations in computer vision

[LP and Bednar, 2015]

Sparse representations in computer vision

[LP, 2021]

Sparse representations in neuromorphic engineering

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Sparse representations in neuromorphic engineering

The HD-SNN neural network.

Sparse representations in neuromorphic engineering

The HD-SNN neural network.

Sparse representations in neuroscience

[Brunel, 2001]

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations?

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations in a nutshell

Neural implementation = gradient descent

LASSO = least absolute shrinkage and selection operator

Matching pursuit algorithm

Init : Residual $R = I$, sparse vector $a$ such that $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- compute $c[i] = \sum_{x, y} (R[x, y] - a[i] \cdot \phi[i, x, y])^2$
- Match: $i^\ast = \arg \min_i c[i]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} ( I[x, y] \cdot \phi[i, x, y])$
- Assign : $a[i^\ast] = \frac{\sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]}{\sum_{x, y} \phi[i^\ast, x, y] \cdot \phi[i^\ast, x, y]}$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, and normalize $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$
- Pursuit : $R[x, y] \leftarrow R[x, y] - a[i^\ast] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
compute $c[i] = \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
compute $X[i, j] = \sum_{x, y} \phi[i, x, y] \cdot \phi[j, x, y]$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i c[i]$
- Assign : $a[i^\ast] = c[i^\ast]$
- Pursuit : $c[i] \leftarrow c[i] - a[i^\ast] \cdot X[i, i^\ast] $

[LP (2004)]

Matching pursuit algorithm

Matching pursuit algorithm

Hebbian learning (once the sparse code is known):

$$ \phi_{i}[x, y] \leftarrow \phi_{i}[x, y] + \eta \cdot a[i] \cdot (I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi_{i}[x, y] ) $$

Matching pursuit algorithm

Sparse representations in a nutshell

Convolutional Sparse Coding

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolution: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolution: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolution: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolution: Mathematics

Convolution: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolution: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Convolutional Sparse Coding

Convolutional Sparse Coding

Convolutional Sparse Coding

[LP, 2015]

Code @ SparseEdges

Convolutional Sparse Coding

[Ladret et al, 2024]

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2024-04-17]

Code / Contact me @ laurent.perrinet@univ-amu.fr

2024-03-27-emergences.md

Wed, 27 Mar 2024 00:00:00 +0000

Analyser de larges volumes de données neurobiologiques

Laurent Perrinet

[2024-03-27] Emergences workshop, Autrans, France

laurent.perrinet@univ-amu.fr

Hello, can you hear me in the back? First of all, I’d like to thank the organizers for this opportunity and all of you for coming.

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and I’m a computational neuroscientist interested in large-scale models of vision.

Alors que ce projet vient juste de commencer, je voudrais déjà parler de quelques idées pour l’avenir. En effet, la question peut se poser quant aux applications futures des puces neuromorphiques qui vont être développées dans le cadre du projet “Emergences”. pour ce développement technologique, on va souvent penser à des applications technologiques, comme les voitures autonome ou la vision robotique. Mais il y a aussi des applications qui peuvent viser à la compréhension du fonctionnement du cerveau et de la cognition en général. Et ceci passe par une meilleure connaissance de la façon dont celle-ci est contenues dans l’activité neurale.

If you wish to go further, these slides along with a number of references and useful links are available on my website.

Techniques d’enregistrement de données neurobiologiques

Enregistrement extracellulaire

[Hubel & Wiesel, 1962]

Même si ce ne sont pas les premiers à avoir enregistré l’activité électrique de neurones (ce sont physiologistes allemands Emil du Bois-Reymond et Hermann von Helmholtz au milieu du 19e siècle), David Hubel et Torsten Wiesel ont marqué leur époque. En 1962, ils ont mené des expériences révolutionnaires qui ont permis de comprendre les mécanismes de base de la perception visuelle et ont jeté les bases de la compréhension de l’organisation fonctionnelle du cortex visuel. Leur travail a valu à Hubel et Wiesel le prix Nobel de physiologie ou médecine en 1981.

La technique principale utilisée par Hubel et Wiesel dans leurs expériences était la microélectrode d’enregistrement extracellulaire. Ils ont inséré de fines électrodes dans le cortex visuel primaire (aussi appelé cortex strié) de chats et de singes anesthésiés. Ces électrodes leur ont permis d’enregistrer l’activité électrique des neurones individuels lors de la présentation de stimuli visuels.

Aire visuelle primaire

Enregistrement extracellulaire

[Hubel & Wiesel, 1962]

Hubel et Wiesel ont utilisé une variété de stimuli visuels, tels que des lignes, des barres, des points lumineux et des motifs en mouvement, qu’ils ont présentés à des animaux dans des conditions contrôlées. En enregistrant les réponses des neurones visuels, ils ont pu observer des motifs caractéristiques d’activité neuronale en fonction des propriétés visuelles des stimuli.

Leur travail a révélé l’existence de neurones spécifiques, appelés neurones simples et neurones complexes, qui répondent de manière sélective à des caractéristiques visuelles spécifiques, telles que l’orientation, la direction du mouvement et la taille des stimuli. Ils ont également découvert que ces neurones étaient organisés de manière hiérarchique, avec des neurones simples détectant des caractéristiques visuelles élémentaires et des neurones complexes intégrant ces informations pour former des représentations plus complexes.

mais aussi: sharp electrodes, patch-clamp

Multi-électrodes

[Microelectrode array (MEAs)]

Différentes échelles

[Chemla et al, 2017]

Vers des données massives

[Stevenson and Kording, 2011]

Vers des données massives

[Steinmetz et al, 2017]

Techniques d’analyse des données neurobiologiques

Méthodes statistiques

Méthodes statistiques

Méthodes statistiques

[Churchland & Cunningham et al. (2012)]

… et au-delà!

… et au-delà: le décodage

… et au-delà: le décodage

… et au-delà: le décodage

Brain-Computer Interface (BCI)

[Interface neuronale directe (BCI)]

Perspectives et opportunités du neuromorphique

Exploitation d’un timing précis

Exploitation d’un timing précis

Exploitation d’un timing précis

Exploitation d’un timing précis

Codage par latence

Codage par latence

[Thorpe (2001)]

Latences et rapidité

Visual latencies (see review).

Algorithmes neuromorphiques

Always-on classification using HOTS

Always-on classification using HOTS

Always-on classification using HOTS

Spiking motifs in vision

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

http://4.bp.blogspot.com/-AHprBxkfu5o/UJ-lqR7GsmI/AAAAAAAAHpo/VJzY7HMuXe0/s1600/The+Horse+in+Motion,+1878.%C2%A0Eadweard+Muybridge+(b.+9+April,+1830)The+first+movie+ever+made,+from+still+photographs..gif https://upload.wikimedia.org/wikipedia/commons/0/07/The_Horse_in_Motion-anim.gif

Spiking motifs pour la bio (HD-SNN)

LP (2023)

Future steps

unsupervised

high-throughput

real-time

Analyser de larges volumes de données neurobiologiques

Laurent Perrinet

[2024-03-27] Emergences workshop, Autrans, France

laurent.perrinet@univ-amu.fr

2024-02-05-udem.md

Mon, 05 Feb 2024 00:00:00 +0000

Neuromorphic models of vision

Laurent Perrinet

[2024-02-05] Seminar at UdeM’s School of Optometry, Montréal

laurent.perrinet@univ-amu.fr

When brains meet computing machines

Hello, can you hear me in the back? First of all, I’d like to thank the organizers for this opportunity and all of you for coming.

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and I’m a computational neuroscientist interested in large-scale models of vision. During this seminar for the “groupe de recherche de la vision de l’UdeM”, I’ll focus on neuromorphic models by introducing you to event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. The outline of the talk is as follows: first, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Then we’ll explore some applications of these cameras using specific algorithms. Finally, we’ll look at how our understanding of neuroscience can improve these algorithms.

Relax, these slides along with a number of references and useful links are available on my website.

Sensing light

… and this example demonstrates that since numerous years imaging techniques have also opened the door to new scientific discoveries. For example, in the late 19th century, scientists wondered if horses lifted all four hooves off the ground when they galloped. It was too fast for the human eye to see. Eadweard Muybridge solved this mystery using chronophotography, an early form of photography that captures motion. He took a series of photographs of a horse running and showed that there are moments when all four hooves are in the air. This breakthrough helped us better understand animal movement and paved the way for modern cameras.

This technique is inspired by the research of [Etienne-Jules Marey] (https://en.wikipedia.org/wiki/Etienne-Jules_Marey), under the term chronophotography, which is the use of a rifle-like apparatus to photograph a visual scene. This technique allowed Muybridge, in particular, to scientifically demonstrate the mechanism of a horse’s gallop. The movie theater became popular only afterwards.

Representing light

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Temporal Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Cameras

An event-based camera is equipped with a sensor that converts light into an electrical current, similar to conventional CMOS sensors. However, it differs from standard frame-based cameras in that it is inspired by the human retina. There are two main differences from a frame-based camera (middle graph) that lead to an event-based representation (right graph):

First, each pixel of an event-based camera is independent, operating without a synchronized global clock.
Second, each pixel detects changes in logarithmic light intensity and generates a binary event only if the change exceeds a threshold. If the change is an increment - that is, the log intensity has increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is generated asynchronously when a pixel-level change in brightness is detected. This results in superior temporal resolution and reduced susceptibility to motion blur, making event cameras ideal for capturing fast-moving scenes.

Event-Based Cameras: DVS gesture

Event-Based Cameras

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Cameras

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-driven cameras boast several remarkable properties. Firstly, their temporal precision is in the microsecond range, allowing for a theoretical frame rate of up to a million images per second. In contrast, a conventional camera typically captures around a hundred images per second, while a high-speed camera may reach 10,000 images per second. Estimating the sampling frequency of human perception is challenging; although 25 frames per second usually suffice for movies, the human eye can discern temporal details at rates between 300 and 1,000 frames per second. It’s also noteworthy that the spatial resolution of event cameras is generally modest, often in the megapixel range. This is not due to technical constraints but rather reflects the cameras’ common technological applications. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another key feature is their ability to detect a very wide range of luminosity, reaching 120 dB, which is a million times greater than conventional cameras and thousand times greater than an human eye.

Event-Based Cameras

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition —for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror different temporal sequence of events, termed the temporal surface which provides with a representation of recent activity. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at about 8.3%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis —an important concept from neuroscience— we enhanced the algorithm’s performance to 82%. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons. This underscores the value of incorporating neuroscientific principles into machine learning.

Furthermore, we leveraged a key trait of biological systems: the ability to process information continuously, in real time. Traditional algorithms wait to classify until all events are processed. We innovated by enabling our algorithm to classify on-the-fly, in real-time, with each incoming event. This means that as events occur, they’re instantly processed through the layers, reaching the classification layer without delay.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of our algorithm’s average performance relative to the dataset and the number of processed events. The blue curve reveals that with fewer than 10 events, performance hovers at chance levels. However, as more events are processed, we observe a steady improvement. Remarkably, with 10,000 events, performance matches that of the original algorithm and further excels with an additional tenfold increase in events. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in real-time, at any point during the event stream —not just after the entire signal is processed. Online processing is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the time to wait for the video sequence to finish processing before making the right decision, which is to flee. We’ve also refined our algorithm to select classification events based on precision calculations for each event. By adding a precision threshold, we achieve high performance with merely a hundred events. This reflects a biological network trait where decisions aren’t made incrementally but rather emerge abruptly here after 200 events — and then continue to improve and stabilize.

Spiking Neural Networks

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE Spiking neural networks show great potential for processing data from event-driven cameras. However, neurophysiology studies reveal some unexpected behaviors, very different from the classical perceptron. I will highlight these differences with three examples. The first example is a 1995 study by Mainen and Sejnowski examined a neuron’s reaction to repeated stimulations. Panel A at the top presents the neuron’s response to multiple stimulations with a 200 picoampere current step. The membrane potential varied across trials, indicating an unpredictable response. Initially, the spikes were synchronized at the onset of stimulation, but coherence diminished over time, leading to no alignment after approximately 750 milliseconds. In contrast, Panel B at the botton shows the neuron’s response to stimulation with noise. Here, the neuron exhibited highly consistent responses across trials, with membrane potential traces nearly identical. This precision was achieved using frozen noise, a repeated, unchanging stimulus. The study highlights that neurons are less responsive to constant analog values, such as square pulses, and more selective to dynamic signals, responding with remarkable precision in the temporal domain.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

Neuromorphic models of vision

Laurent Perrinet

[2024-02-05] Seminar at UdeM’s School of Optometry, Montréal

laurent.perrinet@univ-amu.fr

In conclusion, we have seen that event-driven cameras open the door to new applications that mimic the performance of the human eye, in terms of computational dynamics, adaptation to light conditions and energy constraints. This technological development has recently been accompanied by the development of neuromorphic chips and innovative algorithms in the form of spiking neural networks. However, there is still a great deal of progress to be made at theoretical level, particularly in the understanding of these spiking neural networks, and we have shown the potential progress that can be made by exploiting the richness of temporal representations, particularly by taking advantage of heterogeneous delays. Beyond these particular applications to natural image processing, I hope to have succeeded in demonstrating the importance of cross-fertilizing the field of engineering applications in general with biological neuroscience. This new line of research - known as NeuroAI or, more generally, as computational neuroscience - is likely to develop over the next few years. Thank you for your attention.

To conclude, we’ve explored how event-driven cameras pave the way for new applications. These applications mirror the human eye’s performance in terms of computational dynamics, rapid light condition adaptation, and energy efficiency. This tech advancement is complemented by the emergence of neuromorphic chips and innovative algorithms, specifically spiking neural networks. These networks emulate biological neurons, which communicate through binary events known as spikes rather than analog values used in traditionnal neural networks.

Despite these advancements, there’s still much to learn, especially in understanding how spiking neural networks process information. I hope I’ve successfully highlighted the importance of integrating engineering applications with neuroscience. This emerging research area, known as NeuroAI or computational neuroscience, is evolving rapidly. The ultimate aim of NeuroAI is to emulate the brain’s performance: it’s like having the computational power of a supercomputer compacted into the size of a soccer ball, using only around 20W of power, which is comparable to the energy consumption of a light bulb. This emerging research area, known as NeuroAI or computational neuroscience, is set

Event-based vision

Thu, 14 Dec 2023 14:00:00 +0000

Journées sur l’apprentissage frugal (JRAF)
13-14 décembre 2023
Grenoble (France)
https://jraf-2023.sciencesconf.org/

2023-12-14-jraf.md

Thu, 14 Dec 2023 00:00:00 +0000

Event-based vision

Adrien Fois & Laurent Perrinet

[2023-12-14] Journées sur l’apprentissage frugal (JRAF)

Hello, can you hear me in the back?

I’m Adrien Fois from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit. I’m a post-doctoral researcher under the supervision of Laurent Perrinet, and during this seminar, I’ll be presenting event-driven cameras. This innovative imaging technology and its influence on our understanding of vision will be our focus. I’d like to thank organizers for this opportunity, and all of you for coming. You can find these slides and related references on Laurent Perrinet’s website. The outline of the talk is as follows: initially, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Following that, we’ll explore some applications of these cameras using specific algorithms. Lastly, we’ll delve into how our understanding of neuroscience can enhance these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

An event-based camera is equipped with a sensor that, much like common CMOS sensors, converts light into electrical current. Yet, it stands apart from standard frame-based cameras by taking inspiration from the human retina. There are two main differences with respect to frame-based camera: Firstly each pixel of an event-based camera is independent, functioning without a synchonized global clock. Secondly, each pixel detect shifts in logarithmic light intensity, generating an binary event only when the change exceeds a threshold. If the change is an increment - meaning the log intensity increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is asynchronously generated when a pixel-level change in brightness is detected. This leads to a superior temporal resolution and a reduced susceptibility to motion blur, making event-camera ideal for capturing fast-moving scenes. Now, let’s explain how discrete events are produced in response to an analog signal that evolves continuously over time.

Event-Based Camera

Ultimately, we obtain a list of events for each pixels which can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence and polarities. Note that as events are generated over time, they are naturally sorted by their time of occurences. These events are then transmitted in real-time to the output bus, often through a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve, which connects our retina to the brain. In fact, the retina’s output is composed of a million ganglion cells that emit action potentials, constituting the only source of information transmitted through the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition—for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror the temporal sequence of events, termed the temporal surface. A temporal surface provides a representation of recent activity, it jumps to one on an event and then exponentially decays through time. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at 10%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis—an important concept from neuroscience—we enhanced the algorithm’s performance to 82%. This underscores the value of incorporating neuroscientific principles into machine learning. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons.

Always-on Object Gesture Recognition

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

Event-based vision

Adrien Fois & Laurent Perrinet

[2023-12-14] Journées sur l’apprentissage frugal (JRAF)

Spiking Neural Networks

[Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. vim Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Event-based vision

Fri, 01 Dec 2023 09:45:00 +0000

Event-based vision

Fri, 01 Dec 2023 09:45:00 +0000

When brains meet computing machines

2023-12-01-biocomp.md

Fri, 01 Dec 2023 00:00:00 +0000

Event-based vision

Laurent Perrinet

[2023-12-01] Séminaire colloque BioComp 2023

Hello, can you hear me in the back?

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this seminar at the BioComp 2023 colloquium, I’ll be presenting event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. I’d like to thank organizers for this opportunity, and all of you for coming. These slides are available from my website, along with a number of references. The outline of the talk is as follows: first, we’ll describe what an event-driven camera is - in particular, by comparing it to a conventional camera; then, we’ll show some examples of applications of these cameras with dedicated algorithms; and finally, we’ll present how our knowledge of biological mechanisms in neuroscience can enable us to improve these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

Finally, we obtain a list of events for each pixels which can be merged for the image as a whole, forming a list of events, including pixel addresses, times of occurrence and polarities. As they are generated over time, they are naturally arranged in order of occurrence. All these events are then transmitted in real time to the output bus, typically by means of a USB3 connection. Note the analogy between this representation and the one made in the optic nerve that connects our retina to the rest of the brain: indeed, the million ganglion cells that make up the retina’s output emit action potentials, which are the only source of information that leaves the retina via the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

There are several properties of event-driven cameras that make them remarkable. First of all, the temporal precision of events is of the order of microseconds, enabling a theoretical frame rate of the order of a million images per second to be reached. This can be compared with a conventional camera, which is of the order of a hundred images per second, or with a high-speed camera, which can reach 10,000 images per second. It is difficult to estimate the sampling frequency of human perception, because while 25 frames per second is often sufficient for movie viewing, it has been shown that the human eye can distinguish temporal details up to 300 or even 1,000 frames per second. It’s worth noting that the spatial resolution of these event cameras is often relatively modest, in the order of megapixels, but this is not a technical limitation, but rather due to the technological applications in which these cameras are commonly used. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another important feature of these cameras is their ability to detect a very wide range of luminosity, far exceeding that of conventional cameras at 120 dB (a factor of a million, compared with the human eye’s factor of 1 in a thousand between full moon and full sun),

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

The first algorithm we developed with Antoine Grimaldi, who is a PhD student, and in collaboration with Sio Ieng and Ryad Benosman of Sorbonne University, who are recognized researchers in the development of this type of camera, is an improvement on an existing algorithm, HOTS. This algorithm uses a relatively classical convolutional and hierarchical information processing architecture, which passes information “forward” from the camera and its event representation, and then through different processing layers to converge on a high-level representation that can be used for classification, in this case to recognize the identity of the digit presented as input, i.e. an eight digit. A fundamental feature of this algorithm is that it transforms the event representation into multiplexed, parallel channels, which analogously represent the temporal pattern of events, or “temporal surface”. These are represented in the different layers by the individual temporal surfaces. An interesting feature of this algorithm is that learning in each of the layers is unsupervised, which is a significant improvement over conventional deep learning algorithms that assume that a classification error signal can be back-propagated along the entire hierarchy, which is notoriously incorrect. Starting from this algorithm, we improved it by including neuro-biological knowledge, especially about the balance between different parallel communication pathways by including homeostasis rules.

Always-on Object Gesture Recognition

To illustrate the results of our algorithm, we applied a classic camera dataset involving the classification of 10 different types of human gestures. These biological movements are, for example, clapping hands, saying hello or a drum movement. The chance level is therefore at 10%, and we have observed that when all events have been processed, the original algorithm achieves a performance of around 70%. By adding homeostasis, we have reached a higher level of 82%, demonstrating the usefulness of using neuroscientific knowledge to improve machine learning algorithms.

We also built on a fundamental characteristic of biological systems. In fact, this kind of algorithm is classically used to process the flow of events, but classification is only used as a last resort when all the events have been processed. We have modified the algorithm so that this classification can be done online, in real time, event by event. In this way, processing in the various layers is triggered by the arrival of each event, which is propagated from the camera through all the layers to the classification layer.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of the average performance obtained on a data set, and as a function of the number of events processed by the algorithm. The blue curve shows that if below 10 events, we remain at the level of chance, we then experience a gradual increase in performance that reaches the level of the original algorithm with ten thousand events, and exceeds this performance when we have even 10 times more. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in its event camera, not once the entire signal has been processed by the system, but at any time. This characteristic is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the flexibility to wait for the video sequence to finish processing before making the right decision, which is to flee. Another variant in our algorithm consists of selecting the output classification events based on a calculation of the precision for each event. By using a threshold on this precision, we can achieve a very good level of performance, with just a hundred events, and so achieve a characteristic that is common in biological networks, i.e. that a decision is not taken gradually, but emerges abruptly (here after 200 events) and then improves and stabilizes.

Spiking Neural Networks

[Tonic manual]

Indeed, most neural networks used in deep learning use an analog representation. This is illustrated in this figure, which represents the various analog inputs to a formal neuron as they are linearly integrated by the synapses, then transformed by a non-linear function to generate an activation which is itself analog. This basic perceptron principle is at the foundation of all existing neural networks, and in particular enables the construction of convolutional-type networks which are currently the champions for image classification, having outperformed human performance for several years. However, while this is true for static images, it can become prohibitively expensive with videos. This is why it can be interesting to use spiking neurons instead, which, instead of receiving an analog input, will receive events that will trigger cascades of mechanisms in the neuronal cell, notably represented by the cell’s membrane potential. Typically, we’ll include a threshold for triggering action potential in this cell, which will generate new output events on the cell’s axon.

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

This new type of representation represents a paradigm shift in computation, in the same way that event-driven cameras have brought with them a paradigm shift in image representation. The development of these two new algorithms, which use impulse neural networks, is accompanied by the development of new neuromorphic chips, such as the Loihi 2 chip developed by Intel, which replaces a central computing unit with a massively parallelized array of elementary event-driven computing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. Other types of neuromorphic chips are currently being developed and may soon be used instead of conventional CPUs or GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking neural networks therefore seem very promising for processing the output of event-driven cameras, but the study of neurophysiology shows us that their operation can sometimes seem incongruous and far from the perceptron. In this first example, taken from an article by Mainen and Sejnowski from 1995, we see the response of the same neuron to several repetitions of a stimulation in panel A. At the top, we see the membrane potential of this neuron in response to a 200 Pico ampere current step, which shows that the membrane potential is not reproducible across different trials. This is illustrated by showing the spike response over time for the different trials, which shows a strong alignment at the start of stimulation, but that this diffuses little by little, so that after around 750 milliseconds there is no longer any coherence between the different trials. The situation is different in panel B, where the neuron is stimulated with noise. In this case, the responses are so precise for the different trials that the membrane potential traces are overlapping almost exactly. The subtlety of this paper lies in its use of a frozen noise, i.e. one that is repeated unchanged across trials. In this way, it demonstrates that neurons are not so much sensitive to analog values presented in the form of square pulses, but rather to dynamic signals for which they will respond with very high precision in the dynamic domain.

Spiking Neural Networks in neurobiology

2 MINUTE

In this other example, I show a simulation that reproduces the 1999 paper by Diesmann and colleagues. This theoretical model considers ten groups of 100 neurons that are connected from group to group. An interesting property of this system is to show that for the same stimulation, i.e. for the same number of spikes, information can propagate from group to group only if it is sufficiently concentrated in time. For the first two groups, the information is too dispersed in the first group and spreads progressively and increasingly in subsequent groups. Above a certain threshold, the information formed by a group of relatively synchronous spikes is correctly transmitted to the various groups in the network. This non-linear behavior is one of the characteristics of spiking networks, giving them a certain richness, but also a certain complexity.

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Event-based vision

Laurent Perrinet

[2023-12-01] Séminaire colloque BioComp 2023

2023-11-07-snufa.md

Tue, 07 Nov 2023 00:00:00 +0000

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

SNUFA: Spiking Neural networks as Universal Function Approximators

^{https://laurentperrinet.github.io/talk/2023-11-07-snufa}

Core Mechanism of Spiking Motif Detection

From generating raster plots to inferring spiking motifs

A In this work, this principle was framed in a probabilistic setting such that we could provide an optimal scheme for detecting generic spiking motifs which may be superposed at random times. Starting with 10 presynaptic inputs, this model allows to generate a synthetic raster plot as the combination of four different spiking motifs. B These motifs are defined by a positive (red) or negative (blue) contribution to the spiking probability which are represented here. C Applying a Bayesian approach, we may define four formal spiking neurons which will integrate the incoming spiking information from the presynaptic neurons - this analog signal can then be thresholded to give the detection of each spiking motif (vertical) bar which was here always exact with respect to the ground truth (stars). D The beauty of this is that we can recover in the presynaptic raster plot the contribution of each spiking motif to the original raster plot.

Detecting spiking motifs using heterogeneous delays

Detecting spiking motifs using heterogeneous delays: supervised learning

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

SNUFA: Spiking Neural networks as Universal Function Approximators

^{https://laurentperrinet.github.io/talk/2023-11-07-snufa}

ANR MarmoCatch (2023-10/2029-04)

Fri, 27 Oct 2023 00:00:00 +0000

Under natural conditions, many animals perform interceptive movements to catch small prey (Shaw22 for a marmoset study). Such movements require the processing of several features of the prey, such as its size, orientation, and position, which are ultimately expressed into the coordinated control of the arm and hand during movement execution. Furthermore, in the case of a moving prey, catching movements must also take into account the highly dynamic and intricate nature of these visual features in order to precisely control the movement to catch at the appropriate location, orientation, and timing. This behaviour relies on the capacity of our brain to overcome the intrinsic neuronal delays in visual and motor systems to anticipate as accurately as possible the prey’s trajectory in the multiple features of interest. At the visual level, moving stimuli are known to induce predictions along their trajectory (Krekelberg01, Nijhawan08), yet with fewer neurophysiological (Jancke04, Guo07, Subramaniyan2018, Benvenuti21) or theoretical evidence (Grzywacz95, Perrinet12). At the motor level, neurons in motor and parietal cortex have been reported to be involved in the predictive control of interception movements that require a precise estimation of the movement time (Port01, Merchant04, Li22). However, how visual and motor predictive responses relate to one-another remains an open issue, even more so under naturalistic conditions in which several target features may (co-)vary in parallel. Our objective is to shed light on the mechanisms underlying the coordination between visual and motor cortices in the marmoset while it prepares and executes the grasp of a real physical target in complex motion.

Acknowledgement

This work was supported by ANR project “MarmoCatch” N° ANR-XXX-YYY.

Time-to-Contact Map by Joint Estimation of Up-to-Scale Inverse Depth and Global Motion using a Single Event Camera

Fri, 06 Oct 2023 00:00:00 +0000

the code is openly available on GitHub with the accompanying data VL.zip.

2023-09-27_icann.md

Wed, 27 Sep 2023 00:00:00 +0000

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

ICANN workshop on Recent Advances in SNNs

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this talk at this ICANN workshop on Recent Advances in SNNs, I’ll be presenting a method for the Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network, and how it may also impact the design of SNNs. I’d like to thank Sander Bohté and Sebastian Otte for the organization of this workshop and you for listening. These slides are available from my web-site, along with a number of references. The outline of the talk is as follows: first, I’ll describe how one may perform computations using Heterogeneous Delays - and present a toy model example; then, I’ll show real scale example quantifying the performance on synthetic data ; and finally, I’ll present how this SNN is in fact differentiable and may be extended for future applications.

Core Mechanism of Spiking Motif Detection

From generating raster plots to inferring spiking motifs

Detecting spiking motifs using heterogeneous delays

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

ICANN workshop on Recent Advances in SNNs

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

Learning heterogeneous delays in a layer of spiking neurons for fast motion detection

Mon, 11 Sep 2023 00:00:00 +0000

read the paper online (paywall) or read the reprint as PDF
full code with extensive Supplementary Material
join the Zotero group to add and discuss more items
this paper is a follow-up of
Antoine Grimaldi, Camille Besnainou, Hugo Ladret, Laurent U Perrinet (2022). Learning heterogeneous delays of spiking neurons for motion detection. Proceedings of ICIP 2022.
PDF Cite DOI Venue URL

Event-based vision

Fri, 08 Sep 2023 11:00:00 +0000

2023-09-08_fresnel.md

Fri, 08 Sep 2023 00:00:00 +0000

Event-based vision

Laurent Perrinet

[2023-09-08] Séminaire institut Fresnel

https://laurentperrinet.github.io/talk/2023-09-08-fresnel/

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this seminar at the Institut Fresnel, I’ll be presenting event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. I’d like to thank Loic le Goff for his kind invitation, and all of you for coming. These slides are available from my website, along with a number of references. The outline of the talk is as follows: first, we’ll describe what an event-driven camera is - in particular, by comparing it to a conventional camera; then, we’ll show some examples of applications of these cameras with dedicated algorithms; and finally, we’ll present how our knowledge of biological mechanisms in neuroscience can enable us to improve these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition

Always-on Object Gesture Recognition

Always-on Object Gesture Recognition

Spiking Neural Networks

[Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Event-based vision

Laurent Perrinet

[2023-09-08] Séminaire institut Fresnel

Cortical recurrence supports resilience to sensory variance in the primary visual cortex

Tue, 06 Jun 2023 00:00:00 +0000

open access: https://www.nature.com/articles/s42003-023-05042-3
5 minutes summary: https://hugoladret.github.io/publications/ladret_et_al_variance_v1/
This neurophysiological work accompanies a similar study in theoretical neuroscience :
Hugo Ladret, Laurent U Perrinet (2023). Convolutional Sparse Coding is improved by heterogeneous uncertainty modeling. ICLR 2023 SNN Workshop.
PDF Cite URL

Communiqué de presse: Comment le cerveau fait face à l’incertitude ?

[Introduction :] Nous vivons dans un monde fait d’incertitudes, qui pourtant ne nous empêchepas d’effectuer nos tâches quotidiennes. Vous ne traverseriez pas la route avant d’être certain que le conducteur de la voiture passante vous a vu, pas d’avantage que vous ne vous approcheriez pas d’un buisson avant d’être sûr qu’il est occupé par un oiseau plutôt que par un lion. Malgré la nécessité fondamentale de résoudre ces incertitudes au quotidien, nous savons relativement peu sur la manière dont notre cerveau procède pour ce faire. Dans cet article publié dans Nature Communications Biology, les scientifiques présentent des enregistrements des neurones du cerveau, et mettent en évidence un nouveau type de neurone qui encode cette incertitude. Cette recherche est clé pour avancer la compréhension de notre cerveau et construire des modèles artificiels qui peuvent prendre en compte leurs certitudes. Imaginez que vous vous promeniez dans une forêt. Le vent bruisse dans les feuilles, quand soudain un bruit étrange attire votre attention. S’agit-il d’un écureuil qui se précipite sur le sentier à la vue de tous ? Ou peut-être d’un oiseau niché derrière les buissons, caché dans le feuillage ? Dans ce dernier cas, prenez-vous le temps de voir l’oiseau, ou en déduirez-vous que le bruissement est plutôt celui d’un lion, et vous enfuirez-vous le plus vite possible ? Ce simple scénario illustre un problème quotidien auquel nous sommes confrontés : comment notre cerveau peut-il donner un sens au monde, alors que nos sens sont bombardés d’informations peu fiables ? Ce manque de précision - l’inverse de la variance d’une information - est marquant dans le domaine de la vision. En effet, une image peut être décomposée en de nombreuses lignes ou “bords” qui forment sa structure, à l’instar d’un puzzle composé de nombreuses pièces différentes.

Figure 1. Les images naturelles (ici, une vue des calanques de Marseille), sont décomposées en éléments orientés (en bas à gauche) par des réseaux de neurones, dont les interactions sont contraintes par l’incertitude locale qui décrit des parties du champ visuel.

Cependant, toutes les pièces du puzzle ne sont pas coupées de la même manière, et certaines ont des bords plus variables que d’autres. C’est un problème pour la toute première zone de notre cerveau qui commence à donner un sens à ces “pièces de puzzle” visuelles, le cortex visuel primaire. Jusqu’à récemment, notre compréhension de la manière dont le cerveau traite ces données visuelles complexes reposait en grande partie sur l’observation du comportement humain [1,2]. Ces dernières années, cependant, les chercheurs ont commencé à sonder le cortex visuel primaire des macaques et ont découvert que cette zone du cerveau présente des comportements complexes qui reflètent les processus de prise de décision complexes que nous entreprenons en tant qu’êtres humains [3]. En effectuant des enregistrements dans le cortex visuel primaire, la zone responsable du traitement de l’information visuelle dans le cerveau, les chercheurs ont découvert un phénomène remarquable : les neurones de notre cortex visuel primaire ont des réponses distinctes à la complexité des images. Deux types principaux de neurones ont été identifiés sur la base de leurs réponses : certains sont relativement indifférents à l’augmentation de la variance, tandis que d’autres montrent une décroissance rapide de leur capacité d’encodage face à cette variance (non linéaires).

Figure 2 La variation du code des neurones face a une augmentation d’incertitude dépend de leur position dans le cortex (a, b), un phénomène expliqué par une activité récurrente plus intense pour les neurones encodant l’incertitude.

Globalement, la récurrence peut expliquer comment différents neurones encodent (ou non) la variance de leur entrée. Ces résultats vont dans le sens d’une compréhension plus complète du cerveau, qui ne se contente pas d’encoder des caractéristiques moyennes, comme le suggéraient les modèles précédents, mais prend également en compte la complexité des entrées, grâce à la connectivité entre les neurones. Il s’agit d’une étape cruciale pour comprendre comment notre cortex gère les “puzzles visuels” que nous rencontrons tous les jours, permettant au cerveau d’effectuer des calculs complexes sur des distributions probabilistes - un modèle qui gagne en popularité dans les neurosciences [5].

Références

[1] Von Helmholtz, H. (1925). Helmholtz’s treatise on physiological optics (Vol. 3). Optical Society of America. [2] Barthelmé, S., & Mamassian, P. (2009). Evaluation of objective uncertainty in the visual system. PLoS computational biology, 5(9), e1000504. [3] Hénaff, O. J., Boundy-Singer, Z. M., Meding, K., Ziemba, C. M., & Goris, R. L. (2020). Representation of visual uncertainty through neural gain variability. Nature communications, 11(1), 2513. [4] Leon, P. S., Vanzetta, I., Masson, G. S., & Perrinet, L. U. (2012). Motion clouds: model-based stimulus synthesis of natural-like random textures for the study of motion perception. Journal of neurophysiology, 107(11), 3217-3226. [5] Spratling, M. W. (2016). A neural implementation of Bayesian inference based on predictive coding. Connection Science, 28(4), 346-383.

2023-05-10-phd-program_neurosciences-computationnelles.md

Wed, 10 May 2023 00:00:00 +0000

Interactions between machine learning, artificial neural networks and our understanding of biological vision

Laurent Perrinet

[2023-05-10] NeuroSchool PhD Program in Neuroscience: Computation Neuroscience

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Cydonia Mensae, 1976, Viking Orbiter image

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Cydonia Mensae, 2007, Mars Global Surveyor

Visual illusions : Pareidolia

Cydonia Mensae, 2007, Mars Global Surveyor

Visual illusions: Context

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions: Context

Visual illusions: Context

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

[Sejnowski, Koch & Churchland, 1998]

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks (CNNs)

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962] - from @Neuroslicer

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision

Interactions between machine learning, artificial neural networks and our understanding of biological vision

Laurent Perrinet

[2023-05-10] NeuroSchool PhD Program in Neuroscience: Computation Neuroscience

Contact me @ laurent.perrinet@univ-amu.fr

Postdoc position "Accurate detection of precise spiking motifs in neurobiological data"

Mon, 01 May 2023 09:00:00 +0000

THE POSITION HAS BEEN FILLED.

Dear colleagues,

Applications are welcome for a fully funded 18-month postdoctoral position for the development of an algorithm for the accurate detection of precise spiking motifs in neurobiological data. The position will be located at the INT in Marseille, France. The project is funded by the polychronies grant (AMX-21-RID-025) and coordinated by Laurent Perrinet together with Thomas Schatz (theory) and Rosa Cossart (neurobiology).

Candidates should have experience in computational neuroscience, physics, engineering, or related fields, and a strong background in machine learning. The candidate must have good computer science skills (programming skills, git versioning, …) and Python programming experience is required. A multidisciplinary background would be highly appreciated, especially an advanced knowledge of mathematics. The candidate must have a strong interest in neuroscience. The candidate must be fluent in English and willing to proactively interact with partners in different communities, including theoretical neuroscience, machine learning, or neurobiology. The preferred candidate should have the ability to work independently and be flexible to adapt to the working methods of the supervisors.

More details on the “polychronies” grant:
Rosa Cossart, Thomas Schatz, Laurent U Perrinet (2022). Polychronies (2022 / 2025).
Our recent review on Precise spiking motifs in neurobiological and neuromorphic data:
Antoine Grimaldi, Amélie Gruel, Camille Besnainou, Jean-Nicolas Jérémie, Jean Martinet, Laurent U Perrinet (2022). Precise spiking motifs in neurobiological and neuromorphic data.
Cite DOI Hal Code URL arXiv
Application of detecting spiking motifs in neuromorphic data:
Antoine Grimaldi, Laurent U Perrinet (2023). Learning heterogeneous delays in a layer of spiking neurons for fast motion detection. Biological Cybernetics.
PDF Cite DOI URL Code Supplementary Material
A theoretical framework on the accurate (supervised) detection of spiking motifs in (synthetic) multi-unit raster plots
Laurent U Perrinet (2023). Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network. ICANN Special Session on Recent Advances in Spiking Neural Networks.
PDF Cite Slides DOI Code arXiv
job offer posted on: Euraxess - jobrXiv - academic positions

Research context

The position will be carried out in the team “NEuronal OPerations in visual TOpographic maps” (NeOpTo) within the Institut de Neurosciences de la Timone in Marseille, a welcoming and lively town by the Mediterranean Sea in the south of France. The research team is led by F. Chavane (DR, CNRS) and currently hosts 4 permanent staff, 3 post-docs and 4 PhD students. The research themes of the team are focused on neuronal operations within visual cortical maps. Indeed, along the cortical hierarchy, low-level features such as the position and orientation of the visual stimulus (but also auditory tone, somatosensory touch, etc…) but also higher-level features (such as faces, viewpoints of objects, etc…) are represented topographically on the cortical surface.

Artificial neural networks and machine learning applied to the understanding of biological vision

Wed, 05 Apr 2023 09:00:00 +0000

2023-04-05-ue-neurosciences-computationnelles

Wed, 05 Apr 2023 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-05] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : the HMAX model

Convolutional Neural Networks (CNNs)

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-05] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

Artificial neural networks and machine learning applied to the understanding of biological vision

Mon, 03 Apr 2023 14:00:00 +0000

2023-04-03-master-m-4-nc

Mon, 03 Apr 2023 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-03] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : the HMAX model

Convolutional Neural Networks (CNNs)

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision