Slides | Next-generation neural computations

Spiking Neural Networks: neurobiology

Spiking Neural Networks: neurobiology

Spiking Neural Networks: neurobiology

Etienne Rey (2010) Spectre audiographique

Spiking Neural Networks: Leaky Integrate-and-Fire

Review on Precise Spiking Motifs.

Spiking Neural Networks: Heterogeneous Delays

Review on Precise Spiking Motifs.

Heterogeneous Delays Spiking Neural Network: HD-SNN

Spiking Neural Network: Polychronization

Izhikevich (2006)

Spiking Neural Network: Polychronization

Izhikevich (2006)

Spiking Neural Network: Polychronization

LP (2026)

Methods : BPTT (snn Torch) - synthetic target

Methods : Weight initialization

$$ I_j(t) = \sum_{i=1}^{N} \bigl ( \sum_{d=1}^{D} \mathbf{W}_{j, i, d} \cdot s_i(t-d) \bigr ) $$ $$ u_j(t) = \beta \cdot u_j(t-1) \cdot (1 - s_j(t-1)) + I_j(t) $$ $$ s_j(t) = \mathbf{H}[u_j(t) \geq \vartheta] $$ $$ \mathbf{W} \mathbf{C} \approx \mathbf{S} $$ $$ w_{j, i, d} = \frac{1}{N \cdot D \cdot p_A \cdot M} \sum_{\mu=1}^{M} \sum_{t=D+1}^{T} s_{j}^{\mu}(t) \cdot s_i^{\mu}(t-d) $$

Results : recall of target

Results : memory retrieval

Results : recall of target with noise

Results : role of parameters

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Cerco seminar

[2026-04-16]

Contact me @ laurent.perrinet@univ-amu.fr

Working Memory in SNNs

Wed, 15 Apr 2026 09:00:00 +0000

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Austrian Symposium on AI, Robotics and Vision

[2026-04-15]

Contact me @ laurent.perrinet@univ-amu.fr

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this talk at this AIROV workshop on Recent Advances in SNNs.

I’d like to thank Sander Bohté and Sebastian Otte for the organization of this workshop and you for listening. These slides are available from my web-site, along with a number of references. The outline of the talk is as follows: first, I’ll describe how one may perform computations using Heterogeneous Delays - and present a toy model example; then, I’ll show real scale example quantifying the performance on synthetic data.

Polychronization

Methods : BPTT (snn Torch) - synthetic target

Methods : Weight initialization

Results : recall of target

Results : role of parameters

Results : memory retrieval

Learning Working Memory in Recurrent Spiking Neural Networks Using Heterogeneous Delays

Laurent Perrinet

Austrian Symposium on AI, Robotics and Vision

[2026-04-15]

Contact me @ laurent.perrinet@univ-amu.fr

2026-04-11-intelligence-du-regard

Sat, 11 Apr 2026 00:00:00 +0000

L’intelligence du regard

Laurent Perrinet

Forum des Sciences Cognitives 2026

[2026-04-11]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

Bonjour, je me présente : Laurent Perrinet. Je suis très heureux de participer au Forum des Sciences Cognitives et je remercie les organisateurs pour cette invitation. Je suis d’autant plus ravi d’y participer que je ne vais pas parler de mes recherches habituelles, mais plutôt exposer la collaboration que j’ai avec un artiste plasticien à Marseille. Mon objectif : vous convaincre des bénéfices que l’on peut tirer à s’ouvrir au monde artistique pour mieux percer les mystères de la cognition dans toute sa diversité.

Les objectifs de cet exposé seront multiples :

Vous faire découvrir certains artistes contemporains qui questionnent notre rapport aux nombres visuels
Montrer la diversité de la vision à travers les interactions entre art et science
Dévoiler certains mystères de la vision au travers de l’expérience artistique et les applications que cela peut avoir sur notre compréhension de la cognition

Art & Sciences révèlent la diversité de notre vision

Etienne Rey

Art & Sciences révèlent la diversité de notre vision

Notre collaboration a commencé quand il m’a invité à présenter mon travail sur la perception visuelle au vernissage de cette œuvre qui représente une visualisation spatio-temporelle du spectre audiographique du son d’une cloche. Il est composé de multiples plaques semi-transparentes et dichroïques, c’est-à-dire ayant la capacité de présenter différentes couleurs selon l’angle de vue. Ce volume sculptural donne, de façon furtive, toute la profondeur de cette expérience sensorielle.

C’est là que se révèle « L’irraisonnable efficacité de la vision » — je reprends les mots de Wigner à propos de la capacité des mathématiques à sonder le monde. Car nous nous retrouvons devant un constat similaire : comment est-il possible avec aussi peu de moyens d’obtenir une perception si vivante du monde qui nous entoure ?

“L’irraisonnable efficacité de la vision”

Nage de la raie, 1894 [Étienne-Jules Marey]

J’espère vous surprendre en vous montrant ce vol de raie, une nage capturée par Étienne-Jules Marey grâce au procédé de chronophotographie. Je trouve cette image animée remarquable par plusieurs aspects.

D’abord, Marey utilisait carrément un appareil en forme de fusil mitrailleur avec des plaques photographiques en guise de balles pour « shooter » une scène dynamique que l’œil humain aurait du mal à décomposer.

L’enjeu est d’abord scientifique : comprendre le mouvement. Il a d’ailleurs donné son nom à l’ISM, l’Institute for Scientific Motion.

Il y a aussi un plaisir artistique, celui qui a été développé jusqu’à devenir l’industrie cinématographique : une succession d’images peut donner la perception d’un mouvement fluide. C’est là que c’est irraisonnable — des images statiques, de qualité médiocre, donnent pourtant une impression vive.

Et cette capacité n’est pas nouvelle.

“L’irraisonnable efficacité de la vision”

Panneau Des Lions [Grotte chauvet, -30 kA]

“L’irraisonnable efficacité de la vision”

Comment la vision a évolué… [LP, 2024, The Conversation]

Illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Illusions visuelles

Illusions visuelles

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Victor Vasarely (1971) Gare Montparnasse

La perception comme processus émergent

La perception comme processus émergent

François Morrelet (1962) Mönchengladbach, Sphère - trames

La perception comme processus émergent

Etienne Rey, Trames

C’est dans ce cadre que nous avons expérimenté avec Etienne Rey sur des trames, qui crée ces interférences. Émergence de nouvelles formes — hexagones comme utilisés à l’Alhambra — espaces tridimensionnels.

Je reviendrai sur le fait que vous pouvez transformer l’image en bougeant les yeux.

La perception comme processus émergent

Etienne Rey (2025) Variable Density, série Delaunay

La vibration des apparences

Paul Cézanne, Montagne Sainte-Victoire, 1904

La vibration des apparences

Merleau-Ponty, Sens et non-sens

Etienne Rey (2019) Horizon faille - Densité flou - Sans gravité - une poétique de l’air à Ardenome Avignon https://www.enrevenantdelexpo.com

La vibration des apparences

Etienne rey (2025) Polaire

La vibration des apparences


def retino_grid(cr, N_rho=34, N_phi=233, N_H, N_V, offset,
 size_mag, ecc_max, alpha, c1, c2, power):
 # https://laurentperrinet.github.io/sciblog/posts/2020-04-16-creating-an-hexagonal-grid.html
 phi_v, rho_v = np.meshgrid(np.linspace(0, 2*np.pi, N_phi, endpoint=False),
 np.linspace(0, ecc_max, N_rho+1, endpoint=True)[1:], sparse=False, indexing='xy')
 phi_v[::2, :] += np.pi/N_phi

 offsets, colors = [-offset, offset], [c1, c2]
 for offset_, color in zip(offsets, colors):
 # convert to cartesian coordinates
 X, Y = (rho_v * np.sin(phi_v) + offset_+1)/2, (rho_v * np.cos(phi_v)+1)/2
 R = size_mag * rho_v**power / N_rho

 for x, y, r in zip(X.ravel(), Y.ravel(), R.ravel()):
 circle(cr, x, y, r)
 cr.set_source_rgba(*hue_to_rgba(color, alpha))
 cr.fill()
 return cr
c_blue, dc = 240, 60
opts = dict(N_rho=N_rho, N_phi=N_phi, N_H=N_H, N_V=N_V,
 offset=0.07, size_mag=0.3, ecc_max=0.8, alpha=0.80,
 c1=c_blue-dc, c2=c_blue+dc)
@disp
def draw(cr, N_H=N_H, N_V=N_V): cr = retino_grid(cr, **opts)

À quoi sert la vision ?

Ilya Repin (1884) An Unexpected Visitor

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor How long?

À quoi sert la vision ?

Yarbus (1965) An Unexpected Visitor - Age?

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

E Rey et LP (2026) Formes & perception

À quoi sert la vision : rétinotopie fovéale

À quoi sert la vision : rétinotopie fovéale

Illusions visuelles

Etienne Rey, Spectre audiographique

Art & Sciences révèlent la vision en action

Art & Sciences révèlent la vision en action

Carlos Cruz-Diez (2013) Chromosaturation

Felice Varini.

Art & Sciences révèlent la vision en action : Tropique

Etienne Rey, Tropique

Art & Sciences révèlent la vision en action : Tropique

Etienne Rey, TRAME ÉLASTICITÉ

Art & Sciences révèlent la vision en action : TRAME ÉLASTICITÉ

Art & Sciences révèlent la vision en action

Ede Rancz, Role of neuromodulators in active perception

Art & Sciences révèlent la vision en action

Ede Rancz, Role of neuromodulators in active perception

L’intelligence du regard

Laurent Perrinet

Forum des Sciences Cognitives 2026

[2026-04-11]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

Diapositives supplémentaires

Victor Vasarely

Victor Vasarely (1962) Mönchengladbach, Sphère - trames

Victor Vasarely

Victor Vasarely (1977)Outdoor Vasarely artwork at the church of Pálos in Pécs.

Victor Vasarely

Victor Vasarely (1962) Supernovae

Victor Vasarely

Victor Vasarely (1976) Fondation Vasarely, Aix-en-Provence

Etienne Rey, Cristal n2

2026-03-05-ue-natural-cognition

Thu, 05 Mar 2026 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2026-03-05] Master 1 Neuroscience, UE Natural Cognition, Artificial Cognition

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

[Serre and Poggio, 2007]

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Hybrid IA models

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Principles of vision?

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Principles of Vision

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2026-03-05] Master 1 Neuroscience, UE Natural Cognition, Artificial Cognition

Contact me @ laurent.perrinet@univ-amu.fr

2026-02-10-biomplus

Tue, 10 Feb 2026 00:00:00 +0000

Recréer des réseaux neuronaux pour améliorer la compréhension de notre cerveau

Laurent Perrinet

Webinaire Biome+ [Biomimétisme & Neurosciences]

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

Attention in Vision Transformers and in Natural Vision

Saccade selection method. Matthis Dallain with the EDGE Team @ LEAT Laboratory

Learning where to look

Spiking Neural Networks

From frame-based to event-based cameras.

Spiking Neural Networks

The HD-SNN neural network.

Recréer des réseaux neuronaux pour améliorer la compréhension de notre cerveau

Laurent Perrinet

Webinaire Biome+ [Biomimétisme & Neurosciences]

[2026-02-03]

2026-02-03-ai-and-neuroscience-day

Tue, 03 Feb 2026 00:00:00 +0000

Neuroscience & AI: Energy-efficient visual processing algorithms

Laurent Perrinet

Journée Neurosciences et IA / IA et Neurosciences de NeuroMarseille

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

Spiking Neural Networks

From frame-based to event-based cameras.

Spiking Neural Networks

The HD-SNN neural network.

Attention in Vision Transformers and in Natural Vision

Saccade selection method. Matthis Dallain with the EDGE Team @ LEAT Laboratory

Learning where to look

Neuroscience & AI: Energy-efficient visual processing algorithms

Laurent Perrinet

Journée Neurosciences et IA / IA et Neurosciences de NeuroMarseille

[2026-02-03]

Contact me @ laurent.perrinet@univ-amu.fr

2026-01-29-emergences

Thu, 29 Jan 2026 00:00:00 +0000

Neurosciences and sparsity

Laurent Perrinet

Séminaire à l’atelier “IA embarquée” du PEPR IA

[2026-01-29]

Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations in computer vision

[Lennie, 2003, The Cost of Cortical Computation]

Neurosciences and sparsity: a survey

Go to wooclap.com
Enter the code HLEQUP
Or directly follow https://app.wooclap.com/HLEQUP?from=instruction-slide

Neurosciences and sparsity: a survey

Neurosciences and sparsity

Neurosciences and sparsity

[Brunel, 2001]

Neurosciences and sparsity

Neurosciences and sparsity

Neurosciences and sparsity

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ & = \frac{1}{2\sigma_n^2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 - \sum_{i=1}^{K} \log Pr( a[i] ) \end{aligned} $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations and learning

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Neurosciences and sparsity

Laurent Perrinet

Séminaire à l’atelier “IA embarquée” du PEPR IA

[2026-01-29]

Contact me @ laurent.perrinet@univ-amu.fr

Markdown Slides Demo

Mon, 15 Dec 2025 00:00:00 +0000

Markdown Slides

Write in Markdown. Present Anywhere.

What You Can Do

Write slides in pure Markdown
Include code, math, and diagrams
Add speaker notes for presenter view
Use progressive reveals for storytelling
Customize themes and transitions

Code Highlighting

def fibonacci(n: int) -> int:
 if n <= 1:
 return n
 return fibonacci(n-1) + fibonacci(n-2)

# Calculate the 10th Fibonacci number
print(fibonacci(10)) # Output: 55

Mathematical Equations

Einstein’s famous equation:

$$E = mc^2$$

The quadratic formula:

$$x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}$$

Mermaid Diagrams

graph LR A[Markdown] --> B[Hugo] B --> C[Reveal.js] C --> D[Beautiful Slides]

Progressive Reveals

Build your narrative step by step:

First, introduce the concept Then, add supporting details Finally, deliver the conclusion

Speaker Notes

Press S to open presenter view!

Note:

These notes are only visible in presenter mode
Perfect for talking points and reminders
Supports Markdown formatting
Add timing cues and references here

Dual Column Layout

Benefits

Open source
Version control
No vendor lock-in
Works offline

Use Cases

Tech talks
Academic papers
Team updates
Training sessions

Custom Backgrounds

Slides can have custom colors or images.

Use {{< slide background-color="#hex" >}}

Keyboard Shortcuts

Key	Action
`→` / `←`	Navigate slides
`S`	Speaker notes
`F`	Fullscreen
`O`	Overview mode
`ESC`	Exit modes

Get Started

Create a file in content/slides/
Add front matter with type: slides
Write your content in Markdown
Separate slides with ---

Thank You!

Questions?

GitHub: HugoBlox/kit
Docs: docs.hugoblox.com

Built with Markdown Slides

🎨 Branding Your Slides

Add your identity to every slide with simple configuration!

What you can add:

Element	Position Options
Logo	top-left, top-right, bottom-left, bottom-right
Title	Same as above
Author	Same as above
Footer Text	Same + bottom-center

Edit the branding: section in your slide’s front matter (top of file).

📁 Adding Your Logo

Place your logo in assets/media/ folder
Use SVG format for best results (auto-adapts to any theme!)
Add to front matter:

branding:
 logo:
 filename: "your-logo.svg" # Must be in assets/media/
 position: "top-right"
 width: "60px"

Tip: SVGs with fill="currentColor" automatically match theme colors!

📝 Title & Author Overlays

Show presentation title and/or author on every slide:

branding:
 title:
 show: true
 position: "bottom-left"
 text: "Short Title" # Optional: override long page title

 author:
 show: true
 position: "bottom-right"

Author is auto-detected from page front matter (author: or authors:).

📄 Footer Text

Add copyright, conference name, or any persistent text:

branding:
 footer:
 text: "© 2024 Your Name · ICML 2024"
 position: "bottom-center"

Tip: Supports Markdown! Use [Link](url) for clickable links.

🔇 Hiding Branding Per-Slide

Sometimes you want a clean slide (title slides, full-screen images).

Add this comment at the start of your slide content:

<!-- no-branding -->
## My Clean Slide

Content here...

☝️ This slide uses  — notice no logo or overlays!

🔇 Selective Hiding

Hide just the header (logo + title):

<!-- no-header -->

Or just the footer (author + footer text):

<!-- no-footer -->

☝️ This slide uses  — footer still visible below!

✅ Quick Reference

Comment	Hides
`<!-- no-branding -->`	Everything (logo, title, author, footer)
`<!-- no-header -->`	Logo + Title overlay
`<!-- no-footer -->`	Author + Footer text

☝️ This slide uses  — logo still visible above!

2025-10-16-flash-lag-effect

Thu, 16 Oct 2025 00:00:00 +0000

Mislocalization by Design
The Flash-Lag Effect as Prediction

Laurent Perrinet, CNRS/AMU, Marseille, France

[2025-10-16] Suresh Krishna’s lab meeting

Contact me @ laurent.perrinet@univ-amu.fr

Mislocalization by Design: The Flash-Lag Effect as Prediction »

Why do we sometimes misjudge where visual objects are? This talk explores how predictive processing may cause systematic perceptual mislocalizations. Indeed, the early visual system doesn’t passively process information—it actively predicts the world, compensating for neural delays by extrapolating motion trajectories. Using a Bayesian computational model, I show how this predictive mechanism explains the flash-lag effect: moving objects appear ahead of flashed ones because the brain forecasts their current position while the unpredictable flash cannot be anticipated. This framework reveals that mislocalization isn’t a bug but a feature of efficient visual coding. I’ll discuss how these principles illuminate both biological vision and artificial visual system design, demonstrating that what we perceive as “now” is actually the brain’s best prediction of the present.

Timing in the visual pathways

Ultra-rapid visual processing (see review).

Compensating visual delays (Perrinet, Adams & Friston 2014).

Compensating visual delays (Perrinet Adams & Friston, 2014).

Suppressive travelling waves (Chemla et al, 2019).

Predictive processing

Motion-based prediction (Perrinet et al, 2012).

Flash-lag effect

Flash-lag effect (Khoei et al, 2017).

Diagonal Markov model (Khoei et al, 2017).

Flash-lag effect: MBP (Khoei et al, 2017).

Flash-lag effect (Khoei et al, 2017).

Space-time probability distributions (Khoei et al, 2017).

Motion reversal (Khoei et al, 2017).

Motion reversal (smoothed) (Khoei et al, 2017).

Space-time probability distributions (Khoei et al, 2017).

Limit cycles (Khoei et al, 2017).

Mislocalization by Design
The Flash-Lag Effect as Prediction

Laurent Perrinet, CNRS/AMU, Marseille, France

[2025-10-16] Suresh Krishna’s lab meeting

Contact me @ laurent.perrinet@univ-amu.fr

Mislocalization by Design: The Flash-Lag Effect as Prediction »

2025-05-26-master-m-4-nc

Mon, 26 May 2025 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2025-05-26] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

[Serre and Poggio, 2007]

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Hybrid IA models

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2025-05-26] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

2025-04-18-vibration-apparences

Fri, 18 Apr 2025 00:00:00 +0000

La vibration des apparences

Laurent Perrinet

Journées d’Ouverture Scientifique (JOS)

[2025-04-18]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

“L’irraisonnable efficacité de la vision”

Comment la vision a évolué… [Perrinet, 2024]

Illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Illusions visuelles

Illusions visuelles

Illusions visuelles

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Neurosciences computationnelles de la vision

Etienne Rey, SPECTRE AUDIOGRAPHIQUE – DIFFRACTION

Anatomie du système visuel humain

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Modèles hybrides d’IA

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Art & Sciences

Etienne Rey

Tropique

Etienne Rey, Tropique

Tropique

Etienne Rey, Cristal n2

Etienne Rey, TRAME ÉLASTICITÉ

TRAME ÉLASTICITÉ

De la nature des choses

Phyllotaxie

Par Cmglee — Travail personnel, CC BY-SA 4.0, Lien

Etienne Rey, Densité flou

Etienne Rey, Horizon Faille

Caustiques

La vibration des apparences

Paul Cézanne, Montagne Sainte-Victoire, 1904

La vibration des apparences

Merleau-Ponty, Sens et non-sens

Etienne Rey, Trames

La vibration des apparences

N_rho, N_phi = 34, 233

def retino_grid(cr, N_rho, N_phi, N_H, N_V, offset, size_mag,
 ecc_max, alpha, c1, c2, power, operator,
 channel='both'):

 cr.scale(N_H, N_V)
 cr.set_operator(operator)

 # https://laurentperrinet.github.io/sciblog/posts/2020-04-16-creating-an-hexagonal-grid.html
 phi_v, rho_v = np.meshgrid(np.linspace(0, 2*np.pi, N_phi, endpoint=False),
 np.linspace(0, ecc_max, N_rho+1, endpoint=True)[1:], sparse=False, indexing='xy')
 phi_v[::2, :] += np.pi/N_phi

 offsets = [-offset, offset]
 colors = [c1, c2]

 for offset_, color in zip(offsets, colors):
 # convert to cartesian coordinates
 X = rho_v * np.sin(phi_v) + offset_
 Y = rho_v * np.cos(phi_v)
 X = (X+1)/2
 Y = (Y+1)/2
 R = size_mag * rho_v**power / N_rho

 # draw 
 for x, y, r in zip(X.ravel(), Y.ravel(), R.ravel()):
 circle(cr, x, y, r)
 cr.set_source_rgba(*hue_to_rgba(color, alpha))
 cr.fill()

 return cr

c_blue = 240
dc = 60
opts = dict(N_rho=N_rho, N_phi=N_phi, N_H=N_H, N_V=N_V,
 offset=0.07, size_mag=0.3, ecc_max=0.8, alpha=0.80, c1=c_blue-dc, c2=c_blue+dc, power=.5, operator=cairo.OPERATOR_MULTIPLY)

@disp
def draw(cr, N_H=N_H, N_V=N_V): cr = retino_grid(cr, **opts)

Etienne Rey, La vibration des apparences

La vibration des apparences

Laurent Perrinet

Journées d’Ouverture Scientifique (JOS)

[2025-04-18]

Art-Sciences / Contact me @ laurent.perrinet@univ-amu.fr

2025-03-11-phd-program-sparse-representations

Tue, 11 Mar 2025 00:00:00 +0000

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2025-03-11]

Code / Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations?

Paysage catalan (Le Chasseur)

to rephrase the expression “The Unreasonable Effectiveness of Mathematics” by Wigner, the “Unreasonable efficiency of vision” is playfully illustrated in this painting from Joan Miró, which allows us to depict this Catalan landscape with the a few strokes where our imagination will fill the gaps and signify the landscape, allowing us to imagine the hunter, the sardine or the plane.

the whole is the sum of a few parts

Sparse coding is a technique used in signal processing and machine learning to represent data in a more concise and efficient manner. It aims to find a sparse representation of the data, which means representing the data with only a small number of non-zero coefficients or activations. In sparse coding, a set of basis functions or atoms is typically defined, and the goal is to find a linear combination of these atoms that best represents the input data. The coefficients of this linear combination are often constrained to be sparse, meaning that only a few of them are allowed to be non-zero.

Sparse representations in computer vision

Sparse representations in computer vision

Sparse representations in computer vision

[LP and Bednar, 2015]

Sparse representations in computer vision

[LP, 2021]

Sparse representations in neuromorphic engineering

Ultimately, we get a list of events for each pixel that can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence, and polarities. Note that since events are generated over time, they are naturally sorted by their time of occurrence. These events are then transmitted in real time to the output bus, often via a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve that connects our retina to the brain. In fact, the output of the retina consists of a million ganglion cells that emit action potentials, which are the only source of information transmitted by the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Sparse representations in neuroscience

[Brunel, 2001]

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations?

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations in a nutshell

Neural implementation = gradient descent

LASSO = least absolute shrinkage and selection operator

Orthogonal Matching Pursuit (OMP): OMP is an iterative algorithm used for sparse signal recovery. It starts with an initial sparse solution and iteratively selects the most correlated dictionary atoms with the residual signal. OMP aims to minimize the L2 norm of the residual while maintaining sparsity. It has a greedy nature and can provide a near-optimal sparse solution.

Basis Pursuit (BP): Basis Pursuit is an optimization problem that seeks the sparsest solution to an underdetermined linear system of equations. It involves minimizing the L1 norm of the coefficient vector subject to a linear constraint. BP can be solved using linear programming techniques or convex optimization algorithms.

Iterative Soft Thresholding Algorithm (ISTA): ISTA is an iterative optimization algorithm commonly used in sparse coding. It alternates between a gradient descent step and a soft thresholding step. The gradient descent step minimizes the data fidelity term, and the soft thresholding step enforces sparsity by setting small coefficients to zero. ISTA converges to a sparse solution and can be used for dictionary learning.

FISTA (Fast Iterative Shrinkage-Thresholding Algorithm): FISTA is an accelerated version of ISTA that improves convergence speed. It incorporates momentum into the optimization process and achieves faster convergence rates.

ADMM (Alternating Direction Method of Multipliers): ADMM is an optimization technique that decomposes the original problem into smaller subproblems and solves them iteratively. It is often used for convex optimization problems with L1 regularization. ADMM has been applied to solve sparse coding problems efficiently.

Matching pursuit algorithm

Init : Residual $R = I$, sparse vector $a$ such that $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- compute $c[i] = \sum_{x, y} (R[x, y] - a[i] \cdot \phi[i, x, y])^2$
- Match: $i^\ast = \arg \min_i c[i]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} ( I[x, y] \cdot \phi[i, x, y])$
- Assign : $a[i^\ast] = \frac{\sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]}{\sum_{x, y} \phi[i^\ast, x, y] \cdot \phi[i^\ast, x, y]}$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, and normalize $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$
- Pursuit : $R[x, y] \leftarrow R[x, y] - a[i^\ast] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
compute $c[i] = \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
compute $X[i, j] = \sum_{x, y} \phi[i, x, y] \cdot \phi[j, x, y]$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i c[i]$
- Assign : $a[i^\ast] = c[i^\ast]$
- Pursuit : $c[i] \leftarrow c[i] - a[i^\ast] \cdot X[i, i^\ast] $

[LP (2004)]

Matching pursuit algorithm

Matching pursuit algorithm

Hebbian learning (once the sparse code is known):

$$ \phi_{i}[x, y] \leftarrow \phi_{i}[x, y] + \eta \cdot a[i] \cdot (I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi_{i}[x, y] ) $$

Matching pursuit algorithm

Sparse representations in a nutshell

Convolutional Sparse Coding

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolution: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolution: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolution: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolution: Mathematics

Convolution: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolution: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Convolutional Sparse Coding

Convolutional Sparse Coding

Convolutional Sparse Coding

[LP, 2015]

Code @ SparseEdges

Convolutional Sparse Coding

[Ladret et al, 2024]

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2025-03-11]

Code / Contact me @ laurent.perrinet@univ-amu.fr

2025-02-14-supaero

Fri, 14 Feb 2025 00:00:00 +0000

Qu'est-ce que les Neurosciences peuvent apporter à l'Intelligence Artificielle ?
[2025-02-14] Airbus Helicopters Laurent Perrinet ― https://laurentperrinet.github.io

Mais que fait un neuroscientifique à Airbus Helictopters?

Je suis moi-même un passionné d’aéronautique et de spatial, ce qui m’a amené à suivre l’école d’aéronautique SUPAERO. Puis vers l’imagerie satellitaire, qui dépendait déjà de l’IA sous la forme des réseaux de neurones. C’est à partir de là, grâce à la rencontre avec mon professeur de mathématiques Manuel Samuelides, que j’ai découvert les neurosciences computationnelles et les pouvoirs qu’elles peuvent offrir pour mieux comprendre le cerveau et pour créer de nouveaux systèmes d’intelligence artificielle. Voici un

Le jury était consistué (de gauche à droite) de Jeanny Hérault (Rapporteur), Michel Imbert (Président), Yves Burnod (Rapporteur, absent de la photo), Manuel Samuelides (Directeur de thèse) et Simon Thorpe (Co-directeur de thèse).

Depuis ce temps là, je développe des réseaux de neurones concus comme des algorithmes / processus d’optimisation numérique, que j’applique pour le traitement automatisé des images. une optique nouvelle n’est pas simplement d’utiliser l’inspiration neuro-mimétique mais de faire des aller retours avec l’expérimentation

mais d’abord quid AI ?

L’intelligence artificielle est-elle “intelligente” ?

L’intelligence artificielle (IA) est-elle “intelligente” ?

L’IA est une science multi-disciplinaire qui vise à créer des machines capables d’exécuter des tâches intelligentes, similaires à celles effectuées par l’être humain.
années 1950-1970 : approches logiques et symboliques
années 1980-2010 : machine learning (apprentissage automatique)
années 2010-2020 : deep learning
années 2020-… : la révolution des transformers

L’intelligence artificielle est-elle “intelligente” ?

Sommet de l’IA de 2025

Enjeux de l’IA embarquée : latence de réponse

Visual latencies [Grimaldi et al, 2022]

Tout d’abord, les systèmes sensoriels biologiques sont composés de séquences de traitement qui possèdent des délais de traitement. Je décris ici la chaîne de traitement d’une image visuelle, ici pour un enfant jouant à un jeu et devant cliquer sur le bon bouton, et qui illustre les différentes latences du traitement de l’information de la vision à l’action.

Si les délais dans un système embarqué sont plus rapides, il reste que les informations dans les différentes étapes de traitement peuvent être décalées et nécessitent un traitement adapté afin de répondre de la façon la plus immédiate possible. Je pense notamment à la détection d’objets en mouvement très rapide dans le cadre spatial.

Tout d’abord, la plupart de ces systèmes traitent des données statiques. Ils ignorent notamment l’aspect dynamique, comme la nécessité de pouvoir répondre à tout moment ou de compenser les délais de traitement. Dans un premier temps, je présenterai un nouveau type de caméra, inspirée du fonctionnement de la rétine et du codage neural par potentiels d’actions ou « spikes ». Ces caméras permettent de capturer l’information sous forme d’événements et nécessitent d’adapter les algorithmes de traitement de l’information, qui sont plus proches de ceux utilisés par le cerveau.

Enjeux de l’IA embarquée : budget énergétique

“L’irraisonnable efficacité de la vision”

Comment la vision a évolué… [Perrinet, 2024]

Illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Illusions visuelles

Illusions visuelles

Illusions visuelles

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Illusions visuelles : Paréidolie

Neurosciences computationnelles de la vision

Anatomie du système visuel humain

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Cortex visuel primaire

[Hubel & Wiesel, 1962]

Modèles hybrides d’IA

Using goal-driven deep learning models to understand sensory cortex [Yamins & DiCarlo, 2016]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

From frame-based to event-based cameras.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Les caméras événementielles présentent plusieurs propriétés qui les rendent remarquables. Tout d’abord, la précision temporelle des événements est de l’ordre de la microseconde, ce qui permet d’atteindre une cadence théorique de l’ordre du million d’images par seconde. On peut la comparer à celle d’une caméra classique, qui est de l’ordre de la centaine d’images par seconde, ou à celle d’une caméra à grande vitesse, qui peut atteindre 10 000 images par seconde. Il est difficile d’estimer la fréquence d’échantillonnage de la perception humaine, car si 25 images par seconde sont souvent suffisantes pour visionner un film, il a été démontré que l’œil humain peut distinguer des détails temporels jusqu’à la milliseconde.

Une autre caractéristique importante de ces caméras est leur capacité à détecter une très large gamme de luminosité, dépassant de loin celle des caméras conventionnelles à 120 dB (un facteur d’un million, comparé au facteur de un sur mille de l’œil humain entre la pleine lune et le soleil),

Il convient de noter que la « résolution spatiale » de ces caméras est souvent relativement modeste, de l’ordre du mégapixel. Cependant, il ne s’agit pas d’une limitation technique, mais plutôt d’une conséquence des applications technologiques dans lesquelles ces caméras sont couramment utilisées.

Par rapport aux caméras classiques, qui consomment plusieurs watts, les caméras événementielles consomment très peu d’énergie électrique, de l’ordre de 10 milliwatts, soit une consommation équivalente à celle de l’œil humain. https://en.wikipedia.org/wiki/Event_camera#Functional_description

Levier #1: Réseaux de neurones impulsionnels (SNNs)

[Grimaldi et al, 2023, Precise Spiking Motifs]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Loihi 2

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Ces caméras ne présentent que des avantages, mais alors, comment traiter cette nouvelle représentation des données ? En effet, les neurosciences montrent que les neurones ne manipulent pas des données continues (comme ceux du deep learning), mais communiquent exactement de la même manière en échangeant de brèves impulsions prototypiques, les potentiels d’action (spikes).
Notre solution : une architecture similaire au deep learning, mais chaque neurone (brique élémentaire) est un modèle simplifié de neurone biologique impulsionnel. Cependant, nous nous retrouvons avec un problème par rapport à l’établissement que nous avons réussi à résoudre théoriquement. Un avantage supplémentaire est que ce genre de calcul est actuellement développé sur des puces embarquées (comme les pixels de la caméra évanementielle).
notre architecture fonctionne ainsi directement sur cette même représentation. Un autre avantage : le « always on computing ».

Quels résultats ? Peut-on les évaluer avant d’avoir ces puces ?

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Time-to-Contact maps [Nunes et al, 2023]

2 MINUTES

Nos simulations montrent ainsi une très grande efficacité (ici pour catégoriser un type de flux optique, ce qui peut guider la navigation).
un aspect innovant de notre technologie réside dans notre capacité à utiliser autant de neurones, mais moins de connexions. Nous avons par ailleurs montré que l’efficacité restait acceptable. Par rapport à une technologie classique (en orange) qui montre une baisse rapide, nos résultats montrent une bonne efficacité avec une demi-valeur critique donnée pour un gain de 700x (noter l’axe log). C’est ce qu’on appelle le « frugal computing » et nous œuvrons maintenant à son implémentation dans un PEPR IA.
c’est une étape importante, mais on peut aller plus loin, et je vais vous présenter un deuxième levier : éviter de tout traiter pour ne traiter que ce qui est nécessaire.

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

[Kremkow et al, 2018]

Levier #2: Vision active / Active Vision

---
## Levier #2: Vision active / *Active Vision*




[Jérémie et al, 2024]

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

From: https://pressbooks.umn.edu/sensationandperception/chapter/columns-and-hypercolumns-in-v1/

Qu'est-ce que les Neurosciences peuvent apporter à l'Intelligence Artificielle ?
[2025-02-14] Airbus Helicopters Laurent Perrinet ― https://laurentperrinet.github.io

2025-02-11-neuromath

Tue, 11 Feb 2025 00:00:00 +0000

[2025-02-11] When Cortical Neurons Talk Sideways: Beyond Feedforward Visual Processing

Laurent Perrinet - https://laurentperrinet.github.io Séminaire Neuromathématiques, Collège de France

Hi, thanks for the introduction! I am Laurent Perrinet, a researcher in computational neuroscience and currently a research director at CNRS at the Institute of Neuroscience of la Timone in Marseille. Thank you for inviting me to participate in this “NeuroMathematics” seminar at the intersection of mathematics and neuroscience.

As an engineer by training, I could have pursued a career in aeronautics rather than becoming a neuroscientist. It is thanks to my mathematics professor Manuel Samuelides that I discovered the beauty of neural networks at the end of my engineering studies. This developped a curiosity, and thanks to him, I was also able to study in a mastere of cognitive sciences (now called CogMaster) in 1998. This is where I particularly want to acknowledge Jean Petitot - for his course I discovered how natural image statistics could link to principles in the central nervous system. This was a vivid revelation, and I’m grateful for his guidance in my academic path. Today’s seminar represents a return to these roots, as I’ll present my research progress since my mastere thesis on this very topic.

Today, I will address our current knowledge about horizontal connectivity rules in V1. Why is this important? As a matter of fact, one main function of sensory systems, such as the pivotal role of the primary visual cortex for vision, is to bind together the different visual features to help ultimately build a global perception.

Paysage catalan (Le Chasseur), Joan Miró (1924)

Trames (Etienne Rey)

Contour detection and the Association Field

[Field et al, 1993]

Contour detection and the Association Field

[Field et al, 1993]

Contour detection and the Association Field

[Field et al, 1993]

Natural Images : Edges are on a common circle

[Sigman et al, 2001]

A significant contribution to understanding the association field came from studying edge co-occurrences in natural images by Sigman et al. (2001). They quantified the probability density function of edge co-occurrences based on their relative positions and orientations. The figure demonstrates this by showing the spatial distribution patterns for edges relative to a reference edge at different orientations. For iso-oriented edges (a), the co-occurrence pattern shows clear structure. As the relative orientation increases through 22.5° (b), 45° (c), 67.5° (d), to 90° (e), distinct spatial patterns emerge.

A key finding was that for any given relative orientation between edges, the angle of maximal interaction occurs at the bisector between the orientations. This suggests that co-occurring edges tend to lie on a common circle - a property known as cocircularity. Panel (f) illustrates this geometrical principle: given two edges at angles w (red, 20°) and c (blue, 40°), the cocircularity solutions (green lines at 30° and 120°) represent the possible orientations of connecting circular arcs. This mathematical relationship provides insights into how the visual system might leverage statistical regularities in natural scenes for contour integration. We will go back into the details of this a bit further in the talk.

This association field concept provided a compelling framework for understanding how the visual system may implement contour integration through neural connectivity patterns. but before going there we should go back to the basic anatomy of the visual cortex.

Contour detection and the Association Field

[Field et al, 1993]

Dynamics of vision

Human Visual system (Grimaldi et al 2022)

Anatomy of the Human Visual system

Human Visual system (Grimaldi et al 2022)

Thalamic, short- & long-range lateral, inter-areal

A key feature of primary visual cortex is its layered organization, which is shared across cortical areas. The main thalamic input arrives in layer 4, which connects to a dense network of vertical connections across layers. These columns can then communicate via horizontal connections within layers.

Hubel and Wiesel also proposed the ice-cube model that every point in the visual field produces a response in a 2 mm x 2 mm area of the cortex. Such an area can contain two complete groups of ocular dominance columns, 16 blobs and interblobs that may contain more than two times all of the orientations possible across 180 degrees. This region of the cortex, which Hubel and Wiesel called a hypercolumn (or, more generally, a cortical module) seems both necessary and sufficient for analyzing the image of a point in visual space. Because the cortex is a continuous cellular layer and because it is very hard to establish the boundaries of these modules physically, their existence from a functional standpoint is still the subject of debate. https://thebrain.mcgill.ca/flash/a/a_02/a_02_cl/a_02_cl_vis/a_02_cl_vis.html

Figure 9.2. Hypercolumn Diagram. Ocular dominance columns are segregated into left and right eye inputs. Orientation columns are neurons that get excited at different orientations and a cluster of these is called a pinwheel. Blobs are color selective and for every pinwheel there is a blob. (Credit: McGill: The Brain from Top to Bottom, Figure of hypercolumns, Copyleft https://copyleft.org/, https://thebrain.mcgill.ca/flash/a/a_02/a_02_cl/a_02_cl_vis/a_02_cl_vis.html. No modifications.)

Thalamic, short- & long-range lateral, inter-areal

[Markov et al 2011]

Anatomy of the Primary Visual Cortex

[Kaschube et al (2010)]

V1 is central to these pathways and shows distinctive anatomical and functional properties along with a complex topographical organization.

This figure from Kaschube et al. (2010) illustrates the organization of orientation preference maps in primary visual cortex (V1). Individual V1 neurons exhibit selective responses to oriented visual stimuli (as denoted by varying hues Colors code preferred ORs as indicated by the bars in (C)), with their spatial arrangement following highly structured patterns across the cortical surface. Panel B shows Synthetic orientation-maps of equal column spacing Λ but widely different pinwheel densities ρ. Left to right: solutions of different models: (13–16).. (C) High (blue frame) and low (orange frame) pinwheel density regions in tree shrew visual cortex. (D to F), Optically recorded orientation-maps in tree shrew (D), galago (E), and ferret (F) visual cortex. Regions shown in (C) are marked in (D). White arrows in (F) mark selected pinwheel centers. Framed regions in (C) and (F) are magnified. In many mammals including cats, monkeys and ferrets, orientation preference is organized in a quasi-periodic manner, forming what are known as orientation preference maps. These maps show remarkable consistency in their geometric properties across species, particularly in the spatial organization of pinwheel centers where orientation preferences converge.

However, this organization shows important species-specific variations. Most notably, while primates and carnivores display orderly orientation maps with smooth transitions between preferred orientations, rodents lack such maps and instead show a “salt-and-pepper” arrangement where neighboring neurons have seemingly random orientation preferences. This organizational diversity raises interesting questions about the computational advantages of these different architectures and their relationship to visual processing requirements and behavioral needs across species.

Horizontal connectivity links different hypercolumns

[Bosking et al, 1997]

Contour detection and the Association Field

[Field et al, 1993]

The like-to-like hypothesis

[Field et al, 2013]

The resemblance between what was shown by Bosking and the structure of the association field that we saw above is such that it is tempting to align both and state that the function of horizontal connections is to bind neurons with a selectivity to similar orientations* over long distances. This like-to-like hypothesis has been influential in understanding horizontal connectivity patterns.

However, we should be cautious about overstating these relationships. While horizontal connections show some orientation specificity, recent evidence indicates the connectivity patterns are more complex and heterogeneous than initially proposed. The functional role of this diverse connectivity remains an active area of investigation.

During the remainder of this talk, I will try to shed light on our current knowledege on horizontal connectivities.

Supplementary: the HMAX model

[Serre and Poggio, 2007]

Supplementary: Convolutional Neural Nets (CNN)

[Chavane, LP and Rankin, 2022]

Supplementary: Orientation selectivity in V1

[Hubel & Wiesel, 1962]

Supplementary: Orientation selectivity in V1

[Hubel & Wiesel, 1962]

Supplementary: Marr’s three levels of analysis

Marr, 1982

cut in different levels: Marr (+ Poggio)
arbitrary, but useful division of labor= computational / algorithm / hardware
here:
anatomy
algorithm / model
function

First: What is the anatomy of horizontal connections?

<!--

[Marr, 1982]]

-->

Challenging the like-to-like hypothesis

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

This figure illustrates different hypothetical connectivity rules for horizontal connections in V1. The target neuron (large circle on left) has a specific orientation preference indicated by its color. Following the classical like-to-like hypothesis (shown in panel A), this neuron would preferentially connect to other neurons with matching orientation preference (similar colors) across multiple hypercolumns, as indicated by the vertical red arrows. The radial spread of connections spans approximately three hypercolumns, consistent with anatomical observations. Each hypercolumn contains a complete set of orientation preferences, represented by the different colored neurons.

This first schematic (noted A) represents one of the like-to-like connectivity rules, where horizontal connections strictly follow orientation similarity.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Let’s first shows some functional evidence.

This video shows voltage-sensitive dye imaging (VSDI) data from cat primary visual cortex (area 17) in response to a local oriented grating stimulus. The visualization reveals two key aspects:

The broader activation pattern shown by overall fluorescence changes (gray)
The more restricted orientation-selective response pattern (colored regions)

Two contours are overlaid: a red line marking the boundary of significant activation, and a white line delineating regions with statistically significant orientation selectivity. The orientation selectivity is encoded by color hue.

The bottom plots quantify the spatiotemporal dynamics by showing:

Left: The total activated cortical area over time
Right: The extent of orientation-selective regions over time

Together, these measurements demonstrate how orientation-selective signals propagate laterally beyond the classical feedforward input zone through horizontal connections, while maintaining some degree of feature selectivity.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

This figure shows spatial and temporal dynamics of orientation selectivity in cat V1 analyzed from voltage-sensitive dye imaging data. Panel A displays a cortical orientation map averaged over the final 145ms of the response, where hue indicates preferred orientation and brightness shows orientation tuning strength. The dotted red line delineates the expected retinotopic boundary of feedforward input based on Albus (2004).

The inset quantitatively compares the spatial extent of:

Total cortical activation (grey contour)
Orientation-selective activation (black contour)
Theoretical feedforward input boundary (red contour)

This data demonstrates that orientation-selective responses propagate laterally beyond the classical feedforward input zone through horizontal connections, while maintaining some degree of feature selectivity. The systematic comparison between total activation and selective activation provides direct evidence for how horizontal connectivity shapes the spatiotemporal dynamics of orientation processing in V1.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel C displays intracellular recordings of subthreshold responses visualized as a visuotopic orientation polar map. The color hue represents preferred orientation while brightness indicates the strength of orientation tuning in the membrane potential. White contours outline regions showing statistically significant responses based on both amplitude and orientation selectivity criteria. The middle plots show averaged subthreshold responses to four different oriented stimuli (color-coded) at specific recording locations (marked by circle, triangle and square symbols), with scale bars indicating 50 ms and 1 mV. On the right, normalized orientation tuning curves are shown, computed by integrating responses within a fixed temporal window (shaded region in middle panel). The black circle marks the spontaneous activity level for the depolarizing integral measurement.

These shows a direct functional evidence for a diversity of tuning profile in th horizontal connectivity.

Challenging the like-to-like hypothesis

[Voges and LP, 2012]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Figure 4 illustrates a neural field model that bridges anatomical structure with functional observations in V1, as developed by Rankin and Chavane (2017).

Panel A depicts radial connectivity profiles with Gaussian-decaying inhibition and distance-dependent excitation that peaks periodically at multiples of distance L. The Ring Width (RW) parameter controls the spread of these excitatory peaks.

Panel B shows how local orientation preference maps influence lateral connectivity patterns under different orientation bias (BR) values in the recurrent connections.

Panel C quantifies the orientation tuning that emerges from these connectivity patterns. While orientations are uniformly represented globally, the local excitatory component shows strong bias around -60°. As BR increases above 0.5, the lateral connection orientation bias strengthens, reaching values around k=1 (consistent with Buzás et al. 2006).

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel F maps the normalized selective area (relative to the feedforward footprint) across Ring Width (RW) and orientation bias (BR) parameters. White contours delineate anatomically plausible ranges where k values fall between 0.7-1.2, consistent with experimental measurements. The green region indicates parameter combinations that additionally satisfy constraints on both orientation preference and the observed radial decay of selectivity.

The neural field model effectively connects anatomical connectivity patterns with functional observations of orientation selectivity propagation in V1. The resulting connectivity structure exhibits similarities with “association field” patterns, suggesting potential optimization for encoding natural image statistics. This framework provides a quantitative basis for investigating computational principles underlying horizontal connectivity in visual cortex.

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Modelling the Association field

[Field et al, 2013]

Edge co-occurences in natural images

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

Panel A illustrates the groundbreaking approach developed by Geisler et al. (2001) for analyzing edge statistics in natural images. The method involves:

Detecting oriented edge elements in natural images (shown as red segments)
For each edge pair, measuring:
- Their relative orientation difference (𝜃)
- The relative position angle (𝜙)
- Center-to-center distance (d)
- Azimuth difference (φ)
- Co-circularity parameter ψ = φ - θ/2

This quantitative analysis reveals two key distributions:

A strong bias for parallel edge arrangements, evident in the orientation difference histogram
A marked preference for co-circular alignments, shown in the relative position histogram

These statistics vary significantly across image databases. For example, images containing animals exhibit enhanced co-circularity compared to general natural scenes. This suggests that rather than implementing a single fixed association field, the visual system may need to handle diverse statistical regularities present in natural inputs.

The next section will examine how these statistical regularities inform computational models of the association field.

Edge co-occurences in natural images

[Geisler, 2001]

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

The probability distribution function p(ψ,θ) represents the distribution of the different geometrical arrangements of edges’ angles, which we call a “chevron map”. We show here the histogram for non-animal natural images, illustrating the preference for co-linear edge configurations. For each chevron configuration, deeper and deeper red circles indicate configurations that are more and more likely with respect to a uniform prior, with an average maximum of about 3 times more likely, and deeper and deeper blue circles indicate configurations less likely than a flat prior (with a minimum of about 0.8 times as likely). Conveniently, this “chevron map” shows in one graph that non-animal natural images have on average a preference for co-linear and parallel edges, (the horizontal middle axis) and orthogonal angles (the top and bottom rows), along with a slight preference for co-circular configurations (for ψ =0 and ψ = ± π/2, just above and below the central row).

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

This figure shows classification performance across image categories using different statistical features. We used an SVM classifier with three feature sets: first-order orientation statistics (FO), the reduced 2D “chevron map” (CM), and full 4D second-order statistics (SO). The classification accuracy (F1 score) was tested for distinguishing between image categories. Results show strong performance in separating man-made from natural images, as expected. More notably, the classifier achieved ~80% accuracy in discriminating animal vs non-animal natural images, matching human performance levels reported by Serre et al. This suggests that relatively simple edge co-occurrence statistics contain sufficient information for basic image categorization tasks, without requiring higher-level semantic processing.

We also found that our model made the same errors as humans do: if an image without an animal contains more co-circular edges, it is more likely to be falsely categorized as containing an animal.

Edge co-occurences in natural images

Edge co-occurrences can account for rapid categorization of natural versus animal images [LP and Bednar, 2015]

Can we explain the diversity ?

[Bosking et al, 1997]

Indeed, this diversity is revealed in the anatomical data: V1 horizontal connectivity exhibits more complexity than suggested by the classical like-to-like hypothesis. While orientation-specific connections exist, they coexist with non-selective connections that link neurons irrespective of their tuning preferences. This diversity likely serves multiple computational functions:

Specific connections could support contour integration and feature binding
Non-selective connections may enable broad contextual modulation
Mixed connectivity patterns could help maintain network stability while preserving functional specificity

This anatomical heterogeneity aligns with V1’s role in both specialized feature detection and broader contextual processing. Understanding how these distinct connectivity patterns interact remains an active area of research in visual neuroscience.

Predictive processing

Predictive processing

To bridge the gap between anatomical observations and functional requirements of visual processing, We added two key ingredients in the sparse deep predictive coding (SDPC) model :

Sparse connectivity patterns:
- Enforcing regularization of the activity map using L1 penalty
- Activity computed via recurrent local connectivity
- Similar to biological observations
Feedback from efferent layers:
- Predicts activity of afferent layer
- Only residual prediction error is processed
- Defines long-range inter-areal connectivity
- Specific influence demonstrated in Neural Computation paper

By defining a cost on minimizing the prediction error in each layer, everything stays derivable, such that we can use a classical gradient descent. These additions should allow us to better understand how feedback shapes visual processing in biological neural networks.

Predictive processing

Predictive processing

More specifically in the context of our focus today, we can look at the co-occurence

llustration of the procedure to generate interaction map. In this illustrative example we consider a V1 representation with only 4 feature maps (represented in the upper-left box). Step 1 is to extract a neighborhood (of size 3x3 in the illustration only) around the most strongly activated neuron (represented with a red square in the illustration) for a given central preferred orientation (denoted ✓ c ). Step 2 is to normalize the neural activity in the extracted neighborhood using the marginal activity (see Eq.8). Step 3 is to compute the resulting orientation and activity at every position of the neighborhood using a circular mean (see Eq. 11 and Eq. 12 respectively). To keep a concise figure we have illustrated the computation of the central edge of the interaction map only. For simplification, the illustration shows only 1 neighborhood extraction whereas the interaction maps shown in the paper are computed by averaging neighborhoods centered on the 10 most strongly activated neurons

Predictive processing

Predictive processing

Predictive processing with pooling

Challenging the like-to-like hypothesis

[Bosking et al, 1997]

Challenging the like-to-like hypothesis

Revisiting Horizontal Connectivity Rules in V1: From like-to-like towards like-to-All [Chavane, LP and Rankin, 2022]

To conclude, our review of horizontal connectivity in V1 reveals patterns more complex than initially theorized. The classical like-to-like hypothesis, while valuable, doesn’t fully capture the diversity of observed connectivity patterns.

Mathematical modeling has proven essential in bridging theory and biology. Our predictive processing framework shows how simple computational principles can explain the emergence of these complex connectivity patterns. The model demonstrates how feedback influences lateral interactions and reproduces key experimental observations.

However, important questions remain unanswered. We need to better understand how precise timing information is encoded in these circuits, how temporal dynamics shape processing, and whether similar principles apply across other cortical areas.

These fundamental questions will guide future experimental and theoretical work as we continue to unravel the computational principles of cortical processing.

[2025-02-11] When Cortical Neurons Talk Sideways: Beyond Feedforward Visual Processing

Laurent Perrinet - https://laurentperrinet.github.io Séminaire Neuromathématiques, Collège de France

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Dynamics of vision: Neural modeling

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

One advantage of this network is that it is differentiable, enabling us to apply classical machine learning methods, notably supervised learning. We then see the emergence of different convolution kernels, and here I represent a subset of its kernels for different directions, as denoted by the red arrows on the left of the graph. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2024-11-18-journee-biomometisme

Mon, 18 Nov 2024 00:00:00 +0000

[2024-11-18] NeuroAI: interactions multiples entre Neurosciences et Intelligence artificielle

Laurent Perrinet - https://laurentperrinet.github.io

outline =
Bonjour. Je suis Laurent Perrinet, directeur de recherche CNRS en neurosciences computationnelles à l’Institut de Neurosciences de la Timone à Marseille. Je vous remercie pour cette invitation à participer à cette Journée Scientifique “Biomimove 2024 : Action, Perception et Traitement” à la croisée entre robotique et science du vivant.
Les neurosciences computationnelles sont les sciences qui essaient d’extraire de nos connaissances en neurosciences biologiques des principes computationnels, comme le neurone formel et sa capacité d’apprentissage, qui est la brique de base des réseaux de neurones. Ces derniers ont conduit à la révolution de l’IA avec les réseaux profonds.
Je suis convaincu que nous sommes au tournant d’une nouvelle ère dans le développement des systèmes embarqués, où l’intelligence artificielle a le potentiel de créer des innovations disruptives à la hauteur des performances de l’intelligence naturelle et pour lesquelles il est essentiel de s’inspirer des neurosciences biologiques.
C’est pourquoi je suis très heureux de vous présenter en premier lieu le projet ANR AgileNeuRobot, un projet de recherche interdisciplinaire visant à développer des robots aériens agiles bio-mimétiques pour le vol en conditions réelles.

Dans cette optique, afin de caractériser certains enjeux de l’IA embarquée, notamment dans le domaine du spatial, je vais vous présenter deux leviers s’inspirant de la biologie et illustrant comment les neurosciences peuvent faire avancer le domaine de façon radicale. L’intégration de connaissances biomimétiques dans les engins embarqués peut améliorer leur résilience et leur adaptabilité face aux environnements hostiles, tout en réduisant la consommation d’énergie. Je serais ravi d’engager ensuite une discussion avec vous sur ces sujets et d’échanger sur vos propres expériences et perspectives.

AgileNeuRobot: Fiche d’identité

Titre : Robots aériens agiles bio-mimetiques pour le vol en conditions réelles
Title : Bio-mimetic agile aerial robots flying in real-life conditions
CES : CE23 - Intelligence Artificielle (ANR-20-CE23-0021)
Durée: 4 ans, du 1er Octobre 2021 au 30 Septembre 2025
Budget total: 435 k€

AgileNeuRobot: Consortium:


Stéphane Viollet	Ryad Benosman	Laurent Perrinet
Julien Diperi	Sio-Hoï Ieng	Emmanuel Daucé
Post-doc 1	Post-doc 2	PhD (JN Jérémie)
Inst Sciences Mouvement	Inst de la Vision	Inst Neurosci de la Timone

Le projet AgileNeuRobot est un projet que je coordonne en collaboration avec plusieurs institutions :

l’Inst Sciences du Mouvement pour la partie robotique bio-inspirée,
l’Institut de la Vision pour le développement de nouveaux capteurs.
l’Institut de Neurosciences de la Timone (Aix-Marseille Université) pour l’aspect théorique et l’intégration de ces disciplines.

Ensemble, nous travaillons à la fois sur les aspects techniques et scientifiques pour créer ces robots aériens. Ce projet a pour but de contribuer non seulement au développement de nouvelles technologies, mais aussi à la compréhension et à l’élaboration de nouveaux modèles théoriques pour expliquer les mécanismes naturels sous-jacents aux capacités d’adaptabilité et d’apprentissage des systèmes biologiques.

AgileNeuRobot: Agile = Performant et efficace

The system includes 3 units to process event-driven visual inputs communicating by feed-forward and feed-back paths.

Le système en développement est un exemple de robotique bio-inspirée. Il est conçu pour être capable de traiter des données visuelles en temps réel et de réagir rapidement aux changements de l’environnement, notamment pour éviter ou intercepter des objets en vol.

performance : garder une bonne acuité tout en répondant rapidement et presque immédiatement.
efficacité : des besoins réduits en énergie pour un fonctionnement autonome.

Pour cela, nous avons utilisé une architecture inspirée des insectes qui combine des capteurs événementiels avec des réseaux de neurones impulsionnels pour créer un système agile et performant, que je vais décrire dans la suite de l’exposé.

Mais d’abord, je voudrais souligner deux contraintes majeures de ce type de systèmes embarqués :

Enjeux de l’IA embarquée : latence de réponse

Visual latencies [Grimaldi et al, 2022]

Enjeux de l’IA embarquée : budget énergétique

Prototype avec caméra événementielle et calculateur.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

From frame-based to event-based cameras.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Il convient de noter que la résolution spatiale de ces caméras est souvent relativement modeste, de l’ordre du mégapixel. Cependant, il ne s’agit pas d’une limitation technique, mais plutôt d’une conséquence des applications technologiques dans lesquelles ces caméras sont couramment utilisées.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Ces caméras ne présentent que des avantages, mais alors, comment traiter cette nouvelle représentation des données ? En effet, les neurosciences montrent que les neurones ne manipulent pas des données continues (comme ceux du deep learning), mais communiquent exactement de la même manière en échangeant de brèves impulsions prototypiques, les potentiels d’action (spikes).
Notre solution : une architecture similaire au deep learning, mais chaque neurone (brique élémentaire) est un modèle simplifié de neurone biologique impulsionnel. Cependant, nous nous retrouvons avec un problème par rapport à l’établissement que nous avons réussi à résoudre théoriquement. Un avantage supplémentaire est que ce genre de calcul est actuellement développé sur des puces embarquées (comme les pixels de la caméra évanementielle).
notre architecture fonctionne ainsi directement sur cette même représentation. Un autre avantage : le « always on computing ».

Quels résultats ? Peut-on les évaluer avant d’avoir ces puces ?

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Time-to-Contact maps [Nunes et al, 2023]

2 MINUTES

Nos simulations montrent ainsi une très grande efficacité (ici pour catégoriser un type de flux optique, ce qui peut guider la navigation).
un aspect innovant de notre technologie réside dans notre capacité à utiliser autant de neurones, mais moins de connexions. Nous avons par ailleurs montré que l’efficacité restait acceptable. Par rapport à une technologie classique (en orange) qui montre une baisse rapide, nos résultats montrent une bonne efficacité avec une demi-valeur critique donnée pour un gain de 700x (noter l’axe log). C’est ce qu’on appelle le « frugal computing » et nous œuvrons maintenant à son implémentation dans un PEPR IA.
c’est une étape importante, mais on peut aller plus loin, et je vais vous présenter un deuxième levier : éviter de tout traiter pour ne traiter que ce qui est nécessaire.

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

[2024-11-18] NeuroAI: interactions multiples entre Neurosciences et Intelligence artificielle

Laurent Perrinet - https://laurentperrinet.github.io

2024-09-09-agileneurobot-anr

Mon, 09 Sep 2024 00:00:00 +0000

Laurent Perrinet (https://laurentperrinet.github.io) [2024-09-09] Enjeux pour l'IA embarquée

outline =
Bonjour. Je suis Laurent Perrinet, directeur de recherche CNRS en neurosciences computationnelles à l’Institut des neurosciences de la Timone à Marseille. Je vous remercie pour cette invitation à participer à cette table ronde sur l’IA embarquée dans le domaine spatial. Je suis moi-même un passionné d’aéronautique et de spatial, ce qui m’a amené à suivre l’école d’aéronautique SUPAERO. Puis vers l’imagerie satellitaire, qui dépendait déjà de l’IA sous la forme des réseaux de neurones. C’est à partir de là, grâce à la rencontre avec mon professeur de mathématiques Manuel Samuelides, que j’ai découvert les neurosciences computationnelles et les pouvoirs qu’elles peuvent offrir pour mieux comprendre le cerveau et pour créer de nouveaux systèmes d’intelligence artificielle.
Les neurosciences computationnelles sont les sciences qui essaient d’extraire de nos connaissances en neurosciences biologiques des principes computationnels, comme le neurone formel et sa capacité d’apprentissage, qui est la brique de base des réseaux de neurones. Ces derniers ont conduit à la révolution de l’IA avec les réseaux profonds.
Je suis convaincu que nous sommes au tournant d’une nouvelle ère dans le développement des systèmes embarqués, où l’intelligence artificielle a le potentiel de créer des innovations disruptives à la hauteur des performances de l’intelligence naturelle et pour lesquelles il est essentiel de s’inspirer des neurosciences biologiques.
C’est pourquoi je suis très heureux de vous présenter en premier lieu le projet ANR AgileNeuRobot, un projet de recherche interdisciplinaire visant à développer des robots aériens agiles bio-mimétiques pour le vol en conditions réelles.

Dans cette optique, afin de caractériser certains enjeux de l’IA embarquée, notamment dans le domaine du spatial, je vais vous présenter deux leviers s’inspirant de la biologie et illustrant comment les neurosciences peuvent faire avancer le domaine de façon radicale. L’intégration de connaissances biomimétiques dans les engins spatiaux peut améliorer leur résilience et leur adaptabilité face aux environnements hostiles, tout en réduisant la consommation d’énergie. Je serais ravi d’engager ensuite une discussion avec vous sur ces sujets et d’échanger sur vos propres expériences et perspectives.

AgileNeuRobot: Fiche d’identité

Titre : Robots aériens agiles bio-mimetiques pour le vol en conditions réelles
Title : Bio-mimetic agile aerial robots flying in real-life conditions
CES : CE23 - Intelligence Artificielle (ANR-20-CE23-0021)
Durée: 4 ans, du 1er Octobre 2021 au 30 Septembre 2025
Budget total: 435 k€

AgileNeuRobot: Consortium:


Stéphane Viollet	Ryad Benosman	Laurent Perrinet
Julien Diperi	Sio-Hoï Ieng	Emmanuel Daucé
Post-doc 1	Post-doc 2	PhD (JN Jérémie)
Inst Sciences Mouvement	Inst de la Vision	Inst Neurosci de la Timone

Le projet AgileNeuRobot est un projet que je coordonne en collaboration avec plusieurs institutions :

l’Inst Sciences du Mouvement pour la partie robotique bio-inspirée,
l’Institut de la Vision pour le développement de nouveaux capteurs.
l’Institut de Neurosciences de la Timone (Aix-Marseille Université) pour l’aspect théorique et l’intégration de ces disciplines.

AgileNeuRobot: Agile = Performant et efficace

The system includes 3 units to process event-driven visual inputs communicating by feed-forward and feed-back paths.

performance : garder une bonne acuité tout en répondant rapidement et presque immédiatement.
efficacité : des besoins réduits en énergie pour un fonctionnement autonome.

Mais d’abord, je voudrais souligner deux contraintes majeures de ce type de systèmes embarqués :

Enjeux de l’IA embarquée : latence de réponse

Visual latencies [Grimaldi et al, 2022]

Enjeux de l’IA embarquée : budget énergétique

Prototype avec caméra événementielle et calculateur.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

From frame-based to event-based cameras.

Levier #1: Réseaux de neurones impulsionnels (SNNs)

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Ces caméras ne présentent que des avantages, mais alors, comment traiter cette nouvelle représentation des données ? En effet, les neurosciences montrent que les neurones ne manipulent pas des données continues (comme ceux du deep learning), mais communiquent exactement de la même manière en échangeant de brèves impulsions prototypiques, les potentiels d’action (spikes).
Notre solution : une architecture similaire au deep learning, mais chaque neurone (brique élémentaire) est un modèle simplifié de neurone biologique impulsionnel. Cependant, nous nous retrouvons avec un problème par rapport à l’établissement que nous avons réussi à résoudre théoriquement. Un avantage supplémentaire est que ce genre de calcul est actuellement développé sur des puces embarquées (comme les pixels de la caméra évanementielle).
notre architecture fonctionne ainsi directement sur cette même représentation. Un autre avantage : le « always on computing ».

Quels résultats ? Peut-on les évaluer avant d’avoir ces puces ?

Levier #1: Réseaux de neurones impulsionnels (SNNs)

The HD-SNN neural network [Grimaldi et al, 2023]

Time-to-Contact maps [Nunes et al, 2023]

2 MINUTES

Nos simulations montrent ainsi une très grande efficacité (ici pour catégoriser un type de flux optique, ce qui peut guider la navigation).
un aspect innovant de notre technologie réside dans notre capacité à utiliser autant de neurones, mais moins de connexions. Nous avons par ailleurs montré que l’efficacité restait acceptable. Par rapport à une technologie classique (en orange) qui montre une baisse rapide, nos résultats montrent une bonne efficacité avec une demi-valeur critique donnée pour un gain de 700x (noter l’axe log). C’est ce qu’on appelle le « frugal computing » et nous œuvrons maintenant à son implémentation dans un PEPR IA.
c’est une étape importante, mais on peut aller plus loin, et je vais vous présenter un deuxième levier : éviter de tout traiter pour ne traiter que ce qui est nécessaire.

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Levier #2: Vision active / Active Vision

Laurent Perrinet (https://laurentperrinet.github.io) [2024-09-09] Enjeux pour l'IA embarquée

2024-05-13-master-m-4-nc

Mon, 13 May 2024 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2024-05-13]Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2024-05-13] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research

Contact me @ laurent.perrinet@univ-amu.fr

2024-04-10-ue-neurosciences-computationnelles

Wed, 10 Apr 2024 00:00:00 +0000

Artificial neural networks applied to the understanding of biological vision

Laurent Perrinet

[2024-04-10] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

appear bent

effect of context -> 3D

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex

[Hubel & Wiesel, 1962]

Primary visual cortex

[Hubel & Wiesel, 1962]

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

CNN: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

CNN: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

CNN: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

CNN: Mathematics

CNN: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

CNN: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks (SNN)

SNN: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN in neurobiology

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN: Spiking motifs

Review on Precise Spiking Motifs.

SNN in neuromorphic engineering

From frame-based to event-based cameras.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

SNN in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks (SNN)

Artificial neural networks applied to the understanding of biological vision

Artificial neural networks applied to the understanding of biological vision

Laurent Perrinet

[2024-04-10] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

2024-04-17-phd-program-sparse-representations

Wed, 10 Apr 2024 00:00:00 +0000

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2024-04-17]

Code / Contact me @ laurent.perrinet@univ-amu.fr

Sparse representations?

Sparse representations in computer vision

Sparse representations in computer vision

[LP and Bednar, 2015]

Sparse representations in computer vision

[LP, 2021]

Sparse representations in neuromorphic engineering

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Sparse representations in neuromorphic engineering

The HD-SNN neural network.

Sparse representations in neuromorphic engineering

The HD-SNN neural network.

Sparse representations in neuroscience

[Brunel, 2001]

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations in neuroscience

Sparse representations?

Sparse representations in a nutshell

Sparse representations in a nutshell

Sparse representations in a nutshell

Generative model of image synthesis:

$I[x, y] = $ $\sum_{i=1}^{K} a[i] \cdot \phi[i, x, y]$ $ + \varepsilon[x, y]$

Where $\phi$ is a dictionary of $K$ atoms, $a$ is a sparse vector of coefficients, and $\varepsilon$ is a noise term.

Sparse representations in a nutshell

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

$$ \begin{aligned} \mathcal{L}(a) & = - \log Pr( a | I ) \\ & = - \log Pr( I | a ) - \log Pr(a) \\ \end{aligned} $$

Sparse representations in a nutshell

Given an observation $I$,

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L} = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_i ( a[i] \neq 0) $$

Sparse representations in a nutshell

The problem is formalized as an optimization problem $a^\ast = \arg \min_a \mathcal{L}(a)$ with:

$$ \mathcal{L}(a) = \frac{1}{2} \sum_{x, y} ( I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi[i, x, y])^2 + \lambda \cdot \sum_{i=1}^{K} | a[i] | $$

Sparse representations in a nutshell

[Rentzeperis et al (2023)]

Sparse representations in a nutshell

Neural implementation = gradient descent

LASSO = least absolute shrinkage and selection operator

Matching pursuit algorithm

Init : Residual $R = I$, sparse vector $a$ such that $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- compute $c[i] = \sum_{x, y} (R[x, y] - a[i] \cdot \phi[i, x, y])^2$
- Match: $i^\ast = \arg \min_i c[i]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} ( I[x, y] \cdot \phi[i, x, y])$
- Assign : $a[i^\ast] = \frac{\sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]}{\sum_{x, y} \phi[i^\ast, x, y] \cdot \phi[i^\ast, x, y]}$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, and normalize $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
- Assign : $a[i^\ast] = \sum_{x, y} R[x, y] \cdot \phi[i^\ast, x, y]$
- Pursuit : $R[x, y] \leftarrow R[x, y] - a[i^\ast] \cdot \phi[i^\ast, x, y]$

Matching pursuit algorithm

Init : $R = I$, $\forall i$, $a[i] = 0$, $\sum_{x, y} \phi[i, x, y]^2 = 1$
compute $c[i] = \sum_{x, y} R[x, y] \cdot \phi[i, x, y]$
compute $X[i, j] = \sum_{x, y} \phi[i, x, y] \cdot \phi[j, x, y]$
while $\frac{1}{2} \sum_{x, y} R[x, y]^2 > \vartheta $, do :
- Match : $i^\ast = \arg \max_i c[i]$
- Assign : $a[i^\ast] = c[i^\ast]$
- Pursuit : $c[i] \leftarrow c[i] - a[i^\ast] \cdot X[i, i^\ast] $

[LP (2004)]

Matching pursuit algorithm

Matching pursuit algorithm

Hebbian learning (once the sparse code is known):

$$ \phi_{i}[x, y] \leftarrow \phi_{i}[x, y] + \eta \cdot a[i] \cdot (I[x, y] - \sum_{i=1}^{K} a[i] \cdot \phi_{i}[x, y] ) $$

Matching pursuit algorithm

Sparse representations in a nutshell

Convolutional Sparse Coding

Convolutional Neural Nets (CNN)

Convolutional Neural Nets (CNN)

Convolution: Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolution: Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolution: Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolution: Mathematics

Convolution: Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{c,i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolution: Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c,i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

CNN: the HMAX model

CNN: challenges

Convolutional Sparse Coding

Convolutional Sparse Coding

Convolutional Sparse Coding

[LP, 2015]

Code @ SparseEdges

Convolutional Sparse Coding

[Ladret et al, 2024]

Convolutional Sparse Coding

Convolutional Sparse Coding

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Predictive processing

CNN: Topography

[Bosking et al, 1997]

CNN: Topography

Sparse representations

Laurent Perrinet

NeuroSchool PhD Program in Neuroscience

[2024-04-17]

Code / Contact me @ laurent.perrinet@univ-amu.fr

2024-03-27-emergences.md

Wed, 27 Mar 2024 00:00:00 +0000

Analyser de larges volumes de données neurobiologiques

Laurent Perrinet

[2024-03-27] Emergences workshop, Autrans, France

laurent.perrinet@univ-amu.fr

Hello, can you hear me in the back? First of all, I’d like to thank the organizers for this opportunity and all of you for coming.

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and I’m a computational neuroscientist interested in large-scale models of vision.

Alors que ce projet vient juste de commencer, je voudrais déjà parler de quelques idées pour l’avenir. En effet, la question peut se poser quant aux applications futures des puces neuromorphiques qui vont être développées dans le cadre du projet “Emergences”. pour ce développement technologique, on va souvent penser à des applications technologiques, comme les voitures autonome ou la vision robotique. Mais il y a aussi des applications qui peuvent viser à la compréhension du fonctionnement du cerveau et de la cognition en général. Et ceci passe par une meilleure connaissance de la façon dont celle-ci est contenues dans l’activité neurale.

If you wish to go further, these slides along with a number of references and useful links are available on my website.

Techniques d’enregistrement de données neurobiologiques

Enregistrement extracellulaire

[Hubel & Wiesel, 1962]

Même si ce ne sont pas les premiers à avoir enregistré l’activité électrique de neurones (ce sont physiologistes allemands Emil du Bois-Reymond et Hermann von Helmholtz au milieu du 19e siècle), David Hubel et Torsten Wiesel ont marqué leur époque. En 1962, ils ont mené des expériences révolutionnaires qui ont permis de comprendre les mécanismes de base de la perception visuelle et ont jeté les bases de la compréhension de l’organisation fonctionnelle du cortex visuel. Leur travail a valu à Hubel et Wiesel le prix Nobel de physiologie ou médecine en 1981.

La technique principale utilisée par Hubel et Wiesel dans leurs expériences était la microélectrode d’enregistrement extracellulaire. Ils ont inséré de fines électrodes dans le cortex visuel primaire (aussi appelé cortex strié) de chats et de singes anesthésiés. Ces électrodes leur ont permis d’enregistrer l’activité électrique des neurones individuels lors de la présentation de stimuli visuels.

Aire visuelle primaire

Enregistrement extracellulaire

[Hubel & Wiesel, 1962]

Hubel et Wiesel ont utilisé une variété de stimuli visuels, tels que des lignes, des barres, des points lumineux et des motifs en mouvement, qu’ils ont présentés à des animaux dans des conditions contrôlées. En enregistrant les réponses des neurones visuels, ils ont pu observer des motifs caractéristiques d’activité neuronale en fonction des propriétés visuelles des stimuli.

Leur travail a révélé l’existence de neurones spécifiques, appelés neurones simples et neurones complexes, qui répondent de manière sélective à des caractéristiques visuelles spécifiques, telles que l’orientation, la direction du mouvement et la taille des stimuli. Ils ont également découvert que ces neurones étaient organisés de manière hiérarchique, avec des neurones simples détectant des caractéristiques visuelles élémentaires et des neurones complexes intégrant ces informations pour former des représentations plus complexes.

mais aussi: sharp electrodes, patch-clamp

Multi-électrodes

[Microelectrode array (MEAs)]

Différentes échelles

[Chemla et al, 2017]

Vers des données massives

[Stevenson and Kording, 2011]

Vers des données massives

[Steinmetz et al, 2017]

Techniques d’analyse des données neurobiologiques

Méthodes statistiques

Méthodes statistiques

Méthodes statistiques

[Churchland & Cunningham et al. (2012)]

… et au-delà!

… et au-delà: le décodage

… et au-delà: le décodage

… et au-delà: le décodage

Brain-Computer Interface (BCI)

[Interface neuronale directe (BCI)]

Perspectives et opportunités du neuromorphique

Exploitation d’un timing précis

Exploitation d’un timing précis

Exploitation d’un timing précis

Exploitation d’un timing précis

Codage par latence

Codage par latence

[Thorpe (2001)]

Latences et rapidité

Visual latencies (see review).

Algorithmes neuromorphiques

Always-on classification using HOTS

Always-on classification using HOTS

Always-on classification using HOTS

Spiking motifs in vision

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

Spiking motifs in vision

http://4.bp.blogspot.com/-AHprBxkfu5o/UJ-lqR7GsmI/AAAAAAAAHpo/VJzY7HMuXe0/s1600/The+Horse+in+Motion,+1878.%C2%A0Eadweard+Muybridge+(b.+9+April,+1830)The+first+movie+ever+made,+from+still+photographs..gif https://upload.wikimedia.org/wikipedia/commons/0/07/The_Horse_in_Motion-anim.gif

Spiking motifs pour la bio (HD-SNN)

LP (2023)

Future steps

unsupervised

high-throughput

real-time

Analyser de larges volumes de données neurobiologiques

Laurent Perrinet

[2024-03-27] Emergences workshop, Autrans, France

laurent.perrinet@univ-amu.fr

2024-02-05-udem.md

Mon, 05 Feb 2024 00:00:00 +0000

Neuromorphic models of vision

Laurent Perrinet

[2024-02-05] Seminar at UdeM’s School of Optometry, Montréal

laurent.perrinet@univ-amu.fr

When brains meet computing machines

Hello, can you hear me in the back? First of all, I’d like to thank the organizers for this opportunity and all of you for coming.

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and I’m a computational neuroscientist interested in large-scale models of vision. During this seminar for the “groupe de recherche de la vision de l’UdeM”, I’ll focus on neuromorphic models by introducing you to event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. The outline of the talk is as follows: first, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Then we’ll explore some applications of these cameras using specific algorithms. Finally, we’ll look at how our understanding of neuroscience can improve these algorithms.

Relax, these slides along with a number of references and useful links are available on my website.

Sensing light

… and this example demonstrates that since numerous years imaging techniques have also opened the door to new scientific discoveries. For example, in the late 19th century, scientists wondered if horses lifted all four hooves off the ground when they galloped. It was too fast for the human eye to see. Eadweard Muybridge solved this mystery using chronophotography, an early form of photography that captures motion. He took a series of photographs of a horse running and showed that there are moments when all four hooves are in the air. This breakthrough helped us better understand animal movement and paved the way for modern cameras.

This technique is inspired by the research of [Etienne-Jules Marey] (https://en.wikipedia.org/wiki/Etienne-Jules_Marey), under the term chronophotography, which is the use of a rifle-like apparatus to photograph a visual scene. This technique allowed Muybridge, in particular, to scientifically demonstrate the mechanism of a horse’s gallop. The movie theater became popular only afterwards.

Representing light

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Temporal Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Cameras

An event-based camera is equipped with a sensor that converts light into an electrical current, similar to conventional CMOS sensors. However, it differs from standard frame-based cameras in that it is inspired by the human retina. There are two main differences from a frame-based camera (middle graph) that lead to an event-based representation (right graph):

First, each pixel of an event-based camera is independent, operating without a synchronized global clock.
Second, each pixel detects changes in logarithmic light intensity and generates a binary event only if the change exceeds a threshold. If the change is an increment - that is, the log intensity has increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is generated asynchronously when a pixel-level change in brightness is detected. This results in superior temporal resolution and reduced susceptibility to motion blur, making event cameras ideal for capturing fast-moving scenes.

Event-Based Cameras: DVS gesture

Event-Based Cameras

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Cameras

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-driven cameras boast several remarkable properties. Firstly, their temporal precision is in the microsecond range, allowing for a theoretical frame rate of up to a million images per second. In contrast, a conventional camera typically captures around a hundred images per second, while a high-speed camera may reach 10,000 images per second. Estimating the sampling frequency of human perception is challenging; although 25 frames per second usually suffice for movies, the human eye can discern temporal details at rates between 300 and 1,000 frames per second. It’s also noteworthy that the spatial resolution of event cameras is generally modest, often in the megapixel range. This is not due to technical constraints but rather reflects the cameras’ common technological applications. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another key feature is their ability to detect a very wide range of luminosity, reaching 120 dB, which is a million times greater than conventional cameras and thousand times greater than an human eye.

Event-Based Cameras

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition —for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror different temporal sequence of events, termed the temporal surface which provides with a representation of recent activity. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at about 8.3%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis —an important concept from neuroscience— we enhanced the algorithm’s performance to 82%. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons. This underscores the value of incorporating neuroscientific principles into machine learning.

Furthermore, we leveraged a key trait of biological systems: the ability to process information continuously, in real time. Traditional algorithms wait to classify until all events are processed. We innovated by enabling our algorithm to classify on-the-fly, in real-time, with each incoming event. This means that as events occur, they’re instantly processed through the layers, reaching the classification layer without delay.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of our algorithm’s average performance relative to the dataset and the number of processed events. The blue curve reveals that with fewer than 10 events, performance hovers at chance levels. However, as more events are processed, we observe a steady improvement. Remarkably, with 10,000 events, performance matches that of the original algorithm and further excels with an additional tenfold increase in events. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in real-time, at any point during the event stream —not just after the entire signal is processed. Online processing is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the time to wait for the video sequence to finish processing before making the right decision, which is to flee. We’ve also refined our algorithm to select classification events based on precision calculations for each event. By adding a precision threshold, we achieve high performance with merely a hundred events. This reflects a biological network trait where decisions aren’t made incrementally but rather emerge abruptly here after 200 events — and then continue to improve and stabilize.

Spiking Neural Networks

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE Spiking neural networks show great potential for processing data from event-driven cameras. However, neurophysiology studies reveal some unexpected behaviors, very different from the classical perceptron. I will highlight these differences with three examples. The first example is a 1995 study by Mainen and Sejnowski examined a neuron’s reaction to repeated stimulations. Panel A at the top presents the neuron’s response to multiple stimulations with a 200 picoampere current step. The membrane potential varied across trials, indicating an unpredictable response. Initially, the spikes were synchronized at the onset of stimulation, but coherence diminished over time, leading to no alignment after approximately 750 milliseconds. In contrast, Panel B at the botton shows the neuron’s response to stimulation with noise. Here, the neuron exhibited highly consistent responses across trials, with membrane potential traces nearly identical. This precision was achieved using frozen noise, a repeated, unchanging stimulus. The study highlights that neurons are less responsive to constant analog values, such as square pulses, and more selective to dynamic signals, responding with remarkable precision in the temporal domain.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

Neuromorphic models of vision

Laurent Perrinet

[2024-02-05] Seminar at UdeM’s School of Optometry, Montréal

laurent.perrinet@univ-amu.fr

In conclusion, we have seen that event-driven cameras open the door to new applications that mimic the performance of the human eye, in terms of computational dynamics, adaptation to light conditions and energy constraints. This technological development has recently been accompanied by the development of neuromorphic chips and innovative algorithms in the form of spiking neural networks. However, there is still a great deal of progress to be made at theoretical level, particularly in the understanding of these spiking neural networks, and we have shown the potential progress that can be made by exploiting the richness of temporal representations, particularly by taking advantage of heterogeneous delays. Beyond these particular applications to natural image processing, I hope to have succeeded in demonstrating the importance of cross-fertilizing the field of engineering applications in general with biological neuroscience. This new line of research - known as NeuroAI or, more generally, as computational neuroscience - is likely to develop over the next few years. Thank you for your attention.

To conclude, we’ve explored how event-driven cameras pave the way for new applications. These applications mirror the human eye’s performance in terms of computational dynamics, rapid light condition adaptation, and energy efficiency. This tech advancement is complemented by the emergence of neuromorphic chips and innovative algorithms, specifically spiking neural networks. These networks emulate biological neurons, which communicate through binary events known as spikes rather than analog values used in traditionnal neural networks.

Despite these advancements, there’s still much to learn, especially in understanding how spiking neural networks process information. I hope I’ve successfully highlighted the importance of integrating engineering applications with neuroscience. This emerging research area, known as NeuroAI or computational neuroscience, is evolving rapidly. The ultimate aim of NeuroAI is to emulate the brain’s performance: it’s like having the computational power of a supercomputer compacted into the size of a soccer ball, using only around 20W of power, which is comparable to the energy consumption of a light bulb. This emerging research area, known as NeuroAI or computational neuroscience, is set

2023-12-14-jraf.md

Thu, 14 Dec 2023 00:00:00 +0000

Event-based vision

Adrien Fois & Laurent Perrinet

[2023-12-14] Journées sur l’apprentissage frugal (JRAF)

Hello, can you hear me in the back?

I’m Adrien Fois from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit. I’m a post-doctoral researcher under the supervision of Laurent Perrinet, and during this seminar, I’ll be presenting event-driven cameras. This innovative imaging technology and its influence on our understanding of vision will be our focus. I’d like to thank organizers for this opportunity, and all of you for coming. You can find these slides and related references on Laurent Perrinet’s website. The outline of the talk is as follows: initially, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Following that, we’ll explore some applications of these cameras using specific algorithms. Lastly, we’ll delve into how our understanding of neuroscience can enhance these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

An event-based camera is equipped with a sensor that, much like common CMOS sensors, converts light into electrical current. Yet, it stands apart from standard frame-based cameras by taking inspiration from the human retina. There are two main differences with respect to frame-based camera: Firstly each pixel of an event-based camera is independent, functioning without a synchonized global clock. Secondly, each pixel detect shifts in logarithmic light intensity, generating an binary event only when the change exceeds a threshold. If the change is an increment - meaning the log intensity increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is asynchronously generated when a pixel-level change in brightness is detected. This leads to a superior temporal resolution and a reduced susceptibility to motion blur, making event-camera ideal for capturing fast-moving scenes. Now, let’s explain how discrete events are produced in response to an analog signal that evolves continuously over time.

Event-Based Camera

Ultimately, we obtain a list of events for each pixels which can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence and polarities. Note that as events are generated over time, they are naturally sorted by their time of occurences. These events are then transmitted in real-time to the output bus, often through a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve, which connects our retina to the brain. In fact, the retina’s output is composed of a million ganglion cells that emit action potentials, constituting the only source of information transmitted through the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition—for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror the temporal sequence of events, termed the temporal surface. A temporal surface provides a representation of recent activity, it jumps to one on an event and then exponentially decays through time. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at 10%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis—an important concept from neuroscience—we enhanced the algorithm’s performance to 82%. This underscores the value of incorporating neuroscientific principles into machine learning. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons.

Always-on Object Gesture Recognition

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

Event-based vision

Adrien Fois & Laurent Perrinet

[2023-12-14] Journées sur l’apprentissage frugal (JRAF)

Spiking Neural Networks

[Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. vim Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2023-12-01-biocomp.md

Fri, 01 Dec 2023 00:00:00 +0000

Event-based vision

Laurent Perrinet

[2023-12-01] Séminaire colloque BioComp 2023

Hello, can you hear me in the back?

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this seminar at the BioComp 2023 colloquium, I’ll be presenting event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. I’d like to thank organizers for this opportunity, and all of you for coming. These slides are available from my website, along with a number of references. The outline of the talk is as follows: first, we’ll describe what an event-driven camera is - in particular, by comparing it to a conventional camera; then, we’ll show some examples of applications of these cameras with dedicated algorithms; and finally, we’ll present how our knowledge of biological mechanisms in neuroscience can enable us to improve these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

Finally, we obtain a list of events for each pixels which can be merged for the image as a whole, forming a list of events, including pixel addresses, times of occurrence and polarities. As they are generated over time, they are naturally arranged in order of occurrence. All these events are then transmitted in real time to the output bus, typically by means of a USB3 connection. Note the analogy between this representation and the one made in the optic nerve that connects our retina to the rest of the brain: indeed, the million ganglion cells that make up the retina’s output emit action potentials, which are the only source of information that leaves the retina via the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

There are several properties of event-driven cameras that make them remarkable. First of all, the temporal precision of events is of the order of microseconds, enabling a theoretical frame rate of the order of a million images per second to be reached. This can be compared with a conventional camera, which is of the order of a hundred images per second, or with a high-speed camera, which can reach 10,000 images per second. It is difficult to estimate the sampling frequency of human perception, because while 25 frames per second is often sufficient for movie viewing, it has been shown that the human eye can distinguish temporal details up to 300 or even 1,000 frames per second. It’s worth noting that the spatial resolution of these event cameras is often relatively modest, in the order of megapixels, but this is not a technical limitation, but rather due to the technological applications in which these cameras are commonly used. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another important feature of these cameras is their ability to detect a very wide range of luminosity, far exceeding that of conventional cameras at 120 dB (a factor of a million, compared with the human eye’s factor of 1 in a thousand between full moon and full sun),

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

The first algorithm we developed with Antoine Grimaldi, who is a PhD student, and in collaboration with Sio Ieng and Ryad Benosman of Sorbonne University, who are recognized researchers in the development of this type of camera, is an improvement on an existing algorithm, HOTS. This algorithm uses a relatively classical convolutional and hierarchical information processing architecture, which passes information “forward” from the camera and its event representation, and then through different processing layers to converge on a high-level representation that can be used for classification, in this case to recognize the identity of the digit presented as input, i.e. an eight digit. A fundamental feature of this algorithm is that it transforms the event representation into multiplexed, parallel channels, which analogously represent the temporal pattern of events, or “temporal surface”. These are represented in the different layers by the individual temporal surfaces. An interesting feature of this algorithm is that learning in each of the layers is unsupervised, which is a significant improvement over conventional deep learning algorithms that assume that a classification error signal can be back-propagated along the entire hierarchy, which is notoriously incorrect. Starting from this algorithm, we improved it by including neuro-biological knowledge, especially about the balance between different parallel communication pathways by including homeostasis rules.

Always-on Object Gesture Recognition

To illustrate the results of our algorithm, we applied a classic camera dataset involving the classification of 10 different types of human gestures. These biological movements are, for example, clapping hands, saying hello or a drum movement. The chance level is therefore at 10%, and we have observed that when all events have been processed, the original algorithm achieves a performance of around 70%. By adding homeostasis, we have reached a higher level of 82%, demonstrating the usefulness of using neuroscientific knowledge to improve machine learning algorithms.

We also built on a fundamental characteristic of biological systems. In fact, this kind of algorithm is classically used to process the flow of events, but classification is only used as a last resort when all the events have been processed. We have modified the algorithm so that this classification can be done online, in real time, event by event. In this way, processing in the various layers is triggered by the arrival of each event, which is propagated from the camera through all the layers to the classification layer.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of the average performance obtained on a data set, and as a function of the number of events processed by the algorithm. The blue curve shows that if below 10 events, we remain at the level of chance, we then experience a gradual increase in performance that reaches the level of the original algorithm with ten thousand events, and exceeds this performance when we have even 10 times more. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in its event camera, not once the entire signal has been processed by the system, but at any time. This characteristic is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the flexibility to wait for the video sequence to finish processing before making the right decision, which is to flee. Another variant in our algorithm consists of selecting the output classification events based on a calculation of the precision for each event. By using a threshold on this precision, we can achieve a very good level of performance, with just a hundred events, and so achieve a characteristic that is common in biological networks, i.e. that a decision is not taken gradually, but emerges abruptly (here after 200 events) and then improves and stabilizes.

Spiking Neural Networks

[Tonic manual]

Indeed, most neural networks used in deep learning use an analog representation. This is illustrated in this figure, which represents the various analog inputs to a formal neuron as they are linearly integrated by the synapses, then transformed by a non-linear function to generate an activation which is itself analog. This basic perceptron principle is at the foundation of all existing neural networks, and in particular enables the construction of convolutional-type networks which are currently the champions for image classification, having outperformed human performance for several years. However, while this is true for static images, it can become prohibitively expensive with videos. This is why it can be interesting to use spiking neurons instead, which, instead of receiving an analog input, will receive events that will trigger cascades of mechanisms in the neuronal cell, notably represented by the cell’s membrane potential. Typically, we’ll include a threshold for triggering action potential in this cell, which will generate new output events on the cell’s axon.

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

This new type of representation represents a paradigm shift in computation, in the same way that event-driven cameras have brought with them a paradigm shift in image representation. The development of these two new algorithms, which use impulse neural networks, is accompanied by the development of new neuromorphic chips, such as the Loihi 2 chip developed by Intel, which replaces a central computing unit with a massively parallelized array of elementary event-driven computing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. Other types of neuromorphic chips are currently being developed and may soon be used instead of conventional CPUs or GPUs.

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking neural networks therefore seem very promising for processing the output of event-driven cameras, but the study of neurophysiology shows us that their operation can sometimes seem incongruous and far from the perceptron. In this first example, taken from an article by Mainen and Sejnowski from 1995, we see the response of the same neuron to several repetitions of a stimulation in panel A. At the top, we see the membrane potential of this neuron in response to a 200 Pico ampere current step, which shows that the membrane potential is not reproducible across different trials. This is illustrated by showing the spike response over time for the different trials, which shows a strong alignment at the start of stimulation, but that this diffuses little by little, so that after around 750 milliseconds there is no longer any coherence between the different trials. The situation is different in panel B, where the neuron is stimulated with noise. In this case, the responses are so precise for the different trials that the membrane potential traces are overlapping almost exactly. The subtlety of this paper lies in its use of a frozen noise, i.e. one that is repeated unchanged across trials. In this way, it demonstrates that neurons are not so much sensitive to analog values presented in the form of square pulses, but rather to dynamic signals for which they will respond with very high precision in the dynamic domain.

Spiking Neural Networks in neurobiology

2 MINUTE

In this other example, I show a simulation that reproduces the 1999 paper by Diesmann and colleagues. This theoretical model considers ten groups of 100 neurons that are connected from group to group. An interesting property of this system is to show that for the same stimulation, i.e. for the same number of spikes, information can propagate from group to group only if it is sufficiently concentrated in time. For the first two groups, the information is too dispersed in the first group and spreads progressively and increasingly in subsequent groups. Above a certain threshold, the information formed by a group of relatively synchronous spikes is correctly transmitted to the various groups in the network. This non-linear behavior is one of the characteristics of spiking networks, giving them a certain richness, but also a certain complexity.

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Event-based vision

Laurent Perrinet

[2023-12-01] Séminaire colloque BioComp 2023

2023-11-07-snufa.md

Tue, 07 Nov 2023 00:00:00 +0000

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

SNUFA: Spiking Neural networks as Universal Function Approximators

^{https://laurentperrinet.github.io/talk/2023-11-07-snufa}

Core Mechanism of Spiking Motif Detection

From generating raster plots to inferring spiking motifs

A In this work, this principle was framed in a probabilistic setting such that we could provide an optimal scheme for detecting generic spiking motifs which may be superposed at random times. Starting with 10 presynaptic inputs, this model allows to generate a synthetic raster plot as the combination of four different spiking motifs. B These motifs are defined by a positive (red) or negative (blue) contribution to the spiking probability which are represented here. C Applying a Bayesian approach, we may define four formal spiking neurons which will integrate the incoming spiking information from the presynaptic neurons - this analog signal can then be thresholded to give the detection of each spiking motif (vertical) bar which was here always exact with respect to the ground truth (stars). D The beauty of this is that we can recover in the presynaptic raster plot the contribution of each spiking motif to the original raster plot.

Detecting spiking motifs using heterogeneous delays

Detecting spiking motifs using heterogeneous delays: supervised learning

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

SNUFA: Spiking Neural networks as Universal Function Approximators

^{https://laurentperrinet.github.io/talk/2023-11-07-snufa}

2023-09-27_icann.md

Wed, 27 Sep 2023 00:00:00 +0000

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

ICANN workshop on Recent Advances in SNNs

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this talk at this ICANN workshop on Recent Advances in SNNs, I’ll be presenting a method for the Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network, and how it may also impact the design of SNNs. I’d like to thank Sander Bohté and Sebastian Otte for the organization of this workshop and you for listening. These slides are available from my web-site, along with a number of references. The outline of the talk is as follows: first, I’ll describe how one may perform computations using Heterogeneous Delays - and present a toy model example; then, I’ll show real scale example quantifying the performance on synthetic data ; and finally, I’ll present how this SNN is in fact differentiable and may be extended for future applications.

Core Mechanism of Spiking Motif Detection

From generating raster plots to inferring spiking motifs

Detecting spiking motifs using heterogeneous delays

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

Laurent Perrinet

ICANN workshop on Recent Advances in SNNs

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a Spiking Neural Network

^{https://laurentperrinet.github.io/talk/2023-09-27-icann}

2023-09-08_fresnel.md

Fri, 08 Sep 2023 00:00:00 +0000

Event-based vision

Laurent Perrinet

[2023-09-08] Séminaire institut Fresnel

https://laurentperrinet.github.io/talk/2023-09-08-fresnel/

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this seminar at the Institut Fresnel, I’ll be presenting event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. I’d like to thank Loic le Goff for his kind invitation, and all of you for coming. These slides are available from my website, along with a number of references. The outline of the talk is as follows: first, we’ll describe what an event-driven camera is - in particular, by comparing it to a conventional camera; then, we’ll show some examples of applications of these cameras with dedicated algorithms; and finally, we’ll present how our knowledge of biological mechanisms in neuroscience can enable us to improve these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

Frame-Based Camera: Wagon-Wheel Illusion

https://en.wikipedia.org/wiki/Event_camera#Functional_description

Event-Based Camera

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition

Always-on Object Gesture Recognition

Always-on Object Gesture Recognition

Spiking Neural Networks

[Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

Loihi 2

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking Neural Networks in neurobiology

2 MINUTE

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

2 MINUTE

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Spiking Neural Networks: HD-SNN

Event-based vision

Laurent Perrinet

[2023-09-08] Séminaire institut Fresnel

2023-05-10-phd-program_neurosciences-computationnelles.md

Wed, 10 May 2023 00:00:00 +0000

Interactions between machine learning, artificial neural networks and our understanding of biological vision

Laurent Perrinet

[2023-05-10] NeuroSchool PhD Program in Neuroscience: Computation Neuroscience

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Visual illusions

Cydonia Mensae, 1976, Viking Orbiter image

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Cydonia Mensae, 2007, Mars Global Surveyor

Visual illusions : Pareidolia

Cydonia Mensae, 2007, Mars Global Surveyor

Visual illusions: Context

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions: Context

Visual illusions: Context

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

[Sejnowski, Koch & Churchland, 1998]

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks (CNNs)

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962] - from @Neuroslicer

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

[Grimaldi et al, 2023, Precise Spiking Motifs]

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision

Interactions between machine learning, artificial neural networks and our understanding of biological vision

Laurent Perrinet

[2023-05-10] NeuroSchool PhD Program in Neuroscience: Computation Neuroscience

Contact me @ laurent.perrinet@univ-amu.fr

2023-04-05-ue-neurosciences-computationnelles

Wed, 05 Apr 2023 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-05] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : the HMAX model

Convolutional Neural Networks (CNNs)

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-05] Master 1 Neurosciences et Sciences Cognitives.

Contact me @ laurent.perrinet@univ-amu.fr

2023-04-03-master-m-4-nc

Mon, 03 Apr 2023 00:00:00 +0000

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-03] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research.

Contact me @ laurent.perrinet@univ-amu.fr

Principles of Vision

What is the function of vision?

What is the function of vision?

What is the function of vision?

What is the function of vision?

Ilusions of brightness or lightness Akiyoshi KITAOKA

Visual illusions

Visual illusions

Visual illusions

Visual illusions

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Principles of vision?

Computational neuroscience of vision

Computational neuroscience of vision

Anatomy of the Human Visual system

Human Visual system : the HMAX model

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Primary visual cortex: Hubel & Wiesel

[Hubel & Wiesel, 1962]

Convolutional Neural Networks : Hierarchy

Convolutional Neural Networks : Mathematics

One-dimensional discrete convolution (eg in time) with a kernel $g$ of radius $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[n-m] \cdot g[m] $$

Convolutional Neural Networks : Mathematics

Convolution of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x-i, y-j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Cross-correlation of an image (two-dimensional) with a kernel $g$ of radius $K\times K$:

$$ (f \ast \tilde{g})[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[x+i, y+j] \cdot g[i, j] $$

Convolutional Neural Networks : Mathematics

Convolutional Neural Networks : Mathematics

Correlation of an image defined on several channels (note the order of the indices):

$$ (f \ast \tilde{g})[x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[c, i, j] $$

Convolutional Neural Networks : Mathematics

Correlation of a multi-channel image for multiple output channels (note the order of the indices):

$$ (f \ast \tilde{g})[k, x, y] = \sum_{c=1}^{C} \sum_{i,j} f[c, x+i, y+j] \cdot g[k, c, i, j] $$

Convolutional Neural Networks : the HMAX model

Convolutional Neural Networks (CNNs)

Convolutional Neural Networks : hierarchy

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Predictive coding

Convolutional Neural Networks : Topography

[Bosking et al, 1997]

Convolutional Neural Networks : Topography

Computational neuroscience of vision

Dynamics of vision

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Visual latencies (see review).

Dynamics of vision

Sensorimotor delays (Perrinet & Friston 2014)

Dynamics of vision

Sensorimotor delays (Perrinet & Friston, 2014)

Dynamics of vision

Diagonal markov model (Khoei et al, 2017).

Dynamics of vision

Flash-lag effect: MBP (Khoei et al, 2017)

Dynamics of vision

Spiking Neural Networks

Spiking Neural Networks: Leaky Integrate-and-Fire Neuron

Review on Precise Spiking Motifs.

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks in neurobiology

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Review on Precise Spiking Motifs.

Spiking Neural Networks in neuromorphic engineering

From frame-based to event-based cameras.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Spiking Neural Networks in neuromorphic engineering

The HD-SNN neural network.

Artificial neural networks and machine learning applied to the understanding of biological vision

https://laurentperrinet.github.io/talk/2023-01-23-game-theory-and-the-brain

Artificial neural networks and machine learning applied to the understanding of biological vision

Laurent Perrinet

[2023-04-03] Master M4NC de l’institut NeuroMod, cours Prospective Innovation and Research.

Contact me @ laurent.perrinet@univ-amu.fr

2023-01-23_game-theory-and-the-brain

Mon, 23 Jan 2023 00:00:00 +0000

Game theory and brain strategies

[2023-01-23] Atelier jeu et cerveau

Game theory and brain strategies

Le jeu du cerveau et du hasard, The Conversation

Aleatoric noise

A Huynh, generating Poisson disk noise

what is noise? it exists at quantum level, but if I were to ask you to draw random points how would it look like?
Aleatoric comes from alea, the Latin word for “dice.” Aleatoric uncertainty is the uncertainty introduced by the randomness of an event. For example, the result of flipping a coin is an aleatoric event.
In your opinion, which of the two is the most random pattern?
from your responses …
the answer is that …

When it comes to true randomness, one of its stranger aspects is that it often behaves differently to people’s expectations. Take the two diagrams below – which one do you think is a random distribution, and which has been deliberately created/adjusted?

randomized dots Only one of these panels shows a random distribution of dots | Source: Bully for Brontosaurus – Stephen Jay Gould

If you said the right panel, you are in good company, as this is most people’s expectation of what randomness looks like. However, this relatively uniform distribution has been adjusted to ensure the dots are evenly spread. In fact, it is the left panel, with its clumps and voids, that reflects a true random distribution. It is also this tendency for randomness to produce clumps and voids that leads to some unintuitive outcomes.

https://theconversation.com/daniel-kahneman-on-noise-the-flaw-in-human-judgement-harder-to-detect-than-cognitive-bias-160525

Instabilité, Etienne Rey.

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Visual illusions : Pareidolia

Sequence prediction


A: 👍👍👍👍🤘👍👍👍👍👍🤘👍👍👍👍🤘👍👍👍👍👍🤘👍🤘👍👍👍👍👍🤘 ?


B: 👍🤘🤘🤘👍👍👍🤘🤘👍🤘👍🤘👍👍🤘👍🤘👍👍👍🤘👍🤘👍🤘🤘🤘👍🤘 ?


C: 👍🤘🤘🤘👍🤘👍👍🤘🤘🤘🤘🤘🤘👍👍🤘👍🤘🤘🤘👍🤘👍🤘🤘🤘🤘🤘👍 ?


D: 🤘🤘🤘🤘🤘👍🤘🤘🤘👍🤘🤘🤘🤘👍🤘👍👍👍👍👍🤘👍🤘👍👍👍👍👍🤘 ?

Sequence prediction

(Pasturel et al, 2020).

Epistemic noise

Representing uncertainty

Visual epistemic uncertainty (Hugo Ladret).

Representing uncertainty

Visual epistemic uncertainty (Hugo Ladret).

Conclusion

Game theory and brain strategies

Aleatoric uncertainty (Pasturel et al, 2020).

Game theory and brain strategies

Epistemic uncertainty (Hugo Ladret).

Questions?

https://laurentperrinet.github.io/slides/2022-11-21_flash-lag-effect

More info @ web-site.

2022-11-21_flash-lag-effect

Mon, 21 Nov 2022 00:00:00 +0000

[2022-11-21] Alex Reynaud’s lab meeting

Timing in the visual pathways

Ultra-rapid visual processing (see review).

Compensating visual delays (Perrinet, Adams & Friston 2014).

Compensating visual delays (Perrinet Adams & Friston, 2014).

Travelling waves?

Suppressive travelling waves (Chemla et al, 2019).

Predictive coding

Motion-based prediction (Perrinet et al, 2012).

Flash-lag effect

Flash-lag effect (Khoei et al, 2017).

Diagonal markov model (Khoei et al, 2017).

Flash-lag effect: MBP (Khoei et al, 2017).

Flash-lag effect (Khoei et al, 2017).

Probability distributions (Khoei et al, 2017).

Motion reversal (Khoei et al, 2017).

Motion reversal (smoothed) (Khoei et al, 2017).

Probability distributions (Khoei et al, 2017).

Limit cycles (Khoei et al, 2017).

Application to the Pulfrich phenomenon?

Questions?

More info @ web-site + paper

2022-03-23_UE-neurosciences-computationnelles

Wed, 23 Mar 2022 09:00:00 +0000

Réseaux de neurones artificiels et apprentissage machine appliqués à la compréhension de la vision

Laurent Perrinet

[2022-03-23] Master 1 Neurosciences et Sciences Cognitives

Principes de la Vision

À quoi sert la vision?

À quoi sert la vision?

À quoi sert la vision?

À quoi sert la vision?

Les illusions visuelles

Les illusions visuelles

Ilusions of brightness or lightness Akiyoshi KITAOKA

Les illusions visuelles

Les illusions visuelles

Les illusions visuelles : Paréidolie

Les illusions visuelles : Paréidolie

Les illusions visuelles : Paréidolie

Les neurosciences computationnelles

Système visuel humain (Wikipedia)

De V1 aux réseaux convolutionnels

Le système visuel

Le cortex visuel primaire

[Hubel & Wiesel, 1962]

Hubel & Wiesel

[Hubel & Wiesel, 1962]

Réseaux convolutionnels : hiérarchie

Réseaux convolutionnels : Math

Convolution discrète uni-dimensionnelle (eg dans le temps) avec un noyau f de rayon $K$: $$ (f \ast g)[n]=\sum_{m=-K}^{K} f[m] g[n-m] $$

Réseaux convolutionnels : Math

Convolution discrète d’une image (bi-dimensionnelle):

$$ (f \ast g)[x, y] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[i, j] g[i-x, j-y] $$

Réseaux convolutionnels : l’opération de convolution

Réseaux convolutionnels : Math

Convolution discrète d’une image sur plusieurs canaux de sortie:

$$ (f \ast g)[x, y, k] = \sum_{i=-K}^{K} \sum_{j=-K}^{K} f[k, i, j, k] g[i-x, j-y] $$

Réseaux convolutionnels : Math

Convolution discrète d’une image multi-canaux (eg. RGB) sur plusieurs canaux de sortie (noter l’ordre des indices):

$$ (f \ast g)[x, y, k] = \ \sum_{i=-K}^{K} \sum_{j=-K}^{K} \sum_{c=1}^{C} f[k, c, i, j] g[i-x, j-y, c] $$

Réseaux convolutionnels : CNN

Mise en pratique: détecter & apprendre

Perspectives

Réseaux convolutionnels : hiérarchie

Réseaux prédictifs

Topographie dans V1

[Bosking et al, 1997]

Spiking Neural Networks

From frame-based to event-based cameras.

Recurrent processing

Dynamique de la vision

[Thorpe (2001)]

Applications robotiques

Our system is divided into 3 units to process visual inputs communicating by event-driven, feed-forward and feed-back communications.

Questions?

More info @ web-site

2020-12-10_agileneurobot_anr

Mon, 01 Jan 0001 00:00:00 +0000

Présentation du projet - L. Perrinet [2020-12-10] Réunion de lancement

AgileNeuRobot: Fiche d’identité

Titre : Robots aériens agiles bio-mimetiques pour le vol en conditions réelles
Title : Bio-mimetic agile aerial robots flying in real-life conditions
CES : CE23 - Intelligence Artificielle (ANR-20-CE23-0021)
Durée: 3 ans, à partir du 1er mars 2021
Budget total: 435 k€

Spiking Neural Networks

From frame-based to event-based cameras.

Recurrent processing

Our system is divided into 3 units to process visual inputs communicating by event-driven, feed-forward and feed-back communications.

Consortium:


Stéphane Viollet	Ryad Benosman	Laurent Perrinet
Julien Diperi	Sio-Hoï Ieng	Emmanuel Daucé
Inst Sciences Mouvement	Inst de la Vision	Inst Neurosci de la Timone

Gantt Chart of project

Questions?

More info @ web-site

2022-07-01_grimaldi-22-areadne

Mon, 01 Jan 0001 00:00:00 +0000

Decoding spiking motifs using neurons with heterogeneous delays [2022-07-01] AREADNE 2022 conference

Spiking Neural Networks

From frame-based to event-based cameras.

A raster plot..

.. as a mixture of motifs

… defined as list of weights and delays..

occurring from a new raster plot..

supervised learning

Learned heterogeneous weights

Heterogeneous delays as convolution kernels.

Mask applied on the weights.

Scatter of ON versus OFF weights.

Frugal computing

Stable accuracy while pruning ~99% weights.

Questions?