$logos$

laurent.perrinet@univ-amu.fr

When brains meet computing machines

Hello, can you hear me in the back? First of all, I’d like to thank the organizers for this opportunity and all of you for coming.

I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and I’m a computational neuroscientist interested in large-scale models of vision. During this seminar for the “groupe de recherche de la vision de l’UdeM”, I’ll focus on neuromorphic models by introducing you to event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. The outline of the talk is as follows: first, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Then we’ll explore some applications of these cameras using specific algorithms. Finally, we’ll look at how our understanding of neuroscience can improve these algorithms.

Relax, these slides along with a number of references and useful links are available on my website.

Sensing light

… and this example demonstrates that since numerous years imaging techniques have also opened the door to new scientific discoveries. For example, in the late 19th century, scientists wondered if horses lifted all four hooves off the ground when they galloped. It was too fast for the human eye to see. Eadweard Muybridge solved this mystery using chronophotography, an early form of photography that captures motion. He took a series of photographs of a horse running and showed that there are moments when all four hooves are in the air. This breakthrough helped us better understand animal movement and paved the way for modern cameras.

This technique is inspired by the research of [Etienne-Jules Marey] (https://en.wikipedia.org/wiki/Etienne-Jules_Marey), under the term chronophotography, which is the use of a rifle-like apparatus to photograph a visual scene. This technique allowed Muybridge, in particular, to scientifically demonstrate the mechanism of a horse’s gallop. The movie theater became popular only afterwards.

http://4.bp.blogspot.com/-AHprBxkfu5o/UJ-lqR7GsmI/AAAAAAAAHpo/VJzY7HMuXe0/s1600/The+Horse+in+Motion,+1878.%C2%A0Eadweard+Muybridge+(b.+9+April,+1830)The+first+movie+ever+made,+from+still+photographs..gif https://upload.wikimedia.org/wikipedia/commons/0/07/The_Horse_in_Motion-anim.gif

Representing light

Representing light

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Temporal Aliasing

[[Gregor Lenz, 2020](https://lenzgregor.com/posts/event-cameras/)] — [Gregor Lenz, 2020]

Frame-Based Camera: Wagon-Wheel Illusion

[[Sam Brinson, 2020](https://www.sambrinson.com/nature-of-perception/)] — [Sam Brinson, 2020]

Event-Based Cameras

An event-based camera is equipped with a sensor that converts light into an electrical current, similar to conventional CMOS sensors. However, it differs from standard frame-based cameras in that it is inspired by the human retina. There are two main differences from a frame-based camera (middle graph) that lead to an event-based representation (right graph):

First, each pixel of an event-based camera is independent, operating without a synchronized global clock.
Second, each pixel detects changes in logarithmic light intensity and generates a binary event only if the change exceeds a threshold. If the change is an increment - that is, the log intensity has increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is generated asynchronously when a pixel-level change in brightness is detected. This results in superior temporal resolution and reduced susceptibility to motion blur, making event cameras ideal for capturing fast-moving scenes.

Event-Based Cameras: DVS gesture

Event-Based Cameras

Ultimately, we get a list of events for each pixel that can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence, and polarities. Note that since events are generated over time, they are naturally sorted by their time of occurrence. These events are then transmitted in real time to the output bus, often via a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve that connects our retina to the brain. In fact, the output of the retina consists of a million ganglion cells that emit action potentials, which are the only source of information transmitted by the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Cameras

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW

Event-driven cameras boast several remarkable properties. Firstly, their temporal precision is in the microsecond range, allowing for a theoretical frame rate of up to a million images per second. In contrast, a conventional camera typically captures around a hundred images per second, while a high-speed camera may reach 10,000 images per second. Estimating the sampling frequency of human perception is challenging; although 25 frames per second usually suffice for movies, the human eye can discern temporal details at rates between 300 and 1,000 frames per second. It’s also noteworthy that the spatial resolution of event cameras is generally modest, often in the megapixel range. This is not due to technical constraints but rather reflects the cameras’ common technological applications. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another key feature is their ability to detect a very wide range of luminosity, reaching 120 dB, which is a million times greater than conventional cameras and thousand times greater than an human eye.

https://en.wikipedia.org/wiki/Event_camera#Functional_description

more in https://arxiv.org/pdf/1904.08405.pdf

Event-Based Cameras

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

[[Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023](https://laurentperrinet.github.io/publication/grimaldi-24/)] — [Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023]

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition —for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror different temporal sequence of events, termed the temporal surface which provides with a representation of recent activity. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at about 8.3%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis —an important concept from neuroscience— we enhanced the algorithm’s performance to 82%. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons. This underscores the value of incorporating neuroscientific principles into machine learning.

Furthermore, we leveraged a key trait of biological systems: the ability to process information continuously, in real time. Traditional algorithms wait to classify until all events are processed. We innovated by enabling our algorithm to classify on-the-fly, in real-time, with each incoming event. This means that as events occur, they’re instantly processed through the layers, reaching the classification layer without delay.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of our algorithm’s average performance relative to the dataset and the number of processed events. The blue curve reveals that with fewer than 10 events, performance hovers at chance levels. However, as more events are processed, we observe a steady improvement. Remarkably, with 10,000 events, performance matches that of the original algorithm and further excels with an additional tenfold increase in events. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in real-time, at any point during the event stream —not just after the entire signal is processed. Online processing is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the time to wait for the video sequence to finish processing before making the right decision, which is to flee. We’ve also refined our algorithm to select classification events based on precision calculations for each event. By adding a precision threshold, we achieve high performance with merely a hundred events. This reflects a biological network trait where decisions aren’t made incrementally but rather emerge abruptly here after 200 events — and then continue to improve and stabilize.

Spiking Neural Networks

© Gregor Lenz, [[Tonic manual](https://tonic.readthedocs.io/en/latest/)] — © Gregor Lenz, [Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi *et al*, 2023, [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Grimaldi *et al*, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Prophesee

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

[[Mainen & Sejnowski, 1995](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_2_MainenSejnowski1995.ipynb)] — [Mainen & Sejnowski, 1995]

2 MINUTE Spiking neural networks show great potential for processing data from event-driven cameras. However, neurophysiology studies reveal some unexpected behaviors, very different from the classical perceptron. I will highlight these differences with three examples. The first example is a 1995 study by Mainen and Sejnowski examined a neuron’s reaction to repeated stimulations. Panel A at the top presents the neuron’s response to multiple stimulations with a 200 picoampere current step. The membrane potential varied across trials, indicating an unpredictable response. Initially, the spikes were synchronized at the onset of stimulation, but coherence diminished over time, leading to no alignment after approximately 750 milliseconds. In contrast, Panel B at the botton shows the neuron’s response to stimulation with noise. Here, the neuron exhibited highly consistent responses across trials, with membrane potential traces nearly identical. This precision was achieved using frozen noise, a repeated, unchanging stimulus. The study highlights that neurons are less responsive to constant analog values, such as square pulses, and more selective to dynamic signals, responding with remarkable precision in the temporal domain.

Spiking Neural Networks in neurobiology

[[Diesmann et al. 1999](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_3_Diesmann_et_al_1999.py)] — [Diesmann et al. 1999]

Spiking Neural Networks in neurobiology

[[Haimerl et al, 2019](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Haimerl et al, 2019]

Spiking Neural Networks: Spiking motifs

Review on [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/). — Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Spiking Neural Networks: HD-SNN

[Grimaldi & LP (2023) Biol Cybernetics](https://laurentperrinet.github.io/publication/grimaldi-23-bc/) — Grimaldi & LP (2023) Biol Cybernetics

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Neuromorphic models of vision

Laurent Perrinet

[2024-02-05] Seminar at UdeM’s School of Optometry, Montréal

$logos$

laurent.perrinet@univ-amu.fr

In conclusion, we have seen that event-driven cameras open the door to new applications that mimic the performance of the human eye, in terms of computational dynamics, adaptation to light conditions and energy constraints. This technological development has recently been accompanied by the development of neuromorphic chips and innovative algorithms in the form of spiking neural networks. However, there is still a great deal of progress to be made at theoretical level, particularly in the understanding of these spiking neural networks, and we have shown the potential progress that can be made by exploiting the richness of temporal representations, particularly by taking advantage of heterogeneous delays. Beyond these particular applications to natural image processing, I hope to have succeeded in demonstrating the importance of cross-fertilizing the field of engineering applications in general with biological neuroscience. This new line of research - known as NeuroAI or, more generally, as computational neuroscience - is likely to develop over the next few years. Thank you for your attention.

To conclude, we’ve explored how event-driven cameras pave the way for new applications. These applications mirror the human eye’s performance in terms of computational dynamics, rapid light condition adaptation, and energy efficiency. This tech advancement is complemented by the emergence of neuromorphic chips and innovative algorithms, specifically spiking neural networks. These networks emulate biological neurons, which communicate through binary events known as spikes rather than analog values used in traditionnal neural networks.

Despite these advancements, there’s still much to learn, especially in understanding how spiking neural networks process information. I hope I’ve successfully highlighted the importance of integrating engineering applications with neuroscience. This emerging research area, known as NeuroAI or computational neuroscience, is evolving rapidly. The ultimate aim of NeuroAI is to emulate the brain’s performance: it’s like having the computational power of a supercomputer compacted into the size of a soccer ball, using only around 20W of power, which is comparable to the energy consumption of a light bulb. This emerging research area, known as NeuroAI or computational neuroscience, is set