$logos$

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

Hello, can you hear me in the back?

I’m Adrien Fois from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit. I’m a post-doctoral researcher under the supervision of Laurent Perrinet, and during this seminar, I’ll be presenting event-driven cameras. This innovative imaging technology and its influence on our understanding of vision will be our focus. I’d like to thank organizers for this opportunity, and all of you for coming. You can find these slides and related references on Laurent Perrinet’s website. The outline of the talk is as follows: initially, I will explain the concept of an event-driven camera, especially in comparison to a traditional frame-based camera. Following that, we’ll explore some applications of these cameras using specific algorithms. Lastly, we’ll delve into how our understanding of neuroscience can enhance these algorithms.

Sensing light

Representing spatio-temporal luminous information

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

[[Gregor Lenz, 2020](https://lenzgregor.com/posts/event-cameras/)] — [Gregor Lenz, 2020]

Frame-Based Camera: Wagon-Wheel Illusion

[[Sam Brinson, 2020](https://www.sambrinson.com/nature-of-perception/)] — [Sam Brinson, 2020]

Event-Based Camera

An event-based camera is equipped with a sensor that, much like common CMOS sensors, converts light into electrical current. Yet, it stands apart from standard frame-based cameras by taking inspiration from the human retina. There are two main differences with respect to frame-based camera: Firstly each pixel of an event-based camera is independent, functioning without a synchonized global clock. Secondly, each pixel detect shifts in logarithmic light intensity, generating an binary event only when the change exceeds a threshold. If the change is an increment - meaning the log intensity increased - the event has positive polarity; if it’s a decrement, the event has negative polarity.

In summary, an event is asynchronously generated when a pixel-level change in brightness is detected. This leads to a superior temporal resolution and a reduced susceptibility to motion blur, making event-camera ideal for capturing fast-moving scenes. Now, let’s explain how discrete events are produced in response to an analog signal that evolves continuously over time.

Event-Based Camera

Ultimately, we obtain a list of events for each pixels which can be merged to represent the entire image. This list of events includes pixel addresses, times of occurrence and polarities. Note that as events are generated over time, they are naturally sorted by their time of occurences. These events are then transmitted in real-time to the output bus, often through a USB3 connection. It’s interesting to draw a parallel between this process and the optic nerve, which connects our retina to the brain. In fact, the retina’s output is composed of a million ganglion cells that emit action potentials, constituting the only source of information transmitted through the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW
Event-driven cameras boast several remarkable properties. Firstly, their temporal precision is in the microsecond range, allowing for a theoretical frame rate of up to a million images per second. In contrast, a conventional camera typically captures around a hundred images per second, while a high-speed camera may reach 10,000 images per second. Estimating the sampling frequency of human perception is challenging; although 25 frames per second usually suffice for movies, the human eye can discern temporal details at rates between 300 and 1,000 frames per second. It’s also noteworthy that the spatial resolution of event cameras is generally modest, often in the megapixel range. This is not due to technical constraints but rather reflects the cameras’ common technological applications. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another key feature is their ability to detect a very wide range of luminosity, reaching 120 dB, which is a million times greater than conventional cameras and thousand times greater than an human eye. https://en.wikipedia.org/wiki/Event_camera#Functional_description more in https://arxiv.org/pdf/1904.08405.pdf

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition: DVS gesture

Always-on Object Recognition

[[Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023](https://laurentperrinet.github.io/publication/grimaldi-24/)] — [Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023]

So how can we process and learn data coming from an event-based camera ? My team, including PhD student Antoine Grimaldi, has enhanced an existing algorithm known as HOTS. This algorithm employs a traditional convolutional and hierarchical structure to process information. It begins with the camera’s event data that are processed three stacked layers, the last layer giving a high-level representation suitable for tasks like digit recognition—for example, identifying the number eight. A key aspect of HOTS is its conversion of event data into multiplexed, parallel channels that mirror the temporal sequence of events, termed the temporal surface. A temporal surface provides a representation of recent activity, it jumps to one on an event and then exponentially decays through time. Each layer represents these temporal surfaces individually. Notably, the algorithm’s learning process is unsupervised at every layer, marking a significant advancement over typical deep learning methods that rely on back-propagating classification errors—which is biologically implausible. Building on HOTS, we’ve improved it by incorporated neurobiological insights, particularly the principle of homeostasis, to better balance the various parallel communication pathways.

Always-on Object Gesture Recognition

To demonstrate our algorithm’s effectiveness, we tested it on a standard dataset that I presented before for classifying 10 distinct human gestures, such as clapping, waving, or drumming. With random guessing at 10%, the original HOTS algorithm achieved 70% accuracy after processing all events. However, by adding homeostasis—an important concept from neuroscience—we enhanced the algorithm’s performance to 82%. This underscores the value of incorporating neuroscientific principles into machine learning. Homeostasis is used to balance the firing rates accross neurons in a neural network. It ensures that all neurons contribute equally over time, avoiding dominance by a few neurons.

Furthermore, we leveraged a key trait of biological systems: the ability to process information continuously, in real time. Traditional algorithms wait to classify until all events are processed. We innovated by enabling our algorithm to classify on-the-fly, in real-time, with each incoming event. This means that as events occur, they’re instantly processed through the layers, reaching the classification layer without delay.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of our algorithm’s average performance relative to the dataset and the number of processed events. The blue curve reveals that with fewer than 10 events, performance hovers at chance levels. However, as more events are processed, we observe a steady improvement. Remarkably, with 10,000 events, performance matches that of the original algorithm and further excels with an additional tenfold increase in events. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in real-time, at any point during the event stream —not just after the entire signal is processed. Online processing is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the time to wait for the video sequence to finish processing before making the right decision, which is to flee. We’ve also refined our algorithm to select classification events based on precision calculations for each event. By adding a precision threshold, we achieve high performance with merely a hundred events. This reflects a biological network trait where decisions aren’t made incrementally but rather emerge abruptly here after 200 events — and then continue to improve and stabilize.

Event-based vision

Adrien Fois & Laurent Perrinet

[2023-12-14] Journées sur l’apprentissage frugal (JRAF)

$logos$

adrien.fois@univ-amu.fr laurent.perrinet@univ-amu.fr

To conclude, we’ve explored how event-driven cameras pave the way for new applications. These applications mirror the human eye’s performance in terms of computational dynamics, rapid light condition adaptation, and energy efficiency. This tech advancement is complemented by the emergence of neuromorphic chips and innovative algorithms, specifically spiking neural networks. These networks emulate biological neurons, which communicate through binary events known as spikes rather than analog values used in traditionnal neural networks.

Despite these advancements, there’s still much to learn, especially in understanding how spiking neural networks process information. I hope I’ve successfully highlighted the importance of integrating engineering applications with neuroscience. This emerging research area, known as NeuroAI or computational neuroscience, is evolving rapidly. The ultimate aim of NeuroAI is to emulate the brain’s performance: it’s like having the computational power of a supercomputer compacted into the size of a soccer ball, using only around 20W of power, which is comparable to the energy consumption of a light bulb. This emerging research area, known as NeuroAI or computational neuroscience, is set to evolve in the coming years. Thank you for your attention.

Spiking Neural Networks

[[Tonic manual](https://tonic.readthedocs.io/en/latest/_images/neuron-models.png)] — [Tonic manual]

Spiking Neural Networks: LIF Neuron

[Grimaldi *et al*, 2023, [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Grimaldi *et al*, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

The introduction of spiking neural networks marks a paradigm shift in computation, in the same way that event-driven cameras have brought a paradigm shift in image representation. These spiking neural networks have led to the creation of innovative algorithms and the development of neuromorphic chips like Intel’s Loihi 2. This chip departs from traditional computing by utilizing a massively parallel array of event-driven processing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. The field continues to advance, with new neuromorphic chips being developed that could potentially replace standard CPUs and GPUs.

Prophesee

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

[[Mainen & Sejnowski, 1995](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_2_MainenSejnowski1995.ipynb)] — [Mainen & Sejnowski, 1995]

2 MINUTE Spiking neural networks show great potential for processing data from event-driven cameras. However, neurophysiology studies reveal some unexpected behaviors, very different from the classical perceptron. I will highlight these differences with three examples. The first example is a 1995 study by Mainen and Sejnowski examined a neuron’s reaction to repeated stimulations. Panel A at the top presents the neuron’s response to multiple stimulations with a 200 picoampere current step. The membrane potential varied across trials, indicating an unpredictable response. Initially, the spikes were synchronized at the onset of stimulation, but coherence diminished over time, leading to no alignment after approximately 750 milliseconds. In contrast, Panel B at the botton shows the neuron’s response to stimulation with noise. Here, the neuron exhibited highly consistent responses across trials, with membrane potential traces nearly identical. This precision was achieved using frozen noise, a repeated, unchanging stimulus. The study highlights that neurons are less responsive to constant analog values, such as square pulses, and more selective to dynamic signals, responding with remarkable precision in the temporal domain.

Spiking Neural Networks in neurobiology

[[Diesmann et al. 1999](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_3_Diesmann_et_al_1999.py)] — [Diesmann et al. 1999]

Spiking Neural Networks in neurobiology

[[Haimerl et al, 2019](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Haimerl et al, 2019]

Spiking Neural Networks: Spiking motifs

Review on [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/). — Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Spiking Neural Networks: HD-SNN

[Grimaldi & LP (2023) Biol Cybernetics](https://laurentperrinet.github.io/publication/grimaldi-23-bc/) — Grimaldi & LP (2023) Biol Cybernetics

Spiking Neural Networks: HD-SNN

2 MINUTE A key advantage of this network is its differentiability, which allows the application of traditional machine learning techniques, such as supervised learning. We then see the emergence of various convolution kernels. The graph on the left, marked by red arrows, displays a selection of these kernels oriented in different directions. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. vim Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.