$logos$

https://laurentperrinet.github.io/talk/2023-09-08-fresnel/

Hello, I’m Laurent Perrinet from the Institut des Neurosciences de la Timone, a joint AMU / CNRS unit, and during this seminar at the Institut Fresnel, I’ll be presenting event-driven cameras, a new technology in the field of imaging, and the impact of this technology on our understanding of vision. I’d like to thank Loic le Goff for his kind invitation, and all of you for coming. These slides are available from my website, along with a number of references. The outline of the talk is as follows: first, we’ll describe what an event-driven camera is - in particular, by comparing it to a conventional camera; then, we’ll show some examples of applications of these cameras with dedicated algorithms; and finally, we’ll present how our knowledge of biological mechanisms in neuroscience can enable us to improve these algorithms.

Sensing light

Representing spatio-temporal luminous information

Frame-Based Camera: Temporal discretization

Frame-Based Camera: Aliasing

[[Gregor Lenz, 2020](https://lenzgregor.com/posts/event-cameras/)] — [Gregor Lenz, 2020]

Frame-Based Camera: Wagon-Wheel Illusion

[[Sam Brinson, 2020](https://www.sambrinson.com/nature-of-perception/)] — [Sam Brinson, 2020]

Event-Based Camera

Event-Based Camera

Finally, we obtain a list of events for each pixels which can be merged for the image as a whole, forming a list of events, including pixel addresses, times of occurrence and polarities. As they are generated over time, they are naturally arranged in order of occurrence. All these events are then transmitted in real time to the output bus, typically by means of a USB3 connection. Note the analogy between this representation and the one made in the optic nerve that connects our retina to the rest of the brain: indeed, the million ganglion cells that make up the retina’s output emit action potentials, which are the only source of information that leaves the retina via the optic nerve.

https://www.researchgate.net/profile/Guido-Croon/publication/313221316/figure/fig2/AS:668997448134663@1536512829861/Picture-of-the-event-based-camera-employed-in-this-work-the-DVS_W640.jpg

Event-Based Camera

Sensor	Range	Framerate	Resolution	Power
Human eye	60 (?) dB	300 (?) fps	100 (?) Mpx	10 mW
DSLR	44.6 dB	120 fps	2–20 Mpx	30 W
Ultra-high speed	64 dB	10^4 fps	0.3–4 Mpx	300 W
Event-based	120 dB	10^6 fps	0.1–2 Mpx	30 mW
There are several properties of event-driven cameras that make them remarkable. First of all, the temporal precision of events is of the order of microseconds, enabling a theoretical frame rate of the order of a million images per second to be reached. This can be compared with a conventional camera, which is of the order of a hundred images per second, or with a high-speed camera, which can reach 10,000 images per second. It is difficult to estimate the sampling frequency of human perception, because while 25 frames per second is often sufficient for movie viewing, it has been shown that the human eye can distinguish temporal details up to 300 or even 1,000 frames per second. It’s worth noting that the spatial resolution of these event cameras is often relatively modest, in the order of megapixels, but this is not a technical limitation, but rather due to the technological applications in which these cameras are commonly used. Compared with conventional cameras, which will consume several watts, event cameras consume very little electrical energy, in the order of 10 milliwatts, a consumption equivalent to that of the human eye. Another important feature of these cameras is their ability to detect a very wide range of luminosity, far exceeding that of conventional cameras at 120 dB (a factor of a million, compared with the human eye’s factor of 1 in a thousand between full moon and full sun), https://en.wikipedia.org/wiki/Event_camera#Functional_description more in https://arxiv.org/pdf/1904.08405.pdf

Event-Based Camera

Event-Based Computer vision

Always-on Object Recognition

[[Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023](https://laurentperrinet.github.io/publication/grimaldi-24/)] — [Grimaldi, Boutin, Sio-Ieng, Benosman & LP, 2023]

The first algorithm we developed with Antoine Grimaldi, who is a PhD student, and in collaboration with Sio Ieng and Ryad Benosman of Sorbonne University, who are recognized researchers in the development of this type of camera, is an improvement on an existing algorithm, HOTS. This algorithm uses a relatively classical convolutional and hierarchical information processing architecture, which passes information “forward” from the camera and its event representation, and then through different processing layers to converge on a high-level representation that can be used for classification, in this case to recognize the identity of the digit presented as input, i.e. an eight digit. A fundamental feature of this algorithm is that it transforms the event representation into multiplexed, parallel channels, which analogously represent the temporal pattern of events, or “temporal surface”. These are represented in the different layers by the individual temporal surfaces. An interesting feature of this algorithm is that learning in each of the layers is unsupervised, which is a significant improvement over conventional deep learning algorithms that assume that a classification error signal can be back-propagated along the entire hierarchy, which is notoriously incorrect. Starting from this algorithm, we improved it by including neuro-biological knowledge, especially about the balance between different parallel communication pathways by including homeostasis rules.

Always-on Object Gesture Recognition

To illustrate the results of our algorithm, we applied a classic camera dataset involving the classification of 10 different types of human gestures. These biological movements are, for example, clapping hands, saying hello or a drum movement. The chance level is therefore at 10%, and we have observed that when all events have been processed, the original algorithm achieves a performance of around 70%. By adding homeostasis, we have reached a higher level of 82%, demonstrating the usefulness of using neuroscientific knowledge to improve machine learning algorithms.

We also built on a fundamental characteristic of biological systems. In fact, this kind of algorithm is classically used to process the flow of events, but classification is only used as a last resort when all the events have been processed. We have modified the algorithm so that this classification can be done online, in real time, event by event. In this way, processing in the various layers is triggered by the arrival of each event, which is propagated from the camera through all the layers to the classification layer.

Always-on Object Gesture Recognition

What’s more interesting is that we were also able to show the evolution of the average performance obtained on a data set, and as a function of the number of events processed by the algorithm. The blue curve shows that if below 10 events, we remain at the level of chance, we then experience a gradual increase in performance that reaches the level of the original algorithm with ten thousand events, and exceeds this performance when we have even 10 times more. A major advantage of this algorithm is that it can be asked to classify the nature of what it sees in its event camera, not once the entire signal has been processed by the system, but at any time. This characteristic is essential in biology. For example, imagine you’re on the savannah and a lion jumps out at you. You won’t have the flexibility to wait for the video sequence to finish processing before making the right decision, which is to flee. Another variant in our algorithm consists of selecting the output classification events based on a calculation of the precision for each event. By using a threshold on this precision, we can achieve a very good level of performance, with just a hundred events, and so achieve a characteristic that is common in biological networks, i.e. that a decision is not taken gradually, but emerges abruptly (here after 200 events) and then improves and stabilizes.

Spiking Neural Networks

[[Tonic manual](https://tonic.readthedocs.io/en/latest/_images/neuron-models.png)] — [Tonic manual]

Indeed, most neural networks used in deep learning use an analog representation. This is illustrated in this figure, which represents the various analog inputs to a formal neuron as they are linearly integrated by the synapses, then transformed by a non-linear function to generate an activation which is itself analog. This basic perceptron principle is at the foundation of all existing neural networks, and in particular enables the construction of convolutional-type networks which are currently the champions for image classification, having outperformed human performance for several years. However, while this is true for static images, it can become prohibitively expensive with videos. This is why it can be interesting to use spiking neurons instead, which, instead of receiving an analog input, will receive events that will trigger cascades of mechanisms in the neuronal cell, notably represented by the cell’s membrane potential. Typically, we’ll include a threshold for triggering action potential in this cell, which will generate new output events on the cell’s axon.

Spiking Neural Networks: LIF Neuron

[Grimaldi *et al*, 2023, [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Grimaldi *et al*, 2023, Precise Spiking Motifs]

Spiking Neural Networks: neuromorphic hardware

This new type of representation represents a paradigm shift in computation, in the same way that event-driven cameras have brought with them a paradigm shift in image representation. The development of these two new algorithms, which use impulse neural networks, is accompanied by the development of new neuromorphic chips, such as the Loihi 2 chip developed by Intel, which replaces a central computing unit with a massively parallelized array of elementary event-driven computing units. As with event-driven cameras, this has the dual advantage of being very fast and consuming very little energy. Other types of neuromorphic chips are currently being developed and may soon be used instead of conventional CPUs or GPUs.

Prophesee

Loihi: https://d1fmx1rbmqrxrr.cloudfront.net/zdnet/optim/i/edit/ne/2019/Pierre%20temp/Intel%20Loihi__w630.jpg

https://cdn.cnx-software.com/wp-content/uploads/2022/09/Intel-Loihi-2.jpg?lossy=0&strip=none&ssl=1

Spiking Neural Networks in neurobiology

[[Mainen & Sejnowski, 1995](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_2_MainenSejnowski1995.ipynb)] — [Mainen & Sejnowski, 1995]

2 MINUTE

Spiking neural networks therefore seem very promising for processing the output of event-driven cameras, but the study of neurophysiology shows us that their operation can sometimes seem incongruous and far from the perceptron. In this first example, taken from an article by Mainen and Sejnowski from 1995, we see the response of the same neuron to several repetitions of a stimulation in panel A. At the top, we see the membrane potential of this neuron in response to a 200 Pico ampere current step, which shows that the membrane potential is not reproducible across different trials. This is illustrated by showing the spike response over time for the different trials, which shows a strong alignment at the start of stimulation, but that this diffuses little by little, so that after around 750 milliseconds there is no longer any coherence between the different trials. The situation is different in panel B, where the neuron is stimulated with noise. In this case, the responses are so precise for the different trials that the membrane potential traces are overlapping almost exactly. The subtlety of this paper lies in its use of a frozen noise, i.e. one that is repeated unchanged across trials. In this way, it demonstrates that neurons are not so much sensitive to analog values presented in the form of square pulses, but rather to dynamic signals for which they will respond with very high precision in the dynamic domain.

Spiking Neural Networks in neurobiology

[[Diesmann et al. 1999](https://github.com/SpikeAI/2022_polychronies-review/blob/main/src/Figure_3_Diesmann_et_al_1999.py)] — [Diesmann et al. 1999]

2 MINUTE

In this other example, I show a simulation that reproduces the 1999 paper by Diesmann and colleagues. This theoretical model considers ten groups of 100 neurons that are connected from group to group. An interesting property of this system is to show that for the same stimulation, i.e. for the same number of spikes, information can propagate from group to group only if it is sufficiently concentrated in time. For the first two groups, the information is too dispersed in the first group and spreads progressively and increasingly in subsequent groups. Above a certain threshold, the information formed by a group of relatively synchronous spikes is correctly transmitted to the various groups in the network. This non-linear behavior is one of the characteristics of spiking networks, giving them a certain richness, but also a certain complexity.

Spiking Neural Networks in neurobiology

[[Haimerl et al, 2019](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/)] — [Haimerl et al, 2019]

Spiking Neural Networks: Spiking motifs

Review on [Precise Spiking Motifs](https://laurentperrinet.github.io/publication/grimaldi-22-polychronies/). — Review on Precise Spiking Motifs.

Spiking Neural Networks: Spiking motifs

Spiking Neural Networks: HD-SNN

[Grimaldi & LP (2023) Biol Cybernetics](https://laurentperrinet.github.io/publication/grimaldi-23-bc/) — Grimaldi & LP (2023) Biol Cybernetics

Spiking Neural Networks: HD-SNN

2 MINUTE

One advantage of this network is that it is differentiable, enabling us to apply classical machine learning methods, notably supervised learning. We then see the emergence of different convolution kernels, and here I represent a subset of its kernels for different directions, as denoted by the red arrows on the left of the graph. It shows the kernels obtained on the spatial representation according to the different columns, and each row represents the different delays from a delay of one on the right to a delay of 12 time steps on the left. Detectors that follow the motion emerge. For example, for the top line from top to bottom. These kernels integrate both positive neurons in red and negative polarity inputs in blue. Such spatio-temporal filtering is observed in neurobiology, but to my knowledge had never been observed in a model of spiking neurons trained under natural conditions.

Spiking Neural Networks: HD-SNN

Event-based vision

Laurent Perrinet

[2023-09-08] Séminaire institut Fresnel

$logos$ [laurent.perrinet@univ-amu.fr](mailto:laurent.perrinet@univ-amu.fr)

In conclusion, we have seen that event-driven cameras open the door to new applications that mimic the performance of the human eye, in terms of computational dynamics, adaptation to light conditions and energy constraints. This technological development has recently been accompanied by the development of neuromorphic chips and innovative algorithms in the form of spiking neural networks. However, there is still a great deal of progress to be made at theoretical level, particularly in the understanding of these spiking neural networks, and we have shown the potential progress that can be made by exploiting the richness of temporal representations, particularly by taking advantage of heterogeneous delays. Beyond these particular applications to natural image processing, I hope to have succeeded in demonstrating the importance of cross-fertilizing the field of engineering applications in general with biological neuroscience. This new line of research - known as NeuroAI or, more generally, as computational neuroscience - is likely to develop over the next few years. Thank you for your attention.