Abstract
Both biological and artificial neural networks inherently balance their performance with their operational cost, which balances their computational abilities. Typically, an efficient neuromorphic neural network is one that learns representations that reduce the redundancies and dimensionality of its input. This is for instance achieved in sparse coding, and sparse representations derived from natural images yield representations that are heterogeneous, both in their sampling of input features and in the variance of those features. Here, we investigated the connection between natural images’ structure, particularly oriented features, and their corresponding sparse codes. We showed that representations of input features scattered across multiple levels of variance substantially improve the sparseness and resilience of sparse codes, at the cost of reconstruction performance. This echoes the structure of the model’s input, allowing to account for the heterogeneously aleatoric structures of natural images. We demonstrate that learning kernel from natural images produces heterogeneity by balancing between approximate and dense representations, which improves all reconstruction metrics. Using a parametrized control of the kernels’ heterogeneity used by a convolutional sparse coding algorithm, we show that heterogeneity emphasizes sparseness, while homogeneity improves representation granularity. In a broader context, these encoding strategy can serve as inputs to deep convolutional neural networks. We prove that such variance-encoded sparse image datasets enhance computational efficiency, emphasizing the benefits of kernel heterogeneity to leverage naturalistic and variant input structures and possible applications to improve the throughput of neuromorphic hardware.
- 5 minutes summary: https://hugoladret.github.io/publications/ladret_et_al_sparsecoding/
1/9 New paper out @IOPneuromorphic !
— Hugo Ladret (@hugoladret) August 22, 2024
All natural images are equal(ly complex to model), but some are more equal than others.
-> How should we model them ?
-> https://t.co/Ww8U5h4z8Q
(cool pictures + deep learning in this thread)https://t.co/N0iKkPtSVe pic.twitter.com/JMCyauSZtF - In a nutshell: We found that sparse coding of images (here extended in a convolutional framework) is improved when using kernels with heterogeneous precision in how they encode orientation information. This was confirmed by learning, but also by comparison with what is observed in the statistics of natural images and in our recordings from neurons in primary visual cortex.
Epistemic uncertainty in a CSC dictionary improves both sparseness and reconstruction performance. (a) Elements from dictionaries with fixed epistemic uncertainty before (green) and after dictionary learning (orange). (b) Elements from a dictionary with heterogeneous epistemic uncertainty before (blue) and after dictionary learning (purple). (c) Elements from a dictionary learned from scratch. (d) Distribution of the sparseness (top) and Peak Signal-to-Noise Ratio (PSNR, right) of the five dictionaries, shown as a scatter plot for each of the 600 images of the dataset (center). Median values are shown as dashed line on the histograms. - open access: https://iopscience.iop.org/article/10.1088/2634-4386/ad5d0f
- This work is a followup of(2023). Convolutional Sparse Coding is improved by heterogeneous uncertainty modeling. ICLR 2023 SNN Workshop.
- This theoretical work accompanies a similar study in neurophysiology:
Hugo Ladret
Phd in Computational Neuroscience
My PhD subject focused on the role of precision in natural and artificial neural networks.
Laurent U Perrinet
Researcher in Computational Neuroscience
My research interests include Machine Learning and computational neuroscience applied to Vision.