Scientific logbook (Posts about learning)

Understanding Image Normalization in CNNs

Laurent Perrinet — Mon, 09 Dec 2024 15:00:53 GMT

Architectural innovations in deep learning occur at a breakneck pace, yet fragments of legacy code often persist, carrying assumptions and practices whose necessity remains unquestioned. Practitioners frequently accept these inherited elements as optimal by default, rarely stopping to reevaluate their continued relevance or efficiency.

Input normalization for convolutional neural networks, particularly in image processing, is a prime example of these unscrutinized practices. This is especially evident in the widespread use of pre-trained models like VGG or ResNet, where specific normalization values have become standard across the community. In particular, the standard values used for ImageNet training are:

Mean: [0.485, 0.456, 0.406] (for R,G,B channels respectively)
Std: [0.229, 0.224, 0.225]

What is the origin of these values? Are they really important?

Experimenting with transfer learning for visual categorization

Laurent Perrinet — Fri, 03 Dec 2021 15:23:51 GMT

Hi! I am Jean-Nicolas Jérémie and the goal of this notebook is to provide a framework to implement (and experiment with) transfer learning on deep convolutional neuronal network (DCNN). In a nutshell, transfer learning allows to re-use the knowlegde learned on a problem, such as categorizing images from a large dataset, and apply it to a different (yet related) problem, performing the categorization on a smaller dataset. It is a powerful method as it allows to implement complex task de novo quite rapidly (in a few hours) without having to retrain the millions of parameters of a DCNN (which takes days of computations). The basic hypothesis is that it suffices to re-train the last classification layers (the head) while keeping the first layers fixed. Here, these networks teach us also some interesting insights into how living systems may perform such categorization tasks.

Based on our previous work, we will start from a VGG16 network loaded from the torchvision.models library and pre-trained on the Imagenet dataset wich allows to perform label detection on naturals images for $K = 1000$ labels. Our goal here will be to re-train the last fully-Connected layer of the network to perfom the same task but in a sub-set of $K = 10$ labels from the Imagenet dataset.

Moreover, we are going to evaluate different strategies of transfer learning:

VGG General : Substitute the last layer of the pyTorch VGG16 network ($K = 1000$ labels) with a new layer build from a specific subset ($K = 10$ labels).
VGG Linear : Add a new layer build from a specific subset ($K = 10$ labels) after the last Fully-Connected layer of the the pyTorch VGG16 network.
VGG Gray : Same architecture as the VGG General network but trained with grayscale images.
VGG Scale : Same architecture as the VGG General network but trained with images of different size.
VGG Full : Same architecture as the VGG General network but all the layers are trained (otherwise I trained the last Fully-Connected layer).

In this notebook, I will use the pyTorch library for running the networks and the pandas library to collect and display the results. This notebook was done during a master 2 internship at the Neurosciences Institute of Timone (INT) under the supervision of Laurent Perrinet. It is curated in the following github repo.

Time lapsing an orchid's flower

Laurent Perrinet — Sun, 21 Feb 2021 19:57:52 GMT

The goal here is to realize a time-lapse using a raspberry π and some python code and finally get this:

(This was done over 10 days, with almost each day an irregular session the morning and one in the evening)

Kuramoto model

Laurent Perrinet — Fri, 08 Jan 2021 13:42:21 GMT

The goal here is to re-implement the Kuramoto model, following a lecture from Joana Cabral and the code that is provided, but using python instead of matlab.

Sparse coding of large images

Laurent Perrinet — Thu, 13 Dec 2018 09:53:03 GMT

In this post we would try to show how one could infer the sparse representation of an image knowing an appropriate generative model for its synthesis. We will start by a linear inversion (pseudo-inverse deconvolution), and then move to a gradient descent algorithm. Finally, we will implement a convolutional version of the iterative shrinkage thresholding algorithm (ISTA) and its fast version, the FISTA.

For computational efficiency, all convolutions will be implemented by a Fast Fourier Tansform, so that a standard convolution will be mathematically exactly similar. We will benchmark this on a realistic image size of $512 \times 512$ giving some timing results on a standard laptop.

Read more… (795 min remaining to read)

Extending datasets in pyTorch

Laurent Perrinet — Fri, 07 Sep 2018 10:27:10 GMT

PyTorch is a great library for machine learning. You can in a few lines of codes retrieve a dataset, define your model, add a cost function and then train your model. It's quite magic to copy and paste code from the internet and get the LeNet network working in a few seconds to achieve more than 98% accuracy.

However, it can be tedious sometimes to extend existing objects and here, I will manipulate some ways to define the right dataset for your application. In particular I will modify the call to a standard dataset (MNIST) to place the characters at random places in a large image.

MEUL with a non-parametric homeostasis

Laurent Perrinet — Tue, 07 Nov 2017 14:13:04 GMT

In this notebook, we will study how homeostasis (cooperation) may be an essential ingredient to this algorithm working on a winner-take-all basis (competition). This extension has been published as Perrinet, Neural Computation (2010) (see https://laurentperrinet.github.io/publication/perrinet-10-shl ). Compared to other posts, such as this previous post, we improve the code to not depend on any parameter (namely the Cparameter of the rescaling function). For that, we will use a non-parametric approach based on the use of cumulative histograms.

This is joint work with Victor Boutin and Angelo Francisioni. See also the other posts on unsupervised learning.

Improving calls to the LogGabor library

Laurent Perrinet — Fri, 06 Oct 2017 09:04:04 GMT

To code image as edges, for instance in the SparseEdges sparse coding scheme, we use a model of edges in images. A good model for these edges are bidimensional Log Gabor filter. This is implemented for instance in the LogGabor library. The library was designed to be precise, but not particularly for efficiency. In order to improve its speed, we demonstrate here the use of a cache to avoid redundant computations.

The fastest 2D convolution in the world

Laurent Perrinet — Wed, 20 Sep 2017 09:13:10 GMT

Convolutions are essential components of any neural networks, image processing, computer vision ... but these are also a bottleneck in terms of computations... I will here benchmark different solutions using numpy, scipy or pytorch. This is work-in-progress, so that any suggestion is welcome, for instance on StackExchange or in the comments below this post.

Le jeu de l'urne

Laurent Perrinet — Thu, 15 Jun 2017 14:42:44 GMT

Lors de la visite au laboratoire d'une brillante élève de seconde (salut Lena !), nous avons inventé ensemble un petit jeu aux apparences simples mais aux implications surprenantes : le jeu de l'urne.

L'idée de départ : peut-on mesurer l'impulsivité d'une personne rien qu'en observant ses choix ? C'est le pari de ce jeu.

Le principe¶

Imaginez une urne opaque contenant autant de balles rouges que noires — disons 8 balles au total, 4 rouges, 4 noires. À chaque tour, vous avez deux options :

Passer : vous tirez une balle, observez sa couleur, et la mettez de côté. Vous en savez un peu plus sur ce qu'il reste dans l'urne.
Deviner : vous pariez sur la couleur de la prochaine balle avant de la tirer. Si vous avez raison, vous gagnez autant de points qu'il reste de balles dans l'urne. Si vous avez tort... vous perdez ces mêmes points.

Le dilemme¶

Là où ça devient intéressant : plus vous laissez passer de balles, plus vous avez d'informations et plus votre prédiction devient fiable. Mais les balles restantes sont aussi de moins en moins nombreuses — et donc les points à gagner, de moins en moins élevés.

Il faut donc trouver le bon moment pour se lancer. Trop tôt, c'est risqué. Trop tard, c'est peu rentable. Ce compromis entre exploration (observer pour mieux savoir) et exploitation (agir pour gagner) est précisément ce qui révèle votre niveau d'impulsivité.

Les règles formelles¶

$N$ balles au total ($N$ pair, ex. $N = 8$), moitié rouges, moitié noires.
On tire les balles une à une, sans remise. On peut en observer autant qu'on veut avant de décider.
Quand on choisit de deviner : bonne prédiction → $+k$ points (où $k$ = nombre de balles restantes) ; mauvaise prédiction → $-k$ points.
L'objectif à long terme : trouver la stratégie qui maximise son score.

La première version¶

Nous avons d'abord donné vie à ce jeu grâce au langage de programmation Scratch, visible ici : scratch.mit.edu/projects/165806365.

Ici, nous allons essayer de l'analyser plus finement.