Visual search as active inference

Emmanuel Daucé & Laurent Perrinet

1st International Workshop on Active Inference (IWAI*2020), 14/9/2020

(AUTHOR) Hi, I am Emmanuel Daucé. I will present this work about modelling visual search as active inference. This is a joint work with Laurent Perrinet, from the Institute of Neurosciences la Timone, in Marseille.

Many thanks to the organizers for setting up this first conference on Active Inference.

This work is about the design of an artificial vision system that utilizes active vision principles to guide a visual sensor toward visual targets, like in a classic visual search task.

Today, I wish to view an important aspect of visual cognitive functions, Visual search, in light of active inference theories. As classicly formalized by past authors like Anne Treisman or Jeremy Wolfe, visual search is the common task of looking for a visual object in a cluttered visual environment...

Visual search as active inference

Second, one needs to make predictions. In general, the observer predicts that its saccade will improve its understanding of the visual data.

Starting from the information gain metric, We make the assumption that the visual processing separates in two parts, namely a ventral and a dorsal pathway.

The ventral (or identity) predics the target identity from the current foveal data

the dorsal (position) pathways predicts the future classificatin accuracy from the peripheral data

This is very specific : in the current visual field you don't have enough informatio to identify the peripheral target, but you have enough information to guess that if you move the eye, then you will be good at recognizing the target. you don't know what it is but you guess that its important looking there.

so if you want the detail of our implementation, you have to listen to the full presentation.

Then, the agent must bring the fovea toward visual data that will maximize the information gain.

We will see that -given the retinotopic arrangment of photoreceptor, this amounts to guess where an object is before knowing axactly what it is, because it is easier to predict the recognition accuracy than the category itself.

Outline

Motivation

Methods

Results

Conclusion

Computer vision

Human vision

Attention vs. Scene Understanding

Given a generative model of the environment, one can define a quantity called the bayesian surprise that tells how different is the visual data from your initial guess. Itti and Baldi predict that that the eye is attracted by the bayesian surprise, i.e. by the regions of the image that depart the most from the baseline image statistics. This allows to define salient regions in an image and draw saliency maps over an image that can predict where the eye is attracted the most. This may explain up to 50% of the human scan path, but it is purely phenomenological.

Laurent Itti and Christof Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. In: Vision Research 40.10-12 (2000), pp. 1489--1506.
M. Kümmerer, L. Theis, and M. Bethge Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet ICLR Workshop, 2015

Top down : (sequential decision)

A more detailed modelling originally proposed by Najemnik and Geisler proposes a sequential model of natural vision in a visual search task. Given a generative model of the visual field, the model decides where to look next : choose the next viewpoint that will provide the best information gain. The selection is reiterated several times until enough evidence is gathered.

(In general, the active inference setup means using a generative model to quantify the benefit of doing a certain action (changing viewpoint) to reduce the posterior entropy given an history of past actions (viewpoints), that corresponds to a better understanding of the visual scene.)

J Najemnik and Wilson S. Geisler. Optimal eye movement strategies in visual search. In: Nature reviews. Neuroscience 434 (2005)
Nicholas J Butko and Javier R Movellan. Infomax control of eye movements. In: Autonomous Mental Development, IEEE Transactions on 2.2 (2010)
Fu, J., Zheng, H., & Mei, T. (2017). Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4438-4446).

Outline

Motivation

Methods

Results

Conclusion

Principles for central and peripheric vision

So what we propose here is to go a little further in a biomimetic implementation of an artificial vision system. (Why : biomimetic systems are the result of a continual optimization throughout ages of evolution: they optimize signal processing under strong material and energy constraints, for specific surfival purposes.)

We concentrate her on the foveal vision case

What is specific with foveal vision? Foveal vision is a trick that was selected by natural selection : a compromise between resource saving and accuracy (budgeted vision) The fovea that concentrates most of the photoreceptors, represents less than 2% of the total visual field

So in order to analyze a complex visual scene, there are two types of processing that need to be done. On the one side, you need to process in detail what is at the center of fixation, that is the region of interest currently processed. On the other side, you also need to analyze the surrounding part, even if the resolution is low, in order to choose what is the next position of fixation.

(If we consider now the information gain metric, it shows an interesting correspondence with the central/peripheral processing trade-off. In a sequential setup, the rightmost term can be interpreted as the current state if understanding before the saccade is actuated, that is the information present at the center of the retina -- and the left term can be seen as the future state of understanding after the saccade is executed, that relies on interpreting the peripheral information.)

Methods: What/Where separation

We consider a separate processing of the central part of the visual field and the periphery.

(IN order to predict the information gain, we consider a slight simplification, that is sampling the prior and the posterior on the true label.) The information gain becomes the difference of the future accuracy and the central accuracy. Importantly, the future accuracy is a score that does not predict the future label. It just tells how correct the response will be while doing saccade a.

The separation into current accuracy and future accuracy is reminiscent of the What/where visual processing separation observed in monkeys and humans... with a separate processing of the object detailed shape and identity through the ventral pathway and the visuo-spatial information through the dorsal pathway.

(//Here we interpret the what/where separation in a slightly different manner, with the what devoted to analyzing the central part of the visual field, and the where devoted to choosing the next saccade. The "Where" is not exactly where but rather: where should I look next in order to increase my accuracy?//)

Methods: Computational Graph

Methods: What

WHAT :

On the one side, a ventral pathway predicts the target identity from the current foveal data.

The what pathway is a classic convolutional clasifier.

It shows some translation invariance.

It can quantify its uncertainty.

It monitors the where pathway.

TODO: mettre le résultat de l'accuracy map pour faire la transition?

Methods: Where

WHERE :

On the other side, a dorsal pathway utilizes all the peripheral visual data.

Here we make the assumption that the same logpolar compression pattern is conserved from the retina up to the primary motor layers. Each possible future saccade has an expected accuracy, that can be trained from the what pathway output. To accelerate the training, we use a shortcut that is training the network on a translated accuracy map (with logpolar encoding). The ouput is thus a logpolar accuracy map, that tells for each possible visuo-motor displacement the value of the future accuracy.Thus, the saccadic motor ouput (colliculus) shows a similar log-polar compression than the visual input. The saccades are more precise at short than at long distance (and severals accades may be necessary to precisely reach distant targets).

Outline

Motivation

Methods

Results

Conclusion

Results: success

Results: failure

Effect of eccentricity

Saccades distribution

       done

Outline

Motivation

Methods

Results

Conclusion

Visual search as active inference

Emmanuel Daucé & Laurent Perrinet

1st International Workshop on Active Inference (IWAI*2020), 14/9/2020

https://laurentperrinet.github.io/talk/2020-09-14-iwai