HTML("""
<h1 class="title">Decoding of feature selectivity in neural activity</h1>
<h1>Concrete applications in visual data</h1>
<h2>Wahiba Taouali, INMED - Laurent U Perrinet, INT</h2>
 <img src="{0}" height=61%/>
 """.format(figpath + 'troislogos.png'))

Decoding of feature selectivity in neural activity

Concrete applications in visual data

Wahiba Taouali, INMED - Laurent U Perrinet, INT

Acknowledgements:

• PhD program: Anna Montagnini, Frédéric Chavane, Nadia Pittet, INT, Marseille
• Material: Peggy Series, Christopher Olah
• NeuroPhysiology: Giacomo Benvenuti, Frédéric Chavane, INT, Marseille

ⓦ https://laurentperrinet.github.io/sciblog/files/2015-12-08_cours_neurocomp/PerrinetEtAl12neurocomp-intro.slides.html

HTML("""
 <h2>Reverse engineering : from encoding to decoding</h2>
 <img src="{0}" width=100%/>
 """.format(figpath + 'encoding_problem.png'))

Reverse engineering : from encoding to decoding

Aim of experimental neuroscience : describing the activity of neurons : what are they responding to??

HTML("""
 <h2>Reverse engineering : from encoding to decoding</h2>
 <img src="{0}" width=100%/>
 """.format(figpath + 'decoding.jpg'))

Reverse engineering : from encoding to decoding

The common steps for analysing how a population of neurons encodes information about visual inputs are shown. First, recordings are taken at different sites with implanted electrodes. Second, the simulated activity of single neurons is extracted from the continuous data using spike-sorting algorithms. Third, information is inferred from the multiple spike trains with decoding algorithms (which can predict that the stimulus was an apple), or information theory (which quantifies the knowledge about the stimulus gained by observing the population response). The population analysis allows the study of the information carried by the different features of the multiple spike trains. For example, it can be established whether the information of the apple is given by an increase in firing (neuron in red), by a particular temporal firing pattern (neuron in green) or by the simultaneous firing of a subset of neurons (neurons in blue and grey). The vertical dotted line marks stimulus onset.

Outline

• Neural data analysis
• Decoding Approach
• Some Python scripts

Outline

• Neural data analysis
• Decoding Approach
• Some Python scripts

Neural data analysis

HTML("""
 <h2>Experimental setup</h2>
 <center><img src="{0}" height=60%/>
 """.format(figpath + 'ExperimentalSetup_1.png'))

Experimental setup

HTML("""
 <h2>Experimental setup</h2>
 <center><img src="{0}" height=60%/>
 """.format(figpath + 'ExperimentalSetup_2.png'))

Experimental setup

1. behaving macaque monkey, extracellular
2. visual stimulation - training
3. characterization of the RF with sparse noise
4. visual stimulation = long segment (wider than RF) - along a trajectory, T= / L= / speed = (tuned to psychophysics / lateral interactions)

HTML("""
 <h2>Describing neurons's activity </h2>
    <table border="0" width=100%/> 
    <tr> 
    <td width=50%/><img src="{0}" width=100%/> </td>
    <td width=50%/><img src="{1}" width=100%/> </td>
    </tr> 
    </table>
 """.format(figpath+ 'cell_response_c.png',figpath+ 'cell_response_all.png')
    )

Describing neurons's activity

1. We focus the description of a cell response on the variation of the average firing rate while changing the stimulus --> we obtain what we call tuning curves
2. The results show a typical profile for which the firing rate increases as the bar reaches the receptive field.
3. The average response of each cell is modulated by the stimulus change. A modulated cell responds preferentially to a given stimulus feature value

HTML("""
 <h2>Tuning and Population statistics in RFc</h2>
 <table border="0" width=100%/> 
    <tr> 
    <td width=30%/><embed src="{0}" width=100% > </td>
    <td width=30%/><embed src="{1}" width=100% ></td>
    <td width=30%/><embed src="{2}" width=100% > </td>
    </tr> 
    </table>""".format(figpath+ 'TuningV1_population_average_1.png',figpath+ 'TuningV1_preferred_direction.png',figpath+ 'TuningV1_preferred_orientation_.png'))

Tuning and Population statistics in RFc

In the visual cortex, the activity of neurons (spike count ) is correlated with some aspects of the visual image (contrast, orientation, color, spatial frequency, ...) towards more complicated features sych as faces and shapes. Our task allow to consider to features orientation and direction of movement. V1 is known to be highly selective to orientation which explains the form of the tuning ...

Decoding : Output + Population model ----> Input guess

HTML("""
 <h2>Population dynamic decoding : orientation</h2>
  <center><img src="{0}" width=100%/>
 """.format(figpath+ 'decoding_orientation.png'))

Population dynamic decoding : orientation

Given a decoder (black box) : input response and population knowledge gives output

1. we show here the results of the decoding as individual polar plots as a function of time (horizontal axis) and for different orientations of the bar
2. this shows that as the bar enters the receptive field, there is a progressive decoding of the bar's orientation

HTML("""
 <h2>Population dynamic decoding : direction</h2>
 <center><img src="{0}" width=100%/>
 """.format(figpath+ 'decoding_direction.png'))

Population dynamic decoding : direction

Similar plots for direction, the decoding is less performant. V1 is not the best candidate for the encoding of direction. Even a low decoding performance is interesting to occur in V1

HTML("""
 <h2>How to construct a decoder??</h2>
 <img src="{0}" width=100%/>
 """.format(figpath + 'decoding_problem.png'))

How to construct a decoder??

Outline

• Neural Data
• Decoding Approach
• Some Python scripts

Decoding approach

HTML("""
 <h2>Poisson distribution as a model of variability </h2>
 <table border="0" width=100%/> 
    <tr> 
    <td width=50%/><embed src="{0}" width=100% '> </td>
    <td width=50%/><embed src="{1}" width=100%  '> </td>
    </tr> 
    </table>""".format(figpath+ 'CellRasterV1.png',figpath+ 'DirectionTuningV1.png'))

Poisson distribution as a model of variability

(Left)Let's start with raw neural data : Raster plots of ONE representative cell of area V1 of macaque monkeys in response to an oriented moving bar. Subplots correspond to responses to 12 equally distributed motion directions with orientation orthogonal to direction. % (Right) Preliminary observations of variability: Directional tuning curves (mean spike count, black lines) along with the range of {±} the standard deviation (grey lines) for a population of 16 cells from the same area in response to the 12 directions. The area corresponding to the mean {±} the standard deviation expected under Poisson distribution hypothesis {(σ=μ⎯⎯√)} is plotted in blue. Most of cells show similar variability to that expected from the Poisson model.

HTML("""
 <h2>Von Mises as a model of Tuning </h2>
 <center>
    <table border="0" width=100%/> 
    <tr> 
    <td width=50%/><embed src="{0}" width=100%'> </td>
    <td width=50%/><embed src="{1}" width=100%'> </td>
    </tr> 
    </table>
 """.format(figpath+ '2D_tuning.png',figpath+ 'TuningV1.png')
    )

Von Mises as a model of Tuning

To better quantify the tuning information while dealing with variability, we proposed a model of a generative, directional, tuning function, assuming Poisson distribution as a model of inter-trail variability we details the reason why we make this assumption). Indeed, we defined the generative tuning function on the base of two qualitative observations we made on the TCs: 1) they generally presents two lobes corresponding to the two directions perpendicular to the preferred orientation of the cell (this can be defined as the "orientation component" of the TC), 2) Some cells present a preference for one of these two directions or are tuned for only one direction (it can be defined as the "direction component" of the TC). \

Therefore we defined the generative tuning function as a product of two von Mises functions\footnote{Von Mises are circular Gaussian functions centered on the preferred direction\cite{Swindale1998}}, one encoding the orientation and the other the direction selectivity (fig\ref{fxTun_f}b) of the cell

The function part encoding the orientation tuning of the cell, is actually tuned for the two directions orthogonal to the preferred orientation of the cell $\theta$. The other can present a positive lobe peaking on the preferred direction of motion ($\phi$, fig~\ref{fxTun_f}b). For the special conditions of the stimulation paradigm \textbf{DoD}, $\theta$ is equal to $\phi$ (due to the construction of the experiment) and therefore, for the moment, we are going to enunciate the tuning function only for the parameter $\phi$. We defined this function as "Multiplicative Von Mises" (scaled product of two Von Mises functions), to distinguish it from the basic Von Mises.

The fit was carried out with a minimization process\footnote{In more details, we minimized the Kullback Leibler divergence, between the distribution of spike numbers over trials for each condition and the Poisson distribution with $\lambda$ equal to the correspondent value of the tuning function.}, in which the parameters corresponding to the scaling, un-selective component and maximum values ($\kappa_o$, $\kappa_d$, $R_0$, $R_m$) were free to vary. In Fig\ref{fxTUn_C} are presented some examples of tuning fits.

$$\bf\Large{ f(\Phi, \Theta, t) = R_{0}+(R_m-R_{0})\cdot A(t)\cdot (b+(1-b)\cdot e^{\kappa_\theta\cdot(\cos(2(\theta-\phi_0))-1)} \cdot e^{\kappa_\phi\cdot(\cos(\phi-\phi_0)-1)})}$$

with :

• a Gaussian activation profile: $$\bf\Large{ A(t) = \frac{1}{{\sigma \sqrt {2\pi } }} e^{{\frac{ - \left( {t - \mu } \right)^2 } {2\sigma ^2}}} }$$
• a constant offset-to-gain ratio : b

HTML("""
 <h2>Dynamic Tuning </h2>
    <table border="0" width=100%/> 
    <tr height=30%/><embed src="{0}" width=100%'></tr> 
    <tr height=30%/><embed src="{1}" width=100%'></tr> 
    <tr height=30%/><embed src="{2}" width=100%'></tr> 
    </table>
 """.format(figpath+ 'dynamic_tuning.png',figpath+ 'dynamic_tuning_1.png',figpath+ 'dynamic_tuning_2.png')
    )

Dynamic Tuning

To characterize this dynamic modulation of the TC, we extended our tuning function model to take in account also the different amplitude of gain and offset of the TC in the different time windows. As a first approximation, we supposed that the offset and the gain evolutions in time are correlated. Thus, we defined an offset-to-gain ratio parameter "b", that should be estimated from the data (equation \ref{exOGR}).

HTML("""
 <h2>Evaluation of decoding</h2>Leave-One-Out Cross Validation
 <img src="{0}" width=80%/>
 """.format(figpath+ 'loo.png'))

Evaluation of decoding

Leave-One-Out Cross Validation

Here, we focus on a special form of cross-validation, called leave-one-out cross-validation (LOOCV). In this case, we pick only one point as the test set. We then build a model on all the remaining, complementary points, and evaluate its error on the single-point held out. A generalization error estimate is obtained by repeating this procedure for each of the training points available, averaging the results.

HTML("""
<h2>Evaluation of decoding</h2>Receiver operating characteristic (ROC)
    
    <center><embed src="{0}" height=400'></tr> 
    
 """.format(figpath+ 'DT_1D_decoding_AUROC1.png'))

Evaluation of decoding

Receiver operating characteristic (ROC)

In statistics, a receiver operating characteristic (ROC), or ROC curve, is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The true-positive rate is also known as sensitivity or the sensitivity index d', known as "d-prime" in signal detection and biomedical informatics, or recall in machine learning. The false-positive rate is also known as the fall-out and can be calculated as (1 - specificity).

To quantify the goodness of the decoding performance, we took advantage of a statistical method used in signal detection theory named: the area under the ROC (Receiver Operating Characteristic) curve. The ROC curve compares the true positive rate vs the false positive rate at various threshold settings. In our case the threshold corresponded to the bin size (centered on direction 0) of the polar histogram of the DD. The true positives (or Hits) corresponds to the DD falling in the bin centered on the global direction of the stimulus and the false positive (or False alarm) to all the other DD. In fig~\ref{AUC_DT} is presented an example of ROC curves for the time window centered on the RF center, for all the condition DT .

Example of ROC curve statistic on the decoded directions} ROC curves of the decoded directions in a time window of 100 ms centered on the RF center for the 12 condition of \textbf{DoO} paradigm using DoO tunings.

HTML("""
 <h2>Evaluation of decoding</h2>Area under the ROC curve
    <center><table border="0" width=100%/> 
    <tr height=100%/><embed src="{0}"  height=300'></tr> 
    </table>
 """.format(figpath+ 'DT_AUROC.png'))

Evaluation of decoding

Area under the ROC curve

Average time courses of the AUC over conditions of \textbf{DoO} paradigm. Direction decoding performance using DoO tuning in black, DiO tuning in blue and Orientation decoding performance using DiO tuning in red.

The area under the curve (AUC) of the ROC curve quantifies the overall ability of the decoding method to discriminate the actual global direction of the stimulus (it ranges between 0-1). is shown the time course of the AUC, relative to the decoding of direction (using DoO and DiO tuning) and orientation (Using DiO tuning), averaged over the 12 conditions . In the time step at -500 ms, the bar appears (at -455 ms). The decoding starts to display good performances (corresponding to a distance of 3 degree from the RFc). Between -300 and -200 ms the decoding performance is mainly good for the main part of the conditions. The results show that a main part of the decoding of direction using DoO tuning is due to the orientational component (black versus red) that is better decoded than direction ( red versus black and blue).

Use of simple decoding methods for prosthetics

https://www.youtube.com/watch?v=7kctOHnrvuM

HTML("""
    <center><table border="0" width=100%/> 
    <tr height=100%/><img src="{0}" width=290'></tr> 
    </table>
 """.format(figpath+ 'conclude.png'))

"... Scientists in the US have developed a micro chip which is implanted in the brain to operate computers and robots. It could transform the lives of people who are paralysed, or have lost limbs. Scientists are now ready to start testing on humans."

Outline

• Neural Data
• Decoding Approach
• Some Python scripts

Some Python scripts

HTML("""
 <h2>Object oriented programming</h2>
 <textarea rows="28" cols="60">
class Cell:
    def __init__(self,spikeTime,barOnsetTime,baselineTime,params={}):
        self.parameters = 
        self.spikeTime = 
        self.barOnsetTime = 
        self.trialOnsetTime = 
        self.firingRate =
        
    #---------------------------------------------------------   
    def eval_tuningDirOriTime(self,nbDir=12,nbOri=6):
    #----------------------------------------------------------    
    def eval_firingRate(self,window=0.75/6.6*1000):
    #----------------------------------------------------------
    def eval_trialOnsetTime(self,baselineTime):
    #-----------------------------------------------------
    def plot_raster(self,experiment,condition):
    #--------------------------------------------------------
    def plot_meanfr(self,experiment,figsize=(4,2)):
    #---------------------------------------------------------
    def isDtTrajectory(self):
    #--------------------------------------------------------- 
    def meanFiringRate(self,exp):
    #------------------------------------------------------#
    def temporalTuningDT_fit(self,Nest=12):
    #------------------------------------------------------#

</textarea>
 """)

Object oriented programming

class Cell: def __init__(self,spikeTime,barOnsetTime,baselineTime,params={}): self.parameters = self.spikeTime = self.barOnsetTime = self.trialOnsetTime = self.firingRate = #--------------------------------------------------------- def eval_tuningDirOriTime(self,nbDir=12,nbOri=6): #---------------------------------------------------------- def eval_firingRate(self,window=0.75/6.6*1000): #---------------------------------------------------------- def eval_trialOnsetTime(self,baselineTime): #----------------------------------------------------- def plot_raster(self,experiment,condition): #-------------------------------------------------------- def plot_meanfr(self,experiment,figsize=(4,2)): #--------------------------------------------------------- def isDtTrajectory(self): #--------------------------------------------------------- def meanFiringRate(self,exp): #------------------------------------------------------# def temporalTuningDT_fit(self,Nest=12): #------------------------------------------------------#

HTML("""
 <h2>Object oriented programming</h2>
 <textarea rows="7" cols="100">
  -> param =&#123"barSpeed":6.6,"tstart":0.,"tstop":2500.,"tbaseline" : 400.,"angleBin":30.&#125
  -> V1DataObject = DataSetV1(pathToMatFolder)
  -> cell_num = 6
  -> cell = Cell(V1DataObject.spikeTime[cell_num],V1DataObject.tbar[cell_num],V1DataObject.parameters["tbaseline"],params)
  -> c.plot_raster(experiment=0,condition=0,figsize=(6,4))
 </textarea>
 <img src="{0}" width=60%/>
 """.format(figpath+ 'cell_response_c.png'))

Object oriented programming

-> param ={"barSpeed":6.6,"tstart":0.,"tstop":2500.,"tbaseline" : 400.,"angleBin":30.} -> V1DataObject = DataSetV1(pathToMatFolder) -> cell_num = 6 -> cell = Cell(V1DataObject.spikeTime[cell_num],V1DataObject.tbar[cell_num],V1DataObject.parameters["tbaseline"],params) -> c.plot_raster(experiment=0,condition=0,figsize=(6,4))

HTML("""
 <h2>Modeling and Statistics : numpy,scipy</h2>

<img src="http://latex.codecogs.com/gif.latex?\Large{ f(\Phi, \Theta, t) = 
R_{0}+(R_m-R_{0})\cdot A(t)\cdot (b+(1-b)\cdot e^{\kappa_\theta\cdot(\cos(2(\theta-\phi_0))-1)}
\cdot e^{\kappa_\phi\cdot(\cos(\phi-\phi_0)-1)})}" border="0"/><p>
  <textarea rows="28" cols="60">
    def model_DirIndepOri(parsdict,Ndir,Nori):
        T=parsdict["T"]
        mA= parsdict['mA']
        sA= parsdict['sA']
        b = parsdict['b']
        R_min = parsdict['R_min']
        R_max = parsdict['R_max']
        theta0 = parsdict['theta0']
        k_o = parsdict['k_o']
        k_d = parsdict['k_d']
        #---------------------
        dirs = np.linspace(0, 2*np.pi, Ndir, endpoint=False)
        oris = np.linspace(np.pi/2,3*np.pi/2, Nori, endpoint=False)
        xv, yv = np.meshgrid (dirs,oris) #shape(Nori,Ndir),inversed
        #---------------------
        theta0 = theta0*np.pi/180.
        if k_o==0:
            f = np.exp(k_d*(-np.sin(xv-theta0)-1))
        else:
            f = np.exp(k_o*(np.cos(2*(yv-theta0))-1))
            f *= np.exp(k_d*(-np.sin(xv-theta0)-1))
        import matplotlib.mlab as mlab
        tc = np.zeros((Ndir,Nori,T))
        for t in range(T):
            A = mlab.normpdf(t,mA,sA)*np.sqrt(2*np.pi)*sA
            tc[:,:,t] = R_min + (R_max-R_min)*A*(b+(1-b)*f.transpose())
        return tc
 </textarea>
 """)

Modeling and Statistics : numpy,scipy

def model_DirIndepOri(parsdict,Ndir,Nori): T=parsdict["T"] mA= parsdict['mA'] sA= parsdict['sA'] b = parsdict['b'] R_min = parsdict['R_min'] R_max = parsdict['R_max'] theta0 = parsdict['theta0'] k_o = parsdict['k_o'] k_d = parsdict['k_d'] #--------------------- dirs = np.linspace(0, 2*np.pi, Ndir, endpoint=False) oris = np.linspace(np.pi/2,3*np.pi/2, Nori, endpoint=False) xv, yv = np.meshgrid (dirs,oris) #shape(Nori,Ndir),inversed #--------------------- theta0 = theta0*np.pi/180. if k_o==0: f = np.exp(k_d*(-np.sin(xv-theta0)-1)) else: f = np.exp(k_o*(np.cos(2*(yv-theta0))-1)) f *= np.exp(k_d*(-np.sin(xv-theta0)-1)) import matplotlib.mlab as mlab tc = np.zeros((Ndir,Nori,T)) for t in range(T): A = mlab.normpdf(t,mA,sA)*np.sqrt(2*np.pi)*sA tc[:,:,t] = R_min + (R_max-R_min)*A*(b+(1-b)*f.transpose()) return tc

HTML("""
 <h2>Ipython notebook + matplotlib</h2>
<img src="{0}" height=61%/>
 """.format(figpath + 'snapshot1.png'))

Ipython notebook + matplotlib

HTML("""
    <center>
    <h1>Thank you !!</h1> 
    <h1>Questions??</h1>
    <table border="0" width=100%/> 
    <tr height=100%/><img src="{0}" width=290 type='application/pdf'></tr> 
    </table>
 """.format(figpath+ 'questions.png'))

Thank you !!

Questions??

HTML("""
<h1 class="title">Decoding of feature selectivity in neural activity</h1>
<h1>Concrete applications in visual data</h1>
<h2>Wahiba Taouali, INMED - Laurent U Perrinet, INT</h2>
 <img src="{0}" height=61%/>
 """.format(figpath + 'troislogos.png'))

Decoding of feature selectivity in neural activity

Concrete applications in visual data

Wahiba Taouali, INMED - Laurent U Perrinet, INT

Acknowledgements:

• PhD program: Anna Montagnini, Frédéric Chavane, Nadia Pittet, INT, Marseille
• Material: Peggy Series, Christopher Olah
• NeuroPhysiology: Giacomo Benvenuti, Frédéric Chavane, INT, Marseille

ⓦ https://laurentperrinet.github.io/sciblog/files/2015-12-08_cours_neurocomp/PerrinetEtAl12neurocomp-intro.slides.html