Abstract:The efficient coding approach proposes that neural systems represent as much sensory information as biological constraints allow. It aims at formalizing encoding as a constrained optimal process. A different approach, that aims at formalizing decoding, proposes that neural systems instantiate a generative model of the sensory world. Here, we put forth a normative framework that characterizes neural systems as jointly optimizing encoding and decoding. It takes the form of a variational autoencoder: sensory stimuli are encoded in the noisy activity of neurons to be interpreted by a flexible decoder; encoding must allow for an accurate stimulus reconstruction from neural activity. Jointly, neural activity is required to represent the statistics of latent features which are mapped by the decoder into distributions over sensory stimuli; decoding correspondingly optimizes the accuracy of the generative model. This framework yields in a family of encoding-decoding models, which result in equally accurate generative models, indexed by a measure of the stimulus-induced deviation of neural activity from the marginal distribution over neural activity. Each member of this family predicts a specific relation between properties of the sensory neurons—such as the arrangement of the tuning curve means (preferred stimuli) and widths (degrees of selectivity) in the population—as a function of the statistics of the sensory world. Our approach thus generalizes the efficient coding approach. Notably, here, the form of the constraint on the optimization derives from the requirement of an accurate generative model, while it is arbitrary in efficient coding models. Moreover, solutions do not require the knowledge of the stimulus distribution, but are learned on the basis of data samples; the constraint further acts as regularizer, allowing the model to generalize beyond the training data. Finally, we characterize the family of models we obtain through alternate measures of performance, such as the error in stimulus reconstruction. We find that a range of models admits comparable performance; in particular, a population of sensory neurons with broad tuning curves as observed experimentally yields both low reconstruction stimulus error and an accurate generative model that generalizes robustly to unseen data.

Revealing nonlinear neural decoding by analyzing choices

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

Reviewer #2 (public Review): Revealing Unexpected Complex Encoding but Simple Decoding Mechanisms in Motor Cortex Via Separating Behaviorally Relevant Neural Signals

Comparisons Between Linear And Nonlinear Methods For Decoding Motor Cortical Activities Of Monkey

Neural stochastic codes, encoding and decoding

Cracking neural coding by nonlinear physics

How Can Single Sensory Neurons Predict Behavior?

Weighting neurons by selectivity produces near-optimal population codes

Decoding dynamic visual scenes across the brain hierarchy

Nonlinear decoding of a complex movie from the mammalian retina

Exploiting correlations across trials and behavioral sessions to improve neural decoding

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Decoding region-level visual functions from invasive EEG data

Choice can be predicted from populations of bursting neurons in superficial layers of monkey V1

Population Coding and Decoding in a Neural Field: A Computational Study

A Novel Brain Decoding Method: A Correlation Network Framework for Revealing Brain Connections

Where is all the nonlinearity: flexible nonlinear modeling of behaviorally relevant neural dynamics using recurrent neural networks

A nonlinear code for event probability in the human brain

Jointly efficient encoding and decoding in neural populations