Abstract:Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired representations -- those that are nonnegative and energy efficient -- modularise with respect to source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather, we show that sources modularise if their support is "sufficiently spread". From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data. First, we explain why two studies that recorded prefrontal activity in working memory tasks conflict on whether memories are encoded in orthogonal subspaces: the support of the sources differed due to a critical discrepancy in experimental protocol. Second, we use similar arguments to understand why preparatory and potent subspaces in RNN models of motor cortex are only sometimes orthogonal. Third, we study spatial and reward information mixing in entorhinal recordings, and show our theory matches data better than previous work. And fourth, we suggest a suite of surprising settings in which neurons can be (or appear) mixed selective, without requiring complex nonlinear readouts as in traditional theories. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.

Co-Evolutionary Modular Neural Networks for Automatic Problem Decomposition

Multi-network Evolutionary Systems and Automatic Decomposition of Complex Problems

Evolving Modular Neural Networks Which Generalise Well

ModularBoost: an Efficient Network Inference Algorithm Based on Module Decomposition.

Neural Network Module Decomposition and Recomposition

Task Decomposition Based on Class Relations: A Modular Neural Network Architecture for Pattern Classification

Breaking Neural Network Scaling Laws with Modularity

Independent Modular Networks

Task Decomposition and Module Combination Based on Class Relations: a Modular Neural Network for Pattern Classification

Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning

Modularity as a Means for Complexity Management in Neural Networks Learning

Automatic modularization by speciation

Decomposition and Parallel Learning of Imbalanced Classification Problems by Min-Max Modular Neural Network

Evolutionary Multi-Task Learning for Modular Extremal Learning Machine

Modularity based linkage model for neuroevolution

Cooperative Co-Evolution with Differential Grouping for Large Scale Optimization

Don't Cut Corners: Exact Conditions for Modularity in Biologically Inspired Representations

Modular neural network via exploring category hierarchy

ReCo: A Modular Neural Framework for Automatically Recommending Connections in Software Models

Efficient Evolutionary Neural Architecture Search by Modular Inheritable Crossover

Automatic Divide-And-Conquer Using Populations and Ensembles