Abstract:Deep learning has advanced from fully connected architectures to structured models organized into components, e.g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. In structured models, an interesting question is how to conduct dynamic and possibly sparse communication among the separate components. Here, we explore the hypothesis that restricting the transmitted information among components to discrete representations is a beneficial bottleneck. The motivating intuition is human language in which communication occurs through discrete symbols. Even though individuals have different understandings of what a "cat" is based on their specific experiences, the shared discrete token makes it possible for communication among individuals to be unimpeded by individual differences in internal representation. To discretize the values of concepts dynamically communicated among specialist components, we extend the quantization mechanism from the Vector-Quantized Variational Autoencoder to multi-headed discretization with shared codebooks and use it for discrete-valued neural communication (DVNC). Our experiments show that DVNC substantially improves systematic generalization in a variety of architectures -- transformers, modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method very useful in practice. Moreover, we establish a theoretical justification of our discretization process, proving that it has the ability to increase noise robustness and reduce the underlying dimensionality of the model.

Discrete and continuous representations and processing in deep learning: Looking forward

Discrete-Valued Neural Communication

Neural Networks and Continuous Time

Emergence of machine language: towards symbolic intelligence with neural networks

Computational principles of intelligence: learning and reasoning with neural networks

Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey

Unified Representations for Learning and Reasoning

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Disentangled Representations in Neural Models

Classification-Driven Discrete Neural Representation Learning for Semantic Communications

A unified neural representation model for spatial and semantic computations

Discrete, compositional, and symbolic representations through attractor dynamics

Rethinking Arithmetic for Deep Neural Networks

Understanding Visual Concepts with Continuation Learning

Visual Interpretability forDeepLearning

Discrete Latent Structure in Neural Networks

Representations and generalization in artificial and brain neural networks

Efficient Learning of Discrete-Continuous Computation Graphs

Discrete Neural Algorithmic Reasoning

Harnessing Discrete Representations For Continual Reinforcement Learning