Abstract:We present a unified computational theory of an agent's perception and memory. In our model, both perception and memory are realized by different operational modes of the oscillating interactions between a symbolic index layer and a subsymbolic representation layer. The two layers form a bilayer tensor network (BTN). The index layer encodes indices for concepts, predicates, and episodic instances. The representation layer broadcasts information and reflects the cognitive brain state; it is our model of what authors have called the "mental canvas" or the "global workspace." As a bridge between perceptual input and the index layer, the representation layer enables the grounding of indices by their subsymbolic embeddings, which are implemented as connection weights linking both layers. The propagation of activation to earlier perceptual processing layers in the brain can lead to embodiments of indices. Perception and memories first create subsymbolic representations, which are subsequently decoded semantically to produce sequences of activated indices that form symbolic triple statements. The brain is a sampling engine: only activated indices are communicated to the remaining parts of the brain. Triple statements are dynamically embedded in the representation layer and embodied in earlier processing layers: the brain speaks to itself. Although memory appears to be about the past, its main purpose is to support the agent in the present and the future. Recent episodic memory provides the agent with a sense of the here and now. Remote episodic memory retrieves relevant past experiences to provide information about possible future scenarios. This aids the agent in decision making. "Future" episodic memory, based on expected future events, guides planning and action. Semantic memory retrieves specific information, which is not delivered by current perception, and defines priors for future observations. We argue that it is important for the agent to encode individual entities, not just classes and attributes. Perception is learning: episodic memories are constantly being formed, and we demonstrate that a form of self-supervised learning can acquire new concepts and refine existing ones. We test our model on a standard benchmark data set, which we expanded to contain richer representations for attributes, classes, and individuals. Our key hypothesis is that obtaining a better understanding of perception and memory is a crucial prerequisite to comprehending human-level intelligence.

Tensor Product Production System: a Modular Architecture and Representation

Basic Reasoning with Tensor Product Representations

Attentive Tensor Product Learning

Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

Product of Tensors and Description of Networks

The Tensor Brain: A Unified Theory of Perception, Memory, and Semantic Decoding

The Tensor Brain: Semantic Decoding for Perception and Memory

Getting symbols out of a neural architecture

Kronecker Product of Tensors and Hypergraphs: Structure and Dynamics

Tensor Network Message Passing

A novel associative memory model based on semi-tensor product (STP)

The smash product of monoidal theories

Attention-based Iterative Decomposition for Tensor Product Representation

Stable Tensor Neural Networks for Rapid Deep Learning

The Tsetlin Machine -- A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic

The Tensor Memory Hypothesis

The SP theory of intelligence and the representation and processing of knowledge in the brain

Production System Rules as Protein Complexes from Genetic Regulatory Networks

Structured Production System (extended abstract)

Event Representations With Tensor-Based Compositions

Cognitive Production Systems: A Mapping Study