Abstract:Vector space models for symbolic processing that encode symbols by random vectors have been proposed in cognitive science and connectionist communities under the names Vector Symbolic Architecture (VSA), and, synonymously, Hyperdimensional (HD) computing. In this paper, we generalize VSAs to function spaces by mapping continuous-valued data into a vector space such that the inner product between the representations of any two data points represents a similarity kernel. By analogy to VSA, we call this new function encoding and computing framework Vector Function Architecture (VFA). In VFAs, vectors can represent individual data points as well as elements of a function space (a reproducing kernel Hilbert space). The algebraic vector operations, inherited from VSA, correspond to well-defined operations in function space. Furthermore, we study a previously proposed method for encoding continuous data, fractional power encoding (FPE), which uses exponentiation of a random base vector to produce randomized representations of data points and fulfills the kernel properties for inducing a VFA. We show that the distribution from which elements of the base vector are sampled determines the shape of the FPE kernel, which in turn induces a VFA for computing with band-limited functions. In particular, VFAs provide an algebraic framework for implementing large-scale kernel machines with random features, extending Rahimi and Recht, 2007. Finally, we demonstrate several applications of VFA models to problems in image recognition, density estimation and nonlinear regression. Our analyses and results suggest that VFAs constitute a powerful new framework for representing and manipulating functions in distributed neural systems, with myriad applications in artificial intelligence.

Vector Symbolic Architectures for Context-Free Grammars

LARS-VSA: A Vector Symbolic Architecture For Learning with Abstract Rules

A comparison of Vector Symbolic Architectures

Computing on Functions Using Randomized Vector Representations

Residual and Attentional Architectures for Vector-Symbols

A Walsh Hadamard Derived Linear Vector Symbolic Architecture

Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures

Vector Symbolic Architectures as a Computing Framework for Emerging Hardware

Hyperdimensional Computing with Spiking-Phasor Neurons

VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images

SymbolicAI: A framework for logic-based approaches combining generative models and solvers

Cognitive modeling and learning with sparse binary hypervectors

From Frequency to Meaning: Vector Space Models of Semantics

Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

stl2vec: Semantic and Interpretable Vector Representation of Temporal Logic

PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols

Neural Vector Conceptualization for Word Vector Space Interpretation

Why concepts are (probably) vectors

Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders

GPT-2 Through the Lens of Vector Symbolic Architectures