Abstract:Distributed sparse block codes (SBCs) exhibit compact representations for encoding and manipulating symbolic data structures using fixed-width vectors. One major challenge however is to disentangle, or factorize, the distributed representation of data structures into their constituent elements without having to search through all possible combinations. This factorization becomes more challenging when SBCs vectors are noisy due to perceptual uncertainty and approximations made by modern neural networks to generate the query SBCs vectors. To address these challenges, we first propose a fast and highly accurate method for factorizing a more flexible and hence generalized form of SBCs, dubbed GSBCs. Our iterative factorizer introduces a threshold-based nonlinear activation, conditional random sampling, and an $\ell_\infty$-based similarity metric. Secondly, the proposed factorizer maintains a high accuracy when queried by noisy product vectors generated using deep convolutional neural networks (CNNs). This facilitates its application in replacing the large fully connected layer (FCL) in CNNs, whereby $C$ trainable class vectors, or attribute combinations, can be implicitly represented by our factorizer having $F$-factor codebooks, each with $\sqrt[\leftroot{-2}\uproot{2}F]{C}$ fixed codevectors. We provide a methodology to flexibly integrate our factorizer in the classification layer of CNNs with a novel loss function. With this integration, the convolutional layers can generate a noisy product vector that our factorizer can still decode, whereby the decoded factors can have different interpretations based on downstream tasks. We demonstrate the feasibility of our method on four deep CNN architectures over CIFAR-100, ImageNet-1K, and RAVEN datasets. In all use cases, the number of parameters and operations are notably reduced compared to the FCL.

Factorizers for Distributed Sparse Block Codes

On the Role of Noise in Factorizers for Disentangling Distributed Representations

Factorization of Broad Expansion for Broad Learning System

Statistically Guided Divide-and-Conquer for Sparse Factorization of Large Matrix

FactorizeNet: Progressive Depth Factorization for Efficient Network Architecture Exploration Under Quantization Constraints

Structural Optimization of Factor Graphs for Symbol Detection via Continuous Clustering and Machine Learning

Factorized Bilinear Models for Image Recognition

Gohberg-Semencul Factorization-Based Fast Implementation of Sparse Bayesian Learning With a Fourier Dictionary

Factor Graph Optimization of Error-Correcting Codes for Belief Propagation Decoding

Block-Wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions

Distributed Flexible Nonlinear Tensor Factorization

Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization

Streaming Probabilistic Deep Tensor Factorization

Factorizable Graph Convolutional Networks.

Sparse Factorization of Large Square Matrices

NetSMF: Large-Scale Network Embedding As Sparse Matrix Factorization

Accelerated Structured Matrix Factorization

Stochastic Subsampling for Factorizing Huge Matrices

Robust Factorization and Completion of Streaming Tensor Data via Variational Bayesian Inference

Deep Factorization for Speech Signal

Discrete Denoising Diffusion Approach to Integer Factorization