Abstract:Distributed sparse block codes (SBCs) exhibit compact representations for encoding and manipulating symbolic data structures using fixed-width vectors. One major challenge however is to disentangle, or factorize, the distributed representation of data structures into their constituent elements without having to search through all possible combinations. This factorization becomes more challenging when SBCs vectors are noisy due to perceptual uncertainty and approximations made by modern neural networks to generate the query SBCs vectors. To address these challenges, we first propose a fast and highly accurate method for factorizing a more flexible and hence generalized form of SBCs, dubbed GSBCs. Our iterative factorizer introduces a threshold-based nonlinear activation, conditional random sampling, and an $\ell_\infty$-based similarity metric. Secondly, the proposed factorizer maintains a high accuracy when queried by noisy product vectors generated using deep convolutional neural networks (CNNs). This facilitates its application in replacing the large fully connected layer (FCL) in CNNs, whereby $C$ trainable class vectors, or attribute combinations, can be implicitly represented by our factorizer having $F$-factor codebooks, each with $\sqrt[\leftroot{-2}\uproot{2}F]{C}$ fixed codevectors. We provide a methodology to flexibly integrate our factorizer in the classification layer of CNNs with a novel loss function. With this integration, the convolutional layers can generate a noisy product vector that our factorizer can still decode, whereby the decoded factors can have different interpretations based on downstream tasks. We demonstrate the feasibility of our method on four deep CNN architectures over CIFAR-100, ImageNet-1K, and RAVEN datasets. In all use cases, the number of parameters and operations are notably reduced compared to the FCL.

Block-Wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

XSepConv: Extremely Separated Convolution for Efficient Deep Networks with Large Kernels

XSepConv: Extremely Separated Convolution

Optimized Separable Convolution: Yet Another Efficient Convolution Operator.

FALCON: Lightweight and Accurate Convolution

Expansion-Squeeze- Block: Linear Over-parameterization with Shortcut Connections to Train Compact Convolutional Networks

A Lightweight Block with Information Flow Enhancement for Convolutional Neural Networks

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Factorizers for Distributed Sparse Block Codes

Flattening Singular Values of Factorized Convolution for Medical Images

Factoring 3D Convolutions for Medical Images by Depth-wise Dependencies-induced Adaptive Attention.

Compressing Convolutional Neural Networks Via Factorized Convolutional Filters

Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks

Optimizing Depthwise Separable Convolution Operations on GPUs

Falcon: lightweight and accurate convolution based on depthwise separable convolution

Comb Convolution for Efficient Convolutional Architecture

Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy

Learning Filter Basis for Convolutional Neural Network Compression

A High-Performance Convolution Block Oriented Accelerator for MBConv-Based CNNs

Factorized Bilinear Models for Image Recognition