Abstract:Coding, which targets compressing and reconstructing data, and intelligence, often regarded at an abstract computational level as being centered around model learning and prediction, interweave recently to give birth to a series of significant progress. The recent trends demonstrate the potential homogeneity of these two fields, especially when deep-learning models aid these two categories for better probability modeling. For better understanding and describing from a unified perspective, inspired by the basic generally recognized principles in cognitive psychology, we formulate a novel problem of Coding for Intelligence from the category theory view. Based on the three axioms: existence of ideal coding, existence of practical coding, and compactness promoting generalization, we derive a general framework to understand existing methodologies, namely that, coding captures the intrinsic relationships of objects as much as possible, while ignoring information irrelevant to downstream tasks. This framework helps identify the challenges and essential elements in solving the specific derived Minimal Description Length (MDL) optimization problem from a broader range, providing opportunities to build a more intelligent system for handling multiple tasks/applications with coding ideas/tools. Centering on those elements, we systematically review recent processes of towards optimizing the MDL problem in more comprehensive ways from data, model, and task perspectives, and reveal their impacts on the potential CfI technical routes. After that, we also present new technique paths to fulfill CfI and provide potential solutions with preliminary experimental evidence. Last, further directions and remaining issues are discussed as well. The discussion shows our theory can reveal many phenomena and insights about large foundation models, which mutually corroborate with recent practices in feature learning.

Generalized Universal Coding of Integers

On the Minimum of the Expansion Factor for Universal Coding of Integers

A Tighter Upper Bound of the Expansion Factor for Universal Coding of Integers and Its Code Constructions

Generalized Pairwise Complementary Codes with Set-Wise Uniform Interference-Free Windows

The Universality of Spatially Coupled Generalized LDPC Codes over Memoryless Symmetric Channels

A Study on Universal Codes with Finite Block Lengths

PIR Codes, Unequal-Data-Demand Codes, and the Griesmer Bound

Bounds and Constructions for Generalized Batch Codes

Generalized Expanded-Blaum-Roth Codes and Their Efficient Encoding/Decoding

Universal coding, intrinsic volumes, and metric complexity

Generalized Low-Density Parity-Check Codes Over Set {0,1,...,2(K)-1}

Generalized Gaussian Multiterminal Source Coding: the Symmetric Case.

Generalized Hamming Weights of Irreducible Cyclic Codes.

A Generalized Construction of Asymptotically Optimal Codebooks.

The Design of Binary Shaping Filter of Binary Code.

Quotient Subcode Set And Inverse Relative Dimension/Length Profile

Coding for Intelligence from the Perspective of Category

A General Coding Scheme for Two-User Fading Interference Channels

The Universality of Generalized Hamming Code for Multiple Sources

Structured Non-Binary LDPC Codes with Large Girth

On the Capacity of Locally Decodable Codes