Abstract:Recent advances in Large Language Models (LLMs) have demonstrated the emergence of capabilities (learned skills) when the number of system parameters and the size of training data surpass certain thresholds. The exact mechanisms behind such phenomena are not fully understood and remain a topic of active research. Inspired by the skill-text bipartite graph model proposed by Arora and Goyal for modeling semantic languages, we develop a mathematical theory to explain the emergence of learned skills, taking the learning (or training) process into account. Our approach models the learning process for skills in the skill-text bipartite graph as an iterative decoding process in Low-Density Parity Check (LDPC) codes and Irregular Repetition Slotted ALOHA (IRSA). Using density evolution analysis, we demonstrate the emergence of learned skills when the ratio of the number of training texts to the number of skills exceeds a certain threshold. Our analysis also yields a scaling law for testing errors relative to this ratio. Upon completion of the training, the association of learned skills can also be acquired to form a skill association graph. We use site percolation analysis to derive the conditions for the existence of a giant component in the skill association graph. Our analysis can also be extended to the setting with a hierarchy of skills, where a fine-tuned model is built upon a foundation model. It is also applicable to the setting with multiple classes of skills and texts. As an important application, we propose a method for semantic compression and discuss its connections to semantic communication.

The Theory and Application of Latent Semantic Analysis

Text Information Retrieval Based on Latent Semantic Analysis

Query Subtopic Mining Via Subtractive Initialization of Non-negative Sparse Latent Semantic Analysis

How Does Latent Semantic Analysis Work? A Visualisation Approach

On the connections between explicit semantic analysis and latent semantic analysis.

LATENT SEMANTIC INDEXING (LSI) AND ITS APPLICATIONS IN CHINESE TEXT PROCESSING

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Latent Semantic Scaling: A Semisupervised Text Analysis Technique for New Domains and Languages

A new latent semantic analysis language model

On the Semantics of LM Latent Space: A Vocabulary-defined Approach

LSASGT:an Approach to Text Categorization Based on Latent Semantic Analysis and Spectral Graph Transducer

Statistical Word Sense Aware Topic Models

Using Word Sense As a Latent Variable in LDA Can Improve Topic Modeling.

Semantic Text Mining with Linked Data

Latent Space Cartography: Visual Analysis of Vector Space Embeddings

fPLSA: Learning Semantic Structures in Document Collections Using Foundation Models

Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey

Search-Based Automatic Web Image Annotation Using Latent Visual and Semantic Analysis

A Mathematical Theory for Learning Semantic Languages by Abstract Learners

A Simplified Latent Semantic Indexing Approach for Multi-Linguistic Information Retrieval.

An Ontology-Based Latent Semantic Indexing Approach Using Long Short-Term Memory Networks.