Abstract:Language models (LMs) are often expected to generate strings in some formal language; for example, structured data, API calls, or code snippets. Although LMs can be tuned to improve their adherence to formal syntax, this does not guarantee conformance, especially with smaller LMs suitable for large-scale deployment. In addition, tuning requires significant resources, making it impractical for uncommon or task-specific formats. To prevent downstream parsing errors we would ideally constrain the LM to only produce valid output, but this is severely complicated by tokenization, which is typically both ambiguous and misaligned with the formal grammar. We solve these issues through the application of automata theory, deriving an efficient closed-form solution for the regular languages, a broad class of formal languages with many practical applications, including API calls or schema-guided JSON and YAML. We also discuss pragmatic extensions for coping with the issue of high branching factor, and extend our techniques to deterministic context-free languages, which similarly admit an efficient closed-form solution. Previous work on this topic (Willard and Louf, 2023) layers bespoke solutions onto automata, leading to problems with speed, correctness, and extensibility. Instead, we reformulate the entire task in terms of automata so we can leverage well-studied and well-optimized algorithms. Our system compiles constraints ~7,000x faster, is provably correct, and can be extended in a modular fashion.

Robust models to infer flexible nondeterministic finite automata

Grammatical Inference as a Satisfiability Modulo Theories Problem

Classifying Words with 3-sort Automata

Statistical Inference and Probabilistic Modelling for Constraint-Based NLP

Rigorous Assessment of Model Inference Accuracy using Language Cardinality

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

Robustness of the Random Language Model

FlexFringe: Modeling Software Behavior by Learning Probabilistic Automata

Recovering Models of Network Protocol Using Grammatical Inference

Fine-Tuning Generative Models as an Inference Method for Robotic Tasks

Automata-based constraints for language model decoding

REINAM: Reinforcement Learning for Input-Grammar Inference.

Unsupervised Grammar Induction with Depth-bounded PCFG

Learning Automata Representation Of Network Protocol By Grammar Induction

Finite Automata for Efficient Graph Recognition

Amortizing intractable inference in large language models

Alembic: Automated Model Inference For Stateful Network Functions

Enhanced Lstm For Natural Language Inference

Low-Rank Constraints for Fast Inference in Structured Models