Abstract:Large language models (LLM) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like LaPM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Large Language Model Performance.

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication

StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text

"We Need Structured Output": Towards User-centered Constraints on Large Language Model Output

LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs

Open-domain Implicit Format Control for Large Language Model Generation

Concise and Organized Perception Facilitates Reasoning in Large Language Models

Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models

Case Study: Testing Model Capabilities in Some Reasoning Tasks

Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL

The Impact of Large Language Models in Academia: from Writing to Speaking

Struct-X: Enhancing Large Language Models Reasoning with Structured Data

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

A Comprehensive Evaluation of Constrained Text Generation for Large Language Models.

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

CLR-Bench: Evaluating Large Language Models in College-level Reasoning

Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks

Reasoning Factual Knowledge in Structured Data with Large Language Models

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness