Abstract:Large language models (LLMs) have significantly advanced the field of natural language processing (NLP), providing a highly useful, task-agnostic foundation for a wide range of applications. However, directly applying LLMs to solve sophisticated problems in specific domains meets many hurdles, caused by the heterogeneity of domain data, the sophistication of domain knowledge, the uniqueness of domain objectives, and the diversity of the constraints (e.g., various social norms, cultural conformity, religious beliefs, and ethical standards in the domain applications). Domain specification techniques are key to make large language models disruptive in many applications. Specifically, to solve these hurdles, there has been a notable increase in research and practices conducted in recent years on the domain specialization of LLMs. This emerging field of study, with its substantial potential for impact, necessitates a comprehensive and systematic review to better summarize and guide ongoing work in this area. In this article, we present a comprehensive survey on domain specification techniques for large language models, an emerging direction critical for large language model applications. First, we propose a systematic taxonomy that categorizes the LLM domain-specialization techniques based on the accessibility to LLMs and summarizes the framework for all the subcategories as well as their relations and differences to each other. Second, we present an extensive taxonomy of critical application domains that can benefit dramatically from specialized LLMs, discussing their practical significance and open challenges. Last, we offer our insights into the current research status and future trends in this area.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How can large language models (LLMs) be made to solve problems more effectively in specific domains through domain specialization? Although large language models have made remarkable progress in the field of natural language processing (NLP), when directly applying these models to complex problems in specific domains, there are still many challenges. Specifically, these challenges include: 1. **Heterogeneity of domain data**: The data formats, structures, and contents in different domains vary greatly. 2. **Complexity of domain knowledge**: Each domain has its own unique terms, concepts, and logical relationships. 3. **Uniqueness of domain goals**: Different domains have different requirements and optimization goals for tasks. 4. **Diversity of constraints**: For example, social norms, cultural consistency, religious beliefs, and ethical standards, etc. To overcome these problems, researchers have proposed domain - specialization techniques. These techniques aim to customize general large - language models according to the context data, professional knowledge, optimization goals, and constraints of specific domains. The main contributions of the paper include: - Proposing a systematic taxonomy, dividing domain - specialization techniques into three levels: black - box, gray - box, and white - box, and discussing in detail the relationships, advantages, and disadvantages of each sub - category. - Summarizing the key application areas that can benefit from domain - specialized large - language models, and discussing the practical significance and open challenges in these areas. - Conducting an in - depth analysis of the current research status, providing insights into future research trends, and pointing out the current bottlenecks, open problems, and possible research directions. ### Specific Problems and Solutions #### Challenge 1: Keeping the LLM's Knowledge Up - to - Date Although the training corpus of the LLM is large, it has a knowledge cut - off period and it is difficult to obtain the latest information in a timely manner. To solve this problem, researchers propose that it is necessary to retrain regularly or introduce a continuous learning mechanism to ensure that the model can keep up with the latest development trends. #### Challenge 2: Learning Professional Knowledge in All Domains By default, the LLM has extensive general knowledge, but there may be a lack of knowledge in some specific domains. For this reason, researchers suggest guiding the model to generate more relevant and accurate responses by providing a small number of task - specific examples, thereby improving the model's performance on specific tasks. #### Challenge 3: Model and Computational Complexity of Downstream Task Learning Downstream task learning for adapting to specific - domain applications requires a large amount of high - quality, task - specific data, which is not only time - consuming but also resource - intensive. In addition, the complex architecture of the LLM also makes it difficult to select an appropriate downstream task learning strategy. Researchers point out that model performance can be optimized by adjusting hyper - parameters, learning rates, and training times while avoiding catastrophic forgetting. ### Taxonomy Overview The paper proposes a systematic taxonomy, dividing domain - specialization techniques into the following categories: 1. **External Augmentation (Black - Box)**: It does not require access to the internal parameter space of the LLM, but injects domain knowledge into the input prompt or the generated output through external resources or tools. 2. **Prompt Crafting (Gray - Box)**: Designs various types of prompts by accessing the gradients or loss values of the LLM to more finely control the model's behavior. 3. **Model Fine - tuning (White - Box)**: Requires full access to the LLM, including parameter settings, training data, and model architecture, and directly incorporates domain knowledge into the model by updating the model parameters. ### Application Area Classification The paper also summarizes the key application areas that can benefit from domain - specialized large - language models, such as medicine, law, finance, etc., and discusses the practical significance and open challenges in these areas. ### Future Trends and Prospects Finally, the paper looks forward to future research trends, points out the current bottlenecks, open problems, and possible research directions, providing valuable references for follow - up research. Through these methods and techniques, researchers hope to be able to better utilize large - language models to solve complex problems in specific domains and promote the application and development of artificial intelligence in more fields.

Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey

Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey

Large Language Models Meet NLP: A Survey

Scientific Large Language Models: A Survey on Biological & Chemical Domains

A Survey of Large Language Models

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

Unraveling the landscape of large language models: a systematic review and future perspectives

Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications

Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers

A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law

History, Development, and Principles of Large Language Models-An Introductory Survey

Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey

A Survey for Large Language Models in Biomedicine

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

Current Status and Trends in Large Language Modeling Research

A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

A Systematic Survey on Large Language Models for Algorithm Design

An Interdisciplinary Outlook on Large Language Models for Scientific Research

A Survey on Evaluation of Large Language Models

Exploring the landscape of large language models: Foundations, techniques, and challenges