Abstract:The availability of vast amounts of publicly accessible data of source code and the advances in modern language models, coupled with increasing computational resources, have led to a remarkable surge in the development of large language models for code (LLM4Code, for short). The interaction between code datasets and models gives rise to a complex ecosystem characterized by intricate dependencies that are worth studying. This paper introduces a pioneering analysis of the code model ecosystem. Utilizing Hugging Face -- the premier hub for transformer-based models -- as our primary source, we curate a list of datasets and models that are manually confirmed to be relevant to software engineering. By analyzing the ecosystem, we first identify the popular and influential datasets, models, and contributors. The popularity is quantified by various metrics, including the number of downloads, the number of likes, the number of reuses, etc. The ecosystem follows a power-law distribution, indicating that users prefer widely recognized models and datasets. Then, we manually categorize how models in the ecosystem are reused into nine categories, analyzing prevalent model reuse practices. The top 3 most popular reuse types are fine-tuning, architecture sharing, and quantization. We also explore the practices surrounding the publication of LLM4Code, specifically focusing on documentation practice and license selection. We find that the documentation in the ecosystem contains less information than that in general artificial intelligence (AI)-related repositories hosted on GitHub. Additionally, the license usage is also different from other software repositories. Models in the ecosystem adopt some AI-specific licenses, e.g., RAIL (Responsible AI Licenses) and AI model license agreement.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to study and analyze the ecosystem of large - scale language models for code (LLM4Code). Specifically, the authors hope to solve related problems through the following aspects: 1. **Define and understand the LLM4Code ecosystem**: - The paper first defines the LLM4Code ecosystem as a dynamic network consisting of multiple language models and datasets focused on software engineering tasks. - Research the interdependencies among these models and their evolution processes, and reveal how they evolve from base models to new task - specific models through reuse strategies such as fine - tuning, quantization, and adversarial training. 2. **Identify important entities**: - By analyzing the data on the Hugging Face platform, identify important datasets, models, and contributors in the ecosystem. This helps to understand which models or datasets are the most popular in the community and which contributors have played a key role in the construction of the ecosystem. - Use statistical testing methods to evaluate the difference in influence between corporate accounts and non - corporate accounts, and find that corporate - owned models are more popular in the community than non - corporate - owned models. 3. **Explore model reuse practices**: - Construct an LLM4Code reuse taxonomy with nine categories, analyze in detail different types of reuse patterns (such as fine - tuning, architecture sharing, quantization, etc.), and explore the motivations and development trends behind these patterns. - The analysis shows that fine - tuning is the most common reuse method, and the growth of quantization and distillation techniques reflects the increasing demand for deploying LLM4Code on resource - constrained devices. 4. **Evaluate documentation and license usage in release practices**: - Compare the documentation quality of LLM4Code with that of AI repositories on GitHub, and find that the documentation of LLM4Code is poorly maintained and provides less information. - Analyze the license usage situation, point out that more than 60% of the models do not provide license information, and the license distribution is different from other traditional software ecosystems. In addition, potential license conflict issues are also discussed. 5. **Automatically build and analyze the ecosystem**: - Use large - language models (LLMs) to assist in building and analyzing the LLM4Code ecosystem, demonstrating the potential of LLMs in automatically expanding datasets, which increases the size of manually - annotated datasets by 16.5 times. - Verify that the conclusions drawn from manually - annotated datasets can be generalized to automatically - created datasets, proving the effectiveness of this method. Through the above research, the authors hope to provide valuable insights for LLM4Code researchers and practitioners, helping them better understand and optimize this complex ecosystem, thereby promoting its healthy development.

Ecosystem of Large Language Models for Code

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code

The Stack: 3 TB of permissively licensed source code

Large Language Models Meet NL2Code: A Survey

ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks

An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code

An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

StarCoder: may the source be with you!

Where Are Large Language Models for Code Generation on GitHub?

A Systematic Evaluation of Large Language Models of Code

Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey

Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

A Survey on Large Language Models for Code Generation

Large Language Models in Computer Science Education: A Systematic Literature Review

Optimizing Code Retrieval: High-Quality and Scalable Dataset Annotation Through Large Language Models

L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models