Challenges of Using Pre-trained Models: the Practitioners' Perspective

Xin Tan,Taichuan Li,Ruohe Chen,Fang Liu,Li Zhang

2024-05-01

Abstract:The challenges associated with using pre-trained models (PTMs) have not been specifically investigated, which hampers their effective utilization. To address this knowledge gap, we collected and analyzed a dataset of 5,896 PTM-related questions on Stack Overflow. We first analyze the popularity and difficulty trends of PTM-related questions. We find that PTM-related questions are becoming more and more popular over time. However, it is noteworthy that PTM-related questions not only have a lower response rate but also exhibit a longer response time compared to many well-researched topics in software engineering. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs. To delve into the specific challenges, we manually annotate 430 PTM-related questions, categorizing them into a hierarchical taxonomy of 42 codes (i.e., leaf nodes) and three categories. This taxonomy encompasses many PTM prominent challenges such as fine-tuning, output understanding, and prompt customization, which reflects the gaps between current techniques and practical needs. We discuss the implications of our study for PTM practitioners, vendors, and educators, and suggest possible directions and solutions for future research.

Software Engineering

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the challenges faced when using pre - trained models (PTMs). Specifically, the authors found that although PTMs have achieved remarkable success in many fields, in practical applications, users and developers encounter numerous difficulties when using these models. These problems have not been systematically investigated and analyzed, which has hindered the effective utilization of PTMs.

To fill this knowledge gap, the authors carried out the following tasks:

1. **Collecting and analyzing data**: They collected and analyzed 5,896 question - and - answer posts related to PTMs from Stack Overflow, spanning from 2018 to 2023.

2. **Trend analysis**: They studied the popularity trends and difficulty differences of PTM - related problems. The results show that although the number of PTM - related problems has increased year by year, the response rate of these problems is low, and the time required for answering is long, which reflects the high complexity and difficulty in the practical application of PTMs.

3. **Classification and coding**: By manually annotating 430 PTM - related questions, a hierarchical taxonomy containing 42 codes (i.e., leaf nodes) and three categories was established. These categories cover the main challenges of PTMs, such as fine - tuning, output understanding, and prompt customization, etc.

4. **Proposing improvement suggestions**: Based on the above analysis, the authors discussed the implications of their research for PTM practitioners, suppliers, and educators, and proposed directions for future research and possible solutions.

### Main contributions of the paper

- **Systematic analysis**: It provides a comprehensive overview of the main challenges and requirements faced by PTM users.
- **Providing guidance**: It provides valuable insights and guidance for PTM practitioners, suppliers, and educators.
- **Public data set**: It created and released a data set containing 5,896 questions, labeled with categories and metadata, which can be used for further research.

Through these efforts, the paper reveals the unique challenges in the practical application of PTMs and provides directions for improving the use of PTMs.