Abstract:Large Language Models (LLMs) represent a leap in artificial intelligence, excelling in tasks using human language(s). Although the main focus of general-purpose LLMs is not code generation, they have shown promising results in the domain. However, the usefulness of LLMs in an academic software engineering project has not been fully explored yet. In this study, we explore the usefulness of LLMs for 214 students working in teams consisting of up to six members. Notably, in the academic course through which this study is conducted, students were encouraged to integrate LLMs into their development tool-chain, in contrast to most other academic courses that explicitly prohibit the use of LLMs.

What problem does this paper attempt to address?

This paper examines the usage and perception of Large Language Models (LLMs) in software engineering projects through empirical research. While LLMs are primarily used for natural language processing tasks, they have also shown potential in code generation. However, the practicality of LLMs in academic software engineering projects has not been fully explored. The research methodology involved observing 214 students grouped for C++ application development, encouraging them to incorporate LLMs into their development toolchain instead of prohibiting its use like other courses. By collecting AI-generated code, prompts for code generation, and the level of manual intervention required to integrate the AI code, the study analyzed the role of LLMs throughout the software development lifecycle. Additionally, a perception study was conducted to understand students' views on the use of LLMs and their future expectations. The study found that LLMs are particularly useful in the early stages of software development, such as creating basic design patterns, data structures, and algorithms, providing C++ syntax assistance, and aiding in error debugging. Students also used LLMs as programming partners to understand error messages, improve code quality, and follow best practices. The perception analysis showed that students believed LLMs increase productivity but also expressed concerns about their potential impact on cultivating coding skills and the software engineering job market. The main contributions of the paper include the first study on the use of LLMs in team software engineering projects and providing insights into their use in an academic environment. The study analyzed students' perceptions of using LLMs in software development, providing valuable insights for further incorporation of LLMs in teaching environments. All relevant data for further analysis by the research community is also provided. The research methodology includes a description of the project setup for the software engineering course, the data extraction methods, descriptions of user studies, and analysis of the results. The paper provides a detailed timeline of the project, key milestones, and assessment of student workload. Additionally, a user survey was conducted, analyzing the acceptance and usage behavior of LLMs in code generation using the UTAUT model. The results indicate that most AI-generated code snippets came from the free version of ChatGPT, while the paid version of ChatGPT required less manual intervention. Students were more inclined to use LLMs in the early stages of the project to obtain initial code structure. The study also revealed trends in prompting LLMs for code generation and the impact of different levels of intervention. Lastly, survey results showed that students with lower coding skills had lower willingness to use AI code generation, emphasizing the need for high-level coding abilities in future software engineers to effectively utilize these new technologies.

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

State of Practice: LLMs in Software Engineering and Software Architecture

LLMs for science: Usage for code generation and data analysis

Analyzing LLM Usage in an Advanced Computing Class in India

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

Using LLMs in Software Requirements Specifications: An Empirical Evaluation

"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students

Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

"With Great Power Comes Great Responsibility!": Student and Instructor Perspectives on the influence of LLMs on Undergraduate Engineering Education

LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

CS1-LLM: Integrating LLMs into CS1 Instruction

Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course

LLMs are Imperfect, Then What? An Empirical Study on LLM Failures in Software Engineering

How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts

Breaking the Silence: the Threats of Using LLMs in Software Engineering

Large Language Models for Software Engineering: Survey and Open Problems

A Survey on Large Language Models for Software Engineering

Large Language Models for Software Engineering: A Systematic Literature Review

Towards an Understanding of Large Language Models in Software Engineering Tasks