LLMs Integration in Software Engineering Team Projects: Roles, Impact, and a Pedagogical Design Space for AI Tools in Computing Education

Ahmed Kharrufa,Sami Alghamdi,Abeer Aziz,Christopher Bull

2024-10-30

Abstract:This work takes a pedagogical lens to explore the implications of generative AI (GenAI) models and tools, such as ChatGPT and GitHub Copilot, in a semester-long 2nd-year undergraduate Software Engineering Team Project. Qualitative findings from survey (39 students) and interviews (eight students) provide insights into the students' views on the impact of GenAI use on their coding experience, learning, and self-efficacy. Our results address a particular gap in understanding the role and implications of GenAI on teamwork, team-efficacy, and team dynamics. The analysis of the learning aspects is distinguished by the application of learning and pedagogy informed lenses to discuss the data. We propose a preliminary design space for GenAI-based programming learning tools highlighting the importance of considering the roles that GenAI can play during the learning process, the varying support-ability patterns that can be applied to each role, and the importance of supporting transparency in GenAI for team members and students in addition to educators.

Software Engineering,Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to understand the role and impact of generative artificial intelligence (GenAI) tools in software engineering team projects, especially from the practical use experience of students, and explore the impact of these tools on students' programming experience, learning effect, self - efficacy and team - work dynamics. Specifically, the paper aims to fill the following research gaps:

1. **Students' Perspectives in the Long - Term View**: Through a semester - long software engineering project, understand the views of computing science students on the use of GenAI tools.
2. **Team - work Perspective**: Understand the impact of GenAI on team - work in the context of software engineering, including team effectiveness and team motivation.
3. **Pedagogical Perspective**: Apply learning theories and pedagogical perspectives to analyze and report the collected data, in order to better understand the impact of GenAI on the students' learning process.
4. **Design Perspective**: Propose a preliminary design space for designing GenAI - based programming learning tools/environments, emphasizing the consideration of the possible roles and support modes that GenAI may play in the learning process.

### Main Research Questions

- **The Impact of GenAI on Students' Programming Experience and Learning**: Through surveys and interviews, the paper explores how GenAI tools affect students' programming experience and learning effect. For example, how do students view the help of these tools to their coding ability, and whether they improve students' self - confidence.

- **The Impact of GenAI on Team - work**: The paper pays special attention to the role of GenAI in team projects and analyzes how it affects the interaction and collaboration among team members. The research finds that GenAI not only changes the way individuals work, but also has an important impact on the overall communication and collaboration of the team.

- **Instructional Design and Transparency**: The paper emphasizes the factors that need to be considered when designing GenAI tools, such as the role of the tool, support mode and transparency. This helps to ensure that these tools can effectively support students' learning and be reasonably utilized in the team environment.

### Research Methods

To answer these questions, the author adopts a mixed - method, combining quantitative surveys and qualitative interviews. The specific steps include:

1. **Survey Questionnaire**: A total of 39 students participated in the survey, and the questionnaire content covered personal background information, the use of GenAI tools, overall experience and impact, etc.
2. **In - depth Interviews**: Subsequently, 8 students received in - depth interviews, providing more detailed and personalized insights, helping researchers to more comprehensively understand the real experience of students.

Through this method, the paper not only reveals the application effect of GenAI tools in actual teaching, but also provides valuable references for future research and practice.