Analyzing LLM Usage in an Advanced Computing Class in India

Anupam Garg,Aryaman Raina,Aryan Gupta,Jaskaran Singh,Manav Saini,Prachi Iiitd,Ronit Mehta,Rupin Oberoi,Sachin Sharma,Samyak Jain,Sarthak Tyagi,Utkarsh Arora,Dhruv Kumar

2024-07-26

Abstract:This study examines the use of large language models (LLMs) by undergraduate and graduate students for programming assignments in advanced computing classes. Unlike existing research, which primarily focuses on introductory classes and lacks in-depth analysis of actual student-LLM interactions, our work fills this gap. We conducted a comprehensive analysis involving 411 students from a Distributed Systems class at an Indian university, where they completed three programming assignments and shared their experiences through Google Form surveys. Our findings reveal that students leveraged LLMs for a variety of tasks, including code generation, debugging, conceptual inquiries, and test case creation. They employed a spectrum of prompting strategies, ranging from basic contextual prompts to advanced techniques like chain-of-thought prompting and iterative refinement. While students generally viewed LLMs as beneficial for enhancing productivity and learning, we noted a concerning trend of over-reliance, with many students submitting entire assignment descriptions to obtain complete solutions. Given the increasing use of LLMs in the software industry, our study highlights the need to update undergraduate curricula to include training on effective prompting strategies and to raise awareness about the benefits and potential drawbacks of LLM usage in academic settings.

Human-Computer Interaction,Computers and Society

What problem does this paper attempt to address?

The paper attempts to address the issue of how large language models (LLMs) are used by students for programming assignments in advanced computing courses, and the impact of this usage on student learning outcomes and productivity. Specifically, the paper aims to fill a gap in existing research, which primarily focuses on introductory courses and lacks in-depth analysis of actual student interactions with LLMs. By analyzing data from 411 students in a distributed systems course at an Indian university, the paper explores the following aspects:

1. **Usage Patterns of LLM Tools**: Identifying how students use LLM-based tools, with a focus on the types of queries they make and the prompting strategies they employ.
2. **Effectiveness of LLM Responses**: Evaluating the effectiveness of various LLMs in assisting students with their programming assignments.
3. **Student Perceptions of LLM Impact**: Understanding how students perceive the impact of LLMs on their learning outcomes and productivity.

Through these objectives, the paper aims to provide insights for educators and students on how to effectively utilize LLMs and to highlight potential issues and improvements when using LLMs in an academic setting.