Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course

Aadarsh Padiyath,Xinying Hou,Amy Pang,Diego Viramontes Vargas,Xingjian Gu,Tamara Nelson-Fromm,Zihan Wu,Mark Guzdial,Barbara Ericson

DOI: https://doi.org/10.1145/3632620.3671098

2024-06-11

Abstract:The capability of large language models (LLMs) to generate, debug, and explain code has sparked the interest of researchers and educators in undergraduate programming, with many anticipating their transformative potential in programming education. However, decisions about why and how to use LLMs in programming education may involve more than just the assessment of an LLM's technical capabilities. Using the social shaping of technology theory as a guiding framework, our study explores how students' social perceptions influence their own LLM usage. We then examine the correlation of self-reported LLM usage with students' self-efficacy and midterm performances in an undergraduate programming course. Triangulating data from an anonymous end-of-course student survey (n = 158), a mid-course self-efficacy survey (n=158), student interviews (n = 10), self-reported LLM usage on homework, and midterm performances, we discovered that students' use of LLMs was associated with their expectations for their future careers and their perceptions of peer usage. Additionally, early self-reported LLM usage in our context correlated with lower self-efficacy and lower midterm scores, while students' perceived over-reliance on LLMs, rather than their usage itself, correlated with decreased self-efficacy later in the course.

Human-Computer Interaction,Artificial Intelligence,Computers and Society

What problem does this paper attempt to address?

This paper attempts to explore the use of large language models (LLMs) in undergraduate programming courses. In particular, from the perspective of the social - technical shaping theory, it studies how students' social perceptions of LLMs affect their usage behavior, and the relationship between this use and students' programming self - efficacy and mid - term grades. Specifically, the researchers guided their research through the following two research questions:

1. **RQ1**: How do social perceptions affect the use of large language models in undergraduate intermediate - level programming courses?
2. **RQ2**: What is the association between the use of LLMs and undergraduates' programming self - efficacy and mid - term grades in intermediate - level programming courses?

To answer these questions, the researchers adopted a mixed - methods research design, including anonymous student surveys, student interviews, and regression analysis of mid - term grade data. The main purpose of the research is to provide evidence regarding the social dynamics of using LLMs in course work and their impact on the undergraduate programming learning experience.