Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

Deborah Etsenake,Meiyappan Nagappan

2024-10-02

Abstract:Large Language Models (LLMs) are transforming programming practices, offering significant capabilities for code generation activities. While researchers have explored the potential of LLMs in various domains, this paper focuses on their use in programming tasks, drawing insights from user studies that assess the impact of LLMs on programming tasks. We first examined the user interaction behaviors with LLMs observed in these studies, from the types of requests made to task completion strategies. Additionally, our analysis reveals both benefits and weaknesses of LLMs showing mixed effects on the human and task. Lastly, we looked into what factors from the human, LLM or the interaction of both, affect the human's enhancement as well as the task performance. Our findings highlight the variability in human-LLM interactions due to the non-deterministic nature of both parties (humans and LLMs), underscoring the need for a deeper understanding of these interaction patterns. We conclude by providing some practical suggestions for researchers as well as programmers.

Software Engineering,Human-Computer Interaction

What problem does this paper attempt to address?

The paper attempts to address the issue of understanding and evaluating the use of large language models (LLMs) in programming tasks, particularly the interaction patterns between humans and LLMs and their impact on human capabilities and task performance. Specifically, the study focuses on the following aspects: 1. **How programmers interact with LLMs**: By observing user interactions with LLMs, understanding their specific behaviors in terms of request types, task completion strategies, etc. 2. **Whether using LLMs can enhance human capabilities**: Evaluating the effectiveness of LLMs in improving human programming abilities, including productivity, learning outcomes, etc. 3. **Whether using LLMs can improve task performance**: Assessing the performance of LLM-generated code in terms of quality, readability, correctness, etc. 4. **Which interaction behaviors can enhance human capabilities or improve task performance**: Analyzing specific interaction patterns and strategies to identify methods that can effectively enhance human capabilities and task outcomes. The paper systematically analyzes existing user studies through a literature review, aiming to provide practical recommendations for researchers and programmers and to reveal the shortcomings in current research.

Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?

Understanding User Experience in Large Language Model Interactions

Analyzing LLM Usage in an Advanced Computing Class in India

A Survey on Human-Centric LLMs

LLMs for science: Usage for code generation and data analysis

Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course

CS1-LLM: Integrating LLMs into CS1 Instruction

Large Language Models Humanize Technology

"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students

Low-code LLM: Graphical User Interface over Large Language Models

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

"I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices

State of Practice: LLMs in Software Engineering and Software Architecture

Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics

An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support

A Survey of Useful LLM Evaluation

How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts

On the Opportunities of Large Language Models for Programming Process Data