The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Steven I. Ross,Fernando Martinez,Stephanie Houde,Michael Muller,Justin D. Weisz

DOI: https://doi.org/10.1145/3581641.3584037

2023-02-14

Abstract:Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model's responses. We developed a prototype system -- the Programmer's Assistant -- in order to explore the utility of conversational interactions grounded in code, as well as software engineers' receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant's capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.

Human-Computer Interaction

What problem does this paper attempt to address?

The problem this paper attempts to address is whether modern code-fluent foundational models (i.e., large language models fine-tuned on source code data) are sufficient to support a conversational agent that can act as an assistant in the software development process. Specifically, the authors developed a prototype system called "Programmer Assistant" to explore the capabilities that conversational interaction can achieve, as well as the needs and acceptance of users for conversational assistance in programming tasks.

The paper mentions that although existing tools based on large language models (such as GitHub Copilot) perform well in code auto-completion, these tools typically treat each model invocation as an independent event, lacking contextual connection with prior user interactions, thereby limiting their potential helpfulness. Therefore, the paper explores how to more deeply leverage the capabilities of large language models by constructing a conversational system that can engage in multi-turn dialogues, understand user needs, and provide relevant code generation and other forms of assistance, aiming to enhance the productivity and creativity of software engineers.