Abstract:Recent In-IDE AI coding assistant tools (ACATs) like GitHub Copilot have significantly impacted developers' coding habits. While some studies have examined their effectiveness, there lacks in-depth investigation into the actual assistance process. To bridge this gap, we simulate real development scenarios encompassing three typical types of software development tasks and recruit 27 computer science students to investigate their behavior with three popular ACATs. Our goal is to comprehensively assess ACATs' effectiveness, explore characteristics of recommended code, identify reasons for modifications, and understand users' challenges and expectations. To facilitate the study, we develop an experimental platform that includes a data collection plugin for VSCode IDE and provides functions for screen recording, code evaluation, and automatic generation of personalized interview and survey questions. Through analysis of the collected data, we find that ACATs generally enhance task completion rates, reduce time, improve code quality, and increase self-perceived productivity. However, the improvement is influenced by both the nature of coding tasks and users' experience level. Notably, for experienced participants, the use of ACATs may even increase completion time. We observe that "edited line completion" is the most frequently recommended way, while "comments completion" and "string completion" have the lowest acceptance rates. The primary reasons for modifying recommended code are disparities between output formats and requirements, flawed logic, and inconsistent code styles. In terms of challenges and expectations, optimization of service access and help documentation is also concerned by participants except for functionality and performance. Our study provides valuable insights into the effectiveness and usability of ACATs, informing further improvements in their design and implementation.

How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

How Do Analysts Understand and Verify AI-Assisted Data Analyses?

Understanding the Usability of AI Programming Assistants

Information Seeking Using AI Assistants

Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward

A Wizard of Oz Study Simulating API Usage Dialogues with a Virtual Assistant

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

Beyond Recommender: An Exploratory Study of the Effects of Different AI Roles in AI-Assisted Decision Making

A Wizard of Oz Study of Guidance Strategies and Dynamics

Need Help? Designing Proactive AI Assistants for Programming

"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study

Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns

Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise

How far are AI-powered programming assistants from meeting developers' needs?

The Role of Heuristics and Biases During Complex Choices with an AI Teammate

Does Co-Development with AI Assistants Lead to More Maintainable Code? A Registered Report

Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions

Three Challenges for AI-Assisted Decision-Making

Do Users Write More Insecure Code with AI Assistants?

Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes