Abstract:AI assistants are being created to help software engineers conduct a variety of coding-related tasks, such as writing, documenting, and testing code. We describe the use of the watsonx Code Assistant (WCA), an LLM-powered coding assistant deployed internally within IBM. Through surveys of two user cohorts (N=669) and unmoderated usability testing (N=15), we examined developers' experiences with WCA and its impact on their productivity. We learned about their motivations for using (or not using) WCA, we examined their expectations of its speed and quality, and we identified new considerations regarding ownership of and responsibility for generated code. Our case study characterizes the impact of an LLM-powered assistant on developers' perceptions of productivity and it shows that although such tools do often provide net productivity increases, these benefits may not always be experienced by all users.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to explore the impact of AI code assistants (especially IBM's watsonx Code Assistant, abbreviated as WCA) on developers' productivity and experience in the enterprise environment. Specifically, the main research questions include: 1. **Developers' use of AI code assistants**: - Why do developers choose to use or not use WCA? - How do developers use the content generated by WCA? 2. **The impact of WCA on productivity**: - How does WCA affect developers' productivity, including work speed, code quality, and self - efficacy? - Are the feelings of different user groups about productivity improvement consistent? 3. **Code ownership and responsibility**: - When using the code generated by WCA, who should be entitled to the authorship, and who should be responsible for avoiding copyrighted content? - The issue of the distribution of responsibilities between users and AI systems. 4. **The impact on professional roles**: - Will AI assistants change developers' professional roles, for example, enabling them to focus on higher - level tasks or possibly leading to skill degradation? - Will the increase in productivity bring higher work expectations? ### Research methods To answer the above questions, the researchers adopted a mixed - method approach, including: - **Large - scale survey**: A questionnaire survey was conducted among 669 internal WCA users to understand their usage motives, the impact on productivity, etc. - **Unmoderated usability test**: 15 participants were selected for a small - scale unmoderated usability test to evaluate the user experience of specific functions of WCA. Through these methods, the researchers hope to comprehensively understand the application effect of AI code assistants in the actual work environment and their impact on developers. ### Main findings - **Main uses**: The main uses of WCA are concentrated on code understanding (such as explaining code and answering programming questions), rather than simply generating code. - **Impact on productivity**: Although overall WCA has improved net productivity, this improvement is not evenly distributed among all users. Some users reported significant productivity improvements, while others encountered quality problems or needed additional verification of the generated content. - **Code ownership and responsibility**: Users believe that WCA should be entitled to partial authorship in the co - creation process, but the ultimate responsibility still lies with human developers. - **Changes in professional roles**: AI assistants enable developers to focus on higher - level tasks, but may also increase productivity expectations. Through these findings, the paper provides valuable insights for understanding the application of AI code assistants in enterprises and points out the direction for further research.

Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise

Measuring the Impact of AI on Information Worker Productivity

Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward

Does Co-Development with AI Assistants Lead to More Maintainable Code? A Registered Report

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

Understanding the Usability of AI Programming Assistants

How far are AI-powered programming assistants from meeting developers' needs?

Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes

Do Users Write More Insecure Code with AI Assistants?

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Information Seeking Using AI Assistants

Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns

Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality

Not Merely Useful but Also Amusing: Impact of Perceived Usefulness and Perceived Enjoyment on the Adoption of AI-Powered Coding Assistant

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers

Exploring an AI-based writing Assistant's impact on English language learners

Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace

Envisioning the Next-Generation AI Coding Assistants: Insights & Proposals