Abstract:A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code , especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries , but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this article, we perform the first comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking, “At the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?” To facilitate the study, we first develop a plugin for the PyCharm IDE that implements a hybrid of code generation and code retrieval functionality, and we orchestrate virtual environments to enable collection of many user events (e.g., web browsing, keystrokes, fine-grained code edits). We ask developers with various backgrounds to complete 7 varieties of 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Further analysis identifies several pain points that could improve the effectiveness of future machine learning-based code generation/retrieval developer assistants and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies on this topic, as well as development of better code generation models.

Hidden Gems in the Rough: Computational Notebooks as an Uncharted Oasis for IDEs

What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities

Assessing the Quality of Computational Notebooks for a Frictionless Transition from Exploration to Production

WHATSNEXT: Guidance-enriched Exploratory Data Analysis with Interactive, Low-Code Notebooks

StickyLand: Breaking the Linear Presentation of Computational Notebooks

Promoting Computational Thinking Skills in Non-Computer-Science Students: Gamifying Computational Notebooks to Increase Student Engagement

Facilitating Mixed-Methods Analysis with Computational Notebooks

From Computational to Conversational Notebooks

In-IDE Code Generation from Natural Language: Promise and Challenges

NoteG: A Computational Notebook to Facilitate Rapid Game Prototyping

GIS and Computational Notebooks

A Study of Editor Features in a Creative Coding Classroom

Notably Inaccessible -- Data Driven Understanding of Data Science Notebook (In)Accessibility

JetTrain: IDE-Native Machine Learning Experiments

Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks

Design Concerns for Integrated Scripting and Interactive Visualization in Notebook Environments

Better Code, Better Sharing: On the Need of Analyzing Jupyter Notebooks

A New Generation of Intelligent Development Environments

Notable: On-the-fly Assistant for Data Storytelling in Computational Notebooks

Notable: On-the-fly assistant for data storytelling in computational notebooks