Abstract:A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code , especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries , but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this article, we perform the first comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking, “At the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?” To facilitate the study, we first develop a plugin for the PyCharm IDE that implements a hybrid of code generation and code retrieval functionality, and we orchestrate virtual environments to enable collection of many user events (e.g., web browsing, keystrokes, fine-grained code edits). We ask developers with various backgrounds to complete 7 varieties of 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Further analysis identifies several pain points that could improve the effectiveness of future machine learning-based code generation/retrieval developer assistants and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies on this topic, as well as development of better code generation models.

Sorrel: an IDE Plugin for Managing Licenses and Detecting License Incompatibilities

JITO: a Tool for Just-in-time Defect Identification and Localization

A Large-Scale Empirical Study of Open Source License Usage: Practices and Challenges

LicenseRec: Knowledge Based Open Source License Recommendation for OSS Projects

A Study of Potential Code Borrowing and License Violations in Java Projects on GitHub

EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs

Your "notice" is Missing: Detecting and Fixing Violations of Modification Terms in Open Source Licenses During Forking

In-IDE Code Generation from Natural Language: Promise and Challenges

Detecting and Fixing Violations of Modification Terms in Open Source Licenses during Forking

Bridging Education and Development: IDEs as Interactive Learning Platforms

Catch the Butterfly: Peeking into the Terms and Conflicts among SPDX Licenses

"The Law Doesn't Work Like a Computer": Exploring Software Licensing Issues Faced by Legal Practitioners

Interactive support for secure programming education.

Interactive Static Software Performance Analysis in the IDE

JLLAR: A Logging Recommendation Plug-in Tool for Java

Parichayana: An Eclipse Plugin for Detecting Exception Handling Anti-Patterns and Code Smells in Java Programs

Understanding and Remediating Open-Source License Incompatibilities in the PyPI Ecosystem

I3DE: An IDE for Inspecting Inconsistencies in PL/SQL Code

Sourcerer: An infrastructure for large-scale collection and analysis of open-source code

Evaluating the Language-Based Security for Plugin Development

Predicting Licenses for Changed Source Code.