Abstract:To release developers from time-consuming software development, many approaches have been proposed to generate source code automatically according to software requirements. With significant advances in deep learning and natural language processing, deep learning-based approaches are proposed to generate source code from natural language descriptions. The key insight is that given a large corpus of software requirements and their corresponding implementations, advanced deep learning techniques may learn how to translate software requirements into source code that fulfill such requirements. Although such approaches are reported to be highly accurate, they are evaluated on datasets that are rather small, lack of diversity, and significantly different from real-world software requirements. To this end, we build a large scale dataset that is composed of longer requirements as well as validated implementations. We evaluate the state-of-the-art approaches on this new dataset, and the results suggest that their performance on our dataset is significantly lower than that on existing datasets concerning the common metrics, i.e., BLEU. Evaluation results also suggest that the generated programs often contain syntactic and semantical errors, and none of them can pass even a single predefined test case. Further analysis reveals that the state-of-the-art approaches learn little from software requirements, and most of the successfully generated statements are popular statements in the training programs. Based on this finding, we propose a popularity-based approach that always generates the most popular statements in training programs regardless of the input (software requirements). Evaluation results suggest that none of the state-of-the-art approaches can outperform this simple statistics-based approach. As a conclusion, deep learning-based program generation requires significant improvement in the future, and our dataset may serve as a basis for future research in this direction.

Deep Learning for Source Code Modeling and Generation

Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges

Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches

Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code.

Code Generation Based on Deep Learning: a Brief Review

Deep Learning for Code Generation: a Survey

Deep Learning Based Code Generation Methods: A Literature Review.

Deep Learning Based Code Generation Methods: Literature Review

A systematic mapping study of source code representation for deep learning in software engineering

Deep Transfer Learning for Source Code Modeling

DeepVS: An Efficient and Generic Approach for Source Code Modeling Usage

Deep Learning for Automated Code Generation: Challenges and Opportunities

Survey of Code Search Based on Deep Learning

From Code to Natural Language: Type-Aware Sketch-Based Seq2Seq Learning

The Source Code Comment Generation Based on Deep Reinforcement Learning and Hierarchical Attention

Deep Learning Based Code Generation from Requirements Text: Are We There Yet?

A Survey of Deep Learning Models for Structural Code Understanding

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

Generative Software Engineering

Learning Semantic Vector Representations of Source Code via a Siamese Neural Network

Deep Learning Based Program Generation from Requirements Text: Are We There Yet?