Abstract:To release developers from time-consuming software development, many approaches have been proposed to generate source code automatically according to software requirements. With significant advances in deep learning and natural language processing, deep learning-based approaches are proposed to generate source code from natural language descriptions. The key insight is that given a large corpus of software requirements and their corresponding implementations, advanced deep learning techniques may learn how to translate software requirements into source code that fulfill such requirements. Although such approaches are reported to be highly accurate, they are evaluated on datasets that are rather small, lack of diversity, and significantly different from real-world software requirements. To this end, we build a large scale dataset that is composed of longer requirements as well as validated implementations. We evaluate the state-of-the-art approaches on this new dataset, and the results suggest that their performance on our dataset is significantly lower than that on existing datasets concerning the common metrics, i.e., BLEU. Evaluation results also suggest that the generated programs often contain syntactic and semantical errors, and none of them can pass even a single predefined test case. Further analysis reveals that the state-of-the-art approaches learn little from software requirements, and most of the successfully generated statements are popular statements in the training programs. Based on this finding, we propose a popularity-based approach that always generates the most popular statements in training programs regardless of the input (software requirements). Evaluation results suggest that none of the state-of-the-art approaches can outperform this simple statistics-based approach. As a conclusion, deep learning-based program generation requires significant improvement in the future, and our dataset may serve as a basis for future research in this direction.

On End-to-End Program Generation from User Intention by Deep Neural Networks

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Deep Reinforcement Learning for Dialogue Generation

Deep Learning Based Program Generation from Requirements Text: Are We There Yet?

A Neural Network Architecture for Program Understanding Inspired by Human Behaviors

Real-time interactive sequence generation and control with Recurrent Neural Network ensembles

Deep Learning Based Code Generation from Requirements Text: Are We There Yet?

Human-Neural Net Collaborative Programming.

Program synthesis from natural language using recurrent neural networks

Towards Automatic Generation of Product Reviews from Aspect-Sentiment Scores

Deep Learning for Automated Code Generation: Challenges and Opportunities

Drawing and Recognizing Chinese Characters with Recurrent Neural Network

Modeling question asking using neural program generation

Neural Programming by Example.

The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks

Neural Text Generation: Past, Present and Beyond

Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches

Deep Learning Based Code Completion Models for Programming Codes.

RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text

Automated Item Generation with Recurrent Neural Networks

Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models