Abstract:In the field of software engineering, applying language models to the token sequence of source code is the state-of-art approach to build a code recommendation system. The syntax tree of source code has hierarchical structures. Ignoring the characteristics of tree structures decreases the model performance. Current LSTM model handles sequential data. The performance of LSTM model will decrease sharply if the noise unseen data is distributed everywhere in the test suite. As code has free naming conventions, it is common for a model trained on one project to encounter many unknown words on another project. If we set many unseen words as UNK just like the solution in natural language processing, the number of UNK will be much greater than the sum of the most frequently appeared words. In an extreme case, just predicting UNK at everywhere may achieve very high prediction accuracy. Thus, such solution cannot reflect the true performance of a model when encountering noise unseen data. In this paper, we only mark a small number of rare words as UNK and show the prediction performance of models under in-project and cross-project evaluation. We propose a novel Hierarchical Language Model (HLM) to improve the robustness of LSTM model to gain the capacity about dealing with the inconsistency of data distribution between training and testing. The newly proposed HLM takes the hierarchical structure of code tree into consideration to predict code. HLM uses BiLSTM to generate embedding for sub-trees according to hierarchies and collects the embedding of sub-trees in context to predict next code. The experiments on inner-project and cross-project data sets indicate that the newly proposed Hierarchical Language Model (HLM) performs better than the state-of-art LSTM model in dealing with the data inconsistency between training and testing and achieves averagely 11.2\% improvement in prediction accuracy.

Improve Language Modeling for Code Completion Through Learning General Token Repetition of Source Code with Optimized Memory

CCMC: Code Completion with a Memory Mechanism and a Copy Mechanism

Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation

A unified multi-task learning model for AST-level and token-level code completion

A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning.

Improve Language Modelling for Code Completion through Statement Level Language Model based on Statement Embedding Generated by BiLSTM

Repetition Neurons: How Do Language Models Produce Repetitions?

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Non-Autoregressive Line-Level Code Completion

Improving AST-Level Code Completion with Graph Retrieval and Multi-Field Attention

Improving the Robustness to Data Inconsistency between Training and Testing for Code Completion by Hierarchical Language Model

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective

Unveiling Memorization in Code Models

Sequence Model Design for Code Completion in the Modern IDE

Repetition Improves Language Model Embeddings

Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries

An Improved Recurrent Neural Network Language Model for Programming Language

Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation

A Hybrid Continuous-Time Dynamic Graph Representation Learning Model by Exploring Both Temporal and Repetitive Information

Bridge and Hint: Extending Pre-trained Language Models for Long-Range Code

Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models