Abstract:With the prevalence of publicly available source code repositories to train deep neural network models, neural program models can do well in source code analysis tasks such as predicting method names in given programs that cannot be easily done by traditional program analysis techniques. Although such neural program models have been tested on various existing datasets, the extent to which they generalize to unforeseen source code is largely unknown. Since it is very challenging to test neural program models on all unforeseen programs, in this paper, we propose to evaluate the generalizability of neural program models with respect to semantic-preserving transformations: a generalizable neural program model should perform equally well on programs that are of the same semantics but of different lexical appearances and syntactical structures. We compare the results of various neural program models for the method name prediction task on programs before and after automated semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code, namely code2vec, code2seq, and GGNN, to build nine such neural program models for evaluation. Our results show that even with small semantically preserving changes to the programs, these neural program models often fail to generalize their performance. Our results also suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees. On the positive side, we observe that as the size of the training dataset grows and diversifies the generalizability of correct predictions produced by the neural program models can be improved too. Our results on the generalizability of neural program models provide insights to measure their limitations and provide a stepping stone for their improvement.

Benchmarking Neural Network Generalization for Grammar Induction

A Benchmark for Systematic Generalization in Grounded Language Understanding

Improving Systematic Generalization Through Modularity and Augmentation

On the generalization capacity of neural networks during generic multimodal reasoning

BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance

Generalization in Neural Networks: A Broad Survey

GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding

GLGE: A New General Language Generation Evaluation Benchmark

On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations

Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specifications

On Compositional Generalization of Neural Machine Translation

Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks

CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

Linguistic generalization and compositionality in modern artificial neural networks

Multilingual Grammar Induction with Continuous Language Identification.

Covered Forest: Fine-grained generalization analysis of graph neural networks

Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

Dependency Grammar Induction with Neural Lexicalization and Big Training Data

Understanding Data Augmentation in Neural Machine Translation: Two Perspectives Towards Generalization.

Benchmarking Large Language Model Capabilities for Conditional Generation

Generalizing and Hybridizing Count-based and Neural Language Models