Abstract:Source code representation learning is the basis of applying artificial intelligence to many software engineering tasks such as code clone detection, algorithm classification, and code summarization. Recently, many works have tried to improve the performance of source code representation from various perspectives, e.g., introducing the structural information of programs into latent representation. However, when dealing with rapidly expanded unlabeled cross-language source code datasets from the Internet, there are still two issues. Firstly, deep learning models for many code-specific tasks still suffer from the lack of high-quality labels. Secondly, the structural differences among programming languages make it more difficult to process multiple languages in a single neural architecture. To address these issues, in this article, we propose a novel Cross -language Code representation with a large-scale pre-training ( XCode ) method. Concretely, we propose to use several abstract syntax trees and ELMo-enhanced variational autoencoders to obtain multiple pre-trained source code language models trained on about 1.5 million code snippets. To fully utilize the knowledge across programming languages, we further propose a Shared Encoder-Decoder (SED) architecture which uses the multi-teacher single-student method to transfer knowledge from the aforementioned pre-trained models to the distilled SED. The pre-trained models and SED will cooperate to better represent the source code. For evaluation, we examine our approach on three typical downstream cross-language tasks, i.e., source code translation, code clone detection, and code-to-code search, on a real-world dataset composed of programming exercises with multiple solutions. Experimental results demonstrate the effectiveness of our proposed approach on cross-language code representations. Meanwhile, our approach performs significantly better than several code representation baselines on different downstream tasks in terms of multiple automatic evaluation metrics.

CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model.

SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation

Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities

Graphcodebert: pre-training code represen-

What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code

TreeBERT: A Tree-Based Pre-Trained Model for Programming Language

GraphCodeBERT: Pre-training Code Representations with Data Flow

TransformCode: A Contrastive Learning Framework for Code Embedding Via Subtree Transformation

CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search

Syntax-Enhanced Pre-trained Model

ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning

CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking

CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search

Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation

Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models

Cross-Modal Contrastive Learning for Code Search

Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees.

XCode: Towards Cross-Language Code Representation with Large-Scale Pre-Training

Code Representation Pre-training with Complements from Program Executions

GrammarT5: Grammar-Integrated Pretrained Encoder-Decoder Neural Model for Code

CodeCSE: A Simple Multilingual Model for Code and Comment Sentence Embeddings