Abstract:Transformers are now widely utilized in code intelligence tasks. To better fit highly structured source code, various structure information is passed into Transformer, such as positional encoding and abstract syntax tree (AST) based structures. However, it is still not clear how these structural features affect code intelligence tasks, such as code summarization. Addressing this problem is of vital importance for designing Transformer-based code models. Existing works are keen to introduce various structural information into Transformers while lacking persuasive analysis to reveal their contributions and interaction effects. In this paper, we conduct an empirical study of frequently-used code structure features for code representation, including two types of position encoding features and AST-based structure features. We propose a couple of probing tasks to detect how these structure features perform in Transformer and conduct comprehensive ablation studies to investigate how these structural features affect code semantic summarization tasks. To further validate the effectiveness of code structure features in code summarization tasks, we assess Transformer models equipped with these code structure features on a structural dependent summarization dataset. Our experimental results reveal several findings that may inspire future study: (1) there is a conflict between the influence of the absolute positional embeddings and relative positional embeddings in Transformer; (2) AST-based code structure features and relative position encoding features show a strong correlation and much contribution overlap for code semantic summarization tasks indeed exists between them; (3) Transformer models still have space for further improvement in explicitly understanding code structure information.

Text Structure Analysis Based on Improving Text Structure Relation Graph

Analysis and Implementation of Extraction Algorithm of Web Hierarchy Structure

Description and Browsing of Video Story Structure

Web Information Segmentation Method Based on DOM Structure Tree

Research on the Structure Recognition of Academic Texts Under Different Characteristics

A 3-Layer Structural Analysis Framework for Chinese Text Understanding

Website Link Structure Evaluation and Improvement Based on User Visiting Patterns.

Research on Automatic Summarization Based on Rules and Statistics for Chinese Texts

Text structure partition for automatic summarization

An In-depth Study on Internal Structure of Chinese Words

On the Structural Generalization in Text-to-SQL

Structure extraction from PDF-based book documents.

A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL

Hierarchical Logical Structure Extraction of Book Documents by Analyzing Tables of Contents

Better AMR-To-Text Generation with Graph Structure Reconstruction.

STRUCTBASE: a Structure Based Chinese Text Database System

Revisiting Structured Sentiment Analysis as Latent Dependency Graph Parsing

Enhancing Structure Modeling for Relation Extraction with Fine-Grained Gating and Co-Attention.

An Extensive Study of the Structure Features in Transformer-based Code Semantic Summarization.

A Two-Stage Long Text Summarization Method Based on Discourse Structure

An Automatic Examination Method for C Language Code Structure