Abstract:Programming languages possess rich semantic information such as data flow that is represented by graphs and not available from the surface form of source code. Recent code language models have scaled to billions of parameters, but model source code solely as text tokens while ignoring any other structural information. Conversely, models that do encode structural information of code make modifications to the Transformer architecture, limiting their scale and compatibility with pretrained LLMs. In this work, we take the best of both worlds with GALLa - Graph Aligned Large Language Model. GALLa utilizes graph neural networks and cross-modal alignment technologies to inject the structural information of code into LLMs as an auxiliary task during finetuning. This framework is both model-agnostic and task-agnostic, as it can be applied to any code LLM for any code downstream task, and requires the structural graph data only at training time from a corpus unrelated to the finetuning data, while incurring no cost at inference time over the baseline LLM. Experiments on five code tasks with four different baseline LLMs ranging in size from 350M to 8B validate the effectiveness of GALLa, demonstrating consistent improvement over the baseline, even for powerful models such as LLaMA3.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to integrate the rich semantic information of programming languages (such as data flow, control flow, etc.) into large - language models (LLMs) to improve their understanding and generation ability of source code. Specifically: 1. **Limitations of Existing Methods**: - Existing code language models mainly regard source code as plain - text tokens, ignoring the structural information of the code. - Some methods that attempt to encode code structural information modify the Transformer architecture, which limits their scale and compatibility with pre - trained LLMs. 2. **Innovations of GALLa**: - GALLa (Graph Aligned Large Language Models) injects the structural information of code into LLMs during the fine - tuning process by combining graph neural networks (GNN) and cross - modal alignment techniques. - This method is both model - agnostic and task - agnostic, can be applied to any code LLM, and only requires structural graph data during training and does not incur additional costs during inference. 3. **Solutions to Specific Problems**: - **Code Understanding**: By introducing graph - structural information, LLMs can better understand complex semantic relationships in the code, such as function calls, data flows, etc. - **Code Generation**: When generating code, LLMs can generate code that is more in line with logical and grammatical norms according to the graph structure. - **Multilingual Generalization**: Even if the alignment data only contains Python and Java programs, GALLa can still improve the performance of tasks in other programming languages, demonstrating cross - language knowledge transfer ability. In summary, this paper aims to enhance the understanding and generation ability of LLMs for source code by introducing graph - structural information, thereby overcoming the limitations of existing methods and achieving significant improvements in multiple code understanding and generation tasks.

GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding

Enhance Graph Alignment for Large Language Models

LLaGA: Large Language and Graph Assistant

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

Large Language Models as Code Executors: An Exploratory Study

Large Language Models on Graphs: A Comprehensive Survey

Language Models are Graph Learners

Improving Natural Language Capability of Code Large Language Model

LLaMo: Large Language Model-based Molecular Graph Assistant

Language is All a Graph Needs

LLaSA: Large Language and Structured Data Assistant

L2G: Repurposing Language Models for Genomics Tasks

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

GCoder: Improving Large Language Model for Generalized Graph Problem Solving

All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks

MaLA-500: Massive Language Adaptation of Large Language Models