Abstract:Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing tasks. Recently, several LLMs-based pipelines have been developed to enhance learning on graphs with text attributes, showcasing promising performance. However, graphs are well-known to be susceptible to adversarial attacks and it remains unclear whether LLMs exhibit robustness in learning on graphs. To address this gap, our work aims to explore the potential of LLMs in the context of adversarial attacks on graphs. Specifically, we investigate the robustness against graph structural and textual perturbations in terms of two dimensions: LLMs-as-Enhancers and LLMs-as-Predictors. Through extensive experiments, we find that, compared to shallow models, both LLMs-as-Enhancers and LLMs-as-Predictors offer superior robustness against structural and textual attacks.Based on these findings, we carried out additional analyses to investigate the underlying causes. Furthermore, we have made our benchmark library openly available to facilitate quick and fair evaluations, and to encourage ongoing innovative research in this field.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to evaluate the robustness of large - language models (LLMs) in graph - data learning, especially their performance when facing adversarial attacks. Specifically, the researchers are concerned with whether LLMs can maintain their performance when the graph structure and text attributes are perturbed. The following are the specific problems that the paper attempts to solve:
1. **Robustness under graph - structure attacks**:
- Evaluate the robustness of LLMs when the graph structure is perturbed. For example, an attacker may change the structure of the graph by adding, deleting, or reconnecting edges, thus affecting the model's prediction results.
- Study the performance of LLMs - as - Enhancers and LLMs - as - Predictors under different attack intensities to understand their advantages over traditional shallow models (such as Bag of Words, TF - IDF).
2. **Robustness under text - attribute attacks**:
- Evaluate the robustness of LLMs when the text attributes are perturbed. For example, an attacker may change the text content by replacing words or modifying sentence structures, thus misleading the model.
- Study the performance of LLMs when processing perturbed text inputs, especially compared with traditional text pre - processing techniques.
3. **Comparison of the robustness of different LLM architectures**:
- Compare the differences in the robustness of different types of LLMs (such as SBert, E5, LLaMA, etc.) when facing structural and text attacks.
- Explore the effects of pre - training and fine - tuning on the robustness of LLMs, especially whether fine - tuned LLMs are more robust than those not fine - tuned.
4. **Construct a reproducible benchmark library**:
- Develop a publicly available benchmark library so that researchers can quickly and fairly evaluate the performance of different graph - learning methods under adversarial attacks.
- Provide detailed experimental settings and evaluation metrics to ensure the reproducibility and fairness of the results.
Through these studies, the paper aims to fill the research gap in the existing literature regarding the robustness of graph - learning models under adversarial attacks and provide valuable references and tools for future research.
### Formula Representation
Some formulas and concepts involved in the paper can be presented in Markdown format, for example:
- Definition of a graph: $G=(V, E)$, where $V$ is the set of nodes and $E$ is the set of edges.
- Node feature matrix: $X\in\mathbb{R}^{N\times d}$, where $N$ is the number of nodes and $d$ is the dimension of node features.
- Adjacency matrix: $A\in\mathbb{R}^{N\times N}$.
- Davies - Bouldin Index (DBI): Used to evaluate the separation degree of clustering, and the formula is:
$$
DBI=\frac{1}{k}\sum_{i = 1}^{k}\max_{j\neq i}\left(\frac{\sigma_i+\sigma_j}{d(c_i, c_j)}\right)
$$
where $k$ is the number of clusters, $\sigma_i$ is the average distance of the $i$-th cluster, $c_i$ is the center of the $i$-th cluster, and $d(c_i, c_j)$ is the distance between the two cluster centers.
Through these formulas and concepts, the paper systematically evaluates the robustness of LLMs in graph learning and proposes new insights and improvement directions.