Abstract:We present an approach to modifying Transformer architectures by integrating graph-aware relational reasoning into the attention mechanism, merging concepts from graph neural networks and language modeling. Building on the inherent connection between attention and graph theory, we reformulate the Transformer's attention mechanism as a graph operation and propose Graph-Aware Isomorphic Attention. This method leverages advanced graph modeling strategies, including Graph Isomorphism Networks (GIN) and Principal Neighborhood Aggregation (PNA), to enrich the representation of relational structures. Our approach captures complex dependencies and generalizes across tasks, as evidenced by a reduced generalization gap and improved learning performance. Additionally, we expand the concept of graph-aware attention to introduce Sparse GIN-Attention, a fine-tuning approach that employs sparse GINs. By interpreting attention matrices as sparse adjacency graphs, this technique enhances the adaptability of pre-trained foundational models with minimal computational overhead, endowing them with graph-aware capabilities. Sparse GIN-Attention fine-tuning achieves improved training dynamics and better generalization compared to alternative methods like low-rank adaption (LoRA). We discuss latent graph-like structures within traditional attention mechanisms, offering a new lens through which Transformers can be understood. By evolving Transformers as hierarchical GIN models for relational reasoning. This perspective suggests profound implications for foundational model development, enabling the design of architectures that dynamically adapt to both local and global dependencies. Applications in bioinformatics, materials science, language modeling, and beyond could benefit from this synthesis of relational and sequential data modeling, setting the stage for interpretable and generalizable modeling strategies.

Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers

KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning

Knowledge-Infused Self Attention Transformers

Adaptivity and Modularity for Efficient Generalization Over Task Complexity

Disentangling and Integrating Relational and Sensory Information in Transformer Architectures

Understanding Transformer Reasoning Capabilities via Graph Algorithms

A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task

Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization

Modularity in Transformers: Investigating Neuron Separability & Specialization

Generalized Probabilistic Attention Mechanism in Transformers

Transformers are Universal In-context Learners

Transformer Module Networks for Systematic Generalization in Visual Question Answering

Knowledge Circuits in Pretrained Transformers

Enhancing Transformers for Generalizable First-Order Logical Entailment

Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer

Beyond Intuition: Rethinking Token Attributions Inside Transformers

Transformers meet Neural Algorithmic Reasoners

Attention as a Hypernetwork