Abstract:Graph Neural Networks (GNNs) have emerged as the most powerful weapon for various graph tasks due to the message-passing mechanism's great local information aggregation ability. However, over-smoothing has always hindered GNNs from going deeper and capturing multi-hop neighbors. Unlike GNNs, Transformers can model global information and multi-hop interactions via multi-head self-attention and a proper Transformer structure can show more immunity to the over-smoothing problem. So, can we propose a novel framework to combine GNN and Transformer, integrating both GNN's local information aggregation and Transformer's global information modeling ability to eliminate the over-smoothing problem? To realize this, this paper proposes a collaborative learning scheme for GNN-Transformer and constructs GTC architecture. GTC leverages the GNN and Transformer branch to encode node information from different views respectively, and establishes contrastive learning tasks based on the encoded cross-view information to realize self-supervised heterogeneous graph representation. For the Transformer branch, we propose Metapath-aware Hop2Token and CG-Hetphormer, which can cooperate with GNN to attentively encode neighborhood information from different levels. As far as we know, this is the first attempt in the field of graph representation learning to utilize both GNN and Transformer to collaboratively capture different view information and conduct cross-view contrastive learning. The experiments on real datasets show that GTC exhibits superior performance compared with state-of-the-art methods. Codes can be available at

GenTC: Generative Transformer Via Contrastive Learning for Receipt Information Extraction

Contrastive Information Extraction with Generative Transformer

Contrastive Triple Extraction with Generative Transformer

Tag, Copy or Predict: A Unified Weakly-Supervised Learning Framework for Visual Information Extraction using Sequences

Small Sample Building Energy Consumption Prediction Using Contrastive Transformer Networks

CT-GAN: A conditional Generative Adversarial Network of transformer architecture for text-to-image

TC-OCR: TableCraft OCR for Efficient Detection & Recognition of Table Structure & Content

GenTron: Diffusion Transformers for Image and Video Generation

GTC: GNN-Transformer Co-contrastive Learning for Self-supervised Heterogeneous Graph Representation

A Transformer-Based Decoupled Attention Network for Text Recognition in Shopping Receipt Images

General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

CAT: Cross Attention in Vision Transformer

Optimizing Non-Autoregressive Transformers with Contrastive Learning

SRRNet: A Transformer Structure with Adaptive 2-D Spatial Attention Mechanism for Cell Phone-Captured Shopping Receipt Recognition

CLE-ViT: Contrastive Learning Encoded Transformer for Ultra-Fine-Grained Visual Categorization

GLaLT: Global-Local Attention-Augmented Light Transformer for Scene Text Recognition

Neighborhood Contrastive Transformer for Change Captioning

PCTrans: Position-Guided Transformer with Query Contrast for Biological Instance Segmentation

Genre: generative multi-turn question answering with contrastive learning for entity–relation extraction

Levenshtein OCR