Vanilla Graph Neural Networks

Zhiyuan Liu,Jie Zhou

DOI: https://doi.org/10.1007/978-3-031-01587-8_4

2020-01-01

Abstract:In this section, we describe the vanilla GNNs proposed in Scarselli et al. [2009]. We also list the limitations of the vanilla GNN in representation capability and training efficiency. After this chapter we will talk about several variants ofthe vanilla GNN model.

What problem does this paper attempt to address?

Variants for Different Graph Types

Zhiyuan Liu,Jie Zhou

DOI: https://doi.org/10.1007/978-3-031-01587-8_9

2020-01-01

Synthesis Lectures on Artificial Intelligence and Machine Learning

Abstract:The original GNN [Scarselli et al., 2009] works on the graphs that consist of nodes with label information and undirected edges, which is the simplest graph format. However, there are many variants of graphs in the world and modeling different graph types requires different GNN structures. In this chapter, we investigate graph neural networks designed for specific graph types.
The graph neural network model

Franco Scarselli,Marco Gori,Ah Chung Tsoi,Markus Hagenbuchner,Gabriele Monfardini

DOI: https://doi.org/10.1109/TNN.2008.2005605

Abstract:Many underlying relationships among data in several areas of science and engineering, e.g., computer vision, molecular chemistry, molecular biology, pattern recognition, and data mining, can be represented in terms of graphs. In this paper, we propose a new neural network model, called graph neural network (GNN) model, that extends existing neural network methods for processing the data represented in graph domains. This GNN model, which can directly process most of the practically useful types of graphs, e.g., acyclic, cyclic, directed, and undirected, implements a function tau(G,n) is an element of IR(m) that maps a graph G and one of its nodes n into an m-dimensional Euclidean space. A supervised learning algorithm is derived to estimate the parameters of the proposed GNN model. The computational cost of the proposed algorithm is also considered. Some experimental results are shown to validate the proposed learning algorithm, and to demonstrate its generalization capabilities.
A Review on the Applications of Graph Neural Networks in Materials Science at the Atomic Scale

Xingyue Shi,Linming Zhou,Yuhui Huang,Yongjun Wu,Zijian Hong

DOI: https://doi.org/10.1002/mgea.50

2024-01-01

Abstract:Abstract In recent years, interdisciplinary research has become increasingly popular within the scientific community. The fields of materials science and chemistry have also gradually begun to apply the machine learning technology developed by scientists from computer science. Graph neural networks (GNNs) are new machine learning models with powerful feature extraction, relationship inference, and compositional generalization capabilities. These advantages drive researchers to design computational models to accelerate material property prediction and new materials design, dramatically reducing the cost of traditional experimental methods. This review focuses on the principles and applications of the GNNs. The basic concepts and advantages of the GNNs are first introduced and compared to the traditional machine learning and neural networks. Then, the principles and highlights of seven classic GNN models, namely crystal graph convolutional neural networks, iCGCNN, Orbital Graph Convolutional Neural Network, MatErials Graph Network, Global Attention mechanism with Graph Neural Network, Atomistic Line Graph Neural Network, and BonDNet are discussed. Their connections and differences are also summarized. Finally, insights and prospects are provided for the rapid development of GNNs in materials science at the atomic scale.
Self-supervised Graph Representations with Generative Adversarial Learning

Xuecheng Sun,Zonghui Wang,Zheming Lu,Ziqian Lu

DOI: https://doi.org/10.1016/j.neucom.2024.127786

IF: 6

2024-01-01

Neurocomputing

Abstract:Graph Neural Networks (GNNs) are powerful neural models for representation learning on graphs. In this work we study a self-supervised learning GNN method GraphSSGAN (Graph Self-Supervised Generative Adversarial Network) that learns generalized node representations using only unlabeled data. The core idea lies in learning representations that capture both the local information of the links and the global information of the plausibility of the subgraph samples. Specifically, node features of the original graph are first encoded into latent representations by optimizing a link prediction objective. Then the generator converts noise vectors to node representations in the latent space and predicts the link probabilities from the generated node representations. The discriminator leverages the graph convolutional network (GCN) architecture to produce permutation-invariant graph-level embeddings, and the intermediate node representations are used by simple classifiers in the downstream tasks. In addition, we introduce several technical tricks including Gumbel-Top-k trick, Gumbel-Softmax trick and mini-batch training via subgraph sampling to improve the training process. Through extensive experiments on node classification and link prediction tasks, we demonstrate the effectiveness of the proposed model and the contribution of GAN framework on graph representation learning.
Feature Expansion for Graph Neural Networks

Jiaqi Sun,Lin Zhang,Guangyi Chen,Peng XU,Kun Zhang,Yujiu Yang

2023-01-01

Abstract:Graph neural networks aim to learn representations for graph-structured data and show impressive performance in node classification. Recently, many methods have studied the representations of GNNs from the perspective of optimization goals and spectral graph theory. However, the feature space that dominates representation learning has not been systematically studied in graph neural networks. In this paper, we propose to fill this gap by analyzing the feature space of both spatial and spectral models. We decompose graph neural networks into determined feature spaces and trainable weights, providing the convenience of studying the feature space explicitly using matrix space analysis. In particular, we find theoretically that the feature space tends to be linearly correlated due to repeated aggregations. In this case, the feature space is bounded by the poor representation of shared weights or the limited dimensionality of node attributes in existing models, leading to poor performance. Motivated by these findings, we propose 1) feature subspaces flattening and 2) structural principal components to expand the feature space. Extensive experiments verify the effectiveness of our proposed more comprehensive feature space, with comparable inference time to the baseline, and demonstrate its efficient convergence capability.
Graph Neural Networks Do Not Always Oversmooth

Bastian Epping,Alexandre René,Moritz Helias,Michael T. Schaub

2024-06-04

Abstract:Graph neural networks (GNNs) have emerged as powerful tools for processing relational data in applications. However, GNNs suffer from the problem of oversmoothing, the property that the features of all nodes exponentially converge to the same vector over layers, prohibiting the design of deep GNNs. In this work we study oversmoothing in graph convolutional networks (GCNs) by using their Gaussian process (GP) equivalence in the limit of infinitely many hidden features. By generalizing methods from conventional deep neural networks (DNNs), we can describe the distribution of features at the output layer of deep GCNs in terms of a GP: as expected, we find that typical parameter choices from the literature lead to oversmoothing. The theory, however, allows us to identify a new, nonoversmoothing phase: if the initial weights of the network have sufficiently large variance, GCNs do not oversmooth, and node features remain informative even at large depth. We demonstrate the validity of this prediction in finite-size GCNs by training a linear classifier on their output. Moreover, using the linearization of the GCN GP, we generalize the concept of propagation depth of information from DNNs to GCNs. This propagation depth diverges at the transition between the oversmoothing and non-oversmoothing phase. We test the predictions of our approach and find good agreement with finite-size GCNs. Initializing GCNs near the transition to the non-oversmoothing phase, we obtain networks which are both deep and expressive.

Machine Learning,Disordered Systems and Neural Networks
VanillaNet: the Power of Minimalism in Deep Learning

Hanting Chen,Yunhe Wang,Jianyuan Guo,Dacheng Tao

2023-05-23

Abstract:At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at <a class="link-external link-https" href="https://github.com/huawei-noah/VanillaNet" rel="external noopener nofollow">this https URL</a> and <a class="link-external link-https" href="https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
The Intelligible and Effective Graph Neural Additive Networks

Maya Bechler-Speicher,Amir Globerson,Ran Gilad-Bachrach

2024-06-28

Abstract:Graph Neural Networks (GNNs) have emerged as the predominant approach for learning over graph-structured data. However, most GNNs operate as black-box models and require post-hoc explanations, which may not suffice in high-stakes scenarios where transparency is crucial. In this paper, we present a GNN that is interpretable by design. Our model, Graph Neural Additive Network (GNAN), is a novel extension of the interpretable class of Generalized Additive Models, and can be visualized and fully understood by humans. GNAN is designed to be fully interpretable, allowing both global and local explanations at the feature and graph levels through direct visualization of the model. These visualizations describe the exact way the model uses the relationships between the target variable, the features, and the graph. We demonstrate the intelligibility of GNANs in a series of examples on different tasks and datasets. In addition, we show that the accuracy of GNAN is on par with black-box GNNs, making it suitable for critical applications where transparency is essential, alongside high accuracy.

Machine Learning,Artificial Intelligence
Graph Neural Machine: A New Model for Learning with Tabular Data

Giannis Nikolentzos,Siyun Wang,Johannes Lutzeyer,Michalis Vazirgiannis

2024-02-05

Abstract:In recent years, there has been a growing interest in mapping data from different domains to graph structures. Among others, neural network models such as the multi-layer perceptron (MLP) can be modeled as graphs. In fact, MLPs can be represented as directed acyclic graphs. Graph neural networks (GNNs) have recently become the standard tool for performing machine learning tasks on graphs. In this work, we show that an MLP is equivalent to an asynchronous message passing GNN model which operates on the MLP's graph representation. We then propose a new machine learning model for tabular data, the so-called Graph Neural Machine (GNM), which replaces the MLP's directed acyclic graph with a nearly complete graph and which employs a synchronous message passing scheme. We show that a single GNM model can simulate multiple MLP models. We evaluate the proposed model in several classification and regression datasets. In most cases, the GNM model outperforms the MLP architecture.

Machine Learning
Graph Information Vanishing Phenomenon in Implicit Graph Neural Networks

Silu He,Jun Cao,Hongyuan Yuan,Zhe Chen,Shijuan Gao,Haifeng Li

DOI: https://doi.org/10.3390/math12172659

IF: 2.4

2024-08-29

Mathematics

Abstract:Graph neural networks (GNNs) have been highly successful in graph representation learning. The goal of GNNs is to enrich node representations by aggregating information from neighboring nodes. Much work has attempted to improve the quality of aggregation by introducing a variety of graph information with representational capabilities. The class of GNNs that improves the quality of aggregation by encoding graph information with representational capabilities into the weights of neighboring nodes through different learnable transformation structures (LTSs) are referred to as implicit GNNs. However, we argue that LTSs only transform graph information into the weights of neighboring nodes in the direction that minimizes the loss function during the learning process and does not actually utilize the effective properties of graph information, a phenomenon that we refer to as graph information vanishing (GIV). To validate this point, we perform thousands of experiments on seven node classification benchmark datasets. We first replace the graph information utilized by five implicit GNNs with random values and surprisingly observe that the variation range of accuracies is less than ± 0.3%. Then, we quantitatively characterize the similarity of the weights generated from graph information and random values by cosine similarity, and the cosine similarities are greater than 0.99. The empirical experiments show that graph information is equivalent to initializing the input of LTSs. We believe that graph information as an additional supervised signal to constrain the training of GNNs can effectively solve GIV. Here, we propose GinfoNN, which utilizes both labels and discrete graph curvature as supervised signals to jointly constrain the training of the model. The experimental results show that the classification accuracies of GinfoNN improve by two percentage points over baselines on large and dense datasets.

mathematics
Graph Neural Network (GNN) in Image and Video Understanding Using Deep Learning for Computer Vision Applications

P Pradhyumna,G P Shreya,Mohana

DOI: https://doi.org/10.1109/icesc51422.2021.9532631

2021-08-04

Abstract:Graph neural networks (GNNs) is an information - processing system that uses message passing among graph nodes. In recent years, GNN variants including graph attention network (GAT), graph convolutional network (GCN), and graph recurrent network (GRN) have shown revolutionary performance in computer vision applications using deep learning and artificial intelligence. These neural network model extensions, collect information in the form of graphs. GNN may be divided into three groups based on the challenges it solves: link prediction, node classification, graph classification. Machines can differentiate and recognise objects in image and video using standard CNNs. Extensive amount of research work needs to be done before robots can have same visual intuition as humans. GNN architectures, on the other hand, may be used to solve various image categorization and video challenges. The number of GNN applications in computer vision not limited, continues to expand. Human-object interaction, actin understanding, image categorization from a few shots and many more. In this paper use of GNN in image and video understanding, design aspects, architecture, applications and implementation challenges towards computer vision is described. GNN is a strong tool for analysing graph data and is still a relatively active area that needs further researches attention to solve many computer vision applications.
Total Variation Graph Neural Networks

Jonas Berg Hansen,Filippo Maria Bianchi

DOI: https://doi.org/10.48550/arXiv.2211.06218

2023-04-28

Abstract:Recently proposed Graph Neural Networks (GNNs) for vertex clustering are trained with an unsupervised minimum cut objective, approximated by a Spectral Clustering (SC) relaxation. However, the SC relaxation is loose and, while it offers a closed-form solution, it also yields overly smooth cluster assignments that poorly separate the vertices. In this paper, we propose a GNN model that computes cluster assignments by optimizing a tighter relaxation of the minimum cut based on graph total variation (GTV). The cluster assignments can be used directly to perform vertex clustering or to implement graph pooling in a graph classification framework. Our model consists of two core components: i) a message-passing layer that minimizes the $\ell_1$ distance in the features of adjacent vertices, which is key to achieving sharp transitions between clusters; ii) an unsupervised loss function that minimizes the GTV of the cluster assignments while ensuring balanced partitions. Experimental results show that our model outperforms other GNNs for vertex clustering and graph classification.

Machine Learning
Architectural Implications of Graph Neural Networks.

Zhihui Zhang,Jingwen Leng,Lingxiao Ma,Youshan Miao,Chao Li,Minyi Guo

DOI: https://doi.org/10.1109/lca.2020.2988991

IF: 2.3

2020-01-01

IEEE Computer Architecture Letters

Abstract:Graph neural networks (GNN) represent an emerging line of deep learning models that operate on graph structures. It is becoming more and more popular due to its high accuracy achieved in many graph-related tasks. However, GNN is not as well understood in the system and architecture community as its counterparts such as multi-layer perceptrons and convolutional neural networks. This letter tries to introduce the GNN to our community. In contrast to prior work that only presents characterizations of GCNs, our work covers a large portion of the varieties for GNN workloads based on a general GNN description framework. By constructing the models on top of two widely-used libraries, we characterize the GNN computation at inference stage concerning general-purpose and application-specific architectures and hope our work can foster more system and architecture research for GNNs.
Global Minima, Recoverability Thresholds, and Higher-Order Structure in GNNS

Drake Brown,Trevor Garrity,Kaden Parker,Jason Oliphant,Stone Carson,Cole Hanson,Zachary Boyd

DOI: https://doi.org/10.48550/arXiv.2310.07667

2023-10-12

Abstract:We analyze the performance of graph neural network (GNN) architectures from the perspective of random graph theory. Our approach promises to complement existing lenses on GNN analysis, such as combinatorial expressive power and worst-case adversarial analysis, by connecting the performance of GNNs to typical-case properties of the training data. First, we theoretically characterize the nodewise accuracy of one- and two-layer GCNs relative to the contextual stochastic block model (cSBM) and related models. We additionally prove that GCNs cannot beat linear models under certain circumstances. Second, we numerically map the recoverability thresholds, in terms of accuracy, of four diverse GNN architectures (GCN, GAT, SAGE, and Graph Transformer) under a variety of assumptions about the data. Sample results of this second analysis include: heavy-tailed degree distributions enhance GNN performance, GNNs can work well on strongly heterophilous graphs, and SAGE and Graph Transformer can perform well on arbitrarily noisy edge data, but no architecture handled sufficiently noisy feature data well. Finally, we show how both specific higher-order structures in synthetic data and the mix of empirical structures in real data have dramatic effects (usually negative) on GNN performance.

Machine Learning,Artificial Intelligence
How Interpretable Are Interpretable Graph Neural Networks?

Yongqiang Chen,Yatao Bian,Bo Han,James Cheng

2024-06-12

Abstract:Interpretable graph neural networks (XGNNs ) are widely adopted in various scientific applications involving graph-structured data. Existing XGNNs predominantly adopt the attention-based mechanism to learn edge or node importance for extracting and making predictions with the interpretable subgraph. However, the representational properties and limitations of these methods remain inadequately explored. In this work, we present a theoretical framework that formulates interpretable subgraph learning with the multilinear extension of the subgraph distribution, coined as subgraph multilinear extension (SubMT). Extracting the desired interpretable subgraph requires an accurate approximation of SubMT, yet we find that the existing XGNNs can have a huge gap in fitting SubMT. Consequently, the SubMT approximation failure will lead to the degenerated interpretability of the extracted subgraphs. To mitigate the issue, we design a new XGNN architecture called Graph Multilinear neT (GMT), which is provably more powerful in approximating SubMT. We empirically validate our theoretical findings on a number of graph classification benchmarks. The results demonstrate that GMT outperforms the state-of-the-art up to 10% in terms of both interpretability and generalizability across 12 regular and geometric graph benchmarks.

Machine Learning
Understanding Virtual Nodes: Oversmoothing, Oversquashing, and Node Heterogeneity

Joshua Southern,Francesco Di Giovanni,Michael Bronstein,Johannes F. Lutzeyer

2024-05-22

Abstract:Message passing neural networks (MPNNs) have been shown to have limitations in terms of expressivity and modeling long-range interactions. Augmenting MPNNs with a virtual node (VN) removes the locality constraint of the layer aggregation and has been found to improve performance on a range of benchmarks. We provide a comprehensive theoretical analysis of the role of VNs and benefits thereof, through the lenses of oversmoothing, oversquashing, and sensitivity analysis. First, in contrast to prior belief, we find that VNs typically avoid replicating anti-smoothing approaches to maintain expressive power. Second, we characterize, precisely, how the improvement afforded by VNs on the mixing abilities of the network and hence in mitigating oversquashing, depends on the underlying topology. Finally, we highlight that, unlike Graph-Transformers (GT), classical instantiations of the VN are often constrained to assign uniform importance to different nodes. Consequently, we propose a variant of VN with the same computational complexity, which can have different sensitivity to nodes based on the graph structure. We show that this is an extremely effective and computationally efficient baseline on graph-level tasks.

Machine Learning
Revisiting graph neural networks from hybrid regularized graph signal reconstruction

Jiaxing Miao,Feilong Cao,Hailiang Ye,Ming Li,Bing Yang

DOI: https://doi.org/10.1016/j.neunet.2022.11.003

IF: 7.8

2022-11-13

Neural Networks

Abstract:Graph neural networks (GNNs) have shown strong graph-structured data processing capabilities. However, most of them are generated based on the message-passing mechanism and lack of the systematic approach to guide their developments. Meanwhile, a unified point of view is hard to explain the design concepts of different GNN models. This paper presents a unified optimization framework from hybrid regularized graph signal reconstruction to establish the connection between the aggregation operations of different GNNs, showing that exploring the optimal solution is the process of GNN information aggregation. We use this new framework to mathematically explain several classic GNN models and summarizes their commonalities and differences from a macro perspective. The proposed framework not only provides convenience to understand GNNs, but also has a guiding significance for the proposal of new GNNs. Moreover, we design a model-driven fixed-point iteration method and a data-driven dictionary learning network according to the corresponding optimization objective and sparse representation. Then the new model, GNN based on model-driven and data-driven (GNN-MD), is established by using alternating iteration methods. We also theoretically analyze its convergence. Numerous node classification experiments on multiple datasets illustrate that the proposed GNN-MD has excellent performance and outperforms all baselines on high-feature-dimension datasets.

computer science, artificial intelligence,neurosciences
Graph Neural Networks Inspired by Classical Iterative Algorithms

Yongyi Yang,Tang Liu,Yangkun Wang,Jinjing Zhou,Quan Gan,Zhewei Wei,Zheng Zhang,Zengfeng Huang,David Wipf

DOI: https://doi.org/10.48550/arxiv.2103.06064

2021-01-01

Abstract:Despite the recent success of graph neural networks (GNN), common architectures often exhibit significant limitations, including sensitivity to oversmoothing, long-range dependencies, and spurious edges, e.g., as can occur as a result of graph heterophily or adversarial attacks. To at least partially address these issues within a simple transparent framework, we consider a new family of GNN layers designed to mimic and integrate the update rules of two classical iterative algorithms, namely, proximal gradient descent and iterative reweighted least squares (IRLS). The former defines an extensible base GNN architecture that is immune to oversmoothing while nonetheless capturing long-range dependencies by allowing arbitrary propagation steps. In contrast, the latter produces a novel attention mechanism that is explicitly anchored to an underlying end-to-end energy function, contributing stability with respect to edge uncertainty. When combined we obtain an extremely simple yet robust model that we evaluate across disparate scenarios including standardized benchmarks, adversarially-perturbated graphs, graphs with heterophily, and graphs involving long-range dependencies. In doing so, we compare against SOTA GNN approaches that have been explicitly designed for the respective task, achieving competitive or superior node classification accuracy. Our code is available at this link. And for an extended version of this work, please see (Yang et al., 2021).

Vanilla Graph Neural Networks

Variants for Different Graph Types

The graph neural network model

A Review on the Applications of Graph Neural Networks in Materials Science at the Atomic Scale

Self-supervised Graph Representations with Generative Adversarial Learning

Feature Expansion for Graph Neural Networks

Graph Neural Networks Do Not Always Oversmooth

VanillaNet: the Power of Minimalism in Deep Learning

The Intelligible and Effective Graph Neural Additive Networks

Graph Neural Machine: A New Model for Learning with Tabular Data

Graph Information Vanishing Phenomenon in Implicit Graph Neural Networks

Graph Neural Network (GNN) in Image and Video Understanding Using Deep Learning for Computer Vision Applications

Total Variation Graph Neural Networks

Architectural Implications of Graph Neural Networks.

Global Minima, Recoverability Thresholds, and Higher-Order Structure in GNNS

How Interpretable Are Interpretable Graph Neural Networks?

Understanding Virtual Nodes: Oversmoothing, Oversquashing, and Node Heterogeneity

Revisiting graph neural networks from hybrid regularized graph signal reconstruction

Graph Neural Networks Inspired by Classical Iterative Algorithms