Abstract:Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge of adapting GCNs on large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to the uncontrollable neighborhood expansion across layers. In this paper, we accelerate the training of GCNs through developing an adaptive layer-wise sampling method. By constructing the network layer by layer in a top-down passway, we sample the lower layer conditioned on the top one, where the sampled neighborhoods are shared by different parent nodes and the over expansion is avoided owing to the fixed-size sampling. More importantly, the proposed sampler is adaptive and applicable for explicit variance reduction, which in turn enhances the training of our method. Furthermore, we propose a novel and economical approach to promote the message passing over distant nodes by applying skip connections. Intensive experiments on several benchmarks verify the effectiveness of our method regarding the classification accuracy while enjoying faster convergence speed.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the scalability issue of Graph Convolutional Networks (GCNs) on large - scale graph data. Specifically, current GCNs face huge challenges in terms of computation and memory consumption when processing large - scale graph data, because they need to recursively expand the neighbor nodes of each layer, which leads to an exponential growth in the amount of computation and a large amount of memory occupation. In particular, when the graph structure is dense or follows a power - law distribution, even for a single node, its neighbor nodes will quickly cover most of the graph area, making the traditional mini - batch training method unable to effectively accelerate the convolution calculation. To solve this problem, the author proposes a new method based on adaptive layer sampling to accelerate the training of GCNs. By constructing the network layer by layer and conditionally sampling the next layer based on the top layer, this method can avoid the problem of over - expansion, and at the same time control the number of nodes in each layer through fixed - size sampling. In addition, the proposed sampler is adaptive and can explicitly reduce the variance during the training process, thereby improving the training effect of the model. To further promote the information transfer between distant nodes, the author also introduces a new mechanism - skip connection. By adding a short connection between the (l + 1) - th layer and the (l - 1) - th layer, the nodes of the (l - 1) - th layer are reused as the 2 - hop neighbors of the (l + 1) - th layer, thus naturally maintaining the second - order approximation without additional computational overhead. In general, the main contributions of this paper are as follows: 1. Developed a novel layer - sampling method to accelerate the training of GCN models, in which information is shared between layers and the number of sampled nodes is controllable. 2. The design of the sampler is adaptive, and the form of the sampler is determined by explicitly reducing the variance. 3. Proposed a simple and effective strategy to maintain the second - order approximation by establishing skip connections between two layers. These contributions not only improve the training efficiency of the model on large - scale graph data, but also perform well in classification accuracy.

Adaptive Sampling Towards Fast Graph Representation Learning

Adaptive sampling towards fast graph representation learning

Accurate, Efficient and Scalable Graph Embedding

A Subgraph Sampling Method for Training Large-Scale Graph Convolutional Network.

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

A learnable sampling method for scalable graph neural networks

Resource-Efficient Training for Large Graph Convolutional Networks with Label-Centric Cumulative Sampling

AdaptiveGCN: Efficient GCN Through Adaptively Sparsifying Graphs

Non-recursive graph convolutional networks

MG-GCN: Fast and Effective Learning with Mix-grained Aggregators for Training Large Graph Convolutional Networks

Graph Convolutional Networks with Adaptive Neighborhood Awareness

Learning by Sampling and Compressing: Efficient Graph Representation Learning with Extremely Limited Annotations

Sampling methods for efficient training of graph convolutional networks: A survey

FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale

SGCN: A Scalable Graph Convolutional Network with Graph-Shaped Kernels and Multi-Channels

Scalable Graph Convolutional Networks with Fast Localized Spectral Filter for Directed Graphs

Efficient Data Loader for Fast Sampling-Based GNN Training on Large Graphs.

Learning Discrete Adaptive Receptive Fields for Graph Convolutional Networks

L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks

Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture

Adaptive Sampling Toward a Dynamic Graph Convolutional Network for Hyperspectral Image Classification