Abstract:Atomic structure prediction and associated property calculations are the bedrock of chemical physics. Since high-fidelity ab initio modeling techniques for computing the structure and properties can be prohibitively expensive, this motivates the development of machine-learning (ML) models that make these predictions more efficiently. Training graph neural networks over large atomistic databases introduces unique computational challenges, such as the need to process millions of small graphs with variable size and support communication patterns that are distinct from learning over large graphs, such as social networks. We demonstrate a novel hardware-software codesign approach to scale up the training of atomistic graph neural networks (GNN) for structure and property prediction. First, to eliminate redundant computation and memory associated with alternative padding techniques and to improve throughput via minimizing communication, we formulate the effective coalescing of the batches of variable-size atomistic graphs as the bin packing problem and introduce a hardware-agnostic algorithm to pack these batches. In addition, we propose hardware-specific optimizations, including a planner and vectorization for the gather-scatter operations targeted for Graphcore's Intelligence Processing Unit (IPU), as well as model-specific optimizations such as merged communication collectives and optimized softplus. Putting these all together, we demonstrate the effectiveness of the proposed codesign approach by providing an implementation of a well-established atomistic GNN on the Graphcore IPUs. We evaluate the training performance on multiple atomistic graph databases with varying degrees of graph counts, sizes, and sparsity. We demonstrate that such a codesign approach can reduce the training time of atomistic GNNs and can improve their performance by up to 1.5× compared to the baseline implementation of the model on the IPUs. Additionally, we compare our IPU implementation with a Nvidia GPU-based implementation and show that our atomistic GNN implementation on the IPUs can run 1.8× faster on average compared to the execution time on the GPUs.

Comparison of Graphcore IPUs and Nvidia GPUsfor cosmology applications

Graphcore C2 Card performance for image-based deep learning application: A Report

The Implementation of the Three-Dimensional Unified Gas-Kinetic Wave-Particle Method on Multiple Graphics Processing Units

On Performance Analysis of Graphcore IPUs: Analyzing Squared and Skewed Matrix Multiplication

Performance Evaluation of Parallel Graphs Algorithms Utilizing Graphcore IPU

Insight Gained from Migrating a Machine Learning Model to Intelligence Processing Units

Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs

Tricking AI chips into Simulating the Human Brain: A Detailed Performance Analysis

Benchmarking the Nvidia GPU Lineage: From Early K80 to Modern A100 with Asynchronous Memory Transfers

Benchmarking Edge AI Platforms for High-Performance ML Inference

Acceleration of Graph Neural Network-Based Prediction Models in Chemistry via Co-Design Optimization on Intelligence Processing Units

Evaluating the Performance of NVIDIA's A100 Ampere GPU for Sparse Linear Algebra Computations

Implementation and evaluation of deep neural networks (DNN) on mainstream heterogeneous systems

DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks

Confidential Machine Learning within Graphcore IPUs

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

POM.gpu-v1.0: a GPU-based Princeton Ocean Model

Accelerating Graph Convolutional Networks Through a PIM-Accelerated Approach

Benchmarking Edge Computing Devices for Grape Bunches and Trunks Detection using Accelerated Object Detection Single Shot MultiBox Deep Learning Models