Abstract:Parallelizing sequentially written programs is a challenging task. Even experienced developers need to spend considerable time finding parallelism opportunities and then actually writing parallel versions of sequentially written programs. To address this issue, we present AUTOPARLLM, a framework for automatically discovering parallelism and generating the parallel version of the sequentially written program. Our framework consists of two major components: i) a heterogeneous Graph Neural Network (GNN) based parallelism discovery and parallel pattern detection module, and ii) an LLM-based code generator to generate the parallel counterpart of the sequential programs. We use the GNN to learn the flow-aware characteristics of the programs to identify parallel regions in sequential programs and then construct an enhanced prompt using the GNN's results for the LLM-based generator to finally produce the parallel counterparts of the sequential programs. We evaluate AUTOPARLLM on 11 applications of 2 well-known benchmark suites: NAS Parallel Benchmark and Rodinia Benchmark. Our results show that AUTOPARLLM is indeed effective in improving the state-of-the-art LLM-based models for the task of parallel code generation in terms of multiple code generation metrics. AUTOPARLLM also improves the average runtime of the parallel code generated by the state-of-the-art LLMs by as high as 3.4% and 2.9% for the NAS Parallel Benchmark and Rodinia Benchmark respectively. Additionally, to overcome the issue that well-known metrics for translation evaluation have not been optimized to evaluate the quality of the generated parallel code, we propose OMPScore for evaluating the quality of the generated code. We show that OMPScore exhibits a better correlation with human judgment than existing metrics, measured by up to 75% improvement of Spearman correlation.

Parallelization Techniques for Verifying Neural Networks

CLEVEREST: Accelerating CEGAR-based Neural Network Verification via Adversarial Attacks

Online Verification of Deep Neural Networks under Domain Shift or Network Updates

Bound Tightening using Rolling-Horizon Decomposition for Neural Network Verification

Parallel Algorithms Align with Neural Execution

A Dual Approach to Scalable Verification of Deep Networks

PaSE: Parallelization Strategies for Efficient DNN Training

Expediting Neural Network Verification via Network Reduction

Scaling the Convex Barrier with Sparse Dual Algorithms

Incremental Verification of Neural Networks

Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks

Efficient verification of neural networks based on neuron branching and LP abstraction

Probabilistic Verification of Neural Networks using Branch and Bound

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

An Empirical Study on Improving the Speed and Generalization of Neural Networks Using a Parallel Circuit Approach

Integrated Model, Batch and Domain Parallelism in Training Neural Networks

Parallel Verification of Natural Deduction Proof Graphs

Pruning and Slicing Neural Networks using Formal Verification

AUTOPARLLM: GNN-Guided Automatic Code Parallelization using Large Language Models

Provably Bounding Neural Network Preimages

GPU-Accelerated Verification of Machine Learning Models for Power Systems