Abstract:The prosperous trend of deploying deep neural network (DNN) models to diverse hardware platforms has boosted the development of deep learning (DL) compilers. DL compilers take the high-level DNN model specifications as input and generate optimized DNN executables for diverse hardware architectures like CPUs, GPUs, and various hardware accelerators. Compiling DNN models into high-efficiency executables is not easy: the compilation procedure often involves converting high-level model specifications into several different intermediate representations (IR), e.g., graph IR and operator IR, and performing rule-based or learning-based optimizations from both platform-independent and platform-dependent perspectives. Despite the prosperous adoption of DL compilers in real-world scenarios, principled and systematic understanding toward the correctness of DL compilers does not yet exist. To fill this critical gap, this paper introduces MT-DLComp, a metamorphic testing framework specifically designed for DL compilers to effectively uncover erroneous compilations. Our approach leverages deliberately-designed metamorphic relations (MRs) to launch semantics-preserving mutations toward DNN models to generate their variants. This way, DL compilers can be automatically examined for compilation correctness utilizing DNN models and their variants without requiring manual intervention. We also develop a set of practical techniques to realize an effective workflow and localize identified error-revealing inputs. Real-world DL compilers exhibit a high level of engineering quality. Nevertheless, we detected over 435 inputs that can result in erroneous compilations in four popular DL compilers, all of which are industry-strength products maintained by Amazon, Facebook, Microsoft, and Google. While the discovered error-triggering inputs do not cause the DL compilers to crash directly, they can lead to the generation of incorrect DNN executables. With substantial manual effort and help from the DL compiler developers, we uncovered four bugs in these DL compilers by debugging them using the error-triggering inputs. Our proposed testing frameworks and findings can be used to guide developers in their efforts to improve DL compilers.

Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

Visualizing and Understanding Neural Models in NLP

Testing Neural Network Classifiers Based on Metamorphic Relations

A Declarative Metamorphic Testing Framework for Autonomous Driving

Metamorphic Testing in Autonomous System Simulations

Unveiling Hidden DNN Defects with Decision-Based Metamorphic Testing

Improving the Quality of Computational Science Software by Using Metamorphic Relations to Test Machine Learning Applications

Composite metamorphic relations for integration testing

Towards Automated Metamorphic Test Identification for Ocean System Models

Towards Generating Executable Metamorphic Relations Using Large Language Models

Metamorphic Relation Generation: State of the Art and Visions for Future Research

Testing multiple linear regression systems with metamorphic testing

Towards a Complete Metamorphic Testing Pipeline

Evaluation of Chinese Natural Language Processing System Based on Metamorphic Testing

Adaptive Metamorphic Testing with Contextual Bandits

Metamorphic Testing of Deep Learning Compilers

Identifying implementation bugs in machine learning based image classifiers using metamorphic testing

A study of the interactive role of metamorphic testing and machine learning in the quality assurance of a deep learning forecasting application

One Step Further: Evaluating Interpreters Using Metamorphic Testing

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing