Abstract:The explosive growth of various types of big data and advances in AI technologies have catalyzed a new type of workloads called multi-modal DNNs. Multi-modal DNNs are capable of interpreting and reasoning about information from multiple modalities, making them more applicable to real-world AI scenarios. In recent research, multi-modal DNNs have outperformed the best uni-modal DNN in a wide range of distributed computing applications from traditional multimedia systems to emerging autonomous edge systems. However, despite their importance and superiority, very limited research attention has been devoted to understand the characteristics of multi-modal DNNs and their implications on current computing software/hardware platforms. Existing benchmarks either target uni-modal DNNs or only focus on the algorithm characteristics of multi-modal DNNs. There lacks representative benchmark suites that provide comprehensive system and architecture level analysis of multi-modal networks. To advance the understanding of these multi-modal DNN workloads and facilitate related research, we present MMBench, an open-source, end-to-end benchmark suite consisting of a set of real-world multi-modal DNN workloads with relevant performance metrics for evaluation. We then use MMBench to conduct an in-depth analysis on the characteristics of multi-modal DNNs. We demonstrate their unique characteristics of clear multi-stage execution, frequent synchronization and high heterogeneity, which distinguish them from conventional uni-modal DNNs. Finally, we conduct a case study and extend our benchmark to edge devices. We hope that our work can provide insights for future software/hardware design and optimization to underpin multi-modal DNNs on both cloud and edge computing platforms.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the unique characteristics and challenges in system and architecture design of multi - modal deep neural networks (multi - modal DNNs). Although multi - modal DNNs show higher prediction accuracy than single - modal DNNs in various applications, their inherent characteristics, such as the three - stage execution mode, frequent synchronization operations and high workload heterogeneity, pose new challenges to existing computing software/hardware platforms. Currently, there is a lack of benchmarking suites at the system and architecture levels specifically for multi - modal DNNs, which limits in - depth understanding and optimization of these network characteristics. Therefore, the paper proposes an open - source end - to - end benchmarking suite named MMBench, aiming to fill this gap and provide comprehensive system and architecture analysis tools to promote the research and development of multi - modal DNNs. Specifically, MMBench aims to solve the following problems: 1. **Lack of representative benchmarks**: Existing benchmarks mainly focus on single - modal DNNs or are limited to algorithm - level feature analysis and fail to cover the system and architecture characteristics of multi - modal DNNs. 2. **Insufficient understanding at the system and architecture levels**: Multi - modal DNNs have a unique three - stage execution mode, high heterogeneity and frequent synchronization operations, and these characteristics need to be analyzed in detail at the system and architecture levels. 3. **Cross - platform support**: Existing multi - modal DNN benchmarks usually do not support edge devices, while MMBench not only supports cloud platforms but also extends to edge devices, providing broader applicability. 4. **Performance bottleneck identification**: Through detailed performance analysis, MMBench can help researchers identify performance bottlenecks of multi - modal DNNs on different platforms, thereby guiding future software and hardware optimization designs. Through the introduction of MMBench, the paper hopes to provide valuable insights for future software/hardware design and optimization of multi - modal DNNs, especially on cloud computing and edge computing platforms.

MMBench: Benchmarking End-to-End Multi-modal DNNs and Understanding Their Hardware-Software Implications

Analyzing the Hardware-Software Implications of Multi-modal DNN Workloads Using MMBench.

An End-to-End Benchmarking Tool for Analyzing the Hardware-Software Implications of Multi-modal DNNs

Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking.

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking

Characterizing and Understanding End-to-End Multi-Modal Neural Networks on GPUs.

AIBench: an Industry Standard AI Benchmark Suite from Internet Services.

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

TBD: Benchmarking and Analyzing Deep Neural Network Training

EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices

MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite

Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-chip Training

Benchmarking State-of-the-Art Deep Learning Software Tools

A Survey on Multimodal Benchmarks: In the Era of Large AI Models

Sub-model Parallelism: A Scale-out Deployment Method for Large Multi-modal DNNs