E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

Jiaqing Zhang,Mingxiang Cao,Xue Yang,Weiying Xie,Jie Lei,Daixun Li,Wenbo Huang,Yunsong Li

2024-05-23

Abstract:Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the application issues of multimodal image fusion and object detection in fields such as autonomous driving. Specifically:

1. **Simplifying the Training Process**: Existing multimodal fusion and object detection methods typically adopt a staged training approach, which limits their widespread application due to its complexity. Therefore, the paper proposes an end-to-end algorithm (E2E-MFD) that can achieve high-performance multimodal fusion detection in a single training phase.

2. **Synchronous Joint Optimization**: E2E-MFD synchronously optimizes various components to avoid suboptimal solutions caused by individual tasks. Additionally, it introduces the Gradient Matrix Task Alignment (GMTA) technique to ensure comprehensive optimization of shared parameters, thereby achieving the optimal fusion detection configuration.

3. **Enhancing Detection Performance**: Through extensive testing on multiple public datasets, E2E-MFD demonstrates its superior capabilities in visual effects and detection results. Notably, it improves the mAP 50 value by 3.9% on the horizontal object detection dataset M3FD and by 2.0% on the inclined object detection dataset DroneVehicle.

4. **Unified Feature Representation**: Existing methods mostly focus on how to utilize object detection information to enhance fusion effects but often struggle to balance parameters and easily fall into local optima. E2E-MFD aims to find a unified feature set that meets the needs of each task, thereby overcoming these challenges.

In summary, the main goal of this paper is to simplify the training process of multimodal image fusion and object detection, improve detection performance, and address optimization challenges in existing methods by introducing the E2E-MFD algorithm.