Abstract:Data augmentation methods offer a cost-effective and efficient alternative to the acquisition of additional data, significantly enhancing data diversity and model generalization, making them particularly favored in object detection tasks. However, existing data augmentation techniques primarily focus on the visible spectrum and are directly applied to RGB-T object detection tasks, overlooking the inherent differences in image data between the two tasks. Visible images capture rich color and texture information during the daytime, while infrared images are capable of imaging under low-light complex scenarios during the nighttime. By integrating image information from both modalities, their complementary characteristics can be exploited to improve the overall effectiveness of data augmentation methods. To address this, we propose a cross-modality data augmentation method tailored for RGB-T object detection, leveraging masked image modeling within representation learning. Specifically, we focus on the temporal consistency of infrared images and combine them with visible images under varying lighting conditions for joint data augmentation, thereby enhancing the realism of the augmented images. Utilizing the masked image modeling method, we reconstruct images by integrating multimodal features, achieving cross-modality data augmentation in feature space. Additionally, we investigate the differences and complementarities between data augmentation methods in data space and feature space. Building upon existing theoretical foundations, we propose an integrative framework that combines these methods for improved augmentation effectiveness. Furthermore, we address the slow convergence observed with the existing Mosaic method in aerial imagery by introducing a multi-scale training strategy and proposing a full-scale Mosaic method as a complement. This optimization significantly accelerates network convergence. The experimental results validate the effectiveness of our proposed method and highlight its potential for further advancements in cross-modality object detection tasks.

Cross-modal Augmentation: A Data Augmentation Scheme for RGB-Thermal Semantic Segmentation

Cross-Modality Data Augmentation for Aerial Object Detection with Representation Learning

Back-Modality: Leveraging Modal Transformation for Data Augmentation.

Mitigating Modality Discrepancies for RGB-T Semantic Segmentation

Region-adaptive and context-complementary cross modulation for RGB-T semantic segmentation

A Data Augmentation Method Based on Multi-Modal Image Fusion for Detection and Segmentation

Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification

Cross-Modal Data Augmentation for Tasks of Different Modalities

MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

Discriminative Cross-Modal Data Augmentation for Medical Imaging Applications

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Inter-Class and Inter-Domain Semantic Augmentation for Domain Generalization

SFAF-MA: Spatial Feature Aggregation and Fusion With Modality Adaptation for RGB-Thermal Semantic Segmentation

Dual-branch deep cross-modal interaction network for semantic segmentation with thermal images

Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation

Channel Augmentation for Visible-Infrared Re-Identification.

Mask-guided Modality Difference Reduction Network for RGB-T Semantic Segmentation

Don't Judge by the Look: A Motion Coherent Augmentation for Video Recognition

Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation

Exploring Data Augmentation for Multi-Modality 3D Object Detection