Abstract:Data augmentation methods offer a cost-effective and efficient alternative to the acquisition of additional data, significantly enhancing data diversity and model generalization, making them particularly favored in object detection tasks. However, existing data augmentation techniques primarily focus on the visible spectrum and are directly applied to RGB-T object detection tasks, overlooking the inherent differences in image data between the two tasks. Visible images capture rich color and texture information during the daytime, while infrared images are capable of imaging under low-light complex scenarios during the nighttime. By integrating image information from both modalities, their complementary characteristics can be exploited to improve the overall effectiveness of data augmentation methods. To address this, we propose a cross-modality data augmentation method tailored for RGB-T object detection, leveraging masked image modeling within representation learning. Specifically, we focus on the temporal consistency of infrared images and combine them with visible images under varying lighting conditions for joint data augmentation, thereby enhancing the realism of the augmented images. Utilizing the masked image modeling method, we reconstruct images by integrating multimodal features, achieving cross-modality data augmentation in feature space. Additionally, we investigate the differences and complementarities between data augmentation methods in data space and feature space. Building upon existing theoretical foundations, we propose an integrative framework that combines these methods for improved augmentation effectiveness. Furthermore, we address the slow convergence observed with the existing Mosaic method in aerial imagery by introducing a multi-scale training strategy and proposing a full-scale Mosaic method as a complement. This optimization significantly accelerates network convergence. The experimental results validate the effectiveness of our proposed method and highlight its potential for further advancements in cross-modality object detection tasks.

Data Augmentation via Mixed Class Interpolation using Cycle-Consistent Generative Adversarial Networks Applied to Cross-Domain Imagery

Conditional Generative Adversarial Networks for Data Augmentation and Adaptation in Remotely Sensed Imagery

Cross-Modality Data Augmentation for Aerial Object Detection with Representation Learning

Synthesis of Multispectral Optical Images From SAR/Optical Multitemporal Data Using Conditional Generative Adversarial Networks

The Synthetic Aperture Radar-based Image Augmentation Using Generative Adversarial Network

Synthetic Data Augmentation Using Multiscale Attention CycleGAN for Aircraft Detection in Remote Sensing Images

Data Augmentation of Sar Sensor Image via Information Maximizing Generative Adversarial Net

Image Data Augmentation for SAR Sensor Via Generative Adversarial Nets.

Building Cross-Domain Mapping Chains from Multi-CycleGAN for Hyperspectral Image Classification

Cross-Scene Hyperspectral Image Classification Based on Cycle-Consistent Adversarial Networks.

Data Augmentation for Image Classification using Generative AI

Learning to Augment: Hallucinating Data for Domain Generalized Segmentation

Target Recognition in SAR Images by Deep Learning with Training Data Augmentation

Inter-Class and Inter-Domain Semantic Augmentation for Domain Generalization

Improving the Performance of Fine-Grain Image Classifiers via Generative Data Augmentation

Diverse Sample Generation with Multi-Branch Conditional Generative Adversarial Network for Remote Sensing Objects Detection.

How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?

Hybrid Cgan: Coupling Global and Local Features for SAR-to-Optical Image Translation

IAM-ACGAN: A High-Accuracy Approach for SAR Image Augmentation

Generative data augmentation by conditional inpainting for multi-class object detection in infrared images

SAR Image Colorization Using Multidomain Cycle-Consistency Generative Adversarial Network