Abstract:Automatically translating chromaticity-free thermal infrared (TIR) images into realistic color visible (CV) images is of great significance for autonomous vehicles, emergency rescue, robot navigation, nighttime video surveillance, and many other fields. Most recent designs use end-to-end neural networks to translate TIR directly to CV; however, compared to these networks, TIR has low contrast and an unclear texture for CV translation. Thus, directly translating the TIR temperature value of only one channel to the RGB color value of three channels without adding additional constraints or semantic information does not handle the one-to-three mapping problem between different domains in a good way, causing the translated CV images not only to have blurred edges but also color confusion. As for the methodology of the work, considering that in the translation from TIR to CV the most important process is to map information from the temperature domain into the color domain, an improved CycleGAN (GMA-CycleGAN) is proposed in this work in order to translate TIR images to grayscale visible (GV) images. Although the two domains have different properties, the numerical mapping is one-to-one, which reduces the color confusion caused by one-to-three mapping when translating TIR to CV. Then, a GV-CV translation network is applied to obtain CV images. Since the process of decomposing GV images into CV images is carried out in the same domain, edge blurring can be avoided. To enhance the boundary gradient between the object (pedestrian and vehicle) and the background, a mask attention module based on the TIR temperature mask and the CV semantic mask is designed without increasing the network parameters, and it is added to the feature encoding and decoding convolution layers of the CycleGAN generator. Moreover, a perceptual loss term is applied to the original CycleGAN loss function to bring the translated images closer to the real images regarding the space feature. In order to verify the effectiveness of the proposed method, the FLIR dataset is used for experiments, and the obtained results show that, compared to the state-of-the-art model, the subjective quality of the translated CV images obtained by the proposed method is better, as the objective evaluation metric FID (Fréchet inception distance) is reduced by 2.42 and the PSNR (peak signal-to-noise ratio) is improved by 1.43.

Thermal Infrared Colorization Via Conditional Generative Adversarial Network

Modality-transfer Generative Adversarial Network and Dual-Level Unified Latent Representation for Visible Thermal Person Re-Identification

Exploring efficient and effective generative adversarial network for thermal infrared image colorization

Thermal Infrared Image Inpainting Via Edge-Aware Guidance

DBSF-Net: Infrared Image Colorization Based on the Generative Adversarial Model with Dual-Branch Feature Extraction and Spatial-Frequency-Domain Discrimination

A feature refinement and adaptive generative adversarial network for thermal infrared image colorization

Thermal image visualization using multi-discriminator CycleGAN with unpaired thermal–visible image training set

An Unpaired Thermal Infrared Image Translation Method Using GMA-CycleGAN

Contrastive learning with feature fusion for unpaired thermal infrared image colorization

Thermal Infrared Image Colorization for Nighttime Driving Scenes With Top-Down Guided Attention

Attention-Guided NIR Image Colorization via Adaptive Fusion of Semantic and Texture Clues

Nighttime Thermal Infrared Image Colorization with Feedback-based Object Appearance Learning

MUGAN: Thermal Infrared Image Colorization Using Mixed-Skipping UNet and Generative Adversarial Network

Infrared and Visible Image Fusion via Texture Conditional Generative Adversarial Network

Deep Learning Thermal Image Translation for Night Vision Perception

Spectral Image Visualization Using Generative Adversarial Networks

V2T-GAN: Three-Level Refined Light-Weight GAN with Cascaded Guidance for Visible-to-Thermal Translation

Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model

ClawGAN: Claw connection-based generative adversarial networks for facial image translation in thermal to RGB visible light

Double-Channel Guided Generative Adversarial Network for Image Colorization