IVF-Net: an Infrared and Visible Data Fusion Deep Network for Traffic Object Enhancement in Intelligent Transportation Systems
Mingye Ju,Chunming He,Juping Liu,Bin Kang,Jian Su,Dengyin Zhang
DOI: https://doi.org/10.1109/tits.2022.3210693
IF: 8.5
2023-01-01
IEEE Transactions on Intelligent Transportation Systems
Abstract:Infrared and visible data fusion (IVF) aims to generate a fused output that simultaneously highlights salient thermal radiation features and preserves texture information, which can not only grasp the necessary information for traffic movement, but also highlight the invisible objects that need to be dodged in intelligent transportation system (ITS). Therefore, IVF is capable of improving the environmental perception ability for various challenging traffic situations, e.g., foggy scenarios, rainy environments, and low-light illumination. However, current available IVF algorithms cannot offer a theoretical manner to integrate a priori knowledge and the network structure into a unified model. Moreover, they always fail to handle infrared and visible data pairs with different resolutions, which is a common occurrence in real ITS scenarios. To this end, this study develops a novel model-inspired unsupervised network termed IVF-Net. Specifically, an enhanced IVF model (IVFM), which pays more attention on detailed texture information and salient objects, is first established. According to proximal gradient theory, then we map this model into a deep network with learnable feature extraction parameters, aiming to draw on the strengths of the fusion model and deep learning to better describe the IVF task. Finally, a multiple task-driven loss function is designed to train the mapped network. Unlike previous work, our IVF-Net is motivated by IVFM, each layer in which has a semantic interpretability and a clear mission, thereby leading to a significantly enhanced fusion effect. Another advantage is that it is only composed of simple convolution-based structures, which ensures its lightweight and efficiency. Experiments demonstrate that IVF-Net can have a stronger ability to capture the key traffic information and highlight the salient feature of imperceptible objects, which makes it an excellent candidate to improve the reliability of subsequent applications in ITS.