OFPF-MEF: An Optical Flow Guided Dynamic Multi-exposure Image Fusion Network with Progressive Frequencies Learning
Wenhui Hong,Hao Zhang,Jiayi Ma
DOI: https://doi.org/10.1109/tmm.2024.3379883
IF: 7.3
2024-01-01
IEEE Transactions on Multimedia
Abstract:In this paper, we propose a novel optical flow guided network with progressive frequencies learning, achieving promising dynamic multi-exposure image fusion. Specifically, the proposed method consists of the optical flow alignment block and the progressive frequencies fusion block, where the former is to alleviate the ghost caused by the camera and object motions, and the latter is dedicated to synthesizing the desired color and details. First, in the optical flow alignment block, we estimate the optical flow between two source images and utilize the deformable convolutional network to achieve spatial alignment guided by the estimated optical flow. Second, in the progressive frequencies fusion block, color correction and details preservation are implemented in two gradual phases, i.e., low and high frequencies. For the low frequency fusion phase, we combine the convolutional neural network and swin transformer to capture local and global features, so as to consider the color correction from a complete perspective. For the high frequency fusion phase, an attention gate is designed to evaluate the important details from source images, bringing fewer artifacts and halos. Finally, the low and high frequency fusion phases are connected through a residual mapping strategy to generate a desired image with reasonable colors and rich details. Extensive experiments on publicly available datasets reveal that our method outperforms the state-of-the-art for both static and dynamic scenarios. Moreover, our method is superior in running efficiency over most of the state-of-the-art methods.
computer science, information systems,telecommunications, software engineering