Stereoscopic video deblurring transformer

Hassan Imani,Md Baharul Islam,Masum Shah Junayed,Md Atiqur Rahman Ahad
DOI: https://doi.org/10.1038/s41598-024-63860-9
IF: 4.6
2024-06-23
Scientific Reports
Abstract:Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.
multidisciplinary sciences
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address the problem of stereoscopic video deblurring. Specifically, the authors focus on how to restore clear frames in stereoscopic videos to eliminate blurring caused by camera or object motion. Unlike deblurring methods for monocular images or regular videos, stereoscopic video deblurring requires maintaining both disparity and temporal coherence, making the problem more complex. ### Background and Challenges 1. **Blur in Dynamic Scenes**: Camera shake and object motion are the main causes of video blur, especially in dynamic scenes. 2. **Limitations of Existing Methods**: - Deblurring methods for monocular images and videos are relatively mature, but these methods are usually not suitable for stereoscopic videos. - Stereoscopic video deblurring needs to handle information from two perspectives and consider temporal coherence. - Traditional fully connected self-attention layers and Vision Transformers have limitations in utilizing data locality. - 3D Convolutional Neural Networks (3D CNNs) can handle consecutive frames to correct motion blur but fail to fully utilize stereoscopic information. ### Solution The authors propose a Transformer-based framework for stereoscopic video deblurring, which includes the following key components: 1. **Self-Attention Layer**: Used to model the relationships between pixels within a frame, addressing spatial attention issues. 2. **Feed-Forward Layer**: Based on optical flow methods, used to discover correlations between different video frames and align features, addressing temporal attention issues. 3. **Modified Parallax Attention Mechanism (mPAM)**: Combines stereoscopic information to improve deblurring performance. ### Main Contributions 1. **Proposed a New Transformer Model**: This model deblurs target frames by combining cross-view information and information from adjacent frames. 2. **Designed a New Feed-Forward Layer**: This layer spatially aligns features by calculating relationships between all adjacent frames. 3. **Significantly Improved the PAM Module**: The mPAM is used to combine stereoscopic features, enhancing deblurring performance. 4. **Experimental Validation**: Extensive ablation studies were conducted on two public stereoscopic video datasets, and the experimental results show that this method outperforms existing image and video deblurring techniques in terms of performance. ### Conclusion By combining spatial and temporal attention mechanisms and an improved parallax attention module, this method effectively addresses motion blur in stereoscopic videos, providing a new solution for stereoscopic video deblurring.