Abstract:On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoods for information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed reference-based feature enrichment mechanism, which facilitates handling complex motions. After multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-of-the-art methods on burst super-resolution, burst denoising and burst low-light enhancement. Our codes and pretrained models are available at https:// <a class="link-external link-http" href="http://github.com/akshaydudhane16/Burstormer" rel="external noopener nofollow">this http URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in the image sequences (i.e., burst images) captured by handheld devices (such as smart phones), due to inevitable motion and various degradation factors (such as noise, low - light, etc.), there are misalignments and quality degradations between individual frames. Specifically, modern handheld cameras will quickly and continuously capture multiple images when the shutter is pressed and combine them to generate a high - quality image. However, due to inevitable motion (such as hand - shaking or object movement), these individual frames are misaligned in the burst sequence and contain various degradation problems. To address these problems, the author proposes a novel Transformer architecture named Burstormer for burst image restoration and enhancement. Its main objectives include: 1. **Align burst image frames**: Ensure that each frame in the burst sequence can be correctly aligned, thereby reducing motion - induced blurring and ghosting artifacts. 2. **Fuse complementary information**: Effectively fuse the complementary information between different frames in the burst sequence to achieve a high - quality output image. 3. **Handle complex motion situations**: Design a mechanism so that burst features can be effectively aligned and fused even in complex motion situations. 4. **Improve computational efficiency**: Compared with existing methods, Burstormer aims to provide higher computational efficiency and a more lightweight design. To this end, the author proposes a series of innovative modules and techniques, including: - **Enhanced Deformable Alignment (EDA)**: Used to align burst features and reduce noise while extracting local and non - local features. - **Reference - Based Feature Enrichment (RBFE)**: Used to further correct the aligned burst features, especially in complex motion situations. - **No - Reference Feature Enrichment (NRFE)**: Used to gradually aggregate and upsample burst features to generate the final high - resolution image. - **Cyclic Burst Sampling (CBS)**: Used to flexibly perform inter - frame communication without significantly increasing computational overhead. Through these techniques, Burstormer performs excellently in burst super - resolution, burst denoising, and burst low - light enhancement tasks, and has higher accuracy and lower computational cost compared with existing methods.

Burstormer: Burst Image Restoration and Enhancement Transformer

Burst Image Restoration and Enhancement

Dual-constraint Burst Image Denoising Method

Image Restoration Using Dual-Domain Fusion Network for Rotating Rectangular Synthetic Aperture System

BurstM: Deep Burst Multi-scale SR using Fourier Space with Optical Flow

Unformer: A Transformer-Based Approach for Adaptive Multi-Scale Feature Aggregation in Underwater Image Enhancement

Illuminate the night: lightweight fusion and enhancement model for extreme low-light burst images

Burst-Enhanced Super-Resolution Network (BESR)

BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment

Neural Spline Fields for Burst Image Fusion and Layer Separation

CANDID: Correspondence AligNment for Deep-burst Image Denoising

RBSR: Efficient and Flexible Recurrent Network for Burst Super-Resolution

Burst photography for high dynamic range and low-light imaging on mobile cameras

Burst Image Super-Resolution with Base Frame Selection

Towards Real-World Burst Image Super-Resolution: Benchmark and Method

Fine Dense Alignment of Image Bursts through Camera Pose and Depth Estimation

Burst Photography for Learning to Enhance Extremely Dark Images

Restormer: Efficient Transformer for High-Resolution Image Restoration

Burst Super-Resolution with Diffusion Models for Improving Perceptual Quality

Efficient Burst Raw Denoising with Variance Stabilization and Multi-frequency Denoising Network

Deep Burst Denoising