Abstract:Event cameras sense brightness changes in each pixel and yield asynchronous event streams instead of producing intensity images. They have distinct advantages over conventional cameras, such as a high dynamic range (HDR) and no motion blur. To take advantage of event cameras with existing image-based algorithms, a few methods have been proposed to reconstruct images from event streams. However, the output images have a low resolution (LR) and are unrealistic. Low-quality outputs stem from broader applications of event cameras, where high-quality and high-resolution (HR) images are needed. In this work, we consider the problem of reconstructing and super-resolving images from LR events when no ground truth (GT) HR images and degradation models are available. We propose a novel end-to-end joint framework for single image reconstruction and super-resolution from LR event data. Our method is primarily unsupervised to handle the absence of real inputs from GT and deploys adversarial learning. To train our framework, we constructed an open dataset, including simulated events and real-world images. The use of the dataset boosts the network performance, and the network architectures and various loss functions in each phase help improve the quality of the resulting image. Various experiments showed that our method surpasses the state-of-the-art LR image reconstruction methods for real-world and synthetic datasets. The experiments for super-resolution (SR) image reconstruction also substantiate the effectiveness of the proposed method. We further extended our method to more challenging problems of HDR, sharp image reconstruction, and color events. In addition, we demonstrate that the reconstruction and super-resolution results serve as intermediate representations of events for high-level tasks, such as semantic segmentation, object recognition, and detection. We further examined how events affect the outputs of the three phases and analyze our method's efficacy through an ablation study.

EvIntSR-Net: Event Guided Multiple Latent Frames Reconstruction and Super-resolution

Event-Based Fusion for Motion Deblurring with Cross-modal Attention

Deep Networks with Detail Enhancement for Infrared Image Super-Resolution

Joint Framework for Single Image Reconstruction and Super-Resolution with an Event Camera

Event Enhanced High-Quality Image Recovery

Boosting Event Stream Super-Resolution with a Recurrent Neural Network.

Intensity-image reconstruction for event cameras using convolutional neural network

Asymmetric Event-Guided Video Super-Resolution

Guided Depth Map Super-Resolution Using Recumbent Y Network

Event Stream Super-Resolution Via Spatiotemporal Constraint Learning

Learning to Super-Resolve Blurry Images with Events

Learning Event Guided High Dynamic Range Video Reconstruction

Learning to Reconstruct High Speed and High Dynamic Range Videos from Events

EventHDR: from Event to High-Speed HDR Videos and Beyond

Super-Resolving Blurry Images with Events

Video super-resolution via event-driven temporal alignment

Robust Intensity Image Reconstruciton Based on Event Cameras

A Residual Learning Approach to Deblur and Generate High Frame Rate Video with an Event Camera

Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations

Event-Stream Super Resolution using Sigma-Delta Neural Network

Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution