EvTexture: Event-driven Texture Enhancement for Video Super-Resolution

Dachun Kai,Jiayao Lu,Yueyi Zhang,Xiaoyan Sun

2024-06-19

Abstract:Event-based vision has drawn increasing attention due to its unique characteristics, such as high temporal resolution and high dynamic range. It has been used in video super-resolution (VSR) recently to enhance the flow estimation and temporal alignment. Rather than for motion learning, we propose in this paper the first VSR method that utilizes event signals for texture enhancement. Our method, called EvTexture, leverages high-frequency details of events to better recover texture regions in VSR. In our EvTexture, a new texture enhancement branch is presented. We further introduce an iterative texture enhancement module to progressively explore the high-temporal-resolution event information for texture restoration. This allows for gradual refinement of texture regions across multiple iterations, leading to more accurate and rich high-resolution details. Experimental results show that our EvTexture achieves state-of-the-art performance on four datasets. For the Vid4 dataset with rich textures, our method can get up to 4.67dB gain compared with recent event-based methods. Code: <a class="link-external link-https" href="https://github.com/DachunKai/EvTexture" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

This paper mainly discusses the texture enhancement problem in Video Super-Resolution (VSR). Existing VSR methods usually focus on learning inter-frame motion to restore missing details. However, this paper proposes the first method that utilizes event signals for texture enhancement, called EvTexture. The event signals captured by event cameras have high temporal resolution and dynamic range. The proposed method utilizes these characteristics to progressively restore high-frequency texture information, achieved through a neural network architecture that consists of two branches, motion and texture enhancement. In the texture enhancement branch, an iterative texture enhancement module is designed to gradually explore event information for texture region restoration, thereby refining and improving the accuracy of high-resolution details.

Experimental results show that EvTexture achieves state-of-the-art performance on four datasets, especially in video clips with rich textures, where it outperforms other event-based methods by up to 4.67dB. The paper also compares RGB baseline methods with event baseline methods, and demonstrates that EvTexture effectively utilizes event signals, while improving the clarity and consistency of texture regions while maintaining high-quality reconstruction.

In summary, the main contributions of the paper include:

1. Proposing an event-driven texture restoration solution for VSR.
2. Designing a texture enhancement branch and an iterative texture enhancement module to progressively restore high-frequency texture information.
3. Experimental results demonstrate the superior performance of the proposed method on four VSR benchmark tests, particularly in handling video clips with rich textures.