Abstract:In the evolving landscape of video enhancement and editing methodologies, a majority of deep learning techniques often rely on extensive datasets of observed input and ground truth sequence pairs for optimal performance. Such reliance often falters when acquiring data becomes challenging, especially in tasks like video dehazing and relighting, where replicating identical motions and camera angles in both corrupted and ground truth sequences is complicated. Moreover, these conventional methodologies perform best when the test distribution closely mirrors the training distribution. Recognizing these challenges, this paper introduces a novel video decomposition prior `\texttt{VDP}' framework which derives inspiration from professional video editing practices. Our methodology does not mandate task-specific external data corpus collection, instead pivots to utilizing the motion and appearance of the input video. \texttt{VDP} framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels. These set of layers are then manipulated individually to obtain the desired results. We addresses tasks such as video object segmentation, dehazing, and relighting. Moreover, we introduce a novel logarithmic video decomposition formulation for video relighting tasks, setting a new benchmark over the existing methodologies. We observe the property of relighting emerge as we optimize for our novel relighting decomposition formulation. We evaluate our approach on standard video datasets like DAVIS, REVIDE, \& SDSD and show qualitative results on a diverse array of internet videos. Project Page - <a class="link-external link-https" href="https://www.cs.umd.edu/~gauravsh/video_decomposition/index.html" rel="external noopener nofollow">this https URL</a> for video results.
What problem does this paper attempt to address?
The problems that this paper attempts to solve mainly lie in the limitations of existing deep - learning methods in the fields of video enhancement and editing. Specifically, these limitations include:
1. **High data - dependence**: Most existing deep - learning techniques rely on a large number of input and ground - truth sequence pairs (i.e., training data) to achieve optimal performance. However, in some tasks (such as video defogging and relighting), it is very difficult to obtain such data, especially when the same motion and camera angles need to be replicated.
2. **Poor generalization ability**: When the distribution of test data is inconsistent with that of training data, the performance of pre - trained models will decline significantly. This means that these models perform poorly when dealing with unseen data.
To solve the above problems, this paper proposes a new framework - **Video Decomposition Prior (VDP)**. The main features of the VDP framework are as follows:
- **No need for external datasets**: The VDP framework does not need to collect external datasets for specific tasks, but utilizes the motion and appearance information of the input video itself.
- **Multi - layer decomposition**: VDP decomposes the video sequence into multiple RGB layers and corresponding transparency levels, and achieves the desired effects by manipulating these layers.
- **Applicable to multiple tasks**: The VDP framework can be applied to multiple video processing tasks, such as video object segmentation, defogging, and relighting.
### Specific contributions
1. **Introduction of FlowRGBs**: For the first time, FlowRGBs are utilized in the inference - time optimization technique, thereby using the motion cues in the query video.
2. **New logarithmic decomposition formula**: A new logarithmic decomposition formula is proposed for the video relighting task. This new formula makes relighting a naturally occurring property and significantly improves performance.
3. **Excellent performance in downstream applications**: State - of - the - art performance has been achieved in tasks such as video defogging and relighting, especially in video object segmentation, surpassing existing inference - time optimization techniques.
### Technical details
The VDP framework consists of two main modules:
- **RGB - layer prediction module (RGBnet)**: Responsible for handling the appearance aspect of the input video and outputting RGB layers.
- **α - layer prediction module (α - net)**: Responsible for handling the forward optical - flow RGB feature maps related to the input video and outputting transparency layers (Tmap).
These two modules are jointly optimized to achieve video decomposition and reconstruction. By defining appropriate loss functions (such as reconstruction loss and optical - flow warping loss), it is ensured that the decomposed layers can faithfully reconstruct the original video and maintain temporal consistency.
### Application examples
- **Video relighting**: By introducing the logarithmic decomposition formula, VDP can improve video quality under low - light conditions at night and restore good lighting effects.
- **Video defogging**: The defogging problem is regarded as a video decomposition problem, where one layer is a fog - free video and the other layer is an airlight map.
- **Unsupervised video object segmentation**: By decomposing the foreground and background layers in the video, the tracking and segmentation of the most prominent objects are achieved.
In conclusion, by proposing the VDP framework, this paper aims to overcome the problems of data - dependence and poor generalization ability existing in existing video enhancement and editing methods, and provides a more flexible and efficient solution.