Abstract:Traditional neural network-driven inpainting methods struggle to deliver high-quality results within the constraints of mobile device processing power and memory. Our research introduces an innovative approach to optimize memory usage by altering the composition of input data. Typically, video inpainting relies on a predetermined set of input frames, such as neighboring and reference frames, often limited to five-frame sets. Our focus is to examine how varying the proportion of these input frames impacts the quality of the inpainted video. By dynamically adjusting the input frame composition based on optical flow and changes of the mask, we have observed an improvement in various contents including rapid visual context changes.
What problem does this paper attempt to address?
### What problems does this paper attempt to solve?
This paper aims to solve several key problems in video inpainting, especially the limitations of memory and computing resources when processing videos on mobile devices. Specifically:
1. **Memory usage efficiency problem**:
- When traditional neural - network - based video inpainting methods run on mobile devices, due to limited processing power and memory, it is difficult to provide high - quality results.
- Video inpainting usually depends on a preset set of input frames (such as neighboring frames and reference frames), and the number of these frames is usually fixed (for example, 5 frames). This method is less efficient in memory - constrained environments.
2. **The influence of input frame configuration on inpainting quality**:
- The paper explores how different input frame combinations (i.e., the ratio of neighboring frames to reference frames) affect the quality of video inpainting.
- The author assumes that in a static visual context, the reference frame is more important; while in a dynamic context, the neighboring frames have a greater impact.
3. **Dynamically adjusting input frame combinations to improve inpainting quality**:
- By introducing a dynamic adjustment mechanism based on optical flow and mask changes, the paper proposes a new method to optimize the configuration of input frames, thereby improving the inpainting quality in rapidly changing visual contexts.
4. **Design of an adaptive input configuration framework**:
- A framework named AdaptIn is proposed. This framework can dynamically select input frames according to the changes in the visual context, thereby maintaining efficient memory usage and high - quality inpainting effects under different video dynamic conditions.
### Main contributions
- **Explore the relationship between input frame configuration and inpainting quality**: Through experiments, the influence of different input frame ratios on inpainting quality is verified, and optical flow and mask changes are proposed as key factors for adjusting input frame combinations.
- **Propose the AdaptIn framework**: This framework can improve the quality of video inpainting by dynamically adjusting input frame combinations in a memory - constrained environment, especially suitable for scenarios with limited computing resources such as mobile devices.
- **Empirical research**: Through experiments on multiple datasets (such as DAVIS 2017 and MOSE), the effectiveness of the AdaptIn framework is verified, and its performance improvement under different video dynamic conditions is demonstrated.
### Formula representation
Some formulas involved in the paper are as follows:
- **Normalized optical flow**:
\[
\text{normalized optical flow} = \sqrt{\frac{u^2 + v^2}{\text{mask\_size}}}
\]
where \( (u, v)_{\text{mask}}=\left(\frac{dx}{dt},\frac{dy}{dt}\right)_{\text{mask}} \) represents the optical flow within the mask area.
- **Mask change amount**:
\[
\text{mask change at time }t = \sum_{k = 1}^{m\times n}|M_{k,t}-M_{k,t - 1}|
\]
where \( M_{k,t} \) represents the mask value of the \(k\) - th pixel at time \(t\), and \(m\times n\) is the frame size.
- **PSNR change rate**:
\[
\text{Signed Maximum Change Rate in PSNR}=\text{SIGN}(\arg\max_r P-\arg\min_r P)\times\frac{\max P-\min P}{\max P}
\]
where \(P = \{ \text{psnr}_r\mid r\in[0.125,0.25,\ldots,0.875]\}\), and \(\text{psnr}_r\) is the PSNR value calculated according to the ratio \(r\) of reference frames in the input frames.