Abstract:Video inpainting is the task of filling a region in a video in a visually convincing manner. It is very challenging due to the high dimensionality of the data and the temporal consistency required for obtaining convincing results. Recently, diffusion models have shown impressive results in modeling complex data distributions, including images and videos. Such models remain nonetheless very expensive to train and to perform inference with, which strongly reduce their applicability to videos, and yields unreasonable computational loads. We show that in the case of video inpainting, thanks to the highly auto-similar nature of videos, the training data of a diffusion model can be restricted to the input video and still produce very satisfying results. This leads us to adopt an internal learning approach, which also allows us to greatly reduce the neural network size by about three orders of magnitude less than current diffusion models used for image inpainting. We also introduce a new method for efficient training and inference of diffusion models in the context of internal learning, by splitting the diffusion process into different learning intervals corresponding to different noise levels of the diffusion process. To the best of our knowledge, this is the first video inpainting method based purely on diffusion. Other methods require additional components such as optical flow estimation, which limits their performance in the case of dynamic textures and complex motions. We show qualitative and quantitative results, demonstrating that our method reaches state of the art performance in the case of dynamic textures and complex dynamic backgrounds.

Static and Dynamic Texture Mixing Using Optimal Transport.

Image and Video Retexturing

Dynamic texture analysis with diffusion in networks

Dynamic Optimal Transport with Mixed Boundary Condition for Color Image Processing

Bernoulli Embedding Model and Its Application in Texture Mapping

Dynamic texture synthesis using a spatial temporal descriptor

Two-Stream Convolutional Networks for Dynamic Texture Synthesis

A generalized model for optimal transport of images including dissipation and density modulation

Texture Mixing and Texture Transfer

Dynamic Texture Transfer using PatchMatch and Transformers

Text-Guided Texturing by Synchronized Multi-View Diffusion

Eulerian models and algorithms for unbalanced optimal transport

Texture synthesis and nonparametric resampling of random fields

Infusion: internal diffusion for inpainting of dynamic textures and complex motion

Texture-aware Video Frame Interpolation

Optimal transport for Gaussian mixture models

Generating Non-Stationary Textures using Self-Rectification

Optimal Transportation by Orthogonal Coupling Dynamics

Dynamic Texture Recognition by Spatio-Temporal Multiresolution Histograms

Optimal Transport for Domain Adaptation through Gaussian Mixture Models

Learning subsurface scattering solutions of tightly-packed granular media using optimal transport