Abstract:We present a deep learning-based framework to synthesize motion in-betweening in a two-stage manner. Given some context frames and a target frame, the system can generate plausible transitions with variable lengths in a non-autoregressive fashion. The framework consists of two Transformer Encoder-based networks operating in two stages: in the first stage a Context Transformer is designed to generate rough transitions based on the context and in the second stage a Detail Transformer is employed to refine motion details. Compared to existing Transformer-based methods which either use a complete Transformer Encoder-Decoder architecture or additional 1D convolutions to generate motion transitions, our framework achieves superior performance with less trainable parameters by only leveraging the Transformer Encoder and masked self-attention mechanism. To enhance the generalization of our transformer-based framework, we further introduce Keyframe Positional Encoding and Learned Relative Positional Encoding to make our method robust in synthesizing longer transitions exceeding the maximum transition length during training. Our framework is also artist-friendly by supporting full and partial pose constraints within the transition, giving artists fine control over the synthesized results. We benchmark our framework on the LAFAN1 dataset, and experiments show that our method outperforms the current state-of-the-art methods at a large margin (an average of 16% for normal-length sequences and 55% for excessive-length sequences). Our method trains faster than the RNN-based method and achieves a four-time speedup during inference. We implement our framework into a production-ready tool inside an animation authoring software and conduct a pilot study to validate the practical value of our method.

A filter based approach for inbetweening

Multiframe Interpolation for Video Using Phase Features

Frame Interpolation Using Phase and Amplitude Feature Pyramids

Deep Geometrized Cartoon Line Inbetweening

Correspondence specification learned from master frames for automatic inbetweening

Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation

Deep Sketch-guided Cartoon Video Inbetweening

Thin-Plate Spline-based Interpolation for Animation Line Inbetweening

Automatic Inbetweening for Stroke‐Based Painterly Animation

Motion In-Betweening Via Two-Stage Transformers.

Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening

Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Robust Diffusion‐based Motion In‐betweening

Interactive Separation Network for Image Inpainting

Fine-Grained Motion Estimation for Video Frame Interpolation

Video Frame Interpolation Based on Multi-scale Convolutional Network and Adversarial Training.

A Temporally-Aware Interpolation Network for Video Frame Inpainting

A Novel All-In-One Grid Network for Video Frame Interpolation

The Invertible U-Net for Optical-Flow-free Video Interframe Generation

Video Frame Interpolation Using Recurrent Convolutional Layers

Channel Attention Is All You Need for Video Frame Interpolation