Abstract:Talking-head video editing aims to efficiently insert, delete, and substitute the word of a pre-recorded video through a text transcript editor. The key challenge for this task is obtaining an editing model that generates new talking-head video clips which simultaneously have accurate lip synchronization and motion smoothness. Previous approaches, including 3DMM-based (3D Morphable Model) methods and NeRF-based (Neural Radiance Field) methods, are sub-optimal in that they either require minutes of source videos and days of training time or lack the disentangled control of verbal (e.g., lip motion) and non-verbal (e.g., head pose and expression) representations for video clip insertion. In this work, we fully utilize the video context to design a novel framework for talking-head video editing, which achieves efficiency, disentangled motion control, and sequential smoothness. Specifically, we decompose this framework to motion prediction and motion-conditioned rendering: (1) We first design an animation prediction module that efficiently obtains smooth and lip-sync motion sequences conditioned on the driven speech. This module adopts a non-autoregressive network to obtain context prior and improve the prediction efficiency, and it learns a speech-animation mapping prior with better generalization to novel speech from a multi-identity video dataset. (2) We then introduce a neural rendering module to synthesize the photo-realistic and full-head video frames given the predicted motion sequence. This module adopts a pre-trained head topology and uses only few frames for efficient fine-tuning to obtain a person-specific rendering model. Extensive experiments demonstrate that our method efficiently achieves smoother editing results with higher image quality and lip accuracy using less data than previous methods.

Clarification of Video Retrieval Query Results by the Automated Insertion of Supporting Shots

Refilming with Depth-Inferred Videos

Transcript to Video: Efficient Clip Sequencing from Texts

Video Editing for Video Retrieval

Editing like Humans: A Contextual, Multimodal Framework for Automated Video Editing

Interactive Videos: Plausible Video Editing Using Sparse Structure Points

Automatic Non-Linear Video Editing Transfer

Automated aesthetic enhancement of videos

Analogies based video editing

A Metadata Generation System with Semantic Understanding for Video Retrieval in Film Production

Story-driven Video Editing

The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing

Use What You Have: Video Retrieval Using Representations from Collaborative Experts.

B-Script: Transcript-based B-roll Video Editing with Recommendations

VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing

Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

Video scene segmentation and semantic representation using a novel scheme

ReVideo: Remake a Video with Motion and Content Control

Multimodal Contextualized Support for Enhancing Video Retrieval System

InstructVid2Vid: Controllable Video Editing with Natural Language Instructions

Context-Aware Talking-Head Video Editing