EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

Linrui Tian,Qi Wang,Bang Zhang,Liefeng Bo

2024-02-27

Abstract:In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper addresses the problem of generating realistic and expressive audio-driven character portrait videos. Traditional methods often rely on 3D models or facial landmarks, which limit the capturing of expressions and individual styles. The paper proposes a new framework called EMO, which uses a diffusion model to directly synthesize videos from audio without the need for intermediate representations, and can generate videos with continuous identity preservation, natural head pose variations, and rich expressions. Experiments show that EMO outperforms existing methods in terms of expressiveness and realism.