Abstract:This paper aims to bring fine-grained expression control to identity-preserving portrait generation. Existing methods tend to synthesize portraits with either neutral or stereotypical expressions. Even when supplemented with control signals like facial landmarks, these models struggle to generate accurate and vivid expressions following user instructions. To solve this, we introduce EmojiDiff, an end-to-end solution to facilitate simultaneous dual control of fine expression and identity. Unlike the conventional methods using coarse control signals, our method directly accepts RGB expression images as input templates to provide extremely accurate and fine-grained expression control in the diffusion process. As its core, an innovative decoupled scheme is proposed to disentangle expression features in the expression template from other extraneous information, such as identity, skin, and style. On one hand, we introduce \textbf{I}D-irrelevant \textbf{D}ata \textbf{I}teration (IDI) to synthesize extremely high-quality cross-identity expression pairs for decoupled training, which is the crucial foundation to filter out identity information hidden in the expressions. On the other hand, we meticulously investigate network layer function and select expression-sensitive layers to inject reference expression features, effectively preventing style leakage from expression signals. To further improve identity fidelity, we propose a novel fine-tuning strategy named \textbf{I}D-enhanced \textbf{C}ontrast \textbf{A}lignment (ICA), which eliminates the negative impact of expression control on original identity preservation. Experimental results demonstrate that our method remarkably outperforms counterparts, achieves precise expression control with highly maintained identity, and generalizes well to various diffusion models.

Expressive Facial Style Transfer for Personalized Memes Mimic

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

Exaggerated Portrait Caricatures Synthesis

Unconstrained Facial Expression Transfer using Style-based Generator

CAST: Learning Both Geometric and Texture Style Transfers for Effective Caricature Generation

Modeling Caricature Expressions by 3D Blendshape and Dynamic Texture

A Facial Expression Transfer Method Based on 3DMM and Diffusion Models

A Stylized Method for Facial Cartoon Based on Feature Tracking and Rending

Towards harmonized regional style transfer and manipulation for facial images

Facial Expression Retargeting from Human to Avatar Made Easy

Non-corresponding and topology-free 3D face expression transfer

EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model

Controllable Facial Caricaturization with Localized Deformation and Personalized Semantic Attentions

Learning to Caricature Via Semantic Shape Transform

MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes

Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation

High-Quality Face Caricature via Style Translation

Your3dEmoji: Creating Personalized Emojis Via One-shot 3D-Aware Cartoon Avatar Synthesis.

Synthesis, Style Editing, and Animation of 3D Cartoon Face

WEM-GAN: Wavelet transform based facial expression manipulation

EmojiDiff: Advanced Facial Expression Control with High Identity Preservation in Portrait Generation