Abstract:PDF HTML XML Export Cite reminder Text-driven Face Image Generation and Manipulation via Multi-level Residual Mapper DOI: 10.21655/ijsi.1673-7288.00313 Author: Affiliation: Clc Number: Fund Project: Article | Figures | Metrics | Reference | Related | Cited by | Materials | Comments Abstract:Although Generative Adversarial Networks (GANs) have achieved great success in face image generation and manipulation, discovering meaningful directions in the latent space of GANs to manipulate semantic attributes is a difficult but meaningful challenge in computer vision. The realization of this challenge typically requires large amounts of labeled data and several hours of network fine-tuning. However, obtaining an annotated collection of images for each desired manipulation is usually very expensive and time-consuming. Recent works aim to overcome this limitation by leveraging the pre-trained models. While they are promising, the accuracy of the manipulation and the authenticity of the results cannot meet the needs of real face editing scenarios. To address these problems, this paper encodes the image and text description into a shared embedding space and proposes a unified image generation and manipulation framework by leveraging the powerful joint representation capability from Contrastive Language-Image Pre-training (CLIP). With carefully designed network structures and loss functions, the proposed framework can learn a latent residual mapper network to map the input conditions into corresponding latent code residuals. This scheme enables the proposed method to perform high-quality face image generation and manipulation by leveraging the generative power from the pre-trained StyleGAN2 model. Extensive experiments demonstrate the superiority of the proposed approach in terms of manipulation accuracy, visual realism, and irrelevant attribute preservation. Reference Related Cited by

Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Generative Adversarial Network for Text-to-Face Synthesis and Manipulation

DualG-GAN, a Dual-channel Generator based Generative Adversarial Network for text-to-face synthesis

Two Birds with One Stone: Iteratively Learn Facial Attributes with GANs.

Audio-driven Talking Face Video Generation with Natural Head Pose

Generative Adversarial Networks for text-to-face synthesis & generation: A quantitative–qualitative analysis of Natural Language Processing encoders for Spanish

Attention Based Facial Expression Manipulation

Text2FaceGAN: Face Generation from Fine Grained Textual Descriptions

Two Birds with One Stone: Transforming and Generating Facial Images with Iterative GAN

Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models

Towards Open-World Text-Guided Face Image Generation and Manipulation

Semantic Text-to-Face GAN -ST^2FG

AnyFace: Free-style Text-to-Face Synthesis and Manipulation

Face Generation from Textual Features using Conditionally Trained Inputs to Generative Adversarial Networks

A face template: Improving the face generation quality of multi-stage generative adversarial networks using coarse-grained facial priors

Optimal text-to-image synthesis model for generating portrait images using generative adversarial network techniques

DMF-GAN: Deep Multimodal Fusion Generative Adversarial Networks for Text-to-Image Synthesis

Audio2Face: Generating Speech/Face Animation from Single Audio with Attention-Based Bidirectional LSTM Networks

Text-driven Face Image Generation and Manipulation via Multi-level Residual Mapper