Abstract:Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.

Novel Self-Portrait Enhancement Via Multi-Photo Fusing

1+N Fusion: Cascaded Self-Portrait Enhancement

Selfie Retoucher: Subject-Oriented Self-Portrait Enhancement

Patch-Based Approach for the Fusion of Low-Light Image Pairs

A Fusion Framework For Face Recognition Under Varying Illumination Based On Multi-Scale Analysis

Re-Compositable Panoramic Selfie With Robust Multi-Frame Segmentation And Stitching

Multi-source Remote Sensing Image Registration Based on Sift and Optimization of Local Self-Similarity Mutual Information

Total Selfie: Generating Full-Body Selfies

A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement

Multi-focus Image Fusion with Structure-Driven Adaptive Regions

Unified Photo Enhancement by Discovering Aesthetic Communities From Flickr

Point-and-Shoot All-in-Focus Photo Synthesis from Smartphone Camera Pair

A Fusion-Based Method For Single Backlit Image Enhancement

Relightful Harmonization: Lighting-aware Portrait Background Replacement

Object Recognition Using Multi-View Imaging

A novel multi-scale fusion framework for detail-preserving low-light image enhancement

Research on Feature Matching of Multi-pose Face Based on SIFT

FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing

A Fast and Automatic Fusion Algorithm for Unregistered Multi-Exposure Image Sequence

Robust and Accurate 3D Self-Portraits in Seconds

Multiple Representations-Based Face Sketch–Photo Synthesis