Abstract:We propose ETBHD‐HMF, a novel hierarchical multimodal fusion network that comprehensively learns to align and fuse text instructions into complex image latent code space, achieving high‐quality and accurate hair design. Here, we offer an illustrative example of joint hair colour and hairstyle text as conditional inputs. Text‐based hair design (TBHD) represents an innovative approach that utilizes text instructions for crafting hairstyle and colour, renowned for its flexibility and scalability. However, enhancing TBHD algorithms to improve generation quality and editing accuracy remains a current research difficulty. One important reason is that existing models fall short in alignment and fusion designs. Therefore, we propose a new layered multimodal fusion network called ETBHD‐HMF, which decouples the input image and hair text information into layered hair colour and hairstyle representations. Within this network, the channel enhancement separation (CES) module is proposed to enhance important signals and suppress noise for text representation obtained from CLIP, thus improving generation quality. Based on this, we develop the weighted mapping fusion (WMF) sub‐networks for hair colour and hairstyle. This sub‐network applies the mapper operations to input image and text representations, acquiring joint information. The WMF then selectively merges image representation and joint information from various style layers using weighted operations, ultimately achieving fine‐grained hairstyle designs. Additionally, to enhance editing accuracy and quality, we design a modality alignment loss to refine and optimize the information transmission and integration of the network. The experimental results of applying the network to the CelebA‐HQ dataset demonstrate that our proposed model exhibits superior overall performance in terms of generation quality, visual realism, and editing accuracy. ETBHD‐HMF (27.8 PSNR, 0.864 IDS) outperformed HairCLIP (26.9 PSNR, 0.828 IDS), with a 3% higher PSNR and a 4% higher IDS.

Hair Attribute Transfer Via Deep Feature Fusion

Ihair Recolorer: Deep Image-to-video Hair Color Transfer

CFMNet - Coarse-to-Fine Cascaded Feature Mapping Network for Hair Attribute Transfer.

Hair-GAN: Recovering 3D Hair Structure from a Single Image Using Generative Adversarial Networks

GLStyleNet: Exquisite Style Transfer Combining Global and Local Pyramid Features

Stable-Hair: Real-World Hair Transfer via Diffusion Model

What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer

HairNet: Single-View Hair Reconstruction using Convolutional Neural Networks

HairGAN: Spatial-Aware Palette GAN for Hair Color Transfer

HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling

Deep Identity-aware Transfer of Facial Attributes

Fully-Featured Attribute Transfer

Transferring Rich Deep Features for Facial Beauty Prediction

Personalized hairstyle and hair color editing based on multi-feature fusion

ETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair Design

Hairstyle transfer via manipulating decoupled latent codes of StyleGAN2

NeuralHDHair: Automatic High-fidelity Hair Modeling from a Single Image Using Implicit Neural Representations

Pose-Guided High-Resolution Appearance Transfer via Progressive Training

HairFIT: Pose-Invariant Hairstyle Transfer via Flow-based Hair Alignment and Semantic-Region-Aware Inpainting

HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach

Real-Time Hair Simulation With Neural Interpolation