PortraitNet: Real-time portrait segmentation network for mobile device

Song-Hai Zhang,Xin Dong,Hui Li,Ruilong Li,Yong-Liang Yang

DOI: https://doi.org/10.1016/j.cag.2019.03.007

IF: 1.821

2019-01-01

Computers & Graphics

Abstract:Image, graphical abstract

What problem does this paper attempt to address?

A Scalable Real-time Semantic Segmentation Network for Autonomous Driving

Jun Liu,Chao Wu,Geng Yuan,Wei Niu,Wenbin Zhang,Houbing Herbert Song

DOI: https://doi.org/10.1145/3606042.3616451

2023-01-01

Abstract:Computational resources are limited on real-time embedded devices, so the available computing cost of deployment on the target platform must be considered. We develop a feature extraction module based on the MobileNet backbone that can be adjusted in terms of computational complexity and capacity using the depth multiplier parameter, classifier depth, and kernel depth. These three parameters allow us to control the count of channels within the network, effectively managing the model's capacity and computational requirements. To achieve semantic segmentation, we incorporate additional components, such as an extension module. This extension module typically includes 1x1 pointwise convolutional layers for pixel-level classification and a transposed convolutional layer for upsampling the output to the original input image size. By combining the feature extraction module with this extension module, we can create a complete architecture capable of performing semantic segmentation tasks. The feature extraction module provides the initial feature extraction and the extension module adds the necessary components for accurate pixel-wise classification and upsampling. Compared to Hardware-aware Neural Architecture Search (NAS), pruning, runtime pruning, and knowledge distillation methods, our model has several advantages in terms of modular design, structural controllability, ease of implementation, and cost-effectiveness. Our computational efficiency, as measured by FLOPS, is highly competitive. Our method is distinguished by solving the problem of MobileNet's inability to adjust the size and number of convolution kernels. It achieves this through adaptable parameter tuning, including MobileNet's depth multiplier, the kernel size in the FCN head's Separable Convolution layer, and the depth of the first Point-wise Convolution layer. These adjustments are customized to match hardware's max multiply-accumulates (MACs), optimizing network capacity and maximizing resource utilization.
Multi-objects Real Time Recognition Based on Color Information

Liang Zhang,Rong Xiong,Jian Chu

DOI: https://doi.org/10.1109/wcica.2004.1342284

2004-01-01

Abstract:Multi-objects real time recognition encounters several difficulties, such as the large quantity of data that have to be processed, the fast speed of recognition that guarantees the real time and the disturbance from the change of environmental illuminating level. In this paper, we present a new technique to pre-sampling and real-time segmenting, which combined two image segmentation techniques: the simplified nearest neighbor classification and color space thresholding. This method simplifies sampling and result in good segmentation and real-time performance. We also describe the bit mark method used to represent object's color and the real-time recognition arithmetic based on grid sequential lookup. Experiments in real applications showed that the sensitivity in pixels classification is decreased, anti-jamming and real-time performance are increased, and it is robust to the change of lighting.
Portrait Segmentation Using Deep Learning

Sumedh Vilas Datar and,Jesus Gonzales Bernal

DOI: https://doi.org/10.48550/arXiv.2202.02705

2022-02-06

Computer Vision and Pattern Recognition

Abstract:A portrait is a painting, drawing, photograph, or engraving of a person, especially one depicting only the face or head and shoulders. In the digital world the portrait of a person is captured by having the person as a subject in the image and capturing the image of the person such that the background is blurred. DSLRs generally do it by reducing the aperture to focus on very close regions of interest and automatically blur the background. In this paper I have come up with a novel approach to replicate the portrait mode from DSLR using any smartphone to generate high quality portrait images.
Feature-Based Automatic Portrait Generation System

Yin Zhang,Lu Gao,Sanyuan Zhang

DOI: https://doi.org/10.1109/CSIE.2009.683

2009-01-01

Abstract:Creating a portrait in the style of a particular artistic tradition or a particular artist is a difficult and also an interesting problem in Non-photorealistic Rendering. This paper develops an automatic feature-based portrait generation system, which can transform the input facial photo into a portrait in the style of a given artist’s finished work. First, we propose a new face model, which contains 64 feature points. Second, we extract the facial features using rules and geometric information. Based on the statistics, we divide the face into several small regions, each of which only contains one feature. The fixed-direction open active contour models are employed to extract corresponding feature points in each region. Finally, the system generates a portrait by feature-based morphing, which uses the artist’s work as the source image. The results show that the system is capable of producing portraits in different styles effectively.
MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices

Jianwen Jiang,Gaojie Lin,Zhengkun Rong,Chao Liang,Yongming Zhu,Jiaqi Yang,Tianyun Zhong

2024-07-08

Abstract:Existing neural head avatars methods have achieved significant progress in the image quality and motion range of portrait animation. However, these methods neglect the computational overhead, and to the best of our knowledge, none is designed to run on mobile devices. This paper presents MobilePortrait, a lightweight one-shot neural head avatars method that reduces learning complexity by integrating external knowledge into both the motion modeling and image synthesis, enabling real-time inference on mobile devices. Specifically, we introduce a mixed representation of explicit and implicit keypoints for precise motion modeling and precomputed visual features for enhanced foreground and background synthesis. With these two key designs and using simple U-Nets as backbones, our method achieves state-of-the-art performance with less than one-tenth the computational demand. It has been validated to reach speeds of over 100 FPS on mobile devices and support both video and audio-driven inputs.

Computer Vision and Pattern Recognition,Artificial Intelligence
Towards Real-Time Automatic Portrait Matting on Mobile Devices

Seokjun Seo,Seungwoo Choi,Martin Kersner,Beomjun Shin,Hyungsuk Yoon,Hyeongmin Byun,Sungjoo Ha

DOI: https://doi.org/10.48550/arXiv.1904.03816

2019-04-08

Abstract:We tackle the problem of automatic portrait matting on mobile devices. The proposed model is aimed at attaining real-time inference on mobile devices with minimal degradation of model performance. Our model MMNet, based on multi-branch dilated convolution with linear bottleneck blocks, outperforms the state-of-the-art model and is orders of magnitude faster. The model can be accelerated four times to attain 30 FPS on Xiaomi Mi 5 device with moderate increase in the gradient error. Under the same conditions, our model has an order of magnitude less number of parameters and is faster than Mobile DeepLabv3 while maintaining comparable performance. The accompanied implementation can be found at \url{<a class="link-external link-https" href="https://github.com/hyperconnect/MMNet" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning,Neural and Evolutionary Computing
Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation

Xi Chen,Donglian Qi,Jianxin Shen

DOI: https://doi.org/10.48550/arxiv.1901.03814

2019-01-01

Abstract:Compared with other semantic segmentation tasks, portrait segmentation requires both higher precision and faster inference speed. However, this problem has not been well studied in previous works. In this paper, we propose a lightweight network architecture, called Boundary-Aware Network (BANet) which selectively extracts detail information in boundary area to make high-quality segmentation output with real-time( >25FPS) speed. In addition, we design a new loss function called refine loss which supervises the network with image level gradient information. Our model is able to produce finer segmentation results which has richer details than annotations.
Real-time portrait image retouching extended from DualBLN

Xiang Zhang,Dawei Yan,Genqing Bian,Chengzhe Lu,Sifei Wang,Ghulam Mohi-ud-din,Qingsen Yan,Wei Dong

DOI: https://doi.org/10.1016/j.eswa.2023.121958

IF: 8.5

2024-03-01

Expert Systems with Applications

Abstract:Towards the target of image retouching tasks, the three-dimension LookUp Table (3D LUT) is a discrete 3D lattice that represents the color mapping to correct the inconsistent color of images produced from varying devices and environments. We design a network named Dual Branch LUT-aware Network (DualBLN), which innovatively introduces the information representing the pixel value mapping in 3D LUT into the real-time retouching procedure, learning the adaptive weights and the multiple 3D LUTs with strong representation capability. Specifically, we elaborate two branches to conduct feature extraction of the raw image and 3D LUTs, which consider the detail of the dual inputs, then generate the precise weights to fuse LUTs for graceful retouching. To integrate the information of the image and the 3D LUTs meticulously, we adopt bilinear pooling to alleviate the conflict between both features when fusing them from two branches of the network, which avoids the feature distortion caused by concatenation or summation directly. In addition, to meet the needs of portrait photo retouching (PPR) for portrait area priority (HRP), we propose Human Region Mask Generator (HRMG) module to solve the shortcomings of the current PPR task that requires the manual marking of portrait region, which achieve the PPR task more conveniently and efficiently. Extensive experiments verify the effectiveness of our method, e.g., the PSNR metric receives over 0.6 gains on high-resolution and 1.4 gains on the human-centered region at most compared with prior state-of-art works.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
Real-Time Portrait Stylization on the Edge

Yanyu Li,Xuan Shen,Geng Yuan,Jiexiong Guan,Wei Niu,Hao Tang,Bin Ren,Yanzhi Wang

DOI: https://doi.org/10.48550/arXiv.2206.01244

2022-06-03

Abstract:In this work we demonstrate real-time portrait stylization, specifically, translating self-portrait into cartoon or anime style on mobile devices. We propose a latency-driven differentiable architecture search method, maintaining realistic generative quality. With our framework, we obtain $10\times$ computation reduction on the generative model and achieve real-time video stylization on off-the-shelf smartphone using mobile GPUs.

Computer Vision and Pattern Recognition,Image and Video Processing
MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation

Han Yang,Sotiris Anagnostidis,Enis Simsar,Thomas Hofmann

2024-11-07

Abstract:We propose MegaPortrait. It's an innovative system for creating personalized portrait images in computer vision. It has three modules: Identity Net, Shading Net, and Harmonization Net. Identity Net generates learned identity using a customized model fine-tuned with source images. Shading Net re-renders portraits using extracted representations. Harmonization Net fuses pasted faces and the reference image's body for coherent results. Our approach with off-the-shelf Controlnets is better than state-of-the-art AI portrait products in identity preservation and image fidelity. MegaPortrait has a simple but effective design and we compare it with other methods and products to show its superiority.

Computer Vision and Pattern Recognition
MyPortrait: Morphable Prior-Guided Personalized Portrait Generation

Bo Ding,Zhenfeng Fan,Shuang Yang,Shihong Xia

2023-12-05

Abstract:Generating realistic talking faces is an interesting and long-standing topic in the field of computer vision. Although significant progress has been made, it is still challenging to generate high-quality dynamic faces with personalized details. This is mainly due to the inability of the general model to represent personalized details and the generalization problem to unseen controllable parameters. In this work, we propose Myportrait, a simple, general, and flexible framework for neural portrait generation. We incorporate personalized prior in a monocular video and morphable prior in 3D face morphable space for generating personalized details under novel controllable parameters. Our proposed framework supports both video-driven and audio-driven face animation given a monocular video of a single person. Distinguished by whether the test data is sent to training or not, our method provides a real-time online version and a high-quality offline version. Comprehensive experiments in various metrics demonstrate the superior performance of our method over the state-of-the-art methods. The code will be publicly available.

Computer Vision and Pattern Recognition
Portrait Segmentation Using Ensemble of Heterogeneous Deep-Learning Models

Yong-Woon Kim,Yung-Cheol Byun,Addapalli V. N. Krishna

DOI: https://doi.org/10.3390/e23020197

IF: 2.738

2021-02-05

Entropy

Abstract:Image segmentation plays a central role in a broad range of applications, such as medical image analysis, autonomous vehicles, video surveillance and augmented reality. Portrait segmentation, which is a subset of semantic image segmentation, is widely used as a preprocessing step in multiple applications such as security systems, entertainment applications, video conferences, etc. A substantial amount of deep learning-based portrait segmentation approaches have been developed, since the performance and accuracy of semantic image segmentation have improved significantly due to the recent introduction of deep learning technology. However, these approaches are limited to a single portrait segmentation model. In this paper, we propose a novel approach using an ensemble method by combining multiple heterogeneous deep-learning based portrait segmentation models to improve the segmentation performance. The Two-Models ensemble and Three-Models ensemble, using a simple soft voting method and weighted soft voting method, were experimented. Intersection over Union (IoU) metric, IoU standard deviation and false prediction rate were used to evaluate the performance. Cost efficiency was calculated to analyze the efficiency of segmentation. The experiment results show that the proposed ensemble approach can perform with higher accuracy and lower errors than single deep-learning-based portrait segmentation models. The results also show that the ensemble of deep-learning models typically increases the use of memory and computing power, although it also shows that the ensemble of deep-learning models can perform more efficiently than a single model with higher accuracy using less memory and less computing power.

physics, multidisciplinary
RefFaceNet: Reference-based Face Image Generation from Line Art Drawings

Shukai Wu,Weiming Liu,Qingqin Wang,Sanyuan Zhang,Zhenjie Hong,Shuchang Xu

DOI: https://doi.org/10.1016/j.neucom.2022.02.075

IF: 6

2022-01-01

Neurocomputing

Abstract:The flexible artistic creation of painting using deep neural networks has attracted lots of attention recently. Existing image-to-image translation approaches show powerful capabilities in producing photos between various domains. However, little attention has been paid to reference - based and line-based tasks simultaneously. In this paper, we introduce a novel network, RefFaceNet, to synthesize face portraits by utilizing reference face photos and line art drawings that only consist of the outlines of principal facial components. Our model can provide those people who have no experience in painting with the freedom to create. We utilize two separate encoders to focus on learning better feature representations for the line domain and the reference domain. An Attention-based Face Transfer Module composed of several sub-modules is built to capture the spatial correspondence between the two types of features. To construct a more robust encoding ability for our generator, we first learn a direct mapping from the line drawings to their ground truth color images and then take some distortions on the reference examples. With the assistance of optimal transport, we further propose to keep Spatial Distortion Consistency between the reference pictures in different geometric shapes by aligning features in high-dimensional space. A series of experiments have been conducted, and the results demonstrate the superiority of our method in generating more pleasing images compared with state-of-the-art models.
High-Quality Correspondence and Segmentation Estimation for Dual-Lens Smart-Phone Portraits

Xiaoyong Shen,Hongyun Gao,Xin Tao,Chao Zhou,Jiaya Jia

DOI: https://doi.org/10.48550/arXiv.1704.02205

2017-04-07

Abstract:Estimating correspondence between two images and extracting the foreground object are two challenges in computer vision. With dual-lens smart phones, such as iPhone 7Plus and Huawei P9, coming into the market, two images of slightly different views provide us new information to unify the two topics. We propose a joint method to tackle them simultaneously via a joint fully connected conditional random field (CRF) framework. The regional correspondence is used to handle textureless regions in matching and make our CRF system computationally efficient. Our method is evaluated over 2,000 new image pairs, and produces promising results on challenging portrait images.

Computer Vision and Pattern Recognition
Fast Deep Matting for Portrait Animation on Mobile Phone

Bingke Zhu,Yingying Chen,Jinqiao Wang,Si Liu,Bo Zhang,Ming Tang

DOI: https://doi.org/10.1145/3123266.3123286

2017-10-19

Abstract:Image matting plays an important role in image and video editing. However, the formulation of image matting is inherently ill-posed. Traditional methods usually employ interaction to deal with the image matting problem with trimaps and strokes, and cannot run on the mobile phone in real-time. In this paper, we propose a real-time automatic deep matting approach for mobile devices. By leveraging the densely connected blocks and the dilated convolution, a light full convolutional network is designed to predict a coarse binary mask for portrait image. And a feathering block, which is edge-preserving and matting adaptive, is further developed to learn the guided filter and transform the binary mask into alpha matte. Finally, an automatic portrait animation system based on fast deep matting is built on mobile devices, which does not need any interaction and can realize real-time matting with 15 fps. The experiments show that the proposed approach achieves comparable results with the state-of-the-art matting solvers.
Real-time deep hair matting on mobile devices

Alex Levinshtein,Cheng Chang,Edmund Phung,Irina Kezele,Wenzhangzhi Guo,Parham Aarabi

DOI: https://doi.org/10.48550/arXiv.1712.07168

2018-01-11

Abstract:Augmented reality is an emerging technology in many application domains. Among them is the beauty industry, where live virtual try-on of beauty products is of great importance. In this paper, we address the problem of live hair color augmentation. To achieve this goal, hair needs to be segmented quickly and accurately. We show how a modified MobileNet CNN architecture can be used to segment the hair in real-time. Instead of training this network using large amounts of accurate segmentation data, which is difficult to obtain, we use crowd sourced hair segmentation data. While such data is much simpler to obtain, the segmentations there are noisy and coarse. Despite this, we show how our system can produce accurate and fine-detailed hair mattes, while running at over 30 fps on an iPad Pro tablet.

Computer Vision and Pattern Recognition
AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

Huawei Wei,Zejun Yang,Zhisheng Wang

2024-03-26

Abstract:In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at

Computer Vision and Pattern Recognition,Graphics,Image and Video Processing
Temporal consistent portrait video segmentation

Yifan Wang,Wenbo Zhang,Lijun Wang,Fenghua Yang,Huchuan Lu

DOI: https://doi.org/10.1016/j.patcog.2021.108143

IF: 8

2021-12-01

Pattern Recognition

Abstract:<p>We explore a new video segmentation task, named portrait video segmentation (PVS), which aims to automatically segment the dominant person throughout a given portrait video. To achieve accurate and temporal-coherent segmentation results, a feature reconstruction based PVS method is developed under the meta-learning framework. Due to the dramatic pose variation and severe occlusion in portrait videos, feature reconstruction using existing optical flow models usually suffers from severe ghosting effects in reconstructed features. We mitigate this issue by presenting a soft correspondence network (SCN), which learns to facilitate feature reconstruction in an unsupervised fashion by softly assigning each pixel displacement probabilities between portrait frames. Based on the proposed SCN, a novel portrait segmentation network (PSN) is further designed, which explores the reconstructed features through feature aggregation blocks (FABs), yielding more reliable segmentation results. To capture temporal and target-specific cues, the parameters of FABs are determined by a meta-updater network which is trained offline in the meta-level. In addition, we introduce a new PVS dataset with high-quality segmentation annotations. Experimental results clearly demonstrate the effectiveness of the proposed PVS method.</p>

computer science, artificial intelligence,engineering, electrical & electronic
An Open-Source Project for Real-Time Image Semantic Segmentation

Quan Zhou,Yu Wang,Jia Liu,Xin Jin,Longin Jan Latecki

DOI: https://doi.org/10.1007/s11432-019-2685-1

2019-01-01

Science China Information Sciences

Abstract:>Recent years have witnessed great progress of deep convolutional neural networks (DCNNs) for solving scene understanding tasks [1–3]. These advances prefer to construct deeper and larger net-
Semantic Portrait Color Transfer with Internet Images

Yue Yang,Hanli Zhao,Lihua You,Renlong Tu,Xueyi Wu,Xiaogang Jin

DOI: https://doi.org/10.1007/s11042-015-3063-x

IF: 2.577

2015-01-01

Multimedia Tools and Applications

Abstract:We present a novel color transfer method for portraits by exploring their high-level semantic information. First, a database is set up which consists of a collection of portrait images download from the Internet, and each of them is manually segmented using image matting as a preprocessing step. Second, we search the database using Face++ to find the images with similar poses to a given source portrait image, and choose one satisfactory image from the results as the target. Third, we extract portrait foregrounds from both source and target images. Then, the system extracts the semantic information, such as faces, eyes, eyebrows, lips, teeth, etc., from the extracted foreground of the source using image matting algorithms. After that, we perform color transfer between corresponding parts with the same semantic information. We get the final transferred result by seamlessly compositing different parts together using alpha blending. Experimental results show that our semantics-driven approach can generate better color transfer results for portraits than previous methods and provide users a new means to retouch their portraits.

PortraitNet: Real-time portrait segmentation network for mobile device

A Scalable Real-time Semantic Segmentation Network for Autonomous Driving

Multi-objects Real Time Recognition Based on Color Information

Portrait Segmentation Using Deep Learning

Feature-Based Automatic Portrait Generation System

MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices

Towards Real-Time Automatic Portrait Matting on Mobile Devices

Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation

Real-time portrait image retouching extended from DualBLN

Real-Time Portrait Stylization on the Edge

MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation

MyPortrait: Morphable Prior-Guided Personalized Portrait Generation

Portrait Segmentation Using Ensemble of Heterogeneous Deep-Learning Models

RefFaceNet: Reference-based Face Image Generation from Line Art Drawings

High-Quality Correspondence and Segmentation Estimation for Dual-Lens Smart-Phone Portraits

Fast Deep Matting for Portrait Animation on Mobile Phone

Real-time deep hair matting on mobile devices

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

Temporal consistent portrait video segmentation

An Open-Source Project for Real-Time Image Semantic Segmentation

Semantic Portrait Color Transfer with Internet Images