Abstract:Attributes are semantically meaningful characteristics whose applicability widely crosses category boundaries. They are particularly important in describing and recognizing concepts for which no explicit training example is given, e.g., zero-shot learning. Additionally, since attributes are human describable, they can be used for efficient human-computer interaction. In this article, we propose to employ semantic segmentation to improve person-related attribute prediction. The core idea lies in the fact that many attributes describe local properties. In other words, the probability of an attribute to appear in an image is far from being uniform in the spatial domain. We build our attribute prediction model jointly with a deep semantic segmentation network. This harnesses the localization cues learned by the semantic segmentation to guide the attention of the attribute prediction to the regions where different attributes naturally show up. As a result of this approach, in addition to prediction, we are able to localize the attributes despite merely having access to image-level labels (weak supervision) during training. We first propose semantic segmentation-based pooling and gating, respectively denoted as SSP and SSG. In the former, the estimated segmentation masks are used to pool the final activations of the attribute prediction network, from multiple semantically homogeneous regions. This is in contrast to global average pooling which is agnostic with respect to where in the spatial domain activations occur. In SSG, the same idea is applied to the intermediate layers of the network. Specifically, we create multiple copies of the internal activations. In each copy, only values that fall within a certain semantic region are preserved while outside of that, activations are suppressed. This mechanism allows us to prevent pooling operation from blending activations that are associated with semantically different regions. SSP and SSG, while effective, impose heavy memory utilization since each channel of the activations is pooled/gated with all the semantic segmentation masks. To circumvent this, we propose Symbiotic Augmentation (SA), where we learn only one mask per activation channel. SA allows the model to either pick one, or combine (weighted superposition) multiple semantic maps, in order to generate the proper mask for each channel. SA simultaneously applies the same mechanism to the reverse problem by leveraging output logits of attribute prediction to guide the semantic segmentation task. We evaluate our proposed methods for facial attributes on CelebA and LFWA datasets, while benchmarking WIDER Attribute and Berkeley Attributes of People for whole body attributes. Our proposed methods achieve superior results compared to the previous works. Furthermore, we show that in the reverse problem, semantic face parsing significantly improves when its associated task is jointly learned, through our proposed Symbiotic Augmentation (SA), with facial attribute prediction. We confirm that when few training instances are available, indeed image-level facial attribute labels can serve as an effective source of weak supervision to improve semantic face parsing. That reaffirms the need to jointly model these two interconnected tasks.

Automatic labeling of facial zones for digital clinical application: An ensemble of semantic segmentation models

When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

On Symbiosis of Attribute Prediction and Semantic Segmentation

Symmetry-Aware Semantic Segmentation of a 3D Human Body Model

Automatic Location and Semantic Labeling of Landmarks on 3D Human Body Models

Clinically Applicable System For 3D Teeth Segmentation in Intraoral Scans using Deep Learning

Fully automatic 3D facial expression recognition using a region-based approach

Developing an Artificial Intelligence (A.I)-based descriptor of facial appearance that fits with the assessments of makeup experts

Craniomaxillofacial Bone Segmentation and Landmark Detection Using Semantic Segmentation Networks and an Unbiased Heatmap.

Few-Shot Medical Image Segmentation via Generating Multiple Representative Descriptors

Pixel-Level Deep Segmentation: Artificial Intelligence Quantifies Muscle on Computed Tomography for Body Morphometric Analysis

SimSAM: Zero-shot Medical Image Segmentation via Simulated Interaction

Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation

Automatic Facial Skin Feature Detection for Everyone

Non-semantic Facial Parts for Face Verification.

Synthetic Instance Segmentation from Semantic Image Segmentation Masks

AGSAM: Agent-Guided Segment Anything Model for Automatic Segmentation in Few-Shot Scenarios

autoSMIM: Automatic Superpixel-Based Masked Image Modeling for Skin Lesion Segmentation

A semantic segmentation algorithm for fashion images based on modified mask RCNN

Diagnostics in Semantic Segmentation

Automatic Segmentation of Head-and-Shoulder Images by Combining Edge Feature and Shape Prior