Abstract:In this paper, we present effective algorithms to automatically annotate clothes from social media data, such as Facebook and Instagram. Clothing annotation can be informally stated as recognizing, as accurately as possible, the garment items appearing in the query photo. This task brings huge opportunities for recommender and e-commerce systems, such as capturing new fashion trends based on which clothes have been used more recently. It also poses interesting challenges for existing vision and recognition algorithms, such as distinguishing between similar but different types of clothes or identifying a pattern of a cloth even if it has different colors and shapes. We formulate the annotation task as a multi-label and multi-modal classification problem: (i) both image and textual content (i.e., tags about the image) are available for learning classifiers, (ii) the classifiers must recognize a set of labels (i.e., a set of garment items), and (iii) the decision on which labels to assign to the query photo comes from a set of instances that is used to build a function, which separates labels that should be assigned to the query photo, from those that should not be assigned. Using this configuration, we propose two approaches: (i) the pointwise one, called MMCA, which receives a single image as input, and (ii) a multi-instance classification, called M3CA, also known as pairwise approach, which uses pair of images to create the classifiers. We conducted a systematic evaluation of the proposed algorithms using everyday photos collected from two major fashion-related social media, namely pose.com and chictopia.com. Our results show that the proposed approaches provide improvements when compared to popular first choice multi-label, multi-modal, multi-instance algorithms that range from 20 % to 30 % in terms of accuracy.

Tagging the Shoe Images by Semantic Attributes.

Cross-Domain Shoe Retrieval with a Semantic Hierarchy of Attribute Classification Network.

Fashion Recommendation on Street Images.

Automatic Measurement of Garment Sizes Using Image Recognition

Fashion Analysis With A Subordinate Attribute Classification Network

ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model

Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging

DeepShoe: an Improved Multi-Task View-invariant CNN for Street-to-shop Shoe Retrieval.

Cross-domain Shoe Retrieval Using a Three-Level Deep Feature Representation

Semi-automatic Dynamic Auxiliary-Tag-aided Image Annotation

Bridging the Semantic Gap Between Image Contents and Tags

An Approach for Clothing Recommendation Based on Multiple Image Attributes

Discovering Attribute Shades of Meaning with the Crowd

A Deep-Learning-Based Fashion Attributes Detection Model

Learning to Tag.

Decoding the fashion trend of sports shoes with empowered computer vision

Searching for Apparel Products from Images in the Wild

Semantic Classification Based Clothing Image Retrieval

Progressive Fashion Attribute Extraction

A Federated Approach for Fine-Grained Classification of Fashion Apparel

Pointwise and pairwise clothing annotation: combining features from social media