Abstract:In this paper, we present effective algorithms to automatically annotate clothes from social media data, such as Facebook and Instagram. Clothing annotation can be informally stated as recognizing, as accurately as possible, the garment items appearing in the query photo. This task brings huge opportunities for recommender and e-commerce systems, such as capturing new fashion trends based on which clothes have been used more recently. It also poses interesting challenges for existing vision and recognition algorithms, such as distinguishing between similar but different types of clothes or identifying a pattern of a cloth even if it has different colors and shapes. We formulate the annotation task as a multi-label and multi-modal classification problem: (i) both image and textual content (i.e., tags about the image) are available for learning classifiers, (ii) the classifiers must recognize a set of labels (i.e., a set of garment items), and (iii) the decision on which labels to assign to the query photo comes from a set of instances that is used to build a function, which separates labels that should be assigned to the query photo, from those that should not be assigned. Using this configuration, we propose two approaches: (i) the pointwise one, called MMCA, which receives a single image as input, and (ii) a multi-instance classification, called M3CA, also known as pairwise approach, which uses pair of images to create the classifiers. We conducted a systematic evaluation of the proposed algorithms using everyday photos collected from two major fashion-related social media, namely pose.com and chictopia.com. Our results show that the proposed approaches provide improvements when compared to popular first choice multi-label, multi-modal, multi-instance algorithms that range from 20 % to 30 % in terms of accuracy.

Clothing Matching Based on Multi-modal Data.

Learning compatibility knowledge for outfit recommendation with complementary clothing matching

NeuroStylist: Neural Compatibility Modeling for Clothing Matching.

Fashion Recommendation on Street Images.

Towards Better Understanding the Clothing Fashion Styles: A Multimodal Deep Learning Approach

Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences

Combo-Fashion: Fashion Clothes Matching CTR Prediction with Item History

Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network

How to Wear Beautifully? Clothing Pair Recommendation

Research on type-aware fashion compatibility prediction based on a hybrid attention mechanism

Learning Fashion Compatibility with Bidirectional LSTMs

End-to-end multi-modal product matching in fashion e-commerce

Interpretable Fashion Matching with Rich Attributes

Intelligent Clothing Matching Based on Feature Analysis

Learning Tuple Compatibility for Conditional OutfitRecommendation

Complete the Look: Scene-based Complementary Product Recommendation

Smart Fitting Room: A One-stop Framework for Matching-aware Virtual Try-on

Theme-Matters: Fashion Compatibility Learning via Theme Attention

Pointwise and pairwise clothing annotation: combining features from social media

Outfit compatibility prediction with multi-layered feature fusion network

Personalized Capsule Wardrobe Creation with Garment and User Modeling