Abstract:At Pinterest, we utilize image embeddings throughout our search and recommendation systems to help our users navigate through visual content by powering experiences like browsing of related content and searching for exact products for shopping. In this work we describe a multi-task deep metric learning system to learn a single unified image embedding which can be used to power our multiple visual search products. The solution we present not only allows us to train for multiple application objectives in a single deep neural network architecture, but takes advantage of correlated information in the combination of all training data from each application to generate a unified embedding that outperforms all specialized embeddings previously deployed for each product. We discuss the challenges of handling images from different domains such as camera photos, high quality web images, and clean product catalog images. We also detail how to jointly train for multiple product objectives and how to leverage both engagement data and human labeled data. In addition, our trained embeddings can also be binarized for efficient storage and retrieval without compromising precision and recall. Through comprehensive evaluations on offline metrics, user studies, and online A/B experiments, we demonstrate that our proposed unified embedding improves both relevance and engagement of our visual search products for both browsing and searching purposes when compared to existing specialized embeddings. Finally, the deployment of the unified embedding at Pinterest has drastically reduced the operation and engineering cost of maintaining multiple embeddings while improving quality.

Visual Search at Pinterest

Shop The Look: Building a Large Scale Visual Shopping System at Pinterest

Visual Search at eBay

Visual Search at Alibaba

Learning a Unified Embedding for Visual Search at Pinterest

Towards Practical Visual Search Engine Within Elasticsearch.

A Visual Exploratory Search Engine Solution Based on Cloud Computing

Visual search over billions of aerial and satellite images

OmniSearchSage: Multi-Task Multi-Entity Embeddings for Pinterest Search

Efficient Indexing for Large Scale Visual Search

Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest

Improving Pinterest Search Relevance Using Large Language Models

Rethinking Personalized Ranking at Pinterest: An End-to-End Approach

TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest

Social-oriented Visual Image Search

When Amazon Meets Google

PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest

When Amazon Meets Google: Product Visualization by Exploring Multiple Web Sources

Virtual ID Discovery from E-commerce Media at Alibaba: Exploiting Richness of User Click Behavior for Visual Search Relevance

A Novel Deep Learning-Based Visual Search Engine in Digital Marketing for Tourism E-Commerce Platforms

Large-Scale Visual Search with Binary Distributed Graph at Alibaba