Abstract:With the prosperity of e-commerce industry, various modalities, e.g., vision and language, are utilized to describe product items. It is an enormous challenge to understand such diversified data, especially via extracting the attribute-value pairs in text sequences with the aid of helpful image regions. Although a series of previous works have been dedicated to this task, there remain seldomly investigated obstacles that hinder further improvements: 1) Parameters from up-stream single-modal pretraining are inadequately applied, without proper jointly fine-tuning in a down-stream multi-modal task. 2) To select descriptive parts of images, a simple late fusion is widely applied, regardless of priori knowledge that language-related information should be encoded into a common linguistic embedding space by stronger encoders. 3) Due to diversity across products, their attribute sets tend to vary greatly, but current approaches predict with an unnecessary maximal range and lead to more potential false positives. To address these issues, we propose in this paper a novel approach to boost multi-modal e-commerce attribute value extraction via unified learning scheme and dynamic range minimization: 1) Firstly, a unified scheme is designed to jointly train a multi-modal task with pretrained single-modal parameters. 2) Secondly, a text-guided information range minimization method is proposed to adaptively encode descriptive parts of each modality into an identical space with a powerful pretrained linguistic model. 3) Moreover, a prototype-guided attribute range minimization method is proposed to first determine the proper attribute set of the current product, and then select prototypes to guide the prediction of the chosen attributes. Experiments on the popular multi-modal e-commerce benchmarks show that our approach achieves superior performance over the other state-of-the-art techniques.

TaoHighlight: Commodity-Aware Multi-Modal Video Highlight Detection in E-Commerce

Analyzing and Predicting Consumer Response to Short Videos in E-Commerce

Click-Through Rate Prediction with Multi-Modal Hypergraphs

Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Video Highlight Detection Via Deep Ranking Modeling

Comprehensive Information Integration Modeling Framework for Video Titling

Multi-Modal Representation Learning with Self-Adaptive Threshold for Commodity Verification

Detection of Occluded Small Commodities Based on Feature Enhancement under Super-Resolution

Cross-domain Attention Network with Wasserstein Regularizers for E-commerce Search

HighlightMe: Detecting Highlights from Human-Centric Videos

Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis

UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection

Three-Dimensional Attention-Based Deep Ranking Model for Video Highlight Detection

Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce

Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrieval

Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection

Highlight Detection With Pairwise Deep Ranking For First-Person Video Summarization

Video Highlights Detection and Summarization with Lag-Calibration based on Concept-Emotion Mapping of Crowd-sourced Time-Sync Comments

Video Highlight Detection Via Region-Based Deep Ranking Model

Boosting Multi-Modal E-commerce Attribute Value Extraction via Unified Learning Scheme and Dynamic Range Minimization