Abstract:Product attribute value extraction is a pivotal component in Natural Language Processing (NLP) and the contemporary e-commerce industry. The provision of precise product attribute values is fundamental in ensuring high-quality recommendations and enhancing customer satisfaction. The recently emerging Large Language Models (LLMs) have demonstrated state-of-the-art performance in numerous attribute extraction tasks, without the need for domain-specific training data. Nevertheless, varying strengths and weaknesses are exhibited by different LLMs due to the diversity in data, architectures, and hyperparameters. This variation makes them complementary to each other, with no single LLM dominating all others. Considering the diverse strengths and weaknesses of LLMs, it becomes necessary to develop an ensemble method that leverages their complementary potentials. In this paper, we propose a novel algorithm called LLM-ensemble to ensemble different LLMs' outputs for attribute value extraction. We iteratively learn the weights for different LLMs to aggregate the labels with weights to predict the final attribute value. Not only can our proposed method be proven theoretically optimal, but it also ensures efficient computation, fast convergence, and safe deployment. We have also conducted extensive experiments with various state-of-the-art LLMs, including Llama2-13B, Llama2-70B, PaLM-2, GPT-3.5, and GPT-4, on Walmart's internal data. Our offline metrics demonstrate that the LLM-ensemble method outperforms all the state-of-the-art single LLMs on Walmart's internal dataset. This method has been launched in several production models, leading to improved Gross Merchandise Volume (GMV), Click-Through Rate (CTR), Conversion Rate (CVR), and Add-to-Cart Rate (ATC).

What problem does this paper attempt to address?

The paper primarily addresses the issue of extracting product attribute values in the field of e-commerce. Specifically, the research team proposed a new algorithm called LLM-ensemble, which aims to optimize the process of extracting product attribute values by integrating different large language models (LLMs). This method leverages the complementary advantages of different LLMs to improve the accuracy of attribute value extraction. Key points in the paper include: 1. **Problem Background**: Accurate product attribute values are crucial for improving the quality of recommendation systems and enhancing customer satisfaction. However, existing methods often perform poorly when dealing with internal data from e-commerce platforms. 2. **Application of Large Language Models**: Recently emerged LLMs (such as Llama, the GPT series, and PaLM) have shown excellent performance in various attribute extraction tasks without the need for domain-specific training data. However, due to differences in their data sources, architecture designs, and hyperparameters, each of these models has its own strengths and weaknesses. 3. **LLM-ensemble Algorithm**: To address the issue that a single LLM cannot perform well in all tasks, the researchers developed the LLM-ensemble algorithm. This algorithm is based on the Dawid-Skene model and iteratively learns to assign weights to different LLMs, thereby aggregating the outputs of these models to predict the final product attribute values. This method is not only theoretically optimal but also computationally efficient, has a fast convergence rate, and ensures deployment safety. 4. **Experimental Results**: Through comparative experiments, the algorithm outperformed all individual LLMs and traditional baseline models on Walmart's internal dataset. Additionally, this method significantly improved several key metrics of the recommendation system in online A/B testing, such as Gross Merchandise Value (GMV), Click-Through Rate (CTR), Conversion Rate (CVR), and Add-to-Cart Rate (ATC). In summary, this research addresses the critical challenge of accurately extracting product attribute values in the e-commerce domain by proposing a novel LLM integration method, which has already proven its effectiveness in practical applications.

LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extraction

Investigating LLM Applications in E-Commerce

Using LLMs for the Extraction and Normalization of Product Attribute Values

Leveraging Large Language Models to Enhance Personalized Recommendations in E-commerce

Enhanced E-Commerce Attribute Extraction: Innovating with Decorative Relation Correction and LLAMA 2.0-Based Annotation

Leveraging Large Language Models for Enhanced Product Descriptions in eCommerce

Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models

LiLiuM: eBay's Large Language Models for e-commerce

LLaSA: Large Language and E-Commerce Shopping Assistant

ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extraction

Emerging Synergies Between Large Language Models and Machine Learning in Ecommerce Recommendations

eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data

An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commerce

LLaMA-E: Empowering E-commerce Authoring with Object-Interleaved Instruction Following

Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs

Exploring Large Language Models for Product Attribute Value Identification

Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study

Enabling Explainable Recommendation in E-commerce with LLM-powered Product Knowledge Graph

Llms-Based Machine Translation for E-Commerce

Large Language Models for Market Research: A Data-augmentation Approach

V$^2$L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval