Improving Cross-lingual Representation for Semantic Retrieval with Code-switching

Mieradilijiang Maimaiti,Yuanhang Zheng,Ji Zhang,Fei Huang,Yue Zhang,Wenpei Luo,Kaiyu Huang

2024-03-03

Abstract:Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario. The demands for a cross-lingual smart-customer-service system for an e-commerce platform or some particular business conditions have been increasing recently. Most previous studies exploit cross-lingual pre-trained models (PTMs) for multi-lingual knowledge retrieval directly, while some others also leverage the continual pre-training before fine-tuning PTMs on the downstream tasks. However, no matter which schema is used, the previous work ignores to inform PTMs of some features of the downstream task, i.e. train their PTMs without providing any signals related to SR. To this end, in this work, we propose an Alternative Cross-lingual PTM for SR via code-switching. We are the first to utilize the code-switching approach for cross-lingual SR. Besides, we introduce the novel code-switched continual pre-training instead of directly using the PTMs on the SR tasks. The experimental results show that our proposed approach consistently outperforms the previous SOTA methods on SR and semantic textual similarity (STS) tasks with three business corpora and four open datasets in 20+ languages.

Computation and Language

What problem does this paper attempt to address?

This paper focuses on the problem of cross-lingual semantic retrieval (SR), especially in the context of intelligent customer service systems in e-commerce. Most current methods directly use pre-trained models (PTMs) for multilingual knowledge retrieval or continue pre-training on downstream tasks, but they overlook passing the SR-related features to PTMs. The paper proposes an innovative approach to improve cross-lingual PTM for SR by code-switching. They first generate code-switching data based on a bilingual dictionary, then pre-train the model using alternating language modeling loss (ALM) and similarity loss, and finally fine-tune the model on the SR corpus. Experimental results show that this approach outperforms existing state-of-the-art methods in SR and semantic text similarity tasks on multiple business corpora and open datasets. Additionally, this approach enhances the model's robustness in sentence-level SR tasks.