The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA

Yiming Li,Zhao Zhang

2024-02-28

Abstract:Conversational multi-doc question answering aims to answer specific questions based on the retrieved documents as well as the contextual conversations. In this paper, we introduce our winning approach for the "Conversational Multi-Doc QA" challenge in WSDM Cup 2024, which exploits the superior natural language understanding and generation capability of Large Language Models (LLMs). We first adapt LLMs to the task, then devise a hybrid training strategy to make the most of in-domain unlabeled data. Moreover, an advanced text embedding model is adopted to filter out potentially irrelevant documents and several approaches are designed and compared for the model ensemble. Equipped with all these techniques, our solution finally ranked 1st place in WSDM Cup 2024, surpassing its rivals to a large extent. The source codes have been released at

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges in Conversational Multi - Doc QA. Specifically, the researchers hope to develop a method that can accurately answer specific questions based on multiple retrieved documents and the conversation context. The main difficulties of this task include: 1. **Acquisition of up - to - date knowledge**: Up - to - date knowledge of current or popular topics is usually unavailable during the training phase of language models. 2. **Handling a large amount of input information**: Although providing multiple relevant documents as context information seems feasible, the model may be confused or misled by a large amount of input. 3. **Interference from irrelevant documents**: How to filter out potentially irrelevant documents to ensure that the model reasons only based on high - quality reference documents. To solve these problems, the authors proposed a method using large language models (LLMs) and made improvements through the following technical means: - **Adaptive adjustment**: Apply LLMs to multi - round conditional generation tasks. - **Mixed training strategy**: Combine unlabeled data for multi - stage mixed training to make full use of unlabeled data in the domain. - **Advanced text embedding model**: Use advanced embedding models such as Nomic Embed to calculate the similarity score between the input and documents, thereby filtering out potentially irrelevant documents. - **Model integration**: Design and compare multiple model integration methods to evaluate the quality of answers generated by different LLMs and select the best response as the final answer. These techniques enabled their solution to achieve the first place in the WSDM Cup 2024 competition, significantly outperforming other participating teams.

The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA

Leveraging Large Language Models for Multiple Choice Question Answering

Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering

LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models

LibriSQA: Advancing Free-form and Open-ended Spoken Question Answering with a Novel Dataset and Framework.

Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering

Spoken Language Intelligence of Large Language Models for Language Learning

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

Integrating UMLS Knowledge into Large Language Models for Medical Question Answering

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering

Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training

Evaluation Methodology for Large Language Models for Multilingual Document Question and Answer

LMDX: Language Model-based Document Information Extraction and Localization

Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark

Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering

An Interactive Multi-modal Query Answering System with Retrieval-Augmented Large Language Models

LLM×MapReduce: Simplified Long-Sequence Processing Using Large Language Models