M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

Jaemin Cho,Debanjan Mahata,Ozan Irsoy,Yujie He,Mohit Bansal

2024-11-08

Abstract:Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

This paper attempts to address the limitations of existing methods in handling long documents and complex visual information in multi-page, multi-document understanding. Specifically:

1. **Cross-page or cross-document information needs**: Existing multimodal language models (MLMs) struggle to handle information across multiple pages or documents, while text retrieval-based generative models (such as RAG) ignore the visual elements within documents.
2. **Handling complex visual formats**: Many documents contain complex visual formats, such as tables, charts, and mixed layouts, but existing text extraction tools (such as OCR) fail to capture these details, leading to incomplete or inaccurate document interpretation.

To address these issues, the authors introduce M3D OCRAG (Multimodal Multi-page Multi-document Retrieval Augmented Generation Framework), which can flexibly adapt to different document contexts (closed domain and open domain), question hops (single-hop and multi-hop), and evidence modalities (text, charts, images, etc.). M3D OCRAG finds relevant document pages through a multimodal retrieval model and generates answers using a multimodal language model, efficiently handling single or multiple documents while preserving visual information.

Additionally, the authors propose a new benchmark dataset, M3D OCVQA, for evaluating open-domain multimodal multi-page multi-document understanding tasks. This dataset contains over 3,000 PDF documents, totaling more than 40,000 pages, covering various modalities such as text, images, and tables, better reflecting the challenges of real-world document understanding.