A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents

Norman Meuschke,Apurva Jagdale,Timo Spinde,Jelena Mitrović,Bela Gipp

DOI: https://doi.org/10.1007/978-3-031-28032-0_31

2023-03-17

Abstract:Extracting information from academic PDF documents is crucial for numerous indexing, retrieval, and analysis use cases. Choosing the best tool to extract specific content elements is difficult because many, technically diverse tools are available, but recent performance benchmarks are rare. Moreover, such benchmarks typically cover only a few content elements like header metadata or bibliographic references and use smaller datasets from specific academic disciplines. We provide a large and diverse evaluation framework that supports more extraction tasks than most related datasets. Our framework builds upon DocBank, a multi-domain dataset of 1.5M annotated content elements extracted from 500K pages of research papers on arXiv. Using the new framework, we benchmark ten freely available tools in extracting document metadata, bibliographic references, tables, and other content elements from academic PDF documents. GROBID achieves the best metadata and reference extraction results, followed by CERMINE and Science Parse. For table extraction, Adobe Extract outperforms other tools, even though the performance is much lower than for other content elements. All tools struggle to extract lists, footers, and equations. We conclude that more research on improving and combining tools is necessary to achieve satisfactory extraction quality for most content elements. Evaluation datasets and frameworks like the one we present support this line of research. We make our data and code publicly available to contribute toward this goal.

Information Retrieval

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the issue of extracting information from academic PDF documents and provides a comprehensive benchmarking framework to evaluate existing information extraction tools. Specifically, the paper attempts to solve the following aspects:

1. **Difficulty in selecting information extraction tools**: Due to the existence of various information extraction tools with different technologies, it becomes very difficult to choose the best tool to extract specific content elements.
2. **Insufficient existing benchmarks**: Current benchmarks usually cover only a small number of content elements (such as title metadata or references) and use small datasets limited to specific disciplines.
3. **Lack of recent benchmarks**: The latest comprehensive benchmarks were published in 2015 (metadata), 2017 (body text), and 2018 (references), and these benchmarks fail to reflect the latest technological advancements.

To alleviate this knowledge gap, the paper proposes a large-scale and diverse evaluation framework to assess ten state-of-the-art non-commercial information extraction tools, covering 500,000 pages of documents from various academic fields. Through this framework, the authors hope to identify the best tools for extracting specific content elements and to drive future research developments.