OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
Akari Asai,Jacqueline He,Rulin Shao,Weijia Shi,Amanpreet Singh,Joseph Chee Chang,Kyle Lo,Luca Soldaini,Sergey Feldman,Mike D'arcy,David Wadden,Matt Latzke,Minyang Tian,Pan Ji,Shengyan Liu,Hao Tong,Bohao Wu,Yanyu Xiong,Luke Zettlemoyer,Graham Neubig,Dan Weld,Doug Downey,Wen-tau Yih,Pang Wei Koh,Hannaneh Hajishirzi
2024-11-21
Abstract:Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.
Computation and Language,Artificial Intelligence,Information Retrieval,Machine Learning