Abstract:Question Answering (QA) effectively evaluates language models' reasoning and knowledge depth. While QA datasets are plentiful in areas like general domain and biomedicine, academic chemistry is less explored. Chemical QA plays a crucial role in both education and research by effectively translating complex chemical information into readily understandable format. Addressing this gap, we introduce ScholarChemQA, a large-scale QA dataset constructed from chemical papers. This dataset reflects typical real-world challenges, including an imbalanced data distribution and a substantial amount of unlabeled data that can be potentially useful. Correspondingly, we introduce a QAMatch model, specifically designed to effectively answer chemical questions by fully leveraging our collected data. We first address the issue of imbalanced label distribution by re-weighting the instance-wise loss based on the inverse frequency of each class, ensuring minority classes are not dominated by majority ones during optimization. Next, we utilize the unlabeled data to enrich the learning process, generating a variety of augmentations based on a SoftMix operation and ensuring their predictions align with the same target, i.e., pseudo-labels. To ensure the quality of the pseudo-labels, we propose a calibration procedure aimed at closely aligning the pseudo-label estimates of individual samples with a desired ground truth distribution. Experiments show that our QAMatch significantly outperforms the recent similar-scale baselines and Large Language Models (LLMs) not only on our ScholarChemQA dataset but also on four benchmark datasets. We hope our benchmark and model can facilitate and promote more research on chemical QA.

Leveraging Text-to-Text Pretrained Language Models for Question Answering in Chemistry

Question Answering System for Chemistry

Marie and BERT-A Knowledge Graph Embedding Based Question Answering System for Chemistry

ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering

Training Question Answering Models From Synthetic Data

Retrieval Augmented Generation for Domain-specific Question Answering

Answering Science Exam Questions Using Query Rewriting with Background Knowledge

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

Efficient Medical Question Answering with Knowledge-Augmented Question Generation

SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA

MetaQA: Enhancing human-centered data search using Generative Pre-trained Transformer (GPT) language model and artificial intelligence

Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models

XAIQA: Explainer-Based Data Augmentation for Extractive Question Answering

Medical Data Inquiry Using a Question Answering Model.

Question Answering models for information extraction from perovskite materials science literature

KaPQA: Knowledge-Augmented Product Question-Answering

Long-Tailed Question Answering in an Open World.

Fluent Response Generation for Conversational Question Answering

A medical question answering system using large language models and knowledge graphs

KBQA: Learning Question Answering over QA Corpora and Knowledge Bases