Abstract:Background: Mining the vast pool of biomedical literature to extract accurate responses and relevant references is challenging due to the domain's interdisciplinary nature, specialized jargon, and continuous evolution. Early natural language processing (NLP) approaches often led to incorrect answers as they failed to comprehend the nuances of natural language. However, transformer models have significantly advanced the field by enabling the creation of large language models (LLMs), enhancing question-answering (QA) tasks. Despite these advances, current LLM-based solutions for specialized domains like biology and biomedicine still struggle to generate up-to-date responses while avoiding "hallucination" or generating plausible but factually incorrect responses. Results: Our work focuses on enhancing prompts using a retrieval-augmented architecture to guide LLMs in generating meaningful responses for biomedical QA tasks. We evaluated two approaches: one relying on text embedding and vector similarity in a high-dimensional space, and our proposed method, which uses explicit signals in user queries to extract meaningful contexts. For robust evaluation, we tested these methods on 50 specific and challenging questions from diverse biomedical topics, comparing their performance against a baseline model, BM25. Retrieval performance of our method was significantly better than others, achieving a median Precision@10 of 0.95, which indicates the fraction of the top 10 retrieved chunks that are relevant. We used GPT-4, OpenAI's most advanced LLM to maximize the answer quality and manually accessed LLM-generated responses. Our method achieved a median answer quality score of 2.5, surpassing both the baseline model and the text embedding-based approach. We developed a QA bot, WeiseEule ( https://github.com/wasimaftab/WeiseEule-LocalHost ), which utilizes these methods for comparative analysis and also offers advanced features for review writing and identifying relevant articles for citation. Conclusions: Our findings highlight the importance of prompt enhancement methods that utilize explicit signals in user queries over traditional text embedding-based approaches to improve LLM-generated responses for specialized queries in specialized domains such as biology and biomedicine. By providing users complete control over the information fed into the LLM, our approach addresses some of the major drawbacks of existing web-based chatbots and LLM-based QA systems, including hallucinations and the generation of irrelevant or outdated responses.

Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval

Question answering systems for health professionals at the point of care -- a systematic review

Detect, Retrieve, Comprehend: A Flexible Framework for Zero-Shot Document-Level Question Answering

Question Answering for Electronic Health Records: A Scoping Review of datasets and models

Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence

What Would it Take to get Biomedical QA Systems into Practice?

Are my answers medically accurate? Exploiting medical knowledge graphs for medical question answering

Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering

Comparing Knowledge Sources for Open-Domain Scientific Claim Verification

Medical Question Answering for Clinical Decision Support

PubMedQA: A Dataset for Biomedical Research Question Answering

Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation

Learning to Answer Complex Questions with Evidence Graph.

A Health QA with Enhanced User Interfaces

Top K Relevant Passage Retrieval for Biomedical Question Answering

A Question-Entailment Approach to Question Answering

RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions

Optimizing biomedical information retrieval with a keyword frequency-driven prompt enhancement strategy

K-QA: A Real-World Medical Q&A Benchmark

Knowledge-Aided Open-Domain Question Answering