Abstract:$\textbf{Objectives}$: Large Language Models (LLMs) such as ChatGPT and Med-PaLM have excelled in various medical question-answering tasks. However, these English-centric models encounter challenges in non-English clinical settings, primarily due to limited clinical knowledge in respective languages, a consequence of imbalanced training corpora. We systematically evaluate LLMs in the Chinese medical context and develop a novel in-context learning framework to enhance their performance. $\textbf{Materials and Methods}$: The latest China National Medical Licensing Examination (CNMLE-2022) served as the benchmark. We collected 53 medical books and 381,149 medical questions to construct the medical knowledge base and question bank. The proposed Knowledge and Few-shot Enhancement In-context Learning (KFE) framework leverages the in-context learning ability of LLMs to integrate diverse external clinical knowledge sources. We evaluated KFE with ChatGPT(GPT3.5), GPT4, Baichuan2(BC2)-7B, and BC2-13B in CNMLE-2022 and investigated the effectiveness of different pathways for incorporating LLMs with medical knowledge from 7 perspectives. $\textbf{Results}$: Directly applying ChatGPT failed to qualify for the CNMLE-2022 at a score of 51. Cooperated with the KFE, the LLMs with varying sizes yielded consistent and significant improvements. The ChatGPT's performance surged to 70.04 and GPT-4 achieved the highest score of 82.59. This surpasses the qualification threshold (60) and exceeds the average human score of 68.70. It also enabled a smaller BC2-13B to pass the examination, showcasing the great potential in low-resource settings. $\textbf{Conclusion}$: By synergizing medical knowledge through in-context learning, LLM can extend clinical insight beyond language barriers, significantly reducing language-related disparities of LLM applications and ensuring global benefit in healthcare.

Large language model based framework for knowledgebase coverage and correctness using chatbot and human feedback

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Augmented Large Language Models with Parametric Knowledge Guiding

A Principled Framework for Knowledge-enhanced Large Language Model

Large Knowledge Model: Perspectives and Challenges

Supervised Knowledge Makes Large Language Models Better In-context Learners

Exploring Large Language Models for Knowledge Graph Completion

Can Language Models Act as Knowledge Bases at Scale?

Clue-Guided Path Exploration: Optimizing Knowledge Graph Retrieval with Large Language Models to Address the Information Black Box Challenge

A Platform for the Biomedical Application of Large Language Models

Chatbot-Based Ontology Interaction Using Large Language Models and Domain-Specific Standards

Knowledge Boundary of Large Language Models: A Survey

KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases

KBLaM: Knowledge Base augmented Language Model

Systematic Assessment of Factual Knowledge in Large Language Models

Large language models in health care: Development, applications, and challenges

Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases

Learnings from a Large-Scale Deployment of an LLM-Powered Expert-in-the-Loop Healthcare Chatbot

Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries

Enhancing Large Language Models with Knowledge Graphs for Robust Question Answering