Towards Generalist Biomedical AI

Tao Tu,Shekoofeh Azizi,Danny Driess,Mike Schaekermann,Mohamed Amin,Pi-Chuan Chang,Andrew Carroll,Chuck Lau,Ryutaro Tanno,Ira Ktena,Basil Mustafa,Aakanksha Chowdhery,Yun Liu,Simon Kornblith,David Fleet,Philip Mansfield,Sushant Prakash,Renee Wong,Sunny Virmani,Christopher Semturs,S Sara Mahdavi,Bradley Green,Ewa Dominowska,Blaise Aguera y Arcas,Joelle Barral,Dale Webster,Greg S. Corrado,Yossi Matias,Karan Singhal,Pete Florence,Alan Karthikesalingam,Vivek Natarajan

2023-07-27

Abstract:Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system. Med-PaLM M is a large multimodal generative model that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. Med-PaLM M reaches performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. We also report examples of zero-shot generalization to novel medical concepts and tasks, positive transfer learning across tasks, and emergent zero-shot medical reasoning. To further probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist evaluation of model-generated (and human) chest X-ray reports and observe encouraging performance across model scales. In a side-by-side ranking on 246 retrospective chest X-rays, clinicians express a pairwise preference for Med-PaLM M reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.

Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to develop a general artificial intelligence system capable of handling multiple biomedical data modalities, such as text, imaging, and genomics. To achieve this goal, the research team first created a new multimodal biomedical benchmark dataset called MultiMedBench. This benchmark includes 14 different tasks, such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant identification.

The paper introduces Med-PaLM Multimodal (Med-PaLM M for short), a proof-of-concept project that demonstrates the potential of a general biomedical AI system. Med-PaLM M is a large multimodal generative model capable of flexibly encoding and interpreting clinical language, imaging, and genomic data. It performs better than or on par with the current state-of-the-art on all MultiMedBench tasks, and in some cases, significantly outperforms specialized models. Additionally, Med-PaLM M shows the ability to generalize to new medical concepts and tasks with zero-shot learning, as well as positive transfer learning and zero-shot medical reasoning across tasks.

To further evaluate the capabilities and limitations of Med-PaLM M, researchers conducted an assessment where radiologists evaluated the model-generated (and human-written) chest X-ray reports, and observed encouraging results. In a side-by-side ranking of 246 retrospective chest X-rays, clinicians preferred the reports generated by Med-PaLM M in up to 40.50% of cases, indicating its potential clinical application value. Although much work remains to validate the effectiveness of these models in practical applications, the research results represent an important milestone towards developing general biomedical AI systems.