Bora: Biomedical Generalist Video Generation Model

Weixiang Sun,Xiaocao You,Ruizhe Zheng,Zhengqing Yuan,Xiang Li,Lifang He,Quanzheng Li,Lichao Sun

2024-07-16

Abstract:Generative models hold promise for revolutionizing medical education, robot-assisted surgery, and data augmentation for medical AI development. Diffusion models can now generate realistic images from text prompts, while recent advancements have demonstrated their ability to create diverse, high-quality videos. However, these models often struggle with generating accurate representations of medical procedures and detailed anatomical structures. This paper introduces Bora, the first spatio-temporal diffusion probabilistic model designed for text-guided biomedical video generation. Bora leverages Transformer architecture and is pre-trained on general-purpose video generation tasks. It is fine-tuned through model alignment and instruction tuning using a newly established medical video corpus, which includes paired text-video data from various biomedical fields. To the best of our knowledge, this is the first attempt to establish such a comprehensive annotated biomedical video dataset. Bora is capable of generating high-quality video data across four distinct biomedical domains, adhering to medical expert standards and demonstrating consistency and diversity. This generalist video generative model holds significant potential for enhancing medical consultation and decision-making, particularly in resource-limited settings. Additionally, Bora could pave the way for immersive medical training and procedure planning. Extensive experiments on distinct medical modalities such as endoscopy, ultrasound, MRI, and cell tracking validate the effectiveness of our model in understanding biomedical instructions and its superior performance across subjects compared to state-of-the-art generation models.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that when existing generative models generate medical videos, they are often difficult to accurately represent medical procedures and detailed anatomical structures. Although diffusion models have been able to generate realistic images from text prompts, and recent developments have also demonstrated their ability to create diverse and high - quality videos, in the medical field, the performance of these models still needs to be improved. Therefore, this paper introduces a new spatio - temporal diffusion probability model - Bora, which is specifically used for text - guided biomedical video generation. Bora is based on the Transformer architecture and is aligned and instruction - tuned through a newly - established medical video corpus to generate high - quality video data that meets the standards of medical experts and shows consistency and diversity. This general - purpose video generation model has great potential in enhancing medical consultation and decision - making, especially in resource - limited environments, and can also pave the way for immersive medical training and surgical planning.