BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once

Theodore Zhao,Yu Gu,Jianwei Yang,Naoto Usuyama,Ho Hin Lee,Tristan Naumann,Jianfeng Gao,Angela Crabtree,Jacob Abel,Christine Moung-Wen,Brian Piening,Carlo Bifulco,Mu Wei,Hoifung Poon,Sheng Wang

2024-06-05

Abstract:Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem addressed in this paper is how to integrate and optimize segmentation, detection, and recognition tasks in biomedical image analysis to improve efficiency and accuracy.

BiomedParse is a biomedically-oriented basic model for all types of images, aiming to address the limitations of traditional approaches that handle these tasks separately by employing a unified framework to simultaneously perform segmentation, detection, and recognition. Traditional image analysis methods typically treat each subtask independently, focusing only on segmentation while ignoring semantic information in detection and recognition. BiomedParse combines these tasks by leveraging their interdependencies, such as the semantic labels of segmented objects.

The main innovations mentioned in the paper include:
1. Creation of a large dataset, BiomedParseData, consisting of more than six million image-segmentation-text description triplets. This dataset is generated by using standard segmentation datasets and natural language descriptions (cleaned and aligned with biomedical object ontologies using GPT-4).
2. BiomedParse does not require users to specify bounding boxes and can perform segmentation for all relevant objects based on textual cues. This reduces user workload and enables handling of objects with irregular shapes.
3. BiomedParse demonstrates superior performance compared to existing state-of-the-art methods across multiple imaging modes in tasks including image segmentation, object detection, and recognition.

In summary, BiomedParse is a comprehensive biomedical image analysis tool that addresses segmentation, detection, and recognition tasks simultaneously. It is applicable to various major biomedical image patterns and paves the way for efficient and accurate biomedical discoveries based on image analysis.