μ-Bench: A Vision-Language Benchmark for Microscopy Understanding

Alejandro Lozano,Jeffrey Nirschl,James Burgess,Sanket Rajan Gupte,Yuhui Zhang,Alyssa Unell,Serena Yeung-Levy

2024-07-02

Abstract:Recent advances in microscopy have enabled the rapid generation of terabytes of image data in cell biology and biomedical research. Vision-language models (VLMs) offer a promising solution for large-scale biological image analysis, enhancing researchers' efficiency, identifying new image biomarkers, and accelerating hypothesis generation and scientific discovery. However, there is a lack of standardized, diverse, and large-scale vision-language benchmarks to evaluate VLMs' perception and cognition capabilities in biological image understanding. To address this gap, we introduce {\mu}-Bench, an expert-curated benchmark encompassing 22 biomedical tasks across various scientific disciplines (biology, pathology), microscopy modalities (electron, fluorescence, light), scales (subcellular, cellular, tissue), and organisms in both normal and abnormal states. We evaluate state-of-the-art biomedical, pathology, and general VLMs on {\mu}-Bench and find that: i) current models struggle on all categories, even for basic tasks such as distinguishing microscopy modalities; ii) current specialist models fine-tuned on biomedical data often perform worse than generalist models; iii) fine-tuning in specific microscopy domains can cause catastrophic forgetting, eroding prior biomedical knowledge encoded in their base model. iv) weight interpolation between fine-tuned and pre-trained models offers one solution to forgetting and improves general performance across biomedical tasks. We release {\mu}-Bench under a permissive license to accelerate the research and development of microscopy foundation models.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper introduces µ-Bench, a benchmark for biomedical image understanding and visual language, aiming to address the challenges in large-scale biomedical image analysis, particularly in the understanding of microscope images. Currently, although the advancement of microscopy technology has generated a large amount of image data, there is a lack of standardized, diverse, and large-scale visual language benchmarks to evaluate the perception and cognitive abilities of models in biomedical image understanding.

µ-Bench consists of 22 biomedical tasks spanning across multiple scientific disciplines, covering various microscopy techniques such as electron microscopy, fluorescence microscopy, and light microscopy, as well as different scales from subcellular to tissue. The paper mentions that existing models struggle on all categories, even on basic tasks such as differentiating microscope types. Furthermore, models specifically fine-tuned for biomedical data often perform worse than general models and may lead to catastrophic forgetting, i.e., the loss of prior biomedical knowledge encoded in the base model.

The study also found that fine-tuning in specific microscope domains can lead to performance degradation, while weight interpolation between fine-tuned and pre-trained models can alleviate the forgetting issue and improve overall performance. In conclusion, the authors released µ-Bench to facilitate the research and development of microscope base models.