Foundation Models and Information Retrieval in Digital Pathology

H.R. Tizhoosh

2024-03-14

Abstract:The paper reviews the state-of-the-art of foundation models, LLMs, generative AI, information retrieval and CBIR in digital pathology

Information Retrieval,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning,Image and Video Processing

What problem does this paper attempt to address?

Foundation Models in Radiology: What, How, When, Why and Why Not

Magdalini Paschali,Zhihong Chen,Louis Blankemeier,Maya Varma,Alaa Youssef,Christian Bluethgen,Curtis Langlotz,Sergios Gatidis,Akshay Chaudhari

2024-11-28

Abstract:Recent advances in artificial intelligence have witnessed the emergence of large-scale deep learning models capable of interpreting and generating both textual and imaging data. Such models, typically referred to as foundation models, are trained on extensive corpora of unlabeled data and demonstrate high performance across various tasks. Foundation models have recently received extensive attention from academic, industry, and regulatory bodies. Given the potentially transformative impact that foundation models can have on the field of radiology, this review aims to establish a standardized terminology concerning foundation models, with a specific focus on the requirements of training data, model training paradigms, model capabilities, and evaluation strategies. We further outline potential pathways to facilitate the training of radiology-specific foundation models, with a critical emphasis on elucidating both the benefits and challenges associated with such models. Overall, we envision that this review can unify technical advances and clinical needs in the training of foundation models for radiology in a safe and responsible manner, for ultimately benefiting patients, providers, and radiologists.

Machine Learning
Leveraging Foundation Models for Content-Based Medical Image Retrieval in Radiology

Stefan Denner,David Zimmerer,Dimitrios Bounias,Markus Bujotzek,Shuhan Xiao,Lisa Kausch,Philipp Schader,Tobias Penzkofer,Paul F. Jäger,Klaus Maier-Hein

2024-04-17

Abstract:Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. Current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. In response, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based medical image retrieval. By benchmarking these models on a comprehensive dataset of 1.6 million 2D radiological images spanning four modalities and 161 pathologies, we identify weakly-supervised models as superior, achieving a P@1 of up to 0.594. This performance not only competes with a specialized model but does so without the need for fine-tuning. Our analysis further explores the challenges in retrieving pathological versus anatomical structures, indicating that accurate retrieval of pathological features presents greater difficulty. Despite these challenges, our research underscores the vast potential of foundation models for CBIR in radiology, proposing a shift towards versatile, general-purpose medical image retrieval systems that do not require specific tuning.

Computer Vision and Pattern Recognition,Information Retrieval
When is a Foundation Model a Foundation Model

Saghir Alfasly,Peyman Nejat,Sobhan Hemati,Jibran Khan,Isaiah Lahr,Areej Alsaafin,Abubakr Shafique,Nneka Comfere,Dennis Murphree,Chady Meroueh,Saba Yasir,Aaron Mangold,Lisa Boardman,Vijay Shah,Joaquin J. Garcia,H.R. Tizhoosh

2023-09-15

Abstract:Recently, several studies have reported on the fine-tuning of foundation models for image-text modeling in the field of medicine, utilizing images from online data sources such as Twitter and PubMed. Foundation models are large, deep artificial neural networks capable of learning the context of a specific domain through training on exceptionally extensive datasets. Through validation, we have observed that the representations generated by such models exhibit inferior performance in retrieval tasks within digital pathology when compared to those generated by significantly smaller, conventional deep networks.

Information Retrieval,Artificial Intelligence,Computer Vision and Pattern Recognition
On Validation of Search & Retrieval of Tissue Images in Digital Pathology

H.R. Tizhoosh

2024-08-03

Abstract:Medical images play a crucial role in modern healthcare by providing vital information for diagnosis, treatment planning, and disease monitoring. Fields such as radiology and pathology rely heavily on accurate image interpretation, with radiologists examining X-rays, CT scans, and MRIs to diagnose conditions from fractures to cancer, while pathologists use microscopy and digital images to detect cellular abnormalities for diagnosing cancers and infections. The technological advancements have exponentially increased the volume and complexity of medical images, necessitating efficient tools for management and retrieval. Content-Based Image Retrieval (CBIR) systems address this need by searching and retrieving images based on visual content, enhancing diagnostic accuracy by allowing clinicians to find similar cases and compare pathological patterns. Comprehensive validation of image search engines in medical applications involves evaluating performance metrics like accuracy, indexing, and search times, and storage overhead, ensuring reliable and efficient retrieval of accurate results, as demonstrated by recent validations in histopathology.

Image and Video Processing,Computer Vision and Pattern Recognition,Information Retrieval
Benchmarking foundation models as feature extractors for weakly-supervised computational pathology

Peter Neidlinger,Omar S. M. El Nahhas,Hannah Sophie Muti,Tim Lenz,Michael Hoffmeister,Hermann Brenner,Marko van Treeck,Rupert Langer,Bastian Dislich,Hans Michael Behrens,Christoph Röcken,Sebastian Foersch,Daniel Truhn,Antonio Marra,Oliver Lester Saldanha,Jakob Nikolas Kather

2024-08-28

Abstract:Advancements in artificial intelligence have driven the development of numerous pathology foundation models capable of extracting clinically relevant information. However, there is currently limited literature independently evaluating these foundation models on truly external cohorts and clinically-relevant tasks to uncover adjustments for future improvements. In this study, we benchmarked ten histopathology foundation models on 13 patient cohorts with 6,791 patients and 9,493 slides from lung, colorectal, gastric, and breast cancers. The models were evaluated on weakly-supervised tasks related to biomarkers, morphological properties, and prognostic outcomes. We show that a vision-language foundation model, CONCH, yielded the highest performance in 42% of tasks when compared to vision-only foundation models. The experiments reveal that foundation models trained on distinct cohorts learn complementary features to predict the same label, and can be fused to outperform the current state of the art. Creating an ensemble of complementary foundation models outperformed CONCH in 66% of tasks. Moreover, our findings suggest that data diversity outweighs data volume for foundation models. Our work highlights actionable adjustments to improve pathology foundation models.

Image and Video Processing,Computer Vision and Pattern Recognition
Zero-Shot Whole Slide Image Retrieval in Histopathology Using Embeddings of Foundation Models

Saghir Alfasly,Ghazal Alabtah,Sobhan Hemati,Krishna Rani Kalari,H.R. Tizhoosh

2024-09-12

Abstract:We have tested recently published foundation models for histopathology for image retrieval. We report macro average of F1 score for top-1 retrieval, majority of top-3 retrievals, and majority of top-5 retrievals. We perform zero-shot retrievals, i.e., we do not alter embeddings and we do not train any classifier. As test data, we used diagnostic slides of TCGA, The Cancer Genome Atlas, consisting of 23 organs and 117 cancer subtypes. As a search platform we used Yottixel that enabled us to perform WSI search using patches. Achieved F1 scores show low performance, e.g., for top-5 retrievals, 27% +/- 13% (Yottixel-DenseNet), 42% +/- 14% (Yottixel-UNI), 40%+/-13% (Yottixel-Virchow), 41%+/-13% (Yottixel-GigaPath), and 41%+/-14% (GigaPath WSI).

Image and Video Processing,Artificial Intelligence,Computer Vision and Pattern Recognition
RudolfV: A Foundation Model by Pathologists for Pathologists

Jonas Dippel,Barbara Feulner,Tobias Winterhoff,Timo Milbich,Stephan Tietz,Simon Schallenberg,Gabriel Dernbach,Andreas Kunft,Simon Heinke,Marie-Lisa Eich,Julika Ribbat-Idel,Rosemarie Krupar,Philipp Anders,Niklas Prenißl,Philipp Jurmeister,David Horst,Lukas Ruff,Klaus-Robert Müller,Frederick Klauschen,Maximilian Alber

2024-06-12

Abstract:Artificial intelligence has started to transform histopathology impacting clinical diagnostics and biomedical research. However, while many computational pathology approaches have been proposed, most current AI models are limited with respect to generalization, application variety, and handling rare diseases. Recent efforts introduced self-supervised foundation models to address these challenges, yet existing approaches do not leverage pathologist knowledge by design. In this study, we present a novel approach to designing foundation models for computational pathology, incorporating pathologist expertise, semi-automated data curation, and a diverse dataset from over 15 laboratories, including 58 tissue types, and encompassing 129 different histochemical and immunohistochemical staining modalities. We demonstrate that our model "RudolfV" surpasses existing state-of-the-art foundation models across different benchmarks focused on tumor microenvironment profiling, biomarker evaluation, and reference case search while exhibiting favorable robustness properties. Our study shows how domain-specific knowledge can increase the efficiency and performance of pathology foundation models and enable novel application areas.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
Foundation Models for Slide-level Cancer Subtyping in Digital Pathology

Pablo Meseguer,Rocío del Amor,Adrian Colomer,Valery Naranjo

2024-10-21

Abstract:Since the emergence of the ImageNet dataset, the pretraining and fine-tuning approach has become widely adopted in computer vision due to the ability of ImageNet-pretrained models to learn a wide variety of visual features. However, a significant challenge arises when adapting these models to domain-specific fields, such as digital pathology, due to substantial gaps between domains. To address this limitation, foundation models (FM) have been trained on large-scale in-domain datasets to learn the intricate features of histopathology images. In cancer diagnosis, whole-slide image (WSI) prediction is essential for patient prognosis, and multiple instance learning (MIL) has been implemented to handle the giga-pixel size of WSI. As MIL frameworks rely on patch-level feature aggregation, this work aims to compare the performance of various feature extractors developed under different pretraining strategies for cancer subtyping on WSI under a MIL framework. Results demonstrate the ability of foundation models to surpass ImageNet-pretrained models for the prediction of six skin cancer subtypes

Computer Vision and Pattern Recognition
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision

Bobby Azad,Reza Azad,Sania Eskandari,Afshin Bozorgpour,Amirhossein Kazerouni,Islem Rekik,Dorit Merhof

2023-10-28

Abstract:Foundation models, large-scale, pre-trained deep-learning models adapted to a wide range of downstream tasks have gained significant interest lately in various deep-learning problems undergoing a paradigm shift with the rise of these models. Trained on large-scale dataset to bridge the gap between different modalities, foundation models facilitate contextual reasoning, generalization, and prompt capabilities at test time. The predictions of these models can be adjusted for new tasks by augmenting the model input with task-specific hints called prompts without requiring extensive labeled data and retraining. Capitalizing on the advances in computer vision, medical imaging has also marked a growing interest in these models. To assist researchers in navigating this direction, this survey intends to provide a comprehensive overview of foundation models in the domain of medical imaging. Specifically, we initiate our exploration by providing an exposition of the fundamental concepts forming the basis of foundation models. Subsequently, we offer a methodical taxonomy of foundation models within the medical domain, proposing a classification system primarily structured around training strategies, while also incorporating additional facets such as application domains, imaging modalities, specific organs of interest, and the algorithms integral to these models. Furthermore, we emphasize the practical use case of some selected approaches and then discuss the opportunities, applications, and future directions of these large-scale pre-trained models, for analyzing medical images. In the same vein, we address the prevailing challenges and research pathways associated with foundational models in medical imaging. These encompass the areas of interpretability, data management, computational requirements, and the nuanced issue of contextual comprehension.

Computer Vision and Pattern Recognition
A foundation model for clinical-grade computational pathology and rare cancers detection

Eugene Vorontsov,Alican Bozkurt,Adam Casson,George Shaikovski,Michal Zelechowski,Kristen Severson,Eric Zimmermann,James Hall,Neil Tenenholtz,Nicolo Fusi,Ellen Yang,Philippe Mathieu,Alexander van Eck,Donghun Lee,Julian Viret,Eric Robert,Yi Kan Wang,Jeremy D Kunz,Matthew C H Lee,Jan H Bernhard,Ran A Godrich,Gerard Oakley,Ewan Millar,Matthew Hanna,Hannah Wen,Juan A Retamero,William A Moye,Razik Yousfi,Christopher Kanan,David S Klimstra,Brandon Rothrock,Siqi Liu,Thomas J Fuchs

DOI: https://doi.org/10.1038/s41591-024-03141-0

Abstract:The analysis of histopathology images with artificial intelligence aims to enable clinical decision support systems and precision medicine. The success of such applications depends on the ability to model the diverse patterns observed in pathology images. To this end, we present Virchow, the largest foundation model for computational pathology to date. In addition to the evaluation of biomarker prediction and cell identification, we demonstrate that a large foundation model enables pan-cancer detection, achieving 0.95 specimen-level area under the (receiver operating characteristic) curve across nine common and seven rare cancers. Furthermore, we show that with less training data, the pan-cancer detector built on Virchow can achieve similar performance to tissue-specific clinical-grade models in production and outperform them on some rare variants of cancer. Virchow's performance gains highlight the value of a foundation model and open possibilities for many high-impact applications with limited amounts of labeled training data.
PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

George Shaikovski,Adam Casson,Kristen Severson,Eric Zimmermann,Yi Kan Wang,Jeremy D. Kunz,Juan A. Retamero,Gerard Oakley,David Klimstra,Christopher Kanan,Matthew Hanna,Michal Zelechowski,Julian Viret,Neil Tenenholtz,James Hall,Nicolo Fusi,Razik Yousfi,Peter Hamilton,William A. Moye,Eugene Vorontsov,Siqi Liu,Thomas J. Fuchs

2024-05-23

Abstract:Foundation models in computational pathology promise to unlock the development of new clinical decision support systems and models for precision medicine. However, there is a mismatch between most clinical analysis, which is defined at the level of one or more whole slide images, and foundation models to date, which process the thousands of image tiles contained in a whole slide image separately. The requirement to train a network to aggregate information across a large number of tiles in multiple whole slide images limits these models' impact. In this work, we present a slide-level foundation model for H&E-stained histopathology, PRISM, that builds on Virchow tile embeddings and leverages clinical report text for pre-training. Using the tile embeddings, PRISM produces slide-level embeddings with the ability to generate clinical reports, resulting in several modes of use. Using text prompts, PRISM achieves zero-shot cancer detection and sub-typing performance approaching and surpassing that of a supervised aggregator model. Using the slide embeddings with linear classifiers, PRISM surpasses supervised aggregator models. Furthermore, we demonstrate that fine-tuning of the PRISM slide encoder yields label-efficient training for biomarker prediction, a task that typically suffers from low availability of training data; an aggregator initialized with PRISM and trained on as little as 10% of the training data can outperform a supervised baseline that uses all of the data.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

Shengyi Hua,Fang Yan,Tianle Shen,Lei Ma,Xiaofan Zhang

2024-08-06

Abstract:Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. To validate the efficacy of our models, we evaluate the performance over a wide variety of downstream tasks, including patch-level colorectal cancer subtyping and whole slide image (WSI)-level classification in H&E field, together with expression level prediction of IHC marker, tumor identification and slide-level qualitative analysis in IHC field. The experimental results show the superiority of our models over most tasks and the efficacy of proposed pretext tasks. The codes and models are available at <a class="link-external link-https" href="https://github.com/openmedlab/PathoDuet" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence
Foundation model for cancer imaging biomarkers

Suraj Pai,Dennis Bontempi,Ibrahim Hadzic,Vasco Prudente,Mateo Sokač,Tafadzwa L. Chaunzwa,Simon Bernatz,Ahmed Hosny,Raymond H. Mak,Nicolai J. Birkbak,Hugo J. W. L. Aerts

DOI: https://doi.org/10.1038/s42256-024-00807-9

IF: 23.8

2024-03-16

Nature Machine Intelligence

Abstract:Foundation models in deep learning are characterized by a single large-scale model trained on vast amounts of data serving as the foundation for various downstream tasks. Foundation models are generally trained using self-supervised learning and excel in reducing the demand for training samples in downstream applications. This is especially important in medicine, where large labelled datasets are often scarce. Here, we developed a foundation model for cancer imaging biomarker discovery by training a convolutional encoder through self-supervised learning using a comprehensive dataset of 11,467 radiographic lesions. The foundation model was evaluated in distinct and clinically relevant applications of cancer imaging-based biomarkers. We found that it facilitated better and more efficient learning of imaging biomarkers and yielded task-specific models that significantly outperformed conventional supervised and other state-of-the-art pretrained implementations on downstream tasks, especially when training dataset sizes were very limited. Furthermore, the foundation model was more stable to input variations and showed strong associations with underlying biology. Our results demonstrate the tremendous potential of foundation models in discovering new imaging biomarkers that may extend to other clinical use cases and can accelerate the widespread translation of imaging biomarkers into clinical settings.

computer science, artificial intelligence, interdisciplinary applications
The Promises and Perils of Foundation Models in Dermatology

Haiwen Gui,Jesutofunmi A. Omiye,Crystal T. Chang,Roxana Daneshjou

DOI: https://doi.org/10.1016/j.jid.2023.12.019

IF: 7.59

2024-03-06

Journal of Investigative Dermatology

Abstract:Foundation models (FM), which are large-scale artificial intelligence (AI) models that can complete a range of tasks, represent a paradigm shift in AI. These versatile models encompass large language models, vision-language models, and multimodal models. Although these models are often trained for broad tasks, they have been applied either out of the box or after additional fine tuning to tasks in medicine, including dermatology. From addressing administrative tasks to answering dermatology...

dermatology
Histopathology language-image representation learning for fine-grained digital pathology cross-modal retrieval

Dingyi Hu,Zhiguo Jiang,Jun Shi,Fengying Xie,Kun Wu,Kunming Tang,Ming Cao,Jianguo Huai,Yushan Zheng

DOI: https://doi.org/10.1016/j.media.2024.103163

Abstract:Large-scale digital whole slide image (WSI) datasets analysis have gained significant attention in computer-aided cancer diagnosis. Content-based histopathological image retrieval (CBHIR) is a technique that searches a large database for data samples matching input objects in both details and semantics, offering relevant diagnostic information to pathologists. However, the current methods are limited by the difficulty of gigapixels, the variable size of WSIs, and the dependence on manual annotations. In this work, we propose a novel histopathology language-image representation learning framework for fine-grained digital pathology cross-modal retrieval, which utilizes paired diagnosis reports to learn fine-grained semantics from the WSI. An anchor-based WSI encoder is built to extract hierarchical region features and a prompt-based text encoder is introduced to learn fine-grained semantics from the diagnosis reports. The proposed framework is trained with a multivariate cross-modal loss function to learn semantic information from the diagnosis report at both the instance level and region level. After training, it can perform four types of retrieval tasks based on the multi-modal database to support diagnostic requirements. We conducted experiments on an in-house dataset and a public dataset to evaluate the proposed method. Extensive experiments have demonstrated the effectiveness of the proposed method and its advantages to the present histopathology retrieval methods. The code is available at https://github.com/hudingyi/FGCR.
Foundation AI Model for Medical Image Segmentation

Rina Bao,Erfan Darzi,Sheng He,Chuan-Heng Hsiao,Mohammad Arafat Hussain,Jingpeng Li,Atle Bjornerud,Ellen Grant,Yangming Ou

2024-11-05

Abstract:Foundation models refer to artificial intelligence (AI) models that are trained on massive amounts of data and demonstrate broad generalizability across various tasks with high accuracy. These models offer versatile, one-for-many or one-for-all solutions, eliminating the need for developing task-specific AI models. Examples of such foundation models include the Chat Generative Pre-trained Transformer (ChatGPT) and the Segment Anything Model (SAM). These models have been trained on millions to billions of samples and have shown wide-ranging and accurate applications in numerous tasks such as text processing (using ChatGPT) and natural image segmentation (using SAM). In medical image segmentation - finding target regions in medical images - there is a growing need for these one-for-many or one-for-all foundation models. Such models could obviate the need to develop thousands of task-specific AI models, which is currently standard practice in the field. They can also be adapted to tasks with datasets too small for effective training. We discuss two paths to achieve foundation models for medical image segmentation and comment on progress, challenges, and opportunities. One path is to adapt or fine-tune existing models, originally developed for natural images, for use with medical images. The second path entails building models from scratch, exclusively training on medical images.

Image and Video Processing,Computer Vision and Pattern Recognition
Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation

Jiabo Ma,Zhengrui Guo,Fengtao Zhou,Yihui Wang,Yingxue Xu,Yu Cai,Zhengjie Zhu,Cheng Jin,Yi Lin,Xinrui Jiang,Anjia Han,Li Liang,Ronald Cheong Kin Chan,Jiguang Wang,Kwang-Ting Cheng,Hao Chen

2024-08-03

Abstract:Foundation models pretrained on large-scale datasets are revolutionizing the field of computational pathology (CPath). The generalization ability of foundation models is crucial for the success in various downstream clinical tasks. However, current foundation models have only been evaluated on a limited type and number of tasks, leaving their generalization ability and overall performance unclear. To address this gap, we established a most comprehensive benchmark to evaluate the performance of off-the-shelf foundation models across six distinct clinical task types, encompassing a total of 39 specific tasks. Our findings reveal that existing foundation models excel at certain task types but struggle to effectively handle the full breadth of clinical tasks. To improve the generalization of pathology foundation models, we propose a unified knowledge distillation framework consisting of both expert and self knowledge distillation, where the former allows the model to learn from the knowledge of multiple expert models, while the latter leverages self-distillation to enable image representation learning via local-global alignment. Based on this framework, a Generalizable Pathology Foundation Model (GPFM) is pretrained on a large-scale dataset consisting of 190 million images from around 86,000 public H&E whole slides across 34 major tissue types. Evaluated on the established benchmark, GPFM achieves an impressive average rank of 1.36, with 29 tasks ranked 1st, while the the second-best model, UNI, attains an average rank of 2.96, with only 4 tasks ranked 1st. The superior generalization of GPFM demonstrates its exceptional modeling capabilities across a wide range of clinical tasks, positioning it as a new cornerstone for feature representation in CPath.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
A pathology foundation model for cancer diagnosis and prognosis prediction

Xiyue Wang,Junhan Zhao,Eliana Marostica,Wei Yuan,Jietian Jin,Jiayu Zhang,Ruijiang Li,Hongping Tang,Kanran Wang,Yu Li,Fang Wang,Yulong Peng,Junyou Zhu,Jing Zhang,Christopher R Jackson,Jun Zhang,Deborah Dillon,Nancy U Lin,Lynette Sholl,Thomas Denize,David Meredith,Keith L Ligon,Sabina Signoretti,Shuji Ogino,Jeffrey A Golden,MacLean P Nasrallah,Xiao Han,Sen Yang,Kun-Hsing Yu

DOI: https://doi.org/10.1038/s41586-024-07894-z

IF: 64.8

2024-09-04

Nature

Abstract:Histopathology image evaluation is indispensable for cancer diagnoses and subtype classification. Standard artificial intelligence methods for histopathology image analyses have focused on optimizing specialized models for each diagnostic task1,2. Although such methods have achieved some success, they often have limited generalizability to images generated by different digitization protocols or samples collected from different populations3. Here, to address this challenge, we devised the Clinical Histopathology Imaging Evaluation Foundation (CHIEF) model, a general-purpose weakly supervised machine learning framework to extract pathology imaging features for systematic cancer evaluation. CHIEF leverages two complementary pretraining methods to extract diverse pathology representations: unsupervised pretraining for tile-level feature identification and weakly supervised pretraining for whole-slide pattern recognition. We developed CHIEF using 60,530 whole-slide images spanning 19 anatomical sites. Through pretraining on 44 terabytes of high-resolution pathology imaging datasets, CHIEF extracted microscopic representations useful for cancer cell detection, tumour origin identification, molecular profile characterization and prognostic prediction. We successfully validated CHIEF using 19,491 whole-slide images from 32 independent slide sets collected from 24 hospitals and cohorts internationally. Overall, CHIEF outperformed the state-of-the-art deep learning methods by up to 36.1%, showing its ability to address domain shifts observed in samples from diverse populations and processed by different slide preparation methods. CHIEF provides a generalizable foundation for efficient digital pathology evaluation for patients with cancer.
A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models

Gabriele Campanella,Shengjia Chen,Ruchika Verma,Jennifer Zeng,Aryeh Stock,Matt Croken,Brandon Veremis,Abdulkadir Elmas,Kuan-lin Huang,Ricky Kwan,Jane Houldsworth,Adam J. Schoenfeld,Chad Vanderbilt

2024-07-12

Abstract:The use of self-supervised learning (SSL) to train pathology foundation models has increased substantially in the past few years. Notably, several models trained on large quantities of clinical data have been made publicly available in recent months. This will significantly enhance scientific research in computational pathology and help bridge the gap between research and clinical deployment. With the increase in availability of public foundation models of different sizes, trained using different algorithms on different datasets, it becomes important to establish a benchmark to compare the performance of such models on a variety of clinically relevant tasks spanning multiple organs and diseases. In this work, we present a collection of pathology datasets comprising clinical slides associated with clinically relevant endpoints including cancer diagnoses and a variety of biomarkers generated during standard hospital operation from two medical centers. We leverage these datasets to systematically assess the performance of public pathology foundation models and provide insights into best practices for training new foundation models and selecting appropriate pretrained models.

Image and Video Processing,Computer Vision and Pattern Recognition
Review of deep learning-based pathological image classification: From task-specific models to foundation models

Haijing Luan,Kaixing Yang,Taiyuan Hu,Jifang Hu,Siyao Liu,Ruilin Li,Jiayin He,Rui Yan,Xiaobing Guo,Niansong Qian,Beifang Niu

DOI: https://doi.org/10.1016/j.future.2024.107578

IF: 7.307

2024-10-30

Future Generation Computer Systems

Abstract:Pathological diagnosis is considered the gold standard in cancer diagnosis, playing a crucial role in guiding treatment decisions and prognosis assessment for patients. However, achieving accurate diagnosis of pathology images poses several challenges, including the scarcity of pathologists and the inherent subjective variability in their interpretations. The advancements in whole-slide imaging technology and deep learning methods provide new opportunities for digital pathology, especially in low-resource settings, by enabling effective pathological image classification. In this article, we begin by introducing the datasets, which include both unimodal and multimodal types, as essential resources for advancing pathological image classification. We then provide a comprehensive overview of deep learning-based pathological image classification models, covering task-specific models such as supervised, unsupervised, weakly supervised, and semi-supervised learning methods, as well as unimodal and multimodal foundation models. Next, we review tumor-related indicators that can be predicted from pathological images, focusing on two main categories: indicators that can be recognized by pathologists, such as tumor classification, grading, and region recognition; and those that cannot be recognized by pathologists, including molecular subtype prediction, tumor origin prediction, biomarker prediction, and survival prediction. Finally, we summarize the key challenges in digital pathology and propose potential future directions.

computer science, theory & methods

Foundation Models and Information Retrieval in Digital Pathology

Foundation Models in Radiology: What, How, When, Why and Why Not

Leveraging Foundation Models for Content-Based Medical Image Retrieval in Radiology

When is a Foundation Model a Foundation Model

On Validation of Search & Retrieval of Tissue Images in Digital Pathology

Benchmarking foundation models as feature extractors for weakly-supervised computational pathology

Zero-Shot Whole Slide Image Retrieval in Histopathology Using Embeddings of Foundation Models

RudolfV: A Foundation Model by Pathologists for Pathologists

Foundation Models for Slide-level Cancer Subtyping in Digital Pathology

Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision

A foundation model for clinical-grade computational pathology and rare cancers detection

PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

Foundation model for cancer imaging biomarkers

The Promises and Perils of Foundation Models in Dermatology

Histopathology language-image representation learning for fine-grained digital pathology cross-modal retrieval

Foundation AI Model for Medical Image Segmentation

Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation

A pathology foundation model for cancer diagnosis and prognosis prediction

A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models

Review of deep learning-based pathological image classification: From task-specific models to foundation models