Abstract:Due to the large size and lack of fine-grained annotation, Whole Slide Images (WSIs) analysis is commonly approached as a Multiple Instance Learning (MIL) problem. However, previous studies only learn from training data, posing a stark contrast to how human clinicians teach each other and reason about histopathologic entities and factors. Here we present a novel knowledge concept-based MIL framework, named ConcepPath to fill this gap. Specifically, ConcepPath utilizes GPT-4 to induce reliable diseasespecific human expert concepts from medical literature, and incorporate them with a group of purely learnable concepts to extract complementary knowledge from training data. In ConcepPath, WSIs are aligned to these linguistic knowledge concepts by utilizing pathology vision-language model as the basic building component. In the application of lung cancer subtyping, breast cancer HER2 scoring, and gastric cancer immunotherapy-sensitive subtyping task, ConcepPath significantly outperformed previous SOTA methods which lack the guidance of human expert knowledge.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve several key problems in whole - slide image (WSI) analysis: 1. **Large - scale and lack of fine - grained annotation**: The size of WSIs is very large (for example, 150,000 x 150,000 pixels), and they usually lack detailed annotations. This makes it difficult for traditional supervised learning methods to be directly applied to WSIs. 2. **Existing methods rely only on image data**: Most existing computational pathology methods mainly learn from image data, ignoring the knowledge and reasoning methods of human experts. This method is significantly different from how clinicians teach and understand pathological entities and factors. 3. **Limitations of multi - instance learning (MIL) methods**: Although MIL methods can perform weakly - supervised learning under slide - level labels, they perform poorly in handling complex tasks, especially in tasks that require the identification of complex tissue structures and molecular features. 4. **Reliability issues of language - prior generation**: Some studies attempt to use language priors to assist in WSI analysis, but in a fully - trained setting, these methods show unreliable language - prior generation and unsatisfactory performance. To solve these problems, the authors propose a new framework - ConcepPath. This framework improves the accuracy and interpretability of WSI analysis by combining human expert knowledge and new concepts learned from training data. Specifically: - **Introducing human expert knowledge**: ConcepPath uses large - language models (such as GPT - 4) to derive reliable disease - specific human - expert concepts from medical literature and combines them with learnable concepts to extract supplementary knowledge. - **Aligning language and image**: Align WSIs with these language - knowledge concepts through a pathological vision - language model, thereby using expert knowledge more effectively. - **Two - stage concept - guided hierarchical feature aggregation**: ConcepPath adopts a two - stage concept - guided hierarchical feature aggregation paradigm. First, instance features are aggregated into concept - specific bag - level features, and then further aggregated according to the correlation between instance - level concepts and bag - level expert - class prompts. - **Slide adapter**: To address the domain differences between the training data of the pathological vision - language model and downstream WSI analysis tasks, ConcepPath integrates a slide adapter before the final prediction. Through these innovations, ConcepPath significantly outperforms existing state - of - the - art methods in multiple complex WSI analysis tasks, especially in tasks such as lung cancer subtype classification, breast cancer HER2 scoring, and gastric cancer immunotherapy - sensitivity subtype classification.

Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis

Generating Hypergraph-Based High-Order Representations of Whole-Slide Histopathological Images for Survival Prediction

Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis

Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction

Finding Regions of Interest in Whole Slide Images Using Multiple Instance Learning

Automatic Whole Slide Pathology Image Diagnosis Framework Via Unit Stochastic Selection and Attention Fusion

An efficient context-aware approach for whole slide image classification

PathAlign: A vision-language model for whole slide images in histopathology

Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions

Interpretable Classification of Pathology Whole-Slide Images Using Attention Based Context-Aware Graph Convolutional Neural Network

Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis

Semantic-Similarity Collaborative Knowledge Distillation Framework for Whole Slide Image Classification

Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis

The Whole Pathological Slide Classification via Weakly Supervised Learning

Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks

SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification

Overcoming the limitations of patch-based learning to detect cancer in whole slide images

Data-efficient and weakly supervised computational pathology on whole-slide images

A self-supervised framework for learning whole slide representations

Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classification