Abstract:Understanding prognosis and mortality is critical for evaluating the treatment plan of patients. Advances in digital pathology and deep learning techniques have made it practical to perform survival analysis in whole slide images (WSIs). Current methods are usually based on a multi-stage framework which includes patch sampling, feature extraction and prediction. However, the random patch sampling strategy is highly unstable and prone to sampling non-ROI. Feature extraction typically relies on hand-crafted features or convolutional neural networks (CNNs) pre-trained on ImageNet, while the artificial error or domain gaps may affect the survival prediction performance. Besides, the limited information representation of local sampling patches will create a bottleneck limitation on the effectiveness of prediction. To address the above challenges, we propose a novel patch sampling strategy based on image information entropy and construct a Multi-Scale feature Fusion Network (MSFN) based on self-supervised feature extractor. Specifically, we adopt image information entropy as a criterion to select representative sampling patches, thereby avoiding the noise interference caused by random to blank regions. Meanwhile, we pretrain the feature extractor utilizing self-supervised learning mechanism to improve the efficiency of feature extraction. Furthermore, a global-local feature fusion prediction network based on the attention mechanism is constructed to improve the survival prediction effect of WSIs with comprehensive multi-scale information representation. The proposed method is validated by adequate experiments and achieves competitive results on both of the most popular WSIs survival analysis datasets, TCGA-GBM and TCGA-LUSC. Code and trained models are made available at: https://github.com/Mercuriiio/MSFN.

Integration of Patch Features Through Self-supervised Learning and Transformer for Survival Analysis on Whole Slide Images.

Generating Hypergraph-Based High-Order Representations of Whole-Slide Histopathological Images for Survival Prediction

Sparse and Hierarchical Transformer for Survival Analysis on Whole Slide Images

Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics

Transformer-Based Multimodal Fusion for Survival Prediction by Integrating Whole Slide Images, Clinical, and Genomic Data

Self-Supervised Learning Guided Transformer for Survival Prediction of Lung Cancer Using Pathological Images.

Wsisa: Making Survival Prediction From Whole Slide Histopathological Images

HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image

Self-supervised learning-based Multi-Scale feature Fusion Network for survival analysis from whole slide images

Explainable survival analysis with uncertainty using convolution-involved vision transformer

Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

Surformer: an interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images

End-to-end Multi-source Visual Prompt Tuning for Survival Analysis in Whole Slide Images

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Big-Hypergraph Factorization Neural Network for Survival Prediction From Whole Slide Image

Survival Prediction Based on Histopathology Imaging and Clinical Data: A Novel, Whole Slide CNN Approach

Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks

Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction

Ranking-Based Survival Prediction on Histopathological Whole-Slide Images

Transformer-Based Weakly Supervised Learning for Whole Slide Lung Cancer Image Classification

Shared-Specific Feature Learning with Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.