Abstract:3532 Background: Multi-modal artificial intelligence (AI) approaches have substantial promise for capturing the relationships between pathology images, text descriptions, and patients’ clinical characteristics. Previous attempts showed that multi-modal AI methods classified colorectal cancer (CRC) types with an accuracy of 90%. However, it is unclear if this new AI framework can reliably predict histologic grade and the extent of primary tumor at scale (T stage). To address this research gap, here we developed a multi-modal AI system for detailed CRC pathology analyses, with a focus on histologic factors related to patient prognoses. Methods: We obtained whole-slide pathology images, pathology reports, and detailed clinical information from 441 colorectal adenocarcinoma patients who participated in The Cancer Genome Atlas. These patients were divided into training or testing sets using a 75/25 split. We extracted histology types (invasive adenocarcinoma with or without mucinous features), histology grades (well, moderate, or poorly differentiated), and the extent of the primary tumor (pT2, pT3, or pT4) from the pathology reports in the training subset. We employed the CLIP (Contrastive Language-Image Pre-Training) prefix captioning model as a backbone and developed an AI model for predicting these pathology profiles using whole-slide pathology images. We evaluated the performance (sensitivity, specificity, accuracy, precision, and F1 score) of our model by comparing the model-based prediction in the testing set and the ground-truth information from the pathology reports. Results: Our model successfully predicted histology types with an accuracy of 0.89 and an F1 score of 0.93 (sensitivity = 0.92, specificity = 0.76, and precision = 0.94). In addition, our approach further predicted histology grades with an accuracy of 0.79 and an F1 score of 0.87 (sensitivity = 0.96, specificity = 0.31, and precision = 0.80). Furthermore, our model achieved an accuracy of 0.69 and an F1 score of 0.80 (sensitivity = 0.78, specificity = 0.38, and precision = 0.83) in predicting T stages. Conclusions: Our study suggests that multi-modal AI has the potential to assist with the histopathology diagnosis of CRC tissues. Our approaches performed well in identifying CRC types and could provide second opinions on CRC diagnosis. However, the performance of our model varies across prediction tasks. Extensive external validation using larger and more diverse pathology datasets is crucial to moving our novel AI approaches closer to clinical applications. Category Sensitivity Specificity Accuracy Precision F1 score Histology types 0.92 0.76 0.89 0.94 0.93 Histology grades 0.96 0.31 0.79 0.80 0.87 T stages 0.78 0.38 0.69 0.83 0.80

Interpretable Inference and Classification of Tissue Types in Histological Colorectal Cancer Slides Based on Ensembles Adaptive Boosting Prototype Tree.

Multi-modal artificial intelligence enhanced systematic colorectal cancer pathology evaluation.

An Ensemble Framework of Deep Neural Networks for Colorectal Polyp Classification

DeepTree: Pathological Image Classification Through Imitating Tree-Like Strategies of Pathologists

Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Image

Ensemble-based multi-tissue classification approach of colorectal cancer histology images using a novel hybrid deep learning framework

A Computer-Aided Diagnosis System for Breast Pathology: A Deep Learning Approach with Model Interpretability from Pathological Perspective

Double Attention for Pathology Image Diagnosis Network with Visual Interpretability.

A promising deep learning-assistive algorithm for histopathological screening of colorectal cancer

Improving Performance in Colorectal Cancer Histology Decomposition using Deep and Ensemble Machine Learning

An interpretable machine learning system for colorectal cancer diagnosis from pathology slides

Interpretable Classification of Pathology Whole-Slide Images Using Attention Based Context-Aware Graph Convolutional Neural Network

Stacking-based Ensemble Learning of Decision Trees for Interpretable Prostate Cancer Detection

A pathology foundation model for cancer diagnosis and prognosis prediction

Automatic Whole Slide Pathology Image Diagnosis Framework Via Unit Stochastic Selection and Attention Fusion

An attention-based multi-resolution model for prostate whole slide imageclassification and localization

Accurate prediction of colorectal cancer diagnosis using machine learning based on immunohistochemistry pathological images

A multi-resolution model for histopathology image classification and localization with multiple instance learning

Ensemble transformer-based multiple instance learning to predict pathological subtypes and tumor mutational burden from histopathological whole slide images of endometrial and colorectal cancer

Colorectal Cancer Outcome Prediction from H&E Whole Slide Images using Machine Learning and Automatically Inferred Phenotype Profiles