Abstract:3532 Background: Multi-modal artificial intelligence (AI) approaches have substantial promise for capturing the relationships between pathology images, text descriptions, and patients’ clinical characteristics. Previous attempts showed that multi-modal AI methods classified colorectal cancer (CRC) types with an accuracy of 90%. However, it is unclear if this new AI framework can reliably predict histologic grade and the extent of primary tumor at scale (T stage). To address this research gap, here we developed a multi-modal AI system for detailed CRC pathology analyses, with a focus on histologic factors related to patient prognoses. Methods: We obtained whole-slide pathology images, pathology reports, and detailed clinical information from 441 colorectal adenocarcinoma patients who participated in The Cancer Genome Atlas. These patients were divided into training or testing sets using a 75/25 split. We extracted histology types (invasive adenocarcinoma with or without mucinous features), histology grades (well, moderate, or poorly differentiated), and the extent of the primary tumor (pT2, pT3, or pT4) from the pathology reports in the training subset. We employed the CLIP (Contrastive Language-Image Pre-Training) prefix captioning model as a backbone and developed an AI model for predicting these pathology profiles using whole-slide pathology images. We evaluated the performance (sensitivity, specificity, accuracy, precision, and F1 score) of our model by comparing the model-based prediction in the testing set and the ground-truth information from the pathology reports. Results: Our model successfully predicted histology types with an accuracy of 0.89 and an F1 score of 0.93 (sensitivity = 0.92, specificity = 0.76, and precision = 0.94). In addition, our approach further predicted histology grades with an accuracy of 0.79 and an F1 score of 0.87 (sensitivity = 0.96, specificity = 0.31, and precision = 0.80). Furthermore, our model achieved an accuracy of 0.69 and an F1 score of 0.80 (sensitivity = 0.78, specificity = 0.38, and precision = 0.83) in predicting T stages. Conclusions: Our study suggests that multi-modal AI has the potential to assist with the histopathology diagnosis of CRC tissues. Our approaches performed well in identifying CRC types and could provide second opinions on CRC diagnosis. However, the performance of our model varies across prediction tasks. Extensive external validation using larger and more diverse pathology datasets is crucial to moving our novel AI approaches closer to clinical applications. Category Sensitivity Specificity Accuracy Precision F1 score Histology types 0.92 0.76 0.89 0.94 0.93 Histology grades 0.96 0.31 0.79 0.80 0.87 T stages 0.78 0.38 0.69 0.83 0.80

A pathology foundation model for cancer diagnosis and prognosis prediction

A foundation model for clinical-grade computational pathology and rare cancers detection

A foundation model for generalizable cancer diagnosis and survival prediction from histopathological images

Virchow: A Million-Slide Digital Pathology Foundation Model

Towards a general-purpose foundation model for computational pathology

Interpretable Inference and Classification of Tissue Types in Histological Colorectal Cancer Slides Based on Ensembles Adaptive Boosting Prototype Tree.

Pan-Cancer Diagnostic Consensus Through Searching Archival Histopathology Images Using Artificial Intelligence

Pathologist-level interpretable whole-slide cancer diagnosis with deep learning

Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images

Review of deep learning-based pathological image classification: From task-specific models to foundation models

A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis

Accurate Diagnosis of Colorectal Cancer Based on Histopathology Images Using Artificial Intelligence.

A General-Purpose Self-Supervised Model for Computational Pathology

A whole-slide foundation model for digital pathology from real-world data

Multi-modal artificial intelligence enhanced systematic colorectal cancer pathology evaluation.

Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response

A pathologist–AI collaboration framework for enhancing diagnostic accuracies and efficiencies

Multi-modality artificial intelligence in digital pathology

Artificial intelligence in digital histopathology for predicting patient prognosis and treatment efficacy in breast cancer

AI Empowered of Advancements in Microbial and Tumor Cell Image Labeling for Enhanced Medical Insights

Digital whole slide helps artificial intelligence in pathological imaging strategies