Abstract:3532 Background: Multi-modal artificial intelligence (AI) approaches have substantial promise for capturing the relationships between pathology images, text descriptions, and patients’ clinical characteristics. Previous attempts showed that multi-modal AI methods classified colorectal cancer (CRC) types with an accuracy of 90%. However, it is unclear if this new AI framework can reliably predict histologic grade and the extent of primary tumor at scale (T stage). To address this research gap, here we developed a multi-modal AI system for detailed CRC pathology analyses, with a focus on histologic factors related to patient prognoses. Methods: We obtained whole-slide pathology images, pathology reports, and detailed clinical information from 441 colorectal adenocarcinoma patients who participated in The Cancer Genome Atlas. These patients were divided into training or testing sets using a 75/25 split. We extracted histology types (invasive adenocarcinoma with or without mucinous features), histology grades (well, moderate, or poorly differentiated), and the extent of the primary tumor (pT2, pT3, or pT4) from the pathology reports in the training subset. We employed the CLIP (Contrastive Language-Image Pre-Training) prefix captioning model as a backbone and developed an AI model for predicting these pathology profiles using whole-slide pathology images. We evaluated the performance (sensitivity, specificity, accuracy, precision, and F1 score) of our model by comparing the model-based prediction in the testing set and the ground-truth information from the pathology reports. Results: Our model successfully predicted histology types with an accuracy of 0.89 and an F1 score of 0.93 (sensitivity = 0.92, specificity = 0.76, and precision = 0.94). In addition, our approach further predicted histology grades with an accuracy of 0.79 and an F1 score of 0.87 (sensitivity = 0.96, specificity = 0.31, and precision = 0.80). Furthermore, our model achieved an accuracy of 0.69 and an F1 score of 0.80 (sensitivity = 0.78, specificity = 0.38, and precision = 0.83) in predicting T stages. Conclusions: Our study suggests that multi-modal AI has the potential to assist with the histopathology diagnosis of CRC tissues. Our approaches performed well in identifying CRC types and could provide second opinions on CRC diagnosis. However, the performance of our model varies across prediction tasks. Extensive external validation using larger and more diverse pathology datasets is crucial to moving our novel AI approaches closer to clinical applications. Category Sensitivity Specificity Accuracy Precision F1 score Histology types 0.92 0.76 0.89 0.94 0.93 Histology grades 0.96 0.31 0.79 0.80 0.87 T stages 0.78 0.38 0.69 0.83 0.80

Automated histological classification of whole-slide images of gastric biopsy specimens

Two-step artificial intelligence system for endoscopic gastric biopsy improves the diagnostic accuracy of pathologists

Computer-aided evaluation of neuroblastoma on whole-slide histology images: Classifying grade of neuroblastic differentiation

Histopathological Diagnosis System for Gastritis Using Deep Learning Algorithm.

Comparing Accuracy of Helicobacter pylori Identification Using Traditional Hematoxylin and Eosin-Stained Glass Slides With Digital Whole Slide Imaging

Deep neural networks for automated classification of colorectal polyps on histopathology slides: A multi-institutional evaluation

Using less annotation workload to establish a pathological auxiliary diagnosis system for gastric cancer

Clinically applicable histopathological diagnosis system for gastric cancer detection using deep learning

Artificial Intelligence-Based Quality Assessment of Histopathology Whole-Slide Images within a Clinical Workflow: Assessment of 'PathProfiler' in a Diagnostic Pathology Setting

Computer‐aided diagnosis in real‐time endoscopy for all stages of gastric carcinogenesis: Development and validation study

Deep Learning-Based Sparse Whole-Slide Image Analysis for the Diagnosis of Gastric Intestinal Metaplasia

An artificial intelligence system for comprehensive pathologic outcome prediction in early gastric cancer through endoscopic image analysis (with video)

Accuracy and Efficiency of Deep-Learning–Based Automation of Dual Stain Cytology in Cervical Cancer Screening

Computational Image Analysis Identifies Histopathological Image Features Associated With Somatic Mutations and Patient Survival in Gastric Adenocarcinoma

Application of Graph Based Features in Computer Aided Diagnosis for Histopathological Image Classification of Gastric Cancer

Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images

Artificial Intelligence in the Pathology of Gastric Cancer

Implementing an integrated molecular classification for gastric cancer from endoscopic biopsies using on-slide tests

Pan-Cancer Diagnostic Consensus Through Searching Archival Histopathology Images Using Artificial Intelligence

Multi-modal artificial intelligence enhanced systematic colorectal cancer pathology evaluation.