Abstract:3532 Background: Multi-modal artificial intelligence (AI) approaches have substantial promise for capturing the relationships between pathology images, text descriptions, and patients’ clinical characteristics. Previous attempts showed that multi-modal AI methods classified colorectal cancer (CRC) types with an accuracy of 90%. However, it is unclear if this new AI framework can reliably predict histologic grade and the extent of primary tumor at scale (T stage). To address this research gap, here we developed a multi-modal AI system for detailed CRC pathology analyses, with a focus on histologic factors related to patient prognoses. Methods: We obtained whole-slide pathology images, pathology reports, and detailed clinical information from 441 colorectal adenocarcinoma patients who participated in The Cancer Genome Atlas. These patients were divided into training or testing sets using a 75/25 split. We extracted histology types (invasive adenocarcinoma with or without mucinous features), histology grades (well, moderate, or poorly differentiated), and the extent of the primary tumor (pT2, pT3, or pT4) from the pathology reports in the training subset. We employed the CLIP (Contrastive Language-Image Pre-Training) prefix captioning model as a backbone and developed an AI model for predicting these pathology profiles using whole-slide pathology images. We evaluated the performance (sensitivity, specificity, accuracy, precision, and F1 score) of our model by comparing the model-based prediction in the testing set and the ground-truth information from the pathology reports. Results: Our model successfully predicted histology types with an accuracy of 0.89 and an F1 score of 0.93 (sensitivity = 0.92, specificity = 0.76, and precision = 0.94). In addition, our approach further predicted histology grades with an accuracy of 0.79 and an F1 score of 0.87 (sensitivity = 0.96, specificity = 0.31, and precision = 0.80). Furthermore, our model achieved an accuracy of 0.69 and an F1 score of 0.80 (sensitivity = 0.78, specificity = 0.38, and precision = 0.83) in predicting T stages. Conclusions: Our study suggests that multi-modal AI has the potential to assist with the histopathology diagnosis of CRC tissues. Our approaches performed well in identifying CRC types and could provide second opinions on CRC diagnosis. However, the performance of our model varies across prediction tasks. Extensive external validation using larger and more diverse pathology datasets is crucial to moving our novel AI approaches closer to clinical applications. Category Sensitivity Specificity Accuracy Precision F1 score Histology types 0.92 0.76 0.89 0.94 0.93 Histology grades 0.96 0.31 0.79 0.80 0.87 T stages 0.78 0.38 0.69 0.83 0.80

Multi-modal analysis for accurate prediction of preoperative stage and indications of optimal treatment in gastric cancer

A multi-Task Learning based applicable AI model simultaneously predicts stage, histology, grade and LNM for cervical cancer before surgery

Prediction of lymph node metastasis in primary gastric cancer from pathological images and clinical data by multimodal multiscale deep learning

Development and validation of a Radiopathomics model based on CT scans and whole slide images for discriminating between Stage I-II and Stage III gastric cancer

Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data

Artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer

Identifying Prognostic Markers From Clinical, Radiomics, and Deep Learning Imaging Features for Gastric Cancer Survival Prediction

Preoperative Multidetector-Row Computed Tomography-Guided Prognostic Prediction for Disease-Free Survival: A Multicenter Study in Gastric Cancer

Robustly federated learning model for identifying high-risk patients with postoperative gastric cancer recurrence

Application of artificial intelligence for improving early detection and prediction of therapeutic outcomes for gastric cancer in the era of precision oncology

Prediction of HER2 Expression in Gastric Adenocarcinoma Based On Preoperative Noninvasive Multimodal 18F-FDG PET/CT Imaging

Machine Learning-Based Computational Models Derived From Large-Scale Radiographic-Radiomic Images Can Help Predict Adverse Histopathological Status of Gastric Cancer

Development of a deep learning model for early gastric cancer diagnosis using preoperative computed tomography images

A novel multimodal deep learning model for preoperative prediction of microvascular invasion and outcome in hepatocellular carcinoma

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning

An integrated radiopathomics machine learning model to predict pathological response to preoperative chemotherapy in gastric cancer

Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer with Deep Learning

Applying artificial intelligence using routine clinical data for preoperative diagnosis and prognosis evaluation of gastric cancer

The Role of Artificial Intelligence in Gastric Cancer: Surgical and Therapeutic Perspectives: A Comprehensive Review

Multi-modal artificial intelligence enhanced systematic colorectal cancer pathology evaluation.