Abstract:Abstract Background Scoring colonoscopy videos of UC patients requires a high level of expertise, but even among trained expert readers there are disagreements. In clinical trials, this can negatively impact both subject selection and assessment of treatment response. Consistency among central readers might be improved by algorithms that automatically process videos to identify salient features and estimate the level of disease activity using established scoring methods. Methods We propose an end-to-end system using machine learning (ML) models to process colonoscopy videos, both shortening the time required for a human expert reader to score a video by filtering out non-informative parts and supplying a second-opinion of Mayo Endoscopic Score (MES) on each informative frame and the full video. Our dataset included videos from UC patients with a representative range of disease activity acquired at a Ukrainian hospital (n=505) and an Israeli hospital (n=227). Video annotation was performed by 12 reviewers, trained and supervised by an expert gastroenterologist (DM) with >10 years central reading experience. Out of almost 3 million frames over ~34% were classified by human reviewers as non-informative (e.g., out of focus, motion blurring, stool). The remaining informative frames were further annotated by the reviewers as containing ulcers (6%) and hence classified as MES=3, erosions (22%) as MES=2, loss of vascularity or erythema (23%) as MES= 1, or none of the above (49%) classified as MES=0. Informative frames were split to 80% training set, 10% validation set and 10% test set. The data set included low quality frames on which the reviewers could still estimate the MES. Models consisting of image processing algorithms, Convolutional Neural Networks (CNN), CNN+Long Short Term Memory (LSTM) model and classical ML classifiers were trained to filter out non-informative frames and score independently each frame and the full video for MES. Results Model performance was evaluated using Cohen’s Quadratic Weighted Kappa (QWK) with 95% Confidence Interval [95% CI]..Model agreement with humans is shown in the middle column. Inter-reviewer agreement based on a subset of ‘clear frames’ from various videos of the test set is shown in the last column. Conclusion We developed a full end-to-end pipeline for processing colonoscopy videos that estimated MES with accuracy comparable to human-human agreement. Such a model has the potential to make human readers more efficient by highlighting frames with relevant pathology. It can also aid inter-reader agreement by providing a second-opinion MES. Future work will expand model training and testing with new data sources and explore paradigms to combine the model with human readers to improve clinical trial central reading.

S1213 Visual Accuracy of Gemini Pro 1.5 and GPT-4o in Determining Ulcerative Colitis Severity Based on Endoscopic Images Using the Modified Mayo Endoscopic Score

S276 Optical Accuracy of Artificial Intelligence Large Language Models in Classifying Colorectal Polyps Based on Shape, Size, and Histology, Using Endoscopic Images

Development and Validation of a Deep Learning-Based Approach to Predict the Mayo Endoscopic Score of Ulcerative Colitis

Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images

Improving the Computer-Aided Estimation of Ulcerative Colitis Severity According to Mayo Endoscopic Score by Using Regression-Based Deep Learning

Comparative Evaluation of AI Models Such as ChatGPT 3.5, ChatGPT 4.0, and Google Gemini in Neuroradiology Diagnostics

Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

Comparing Vision-Capable Models, GPT-4 and Gemini, With GPT-3.5 on Taiwan's Pulmonologist Exam

Evaluating the Diagnostic Performance of Large Language Models in Identifying Complex Multisystemic Syndromes: A Comparative Study with Radiology Residents

Enhancing Oncological Surveillance Through Large Language Model-Assisted Analysis: A Comparative Study of GPT-4 and Gemini in Evaluating Oncological Issues From Serial Abdominal CT Scan Reports

Evaluating General Vision-Language Models for Clinical Medicine

S2190 AI vs Experts: Evaluating Chat GPT and Gemini in Gastroenterology Board Review Mastery

P123 Development of an end-to-end Mayo endoscopic score machine learning algorithm for use in clinical trials of ulcerative colitis

Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini

Diagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in "Diagnosis Please" cases

Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in Radiology's Diagnosis Please Cases

P213 ArgesMES: A high-performance, generalizable AI model for scoring Ulcerative Colitis severity from endoscopy videos

Gemini-Assisted Deep Learning Classification Model for Automated Diagnosis of High-Resolution Esophageal Manometry Images

Evaluating AI Proficiency in Nuclear Cardiology: Large Language Models take on the Board Preparation Exam

Comparison of Multi-Modal Large Language Models with Deep Learning Models for Medical Image Classification

Accurate, Robust, and Scalable Machine Abstraction of Mayo Endoscopic Subscores From Colonoscopy Reports