Abstract:Background: Thyroid-associated ophthalmopathy (TAO) is the most prevalent autoimmune orbital condition, significantly impacting patients' appearance and quality of life. Early and accurate identification of active TAO along with timely treatment can enhance prognosis and reduce the occurrence of severe cases. Although the Clinical Activity Score (CAS) serves as an effective assessment system for TAO, it is susceptible to assessor experience bias. This study aimed to develop an ensemble deep learning system that combines anterior segment slit-lamp photographs of patients with facial images to simulate expert assessment of TAO. Method: The study included 156 patients with TAO who underwent detailed diagnosis and treatment at Shanxi Eye Hospital Affiliated to Shanxi Medical University from May 2020 to September 2023. Anterior segment slit-lamp photographs and facial images were used as different modalities and analyzed from multiple perspectives. Two ophthalmologists with more than 10 years of clinical experience independently determined the reference CAS for each image. An ensemble deep learning model based on the residual network was constructed under supervised learning to predict five key inflammatory signs (redness of the eyelids and conjunctiva, and swelling of the eyelids, conjunctiva, and caruncle or plica) associated with TAO, and to integrate these objective signs with two subjective symptoms (spontaneous retrobulbar pain and pain on attempted upward or downward gaze) in order to assess TAO activity. Results: The proposed model achieved 0.906 accuracy, 0.833 specificity, 0.906 precision, 0.906 recall, and 0.906 F1-score in active TAO diagnosis, demonstrating advanced performance in predicting CAS and TAO activity signs compared to conventional single-view unimodal approaches. The integration of multiple views and modalities, encompassing both anterior segment slit-lamp photographs and facial images, significantly improved the prediction accuracy of the model for TAO activity and CAS. Conclusion: The ensemble multi-view multimodal deep learning system developed in this study can more accurately assess the clinical activity of TAO than traditional methods that solely rely on facial images. This innovative approach is intended to enhance the efficiency of TAO activity assessment, providing a novel means for its comprehensive, early, and precise evaluation.

IMoVR-Net: A robust interpretable network for multi-ocular lesion recognition from TAO facial images

Detection of active and inactive phases of thyroid-associated ophthalmopathy using deep convolutional neural network

Enhanced diagnosis of thyroid-associated eye diseases based on deep learning: A novel triplet loss design strategy

HTC-retina: A hybrid retinal diseases classification model using transformer-Convolutional Neural Network from optical coherence tomography images

An ensemble deep learning diagnostic system for determining Clinical Activity Scores in thyroid-associated ophthalmopathy: integrating multi-view multimodal images from anterior segment slit-lamp photographs and facial images

Multi-resolution visual Mamba with multi-directional selective mechanism for retinal disease detection

Towards more efficient ophthalmic disease classification and lesion location via convolution transformer

Parallel Multi-Path Network for Ocular Disease Detection Inspired by Visual Cognition Mechanism

Multihead Neural Network for Multiple Segmented Images-Based Diagnosis of Thyroid-Associated Orbitopathy Activity

Multi-label classification of retinal diseases based on fundus images using Resnet and Transformer

An Intelligent Diagnostic System for Thyroid-Associated Ophthalmopathy Based on Facial Images

OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection

Clinical Interpretable Deep Learning Model for Glaucoma Diagnosis

Multiscale attention network for retinal vein occlusion classification with multicolor image

Semantic segmentation using deep learning to extract total extraocular muscles and optic nerve from orbital computed tomography images

DeepDrRVO: A GAN-auxiliary two-step masked transformer framework benefits early recognition and differential diagnosis of retinal vascular occlusion from color fundus photographs

OCTNet: A Modified Multi-Scale Attention Feature Fusion Network with InceptionV3 for Retinal OCT Image Classification

Deep Residual Network for Diagnosis of Retinal Diseases Using Optical Coherence Tomography Images

TranSegNet: Hybrid CNN-Vision Transformers Encoder for Retina Segmentation of Optical Coherence Tomography

Topology and Intersection-Union Constrained Loss Function for Multi-Region Anatomical Segmentation in Ocular Images

Retinal Photograph-based Deep Learning System for Detection of Thyroid-Associated Ophthalmopathy