Deep Learning Models to Automate the Scoring of Hand Radiographs for Rheumatoid Arthritis

Zhiyan Bo,Laura C. Coates,Bartlomiej W. Papiez

2024-06-14

Abstract:The van der Heijde modification of the Sharp (SvdH) score is a widely used radiographic scoring method to quantify damage in Rheumatoid Arthritis (RA) in clinical trials. However, its complexity with a necessity to score each individual joint, and the expertise required limit its application in clinical practice, especially in disease progression measurement. In this work, we addressed this limitation by developing a bespoke, automated pipeline that is capable of predicting the SvdH score and RA severity from hand radiographs without the need to localise the joints first. Using hand radiographs from RA and suspected RA patients, we first investigated the performance of the state-of-the-art architectures in predicting the total SvdH score for hands and wrists and its corresponding severity class. Secondly, we leveraged publicly available data sets to perform transfer learning with different finetuning schemes and ensemble learning, which resulted in substantial improvement in model performance being on par with an experienced human reader. The best model for RA scoring achieved a Pearson's correlation coefficient (PCC) of 0.925 and root mean squared error (RMSE) of 18.02, while the best model for RA severity classification achieved an accuracy of 0.358 and PCC of 0.859. Our score prediction model attained almost comparable accuracy with experienced radiologists (PCC = 0.97, RMSE = 18.75). Finally, using Grad-CAM, we showed that our models could focus on the anatomical structures in hands and wrists which clinicians deemed as relevant to RA progression in the majority of cases.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the scoring problem of automated assessment of rheumatoid arthritis (RA) hand X - rays in clinical practice. Specifically, the paper proposes solutions to the limitations of the van der Heijde - modified Sharp (SvdH) scoring method in clinical applications. The SvdH score is a method widely used to quantify radiological damage in RA patients, but its complexity and the need for professionals limit its wide application in clinical practice, especially in disease - progress measurement.

To overcome these limitations, researchers have developed a customized automated process that can predict SvdH scores and RA severity from hand X - rays without first locating each joint. This process aims to improve the efficiency and accuracy of scoring, reduce dependence on experts, and improve the consistency and reliability of scoring.

The main contributions of the study include:
1. **Model performance investigation**: Researchers first investigated the performance of the state - of - the - art convolutional neural network (CNN) architectures in predicting the total SvdH score of the hand and wrist and its corresponding severity classification.
2. **Transfer learning and ensemble learning**: By using public data sets for transfer learning and adopting different fine - tuning strategies and ensemble learning techniques, the performance of the model was significantly improved, making it perform comparably to experienced radiologists.
3. **Model interpretability**: Using the gradient - weighted class activation mapping (Grad - CAM) technique, researchers showed that the model can focus on the anatomical structures of the hand and wrist, which are considered to be related to RA progression.

Through the above methods, researchers not only improved the accuracy of SvdH score prediction, but also enhanced the clinical applicability and interpretability of the model. Eventually, the best scoring prediction model achieved an accuracy almost comparable to that of experienced radiologists (Pearson correlation coefficient PCC = 0.97, root - mean - square error RMSE = 18.75), and in most cases was able to identify the anatomical structures related to RA progression.