Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information

Shan, Juan
DOI: https://doi.org/10.1007/s00521-023-08214-8
2023-01-21
Neural Computing and Applications
Abstract:The most common form of arthritis is osteoarthritis (OA) which often affects the knee joint. Recent studies are utilizing convolution neural networks (CNNs) to automatically classify OA severity. These deep learning models are designed to analyze either X-ray images or sequences of images from magnetic resonance imaging (MRI). For the first time, we propose a fusion model that combines three different modalities (X-ray, MRI, and the patient's clinical information) into one network to improve the accuracy over the models being used individually. First, we construct the fusion architecture using two models from a previous work that were trained on a small dataset (dataset 1). This includes a classic CNN for X-ray with an accuracy of 50%, and a custom 3D model for MRI with an accuracy of 54%. When combining these two models with the clinical information, our fusion architecture increased performance to 62%. To further test the fusion architecture, we created a custom X-ray model and trained it on a larger dataset (dataset 2) which achieved an accuracy of 70% on the testing set from dataset 1. When combining the MRI (54%) model with the new X-ray model (70%) and the clinical information, the fusion model increased performance to 76%. In addition to the 5-category KL classification, the fusion model also improves the 2-category OA and non-OA classification to AUC of 0.964. The results show the proposed fusion architecture can be generalized to combine different individual models and a holistic multimodal approach can further boost OA classification performance.
computer science, artificial intelligence
What problem does this paper attempt to address?