Abstract:Introduction: Computer vision extracts meaning from pixelated images and holds promise in automating various clinical tasks. Convolutional neural networks (CNNs), a deep learning network used therein, have shown promise in analyzing X-ray images and joint photographs. We studied the performance of a CNN on standardized smartphone photographs in detecting inflammation in three hand joints and compared it to a rheumatologist's diagnosis. Methods: We enrolled 100 consecutive patients with inflammatory arthritis with an onset period of less than 2 years, excluding those with deformities. Each patient was examined by a rheumatologist, and the presence of synovitis in each joint was recorded. Hand photographs were taken in a standardized manner, anonymized, and cropped to include joints of interest. A ResNet-101 backbone modified for two class outputs (inflamed or not) was used for training. We also tested a hue-augmented dataset. We reported accuracy, sensitivity, and specificity for three joints: wrist, index finger proximal interphalangeal (IFPIP), and middle finger proximal interphalangeal (MFPIP), taking the rheumatologist's opinion as the gold standard. Results: The cohort consisted of 100 individuals, of which 22 of them were men, with a mean age of 49.7 (SD 12.9) years. The majority of the cohort (n = 68, 68%) had rheumatoid arthritis. The wrist (125/200, 62.5%), MFPIP (94/200, 47%), and IFPIP (83/200, 41.5%) were the three most commonly inflamed joints. The CNN achieved the highest accuracy, sensitivity, and specificity in detecting synovitis in the MFPIP (83, 77, and 88%, respectively), followed by the IFPIP (74, 74, and 75%, respectively) and the wrist (62, 90, and 21%, respectively). Discussion: We have demonstrated that computer vision was able to detect inflammation in three joints of the hand with reasonable accuracy on standardized photographs despite a small dataset. Feature engineering was not required, and the CNN worked despite a diversity in clinical diagnosis. Larger datasets are likely to improve accuracy and help explain the basis of classification. These data suggest a potential use of computer vision in screening and follow-up of inflammatory arthritis.

Applying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients

Deep learning-based classification of erosion, synovitis and osteitis in hand MRI of patients with inflammatory arthritis

Enhancing Convolutional Neural Network Scheme for Rheumatoid Arthritis Grading with Limited Clinical Data

Radiograph-based rheumatoid arthritis diagnosis via convolutional neural network

DeepRA: Predicting Joint Damage From Radiographs Using CNN with Attention

Pre-trained convolutional neural network with transfer learning by artificial illustrated images classify power Doppler ultrasound images of rheumatoid arthritis joints

Grading of Rheumatoid Arthritis on Ultrasound Images with Deep Convolutional Neural Network

Identification of rheumatoid arthritis using bone image features with radial basis function neural networks

Deep Learning Enables Automatic Detection of Joint Damage Progression in Rheumatoid Arthritis—Model Development and External Validation

Diagnosing the Severity of Knee Osteoarthritis Using Regression Scores From Artificial Intelligence Convolution Neural Networks

Deep Neural Network-based Knee Osteoarthritis Grading Using X-Rays

Computer vision detects inflammatory arthritis in standardized smartphone photographs in an Indian patient cohort

Deep Learning Models to Automate the Scoring of Hand Radiographs for Rheumatoid Arthritis

Deep learning for accurately recognizing common causes of shoulder pain on radiographs

Automatic Grading of Individual Knee Osteoarthritis Features in Plain Radiographs Using Deep Convolutional Neural Networks

Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss

Incorporating computer vision on smart phone photographs into screening for inflammatory arthritis: results from an Indian patient cohort

Rheumatoid Arthritis: Automated Scoring of Radiographic Joint Damage

Automated Coronary Artery Calcium Scoring Using Convolutional Neural Networks: Enhancing Cardiovascular Risk Assessment in Chest CT Scans

Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information