Abstract:Background: Artificial intelligence (AI) seems promising in diagnosing pneumonia on chest x-rays (CXR), but deep learning (DL) algorithms have primarily been compared with radiologists, whose diagnosis can be not completely accurate. Therefore, we evaluated the accuracy of DL in diagnosing pneumonia on CXR using a more robust reference diagnosis. Methods: We trained a DL convolutional neural network model to diagnose pneumonia and evaluated its accuracy in two prospective pneumonia cohorts including 430 patients, for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. The performance of the DL model was compared with that of senior radiologists and emergency physicians reviewing CXRs and that of radiologists reviewing computed tomography (CT) performed concomitantly. Results: Radiologists and DL showed a similar accuracy on CXR for both cohorts (p ≥ 0.269): cohort 1, radiologist 1 75.5% (95% confidence interval 69.1-80.9), radiologist 2 71.0% (64.4-76.8), DL 71.0% (64.4-76.8); cohort 2, radiologist 70.9% (64.7-76.4), DL 72.6% (66.5-78.0). The accuracy of radiologists and DL was significantly higher (p ≤ 0.022) than that of emergency physicians (cohort 1 64.0% [57.1-70.3], cohort 2 63.0% [55.6-69.0]). Accuracy was significantly higher for CT (cohort 1 79.0% [72.8-84.1], cohort 2 89.6% [84.9-92.9]) than for CXR readers including radiologists, clinicians, and DL (all p-values < 0.001). Conclusions: When compared with a robust reference diagnosis, the performance of AI models to identify pneumonia on CXRs was inferior than previously reported but similar to that of radiologists and better than that of emergency physicians. Relevance statement: The clinical relevance of AI models for pneumonia diagnosis may have been overestimated. AI models should be benchmarked against robust reference multimodal diagnosis to avoid overestimating its performance. Trial registration: NCT02467192 , and NCT01574066 . Key point: • We evaluated an openly-access convolutional neural network (CNN) model to diagnose pneumonia on CXRs. • CNN was validated against a strong multimodal reference diagnosis. • In our study, the CNN performance (area under the receiver operating characteristics curve 0.74) was lower than that previously reported when validated against radiologists' diagnosis (0.99 in a recent meta-analysis). • The CNN performance was significantly higher than emergency physicians' (p ≤ 0.022) and comparable to that of board-certified radiologists (p ≥ 0.269).

Automatic ARDS surveillance with chest X-ray recognition using convolutional neural networks

Deep learning to detect acute respiratory distress syndrome on chest radiographs: a retrospective study with external validation

Pneumonia Detection Using Enhanced Convolutional Neural Network Model on Chest X-Ray Images

Enhancing Clinical Diagnosis With Convolutional Neural Networks: Developing High-Accuracy Deep Learning Models for Differentiating Thoracic Pathologies

An Ensemble of Global and Local-Attention Based Convolutional Neural Networks for COVID-19 Diagnosis on Chest X-ray Images

COVID-19 diagnosis from chest x-rays: developing a simple, fast, and accurate neural network

Performance Evaluation of the Deep Learning Based Convolutional Neural Network Approach for the Recognition of Chest X-Ray Images

COVID-19 Pneumonia and Influenza Pneumonia Detection Using Convolutional Neural Networks

Deep Learning-Based Detection of the Acute Respiratory Distress Syndrome: What Are the Models Learning?

Automated pneumonia detection on chest X-ray images: A deep learning approach with different optimizers and transfer learning architectures

Transfer Learning with Deep Convolutional Neural Network (CNN) for Pneumonia Detection Using Chest X-ray

AttCDCNet: Attention-enhanced Chest Disease Classification using X-Ray Images

Validating the accuracy of deep learning for the diagnosis of pneumonia on chest x-ray against a robust multimodal reference diagnosis: a post hoc analysis of two prospective studies

Detection of pneumonia in chest X‐ray image based on separable convolutional neural network

The Use of Chest Radiographs and Machine Learning Model for the Rapid Detection of Pneumonitis in Pediatric

X-COVNet: Externally Validated Model for Computer-Aided Diagnosis of Pneumonia-Like Lung Diseases in Chest X-Rays Based on Deep Transfer Learning

Multi-Task Learning with Recurrent Neural Networks for ARDS Prediction using only EHR Data: Model Development and Validation Study (Preprint)

A Deep Convolutional Neural Network for Pneumonia Detection in X-ray Images with Attention Ensemble

A new hybrid approach for pneumonia detection using chest X-rays based on ACNN-LSTM and attention mechanism

Multitask Learning With Recurrent Neural Networks for Acute Respiratory Distress Syndrome Prediction Using Only Electronic Health Record Data: Model Development and Validation Study

Automated chest screening based on a hybrid model of transfer learning and convolutional sparse denoising autoencoder