Leveraging 3D convolutional neural network and 3D visible-near-infrared multimodal imaging for enhanced contactless oximetry
Wang Liao,Chen Zhang,Belmin Alić,Alina Wildenauer,Sarah Dietz-Terjung,Jose Guillermo Ortiz Sucre,Sivagurunathan Sutharsan,Christoph Schöbel,Karsten Seidl,Gunther Notni
DOI: https://doi.org/10.1117/1.JBO.29.S3.S33309
Abstract:Significance: Monitoring oxygen saturation ( SpO 2 ) is important in healthcare, especially for diagnosing and managing pulmonary diseases. Non-contact approaches broaden the potential applications of SpO 2 measurement by better hygiene, comfort, and capability for long-term monitoring. However, existing studies often encounter challenges such as lower signal-to-noise ratios and stringent environmental conditions. Aim: We aim to develop and validate a contactless SpO 2 measurement approach using 3D convolutional neural networks (3D CNN) and 3D visible-near-infrared (VIS-NIR) multimodal imaging, to offer a convenient, accurate, and robust alternative for SpO 2 monitoring. Approach: We propose an approach that utilizes a 3D VIS-NIR multimodal camera system to capture facial videos, in which SpO 2 is estimated through 3D CNN by simultaneously extracting spatial and temporal features. Our approach includes registration of multimodal images, tracking of the 3D region of interest, spatial and temporal preprocessing, and 3D CNN-based feature extraction and SpO 2 regression. Results: In a breath-holding experiment involving 23 healthy participants, we obtained multimodal video data with reference SpO 2 values ranging from 80% to 99% measured by pulse oximeter on the fingertip. The approach achieved a mean absolute error (MAE) of 2.31% and a Pearson correlation coefficient of 0.64 in the experiment, demonstrating good agreement with traditional pulse oximetry. The discrepancy of estimated SpO 2 values was within 3% of the reference SpO 2 for ∼ 80 % of all 1-s time points. Besides, in clinical trials involving patients with sleep apnea syndrome, our approach demonstrated robust performance, with an MAE of less than 2% in SpO 2 estimations compared to gold-standard polysomnography. Conclusions: The proposed approach offers a promising alternative for non-contact oxygen saturation measurement with good sensitivity to desaturation, showing potential for applications in clinical settings.