Process and Outcome for International Reliability in Sleep Scoring

Xiaozhe Zhang,Xiaosong Dong,Jan W. Kantelhardt,Jing Li,Long Zhao,Carmen Garcia,Martin Glos,Thomas Penzel,Fang Han

DOI: https://doi.org/10.1007/s11325-014-0990-0

2014-01-01

Abstract:The aim was to evaluate the inter-rater reliability in scoring sleep stages in two sleep labs in Berlin Germany and Beijing China.

What problem does this paper attempt to address?

Interrater agreement between American and Chinese sleep centers according to the 2014 AASM standard

Shujian Deng,Xin Zhang,Ying Zhang,He Gao,Eric I-Chao Chang,Yubo Fan,Yan Xu

DOI: https://doi.org/10.1007/s11325-019-01801-x

2019-01-01

Sleep And Breathing

Abstract:Objectives To determine inter-lab reliability in sleep stage scoring using the 2014 American Academy of Sleep Medicine (AASM) manual. To understand in-depth reasons for disagreement and provide suggestions for improvement. Methods This study consisted of 40 all-night polysomnographys (PSGs) from different samples. PSGs were segmented into 37,642 30-s epochs. Five doctors from China and two doctors from America scored the epochs following the 2014 AASM standard. Scoring disagreement between two centers was evaluated using Cohen’s kappa ( κ ). After visual inspection of PSGs of deviating scorings, potential disagreement reasons were analyzed. Results Inter-lab reliability yielded a substantial degree ( κ = 0.75 ± 0.01). Scoring for stage W ( κ = 0.89) and R ( κ = 0.87) achieved the highest agreement, while stage N1 ( κ = 0.45) reflected the lowest. Considering the relative disagreement ratio, N2-N3 (22.09%), W-N1 (19.68%), and N1-N2 (18.75%) were the most frequent combinations of discrepancy. American and Chinese doctors showed certain characteristics in the scoring of discrepancy combination W-N1, N1-N2, and N2-N3. There are seven reasons for disagreement, namely “on-threshold characteristic” (29.21%), “context influence” (18.06%), “characteristic identification difficulty” (8.81%), “arousal-wake confusion” (7.57%), “derivation inconsistence” (2.15%), “on-borderline characteristic” (0.92%), and “misrecognition” (33.27%). Conclusions This study demonstrated the sleep stage scoring agreement of the 2014 AASM manual and explored potential sources of labeling ambiguity. Improvement measures were suggested accordingly to help remove ambiguity for scorers and improve scoring reliability at the international level.
Polysomnography scoring-related training and quantitative assessment for improving interscorer agreement

Ying-Siou Liao,Meng-Chun Wu,Cheng-Xue Li,Wen-Kuei Lin,Cheng-Yu Lin,Sheng-Fu Liang

DOI: https://doi.org/10.5664/jcsm.10852

2024-02-01

Abstract:Study objectives: To efficiently improve the scoring competency of scorers with varying levels of experience across regions in Taiwan, we developed a training program with a cloud-based polysomnography scoring platform to evaluate and improve interscorer agreement. Methods: A total of 70 scorers from 34 sleep centers in Taiwan (job tenure: 0.5-39.0 years) completed a scoring test. All scorers scored a 742-epoch (30 s/epoch) overnight polysomnography recording of a patient with a moderate apnea-hypopnea index. Subsequently, 8 scoring experts delivered 8 interactive online lectures (each lasting 30 minutes). The training program included identifying scoring weaknesses, highlighting the latest scoring rules, and providing physicians' perspectives. Afterward, the scorers completed the second scoring test on the same participant. Changes in agreement from the first to second scoring test were identified. Sleep staging, sleep parameters, and respiratory events were considered for evaluating scoring agreement. Results: The scorers' agreement in overall sleep stage scoring significantly increased from 74.6 to 82.3% (median score). The proportion of scorers with an agreement of ≥ 80% increased from 20.0% (14/70) to 58.6% (41/70) after the online training program. In addition, the scorers' agreement in overall respiratory-event scoring increased to 88.8% (median score) after training. The scorers with a job tenure of 2.0-4.9 years exhibited the highest level of improvement in overall sleep staging (their median agreement increased from 72.8 to 84.9%; P < .001). Conclusions: Our interactive online training program efficiently targeted the scorers' scoring weaknesses identified in the first scoring test, leading to substantial improvements in scoring proficiency. Citation: Liao Y-S, Wu M-C, Li C-X, Lin W-K, Lin C-Y, Liang S-F. Polysomnography scoring-related training and quantitative assessment for improving interscorer agreement. J Clin Sleep Med. 2024;20(2):271-278.
Current status and prospects of automatic sleep stages scoring: Review

Maksym Gaiduk,Ángel Serrano Alarcón,Ralf Seepold,Natividad Martínez Madrid

DOI: https://doi.org/10.1007/s13534-023-00299-3

2023-07-10

Biomedical Engineering Letters

Abstract:The scoring of sleep stages is one of the essential tasks in sleep analysis. Since a manual procedure requires considerable human and financial resources, and incorporates some subjectivity, an automated approach could result in several advantages. There have been many developments in this area, and in order to provide a comprehensive overview, it is essential to review relevant recent works and summarise the characteristics of the approaches, which is the main aim of this article. To achieve it, we examined articles published between 2018 and 2022 that dealt with the automated scoring of sleep stages. In the final selection for in-depth analysis, 125 articles were included after reviewing a total of 515 publications. The results revealed that automatic scoring demonstrates good quality (with Cohen's kappa up to over 0.80 and accuracy up to over 90%) in analysing EEG/EEG + EOG + EMG signals. At the same time, it should be noted that there has been no breakthrough in the quality of results using these signals in recent years. Systems involving other signals that could potentially be acquired more conveniently for the user (e.g. respiratory, cardiac or movement signals) remain more challenging in the implementation with a high level of reliability but have considerable innovation capability. In general, automatic sleep stage scoring has excellent potential to assist medical professionals while providing an objective assessment.
Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: hypnodensity based on multiple expert scorers and auto-scoring

Jessie P Bakker,Marco Ross,Andreas Cerny,Ray Vasko,Edmund Shaw,Samuel Kuna,Ulysses J Magalang,Naresh M Punjabi,Peter Anderer

DOI: https://doi.org/10.1093/sleep/zsac154

IF: 6.313

2022-07-03

SLEEP

Abstract:Abstract Study Objectives To quantify the amount of sleep stage ambiguity across expert scorers and to validate a new auto-scoring platform against sleep staging performed by multiple scorers. Methods We applied a new auto-scoring system to three datasets containing 95 PSGs scored by 6–12 scorers, to compare sleep stage probabilities (hypnodensity; i.e. the probability of each sleep stage being assigned to a given epoch) as the primary output, as well as a single sleep stage per epoch assigned by hierarchical majority rule. Results The percentage of epochs with 100% agreement across scorers was 46 ± 9%, 38 ± 10% and 32 ± 9% for the datasets with 6, 9, and 12 scorers, respectively. The mean intra-class correlation coefficient between sleep stage probabilities from auto- and manual-scoring was 0.91, representing excellent reliability. Within each dataset, agreement between auto-scoring and consensus manual-scoring was significantly higher than agreement between manual-scoring and consensus manual-scoring (0.78 vs. 0.69; 0.74 vs. 0.67; and 0.75 vs. 0.67; all p < 0.01). Conclusions Analysis of scoring performed by multiple scorers reveals that sleep stage ambiguity is the rule rather than the exception. Probabilities of the sleep stages determined by artificial intelligence auto-scoring provide an excellent estimate of this ambiguity. Compared to consensus manual-scoring, sleep staging derived from auto-scoring is for each individual PSG noninferior to manual-scoring meaning that auto-scoring output is ready for interpretation without the need for manual adjustment.

neurosciences,clinical neurology
Exploring the Potential of a New Wearable Sleep Monitoring Device for Clinical Application

Xin Li,Min Li,Mei Tian,Qi Liu,Xiaomeng Zhou,Hu Liu,Rui Li,Zhenzhong Li,Hui Dong,Lijing Jia,Yaling Liu

DOI: https://doi.org/10.1016/j.bspc.2024.106856

IF: 5.1

2025-01-01

Biomedical Signal Processing and Control

Abstract:OBJECTIVES Sleep is critical to health and quality of life, and the gold-standard assessment method for sleep disorders--polysomnography (PSG)--has struggled to become widely used because of its inherent shortcomings, which portable sleep-monitoring devices are expected to improve. This study aims to validate a new EEG-, EOG-, and EMG-based portable sleep-monitoring device against conventional PSG to explore its feasibility for clinical use. Methods Twenty participants completed overnight sleep monitoring using conventional polysomnography and the Wearable PolysomnogramTM (WPSG-I). The similarity of the raw signals recorded by the two devices and the agreement rate of their sleep-staging results were compared. Results The raw electroencephalogram (EEG) signals recorded by the two devices were highly similar, and the agreement rates between WPSG-I manual sleep staging and automatic sleep staging with conventional PSG manual sleep staging were 95.8 % (kappa = 0.92) and 89.7 % (kappa = 0.80), respectively. Conclusion The agreement rate of sleep staging results between the two devices was high. The results of this study demonstrate the validity and feasibility of the WPSG-I for sleep research and clinical applications.
1078 Performance of USleep Algorithm to a Better Than “Gold-Standard” Polysomnogram Validation Data Set

Umaer Hanif,Guillaume Jubien,Alyssa Cairns,Tammie Radke,Vincent Mysliwiec

DOI: https://doi.org/10.1093/sleep/zsae067.01078

IF: 6.313

2024-04-20

SLEEP

Abstract:Abstract Introduction Algorithm based PSG scoring is increasingly used in clinical practice and research. Currently, there is no accepted PSG validation data set to ensure algorithms are developed from an accepted standard; thus algorithm-based scoring has inherent inaccuracies. The purpose of this study was to evaluate the inter-rater agreement between three experienced sleep technicians to develop a PSG validation data set with a high interscorer reliability and compare performance of the USleep algorithm to the PSG validation data set. Methods One-hundred de-identified PSGs from a clinical database with 55 males and 45 females were independently scored by three different technicians and each record was quality controlled by a lead technician. The data set encompassed all sleep-related events: sleep stages, apneas, hypopneas, desaturations, arousals, and PLMs. Consensus annotations were computed when at least two technicians agreed upon a given event. The performance of each technician was compared against the consensus annotations. USleep was evaluated on the validation data set and the performance with respect to sleep stages was compared against the consensus annotations. Results The inter-rater agreement was 96.0% for sleep stages across all epochs, which did not greatly differ between N1 (88.3%), N2 (97.3%), N3 (94.2%) and REM sleep (98.1%). The inter-rater agreement across all records was 88.9% for arousals, 84.0% for obstructive apneas, 80.2% for central apneas, 76.4% for mixed apneas, 89.4% for hypopneas, 94.9% for desaturations, and 81.3% for PLMs. Comparing USleep to the consensus yielded an accuracy of 78.3%, with differences between N1 (46.2%) and N2 (76.8%), but not N3 (99.3%) and REM sleep (92.1%). Conclusion This data set has high accuracy for sleep stages. The USleep algorithm showed poor performance for N1 and moderate for N2, but high for N3 and REM. Frequently scored PSG measures to include arousals, hypopneas and obstructive apneas had high degrees of accuracy; however, this decreased for central and mixed apneas. This dataset can serve as a benchmark for developing and validating algorithms in PSG scoring for sleep stages and certain PSG measures. Further development is required to provide basis for a comprehensive PSG validation data set. Support (if any)

neurosciences,clinical neurology
0606 A Paradigm for Testing the Accuracy of Digital Sleep Staging Systems

Bethany Gerardy,Heather Tomson,Samuel Kuna,Allan Pack,James Walsh,Clete Kushida,Magdy Younes

DOI: https://doi.org/10.1093/sleep/zsac079.603

IF: 6.313

2022-06-04

SLEEP

Abstract:IntroductionDespite evidence showing that agreement between human and some automatic staging systems is generally comparable to agreement between human scorers, automated scoring is rarely used in clinical practice, even though it offers time savings and consistency. We propose a paradigm for testing digital systems that reveals their true accuracy vs. highly experienced academic scorers. As an example of a digital method to be tested, we used Michele Sleep Scoring (abbreviated:Digital).Methods70 PSGs were scored by 6 experienced technologists from 3 academic centers. Staging results were compared to digital staging results using an epoch-by-epoch approach. For each PSG we carried out 6 cycles of comparisons. Each cycle consisted of two steps, one comparing one scorer (tested scorer) with the scoring of the five remaining scorers (judges), and one comparing Digital as the tested scorer with the same 5 judges. Error 1 was assessed when all judges disagreed with the tested scorer but there was disagreement between the judges. Error 2 was assigned when all judges disagreed with the tested scorer but agreed unanimously on the stage. For each PSG the number of epochs with types 1 and 2 errors was counted for each scorer (n=6 scorers) and for Digital. Results of all 70 PSGs were pooled, and percent of types 1 and 2 errors is reported for all scorers and Digital.Results70 PSGs (females aged 51.1 ± 4.2 years) were evaluated. Average times in different sleep stages (manual scoring) were 43±18, 244±47, 30±21, and 81±25 minutes for stages N1, N2, N3 and REM, respectively. TST was 398±52 minutes, and sleep efficiency was 84±8%. There was a total of 65,053 epochs scored by each scorer and Digital. The average percent of type 1 errors made by scorers for all epochs was 6.4% (0-33.2) vs. 7.8% (1.68-26.6) made by Digital. The average percent of type 2 errors made by scorers for all epochs was 3.9% (0-28.6) vs. 4.3% (0-17.3) made by Digital.ConclusionThis study provides an objective way of testing the accuracy of automated scoring systems and supports evidence that the accuracy of Michele Sleep Scoring is comparable to manual scoring.Support (If Any)None

neurosciences,clinical neurology
Multi-modality of Polysomnography Signals' Fusion for Automatic Sleep Scoring

Rui Yan,Chi Zhang,Karen Spruyt,Lai Wei,Zhiqiang Wang,Lili Tian,Xueqiao Li,Tapani Ristaniemi,Jihui Zhang,Fengyu Cong

DOI: https://doi.org/10.1016/j.bspc.2018.10.001

IF: 5.1

2019-01-01

Biomedical Signal Processing and Control

Abstract:Objective: The study aims to develop an automatic sleep scoring method by fusing different polysomnography (PSG) signals and further to investigate PSG signals' contribution to the scoring result. Methods: Eight combinations of four modalities of PSG signals, namely electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), and electrocardiogram (ECG) were considered to find the optimal fusion of PSG signals. A total of 232 features, covering statistical characters, frequency characters, time-frequency characters, fractal characters, entropy characters and nonlinear characters, were derived from these PSG signals. To select the optimal features for each signal fusion, four widely used feature selection methods were compared. At the classification stage, five different classifiers were employed to evaluate the validity of the features and to classify sleep stages. Results: For the database in the present study, the best classifier, random forest, realized the optimal consistency of 86.24% with the sleep macrostructures scored by the technologists trained at the Sleep Center. The optimal accuracy was achieved by fusing four modalities of PSG signals. Specifically, the top twelve features in the optimal feature set were respectively EEG features named zero-crossings, spectral edge, relative power spectral of theta, Petrosian fractal dimension, approximate entropy, permutation entropy and spectral entropy, and EOG features named spectral edge, approximate entropy, permutation entropy and spectral entropy, and the mutual information between EEG and submental EMG. In addition, ECG features (e.g. Petrosian fractal dimension, zero-crossings, mean value of R amplitude and permutation entropy) were useful for the discrimination among W, S1 and R. Conclusions: Through exploring the different fusions of multi-modality signals, the present study concluded that the multi-modality of PSG signals' fusion contributed to higher accuracy, and the optimal feature set was a fusion of multiple types of features. Besides, compared with manual scoring, the proposed automatic scoring methods were cost-effective, which would alleviate the burden of the physicians, speed up sleep scoring, and expedite sleep research. (C) 2018 The Author(s). Published by Elsevier Ltd.
The Reliability and Validity of Sleep Dysfunction Rating Scale in Patients with Insomnia

XIAO Wei-Dong,LIU Ping,MA Hong

DOI: https://doi.org/10.3321/j.issn:1000-6729.2007.01.013

2007-01-01

Abstract:Objective:Psychometric evaluation of Sleep Dysfunction Rating Scale (SDRS) for insomnia. Methods:The study sample consisted of 215 patients with insomnia (CCMD-3).The length of the test-retest interval was either 3 to 7 days. Results:In insomnic patients, the overall SDRS global score correlation coefficient for test-retest reliability was 0.89(P<0.001). The items of SDRS homogeneity were high with Cronbach's alpha of 0.85～0.88. Validity analyses showed high correlations between SDRS and CGI-S (clinical global impression scale-severity) data. The factor analysis showed that the factor loading and structure, which matched the structure of the scale approximately. The range of factor loading was between 0.73 to 0.95.Conclusion:The SDRS has a high reliability and a good validity in patients with insomnia.
The Effect of Different EEG Derivations on Sleep Staging in Rats: the Frontal Midline–parietal Bipolar Electrode for Sleep Scoring

Guangzhan Fang,Chunpeng Zhang,Yang Xia,Yongxiu Lai,Tiejun Liu,Zili You,Dezhong Yao

DOI: https://doi.org/10.1088/0967-3334/30/7/005

2009-01-01

Physiological Measurement

Abstract:Most sleep-staging research has focused on developing and optimizing algorithms for sleep scoring, but little attention has been paid to the effect of different electroencephalogram (EEG) derivations on sleep staging. To explore the possible effects of EEG derivations, an automatic computer method was established and confirmed by agreement analysis between the computer and two independent raters, and four fronto-parietal bipolar leads were compared for sleep scoring in rats. The results demonstrated that different bipolar electrodes have significantly different sleep-staging accuracies, and that the optimal frontal electrode for sleep scoring is located at the anterior midline.
Refining sleep staging accuracy: Transfer learning coupled with scorability models

Wolfgang Ganglberger,Samaneh Nasiri,Haoqi Sun,Soriul Kim,Chol Shin,M Brandon Westover,Robert J Thomas

DOI: https://doi.org/10.1093/sleep/zsae202

IF: 6.313

2024-08-31

SLEEP

Abstract:Study objectives: This study aimed to 1) improve sleep staging accuracy through transfer learning, to achieve or exceede human inter-expert agreement; 2) introduce a scorability model to assess the quality and trustworthiness of automated sleep staging. Methods: A deep neural network (base model) was trained on a large multi-site polysomnography (PSG) dataset from the United States. Transfer learning was used to calibrate the model to a reduced montage and limited samples from the Korean Genome and Epidemiology Study (KoGES) dataset. Model performance was compared to inter-expert reliability among three human experts. A scorability assessment was developed to predict the agreement between the model and human experts. Results: Initial sleep staging by the base model showed lower agreement with experts (κ=0.55) compared to inter-expert agreement (κ=0.62). Calibration with 324 randomly sampled training cases matched expert agreement levels. Further targeted sampling improved performance, with models exceeding inter-expert agreement (κ=0.70). The scorability assessment, combining biosignal quality and model confidence features, predicted model-expert agreement moderately well (R²=0.42). Recordings with higher scorability scores demonstrated greater model-expert agreement than inter-expert agreement. Even with lower scorability scores, model performance was comparable to inter-expert agreement. Conclusions: Fine-tuning a pre-trained neural network through targeted transfer learning significantly enhances sleep staging performance for an atypical montage, achieving and surpassing human expert agreement levels. The introduction of a scorability assessment provides a robust measure of reliability, ensuring quality control and enhancing the practical application of the system before deployment. This approach marks an important advancement in automated sleep analysis, demonstrating the potential for AI to exceed human performance in clinical settings.
Large-Scale Automated Sleep Staging

Haoqi Sun,Jian Jia,Balaji Goparaju,Guang-Bin Huang,Olga Sourina,Matt Travis Bianchi,M Brandon Westover

DOI: https://doi.org/10.1093/sleep/zsx139

IF: 6.313

2017-09-26

SLEEP

Abstract:Study Objectives: Automated sleep staging has been previously limited by a combination of clinical and physiological heterogeneity. Both factors are in principle addressable with large data sets that enable robust calibration. However, the impact of sample size remains uncertain. The objectives are to investigate the extent to which machine learning methods can approximate the performance of human scorers when supplied with sufficient training cases and to investigate how staging performance depends on the number of training patients, contextual information, model complexity, and imbalance between sleep stage proportions.Methods: A total of 102 features were extracted from six electroencephalography (EEG) channels in routine polysomnography. Two thousand nights were partitioned into equal (n = 1000) training and testing sets for validation. We used epoch-by-epoch Cohen's kappa statistics to measure the agreement between classifier output and human scorer according to American Academy of Sleep Medicine scoring criteria.Results: Epoch-by-epoch Cohen's kappa improved with increasing training EEG recordings until saturation occurred (n = ~300). The kappa value was further improved by accounting for contextual (temporal) information, increasing model complexity, and adjusting the model training procedure to account for the imbalance of stage proportions. The final kappa on the testing set was 0.68. Testing on more EEG recordings leads to kappa estimates with lower variance.Conclusion: Training with a large data set enables automated sleep staging that compares favorably with human scorers. Because testing was performed on a large and heterogeneous data set, the performance estimate has low variance and is likely to generalize broadly.

neurosciences,clinical neurology
RASS: A Portable Real-time Automatic Sleep Scoring System

Jin Zhang,Dawei Chen,Jianhui Zhao,Mincong He,Yuanpeng Wang,Qian Zhang

DOI: https://doi.org/10.1109/RTSS.2012.63

2012-01-01

Abstract:It is a well known fact that the quality of sleep is an important factor in health-related quality of life (HRQoL), and people could prevent potential problems by tracking the quality of their sleep. Unfortunately sleep scoring, which is a systematic way to address the sleep staging as well as the scoring of arousals, respiratory, cardiac, and movement events, is usually conducted with specialized equipment which is expensive and operated by specialists in dedicated sleep centers. Related research studies and products (e.g. ZEO) tried to solve this problem, but they either used multiple probes that cause discomfort to the patient, or could not score in real time. In this paper, we design and implement RASS, a portable Real-time Automatic Sleep Scoring system. RASS only requires one probe, which is inexpensive and, as a result, may be used at home or during travel. RASS accurately scores the sleeping state and detects sleep apnea in real-time based on the sensing results of pulse, blood oxygen, activity, sound and light signals. An alarm will be generated when a severely abnormal sleep state is detected. RASS has been tested with 48 patients, and the test results show that RASS could achieve higher than 84% accuracy.
Sleep Staging Using Noncontact-Measured Vital Signs

Zixia Wang,Shuai Zha,Baoxian Yu,Pengbin Chen,Zhiqiang Pang,Han Zhang

DOI: https://doi.org/10.1155/2022/2016598

2022-07-09

Journal of Healthcare Engineering

Abstract:As a physiological phenomenon, sleep takes up approximately 30% of human life and significantly affects people's quality of life. To assess the quality of night sleep, polysomnography (PSG) has been recognized as the gold standard for sleep staging. The drawbacks of such a clinical device, however, are obvious, since PSG limits the patient's mobility during the night, which is inconvenient for in-home monitoring. In this paper, a noncontact vital signs monitoring system using the piezoelectric sensors is deployed. Using the so-designed noncontact sensing system, heartbeat interval (HI), respiratory interval (RI), and body movements (BM) are separated and recorded, from which a new dimension of vital signs, referred to as the coordination of heartbeat interval and respiratory interval (CHR), is obtained. By extracting both the independent features of HI, RI, and BM and the coordinated features of CHR in different timescales, Wake-REM-NREM sleep staging is performed, and a postprocessing of staging fusion algorithm is proposed to refine the accuracy of classification. A total of 17 all-night recordings of noncontact measurement simultaneous with PSG from 10 healthy subjects were examined, and the leave-one-out cross-validation was adopted to assess the performance of Wake-REM-NREM sleep staging. Taking the gold standard of PSG as reference, numerical results show that the proposed sleep staging achieves an averaged accuracy and Cohen's Kappa index of 82.42% and 0.63, respectively, and performs robust to subjects suffering from sleep-disordered breathing.

health care sciences & services
Assessing Measurement Properties of a Simplified Chinese Version of Sleep Condition Indicator (SCI-SC) in Community Residents

Runtang Meng,Esther Yuet Ying Lau,Karen Spruyt,Christopher B. Miller,Lu Dong

DOI: https://doi.org/10.3390/bs12110433

2022-11-03

Behavioral Sciences

Abstract:Background: The study aimed to assess the measurement properties of a simplified Chinese version of the Sleep Condition Indicator (SCI-SC) in the community. Methods: A psychometric evaluation through an observational cross-sectional survey design was conducted. Community residents (N = 751) in Hangzhou, China completed the SCI-SC and the simplified Chinese version of the Sleep Quality Questionnaire (SQQ) in July 2021. Data were randomly split into a development sample (N = 375) for model development by exploratory factor analysis (EFA) and a holdout sample (N = 376) for validation by confirmatory factor analysis (CFA). Multi-group CFA (MGCFA) was used to assess configural, metric, scalar, and strict measurement invariance across gender, age, marital status, body mass index (BMI), napping habits, generic exercise, hobby, and administered survey. Moreover, statistical analyses were performed to determine the reliability (alpha and omega) and construct validity of the instrument. Results: Both factor analyses showed a stable solution with two dimensions of Sleep Pattern and Sleep-Related Impact. Good structural validity, robust internal consistency, and construct validity with the SQQ were demonstrated. There was evidence of strict invariance across gender, BMI, napping habits, generic exercise, hobby, and administered survey subgroups, but only metric and scalar invariances were established across age and marital status groups, respectively. Conclusions: The SCI-SC demonstrated promising psychometric properties, with high SQQ concordance and consistent structure of the original version. The SCI-SC can be used by sleep researchers as well as healthcare professionals in various contexts in detecting risks for insomnia disorder in the community.
Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep staging

Antoine Guillot,Fabien Sauvet,Emmanuel H During,Valentin Thorey

DOI: https://doi.org/10.48550/arXiv.1911.03221

2020-04-27

Abstract:Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85 % only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches. We also developed and benchmarked a new deep learning method, SimpleSleepNet, inspired by current state-of-the-art. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9 % vs 86.8 % on average for human scorers on DOD-H, and an F1 of 88.3 % vs 84.8 % on DOD-O. Our study highlights that using state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Consideration could be made to use automated approaches in the clinical setting.

Quantitative Methods,Machine Learning,Signal Processing
Inter-database validation of a deep learning approach for automatic sleep scoring

Diego Alvarez-Estevez,Roselyne M. Rijsman

DOI: https://doi.org/10.1371/journal.pone.0256111

IF: 3.7

2021-08-16

PLoS ONE

Abstract:Study objectives Development of inter-database generalizable sleep staging algorithms represents a challenge due to increased data variability across different datasets. Sharing data between different centers is also a problem due to potential restrictions due to patient privacy protection. In this work, we describe a new deep learning approach for automatic sleep staging, and address its generalization capabilities on a wide range of public sleep staging databases. We also examine the suitability of a novel approach that uses an ensemble of individual local models and evaluate its impact on the resulting inter-database generalization performance. Methods A general deep learning network architecture for automatic sleep staging is presented. Different preprocessing and architectural variant options are tested. The resulting prediction capabilities are evaluated and compared on a heterogeneous collection of six public sleep staging datasets. Validation is carried out in the context of independent local and external dataset generalization scenarios. Results Best results were achieved using the CNN_LSTM_5 neural network variant. Average prediction capabilities on independent local testing sets achieved 0.80 kappa score. When individual local models predict data from external datasets, average kappa score decreases to 0.54. Using the proposed ensemble-based approach, average kappa performance on the external dataset prediction scenario increases to 0.62. To our knowledge this is the largest study by the number of datasets so far on validating the generalization capabilities of an automatic sleep staging algorithm using external databases. Conclusions Validation results show good general performance of our method, as compared with the expected levels of human agreement, as well as to state-of-the-art automatic sleep staging methods. The proposed ensemble-based approach enables flexible and scalable design, allowing dynamic integration of local models into the final ensemble, preserving data locality, and increasing generalization capabilities of the resulting system at the same time.

multidisciplinary sciences
An Automatic Sleep Scoring Toolbox: Multi-modality of Polysomnography Signals’ Processing

Rui Yan,Fan Li,Xiaoyu Wang,Tapani Ristaniemi,Fengyu Cong

DOI: https://doi.org/10.5220/0007925503010309

2019-01-01

Abstract:Sleep scoring is a fundamental but time-consuming process in any sleep laboratory. To speed up the process of sleep scoring without compromising accuracy, this paper develops an automatic sleep scoring toolbox with the capability of multi-signal processing. It allows the user to choose signal types and the number of target classes. Then, an automatic process containing signal pre-processing, feature extraction, classifier training (or prediction) and result correction will be performed. Finally, the application interface displays predicted sleep structure, related sleep parameters and the sleep quality index for reference. To improve the identification accuracy of minority stages, a layer-wise classification strategy is proposed according to the signal characteristics of sleep stages. The context of the current stage is taken into consideration in the correction phase by employing a Hidden Markov Model to study the transition rules of sleep stages in the training dataset. These transition rules will be used for logic classification results. The performance of proposed toolbox has been tested on 100 subjects with an average accuracy of 85.76%. The proposed automatic scoring toolbox would alleviate the burden of the physicians, speed up sleep scoring, and expedite sleep research.
A longitudinal examination of the measurement properties and invariance of the Sleep Condition Indicator in Chinese healthcare students

Runtang Meng,Yiwei Ying,Yi Luo,Mengyi Huang,Christopher B Miller,Yuhuan Xie,Yuxin Jia,Lianxia Fan,Wukang Chen,Jiayu Yi,Nongnong Yang,Jiale Xu,Chen Jiang,Liping Lu,Haiyan Ma,Karen Spruyt,Esther Yuet Ying Lau

DOI: https://doi.org/10.1186/s12888-024-05844-7

IF: 4.144

2024-07-22

BMC Psychiatry

Abstract:Background: The Sleep Condition Indicator (SCI), an insomnia measurement tool based on the updated Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria with sound psychometric properties when applied in various populations, was evaluated here among healthcare students longitudinally, to demonstrate its measurement properties and invariance in this particularly high-risk population. Methods: Healthcare students of a Chinese university were recruited into this two-wave longitudinal study, completing the simplified Chinese version of the SCI (SCI-SC), Chinese Regularity, Satisfaction, Alertness, Timing, Efficiency, Duration (RU_SATED-C) scale, Chinese Patient Health Questionnaire-4 (PHQ-4-C), and sociodemographic variables questionnaire (Q-SV) between September and November 2022. Structural validity, measurement invariance (MI), convergent and discriminant validity, internal consistency, and test-retest reliability of the SCI-SC were examined. Subgroups of gender, age, home location, part-time job, physical exercise, and stress-coping strategy were surveyed twice to test cross-sectional and longitudinal MI. Results: We identified 343 valid responses (62.9% female, mean age = 19.650 ± 1.414 years) with a time interval of seven days. The two-factor structure was considered satisfactory (comparative fit index = 0.953-0.989, Tucker-Lewis index = 0.931-0.984, root means square error of approximation = 0.040-0.092, standardized root mean square residual = 0.039-0.054), which mostly endorsed strict invariance except for part-time job subgroups, hence establishing longitudinal invariance. The SCI-SC presented acceptable convergent validity with the RU_SATED-C scale (r ≥ 0.500), discriminant validity with the PHQ-4-C (0.300 ≤ r < 0.500), internal consistency (Cronbach's alpha = 0.811-0.835, McDonald's omega = 0.805-0.832), and test-retest reliability (intraclass correlation coefficient = 0.829). Conclusion: The SCI-SC is an appropriate screening instrument available for assessing insomnia symptoms among healthcare students, and the promising measurement properties provide additional evidence about validity and reliability for detecting insomnia in healthcare students.
[Automatic Sleep Stage Classification Based on an Improved K-means Clustering Algorithm].

Shuyuan Xiao,Bei Wang,Jian Zhang,Qunfeng Zhang,Junzhong Zou

DOI: https://doi.org/10.7507/1001-5515.20160137

2016-01-01

Abstract:Sleep stage scoring is a hotspot in the field of medicine and neuroscience.Visual inspection of sleep is laborious and the results may be subjective to different clinicians.Automatic sleep stage classification algorithm can be used to reduce the manual workload.However,there are still limitations when it encounters complicated and changeable clinical cases.The purpose of this paper is to develop an automatic sleep staging algorithm based on the characteristics of actual sleep data.In the proposed improved K-means clustering algorithm,points were selected as the initial centers by using a concept of density to avoid the randomness of the original K-means algorithm.Meanwhile,the cluster centers were updated according to the‘Three-Sigma Rule’during the iteration to abate the influence of the outliers.The proposed method was tested and analyzed on the overnight sleep data of the healthy persons and patients with sleep disorders after continuous positive airway pressure(CPAP)treatment.The automatic sleep stage classification results were compared with the visual inspection by qualified clinicians and the averaged accuracy reached 76%.With the analysis of morphological diversity of sleep data,it was proved that the proposed improved K-means algorithm was feasible and valid for clinical practice.

Process and Outcome for International Reliability in Sleep Scoring

Interrater agreement between American and Chinese sleep centers according to the 2014 AASM standard

Polysomnography scoring-related training and quantitative assessment for improving interscorer agreement

Current status and prospects of automatic sleep stages scoring: Review

Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: hypnodensity based on multiple expert scorers and auto-scoring

Exploring the Potential of a New Wearable Sleep Monitoring Device for Clinical Application

1078 Performance of USleep Algorithm to a Better Than “Gold-Standard” Polysomnogram Validation Data Set

0606 A Paradigm for Testing the Accuracy of Digital Sleep Staging Systems

Multi-modality of Polysomnography Signals' Fusion for Automatic Sleep Scoring

The Reliability and Validity of Sleep Dysfunction Rating Scale in Patients with Insomnia

The Effect of Different EEG Derivations on Sleep Staging in Rats: the Frontal Midline–parietal Bipolar Electrode for Sleep Scoring

Refining sleep staging accuracy: Transfer learning coupled with scorability models

Large-Scale Automated Sleep Staging

RASS: A Portable Real-time Automatic Sleep Scoring System

Sleep Staging Using Noncontact-Measured Vital Signs

Assessing Measurement Properties of a Simplified Chinese Version of Sleep Condition Indicator (SCI-SC) in Community Residents

Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep staging

Inter-database validation of a deep learning approach for automatic sleep scoring

An Automatic Sleep Scoring Toolbox: Multi-modality of Polysomnography Signals’ Processing

A longitudinal examination of the measurement properties and invariance of the Sleep Condition Indicator in Chinese healthcare students

[Automatic Sleep Stage Classification Based on an Improved K-means Clustering Algorithm].