Abstract:Background: Evaluating competency acquisition during residency training is crucial. The Anesthesiology Milestones have been implemented in the United States. The China Consortium of Elite Teaching Hospitals for Residency Education has also developed the Chinese Resident Core Competency Milestone Evaluation System. Despite this, Milestones tailored for anesthesiology have yet to be implemented in China. To address this gap, we have developed Chinese Anesthesiology Milestones. This study aims to assess the reliability and validity of the Chinese Anesthesiology Milestones and their correlation with objective examinations. Methods: In this single-center cross-sectional study, we included anesthesia residents enrolled in the standardized residency training program at our hospital during the academic year 2021 to 2022. The Chinese Anesthesiology Milestones were developed based on the American version of Anesthesiology Milestones 2.0 and the Chinese Resident Core Competency Milestone Evaluation System using the Delphi method. The Delphi panel comprised a diverse group, including education administrators, faculty from teaching hospitals, and anesthesia residents. Five attending anesthesiologists independently assessed the levels achieved by each anesthesia resident based on the Chinese Anesthesiology Milestones. Subsequently, they collaboratively discussed the ratings for each resident until a consensus was reached. The interrater reliability, internal consistency, and construct validity were assessed using Kendall's coefficient, Cronbach's α coefficient/ composite reliability, and average variance extracted, respectively. Higher values indicated better reliability or validity. The correlation between Milestone ratings and objective examination scores, including written examinations and Objective Structured Clinical Examinations, were analyzed using Pearson correlation. Results: The Chinese Anesthesiology Milestones encompassed 6 competencies, including professionalism, medical knowledge and technical skills, patient care, interpersonal and communication skills, teaching ability, and life-long learning. Milestone evaluation data were available and analyzed from 66 residents. The Kendall's coefficient of concordance among raters ranged from 0.799 (95% confidence interval [CI], 0.793-0.918) to 0.942 (95% CI, 0.934-0.982). The average variance extracted, composite reliability, and Cronbach's α coefficient ranged from 0.782 to 0.920, 0.935 to 0.980, and 0.916 to 0.978, respectively. Correlations between objective examination scores and related Milestone subcompetencies were as follows: written examinations: r = 0.52 (95% CI, 0.22-0.71), technical skills stations: r = 0.51 (95% CI, 0.21-0.71), the oral test station: r = 0.66 (95% CI, 0.45-0.79), and the standardized patient station: r = 0.61 (95% CI, 0.36-0.76). Conclusions: The Chinese Anesthesiology Milestones demonstrated satisfactory interrater reliability, internal consistency, construct validity, and correlation with objective examination scores within our hospital.

Differences between resident self-assessments and faculty- assessments on Anesthesiology Milestones and associated factors: a cross-sectional study

Chinese Anesthesiology Milestones in Resident Evaluation: Reliability, Validity, and Correlation with Objective Examination Scores: a Cross-sectional Study

The role of anesthesiologists’ perceived self-efficacy in anesthesia-related adverse events

Multisource feedback in professionalism for anesthesia residents

Evaluating dental student self‐assessment across preclinical anatomy waxing and operative exercises

Do You See What I See? A Comparison of CCC and Self-Assigned Milestones Across Military Medical Specialties

A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Multisource Feedback Through Paired Self- and Peer Evaluations

Exploring differences in symptomatic adverse events assessment between nurses and physicians in the clinical trial setting

Association between gender and self-assessment skills amongst Japanese dental students

Application of oral examination in self-evaluation and evaluation of senior anesthesia resident

A cross‐sectional study of anesthesia medical staff's occupational and health status in nongovernment medical institutions in China

A cross-sectional survey of standardized resident training teachers in anesthesia in Beijing

Physician Wellness Measures and Clinical Performance on a Critically Ill Simulated Patient: Does a Lack of Well-Being Impact Patient Care?

Comparing Research of Doctor-Patient Communication Between Residents under Standardized Training and Eight-Year Medical Students: an Example of Peking Union Medical College

Gender differences in attitudes towards psychological help-seeking among chinese medical students: a comparative analysis

The Influence of Transaction Process With Doctors on Patient Satisfaction, Self-Rating Anxiety and Self-Efficacy Among International Students in China

Factors associated with medical students’ self-regulated learning and its relationship with clinical performance: a cross-sectional study

The prevalence of imposter syndrome and associated factors in Chinese medical students and residents: A single-center pilot study

Comparing Faculty and Trainee Evaluators of First-Year Resident Skills

Comparison Between Impact of Standardized Patient's Oral Feedback and Self-Evaluation on Performances of Medical Students in Objective Structured Clinical Examination

P5: Anesthesia Nurses' Self-Perceived Roles and Competences and Their Recognition of Anesthesia Specialist Nurses' Roles and Competencies