Abstract:Background: The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to complete competency-based assessments of medical trainees based on nationally established Milestones. Previous research demonstrates a strong correlation between CCC and resident scores on the Milestones in surgery, but little is known if this is true between specialties. In this study, we investigated a variety of specialties and sought to determine what factors affect self-assessment of milestones. In addition, a post-hoc analysis was completed on the COVID-19 pandemic effects on self-evaluation. Methods: This is an IRB approved observational study on prospectively collected self-evaluation milestone data that is used within each ACGME program's Clinical Competency Committees. Medical trainees within the San Antonio Uniformed Services Health Education Consortium were approached for possible participation in this study with permission from program directors. Results: There was no significant difference between self-assessments and CCC-assessments based on self-identified gender or residency type (surgical versus nonsurgical) for any milestone domain. Within the postgraduate year (PGY) groups, the PGY5 and PGY6 tended to rate themselves higher than CCC. Chiefs (Internal Medicine PGY2/3, and General Surgery PGY5/6) tended to be more accurate in scoring themselves than the interns (PGY1) within the milestone of Interpersonal Skills and Communication (chiefs 0.5 vs. interns 0.62, p = 0.03). On post hoc analysis of self-rating, during the first wave of the COVID 19 pandemic, Post-Covid residents were more likely to underrate themselves in Systems-Based Practice compared to the Pre-Covid cohort (-0.49 vs 0.10; p = 0.007) and more likely to rate themselves higher in Professionalism (-0.54 vs. -0.10, p = 0.012). Conclusion: Unique to this study and our institution, there was no gender difference found in self vs CCC evaluations. With the change in learning environment from COVID, there was also a change in ability for some learners to self-assess accurately. As medical educators, we should understand the importance of both encouraging learners to practice self-assessment as well as give feedback to trainees on their progress. We also need to educate our faculty on the use of milestones for assessment to create a true gold standard in the CCC.

The Criterion Problem: What Measure of Success in Graduate Education?

The criterion problem: 1917–1992.

Discouraged or Motivated at University ? Goals and Reference Points in Academic Performance

Dealing with the Criterion Problem by Measuring the Quality and Quantity of Creative Activity and Accomplishment

Are individual characteristics and behaviors necessary-but-not-sufficient conditions for academic success?: A demonstration of Dul's (2016) necessary condition analysis

How not to evaluate a psychological measure: Rebuttal to criticism of the Defining Issues Test of moral judgment development by Curzer and colleagues

Quantitative CV-based indicators for research quality, validated by peer review

Adaptive performance: a criterion problem

Comment on "Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics"

Criteria for assessing grant applications: a systematic review

University Admission: Is Achievement a Sufficient Criterion?

How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review

Graduate entry medicine: selection criteria and student performance

The art and science of selecting graduate students in the biomedical sciences: Performance in doctoral study of the foundational sciences

Excellent Measures Precede Measures Of Excellence

Do You See What I See? A Comparison of CCC and Self-Assigned Milestones Across Military Medical Specialties

Quantifying Success in Science: An Overview

A comparison of remedial, failure, and successful secondary school students across self-perception and past and present school performance variables

Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Setting standards for teaching evaluation data: an application of the contrasting groups method