Assessment centers: Reflections, developments, and empirical insights

Duncan J. R. Jackson,Michael D. Blair,Pia V. Ingold

DOI: https://doi.org/10.1017/iop.2024.8

2024-05-07

Industrial and Organizational Psychology

Abstract://static.cambridge.org/content/id/urn%3Acambridge.org%3Aid%3Aarticle%3AS1754942624000087/resource/name/firstPage-pdf-S1754942624000087a.jpg

psychology, applied

What problem does this paper attempt to address?

Clinical reasoning performance assessment: using situated cognition theory as a conceptual framework

Joseph Rencic,Lambert W.T. Schuwirth,Larry D. Gruppen,Steven J. Durning

DOI: https://doi.org/10.1515/dx-2019-0051

2020-02-06

Diagnosis

Abstract:Abstract Developing valid assessment approaches to clinical reasoning performance has been challenging. Situated cognition theory posits that cognition (e.g. clinical reasoning) emerges from interactions between the clinician and situational (contextual) factors and recognizes an opportunity to gain deeper insights into clinical reasoning performance and its assessment through the study of these interactions. The authors apply situated cognition theory to develop a conceptual model to better understand the assessment of clinical reasoning. The model highlights how the interactions between six contextual factors, including assessee, patient, rater, and environment, assessment method, and task, can impact the outcomes of clinical reasoning performance assessment. Exploring the impact of these interactions can provide insights into the nature of clinical reasoning and its assessment. Three significant implications of this model are: (1) credible clinical reasoning performance assessment requires broad sampling of learners by expert raters in diverse workplace-based contexts; (2) contextual factors should be more explicitly defined and explored; and (3) non-linear statistical models are at times necessary to reveal the complex interactions that can impact clinical reasoning performance assessment.
The Expert Assessor Perspective on Assessment Center Taxonomies

Duncan J. R. Jackson,George Michaelides,Chris Dewberry,Wei-Ning Yang

DOI: https://doi.org/10.1080/08959285.2024.2428190

2024-12-01

Human Performance

Abstract:Research on assessment centers (ACs) has advanced the development of taxonomies for the evaluation of dimension and exercise performance. However, largely missing from current AC taxonomies is the perspective of subject matter expert assessors. Assessor perspectives could contribute toward improving construct differentiation and the development of a theoretical understanding of AC dimensions and exercises. In this study, 197 internationally based assessors participated in a series of multidimensional scaling (MDS) tasks involving stimuli from extant dimension and exercise taxonomies. For our dimension taxonomy, results suggested distinctions between (a) task orientation, (b) interpersonal relations, (c) activity, (d) organizing and planning, and (e) tolerance for stress/uncertainty. For our exercise taxonomy, results suggested that exercises are distinguished by varying levels of (a) media richness and (b) interpersonal interaction. We investigated assessor perceptions of illustrative dimension-exercise combinations and found, in contrast to findings for operational ACs, that assessors expected to rate same dimensions across different exercises. Our results suggest updates to and unique perspectives on taxonomies for dimensions and exercises. We discuss novel theoretical and practical insights that contribute to knowledge relevant to AC research and practice. We furthermore offer applied methodological contributions based on our unique application of MDS in the context of assessor perceptions.

psychology, applied
Perceptions of assessment center exercises: between exercises differences and interventions

Sylvia G. Roch

DOI: https://doi.org/10.1017/iop.2024.4

2024-03-19

Industrial and Organizational Psychology

Abstract:Preliminary research has demonstrated that not all assessment center (AC) exercises are viewed as equally just or motivating. The current research builds upon this research and investigates the relationships between six AC exercises and perceptions of self-efficacy, motivation, assessor bias, and fairness. Using a 2 × 2 × 2 experimental design (two informational justice interventions and one rating timing intervention), 286 working adults completed a survey designed to investigate differences between AC exercises and to investigate interventions designed to influence AC exercise perceptions. The results show not only significant perceptual differences between assessor-rated exercises and an ability test but also differences among the rated exercises. The results suggest that an ability test can be perceived as both among the most just and motivating exercises. Lastly, even though the experimental interventions did not have their anticipated effects, the results suggest benefits to having assessors rate recorded participant behaviors versus rating "live" behaviors, benefits that to a certain extent depend on whether participants had previously attended an assessment center.

psychology, applied
An ethical leadership assessment center pilot: Assessing and developing moral person and moral manager dimensions

Alexis d'Amato,Vignesh Murugavel,Kelsey Medeiros,Logan L. Watts

DOI: https://doi.org/10.1017/iop.2024.7

2024-05-10

Industrial and Organizational Psychology

Abstract:Ethical leaders are those who exemplify moral behavior personally, as well as those who facilitate follower ethical behavior. Although recent attention has been given to the ethical leadership construct, there remains a lack of innovation regarding the assessment and development of ethical leaders in organizations. To address these issues, a pilot study was conducted to examine the convergent validity of an ethical leadership assessment center, as well as the efficacy of using assessment center feedback to foster ethical leadership. Assessees completed a battery of pre tests, a virtual business simulation with a novel exercise, and a set of post tests. Half of the assessees were randomly assigned to a feedback condition, whereas the other half did not receive feedback until after the post tests were completed. Due to low statistical power, quantitative results were inconclusive. Nevertheless, qualitative insights were gained that point to implications for validating assessment center methodologies when assessing and developing ethical leadership.

psychology, applied
Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

Anurag Saxena,Loni Desanghere,Kelly Dore,Harold Reiter

DOI: https://doi.org/10.1186/s12909-024-05310-8

IF: 3.263

2024-03-28

BMC Medical Education

Abstract:Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

education & educational research,education, scientific disciplines
Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment

Richard J. Shavelson,Olga Zlatkin-Troitschanskaia,Klaus Beck,Susanne Schmidt,Julian P. Marino

DOI: https://doi.org/10.1080/15305058.2018.1543309

2019-01-24

International Journal of Testing

Abstract:Following employers’ criticisms and recent societal developments, policymakers and educators have called for students to develop a range of generic skills such as critical thinking (“twenty-first century skills”). So far, such skills have typically been assessed by student self-reports or with multiple-choice tests. An alternative approach is criterion-sampling measurement. This approach leads to developing performance assessments using “criterion” tasks, which are drawn from real-world situations in which students are being educated, both within and across academic or professional domains. One current project, iPAL (The international Performance Assessment of Learning), consolidates previous research and focuses on the next generation performance assessments. In this paper, we present iPAL’s assessment framework and show how it guides the development of such performance assessments, exemplify these assessments with a concrete task, and provide preliminary evidence of its reliability and validity, which allows us to draw initial implications for further test design and development.

English Else
Assessment Center and Regularize Postgraduate Reexamination

JIANG Ying

DOI: https://doi.org/10.3969/j.issn.1000-6378.2005.02.016

2005-01-01

Abstract:The Assessment Center, as an internationally popular and highly accurate method to select talented people, has been applied by governments, schools and commercial and industrial enterprises in Europe and America, but in China, it is still in its infancy. This paper has introduced the concept, technical approaches and comparative advantages of the Assessment Center, dissected the significance of introducing the Assessment Center into postgraduate reexamination. Furthermore, integrating the status quo of China' postgraduate reexamination, this paper proposes the introduction of the Assessment Center, in the hope of inducing more valuable contributions and making the postgraduate reexamination more fair, impartial, open, economic, scientific and effective.
How can applicants' reactions to scholastic aptitude tests be improved? A closer look at specific and general tests

Marek Denker,Clara Schütte,Martin Kersting,Daniel Weppert,Stephan Josef Stegt

DOI: https://doi.org/10.3389/feduc.2022.931841

2023-01-06

Frontiers in Education

Abstract:Since the 1980's, scientific interest in applicants' reactions to admission procedures has been growing. Several theoretical frameworks and evaluation questionnaires were developed. Many researchers have asked potential participants about their attitudes—what is missing, however, are studies of applicants who have actually taken part in the selection procedures. In addition, applicants' reactions to student selection procedures receive less attention than applicants' reactions to personnel selection procedures. Furthermore, tests and testing conditions continue to develop, e.g., through online-based testing and test supervision at home ("proctoring"). Therefore, we used a standardized questionnaire for measuring the overall test evaluation and different dimensions of acceptance including face validity, controllability, and the absence of strain to examine six scholastic aptitude tests. We added items to get deeper insights into situational aspects. According to the results of 2,052 test participants, applicants prefer specific tests and shorter tests. Situational aspects, such as privacy, working conditions, and prior information, have an influence on the perceptions of applicants. Proctored testing is evaluated positively, but taking the test at a test center is still rated more favorably. This study discusses the practical implications.
Assessment centers do not measure competencies: why this is now beyond reasonable doubt

Chris Dewberry

DOI: https://doi.org/10.1017/iop.2024.5

2024-05-10

Industrial and Organizational Psychology

Abstract:Although assessment centers (ACs) are usually designed to measure stable competencies (i.e., dimensions), doubt about whether or not they reliably do so has endured for 70 years. Addressing this issue in a novel way, several published Generalizability (G) theory studies have sought to isolate the multiple sources of variance in AC ratings, including variance specifically concerned with competencies. Unlike previous research, these studies can provide a definitive answer to the AC construct validity issue. In this article, the historical context for the construct validity debate is set out, and the results of four large-scale G-theory studies of ACs are reviewed. It is concluded that these studies demonstrate, beyond reasonable doubt, that ACs do not reliably measure stable competencies, but instead measure general, and exercise-related, performance. The possibility that ACs measure unstable competencies is considered, and it is suggested that evidence that they do so may reflect an artefact of typical AC design rather than a "real" effect. For ethical, individual, and organizational reasons, it is argued that the use of ACs to measure competencies can no longer be justified and should be halted.

psychology, applied
Assessing reflection in English enhancement courses: teachers’ views and development of a holistic framework

Ken Lau

DOI: https://doi.org/10.1080/02602938.2015.1048424

2015-05-29

Abstract:The debate about whether and how reflection should/can be assessed has so far yielded inconclusive findings. This paper intends to revisit and further this controversial issue within a specific disciplinary context: the assessment of reflection in English enhancement courses within a second-language environment. Students across majors taking the language enhancement courses in question are often required to provide written or verbal evidence to demonstrate their abilities to reflect, and in turn enhance their linguistic repertoires. However, a potential problem arises when the assessment of ‘reflection’ at times may become the assessment of learners’ English language proficiency, thus casting doubts about the validity and reliability of incorporating reflection as an assessment tool in our context. Drawing on analyses of seven English enhancement courses with a reflection component, and semi-structured interviews with 22 faculty members who had had the experience of assessing reflection, and/or developing assessment criteria for evaluating the quality of students’ reflection, their perceptions of assessing reflection fairly and the challenges encountered during the assessment process were brought to the surface. A holistic framework is proposed to address the challenges encountered, particularly with respect to the differences in emphasis that faculty members place on content vs. language.

education & educational research
Principles for Adapting Assessments of Executive Function across Cultural Contexts

Matthew C. H. Jukes,Ishita Ahmed,Sara Baker,Catherine E. Draper,Steven J. Howard,Dana Charles McCoy,Jelena Obradović,Sharon Wolf

DOI: https://doi.org/10.3390/brainsci14040318

IF: 3.333

2024-03-28

Brain Sciences

Abstract:Direct assessments of executive functions (EFs) are increasingly used in research and clinical settings, with a central assumption that they assess "universal" underlying skills. Their use is spreading globally, raising questions about the cultural appropriateness of assessments devised in Western industrialized countries. We selectively reviewed multidisciplinary evidence and theory to identify sets of cultural preferences that may be at odds with the implicit assumptions of EF assessments. These preferences relate to motivation and compliance; cultural expectations for interpersonal engagement; contextualized vs. academic thinking; cultural notions of speed and time; the willingness to be silly, be incorrect, or do the opposite; and subject-matter familiarity. In each case, we discuss how the cultural preference may be incompatible with the assumptions of assessments, and how future research and practice can address the issue. Many of the cultural preferences discussed differ between interdependent and independent cultures and between schooled and unschooled populations. Adapting testing protocols to these cultural preferences in different contexts will be important for expanding our scientific understanding of EF from the narrow slice of the human population that has participated in the research to date.

neurosciences
Assessment for inclusion: rethinking contemporary strategies in assessment design

Joanna Tai,Rola Ajjawi,Margaret Bearman,David Boud,Phillip Dawson,Trina Jorre de St Jorre

DOI: https://doi.org/10.1080/07294360.2022.2057451

2022-06-17

Abstract:Assessment has multiple purposes, one of which is to judge if students have met outcomes at the requisite level. Underperformance in assessment is frequently positioned as a problem of the student and attributed to student diversity and/or background characteristics. However, the assessment might also be inequitable and therefore exclude students inappropriately. To be inclusive, assessment design needs to be reconsidered, and educators should look beyond simplistic categories of disability or social equity groups, towards considering and accounting for diversity on many spectra. This article introduces the concept of assessment for inclusion , which seeks to ensure diverse students are not disadvantaged through assessment practices. Assumptions in assessment design are problematised from this point of view, and three central concerns relating to assessment traditions, assessment expectations, and academic integrity are interrogated. Contemporary design strategies of authentic assessment, programmatic assessment, and assessment for distinctiveness are then harnessed to illustrate approaches to assessment for inclusion. Assessment for inclusion therefore builds on the synergies between inclusive practice and good assessment design.

education & educational research
The Counseling Center Assessment of Psychological Symptoms (CCAPS): Merging clinical practice, training, and research.

Soo Jeong Youn,Louis G. Castonguay,Henry Xiao,Rebecca Janis,Andrew A. McAleavey,Allison J. Lockard,Benjamin D. Locke,Jeffrey A. Hayes

DOI: https://doi.org/10.1037/pst0000029

2015-01-01

Psychotherapy

Abstract:The goal of this article is to present information about a standardized multidimensional measure of psychological symptoms, the Counseling Center Assessment of Psychological Symptoms (CCAPS; Locke et al., 2011; Locke, McAleavey, et al., 2012; McAleavey, Nordberg, Hayes, et al., 2012), developed to assess difficulties specific to college students' mental health. We provide (a) a brief review and summary of the psychometric and research support for the CCAPS; (b) examples of the use of the CCAPS for various purposes, including clinical, training, policy, and counseling center advocacy; and (c) implications of the integration of routine outcome monitoring and feedback for the future of training, research, and clinical practice. In particular, the article emphasizes how the assimilation of and symbiotic relationship between research and practice can address the scientist-practitioner gap.

psychology, clinical
Student Performance: Assessment Centers Versus Traditional Classroom Evaluation Techniques

Lynn K. Bartels,William H. Bommer,Robert S. Rubin

DOI: https://doi.org/10.1080/08832320009599014

2000-03-01

Journal of Education for Business

Abstract:Assessment center (AC) scores were compared with traditional large-class evaluation techniques (i.e., multiple-choice exams, group project grades, and discussion section scores), grade point average, and intelligence test scores. Undergraduate business students (n = 347) were videotaped during an AC session consisting of an in-basket, two leaderless group discussions, and a speech. Trained raters used behavioral checklists to evaluate performance. AC scores correlated moderately with traditional evaluation techniques, suggesting that ACs measure knowledge acquisition in addition to skills acquired but not captured by traditional techniques. Their use is thus consistent with the emphasis on skill-based learning.

English Else
DMM assessments of attachment and adaptation: Procedures, validity and utility

Steve Farnfield,Airi Hautamäki,Peder Nørbech,Nicola Sahhar

DOI: https://doi.org/10.1177/1359104510364315

Abstract:This article gives a brief over view of the Dynamic-Maturational Model of attachment and adaptation (DMM; Crittenden, 2008) together with the various DMM assessments of attachment that have been developed for specific stages of development. Each assessment is discussed in terms of procedure, outcomes, validity, advantages and limitations, comparable procedures and areas for further research and validation. The aims are twofold: to provide an introduction to DMM theory and its application that underlie the articles in this issue of CCPP; and to provide researchers and clinicians with a guide to DMM assessments.
Assessment IS learning: developing a student‐centred approach for assessment in Higher Education

Stephen Rutherford,Connie Pritchard,Nigel Francis

DOI: https://doi.org/10.1002/2211-5463.13921

2024-11-03

FEBS Open Bio

Abstract:Assessment IS learning, either assessment of learning, assessment for learning, or assessment as learning, or combinations thereof. Good assessment design, literacy and feedback underpin effective student‐centred assessment. The learner is central to a cyclical process involving the development of competences, acting on feedback received, self‐reflection on progress, and the development of self‐regulatory skills; leading to a capable graduate lifelong learner. Assessment and the associated feedback from those assessments are powerful factors in the development of students' learning. We have seen a shift within the Higher Education sector to conceptualise assessment as being more than summative assessment 'of' learning. Instead, there has been a greater emphasis on assessment 'as' learning, or assessment 'for' learning, through the enhanced use of formative assessments. Centralising assessment within the learning process highlights that assessment IS learning and cannot be separated from other elements of the learning process. In particular, assessment has a vital role to play in the development of students' self‐regulated learning skills and the development of independence in learners. However, for assessments to effectively support learning, they need to be meaningful, engaging, well‐integrated into the learning activities and 'student‐focused'. Placing student skills development and personal development at the centre of assessment design has the potential to empower students through assessment. This review focuses on the potential of assessment to support student learning and development, using the 'Equity, Agency, Transparency' ('EAT') framework as a lens for effective assessment and feedback practices. We suggest ways in which we can make our assessment and feedback practices more inclusive, meaningful and authentic to the students' learning needs.

biochemistry & molecular biology
Personality assessment in military - Bright and dark sides of the assessment measures: A review

Marva Sohail,Fatima Naeem,Muhammad Umair Javaid,Afeez Kayode Ibikunle

DOI: https://doi.org/10.3233/WOR-220454

Work

Abstract:Background: Personality assessment has its own bright and dark sides, especially in the military setting. It is at play since the emergence of clinical psychology and its application in the recruitment process during World War I and II. Change in personality assessment trends, the evolution of complex personality traits, and changing geopolitical settings across the world necessitate the development of cost-effective, time-efficient, and reliable assessment measures that overcome issues in current assessment measures and fulfil the needs of present timesOBJECTIVE:The objective of this paper is to explore the evolution of personality assessment measures and their uses in military settings along with the comparison of self-report and non-self-report measures. It would give directions for the development of new measures to overcome the issues and drawbacks associated with current measures of research. Methods: The research question was formulated using the most valid SPIDER technique for a comprehensive search. PubMed, Google Scholar, and PsycNet databases were searched. 19 articles were selected based on inclusion and exclusion criteria out of 775 articles. Results: Thematic analyses show that personality assessment in the military is done to a) detect psychopathology, b) detect aberrant responses, c) recruit new personnel, d) predict training and performance outcomes, and e) predict leadership. The research contributes to the body of knowledge by showing an integrated picture of the most widely used assessment tools and the purpose it serves. Conclusion: There is a need to develop assessment measures that are culture free and can assess more complex personality attributes.
Assessment Accommodations for English Language Learners: Implications for Policy-Based Empirical Research

Jamal Abedi,Carolyn Huie Hofstetter,Carol Lord

DOI: https://doi.org/10.3102/00346543074001001

2004-03-01

Review of Educational Research

Abstract:Increased attention to large-scale assessments, the growing number of English language learners in schools, and recent inclusionary policies have collectively made assessment accommodations a hotly debated issue, especially regarding the validity of test results for English language learners. Decisions about which accommodations to use, for whom, and under what conditions, are based on limited empirical evidence for their effectiveness and validity. Given the potential consequences of test results, it is important that policy-makers and educators understand the empirical base underlying their use. This article reviews test accommodation strategies for English learners, derived from “scientifically based research.” The results caution against a one-size-fits-all approach. The more promising approaches include modified English and customized dictionaries, which can be used for all students, not just English language learners

education & educational research
The Next Era of Assessment Within Medical Education: Exploring Intersections of Context and Implementation

Aliya Kassam,Ingrid de Vries,Sondra Zabar,Steven J Durning,Eric Holmboe,Brian Hodges,Christy Boscardin,Adina Kalet

DOI: https://doi.org/10.5334/pme.1128

2024-10-09

Abstract:In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.
An epistemology for assessment and development: how do we know what we know?

Deborah E. Rupp,George C. Thornton,Tiffany M. Bisbey,Anna N. Hoover,Eduardo Salas,Kevin R. Murphy

DOI: https://doi.org/10.1017/iop.2024.3

2024-03-19

Industrial and Organizational Psychology

Abstract:To make informed decisions, assessment theorists, researchers, and practitioners can evaluate the overlap among (1) relevant theories, (2) empirical contributions, and (3) best practices. Unfortunately, such a task may seem daunting due to the so-called science-practice gap, which can thwart collaboration among these parties. This paper presents an epistemology for delineating the importance of integrating these three sources of knowledge. We then apply this epistemology to show that our current knowledge of assessment and development topics are well integrated in some places, but still quite lacking in others.

psychology, applied

Assessment centers: Reflections, developments, and empirical insights

Clinical reasoning performance assessment: using situated cognition theory as a conceptual framework

The Expert Assessor Perspective on Assessment Center Taxonomies

Perceptions of assessment center exercises: between exercises differences and interventions

An ethical leadership assessment center pilot: Assessing and developing moral person and moral manager dimensions

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment

Assessment Center and Regularize Postgraduate Reexamination

How can applicants' reactions to scholastic aptitude tests be improved? A closer look at specific and general tests

Assessment centers do not measure competencies: why this is now beyond reasonable doubt

Assessing reflection in English enhancement courses: teachers’ views and development of a holistic framework

Principles for Adapting Assessments of Executive Function across Cultural Contexts

Assessment for inclusion: rethinking contemporary strategies in assessment design

The Counseling Center Assessment of Psychological Symptoms (CCAPS): Merging clinical practice, training, and research.

Student Performance: Assessment Centers Versus Traditional Classroom Evaluation Techniques

DMM assessments of attachment and adaptation: Procedures, validity and utility

Assessment IS learning: developing a student‐centred approach for assessment in Higher Education

Personality assessment in military - Bright and dark sides of the assessment measures: A review

Assessment Accommodations for English Language Learners: Implications for Policy-Based Empirical Research

The Next Era of Assessment Within Medical Education: Exploring Intersections of Context and Implementation

An epistemology for assessment and development: how do we know what we know?