Abstract:One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.

Modeling Response Styles in Cross‐Classified Data Using a Cross‐Classified Multidimensional Nominal Response Model

Approaches to Item-Level Data with Cross-Classified Structure: An Illustration with Student Evaluation of Teaching

A Restricted Latent Class Model with Polytomous Attributes and Respondent-Level Covariates

Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling

Investigating Heterogeneity in Response Strategies: A Mixture Multidimensional IRTree Approach

Applying the Network Item Response Model to Student Assessment Data

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Hidden Markov Models for Longitudinal Rating Data with Dynamic Response Styles

Robust Estimation and Inference in Categorical Data

Markov switching stereotype logit models for longitudinal ordinal data affected by unobserved heterogeneity in responding behavior

Loglinear modeling with mixed numerical and categorical predictor variables through an Extended Stereotype Model

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis

Cross-classified multilevel models

Flexible Bayesian Modeling for Longitudinal Binary and Ordinal Responses

Comparing item response theory models for ranking incorrect response options

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items

Response Surface Analysis with Missing Data

Assessment of School Performance Through a Multilevel Latent Markov Rasch Model

Nonresponse Bias Analysis in Longitudinal Studies: A Comparative Review with an Application to the Early Childhood Longitudinal Study

Multimodal Likelihoods in Educational Assessment

Exploring Within-Rater Category Ordering: A Simulation Study Using Adjacent-Categories Mokken Scale Analysis