Abstract:One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.

The influence of cognitive reflection on consistency of responses between reversed and direct items

Competing explanations for inconsistent responding to a mixed-worded self-esteem scale: Cognitive abilities or personality?

Why Do People Agree with Both Regular and Reversed Items? A Logical Response Perspective

Improved Scoring of the Center for Epidemiologic Studies Depression Scale - Revised: An Item Response Theory Analysis

The Development of Cognitive Reflection in China

Method factor due to the use of reverse-keyed items: Is it simply a response style artifact?

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items

Investigating the Factor Structure of the Japanese Version of the Interpersonal Reactivity Index

Validation of the Very Efficient Short‐Form Need for Cognition Scale in Chinese: Detecting the effect of item orientation

Assessing miserly information processing: An expansion of the Cognitive Reflection Test

Individual differences in environmental sensitivity: associations between cognitive emotion regulation and mental health

The Effect of Enumeration of Self-Relevant Words on Self-Focused Attention and Repetitive Negative Thoughts

Does the Cognitive Reflection Test Work with Chinese College Students? Evidence from a Time-Limited Study

Reliability and Factorial Validity of the Core Self-Evaluations Scale

Linking self-perceived cognitive functioning questionnaires using item response theory: The subjective cognitive decline initiative.

Development of the Japanese Version of the Test of Self-Conscious Affect-3 (TOSCA-3): A Study among Student and Parent Population

Factor Structure and Internal Consistency on a Reduced Version of the Revised Test of Need for Cognitive Closure

Low cognitive reflection predicts honesty for men but not for women

The application of item response theory in developing and validating a shortened version of the Rotterdam Emotional Intelligence Scale

Factor structure and measurement invariance of the Health Education Impact Questionnaire: Does the subjectivity of the response perspective threaten the contextual validity of inferences?

Evaluating a Brief Big Five Personality Test in a Diverse Chinese Sample: The Role of Midpoint Designs and Reversely-Worded Items