Abstract:One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.

Single-Item Versus Multiple-Item Measurement Scales: An Empirical Comparison

Testing the Measurement Equivalence of Personality Adjective Items Across Cultures

Developing and investigating the use of single-item measures in organizational research.

Examining the Concurrent and Predictive Validity of Single Items in Ecological Momentary Assessments

Normalizing the Use of Single-Item Measures: Validation of the Single-Item Compendium for Organizational Psychology

The Predictive Validity of Multiple-Item versus Single-Item Measures of the Same Constructs

Psychometric Evaluation of Single-Item Relationship Satisfaction, Love, Conflict, and Commitment Measures

Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models.

Testing the ruler with item response theory: Increasing precision of measurement for relationship satisfaction with the Couples Satisfaction Index.

Measurement invariance of brief forms of the Schizotypal Personality Questionnaire across convenience versus random samples

Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Testing Measurement Invariance of Satisfaction with Life Scale(SWLS)based on the Multi-group Confirmatory Factor Analysis

Appropriate use of single-item measures is here to stay

A Comparison of Unidimensionality and Measurement Precision of the Narcissistic Personality Inventory and the Narcissistic Admiration and Rivalry Questionnaire

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items

The Even Shorter Almost Perfect Scale: Psychometric Evaluation and Cross-National Implications for Psychological Outcomes

What is wrong with individual differences research?

The Necessity of Testing Measurement Invariance in Cross-Cultural Research: Potential Bias in Cross-Cultural Comparisons With Individualism– Collectivism Self-Report Scales

Nonequivalence of measurement in latent variable modeling of multigroup data: A sensitivity analysis.

Assessing Autonomy, Competence, and Relatedness Briefly

Measurement Models, Estimation, and the Study of Change