Abstract:One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population, ignoring their possible effects on the scores may compromise the fairness and the validity of the assessments. Also, response-style-induced errors of measurement can affect the reliability estimates and overestimate convergent validity by correlating higher with other Likert-scale-based measures. Conversely, it can attenuate the predictive power over non-Likert-based indicators, given that the scores contain more errors. This study compares the validity of the Big Five personality scores obtained: (1) ignoring the SDR and ACQ in graded-scale items (GSQ), (2) accounting for SDR and ACQ with a compensatory IRT model, and (3) using forced-choice blocks with a multi-unidimensional pairwise preference model (MUPP) variant for dominance items. The overall results suggest that ignoring SDR and ACQ offered the worst validity evidence, with a higher correlation between personality and SDR scores. The two remaining strategies have their own advantages and disadvantages. The results from the empirical reliability and the convergent validity analysis indicate that when modeling social desirability with graded-scale items, the SDR factor apparently captures part of the variance of the Agreeableness factor. On the other hand, the correlation between the corrected GSQ-based Openness to Experience scores, and the University Access Examination grades was higher than the one with the uncorrected GSQ-based scores, and considerably higher than that using the estimates from the forced-choice data. Conversely, the criterion-related validity of the Forced Choice Questionnaire (FCQ) scores was similar to the results found in meta-analytic studies, correlating higher with Conscientiousness. Nonetheless, the FCQ-scores had considerably lower reliabilities and would demand administering more blocks. Finally, the results are discussed, and some notes are provided for the treatment of SDR and ACQ in future studies.

Evaluation of the Multidimensional Personality Questionnaire (MPQ) Unlikely Virtues Scale in the detection of underreporting.

Development of the Multidimensional Personality Questionnaire-55 abbreviated research scales.

CAT-PD and MMPI-3 Validity Scales Detect Simulated Overreporting and Underreporting

Susceptibility of the Self-Report Psychopathy Scale (SRP 4) to response distortion and the utility of including validity indices to detect deception.

Development and initial validation of the Scale of Scales (SOS) overreporting scores for the MMPI family of instruments

The Generalizability of Overreporting Across Self-Report Measures: An Investigation With the Minnesota Multiphasic Personality Inventory–2–Restructured Form and the Personality Assessment Inventory in a Civil Disability Sample

Development and validation of a brief form of the Multidimensional Personality Questionnaire.

Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires

Evaluation of the validity of the Multidimensional Personality Questionnaire-Simplified-Wording Form (MPQ-SF) in adolescents with treatment histories

Validation of the Multicultural Personality Questionnaire Short Form (MPQ-SF) for use in the context of international education

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items

Less Evaluative Measures of Personality in Job Applicant Contexts: The Effect on Socially Desirable Responding and Criterion Validity

Controlling for Response Bias in Self-Ratings of Personality: A Comparison of Impression Management Scales and the Overclaiming Technique

Development and Preliminary Validation of a Simplified-Wording Form of the Multidimensional Personality Questionnaire

Measurement invariance of brief forms of the Schizotypal Personality Questionnaire across convenience versus random samples

Cross validation of the response bias scale and the response bias scale-19 in active-duty personnel: use on the MMPI-2-RF and MMPI-3

Detecting Feigning in Adolescents on the Personality Assessment Inventory—Adolescent Form

A little garbage in, lots of garbage out: Assessing the impact of careless responding in personality survey data

A comparison of the MMPI-2-RF and PAI overreporting indicators in a civil forensic sample with emphasis on the Response Bias Scale (RBS) and the Cognitive Bias Scale (CBS).

Exploring Potential Ethnic Bias Among MMPI-3 Scales in Assessing Personality Psychopathology

Examining gender differences in the use of multidimensional forced‐choice measures of personality in terms of test‐taker reactions and test fairness