Abstract:BackgroundImplementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).MethodsThirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.ResultsAll but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.ConclusionsThe AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

Developing and investigating the use of single-item measures in organizational research.

Normalizing the Use of Single-Item Measures: Validation of the Single-Item Compendium for Organizational Psychology

Single-Item Versus Multiple-Item Measurement Scales: An Empirical Comparison

Examining the Concurrent and Predictive Validity of Single Items in Ecological Momentary Assessments

Psychometric Evaluation of Single-Item Relationship Satisfaction, Love, Conflict, and Commitment Measures

Undergraduates' workplace learning in health sciences education: psychometric properties of single-item measures

Appropriate use of single-item measures is here to stay

Assessing Autonomy, Competence, and Relatedness Briefly

Less Evaluative Measures of Personality in Job Applicant Contexts: The Effect on Socially Desirable Responding and Criterion Validity

Applying Item Response Theory to the OPD Structure Questionnaire: Identification of a Unidimensional Core Construct and Feasibility of Computer Adaptive Testing

Development and validation of the SDR-O: A new measure of socially desirable responding in organizations

Reliability and validity testing of a single-item physical activity measure

Pragmatic adaptation of implementation research measures for a novel context and multiple professional roles: a factor analysis study

Are Biasing Factors Idiosyncratic to Measures? A Comparison of Interpersonal Conflict, Organizational Constraints, and Workload

The Social Media Use Scale: Development and Validation

Short and sweet: Comparing strategies for the reduction of questionnaires on self‐criticism and social safeness while preserving construct validity

Testing the ruler with item response theory: Increasing precision of measurement for relationship satisfaction with the Couples Satisfaction Index.

The development of a single item FoMO (Fear of Missing Out) scale

Can—and should—automaticity be self‐reported using a single item? A secondary analysis of 16 datasets

When better is the enemy of good: two cautionary tales of conceptual validity versus parsimony in clinical psychometric research

Psychometric assessment of three newly developed implementation outcome measures