Abstract:BackgroundImplementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).MethodsThirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.ResultsAll but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.ConclusionsThe AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

Assessing Clinical Significance: Proposed Extensions to Method

Using clinical significance in psychotherapy outcome research: The need for a common procedure and validity data

Clinical Significance Methods: A Comparison of Statistical Techniques

Assessing clinical significance: Does it matter which method we use?

Clinical significance: a statistical approach to defining meaningful change in psychotherapy research.

The Importance of the Principle of Clinical Significance—Defining Significant to Whom and for What Purpose: A Response to Tingey, Lambert, Burlingame, and Hansen

Clinical significance methods: Which one to use and how useful are they?

Validity of clinically significant change classifications yielded by Jacobson-Truax and Hageman-Arrindell methods

An Introduction to Clinical Significance: An Alternative Index of Intervention Effect for Group Experimental Designs

On the identification of clinically significant client changes: Reinterpretation of Jacobson's cut scores

clinicalsignificance: Clinical Significance Analyses of Intervention Studies in R

An Evidence-Based Approach to Choosing Outcome Measures: A Checklist for the Critical Appraisal of Validity, Reliability and Responsiveness Studies

Why (not) rely on patients' perspectives? Time for a meaningful standard to measure and define patient-specific change in clinical psychology.

A tale of two constructs: confirmatory factor analysis of performance and symptom validity tests

The Therapy Progress Scale: Evaluating Psychometric Properties in an Outpatient Sample of Clients in Private Practice

How do we know whether treatment has failed? Paradoxical outcomes in counseling with young people

When meta-analysis misleads: A critical case study of a meta-analysis of client feedback.

When better is the enemy of good: two cautionary tales of conceptual validity versus parsimony in clinical psychometric research

Systematic Integration of Multi-Informant Externalizing Ratings in Clinical Settings

Constructing arguments for the interpretation and use of patient-reported outcome measures in research: an application of modern validity theory

Psychometric assessment of three newly developed implementation outcome measures