Abstract:Background: Even though investigating predictors of intervention success (e.g Cognitive Training, CT) is gaining more and more interest in the light of an individualized medicine, results on specific predictors of intervention success in the overall field are mixed and inconsistent due to different and sometimes inappropriate statistical methods used. Therefore, the present paper gives a guidance on the appropriate use of multiple regression analyses to identify predictors of CT and similar non-pharmacological interventions. Methods: We simulated data based on a predefined true model and ran a series of different analyses to evaluate their performance in retrieving the true model coefficients. The true model consisted of a 2 (between: experimental vs. control group) × 2 (within: pre- vs. post-treatment) design with two continuous predictors, one of which predicted the success in the intervention group and the other did not. In analyzing the data, we considered four commonly used dependent variables (post-test score, absolute change score, relative change score, residual score), five regression models, eight sample sizes, and four levels of reliability. Results: Our results indicated that a regression model including the investigated predictor, Group (experimental vs. control), pre-test score, and the interaction between the investigated predictor and the Group as predictors, and the absolute change score as the dependent variable seemed most convenient for the given experimental design. Although the pre-test score should be included as a predictor in the regression model for reasons of statistical power, its coefficient should not be interpreted because even if there is no true relationship, a negative and statistically significant regression coefficient commonly emerges. Conclusion: Employing simulation methods, theoretical reasoning, and mathematical derivations, we were able to derive recommendations regarding the analysis of data in one of the most prevalent experimental designs in research on CT and external predictors of CT success. These insights can contribute to the application of considered data analyses in future studies and facilitate cumulative knowledge gain.

What Predicts Within-Participant Replication of Relative Efficiency in Single-Case Comparisons? A Logistic Regression Analysis

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

A Comparison of Single-Case Effect Measures Using Check-In Check-Out Data

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Residuals and Regression Diagnostics: Focusing on Logistic Regression

A comparison of recent approaches to the analysis of repeated measurements

A systematic comparison of non-overlap metrics and visual analysis in single-case experimental design

On the comparison of regression coefficients across multiple logistic models with binary predictors

Nothing wrong about change: the adequate choice of the dependent variable and design in prediction of cognitive training success

A more powerful test method for analyzing unreplicated factorial two-level experiments

Replication: Revisiting Tversky and Shafir’s (1992) Disjunction Effect with an extension comparing between and within subject designs

Retest of the “random Mating” Hypothesis Applying Mini-census Data:On the Comparability of Logistic Regression Cofficients

Performance and Interpretability Comparisons of Supervised Machine Learning Algorithms: An Empirical Study

Approaches to Statistical Efficiency When Comparing the Embedded Adaptive Interventions in a SMART

Doubly robust estimation of average treatment effect revisited

Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

A measure of reliability convergence to select and optimize cognitive tasks for individual differences research

Model-assisted complier average treatment effect estimates in randomized experiments with non-compliance and a binary outcome

A Comparison of Reproducibility Guidelines and Its Implications on Undergraduate Statistical Education

The influence of procedural characteristics on within-case effect sizes for academic outcomes

Model-assisted complier average treatment effect estimates in randomized experiments with non-compliance and a binary outcome