Use of research evidence in U.S. federal policymaking: A reflexive report on intra-stage mixed methods

Brett A Diaz,Jessica Pugel,Aditya Phutane,Liwei Zhang,Lawrie Green,Jayne Hoffmann,Elizabeth C Long,Max Crowley,J Taylor Scott

DOI: https://doi.org/10.1016/j.evalprogplan.2024.102469

Abstract:The policymaking process is largely opaque, especially regarding the actual writing of the policy. To attempt to better understand this complex process, we utilized mixed methods in our evaluation of an intervention. However, the process of mixing methods can be messy, and thus may require recalibration during the evaluation itself. Yet, in comparison to reporting results, relatively little attention is paid to the effects of mixing methods on the evaluation process. In this article, we take a reflexive approach to reporting a mixed methods evaluation of an intervention on the use of research evidence in U.S. federal policymaking. We focus on the research process in a qualitative coding team, and the effects of mixing methods on that process. Additionally, we report in general terms how to interpret multinomial logistic regressions, an underused analysis type applicable to many evaluations. Thus, this reflexive piece contributes (1) findings from evaluation of the intervention on the policymaking process, (2) an example of mixing methods leading to unexpected findings and future directions, (3) a report about the evaluation process itself, and (4) a tutorial for those new to multinomial logistic regressions.