Developing and evaluating complex interventions: the new Medical Research Council guidance

Peter Craig,Paul Dieppe,Sally Macintyre,Susan Michie,Irwin Nazareth,Mark Petticrew

DOI: https://doi.org/10.1136/bmj.a1655

2008-09-29

BMJ

Abstract:Evaluating complex interventions is complicated. The Medical Research Council's evaluation framework (2000) brought welcome clarity to the task. Now the council has updated its guidance

What problem does this paper attempt to address?

A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance

Kathryn Skivington,Lynsay Matthews,Sharon Anne Simpson,Peter Craig,Janis Baird,Jane M. Blazeby,Kathleen Anne Boyd,Neil Craig,David P. French,Emma McIntosh,Mark Petticrew,Jo Rycroft-Malone,Martin White,Laurence Moore

DOI: https://doi.org/10.1016/j.ijnurstu.2024.104705

IF: 8.1

2024-02-01

International Journal of Nursing Studies

Abstract:The UK Medical Research Council's widely used guidance for developing and evaluating complex interventions has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.

nursing
Process evaluation of complex interventions: Medical Research Council guidance

G. F. Moore,S. Audrey,M. Barker,L. Bond,C. Bonell,W. Hardeman,L. Moore,A. O'Cathain,T. Tinati,D. Wight,J. Baird

DOI: https://doi.org/10.1136/bmj.h1258

2015-03-19

BMJ

Abstract:Process evaluation is an essential part of designing and testing complex interventions. New MRC guidance provides a framework for conducting and reporting process evaluation studies
Development and evaluation of complex intervention]

Miao Liu,Zhi-heng Hong,Si-yan Zhan

DOI: https://doi.org/10.3760/cma.j.issn.0254-6450.2010.12.020

2010-01-01

Abstract:Experimental studies with only one intervention are not always sufficient for health care issues, while the method of complex intervention is more useful. But the complex intervention is more complicated and needs carefully design and evaluation. This article will introduce the framework for design and evaluation of complex intervention and the revised version made by Medical Research Council, with five examples showing its application.
[Development, appraisal and synthesis of complex interventions - a methodological challenge]

Ingrid Mühlhauser,Matthias Lenz,Gabriele Meyer

DOI: https://doi.org/10.1016/j.zefq.2011.11.001

Abstract:Background: Many medical interventions are of a complex nature. They comprise interdependent components differently interacting within various complex settings, e.g., stroke units. Appraising the efficacy, benefit and harm of complex interventions is far more difficult than appraising single interventions like specific drug treatments. Detailed methodological procedures for the development, appraisal and synthesis of complex interventions are increasingly discussed internationally. Methods: Systematic inventory of methodological guidance for the development and evaluation of complex interventions through searching scientific literature databases and web resources of international institutes. Review of the current state of discussion and suggestions for the development, appraisal and synthesis of complex interventions. Results: Five methodological guidance papers have been identified. In particular, the UK Medical Research Council's guidance offers a structured introduction to the development and evaluation of complex interventions in health care and medicine. Synthesis of complex interventions using customary methods of systematic reviews is not satisfying. Other approaches to synthesis like realist evaluation as well as statistical procedures exploring the active components of complex interventions have been discussed. However, post hoc analytical procedures could never adequately replace careful prospective development and exploration of complex interventions and interdependencies with contextual factors. Conclusion: Complex interventions require multi-stage development, use of different methods, reporting on all developing phases and new approaches for synthesis. Presentation of the complete evidence on a specific complex intervention might be more useful than synthesis of a variety of different complex interventions by customarily applied methods of (meta-analytical) systematic review.
Development and introduction of complex intervention development and evaluation framework

卢彦娟,路潜,刘春蕾,庞立莎,朱飞

DOI: https://doi.org/10.3760/cma.j.cn115682-20220710-03343

2023-01-01

Abstract:In recent years, the application of complex interventions in fields such as public health, clinical research, and education has become a trend. Due to the complexity of interventions, target populations, or implementation contexts, complex intervention research can help clarify the mechanism of interventions, determine factors such as contexts related to outcome changes, and evaluate the feasibility and scalability of intervention plans. Based on the update of the complex intervention guidelines and research examples, this article summarizes the complex interventions' characteristics and core elements of the update, combs the development of the complex interventions research framework, and explains the specific content of the research phase, with a view to providing reference for the rational development of complex intervention research in the domestic nursing field.
Using natural experiments to evaluate population health interventions: new Medical Research Council guidance

Peter Craig,Cyrus Cooper,David Gunnell,Sally Haw,Kenny Lawson,Sally Macintyre,David Ogilvie,Mark Petticrew,Barney Reeves,Matt Sutton,Simon Thompson

DOI: https://doi.org/10.1136/jech-2011-200375

2012-05-10

Abstract:Natural experimental studies are often recommended as a way of understanding the health impact of policies and other large scale interventions. Although they have certain advantages over planned experiments, and may be the only option when it is impossible to manipulate exposure to the intervention, natural experimental studies are more susceptible to bias. This paper introduces new guidance from the Medical Research Council to help researchers and users, funders and publishers of research evidence make the best use of natural experimental approaches to evaluating population health interventions. The guidance emphasises that natural experiments can provide convincing evidence of impact even when effects are small or take time to appear. However, a good understanding is needed of the process determining exposure to the intervention, and careful choice and combination of methods, testing of assumptions and transparent reporting is vital. More could be learnt from natural experiments in future as experience of promising but lesser used methods accumulates.

public, environmental & occupational health
From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation

Graham F. Moore,Rhiannon E. Evans,Jemma Hawkins,Hannah Littlecott,G.J. Melendez-Torres,Chris Bonell,Simon Murphy

DOI: https://doi.org/10.1177/1356389018803219

2018-10-31

Evaluation

Abstract:Complex systems approaches to social intervention research are increasingly advocated. However, there have been few attempts to consider how models of intervention science, such as the UK’s Medical Research Council complex interventions framework, might be reframed through a complex systems lens. This article identifies some key areas in which this framework might be reconceptualized, and a number of priority areas where further development is needed if alignment with a systems perspective is to be achieved. We argue that a complex systems perspective broadens the parameters of ‘relevant’ evidence and theory for intervention development, before discussing challenges in defining feasibility in dynamic terms. We argue that whole systems evaluations may be neither attainable, nor necessary; acknowledgment of complexity does not mean that evaluations must be complex, or investigate all facets of complexity. However, a systems lens may add value to evaluation design through guiding identification of key uncertainties, and informing decisions such as timings of follow-up assessments.

social sciences, interdisciplinary
Evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health: a methodological research programme

Graham Dunn,Richard Emsley,Hanhua Liu,Sabine Landau,Jonathan Green,Ian White,Andrew Pickles

DOI: https://doi.org/10.3310/hta19930

Abstract:Background: The development of the capability and capacity to evaluate the outcomes of trials of complex interventions is a key priority of the National Institute for Health Research (NIHR) and the Medical Research Council (MRC). The evaluation of complex treatment programmes for mental illness (e.g. cognitive-behavioural therapy for depression or psychosis) not only is a vital component of this research in its own right but also provides a well-established model for the evaluation of complex interventions in other clinical areas. In the context of efficacy and mechanism evaluation (EME) there is a particular need for robust methods for making valid causal inference in explanatory analyses of the mechanisms of treatment-induced change in clinical outcomes in randomised clinical trials. Objectives: The key objective was to produce statistical methods to enable trial investigators to make valid causal inferences about the mechanisms of treatment-induced change in these clinical outcomes. The primary objective of this report is to disseminate this methodology, aiming specifically at trial practitioners. Methods: The three components of the research were (1) the extension of instrumental variable (IV) methods to latent growth curve models and growth mixture models for repeated-measures data; (2) the development of designs and regression methods for parallel trials; and (3) the evaluation of the sensitivity/robustness of findings to the assumptions necessary for model identifiability. We illustrate our methods with applications from psychological and psychosocial intervention trials, keeping the technical details to a minimum, leaving the reporting of the more theoretical and mathematically demanding results for publication in appropriate specialist journals. Results: We show how to estimate treatment effects and introduce methods for EME. We explain the use of IV methods and principal stratification to evaluate the role of putative treatment effect mediators and therapeutic process measures. These results are extended to the analysis of longitudinal data structures. We consider the design of EME trials. We focus on designs to create convincing IVs, bearing in mind assumptions needed to attain model identifiability. A key area of application that has become apparent during this work is the potential role of treatment moderators (predictive markers) in the evaluation of treatment effect mechanisms for personalised therapies (stratified medicine). We consider the role of targeted therapies and multiarm trials and the use of parallel trials to help elucidate the evaluation of mediators working in parallel. Conclusions: In order to demonstrate both efficacy and mechanism, it is necessary to (1) demonstrate a treatment effect on the primary (clinical) outcome, (2) demonstrate a treatment effect on the putative mediator (mechanism) and (3) demonstrate a causal effect from the mediator to the outcome. Appropriate regression models should be applied for (3) or alternative IV procedures, which account for unmeasured confounding, provided that a valid instrument can be identified. Stratified medicine may provide a setting where such instruments can be designed into the trial. This work could be extended by considering improved trial designs, sample size considerations and measurement properties. Funding: The project presents independent research funded under the MRC-NIHR Methodology Research Programme (grant reference G0900678).
Medical assistance in dying for mental illness: a complex intervention requiring a correspondingly complex evaluation approach

Hamer Bastidas-Bilbao,David Castle,Mona Gupta,Vicky Stergiopoulos,Lisa D Hawke,Lisa D. Hawke

DOI: https://doi.org/10.1192/bjp.2024.21

2024-03-05

The British Journal of Psychiatry

Abstract:Medical assistance in dying for mental illness as a sole underlying medical condition (MAiD MI-SUMC) is a controversial and complex policy in terms of psychosocial and ethical medical practice implications. We discuss the status of MAiD MI-SUMC in Canada and argue for the use of the UK Medical Research Council's framework on complex interventions in programme evaluations of MAiD MI-SUMC. It is imperative to carefully and rigorously evaluate the implementation of MAiD MI-SUMC to ensure an understanding of the multiple facets of implementation in contexts permeated by unique social, economic, cultural and historical influences, with a correspondingly diverse array of outcomes. This requires a complexity-informed programme evaluation focused on context-dependent mechanisms and stakeholder experiences, including patients, service providers and other people affected by the policy. It is also important to consider the economic impact on health and social welfare systems. Such evaluations can provide the data needed to guide evidence-informed decision-making that can contribute to safer implementation and refinement of MAiD MI-SUMC.

psychiatry
The barriers and facilitators to the implementation of National Clinical Programmes in Ireland: using the MRC framework for process evaluations

Catherine D. Darker,Gail H. Nicolson,Aine Carroll,Joe M. Barry

DOI: https://doi.org/10.1186/s12913-018-3543-6

2018-09-24

BMC Health Services Research

Abstract:BackgroundA major healthcare reform agenda in Ireland is underway which underpins the establishment of a series of National Clinical Programmes (NCPs), which aim to take an evidence based approach to improve quality, access and value. The current study aimed to determine the enablers and barriers to implementation of the NCPs.MethodsA qualitative methodology advocated by the Medical Research Council (MRC) framework on conducting process evaluations of complex interventions guided this research. Purposive sampling techniques were used to recruit participants from seven NCPs across both acute and chronic healthcare domains, comprised of orthopaedics, rheumatology, elective surgery, emergency medicine, paediatrics, diabetes and chronic obstructive pulmonary disease. A total of 33 participants were interviewed using a semi-structured interview guide. Participants included current and previous Clinical Leads, Programme Managers, Health Service Executive management, hospital Chief Executive Officers, representatives of General Practice, and a Nursing and a Patient representative. Thematic analyses was conducted.ResultsA range of factors of different combinations and co-occurrence were highlighted across a total of six themes, including (i) positive leadership, governance and clinical networks of the NCPs, (ii) the political and social context in which the NCPs operate, (iii) constraints on resources, (iv) a passive attitudinal resistance to change borne from poor consultation and communication, (v) lack of data and information technology, (vi) forces outside of the NCPs such as the general practitioner contract thwarting change of the model of care.ConclusionsThe MRC framework proved a useful tool to conduct this process evaluation. Results from this research provide real world experiences and insight from the people charged with implementing large-scale health system improvement initiatives. The findings highlight the need for measured responses that acknowledge both direct and non-direct challenges and opportunities for successful change. Combined, it is recommended that these elements be considered in the planning and implementation of large-scale initiatives across healthcare delivery systems, both in Ireland and internationally.

health care sciences & services
Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework

Lisa M Pfadenhauer,Ansgar Gerhardus,Kati Mozygemba,Kristin Bakke Lysdahl,Andrew Booth,Bjørn Hofmann,Philip Wahlster,Stephanie Polus,Jacob Burns,Louise Brereton,Eva Rehfuess

DOI: https://doi.org/10.1186/s13012-017-0552-5

2017-02-15

Abstract:Background: The effectiveness of complex interventions, as well as their success in reaching relevant populations, is critically influenced by their implementation in a given context. Current conceptual frameworks often fail to address context and implementation in an integrated way and, where addressed, they tend to focus on organisational context and are mostly concerned with specific health fields. Our objective was to develop a framework to facilitate the structured and comprehensive conceptualisation and assessment of context and implementation of complex interventions. Methods: The Context and Implementation of Complex Interventions (CICI) framework was developed in an iterative manner and underwent extensive application. An initial framework based on a scoping review was tested in rapid assessments, revealing inconsistencies with respect to the underlying concepts. Thus, pragmatic utility concept analysis was undertaken to advance the concepts of context and implementation. Based on these findings, the framework was revised and applied in several systematic reviews, one health technology assessment (HTA) and one applicability assessment of very different complex interventions. Lessons learnt from these applications and from peer review were incorporated, resulting in the CICI framework. Results: The CICI framework comprises three dimensions-context, implementation and setting-which interact with one another and with the intervention dimension. Context comprises seven domains (i.e., geographical, epidemiological, socio-cultural, socio-economic, ethical, legal, political); implementation consists of five domains (i.e., implementation theory, process, strategies, agents and outcomes); setting refers to the specific physical location, in which the intervention is put into practise. The intervention and the way it is implemented in a given setting and context can occur on a micro, meso and macro level. Tools to operationalise the framework comprise a checklist, data extraction tools for qualitative and quantitative reviews and a consultation guide for applicability assessments. Conclusions: The CICI framework addresses and graphically presents context, implementation and setting in an integrated way. It aims at simplifying and structuring complexity in order to advance our understanding of whether and how interventions work. The framework can be applied in systematic reviews and HTA as well as primary research and facilitate communication among teams of researchers and with various stakeholders.
TRIPLE C reporting principles for case study evaluations of the role of context in complex interventions

Sara E Shaw,Sara Paparini,Jamie Murdoch,Judith Green,Trisha Greenhalgh,Benjamin Hanckel,Hannah M James,Mark Petticrew,Gary W Wood,Chrysanthi Papoutsi

DOI: https://doi.org/10.1186/s12874-023-01888-7

2023-05-13

Abstract:Background: Guidance and reporting principles such as CONSORT (for randomised trials) and PRISMA (for systematic reviews) have greatly improved the reporting, discoverability, transparency and consistency of published research. We sought to develop similar guidance for case study evaluations undertaken to explore the influence of context on the processes and outcomes of complex interventions. Methods: A range of experts were recruited to an online Delphi panel, sampling for maximum diversity in disciplines (e.g. public health, health services research, organisational studies), settings (e.g. country), and sectors (e.g. academic, policy, third sector). To inform panel deliberations, we prepared background materials based on: [a] a systematic meta-narrative review of empirical and methodological literatures on case study, context and complex interventions; [b] the collective experience of a network of health systems and public health researchers; and [c] the established RAMESES II standards (which cover one kind of case study). We developed a list of topics and issues based on these sources and encouraged panel members to provide free text comments. Their feedback informed development of a set of items in the form of questions for potential inclusion in the reporting principles. We circulated these by email, asking panel members to rank each potential item twice (for relevance and validity) on a 7-point Likert scale. This sequence was repeated twice. Results: We recruited 51 panel members from 50 organisations across 12 countries, who brought experience of a range of case study research methods and applications. 26 completed all three Delphi rounds, reaching over 80% consensus on 16 items covering title, abstract, definitions of terms, philosophical assumptions, research question(s), rationale, how context and complexity relates to the intervention, ethical approval, empirical methods, findings, use of theory, generalisability and transferability, researcher perspective and influence, conclusions and recommendations, and funding and conflicts of interest. Conclusion: The 'Triple C' (Case study, Context, Complex interventions) reporting principles recognise that case studies are undertaken in different ways for different purposes and based on different philosophical assumptions. They are designed to be enabling rather than prescriptive, and to make case study evaluation reporting on context and complex health interventions more comprehensive, accessible and useable.
Unpacking the Cinderella black box of complex intervention development through the Partners at Care Transitions (PACT) programme of research

Jenni Murray,Ruth Baxter,Rebecca Lawton,Natasha Hardicre,Rosie Shannon,Joseph Langley,Rebecca Partridge,Sally Moore,Jane K. O'Hara

DOI: https://doi.org/10.1111/hex.13682

2023-04-27

Health Expectations

Abstract:Introduction Complex intervention development has been described as the 'Cinderella' black box in health services research. Greater transparency in the intervention development process is urgently needed to help reduce research waste. Methods We applied a new consensus‐based framework for complex intervention development to our programme of research, in which we developed an intervention to improve the safety and experience of care transitions for older people. Through this process, we aimed to reflect on the framework's utility for intervention development and identify any important gaps within it to support its continued development. Findings The framework was a useful tool for transparent reporting of the process of complex intervention development. We identified potential 'action' gaps in the framework including 'consolidation of evidence' and 'development of principles' that could bracket and steer decision‐making in the process. Conclusions We consider that the level of transparency demonstrated in this report, aided through use of the framework, is essential in the quest for reducing research waste. Patient or Public Contribution We have involved our dedicated patient and public involvement group in all work packages of this programme of research. Specifically, they attended and contributed to co‐design workshops and contributed to finalizing the intervention for the pilot evaluation. Staff also participated by attending co‐design workshops, helping us to prioritize content ideas for the intervention and supporting the development of intervention components outside of the workshops.

public, environmental & occupational health,health care sciences & services,health policy & services
Process evaluations of primary care interventions addressing chronic disease: a systematic review

Hueiming Liu,Alim Mohammed,Janani Shanthosh,Madeline News,Tracey-Lea Laba,Maree L Hackett,David Peiris,Stephen Jan

DOI: https://doi.org/10.1136/bmjopen-2018-025127

IF: 3.006

2019-08-06

BMJ Open

Abstract:Objective: Process evaluations (PEs) alongside randomised controlled trials of complex interventions are valuable because they address questions of for whom, how and why interventions had an impact. We synthesised the methods used in PEs of primary care interventions, and their main findings on implementation barriers and facilitators. Design: Systematic review using the UK Medical Research Council guidance for PE as a guide. Data sources: Academic databases (MEDLINE, SCOPUS, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, EMBASE and Global Health) were searched from 1998 until June 2018. Eligibility criteria: We included PE alongside randomised controlled trials of primary care interventions which aimed to improve outcomes for patients with non-communicable diseases. Data extraction and synthesis: Two independent reviewers screened and conducted the data extraction and synthesis, with a third reviewer checking a sample for quality assurance. Results: 69 studies were included. There was an overall lack of consistency in how PEs were conducted and reported. The main weakness is that only 30 studies were underpinned by a clear intervention theory often facilitated by the use of existing theoretical frameworks. The main strengths were robust sampling strategies, and the triangulation of qualitative and quantitative data to understand an intervention's mechanisms. Findings were synthesised into three key themes: (1) a fundamental mismatch between what the intervention was designed to achieve and local needs; (2) the required roles and responsibilities of key actors were often not clearly understood; and (3) the health system context-factors such as governance, financing structures and workforce-if unanticipated could adversely impact implementation. Conclusion: Greater consistency is needed in the reporting and the methods of PEs, in particular greater use of theoretical frameworks to inform intervention theory. More emphasis on formative research in designing interventions is needed to align the intervention with the needs of local stakeholders, and to minimise unanticipated consequences due to context-specific barriers. Prospero registration number: CRD42016035572.
Evaluating interventions to improve ethical decision making in clinical practice: a review of the literature and reflections on the challenges posed

Agnieszka Ignatowicz,Anne Marie Slowther,Christopher Bassford,Frances Griffiths,Samantha Johnson,Karen Rees

DOI: https://doi.org/10.1136/medethics-2021-107966

2022-03-03

Journal of Medical Ethics

Abstract:Since the 1980s, there has been an increasing acknowledgement of the importance of recognising the ethical dimension of clinical decision-making. Medical professional regulatory authorities in some countries now include ethical knowledge and practice in their required competencies for undergraduate and post graduate medical training. Educational interventions and clinical ethics support services have been developed to support and improve ethical decision making in clinical practice, but research evaluating the effectiveness of these interventions has been limited. We undertook a systematic review of the published literature on measures or models of evaluation used to assess the impact of interventions to improve ethical decision making in clinical care. We identified a range of measures to evaluate educational interventions, and one tool used to evaluate a clinical ethics support intervention. Most measures did not evaluate the key impact of interest, that is the quality of ethical decision making in real-world clinical practice. We describe the results of our review and reflect on the challenges of assessing ethical decision making in clinical practice that face both developers of educational and support interventions and the regulatory organisations that set and assess competency standards.

ethics,medical ethics,social issues,social sciences, biomedical
Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop

Susan Michie,Lucy Yardley,Robert West,Kevin Patrick,Felix Greaves

DOI: https://doi.org/10.2196/jmir.7126

IF: 7.076

2017-06-29

Journal of Medical Internet Research

Abstract:Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.

health care sciences & services,medical informatics
Designing studies and reviews to produce informative, trustworthy evidence about complex interventions in rehabilitation: a narrative review and commentary

William M Levack,Douglas P Gross,Rachelle A Martin,Susanna Every-Palmer,Carlotte Kiekens,Claudio Cordani,Stefano Negrini,Participants in the 5th Cochrane Rehabilitation Methodological Meeting ‡,Chiara Arienti,Irene Battel,Maria G Ceravolo,Christopher Colvin,Pierre Côté,Anne Cusick,Bernard Dan,Matteo J Del Furia,Giorgio Ferriero,Peter Feys,Christoph Gutenbrunner,Carsten B Juhl,Stefano G Lazzarini,Wendy Machalicek,Federico Merlo,Thorsten Meyer-Feil,Luca Miranda,Bianca Mosconi,Randolph Nudo,Aydan Oral,Cecile Røe,Heather Shearer,Jessica Wong

DOI: https://doi.org/10.23736/S1973-9087.24.08459-4

2024-06-26

Abstract:According to Cochrane Rehabilitation's recently published definition for research purposes, rehabilitation is inherently complex. Rehabilitation teams frequently implement multiple strategies concurrently, draw on input from a range of different health professionals, target multiple outcomes, and personalize therapeutic plans. The success of rehabilitation lies not only in the specific therapies employed, but also in how they are delivered, when they are delivered, and the capability and willingness of patients to engage in them. In 2021, the UK Medical Research Council (MRC) and the National Institute of Health Research (NIHR) released the second major update of its framework for developing and evaluating complex interventions. This framework has direct relevance to the development and implementation of evidence-based practice in the field of rehabilitation. While previous iterations of this framework positioned complex interventions as anything that involved multiple components, multiple people, multiple settings, multiple targets of effect, and behavior change, this latest framework expanded on this concept of complexity to also include the characteristics and influence of the context in which interventions occur. The revised MRC-NIHR framework presents complex intervention research as comprising the following four inter-related and overlapping phases: 1) development or identification of the intervention; 2) feasibility; 3) evaluation; and 4) implementation, with different methods and tools required to address each of these phases. This paper provides an overview of the MRC-NIHR framework and its application to rehabilitation, with examples from past research. Rehabilitation researchers are encouraged to learn about the MRC-NIHR framework and its application. Funders of rehabilitation research are also encouraged to place greater emphasis on supporting studies that involve the right design to address key uncertainties in rehabilitation clinical practice. This will require investment into a broader range of types of research than simply individual-level randomized controlled trials. Rehabilitation research can both learn from and contribute to future iterations of the MRC-NIHR framework as it is an excellent environment for exploring complexity in clinical practice.
Realist evaluation of UK medical education quality assurance

Paul Crampton,Leila Mehdizadeh,Michael Page,Laura Knight,Ann Griffin

DOI: https://doi.org/10.1136/bmjopen-2019-033614

IF: 3.006

2019-12-29

BMJ Open

Abstract:Objectives: The aim of the study was to explore what components of the General Medical Council's (GMC) Quality Assurance Framework work, for whom, in what circumstances and how? Setting: UK undergraduate and postgraduate medical education and training. Participants: We conducted interviews with a stratified sample of 36 individuals. This included those who had direct experiences, as well as those with external insights, representing local, national and international organisations within and outside medicine. Intervention: The GMC quality assure education to protect patient and public safety utilising complex intervention components including meeting standards, institutional visits and monitoring performance. However, the context in which these are implemented matters. We undertook an innovative realist evaluation to test an initial programme theory. Data were analysed using framework analysis. Results: Across components of the intervention, we identified key mechanisms, including transparent reporting to promote quality improvement; dialogical feedback; partnership working facilitating interactions between regulators and providers, and role clarity in conducting proportionate interventions appropriate to risk. The GMC's framework was commended for being comprehensive and enabling a broad understanding of an organisation's performance. Unintended consequences included confusion over roles and boundaries in different contexts which often undermined effectiveness. Conclusions: This realist evaluation substantiates the literature and reveals deeper understandings about quality assuring medical education. While standardised approaches are implemented, interventions need to be contextually proportionate. Routine communication is beneficial to verify data, share concerns and check risk; however, ongoing partnership working can foster assurance. The study provides a modified programme theory to explicate how education providers and regulators can work more effectively together to uphold education quality, and ultimately protect public safety. The findings have influenced the GMC's approach to quality assurance which impacts on all medical students and doctors in training.
Strengthening causal inference from randomised controlled trials of complex interventions

Jef L Leroy,Edward A Frongillo,Bezawit E Kase,Silvia Alonso,Mario Chen,Ian Dohoo,Lieven Huybregts,Suneetha Kadiyala,Naomi M Saville

DOI: https://doi.org/10.1136/bmjgh-2022-008597

Abstract:Researchers conducting randomised controlled trials (RCTs) of complex interventions face design and analytical challenges that are not fully addressed in existing guidelines. Further guidance is needed to help ensure that these trials of complex interventions are conducted to the highest scientific standards while maximising the evidence that can be extracted from each trial. The key challenge is how to manage the multiplicity of outcomes required for the trial while minimising false positive and false negative findings. To address this challenge, we formulate three principles to conduct RCTs: (1) outcomes chosen should be driven by the intent and programme theory of the intervention and should thus be linked to testable hypotheses; (2) outcomes should be adequately powered and (3) researchers must be explicit and fully transparent about all outcomes and hypotheses before the trial is started and when the results are reported. Multiplicity in trials of complex interventions should be managed through careful planning and interpretation rather than through post hoc analytical adjustment. For trials of complex interventions, the distinction between primary and secondary outcomes as defined in current guidelines does not adequately protect against false positive and negative findings. Primary outcomes should be defined as outcomes that are relevant based on the intervention intent and programme theory, declared (ie, registered), and adequately powered. The possibility of confirmatory causal inference is limited to these outcomes. All other outcomes (either undeclared and/or inadequately powered) are secondary and inference relative to these outcomes will be exploratory.
Development of the Implementability Assessment Tool for Clinical Practice Guidelines Based on the COSMIN Framework and Factorial Experiment: a Study Protocol

Dongmei Zhong,Gregory A Aarons,Alison M Hutchinson,Zhaolan Liu,Yaolong Chen,William CW Wong,Zaiwei Song,Yanni Wu,Lijiao Yan,David Makram Bishai,Nan Yang,Zijun Wang,Hui Liu,Dong (Roman) Xu

DOI: https://doi.org/10.1186/s12913-024-11754-x

2024-01-01

BMC Health Services Research

Abstract:Clinical practice guidelines are essential tools for standardizing medical practices and improving healthcare quality. However, current guideline implementation is unsatisfactory. Barriers to guideline implementation include external environmental factors (e.g., medical personnel, medical institutions, local policies) and intrinsic characteristics (e.g., context, format, language etc.), with the latter being addressable by optimizing the guidelines themselves. This study aims to develop an appraisal tool to promote effective guideline implementation. Existing guideline implementability appraisal tools lack a clear theoretical basis, evidence of reliability and validity, and/or empirically-based guidance for effective implementation, all of which will be addressed in this study. The research is divided into four phases: (1) Generating a theoretical framework for guideline implementability (i.e., scoping review and concept analysis method); (2) Conducting a factorial experiment (i.e., guideline users simulate the use of different guideline versions to examine the relationship between constructs in the framework and implementability), and removing constructs with weak relationships to form the final implementability theoretical framework; (3) Generating dimensions and items for the guideline implementability appraisal tool based on the constructs in the theoretical framework, conducting comprehensive reliability and validity testing, usability testing, and iterative optimization according to the COSMIN guidelines; (4) Integrating the validated tool with the STAR guideline rating system for guideline appraisal and feedback to promote guideline implementability. (1) Our research will follow a standardized and rigorous process for developing measurement scales, ensuring that the resulting evaluation scale for the implementability of CPGs is theoretically grounded, internally consistent between theory and practice, and reliable and valid. (2) The theoretical framework developed in our study will strengthen the validity and rationality of constructs in the real world. (3) In the confirmatory factor analysis, this study will determine the dimensions and items’ weight through factor loadings, allowing for the quantification of the implementability of CPGs and providing users of the guidelines with objective evaluative results. In addition, the clinical practice guideline implementability evaluation tool developed in this study can assist guideline developers in enhancing implementability before, during, and after guideline formulation.

Developing and evaluating complex interventions: the new Medical Research Council guidance

A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance

Process evaluation of complex interventions: Medical Research Council guidance

Development and evaluation of complex intervention]

[Development, appraisal and synthesis of complex interventions - a methodological challenge]

Development and introduction of complex intervention development and evaluation framework

Using natural experiments to evaluate population health interventions: new Medical Research Council guidance

From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation

Evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health: a methodological research programme

Medical assistance in dying for mental illness: a complex intervention requiring a correspondingly complex evaluation approach

The barriers and facilitators to the implementation of National Clinical Programmes in Ireland: using the MRC framework for process evaluations

Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework

TRIPLE C reporting principles for case study evaluations of the role of context in complex interventions

Unpacking the Cinderella black box of complex intervention development through the Partners at Care Transitions (PACT) programme of research

Process evaluations of primary care interventions addressing chronic disease: a systematic review

Evaluating interventions to improve ethical decision making in clinical practice: a review of the literature and reflections on the challenges posed

Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop

Designing studies and reviews to produce informative, trustworthy evidence about complex interventions in rehabilitation: a narrative review and commentary

Realist evaluation of UK medical education quality assurance

Strengthening causal inference from randomised controlled trials of complex interventions

Development of the Implementability Assessment Tool for Clinical Practice Guidelines Based on the COSMIN Framework and Factorial Experiment: a Study Protocol