Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses.

Xin Sun,Matthias Briel,Stephen D Walter,Gordon H Guyatt

DOI: https://doi.org/10.1136/bmj.c117

2010-01-01

Abstract:How can we tell the difference between spurious and real subgroup effects? This article identifies new criteria and proposes a checklist for judging the credibility of subgroup analyses

What problem does this paper attempt to address?

How to Use A Subgroup Analysis

Xin Sun,John P. A. Ioannidis,Thomas Agoritsas,Ana C. Alba,Gordon Guyatt

DOI: https://doi.org/10.1001/jama.2013.285063

2014-01-01

Abstract:Clinicians, when trying to apply trial results to patient care, need to individualize patient care and, potentially, manage patients based on results of subgroup analyses. Apparently compelling subgroup effects often prove spurious, and guidance is needed to differentiate credible from less credible subgroup claims. We therefore provide 5 criteria to use when assessing the validity of subgroup analyses: (1) Can chance explain the apparent subgroup effect; (2) Is the effect consistent across studies; (3) Was the subgroup hypothesis one of a small number of hypotheses developed a priori with direction specified; (4) Is there strong preexisting biological support; and (5) Is the evidence supporting the effect based on within- or between-study comparisons. The first 4 criteria are applicable to individual studies or systematic reviews, the last only to systematic reviews of multiple studies. These criteria will help clinicians deciding whether to use subgroup analyses to guide their patient care.
Credibility of claims of subgroup effects in randomised controlled trials: systematic review.

Xin Sun,Matthias Briel,Jason W Busse,John J You,Elie A Akl,Filip Mejza,Malgorzata M Bala,Dirk Bassler,Dominik Mertz,Natalia Diaz-Granados,Per Olav Vandvik,German Malaga,Sadeesh K Srinathan,Philipp Dahm,Bradley C Johnston,Pablo Alonso-Coello,Basil Hassouneh,Stephen D Walter,Diane Heels-Ansdell,Neera Bhatnagar,Douglas G Altman,Gordon H Guyatt

DOI: https://doi.org/10.1136/bmj.e1553

2012-01-01

Abstract:Objective To investigate the credibility of authors' claims of subgroup effects using a representative sample of recently published randomised controlled trials. Design Systematic review. Data source Core clinical journals, as defined by the National Library of Medicine, in Medline. Study selection Randomised controlled trials published in 2007. Using prespecified criteria, teams of trained reviewers independently judged whether authors claimed subgroup effects and the strength of their claims. Reviewers assessed each of these claims against 10 predefined criteria, developed through a search of existing criteria and a consensus process. Results Of 207 randomised controlled trials reporting subgroup analyses, 64 (31%) made claims for the primary outcome. Of those, 20 were strong claims and 28 claims of a likely effect. Authors included subgroup variables measured at baseline in 60 (94%) trials, used subgroup variable as a stratification factor at randomisation in 13 (20%), clearly prespecified their hypotheses in 26 (41%), correctly prespecified direction in 4 (6%), tested a small number of hypotheses in 28 (44%), carried out a test of interaction that proved statistically significant in 6 (9%), documented replication of a subgroup effect with previous related studies in 21 (33%), identified consistency of a subgroup effect across related outcomes in 19 (30%), and provided a compelling indirect evidence for the effect in 14 (22%). In the 19 trials making more than one claim, only one (5%) checked the independence of the interaction. Of the 64 claims, 54 (84%) met four or fewer of the 10 criteria. For strong claims, more than 50% failed each of the individual criteria, and only three (15%) met more than five criteria. Conclusion Authors often claim subgroup effects in their trial report. However, the credibility of subgroup effects, even when claims are strong, is usually low. Users of the information should treat claims that fail to meet most criteria with scepticism. Trial researchers should report the conduct of subgroup analyses and provide sufficient evidence when claiming a subgroup effect or suggesting a possible effect.
Challenges in Subgroup Analysis—Should We Do More About Implementation?

Stefan Schandelmaier,Gordon Guyatt

DOI: https://doi.org/10.1001/jamanetworkopen.2024.3339

2024-03-29

JAMA Network Open

Abstract:Sherry et al 1 considered the methodological quality of subgroup analyses reported in 379 oncology trials. The authors explored a number of fundamental problems with these analyses: in forest plots, use of linear rather than logarithmic scales and failure to present overall pooled estimates; failure to conduct and highlight tests of interaction; and readiness to make unwarranted inferences regarding the credibility of postulated subgroup effects. They reported a distressingly high frequency of each of these problems. 1 The study thus adds to an already large body of subgroup literature finding that authors' presentations of forest plots are often suboptimal. It also adds indirect evidence for the often-suspected abuse of subgroup analyses for post hoc data dredging in search of interesting findings 2 : trials that failed to show a significant effect reported more subgroup analyses. 1 The primary message of the study is, therefore, that the methodological limitations of subgroup analyses and misleading inferences in oncology trials remain the same as those that the methods community has been discussing for more than 3 decades. 3 ,4 We will briefly discuss these key limitations in subgroup analyses. In only 10% of the subgroup claims did the primary study authors consider prior evidence or prespecify hypotheses. 1 In failing to do so, they fail to inform their audience whether the claim is consistent with prior knowledge. This information is critical for evaluating the credibility of the authors' subgroup claim. The test of interaction is the single most crucial statistic for subgroup analysis: it tells us the extent to which chance can explain the apparent difference in effect sizes across subgroups. Sherry et al 1 found that only 17% of the trials reported a P value or estimate of interaction, leaving evidence users in the dark regarding the extent to which subgroup differences were compatible with random error (and they mostly are 5 ). The more hypotheses one tests, the more likely one will capitalize on a chance finding and then claim a spurious subgroup effect. With an average number of 9 subgroup analyses per oncology trial, 1 these subgroup analyses run a high risk of being misled—and misleading their audience—by the play of chance. For continuous variables such as age, trial authors set thresholds and reported effects on patients above and below their chosen threshold. A superior alternative would be to examine whether effects differ across the range of the continuous variable. The choice of a single threshold results in a high risk of further capitalizing on the play of chance, especially when choosing a threshold that maximizes apparent difference between groups, and weakening the analysis through discarding the extra information that the continuum provides. And finally, as Sherry et al 1 confirmed, none of the included trials applied credibility criteria for subgroup effects that have been available since the 1990s. 3 ,4,6 We recently refined these criteria in the first formal rigorously developed instrument for judging the credibility of subgroup effects. 7 What can we do to improve the methodological quality of subgroup analyses? Learning from the past, publishing more commentaries, meta-studies, simulation studies, and guidance papers about the challenges and solutions in subgroup analysis, even if they are largely repetitive, seems unlikely to help. One strategy, a systematic greater focus on methods implementation, has thus far failed to attract the attention it might deserve. A small number of pioneering studies have started to identify and better understand barriers to methods implementation 8 -10 identify and test strategies for better methods implementation, 11 and raise the issue of whether the principles of implementation science could work in the methods context. 12 One simple application of implementation science would be to make it easier for investigators to identify methods guidance relevant to their studies. 13 Another would be to improve methods guidance by encouraging those providing such guidance to involve their target audience in ensuring the accessibility of the guidance they provide. 14 Reporting guidelines represent another approach that has demonstrated appreciable—although still perhaps somewhat disappointing—improvement in study design and methods implementation. We might utilize the full potential of reporting guidelines by including more methodological details (eg, items addressing the typical limitations of subgroup analyses we have highlighted) in the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guideline and corresponding items in the Consolidated Standards of Reporting Trials (CONSORT -Abstract Truncated-

medicine, general & internal
Evaluating the Validity of Post-Hoc Subgroup Inferences: A Case Study

Joseph J. Lee,Donald B. Rubin

DOI: https://doi.org/10.1080/00031305.2015.1093961

2016-01-01

Abstract:In randomized experiments, the random assignment of units to treatment groups justifies many of the widely used traditional analysis methods for evaluating causal effects. Specifying subgroups of units for further examination after observing outcomes, however, may partially nullify any advantages of randomized assignment when data are analyzed naively. Some previous statistical literature has treated all post-hoc subgroup analyses homogeneously as entirely invalid and thus uninterpretable. The extent of the validity of such analyses and the factors that affect the degree of validity remain largely unstudied. Here, we describe a recent pharmaceutical case with First Amendment legal implications, in which post-hoc subgroup analyses played a pivotal and controversial role. Through Monte Carlo simulation, we show that post-hoc results that seem highly significant make dramatic movements toward insignificance after accounting for the subgrouping procedure presumably used. Finally, we propose a novel, randomization-based method that generates valid post-hoc subgroup p-values, provided we know exactly how the subgroups were constructed. If we do not know the exact subgrouping procedure, our method may still place helpful bounds on the significance level of estimated effects. This randomization-based approach allows us to evaluate causal effects in situations where valid evaluations were previously considered impossible.
A Bayesian approach to subgroup identification.

James O Berger,Xiaojing Wang,Lei Shen

DOI: https://doi.org/10.1080/10543406.2013.856026

2014-01-01

Journal of Biopharmaceutical Statistics

Abstract:This article discusses subgroup identification, the goal of which is to determine the heterogeneity of treatment effects across subpopulations. Searching for differences among subgroups is challenging because it is inherently a multiple testing problem with the complication that test statistics for subgroups are typically highly dependent, making simple multiplicity corrections such as the Bonferroni correction too conservative. In this article, a Bayesian approach to identify subgroup effects is proposed, with a scheme for assigning prior probabilities to possible subgroup effects that accounts for multiplicity and yet allows for (preexperimental) preference to specific subgroups. The analysis utilizes a new Bayesian model selection methodology and, as a by-product, produces individual probabilities of treatment effect that could be of use in personalized medicine. The analysis is illustrated on an example involving subgroup analysis of biomarker effects on treatments.
Subgroup Analysis of Trials Is Rarely Easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials

Xin Sun,Matthias Briel,Jason W Busse,Elie A Akl,John J You,Filip Mejza,Malgorzata Bala,Natalia Diaz-Granados,Dirk Bassler,Dominik Mertz,Sadeesh K Srinathan,Per Olav Vandvik,German Malaga,Mohamed Alshurafa,Philipp Dahm,Pablo Alonso-Coello,Diane M Heels-Ansdell,Neera Bhatnagar,Bradley C Johnston,Li Wang,Stephen D Walter,Douglas G Altman,Gordon H Guyatt

DOI: https://doi.org/10.1186/1745-6215-10-101

IF: 2.728

2009-01-01

Trials

Abstract:Background Subgroup analyses in randomized trials examine whether effects of interventions differ between subgroups of study populations according to characteristics of patients or interventions. However, findings from subgroup analyses may be misleading, potentially resulting in suboptimal clinical and health decision making. Few studies have investigated the reporting and conduct of subgroup analyses and a number of important questions remain unanswered. The objectives of this study are: 1) to describe the reporting of subgroup analyses and claims of subgroup effects in randomized controlled trials, 2) to assess study characteristics associated with reporting of subgroup analyses and with claims of subgroup effects, and 3) to examine the analysis, and interpretation of subgroup effects for each study's primary outcome. Methods We will conduct a systematic review of 464 randomized controlled human trials published in 2007 in the 118 Core Clinical Journals defined by the National Library of Medicine. We will randomly select journal articles, stratified in a 1:1 ratio by higher impact versus lower impact journals. According to 2007 ISI total citations, we consider the New England Journal of Medicine, JAMA, Lancet, Annals of Internal Medicine , and BMJ as higher impact journals. Teams of two reviewers will independently screen full texts of reports for eligibility, and abstract data, using standardized, pilot-tested extraction forms. We will conduct univariable and multivariable logistic regression analyses to examine the association of pre-specified study characteristics with reporting of subgroup analyses and with claims of subgroup effects for the primary and any other outcomes. Discussion A clear understanding of subgroup analyses, as currently conducted and reported in published randomized controlled trials, will reveal both strengths and weaknesses of this practice. Our findings will contribute to a set of recommendations to optimize the conduct and reporting of subgroup analyses, and claim and interpretation of subgroup effects in randomized trials.
Investigating Stability in Subgroup Identification for Stratified Medicine

G. M. Hair,T. Jemielita,S. Mt‐Isa,P. M. Schnell,R. Baumgartner

DOI: https://doi.org/10.1002/pst.2409

2024-06-26

Pharmaceutical Statistics

Abstract:Subgroup analysis may be used to investigate treatment effect heterogeneity among subsets of the study population defined by baseline characteristics. Several methodologies have been proposed in recent years and with these, statistical issues such as multiplicity, complexity, and selection bias have been widely discussed. Some methods adjust for one or more of these issues; however, few of them discuss or consider the stability of the subgroup assignments. We propose exploring the stability of subgroups as a sensitivity analysis step for stratified medicine to assess the robustness of the identified subgroups besides identifying possible factors that may drive this instability. After applying Bayesian credible subgroups, a nonparametric bootstrap can be used to assess stability at subgroup‐level and patient‐level. Our findings illustrate that when the treatment effect is small or not so evident, patients are more likely to switch to different subgroups (jumpers) across bootstrap resamples. In contrast, when the treatment effect is large or extremely convincing, patients generally remain in the same subgroup. While the proposed subgroup stability method is illustrated through Bayesian credible subgroups method on time–to–event data, this general approach can be used with other subgroup identification methods and endpoints.

pharmacology & pharmacy,statistics & probability
Look before you leap: systematic evaluation of tree-based statistical methods in subgroup identification.

Yang Liu,Xiwen Ma,Donghui Zhang,Lijiang Geng,Xiaojing Wang,Wei Zheng,Ming-Hui Chen

DOI: https://doi.org/10.1080/10543406.2019.1584204

2019-01-01

Journal of Biopharmaceutical Statistics

Abstract:Subgroup analysis, as the key component of personalized medicine development, has attracted a lot of interest in recent years. While a number of exploratory subgroup searching approaches have been proposed, informative evaluation criteria and scenario-based systematic comparison of these methods are still underdeveloped topics. In this article, we propose two evaluation criteria in connection with traditional type I error and power concepts, and another criterion to directly assess recovery performance of the underlying treatment effect structure. Extensive simulation studies are carried out to investigate empirical performance of a variety of tree-based exploratory subgroup methods under the proposed criteria. A real data application is also included to illustrate the necessity and importance of method evaluation.
Differential Treatment Effects of Subgroup Analyses in Oncology Trials, 2004-2020

Alexander D. Sherry,Andrew W. Hahn,Zachary R. McCaw,Joseph Abi Jaoude,Ramez Kouzy,Timothy A. Lin,Bruce Minsky,C. David Fuller,Tomer Meirson,Pavlos Msaouel,Ethan B. Ludmir

DOI: https://doi.org/10.1001/jamanetworkopen.2024.3379

2024-03-29

JAMA Network Open

Abstract:This cross-sectional study evaluates the credibility of differential treatment effect claims in published phase 3 randomized clinical oncology trials published prior to 2021 according to the Instrument for Assessing the Credibility of Effect Modification Analyses.

medicine, general & internal
Discovering Statistically Non-Redundant Subgroups

Jiuyong Li,Jixue Liu,Hannu Toivonen,Kenji Satou,Youqiang Sun,Bingyu Sun

DOI: https://doi.org/10.1016/j.knosys.2014.04.030

IF: 8.139

2014-01-01

Knowledge-Based Systems

Abstract:The objective of subgroup discovery is to find groups of individuals who are statistically different from others in a large data set. Most existing measures of the quality of subgroups are intuitive and do not precisely capture statistical differences of a group with the other, and their discovered results contain many redundant subgroups. Odds ratio is a statistically sound measure to quantify the statistical difference of two groups for a certain outcome and it is a very suitable measure for quantifying the quality of subgroups. In this paper, we propose a statistically sound framework for statistically non-redundant subgroup discovery: measuring the quality of subgroups by the odds ratio and defining statistically non-redundant subgroups by the error bounds of odds ratios. We show that our proposed method is faster than most existing methods and discovers complete statistically non-redundant subgroups.
A Systematic Survey Identified 36 Criteria for Assessing Effect Modification Claims in Randomized Trials or Meta-Analyses

Stefan Schandelmaier,Yaping Chang,Niveditha Devasenapathy,Tahira Devji,Joey S. W. Kwong,Luis E. Colunga Lozano,Yung Lee,Arnav Agarwal,Neera Bhatnagar,Hannah Ewald,Ying Zhang,Xin Sun,Lehana Thabane,Michael Walsh,Matthias Briel,Gordon H. Guyatt

DOI: https://doi.org/10.1016/j.jclinepi.2019.05.014

IF: 7.407

2019-01-01

Journal of Clinical Epidemiology

Abstract:Objective: The objective of the study was to systematically survey the methodological literature and collect suggested criteria for assessing the credibility of effect modification and associated rationales. Study Design and Setting: We searched MEDLINE, Embase, and WorldCat up to March 2018 for publications providing guidance for assessing the credibility of effect modification identified in randomized trials or meta-analyses. Teams of two investigators independently identified eligible publications and extracted credibility criteria and authors' rationale, reaching consensus through discussion. We created a taxonomy of criteria that we iteratively refined during data abstraction. Results: We identified 150 eligible publications that provided 36 criteria and associated rationales. Frequent criteria included significant test for interaction (n = 54), a priori hypothesis (n = 49), providing a causal explanation (n = 47), accounting for multiplicity (n = 45), testing a small number of effect modifiers (n = 38), and prespecification of analytic details (n = 39). For some criteria, we found more than one rationale; some criteria were connected through a common rationale. For some criteria, experts disagreed regarding their suitability (e.g., added value of stratified randomization; trustworthiness of biologic rationales). Conclusion: Methodologists have expended substantial intellectual energy providing criteria for critical appraisal of apparent effect modification. Our survey highlights popular criteria, expert agreement and disagreement, and where more work is needed, including testing criteria in practice. (C) 2019 Published by Elsevier Inc.
Subgroup identification in clinical trials: an overview of available methods and their implementations with R

Zhongheng Zhang,Heidi Seibold,Mario Vianna Vettore,Woo-Jung Song,Vieille François

DOI: https://doi.org/10.21037/atm.2018.03.07

IF: 3.616

2018-01-01

Annals of Translational Medicine

Abstract:Randomized controlled trials (RCTs) usually enroll heterogeneous study population, and thus it is interesting to identify subgroups of patients for whom the treatment may be beneficial or harmful. A variety of methods have been developed to do such kind of post hoc analyses. Conventional generalized linear model is able to include prognostic variables as a main effect and predictive variables in an interaction with treatment variable. A statistically significant and large interaction effect usually indicates potential subgroups that may have different responses to the treatment. However, the conventional regression method requires to specify the interaction term, which requires knowledge of predictive variables or becomes infeasible when there is a large number of feature variables. The Least Absolute Shrinkage and Selection Operator (LASSO) method does variable selection by shrinking less clear effects (including interaction effects) to zero and in this way selects only certain variables and interactions for the model. There are many tree-based methods for subgroup identification. For example, model-based recursive partitioning incorporates parametric models such as generalized linear models into trees. The model incorporated is usually a simple model with only the treatment as covariate. Predictive and prognostic variables are found and incorporated automatically via the tree. The present article gives an overview of these methods and explains how to perform them using the free software environment for statistical computing R (version 3.3.2). A simulated dataset is employed for illustrating the performance of these methods.
Subgroup analyses

Michael Borenstein, Larry V Hedges, Julian PT Higgins, Hannah R Rothstein

2009-01-01

Abstract:
Detecting critical treatment effect bias in small subgroups

Piersilvio De Bartolomeis,Javier Abad,Konstantin Donhauser,Fanny Yang

2024-04-30

Abstract:Randomized trials are considered the gold standard for making informed decisions in medicine, yet they often lack generalizability to the patient populations in clinical practice. Observational studies, on the other hand, cover a broader patient population but are prone to various biases. Thus, before using an observational study for decision-making, it is crucial to benchmark its treatment effect estimates against those derived from a randomized trial. We propose a novel strategy to benchmark observational studies beyond the average treatment effect. First, we design a statistical test for the null hypothesis that the treatment effects estimated from the two studies, conditioned on a set of relevant features, differ up to some tolerance. We then estimate an asymptotically valid lower bound on the maximum bias strength for any subgroup in the observational study. Finally, we validate our benchmarking strategy in a real-world setting and show that it leads to conclusions that align with established medical knowledge.

Methodology,Machine Learning
Subgroup Analyses in Randomised Controlled Trials: Cohort Study on Trial Protocols and Journal Publications.

Benjamin Kasenda,Stefan Schandelmaier,Xin Sun,Erik von Elm,John You,Anette Bluemle,Yuki Tomonaga,Ramon Saccilotto,Alain Amstutz,Theresa Bengough,Joerg J. Meerpohl,Mihaela Stegert,Kelechi K. Olu,Kari A. O. Tikkinen,Ignacio Neumann,Alonso Carrasco-Labra,Markus Faulhaber,Sohail M. Mulla,Dominik Mertz,Elie A. Akl,Dirk Bassler,Jason W. Busse,Ignacio Ferreira-Gonzalez,Francois Lamontagne,Alain Nordmann,Viktoria Gloy,Heike Raatz,Lorenzo Moja,Rachel Rosenthal,Shanil Ebrahim,Per O. Vandvik,Bradley C. Johnston,Martin A. Walter,Bernard Burnand,Matthias Schwenkglenks,Lars G. Hemkens,Heiner C. Bucher,Gordon H. Guyatt,Matthias Briel

DOI: https://doi.org/10.1136/bmj.g4921

2014-01-01

BMJ

Abstract:OBJECTIVE:To investigate the planning of subgroup analyses in protocols of randomised controlled trials and the agreement with corresponding full journal publications.DESIGN:Cohort of protocols of randomised controlled trial and subsequent full journal publications.SETTING:Six research ethics committees in Switzerland, Germany, and Canada.DATA SOURCES:894 protocols of randomised controlled trial involving patients approved by participating research ethics committees between 2000 and 2003 and 515 subsequent full journal publications.RESULTS:Of 894 protocols of randomised controlled trials, 252 (28.2%) included one or more planned subgroup analyses. Of those, 17 (6.7%) provided a clear hypothesis for at least one subgroup analysis, 10 (4.0%) anticipated the direction of a subgroup effect, and 87 (34.5%) planned a statistical test for interaction. Industry sponsored trials more often planned subgroup analyses compared with investigator sponsored trials (195/551 (35.4%) v 57/343 (16.6%), P<0.001). Of 515 identified journal publications, 246 (47.8%) reported at least one subgroup analysis. In 81 (32.9%) of the 246 publications reporting subgroup analyses, authors stated that subgroup analyses were prespecified, but this was not supported by 28 (34.6%) corresponding protocols. In 86 publications, authors claimed a subgroup effect, but only 36 (41.9%) corresponding protocols reported a planned subgroup analysis.CONCLUSIONS:Subgroup analyses are insufficiently described in the protocols of randomised controlled trials submitted to research ethics committees, and investigators rarely specify the anticipated direction of subgroup effects. More than one third of statements in publications of randomised controlled trials about subgroup prespecification had no documentation in the corresponding protocols. Definitive judgments regarding credibility of claimed subgroup effects are not possible without access to protocols and analysis plans of randomised controlled trials.
Evaluating sensitivity to classification uncertainty in latent subgroup effect analyses

Wen Wei Loh,Jee-Seon Kim

DOI: https://doi.org/10.1186/s12874-022-01720-8

2022-09-24

Abstract:Background: Increasing attention is being given to assessing treatment effect heterogeneity among individuals belonging to qualitatively different latent subgroups. Inference routinely proceeds by first partitioning the individuals into subgroups, then estimating the subgroup-specific average treatment effects. However, because the subgroups are only latently associated with the observed variables, the actual individual subgroup memberships are rarely known with certainty in practice and thus have to be imputed. Ignoring the uncertainty in the imputed memberships precludes misclassification errors, potentially leading to biased results and incorrect conclusions. Methods: We propose a strategy for assessing the sensitivity of inference to classification uncertainty when using such classify-analyze approaches for subgroup effect analyses. We exploit each individual's typically nonzero predictive or posterior subgroup membership probabilities to gauge the stability of the resultant subgroup-specific average causal effects estimates over different, carefully selected subsets of the individuals. Because the membership probabilities are subject to sampling variability, we propose Monte Carlo confidence intervals that explicitly acknowledge the imprecision in the estimated subgroup memberships via perturbations using a parametric bootstrap. The proposal is widely applicable and avoids stringent causal or structural assumptions that existing bias-adjustment or bias-correction methods rely on. Results: Using two different publicly available real-world datasets, we illustrate how the proposed strategy supplements existing latent subgroup effect analyses to shed light on the potential impact of classification uncertainty on inference. First, individuals are partitioned into latent subgroups based on their medical and health history. Then within each fixed latent subgroup, the average treatment effect is assessed using an augmented inverse propensity score weighted estimator. Finally, utilizing the proposed sensitivity analysis reveals different subgroup-specific effects that are mostly insensitive to potential misclassification. Conclusions: Our proposed sensitivity analysis is straightforward to implement, provides both graphical and numerical summaries, and readily permits assessing the sensitivity of any machine learning-based causal effect estimator to classification uncertainty. We recommend making such sensitivity analyses more routine in latent subgroup effect analyses.
Evaluating the impact of treating the optimal subgroup

Alexander R Luedtke,Mark J van der Laan

DOI: https://doi.org/10.1177/0962280217708664

IF: 2.494

2017-05-08

Statistical Methods in Medical Research

Abstract:Suppose we have a binary treatment used to influence an outcome. Given data from an observational or controlled study, we wish to determine whether or not there exists some subset of observed covariates in which the treatment is more effective than the standard practice of no treatment. Furthermore, we wish to quantify the improvement in population mean outcome that will be seen if this subgroup receives treatment and the rest of the population remains untreated. We show that this problem is surprisingly challenging given how often it is an (at least implicit) study objective. Blindly applying standard techniques fails to yield any apparent asymptotic results, while using existing techniques to confront the non-regularity does not necessarily help at distributions where there is no treatment effect. Here, we describe an approach to estimate the impact of treating the subgroup which benefits from treatment that is valid in a nonparametric model and is able to deal with the case where there is no treatment effect. The approach is a slight modification of an approach that recently appeared in the individualized medicine literature.

health care sciences & services,medical informatics,mathematical & computational biology,statistics & probability
Toward a more credible assessment of the credibility of science by many-analyst studies

Katrin Auspurg,Josef Brüderl

DOI: https://doi.org/10.1073/pnas.2404035121

2024-09-17

Abstract:We discuss a relatively new meta-scientific research design: many-analyst studies that attempt to assess the replicability and credibility of research based on large-scale observational data. In these studies, a large number of analysts try to answer the same research question using the same data. The key idea is the greater the variation in results, the greater the uncertainty in answering the research question and, accordingly, the lower the credibility of any individual research finding. Compared to individual replications, the large crowd of analysts allows for a more systematic investigation of uncertainty and its sources. However, many-analyst studies are also resource-intensive, and there are some doubts about their potential to provide credible assessments. We identify three issues that any many-analyst study must address: 1) identifying the source of variation in the results; 2) providing an incentive structure similar to that of standard research; and 3) conducting a proper meta-analysis of the results. We argue that some recent many-analyst studies have failed to address these issues satisfactorily and have therefore provided an overly pessimistic assessment of the credibility of science. We also provide some concrete guidance on how future many-analyst studies could provide a more constructive assessment.
Bayesian Shrinkage Estimation of Credible Subgroups for Count Data with Excess Zeros

Duy Ngo,Daniel Quartey,Patrick M Schnell,Richard Baumgartner,Shahrul Mt-Isa,Dai Feng

DOI: https://doi.org/10.1080/19466315.2024.2388520

2024-09-21

Statistics in Biopharmaceutical Research

Abstract:Heterogeneity of treatment effects due to heterogeneous patient characteristics often arises in clinical trials. Subgroup analysis and the analysis of interactions are the most common approaches for evaluating such heterogeneous effects but do not explicitly address multiplicity issues. Another common challenge of analyzing treatment effect heterogeneity is the large number of possible covariates which inevitably causes problems related to multiplicity and lack of power. In this article, we develop a Bayesian credible subgroups method using continuous shrinkage priors to assess heterogeneity in treatment effects and multiplicity-adjusted benefiting subgroup identification for zero-inflated count data, which are often encountered in medical and public health studies. Our proposed method provides two bounding subgroups for the true benefiting subgroup: one that is probably contained by the true benefiting subgroup and one that probably contains the true benefiting subgroup. A simulation study has been conducted to compare the performance of the proposed method with other methods through frequentist properties. We apply our method to a clinical bladder tumor trial studying the effect of thiotepa treatment on the reduction of the recurrence of bladder tumor.

mathematical & computational biology,statistics & probability
Online Testing of Subgroup Treatment Effects Based on Value Difference

Miao Yu,Wenbin Lu,Rui Song

DOI: https://doi.org/10.48550/arXiv.2109.00712

2021-09-02

Abstract:Online A/B testing plays a critical role in the high-tech industry to guide product development and accelerate innovation. It performs a null hypothesis statistical test to determine which variant is better. However, a typical A/B test presents two problems: (i) a fixed-horizon framework inflates the false-positive errors under continuous monitoring; (ii) the homogeneous effects assumption fails to identify a subgroup with a beneficial treatment effect. In this paper, we propose a sequential test for subgroup treatment effects based on value difference, named SUBTLE, to address these two problems simultaneously. The SUBTLE allows the experimenters to "peek" at the results during the experiment without harming the statistical guarantees. It assumes heterogeneous treatment effects and aims to test if some subgroup of the population will benefit from the investigative treatment. If the testing result indicates the existence of such a subgroup, a subgroup will be identified using a readily available estimated optimal treatment rule. We examine the empirical performance of our proposed test on both simulations and a real dataset. The results show that the SUBTLE has high detection power with controlled type I error at any time, is more robust to noise covariates, and can achieve early stopping compared with the corresponding fixed-horizon test.

Methodology

Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses.

How to Use A Subgroup Analysis

Credibility of claims of subgroup effects in randomised controlled trials: systematic review.

Challenges in Subgroup Analysis—Should We Do More About Implementation?

Evaluating the Validity of Post-Hoc Subgroup Inferences: A Case Study

A Bayesian approach to subgroup identification.

Subgroup Analysis of Trials Is Rarely Easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials

Investigating Stability in Subgroup Identification for Stratified Medicine

Look before you leap: systematic evaluation of tree-based statistical methods in subgroup identification.

Differential Treatment Effects of Subgroup Analyses in Oncology Trials, 2004-2020

Discovering Statistically Non-Redundant Subgroups

A Systematic Survey Identified 36 Criteria for Assessing Effect Modification Claims in Randomized Trials or Meta-Analyses

Subgroup identification in clinical trials: an overview of available methods and their implementations with R

Subgroup analyses

Detecting critical treatment effect bias in small subgroups

Subgroup Analyses in Randomised Controlled Trials: Cohort Study on Trial Protocols and Journal Publications.

Evaluating sensitivity to classification uncertainty in latent subgroup effect analyses

Evaluating the impact of treating the optimal subgroup

Toward a more credible assessment of the credibility of science by many-analyst studies

Bayesian Shrinkage Estimation of Credible Subgroups for Count Data with Excess Zeros

Online Testing of Subgroup Treatment Effects Based on Value Difference