An approach theoretic version of Anscombe's theorem with an application in biostatistics

Ben Berckmoes

DOI: https://doi.org/10.48550/arXiv.1612.05104

2018-02-10

Abstract:We establish an approach theoretic version of Anscombe's theorem, which we apply to justify the use of confidence intervals based on the sample mean after a group sequential trial.

Probability

What problem does this paper attempt to address?

Bayesian Updating and Sequential Testing: Overcoming Inferential Limitations of Screening Tests

Jacques Balayla

DOI: https://doi.org/10.48550/arXiv.2006.11641

2020-08-30

Abstract:Bayes' Theorem confers inherent limitations on the accuracy of screening tests as a function of disease prevalence. We have shown in previous work that a testing system can tolerate significant drops in prevalence, up until a certain well-defined point known as the $prevalence$ $threshold$, below which the reliability of a positive screening test drops precipitously. Herein, we establish a mathematical model to determine whether sequential testing overcomes the aforementioned Bayesian limitations and thus improves the reliability of screening tests. We show that for a desired positive predictive value of $\rho$ that approaches $k$, the number of positive test iterations $n_i$ needed is: $ n_i =\lim_{\rho \to k}\left\lceil\frac{ln\left[\frac{\rho(\phi-1)}{\phi(\rho-1)}\right]}{ln\left[\frac{a}{1-b}\right]}\right\rceil$ where $n_i$ = number of testing iterations necessary to achieve $\rho$, the desired positive predictive value, a = sensitivity, b = specificity, $\phi$ = disease prevalence and $k$ = constant. Based on the aforementioned derivation, we provide reference tables for the number of test iterations needed to obtain a $\rho(\phi)$ of 50, 75, 95 and 99$\%$ as a function of various levels of sensitivity, specificity and disease prevalence.

Methodology
A constructive approach to selective risk control

Zijun Gao,Wenjie Hu,Qingyuan Zhao

2024-01-30

Abstract:Many modern applications require the use of data to both select the statistical tasks and make valid inference after selection. In this article, we provide a unifying approach to control for a class of selective risks. Our method is motivated by a reformulation of the celebrated Benjamini-Hochberg (BH) procedure for multiple hypothesis testing as the iterative limit of the Benjamini-Yekutieli (BY) procedure for constructing post-selection confidence intervals. Although several earlier authors have made noteworthy observations related to this, our discussion highlights that (1) the BH procedure is precisely the fixed-point iteration of the BY procedure; (2) the fact that the BH procedure controls the false discovery rate is almost an immediate corollary of the fact that the BY procedure controls the false coverage-statement rate. Building on this observation, we propose a constructive approach to control extra-selection risk (selection made after decision) by iterating decision strategies that control the post-selection risk (decision made after selection), and show that many previous methods and results are special cases of this general framework. We further extend this approach to problems with multiple selective risks and demonstrate how new methods can be developed. Our development leads to two surprising results about the BH procedure: (1) in the context of one-sided location testing, the BH procedure not only controls the false discovery rate at the null but also at other locations for free; (2) in the context of permutation tests, the BH procedure with exact permutation p-values can be well approximated by a procedure which only requires a total number of permutations that is almost linear in the total number of hypotheses.

Methodology,Statistics Theory,Applications,Computation
How sure are we? Two approaches to statistical inference

Michael Wood

DOI: https://doi.org/10.48550/arXiv.1803.06214

2018-03-15

Other Statistics

Abstract:Suppose you are told that taking a statin will reduce your risk of a heart attack or stroke by 3% in the next ten years, or that women have better emotional intelligence than men. You may wonder how accurate the 3% is, or how confident we should be about the assertion about women's emotional intelligence, bearing in mind that these conclusions are only based on samples of data? My aim here is to present two statistical approaches to questions like these. Approach 1 is often called null hypothesis testing but I prefer the phrase "baseline hypothesis": this is the standard approach in many areas of inquiry but is fraught with problems. Approach 2 can be viewed as a generalisation of the idea of confidence intervals, or as the application of Bayes' theorem. Unlike Approach 1, Approach 2 provides a tentative estimate of the probability of hypotheses of interest. For both approaches, I explain, from first principles, building only on "common sense" statistical concepts like averages and randomness, both how to derive answers, and the rationale behind the answers. This is achieved by using computer simulation methods (resampling and bootstrapping using a spreadsheet available on the web) which avoid the use of probability distributions (t, normal, etc). Such a minimalist, but reasonably rigorous, analysis is particularly useful in a discipline like statistics which is widely used by people who are not specialists. My intended audience includes both statisticians, and users of statistical methods who are not statistical experts.
On an Approach to Bayesian Sample Sizing in Clinical Trials

Robb J. Muirhead,Adina I. Soaita

DOI: https://doi.org/10.48550/arXiv.1204.4460

2012-04-20

Abstract:This paper explores an approach to Bayesian sample size determination in clinical trials. The approach falls into the category of what is often called "proper Bayesian", in that it does not mix frequentist concepts with Bayesian ones. A criterion for a "successful trial" is defined in terms of a posterior probability, its probability is assessed using the marginal distribution of the data, and this probability forms the basis for choosing sample sizes. We illustrate with a standard problem in clinical trials, that of establishing superiority of a new drug over a control.

Methodology,Statistics Theory
On the Benjamini--Hochberg method

J. A. Ferreira,A. H. Zwinderman

DOI: https://doi.org/10.1214/009053606000000425

2006-11-20

Abstract:We investigate the properties of the Benjamini--Hochberg method for multiple testing and of a variant of Storey's generalization of it, extending and complementing the asymptotic and exact results available in the literature. Results are obtained under two different sets of assumptions and include asymptotic and exact expressions and bounds for the proportion of rejections, the proportion of incorrect rejections out of all rejections and two other proportions used to quantify the efficacy of the method.

Statistics Theory
Anscombe's Model for Sequential Clinical Trials Revisited

Sebastian Jobjörnsson,Sören Christensen

DOI: https://doi.org/10.48550/arXiv.1712.05547

2017-12-15

Abstract:In Anscombe's classical model, the objective is to find the optimal sequential rule for learning about the difference between two alternative treatments and subsequently selecting the superior one. The population for which the procedure is optimised has size $N$ and includes both the patients in the trial and those which are treated with the chosen alternative after the trial. We review earlier work on this problem and give a detailed treatment of the problem itself. In particular, while previous work has mainly focused on the case of conjugate normal priors for the incremental effect, we demonstrate how to handle the problem for priors of a general form. We also discuss methods for numerical solutions and the practical implications of the results for the regulation of clinical trials. Two extensions of the model are proposed and analysed. The first breaks the symmetry of the treatments, giving one the role of the current standard being administered in parallel with the trial. We show how certain asymptotic results due to Chernoff can be adapted to this asymmetric case. The other extension assumes that $N$ is a random variable instead of a known constant.

Methodology
Asymptotic false discovery control of the Benjamini-Hochberg procedure for pairwise comparisons

Weidong Liu,Dennis Leung,Qiman Shao

2023-12-09

Abstract:In a one-way analysis-of-variance (ANOVA) model, the number of all pairwise comparisons can be large even when there are only a moderate number of groups. Motivated by this, we consider a regime with a growing number of groups, and prove that for testing pairwise comparisons the BH procedure can offer asymptotic control on false discoveries, despite that the t-statistics involved do not exhibit the well-known positive dependence structure called the PRDS to guarantee exact false discovery rate (FDR) control. Sharing Tukey's viewpoint that the difference in the means of any two groups cannot be exactly zero, our main result is stated in terms of the control on the directional false discovery rate and directional false discovery proportion. A key technical contribution is that we have shown the dependence among the t-statistics to be weak enough to induce a convergence result typically needed for establishing asymptotic FDR control. Our analysis does not adhere to stylized assumptions such as normality, variance homogeneity and a balanced design, and thus provides a theoretical grounding for applications in more general situations.

Statistics Theory
A Bayesian approach on sample size calculation for comparing means.

Hansheng Wang,Shein-Chung Chow,Murphy Chen

DOI: https://doi.org/10.1081/BIP-200067789

2006-01-01

Journal of Biopharmaceutical Statistics

Abstract:In clinical research, parameters required for sample size calculation are usually unknown. A typical approach is to use estimates from some pilot studies as the true parameters in the calculation. This approach, however, does not take into consideration sampling error. Thus, the resulting sample size could be misleading if the sampling error is substantial. As an alternative, we suggest a Bayesian approach with noninformative prior to reflect the uncertainty of the parameters induced by the sampling error. Based on the informative prior and data from pilot samples, the Bayesian estimators based on appropriate loss functions can be obtained. Then, the traditional sample size calculation procedure can be carried out using the Bayesian estimates instead of the frequentist estimates. The results indicate that the sample size obtained using the Bayesian approach differs from the traditional sample size obtained by a constant inflation factor, which is purely determined by the size of the pilot study. An example is given for illustration purposes.
A Practical Approach for Comparing Means of Two Groups Without Equal Variance Assumption

HS Wang,SC Chow

DOI: https://doi.org/10.1002/sim.1238

2002-01-01

Statistics in Medicine

Abstract:In this paper we consider two-groups of i.i.d. normally distributed random variables (N(mu(x),sigma(x)(2)) and N(mu(y),sigma(y)(2))) without assuming equal variance (sigma(x)(2) = sigma(y)(2)). We propose a simple method for constructing confidence bounds based on Howe's approximation I. Its applications in parallel clinical trial (testing H-0 : mu(x) - mu(y) = 0 versus H-1 : mu(x) - mu(y) < 0) and parallel bioequivalence (BE) trial (testing H-0 : \mu(x) - mu(y)\ greater than or equal to delta versus H-1 : \mu(x) - mu(y)\ < delta) are studied. Sample size calculation formulae for both cases are derived. Their performances are evaluated by simulation. Our study shows that the proposed procedure can control type I error satisfactorily compared with Cochran-Cox's and Satterthwaite's approximations while maintaining a relatively high power. The proposed approach is not only simple for constructing the confidence limit, but also provides a simple and accurate formula for sample size calculation. Copyright (C) 2002 John Wiley Sons, Ltd.
A Unified Approach to the Classical Statistical Analysis of Small Signals

Gary J. Feldman,Robert D. Cousins

DOI: https://doi.org/10.1103/PhysRevD.57.3873

1999-12-16

Abstract:We give a classical confidence belt construction which unifies the treatment of upper confidence limits for null results and two-sided confidence intervals for non-null results. The unified treatment solves a problem (apparently not previously recognized) that the choice of upper limit or two-sided intervals leads to intervals which are not confidence intervals if the choice is based on the data. We apply the construction to two related problems which have recently been a battle-ground between classical and Bayesian statistics: Poisson processes with background, and Gaussian errors with a bounded physical region. In contrast with the usual classical construction for upper limits, our construction avoids unphysical confidence intervals. In contrast with some popular Bayesian intervals, our intervals eliminate conservatism (frequentist coverage greater than the stated confidence) in the Gaussian case and reduce it to a level dictated by discreteness in the Poisson case. We generalize the method in order to apply it to analysis of experiments searching for neutrino oscillations. We show that this technique both gives correct coverage and is powerful, while other classical techniques that have been used by neutrino oscillation search experiments fail one or both of these criteria.

Data Analysis, Statistics and Probability,High Energy Physics - Experiment
Another Look at Confidence Intervals: Proposal for a More Relevant and Transparent Approach

Steven D. Biller,Scott M. Oser

DOI: https://doi.org/10.1016/j.nima.2014.11.081

2015-02-03

Abstract:The behaviors of various confidence/credible interval constructions are explored, particularly in the region of low statistics where methods diverge most. We highlight a number of challenges, such as the treatment of nuisance parameters, and common misconceptions associated with such constructions. An informal survey of the literature suggests that confidence intervals are not always defined in relevant ways and are too often misinterpreted and/or misapplied. This can lead to seemingly paradoxical behaviours and flawed comparisons regarding the relevance of experimental results. We therefore conclude that there is a need for a more pragmatic strategy which recognizes that, while it is critical to objectively convey the information content of the data, there is also a strong desire to derive bounds on models and a natural instinct to interpret things this way. Accordingly, we attempt to put aside philosophical biases in favor of a practical view to propose a more transparent and self-consistent approach that better addresses these issues.

Data Analysis, Statistics and Probability,Instrumentation and Methods for Astrophysics,High Energy Physics - Experiment,High Energy Physics - Phenomenology
On randomized confidence intervals for the binomial probability

Paul Kabaila

DOI: https://doi.org/10.48550/arXiv.1302.6659

2013-02-27

Statistics Theory

Abstract:Suppose that X_1,X_2,...,X_n are independent and identically Bernoulli(theta) distributed. Also suppose that our aim is to find an exact confidence interval for theta that is the intersection of a 1-\alpha/2 upper confidence interval and a 1-\alpha/2 lower confidence interval. The Clopper-Pearson interval is the standard such confidence interval for theta, which is widely used in practice. We consider the randomized confidence interval of Stevens, 1950 and present some extensions, including pseudorandomized confidence intervals. We also consider the "data-randomized" confidence interval of Korn, 1987 and point out some additional attractive features of this interval. We also contribute to the discussion about the practical use of such confidence intervals.
A Bayesian approach for evaluating equivalence over multiple groups, and comparison with frequentist tost

Jos Weusten,Ji Young Kim,Katherine Giacoletti,Jorge Vázquez,Plinio De los Santos

DOI: https://doi.org/10.1080/02664763.2023.2297150

IF: 1.416

2024-01-14

Journal of Applied Statistics

Abstract:Manufacturing and testing of pharmaceutical products frequently occur in multiple facilities within a company's network. It is of interest to demonstrate equivalence among the alternative testing/manufacturing facilities to ensure product consistency and quality regardless of the facility where it was manufactured/tested. In the Frequentist framework, equivalence testing is well established when comparing two labs or manufacturing facilities; however, when considering more than two labs or production sites, the Frequentist approach may not always offer appropriate or interpretable estimates for demonstrating equivalence among all of them simultaneously. This paper demonstrates the utility of Bayesian methods to the equivalence assessment of multiple groups means, with a comparison against traditional Frequentist methods. We conclude that a Bayesian strategy is very useful for addressing the problem of multi-group equivalence. While it is not our intention to argue that Bayesian methods should always replace Frequentist ones, we show that among the advantages of a Bayesian analysis is that it provides a more nuanced understanding of the degree of similarity among sites than the hypothesis testing underpinning the Frequentist approach.

statistics & probability
Asymptotic Properties of Approximate Bayesian Computation

David T. Frazier,Gael M. Martin,Christian P. Robert,Judith Rousseau

DOI: https://doi.org/10.48550/arXiv.1607.06903

2018-05-08

Abstract:Approximate Bayesian computation allows for statistical analysis in models with intractable likelihoods. In this paper we consider the asymptotic behaviour of the posterior distribution obtained by this method. We give general results on the rate at which the posterior distribution concentrates on sets containing the true parameter, its limiting shape, and the asymptotic distribution of the posterior mean. These results hold under given rates for the tolerance used within the method, mild regularity conditions on the summary statistics, and a condition linked to identification of the true parameters. Implications for practitioners are discussed.

Methodology,Statistics Theory,Computation
Theoretical Justification of the Bi Error Method

MaryLena Bleile

DOI: https://doi.org/10.48550/arXiv.2009.12453

2020-09-25

Methodology

Abstract:Incorrect usage of $p$-values, particularly within the context of significance testing using the arbitrary .05 threshold, has become a major problem in modern statistical practice. The prevalence of this problem can be traced back to the context-free 5-step method commonly taught to undergraduates: we teach it because it is what is done, and we do it because it is what we are taught. This hold particularly true for practitioners of statistics who are not formal statisticians. Thus, in order to improve scientific practice and overcome statistical dichotomania, an accessible replacement for the 5-step method is warranted. We propose a method foundational on the utilization of the Youden Index as a potential decision threshold, which has been shown in the literature to be effective in conjunction with neutral zones. Unlike the traditional 5-step method, our 5-step method (the Bi Error method) allows for neutral results, does not require $p$-values, and does not provide any default threshold values. Instead, our method explicitly requires contextual error analysis as well as quantification of statistical power. Furthermore, and in part due to its lack of usage of p-values, the method sports improved accessibility. This accessibility is supported by a generalized analytical derivation of the Youden Index.
A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems

Vered Madar,Sandra Batista

DOI: https://doi.org/10.48550/arXiv.1501.05225

2015-01-22

Abstract:We address a common problem in large-scale data analysis, and especially the field of genetics, the huge-scale testing problem, where millions to billions of hypotheses are tested together creating a computational challenge to perform multiple hypotheses testing procedures. As a solution we propose an alternative algorithm to the well used Linear Step Up procedure of Benjamini and Hochberg (1995). Our algorithm requires linear time and does not require any p-value ordering. It permits separating huge-scale testing problems arbitrarily into computationally feasible sets or chunks. Results from the chunks are combined by our algorithm to produce the same results as the controlling procedure on the entire set of tests, thus controlling the global false discovery rate even when p-values are arbitrarily divided. The practical memory usage may also be determined arbitrarily by the size of available memory.

Methodology
Optimal Stopping for Interval Estimation in Bernoulli Trials

Tony Yaacoub,George V. Moustakides,Yajun Mei

DOI: https://doi.org/10.48550/arXiv.1711.06912

2017-11-19

Abstract:We propose an optimal sequential methodology for obtaining confidence intervals for a binomial proportion $\theta$. Assuming that an i.i.d. random sequence of Benoulli($\theta$) trials is observed sequentially, we are interested in designing a)~a stopping time $T$ that will decide when is the best time to stop sampling the process, and b)~an optimum estimator $\hat{\theta}_{T}$ that will provide the optimum center of the interval estimate of $\theta$. We follow a semi-Bayesian approach, where we assume that there exists a prior distribution for $\theta$, and our goal is to minimize the average number of samples while we guarantee a minimal coverage probability level. The solution is obtained by applying standard optimal stopping theory and computing the optimum pair $(T,\hat{\theta}_{T})$ numerically. Regarding the optimum stopping time component $T$, we demonstrate that it enjoys certain very uncommon characteristics not encountered in solutions of other classical optimal stopping problems. Finally, we compare our method with the optimum fixed-sample-size procedure but also with existing alternative sequential schemes.

Methodology
Asymptotic Representations for Sequential Decisions, Adaptive Experiments, and Batched Bandits

Keisuke Hirano,Jack R. Porter

DOI: https://doi.org/10.48550/arXiv.2302.03117

2023-02-06

Econometrics

Abstract:We develop asymptotic approximation results that can be applied to sequential estimation and inference problems, adaptive randomized controlled trials, and other statistical decision problems that involve multiple decision nodes with structured and possibly endogenous information sets. Our results extend the classic asymptotic representation theorem used extensively in efficiency bound theory and local power analysis. In adaptive settings where the decision at one stage can affect the observation of variables in later stages, we show that a limiting data environment characterizes all limit distributions attainable through a joint choice of an adaptive design rule and statistics applied to the adaptively generated data, under local alternatives. We illustrate how the theory can be applied to study the choice of adaptive rules and end-of-sample statistical inference in batched (groupwise) sequential adaptive experiments.
Joining Forces of Bayesian and Frequentist Methodology: A Study for Inference in the Presence of Non-Identifiability

Andreas Raue,Clemens Kreutz,Fabian Joachim Theis,Jens Timmer

DOI: https://doi.org/10.1098/rsta.2011.0544

2012-02-21

Abstract:Increasingly complex applications involve large datasets in combination with non-linear and high dimensional mathematical models. In this context, statistical inference is a challenging issue that calls for pragmatic approaches that take advantage of both Bayesian and frequentist methods. The elegance of Bayesian methodology is founded in the propagation of information content provided by experimental data and prior assumptions to the posterior probability distribution of model predictions. However, for complex applications experimental data and prior assumptions potentially constrain the posterior probability distribution insufficiently. In these situations Bayesian Markov chain Monte Carlo sampling can be infeasible. From a frequentist point of view insufficient experimental data and prior assumptions can be interpreted as non-identifiability. The profile likelihood approach offers to detect and to resolve non-identifiability by experimental design iteratively. Therefore, it allows one to better constrain the posterior probability distribution until Markov chain Monte Carlo sampling can be used securely. Using an application from cell biology we compare both methods and show that a successive application of both methods facilitates a realistic assessment of uncertainty in model predictions.

Data Analysis, Statistics and Probability
Repro Samples Method for Finite- and Large-Sample Inferences

Min-ge Xie,Peng Wang

DOI: https://doi.org/10.48550/arXiv.2206.06421

2022-06-13

Methodology

Abstract:This article presents a novel, general, and effective simulation-inspired approach, called {\it repro samples method}, to conduct statistical inference. The approach studies the performance of artificial samples, referred to as {\it repro samples}, obtained by mimicking the true observed sample to achieve uncertainty quantification and construct confidence sets for parameters of interest with guaranteed coverage rates. Both exact and asymptotic inferences are developed. An attractive feature of the general framework developed is that it does not rely on the large sample central limit theorem and is likelihood-free. As such, it is thus effective for complicated inference problems which we can not solve using the large sample central limit theorem. The proposed method is applicable to a wide range of problems, including many open questions where solutions were previously unavailable, for example, those involving discrete or non-numerical parameters. To reduce the large computational cost of such inference problems, we develop a unique matching scheme to obtain a data-driven candidate set. Moreover, we show the advantages of the proposed framework over the classical Neyman-Pearson framework. We demonstrate the effectiveness of the proposed approach on various models throughout the paper and provide a case study that addresses an open inference question on how to quantify the uncertainty for the unknown number of components in a normal mixture model. To evaluate the empirical performance of our repro samples method, we conduct simulations and study real data examples with comparisons to existing approaches. Although the development pertains to the settings where the large sample central limit theorem does not apply, it also has direct extensions to the cases where the central limit theorem does hold.

An approach theoretic version of Anscombe's theorem with an application in biostatistics

Bayesian Updating and Sequential Testing: Overcoming Inferential Limitations of Screening Tests

A constructive approach to selective risk control

How sure are we? Two approaches to statistical inference

On an Approach to Bayesian Sample Sizing in Clinical Trials

On the Benjamini--Hochberg method

Anscombe's Model for Sequential Clinical Trials Revisited

Asymptotic false discovery control of the Benjamini-Hochberg procedure for pairwise comparisons

A Bayesian approach on sample size calculation for comparing means.

A Practical Approach for Comparing Means of Two Groups Without Equal Variance Assumption

A Unified Approach to the Classical Statistical Analysis of Small Signals

Another Look at Confidence Intervals: Proposal for a More Relevant and Transparent Approach

On randomized confidence intervals for the binomial probability

A Bayesian approach for evaluating equivalence over multiple groups, and comparison with frequentist tost

Asymptotic Properties of Approximate Bayesian Computation

Theoretical Justification of the Bi Error Method

A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems

Optimal Stopping for Interval Estimation in Bernoulli Trials

Asymptotic Representations for Sequential Decisions, Adaptive Experiments, and Batched Bandits

Joining Forces of Bayesian and Frequentist Methodology: A Study for Inference in the Presence of Non-Identifiability

Repro Samples Method for Finite- and Large-Sample Inferences