A discussion of the paper "Safe testing" by Grünwald, de Heide, and Koolen

Joshua Bon,Christian P Robert

2024-02-09

Abstract:This is a discussion of the paper "Safe testing" by Grünwald, de Heide, and Koolen, Read before The Royal Statistical Society at a meeting organized by the Research Section on Wednesday, 24 January, 2024

Methodology,Statistics Theory

What problem does this paper attempt to address?

Chloe Krakauer and Kenneth Rice’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Chloe Krakauer,Kenneth Rice

DOI: https://doi.org/10.1111/rssa.12660

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 452-453, April 2021.

statistics & probability,social sciences, mathematical methods
Sander Greenland’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Sander Greenland

DOI: https://doi.org/10.1111/rssa.12659

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 450-451, April 2021.

statistics & probability,social sciences, mathematical methods
Three discussions of the paper "sequential quasi-Monte Carlo sampling", by M. Gerber and N. Chopin

Julyan Arbel,Igor Prunster,Christian P. Robert,Robin J. Ryder

DOI: https://doi.org/10.1111/rssb.12104

2015-05-26

Abstract:This is a collection of three written discussions of the paper "sequential quasi-Monte Carlo sampling" by M. Gerber and N. Chopin, following the presentation given before the Royal Statistical Society in London on December 10th, 2014.

Computation
Discussion of the paper "Optimal treatment allocations in space and time for on-line control of an emerging infectious disease'' by E. B. Laber, N. J. Meyer, B. J. Reich, K. Pacifici, J. A. Collazo and J. Drake

Johan Koskinen

DOI: https://doi.org/10.48550/arXiv.2006.16527

2020-06-30

Abstract:This is a written discussion of the paper "Optimal treatment allocations in space and time for on-line control of an emerging infectious disease" by E. B. Laber, N. J. Meyer, B. J. Reich, K. Pacifici, J. A. Collazo and J. Drake, contributed to the Journal of the Royal Statistical Society Series C

Methodology
A distillation of the live chat during the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Paul A. Smith

DOI: https://doi.org/10.1111/rssa.12671

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 448-448, April 2021.

statistics & probability,social sciences, mathematical methods
Safe Testing

Peter Grünwald,Rianne de Heide,Wouter Koolen

DOI: https://doi.org/10.1093/jrsssb/qkae011

2024-03-07

Abstract:Abstract We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2×2 contingency table. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.

statistics & probability
Discussion on the paper by Xia, Tong, Li and Zhu

JT Kent,Adrian Bowman,Santiago Velilla,Wenyang Zhang,Frank Critchley,AC Atkinson,Qiwei Yao,AH Welsh,Hengjian Cui,Guoying Li,Vladimir Spokoiny,KS Chan,Ming Chung Li,Pavel Čížek,Wolfgang Härdle,Lijian Yang,RD Cook,Jianqing Fan,L Ferré,Ker Chau Li,Lexin Li,Oliver Linton,Liqiang Ni,Megu Ohtaki,Yasunori Fujikoshi,James R Schott,CM Setodji,Nils Chr Stenseth,Ole Chr Lingjærde

2002-01-01

Abstract:Kent, JT, Bowman, A, Velilla, S, Zhang, W, Critchley, F, Atkinson, AC, Yao, Q, Welsh, AH, Cui, H, Li, G, Spokoiny, V, Chan, KS, Li, MC, Čížek, P, Härdle, W, Yang, L, Cook, RD, Fan, J, Ferré, L, Li, KC, Li, L, Linton, O, Ni, L, Ohtaki, M, Fujikoshi, Y, Schott, JR, Setodji, CM, Stenseth, NC & Lingjærde, OC 2002, 'Discussion on the paper by Xia, Tong, Li and Zhu', Journal of the Royal Statistical Society. Series B: Statistical Methodology, vol. 64, no. 3, pp. 388-410. … Discussion on the paper by Xia, Tong, Li and Zhu. / Kent, JT; Bowman, Adrian; Velilla, Santiago; Zhang, Wenyang; Critchley, Frank; Atkinson, AC; Yao, Qiwei; Welsh, AH; Cui, Hengjian; Li, Guoying; Spokoiny, Vladimir; Chan, KS; Li, Ming Chung; Čížek, Pavel; Härdle, Wolfgang; Yang, Lijian; Cook, RD; Fan, Jianqing; Ferré, L.; Li, Ker Chau; Li, Lexin; Linton, Oliver; Ni, Liqiang; Ohtaki, Megu; Fujikoshi, Yasunori; Schott, James R.; Setodji, CM; …
Group Testing in Arbitrary Hypergraphs and Related Combinatorial Structures

Annalisa De Bonis

DOI: https://doi.org/10.48550/arXiv.2307.09608

2023-11-28

Abstract:We consider a generalization of group testing where the potentially contaminated sets are the members of a given hypergraph ${\cal F}=(V,E)$. This generalization finds application in contexts where contaminations can be conditioned by some kinds of social and geographical clusterings. We study non-adaptive algorithms, two-stage algorithms, and three-stage algorithms. Non-adaptive group testing algorithms are algorithms in which all tests are decided beforehand and therefore can be performed in parallel, whereas two-stage group testing algorithms and three-stage group testing algorithms are algorithms that consist of two stages and three stages, respectively, with each stage being a completely non-adaptive algorithm. In classical group testing, the potentially infected sets are all subsets of up to a certain number of elements of the given input set. For classical group testing, it is known that there exists a correspondence between classical superimposed codes and non-adaptive algorithms, and between two stage algorithms and selectors. Bounds on the number of tests for those algorithms are derived from the bounds on the dimensions of the corresponding combinatorial structures. Obviously, the upper bounds for the classical case apply also to our group testing model. In the present paper, we aim at improving on those upper bounds by leveraging on the characteristics of the particular hypergraph at hand. In order to cope with our version of the problem, we introduce new combinatorial structures that generalize the notions of classical selectors and superimposed codes.

Data Structures and Algorithms,Discrete Mathematics
Arthur Paul Pedersen’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Arthur Paul Pedersen

DOI: https://doi.org/10.1111/rssa.12655

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 443-444, April 2021.

statistics & probability,social sciences, mathematical methods
Ryan Martin’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Ryan Martin

DOI: https://doi.org/10.1111/rssa.12665

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 456-457, April 2021.

statistics & probability,social sciences, mathematical methods
Paul Vos’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Paul Vos

DOI: https://doi.org/10.1111/rssa.12668

2021-04-01

Abstract:Journal of the Royal Statistical Society: Series A (Statistics in Society), Volume 184, Issue 2, Page 461-462, April 2021.

statistics & probability,social sciences, mathematical methods
Working Across Boundaries.

Robert M. Hierons,Tao Xie

DOI: https://doi.org/10.1002/stvr.1734

2020-01-01

Software Testing Verification and Reliability

Abstract:This editorial was written during a period of extreme difficulty for many individuals, families, and nations in the ongoing COVID-19 outbreak. We can only hope that measures taken are successful and that the situation has improved considerably. We also do not pretend that we have anything to add regarding health, social, or economic issues. However, the crisis has shown the role that Computer Science can play in informing policy. Society requires evidence and computers are often involved in producing such evidence via, for example, simulation. It is here that we, as a community, can contribute through advances in testing, verification, and reliability in areas such as Scientific Computing and Computer Simulations and maybe also AI/data sciences for helping expedite the process of finding treatment. As a recent example discussed in social media, when commenting on pandemic simulation code used to model control measures against COVID-19, Prof. Guido Salvaneschi said in his tweet: “Ever wondered about the “impact“ of research on programming languages and software engineering? Political decisions affecting hundreds of millions are being taken based on thousands of lines of 13+ years old C code that allegedly nobody understands anymore. #COVID19 #cs” (https://twitter.com/guidosalva/status/1242049884347412482). There is already some truly excellent work for making advances in these areas and we are confident that the community will rise to the challenge. This issue contains two papers. In the first paper, Simons and Lefticaru introduce a new Model-Based Testing approach, which is based on the use of a Stream X-machine (SXM) specification. SXMs provide a state-based formalism and there is a traditional approach to testing from an SXM. This approach typically assumes that the underlying functions/operations have been implemented correctly but these functions may be integrated (into a state machine) in the wrong way. There are a number of automated test generation approaches for SXMs and the authors make two main additional contributions to this area. First, they introduce a number of novel optimisations into test generation. Second, they observe that SXM test generation algorithms return abstract test cases (sequences of functions); the paper shows how corresponding concrete test data can be generated. The approach has been implemented and evaluated on case studies, with the tool also checking that a specification satisfies certain desirable properties. (Recommended by Hyunsook Do). In the second paper, Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman, and Arie van Deursen introduce behavioural model seeding, a new seeding approach for learning class usages from both the system source code under test and existing test cases. The learned class usages are represented in a state-machine-based behavioural model. The behavioural model is then used to guide search-based crash reproduction, which generates a test case (i.e., objects and sequences of method calls on those objects) to reproduce a crash given its stack trace. This approach is in contrast to the existing seeding strategies, which simply collect and reuse values and object states from the system source code under test and existing test cases without any abstraction. The approach has been implemented in an open-source implementation named the BOTSING toolset and evaluated on 122 crashes from six open-source applications. (Recommended by Phil McMinn).
Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning

Stefan Pranger,Hana Chockler,Martin Tappler,Bettina Könighofer

2024-11-12

Abstract:In many Deep Reinforcement Learning (RL) problems, decisions in a trained policy vary in significance for the expected safety and performance of the policy. Since RL policies are very complex, testing efforts should concentrate on states in which the agent's decisions have the highest impact on the expected outcome. In this paper, we propose a novel model-based method to rigorously compute a ranking of state importance across the entire state space. We then focus our testing efforts on the highest-ranked states. In this paper, we focus on testing for safety. However, the proposed methods can be easily adapted to test for performance. In each iteration, our testing framework computes optimistic and pessimistic safety estimates. These estimates provide lower and upper bounds on the expected outcomes of the policy execution across all modeled states in the state space. Our approach divides the state space into safe and unsafe regions upon convergence, providing clear insights into the policy's weaknesses. Two important properties characterize our approach. (1) Optimal Test-Case Selection: At any time in the testing process, our approach evaluates the policy in the states that are most critical for safety. (2) Guaranteed Safety: Our approach can provide formal verification guarantees over the entire state space by sampling only a fraction of the policy. Any safety properties assured by the pessimistic estimate are formally proven to hold for the policy. We provide a detailed evaluation of our framework on several examples, showing that our method discovers unsafe policy behavior with low testing effort.

Machine Learning
Remarks on the statistical aspects of the safety analysis

Lenard Pal,Mihaly Makai

DOI: https://doi.org/10.48550/arXiv.1306.1336

2013-06-06

Abstract:We investigate the statistical methods applied throughout safety analysis of complex systems. The tolerance interval method implemented in the widely utilized 0.95|0.95 methodology is analyzed. We point out a remarkable weakness of the tolerance interval method concerning the principle of repeatability. It is proved that repeating twice the procedure, the probability that the second maximum will be larger/smaller than the first one is 50\%. This statement is not surprising, it holds for any random variable with continuous distribution function. In order to demonstrate the undesirable consequences of the tolerance interval method in the decision making, the results of the analysis of an elementary example are discussed. Instead of the tolerance interval method, we suggest another method based on the sign test which has more encouraging features, especially in the case of several output variables. The problematic aspects of the method are also discussed. Finally, we suggest a simple test case which is able to reveal if the tolerance interval method would not be capable of determining the risky states of the system, when there are only a few of them. If there are many, then the method may not explore each one in the analysis.

Data Analysis, Statistics and Probability
Statistical significance testing and p-values: Defending the indefensible? A discussion paper and position statement

Peter Griffiths,Jack Needleman

DOI: https://doi.org/10.1016/j.ijnurstu.2019.07.001

Abstract:Much statistical teaching and many research reports focus on the 'null hypothesis significance test'. Yet the correct meaning and interpretation of statistical significance tests is elusive. Misinterpretations are both common and persistent, leading many to question whether significance tests should be used at all. While most take aim at the arbitrary declaration of p < 0.05 as a threshold for determining 'significance', others extend the critique to suggest the 'p-value' should be dispensed with entirely. P-values and significance tests are still widely used as if they give a measure of the size and importance of relationships, even though this misunderstanding has been observed and discussed for many years. We argue that p-values and significance tests are intrinsically misleading. Point estimates of relationships and confidence intervals give direct information about the effect and the uncertainty of the estimate without recourse to interpreting how a particular p-value might have arisen or indeed referring to them at all. In this paper we briefly outline some of the problems with significance testing, offer a number of examples selected from a recent issue of the International Journal of Nursing Studies and discuss some proposed responses to these problems. We conclude by offering some guidance to authors reporting statistical tests in journals and present a position statement that has been adopted by the International Journal of Nursing Studies to guide its' authors in reporting the results of statistical analyses. While stopping short of calling for an outright ban on reporting p-values and significance tests we urge authors (and journals) to place more emphasis on measures of effect and estimates of precision/uncertainty and, following the position of the American Statistical Association, emphasise that authors (and readers) should avoid using 0.05 or any other cut off for a p-value as the basis for a decision about the meaningfulness/importance of an effect. If point estimates and confidence intervals are used, then the p-value may be redundant and can be omitted from reports. When authors talk about 'significance' they need to be explicit when referring to statistical significance and we recommend authors adopt the language of 'importance' when talking about effect sizes to avoid any confusion.
Discussion of "Statistical Inference: The Big Picture" by R. E. Kass

Robert McCulloch

DOI: https://doi.org/10.1214/11-STS337D

2011-06-17

Abstract:Discussion of "Statistical Inference: The Big Picture" by R. E. Kass [<a class="link-https" data-arxiv-id="1106.2895" href="https://arxiv.org/abs/1106.2895">arXiv:1106.2895</a>]

Methodology
Discussion: On Arguments Concerning Statistical Principles

D. A. S. Fraser

DOI: https://doi.org/10.48550/arXiv.1411.0809

2014-11-04

Methodology

Abstract:Discussion of "On the Birnbaum Argument for the Strong Likelihood Principle" by Deborah G. Mayo [arXiv:1302.7021].
Revisiting nested group testing procedures: new results, comparisons, and robustness

Yaakov Malinovsky,Paul S. Albert

DOI: https://doi.org/10.48550/arXiv.1608.06330

2017-07-26

Abstract:Group testing has its origin in the identification of syphilis in the US army during World War II. Much of the theoretical framework of group testing was developed starting in the late 1950s, with continued work into the 1990s. Recently, with the advent of new laboratory and genetic technologies, there has been an increasing interest in group testing designs for cost saving purposes. In this paper, we compare different nested designs, including Dorfman, Sterrett and an optimal nested procedure obtained through dynamic programming. To elucidate these comparisons, we develop closed-form expressions for the optimal Sterrett procedure and provide a concise review of the prior literature for other commonly used procedures. We consider designs where the prevalence of disease is known as well as investigate the robustness of these procedures when it is incorrectly assumed. This article provides a technical presentation that will be of interest to researchers as well as from a pedagogical perspective. Supplementary material for this article is available online.

Other Statistics
Testing Software in Age of Data Privacy: a Balancing Act.

Kunal Taneja,Mark Grechanik,Rayid Ghani,Tao Xie

DOI: https://doi.org/10.1145/2025113.2025143

2011-01-01

Abstract:ABSTRACTDatabase-centric applications (DCAs) are common in enterprise computing, and they use nontrivial databases. Testing of DCAs is increasingly outsourced to test centers in order to achieve lower cost and higher quality. When proprietary DCAs are released, their databases should also be made available to test engineers. However, different data privacy laws prevent organizations from sharing this data with test centers because databases contain sensitive information. Currently, testing is performed with anonymized data, which often leads to worse test coverage (such as code coverage) and fewer uncovered faults, thereby reducing the quality of DCAs and obliterating benefits of test outsourcing. To address this issue, we offer a novel approach that combines program analysis with a new data privacy framework that we design to address constraints of software testing. With our approach, organizations can balance the level of privacy with needs of testing. We have built a tool for our approach and applied it to nontrivial Java DCAs. Our results show that test coverage can be preserved at a higher level by anonymizing data based on their effect on corresponding DCAs.

A discussion of the paper "Safe testing" by Grünwald, de Heide, and Koolen

Chloe Krakauer and Kenneth Rice’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Sander Greenland’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Three discussions of the paper "sequential quasi-Monte Carlo sampling", by M. Gerber and N. Chopin

Discussion of the paper "Optimal treatment allocations in space and time for on-line control of an emerging infectious disease'' by E. B. Laber, N. J. Meyer, B. J. Reich, K. Pacifici, J. A. Collazo and J. Drake

A distillation of the live chat during the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Safe Testing

Discussion on the paper by Xia, Tong, Li and Zhu

Group Testing in Arbitrary Hypergraphs and Related Combinatorial Structures

Arthur Paul Pedersen’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Ryan Martin’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Paul Vos’s contribution to the Discussion of ‘Testing by betting: A strategy for statistical and scientific communication’ by Glenn Shafer

Working Across Boundaries.

Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning

Remarks on the statistical aspects of the safety analysis

Statistical significance testing and p-values: Defending the indefensible? A discussion paper and position statement

Discussion of "Statistical Inference: The Big Picture" by R. E. Kass

Discussion: On Arguments Concerning Statistical Principles

Revisiting nested group testing procedures: new results, comparisons, and robustness

Testing Software in Age of Data Privacy: a Balancing Act.