Abstract:Estimating the probability of the binomial distribution is a basic problem, which appears in almost all introductory statistics courses and is performed frequently in various studies. In some cases, the parameter of interest is a difference between two probabilities, and the current work studies the construction of confidence intervals for this parameter when the sample size is small. Our goal is to find the shortest confidence intervals under the constraint of coverage probability being at least as large as a predetermined level. For the two-sample case, there is no known algorithm that achieves this goal, but different heuristics procedures have been suggested, and the present work aims at finding optimal confidence intervals. In the one-sample case, there is a known algorithm that finds optimal confidence intervals presented by Blyth and Still (J Am Stat Assoc 78(381):108–116, 1983). It is based on solving small and local optimization problems and then using an inversion step to find the global optimum solution. We show that this approach fails in the two-sample case and therefore, in order to find optimal confidence intervals, one needs to solve a global optimization problem, rather than small and local ones, which is computationally much harder. We present and discuss the suitable global optimization problem. Using the Gurobi package we find near-optimal solutions when the sample sizes are smaller than 15, and we compare these solutions to some existing methods, both approximate and exact. We find that the improvement in terms of lengths with respect to the best competitor varies between 1.5 and 5% for different parameters of the problem. Therefore, we recommend the use of the new confidence intervals when both sample sizes are smaller than 15. Tables of the confidence intervals are given in the Excel file in this link (https://technionmail-my.sharepoint.com/:f:/g/personal/ap_campus_technion_ac_il/El-213Kms51BhQxR8MmQJCYBDfIsvtrK9mQIey1sZnZWIQ?e=hxGunl).

Sample size for testing difference between two proportions for the bilateral-sample design.

Testing the Equality of Proportions for Combined Unilateral and Bilateral Data

Sample size formula for general win ratio analysis

On sample size calculation based on odds ratio in clinical trials.

A Note on Sample Size Calculation for Mean Comparisons Based on Noncentral T-Statistics.

Optimal confidence interval for the difference between proportions

Determining Sample Size in Binary Measurement System

Estimating sample sizes for binary, ordered categorical, and continuous outcomes in two group comparisons

Hypothesis Test to Compare Two Paired Binomial Proportions: Assessment of 24 Methods

Are the tests overpowered or underpowered? A unified solution to correctly specify type I errors in design of clinical trials for two sample proportions

Sample Size for Assessing Agreement Between Two Methods of Measurement by Bland-Altman Method.

A Simple Formula for Sample Size Calculation in Equivalence Studies

On Sample Size Calculation in Bioequivalence Trials

Common odds ratio test and interval estimation for stratified bilateral and unilateral data

Sample Size for AssessingAgreement between TwoMethods ofMeasurement byBland − Altman Method

Sample Size Determination for Mediation Analysis of Longitudinal Data.

Sample Size Calculation for Comparing Variabilities

Estimating sample sizes for continuous, binary, and ordinal outcomes in paired comparisons: practical hints

An optimal exact confidence interval for the difference of two independent binomial proportions

Sample-Size Formula for the Proportional-Hazards Regression Model

Testing Homogeneity of Proportion Ratios for Stratified Bilateral Correlated Data