A note on optimal designs for estimating the slope of a polynomial regression

Holger Dette,Viatcheslav B. Melas,Petr Shpilev

DOI: https://doi.org/10.1016/j.spl.2020.108992

2021-03-01

Abstract:<p>In this note we consider the optimal design problem for estimating the slope of a polynomial regression with no intercept at a given point, say z. In contrast to previous work, we investigate the model on the non-symmetric interval.</p>

statistics & probability

What problem does this paper attempt to address?

Optimal designs for estimating individual coefficients in polynomial regression with no intercept

Holger Dette,Viatcheslav B. Melas,Petr Shpilev

DOI: https://doi.org/10.48550/arXiv.1906.08343

2019-06-20

Abstract:In a seminal paper \cite{studden1968} characterized $c$-optimal designs in regression models, where the regression functions form a Chebyshev system. He used these results to determine the optimal design for estimating the individual coefficients in a polynomial regression model on the interval $[-1,1]$ explicitly. In this note we identify the optimal design for estimating the individual coefficients in a polynomial regression model with no intercept (here the regression functions do not form a Chebyshev system).

Statistics Theory
A note on the de la Garza phenomenon for locally optimal designs

Holger Dette,Viatcheslav B. Melas

DOI: https://doi.org/10.48550/arXiv.1105.3575

2011-05-18

Statistics Theory

Abstract:The celebrated de la Garza phenomenon states that for a polynomial regression model of degree $p-1$ any optimal design can be based on at most $p$ design points. In a remarkable paper, Yang [Ann. Statist. 38 (2010) 2499--2524] showed that this phenomenon exists in many locally optimal design problems for nonlinear models. In the present note, we present a different view point on these findings using results about moment theory and Chebyshev systems. In particular, we show that this phenomenon occurs in an even larger class of models than considered so far.
Change point estimators by local polynomial fits under a dependence assumption

Zhengyan Lin,Degui Li, Chen

DOI: https://doi.org/10.1016/j.jmva.2008.02.036

IF: 1.387

2008-01-01

Journal of Multivariate Analysis

Abstract:We study a random design regression model generated by dependent observations, when the regression function itself (or its @n-th derivative) may have a change or discontinuity point. A method based on the local polynomial fits with one-sided kernels to estimate the location and the jump size of the change point is applied in this paper. When the jump location is known, a central limit theorem for the estimator of the jump size is established; when the jump location is unknown, we first obtain a functional limit theorem for a local dilated-rescaled version estimator of the jump size and then give the asymptotic distributions for the estimators of the location and the jump size of the change point. The asymptotic results obtained in this paper can be viewed as extensions of corresponding results for independent observations. Furthermore, a simulated example is given to show that our theory and method perform well in practice.
The Asymptotics of Optimal Designs for Polynomial Regression

T. Bloom,L. Bos,N. Levenberg

DOI: https://doi.org/10.48550/arXiv.1112.3735

2011-12-16

Abstract:We give the asymptotics for D-optimal (equivalently G-optimal) designs on a compact (possibly complex) design space.

Methodology
Minimizing the Condition Number to Construct Design Points for Polynomial Regression Models

Jane J. Ye,Julie Zhou

DOI: https://doi.org/10.1137/110850268

IF: 2.763

2013-01-01

SIAM Journal on Optimization

Abstract:In this paper we study a new optimality criterion, the $K$-optimality criterion, for constructing optimal experimental designs for polynomial regression models. We focus on the $p$th order polynomial regression model with symmetric design space $[-1,1]$. For this model, we show that there is always a symmetric $K$-optimal design with exactly $p+1$ support points including the boundary points $-1$ and $1$. It is well known that the condition number for a positive definite matrix as the ratio of the maximum eigenvalue to the minimum eigenvalue is usually nonsmooth. We show that for our model, the condition number of the information matrix is continuously differentiable. Theoretical $K$-optimal designs are derived for $p=1$ and $2$. Numerical results are presented for $3 \le p \le 10$.

mathematics, applied
A note on locally optimal designs for generalized linear models with restricted support

Osama Idais

DOI: https://doi.org/10.1016/j.spl.2019.108692

2020-04-01

Abstract:<p>We consider the generalized linear models. Particular assumptions are proposed to derive a locally optimal design for a model without intercept from a locally optimal design for the corresponding model with intercept and vice versa. We concentrate on D- and A-optimal designs. Applications to Poisson and logistic models and Extensions to nonlinear models are provided.</p>

statistics & probability
Efficient Designs of SLOPE Penalty Sequences in Finite Dimension

Yiliang Zhang,Zhiqi Bu

DOI: https://doi.org/10.48550/arXiv.2102.07211

IF: 5.414

2021-02-14

Machine Learning

Abstract:In linear regression, SLOPE is a new convex analysis method that generalizes the Lasso via the sorted L1 penalty: larger fitted coefficients are penalized more heavily. This magnitude-dependent regularization requires an input of penalty sequence $\lambda$, instead of a scalar penalty as in the Lasso case, thus making the design extremely expensive in computation. In this paper, we propose two efficient algorithms to design the possibly high-dimensional SLOPE penalty, in order to minimize the mean squared error. For Gaussian data matrices, we propose a first order Projected Gradient Descent (PGD) under the Approximate Message Passing regime. For general data matrices, we present a zero-th order Coordinate Descent (CD) to design a sub-class of SLOPE, referred to as the k-level SLOPE. Our CD allows a useful trade-off between the accuracy and the computation speed. We demonstrate the performance of SLOPE with our designs via extensive experiments on synthetic data and real-world datasets.
The polytope of optimal approximate designs: extending the selection of informative experiments

Radoslav Harman,Lenka Filová,Samuel Rosa

DOI: https://doi.org/10.1007/s11222-024-10527-0

IF: 2.3241

2024-11-02

Statistics and Computing

Abstract:Consider the problem of constructing an experimental design, optimal for estimating parameters of a given statistical model with respect to a chosen criterion. To address this problem, the literature usually provides a single solution. Often, however, there exists a rich set of optimal designs, and the knowledge of this set can lead to substantially greater freedom to select an appropriate experiment. In this paper, we demonstrate that the set of all optimal approximate designs generally corresponds to a polytope. Particularly important elements of the polytope are its vertices, which we call vertex optimal designs. We prove that the vertex optimal designs possess unique properties, such as small supports, and outline strategies for how they can facilitate the construction of suitable experiments. Moreover, we show that for a variety of situations it is possible to construct the vertex optimal designs with the assistance of a computer, by employing error-free rational-arithmetic calculations. In such cases the vertex optimal designs are exact, often closely related to known combinatorial designs. Using this approach, we were able to determine the polytope of optimal designs for some of the most common multifactor regression models, thereby extending the choice of informative experiments for a large variety of applications.

statistics & probability,computer science, theory & methods
Optimal discriminating designs for several competing regression models

Dietrich Braess,Holger Dette

DOI: https://doi.org/10.1214/13-AOS1103

2013-06-06

Abstract:The problem of constructing optimal discriminating designs for a class of regression models is considered. We investigate a version of the $T_p$-optimality criterion as introduced by Atkinson and Fedorov [Biometrika 62 (1975a) 289-303]. The numerical construction of optimal designs is very hard and challenging, if the number of pairwise comparisons is larger than 2. It is demonstrated that optimal designs with respect to this type of criteria can be obtained by solving (nonlinear) vector-valued approximation problems. We use a characterization of the best approximations to develop an efficient algorithm for the determination of the optimal discriminating designs. The new procedure is compared with the currently available methods in several numerical examples, and we demonstrate that the new method can find optimal discriminating designs in situations where the currently available procedures fail.

Statistics Theory
Optimal Subsampling Design for Polynomial Regression in one Covariate

Torsten Reuter,Rainer Schwabe

DOI: https://doi.org/10.48550/arXiv.2301.03295

2023-02-25

Abstract:Improvements in technology lead to increasing availability of large data sets which makes the need for data reduction and informative subsamples ever more important. In this paper we construct $ D $-optimal subsampling designs for polynomial regression in one covariate for invariant distributions of the covariate. We study quadratic regression more closely for specific distributions. In particular we make statements on the shape of the resulting optimal subsampling designs and the effect of the subsample size on the design. To illustrate the advantage of the optimal subsampling designs we examine the efficiency of uniform random subsampling.

Statistics Theory
Optimal designs for linear regression models with skew-normal errors

Xiao-Dong Zhou,Yun-Juan Wang,Rong-Xian Yue

DOI: https://doi.org/10.1080/02331888.2024.2410802

IF: 2.346

2024-10-02

Statistics

Abstract:This paper investigates optimal designs for linear regression models with skew-normal errors which provide an alternative to traditional models with symmetric normal errors and eliminate the need for data transformation. Using the maximum likelihood estimation approach, we consider various design criteria, including D -, A -, As -optimality. Equivalence theorems are provided to determine the optimality of the designs. Optimal designs for a centered parametrization model are also considered. Two examples are provided to illustrate the theoretical results. In addition, the efficiencies of the optimal designs based on skew-normal errors are compared with those based on normal errors.

statistics & probability
D-Optimal Designs for the Mitscherlich Non-Linear Regression Function

Maliheh Heidari,Md Abu Manju,Pieta C. IJzerman-Boon,Edwin R. van den Heuvel

DOI: https://doi.org/10.3103/S1066530722010033

2022-07-20

Mathematical Methods of Statistics

Abstract:Mitscherlich's function is a well-known three-parameter non-linear regression function that quantifies the relation between a stimulus or a time variable and a response. It has many applications, in particular in the field of measurement reliability. Optimal designs for estimation of this function have been constructed only for normally distributed responses with homoscedastic variances. In this paper we generalize this literature to D-optimal designs for discrete and continuous responses having their distribution function in the exponential family. We also demonstrate that our D-optimal designs can be identical to and different from optimal designs for variance weighted linear regression.
Optimal designs for comparing several regression curves

Chang-Yu Liu,Xin Liu,Rong-Xian Yue

DOI: https://doi.org/10.1007/s42952-024-00272-5

2024-06-15

Journal of the Korean Statistical Society

Abstract:This article is concerned with the optimal design problem of efficient statistical inference for comparing several regression curves estimated from samples of independent measurements. The objective is to find the -optimal designs that minimize an -norm of the asymptotic variance of the prediction for the contrasts of k regression curves. General equivalence theorems are established to verify the -optimality in the set of all approximate designs. Invariant property with respect to model reparameterization are also obtained. The results obtained for the linear models are extended to the situation of generalized linear models. Three examples are presented to illustrate the applications of the obtained results.

statistics & probability
A Characterization of Optimal Prediction Measures via $\ell_1$ Minimization

Len Bos

2023-12-06

Abstract:Suppose that $K\subset\C$ is compact and that $z_0\in\C\backslash K$ is an external point. An optimal prediction measure for regression by polynomials of degree at most $n,$ is one for which the variance of the prediction at $z_0$ is as small as possible. Hoel and Levine (\cite{HL}) have considered the case of $K=[-1,1]$ and $z_0=x_0\in \R\backslash [-1,1],$ where they show that the support of the optimal measure is the $n+1$ extremme points of the Chebyshev polynomial $T_n(x)$ and characterizing the optimal weights in terms of absolute values of fundamental interpolating Lagrange polynomials. More recently, \cite{BLO} has given the equivalence of the optimal prediction problem with that of finding polynomials of extremal growth. They also study in detail the case of $K=[-1,1]$ and $z_0=ia\in i\R,$ purely imaginary. In this work we generalize the Hoel-Levine formula to the general case when the support of the optimal measure is a finite set and give a formula for the optimal weights in terms of a $\ell_1$ minimization problem.

Statistics Theory
CVXSADes: a stochastic algorithm for constructing optimal exact regression designs with single or multiple objectives

Chi-Kuang Yeh,Julie Zhou

DOI: https://doi.org/10.48550/arXiv.2405.02983

2024-05-06

Abstract:We propose an algorithm to construct optimal exact designs (EDs). Most of the work in the optimal regression design literature focuses on the approximate design (AD) paradigm due to its desired properties, including the optimality verification conditions derived by Kiefer (1959, 1974). ADs may have unbalanced weights, and practitioners may have difficulty implementing them with a designated run size $n$. Some EDs are constructed using rounding methods to get an integer number of runs at each support point of an AD, but this approach may not yield optimal results. To construct EDs, one may need to perform new combinatorial constructions for each $n$, and there is no unified approach to construct them. Therefore, we develop a systematic way to construct EDs for any given $n$. Our method can transform ADs into EDs while retaining high statistical efficiency in two steps. The first step involves constructing an AD by utilizing the convex nature of many design criteria. The second step employs a simulated annealing algorithm to search for the ED stochastically. Through several applications, we demonstrate the utility of our method for various design problems. Additionally, we show that the design efficiency approaches unity as the number of design points increases.

Methodology,Applications,Computation
Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit

Zhiqi Bu,Jason Klusowski,Cynthia Rush,Weijie J. Su

DOI: https://doi.org/10.48550/arXiv.2105.13302

2022-06-06

Abstract:Sorted l1 regularization has been incorporated into many methods for solving high-dimensional statistical estimation problems, including the SLOPE estimator in linear regression. In this paper, we study how this relatively new regularization technique improves variable selection by characterizing the optimal SLOPE trade-off between the false discovery proportion (FDP) and true positive proportion (TPP) or, equivalently, between measures of type I error and power. Assuming a regime of linear sparsity and working under Gaussian random designs, we obtain an upper bound on the optimal trade-off for SLOPE, showing its capability of breaking the Donoho-Tanner power limit. To put it into perspective, this limit is the highest possible power that the Lasso, which is perhaps the most popular l1-based method, can achieve even with arbitrarily strong effect sizes. Next, we derive a tight lower bound that delineates the fundamental limit of sorted l1 regularization in optimally trading the FDP off for the TPP. Finally, we show that on any problem instance, SLOPE with a certain regularization sequence outperforms the Lasso, in the sense of having a smaller FDP, larger TPP and smaller l2 estimation risk simultaneously. Our proofs are based on a novel technique that reduces a calculus of variations problem to a class of infinite-dimensional convex optimization problems and a very recent result from approximate message passing theory.

Statistics Theory,Information Theory,Machine Learning,Signal Processing
Optimal Designs for Regression on Lie Groups

Somnath Chakraborty,Holger Dette,Martin Kroll

2024-10-01

Abstract:We consider a linear regression model with complex-valued response and predictors from a compact and connected Lie group. The regression model is formulated in terms of eigenfunctions of the Laplace-Beltrami operator on the Lie group. We show that the normalized Haar measure is an approximate optimal design with respect to all Kiefer's $\Phi_p$-criteria. Inspired by the concept of $t$-designs in the field of algebraic combinatorics, we then consider so-called $\lambda$-designs in order to construct exact $\Phi_p$-optimal designs for fixed sample sizes in the considered regression problem. In particular, we explicitly construct $\Phi_p$-optimal designs for regression models with predictors in the Lie groups $\mathrm{SU}(2)$ and $\mathrm{SO}(3)$, the groups of $2\times 2$ unitary matrices and $3\times 3$ orthogonal matrices with determinant equal to $1$, respectively. We also discuss the advantages of the derived theoretical results in a concrete biological application.

Statistics Theory
Optimal designs for linear models with Fredholm-type errors

Jun Yu,Mingyao Ai,Yaping Wang

DOI: https://doi.org/10.1016/j.jspi.2017.09.018

IF: 1.095

2018-01-01

Journal of Statistical Planning and Inference

Abstract:A new class of estimators, matrix weighted estimators (MWEs), have been proposed for the parameters in linear models with correlated errors. In this paper, we consider the design problem for linear models with Fredholm-type errors, i.e., the errors with continuous covariance kernels, when MWEs are used. By applying the tools of stochastic analysis and measure theory, we derive the analytical expressions of the optimal designs under some restrictions. To treat the general cases, an approximation method for the optimal designs is then introduced, which can reduce the computational cost. Practical implementations for obtaining designs with finite sample sizes are demonstrated. Numerical examples show that the obtained approximate designs are very close to the optimal ones.
Computing optimal experimental designs with respect to a compound Bayes risk criterion

Radoslav Harman,Maryna Prus

DOI: https://doi.org/10.48550/arXiv.1709.02317

2017-09-07

Abstract:We consider the problem of computing optimal experimental design on a finite design space with respect to a compound Bayes risk criterion, which includes the linear criterion for prediction in a random coefficient regression model. We show that the problem can be restated as constrained A-optimality in an artificial model. This permits using recently developed computational tools, for instance the algorithms based on the second-order cone programming for optimal approximate design, and mixed-integer second-order cone programming for optimal exact designs. We demonstrate the use of the proposed method for the problem of computing optimal designs of a random coefficient regression model with respect to an integrated mean squared error criterion.

Computation
$E$-optimal designs for second-order response surface models

Holger Dette,Yuri Grigoriev

DOI: https://doi.org/10.1214/14-AOS1241

2014-08-29

Abstract:$E$-optimal experimental designs for a second-order response surface model with $k\geq1$ predictors are investigated. If the design space is the $k$-dimensional unit cube, Galil and Kiefer [J. Statist. Plann. Inference 1 (1977a) 121-132] determined optimal designs in a restricted class of designs (defined by the multiplicity of the minimal eigenvalue) and stated their universal optimality as a conjecture. In this paper, we prove this claim and show that these designs are in fact $E$-optimal in the class of all approximate designs. Moreover, if the design space is the unit ball, $E$-optimal designs have not been found so far and we also provide a complete solution to this optimal design problem. The main difficulty in the construction of $E$-optimal designs for the second-order response surface model consists in the fact that for the multiplicity of the minimum eigenvalue of the "optimal information matrix" is larger than one (in contrast to the case $k=1$) and as a consequence the corresponding optimality criterion is not differentiable at the optimal solution. These difficulties are solved by considering nonlinear Chebyshev approximation problems, which arise from a corresponding equivalence theorem. The extremal polynomials which solve these Chebyshev problems are constructed explicitly leading to a complete solution of the corresponding $E$-optimal design problems.

Methodology

A note on optimal designs for estimating the slope of a polynomial regression

Optimal designs for estimating individual coefficients in polynomial regression with no intercept

A note on the de la Garza phenomenon for locally optimal designs

Change point estimators by local polynomial fits under a dependence assumption

The Asymptotics of Optimal Designs for Polynomial Regression

Minimizing the Condition Number to Construct Design Points for Polynomial Regression Models

A note on locally optimal designs for generalized linear models with restricted support

Efficient Designs of SLOPE Penalty Sequences in Finite Dimension

The polytope of optimal approximate designs: extending the selection of informative experiments

Optimal discriminating designs for several competing regression models

Optimal Subsampling Design for Polynomial Regression in one Covariate

Optimal designs for linear regression models with skew-normal errors

D-Optimal Designs for the Mitscherlich Non-Linear Regression Function

Optimal designs for comparing several regression curves

A Characterization of Optimal Prediction Measures via $\ell_1$ Minimization

CVXSADes: a stochastic algorithm for constructing optimal exact regression designs with single or multiple objectives

Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit

Optimal Designs for Regression on Lie Groups

Optimal designs for linear models with Fredholm-type errors

Computing optimal experimental designs with respect to a compound Bayes risk criterion

$E$-optimal designs for second-order response surface models