On a Multiplicative Algorithm for Computing Bayesian D-optimal Designs

Yaming Yu

DOI: https://doi.org/10.48550/arXiv.1005.2355

2010-05-13

Abstract:We use the minorization-maximization principle (Lange, Hunter and Yang 2000) to establish the monotonicity of a multiplicative algorithm for computing Bayesian D-optimal designs. This proves a conjecture of Dette, Pepelyshev and Zhigljavsky (2008).

Computation,Methodology

What problem does this paper attempt to address?

A Fast Algorithm For Maximizing A Non-Monotone Dr-Submodular Integer Lattice Function

Qingqin Nong,Jiazhu Fang,Suning Gong,Yan Feng,Xiaoying Qu

DOI: https://doi.org/10.1016/j.tcs.2020.08.011

IF: 1.002

2020-01-01

Theoretical Computer Science

Abstract:In this paper we consider the problem of maximizing a non-monotone and non-negative DR-submodular function on a bounded integer lattice [(B) over right arrow] ={(x(1), ..., x(n)) is an element of Z(+)(n): 0 <= x(k) <= B-k, for all 1 <= k <= n} without any constraint, where (B) over right arrow = (B-1, ..., B-n) is an element of Z(+)(n). We design an algorithm for the problem and measure its performance by its approximation ratio and the number of value oracle queries it needs, where the latter one is the dominating term in the running time of an algorithm. It has been showed that, for the problem considered, any algorithm achieving an approximation ratio greater than 1/2 requires an exponential number of value oracle queries. In the literature there are two algorithms that reach 1/2 approximation guarantee. The first algorithm needs O(n parallel to B parallel to(infinity)) oracle queries. The second one reduces its number of oracle queries to O(n max{1, log parallel to(B) over right arrow parallel to(infinity)}) but it needs large storage. In this paper we present a randomized approximation algorithm that has an approximation guarantee of 1/2, calls O(n max{1, log parallel to(B) over right arrow parallel to(infinity)}) oracle queries and does not need large storage, improving the results of the literature. (C) 2020 Elsevier B.V. All rights reserved.
Approximate Truthful Mechanism Design For Two-Dimensional Orthogonal Knapsack Problem

Deshi Ye,Guochuan Zhang

DOI: https://doi.org/10.1007/978-3-319-21398-9_31

2015-01-01

Abstract:This paper provides a technique for designing truthful mechanisms for a combinatorial optimization problem, which requires composition algorithms. We show that the composition algorithm A o B is monotone if the algorithm A and the algorithm B are both monotone. Then, we apply this technique to the two-dimensional orthogonal knapsack problem with provable approximation bounds, improving the previous results in [5].
D-optimal designs via a cocktail algorithm

Yaming Yu

DOI: https://doi.org/10.1007/s11222-010-9183-2

2010-05-20

Abstract:A fast new algorithm is proposed for numerical computation of (approximate) D-optimal designs. This "cocktail algorithm" extends the well-known vertex direction method (VDM; Fedorov 1972) and the multiplicative algorithm (Silvey, Titterington and Torsney, 1978), and shares their simplicity and monotonic convergence properties. Numerical examples show that the cocktail algorithm can lead to dramatically improved speed, sometimes by orders of magnitude, relative to either the multiplicative algorithm or the vertex exchange method (a variant of VDM). Key to the improved speed is a new nearest neighbor exchange strategy, which acts locally and complements the global effect of the multiplicative algorithm. Possible extensions to related problems such as nonparametric maximum likelihood estimation are mentioned.

Computation,Methodology
Multiplication Algorithms for Approximate Optimal Distributions with Cost Constraints

Lianyan Fu,Faming Ma,Zhuoxi Yu,Zhichuan Zhu

DOI: https://doi.org/10.3390/math11081963

IF: 2.4

2023-04-22

Mathematics

Abstract:In this paper, we study the D- and A-optimal assignment problems for regression models with experimental cost constraints. To solve these two problems, we propose two multiplicative algorithms for obtaining optimal designs and establishing extended D-optimal (ED-optimal) and A-optimal (EA-optimal) criteria. In addition, we give proof of the convergence of the ED-optimal algorithm and draw conjectures about some properties of the EA-optimal algorithm. Compared with the classical D- and A-optimal algorithms, the ED- and EA-optimal algorithms consider not only the accuracy of parameter estimation, but also the experimental cost constraint. The proposed methods work well in the digital example.

mathematics
Efficient computation of Bayesian optimal discriminating designs

Holger Dette,Roman Guchenko,Viatcheslav B. Melas

DOI: https://doi.org/10.48550/arXiv.1508.00279

2015-08-03

Abstract:An efficient algorithm for the determination of Bayesian optimal discriminating designs for competing regression models is developed, where the main focus is on models with general distributional assumptions beyond the "classical" case of normally distributed homoscedastic errors. For this purpose we consider a Bayesian version of the Kullback- Leibler (KL) optimality criterion introduced by López-Fidalgo et al. (2007). Discretizing the prior distribution leads to local KL-optimal discriminating design problems for a large number of competing models. All currently available methods either require a large computation time or fail to calculate the optimal discriminating design, because they can only deal efficiently with a few model comparisons. In this paper we develop a new algorithm for the determination of Bayesian optimal discriminating designs with respect to the Kullback-Leibler criterion. It is demonstrated that the new algorithm is able to calculate the optimal discriminating designs with reasonable accuracy and computational time in situations where all currently available procedures are either slow or fail.

Computation
Algorithmic Design of Majorizers for Large-Scale Inverse Problems

Madison G. McGaffin,Jeffrey A. Fessler

DOI: https://doi.org/10.48550/arXiv.1508.02958

2015-08-12

Computation

Abstract:Iterative majorize-minimize (MM) (also called optimization transfer) algorithms solve challenging numerical optimization problems by solving a series of "easier" optimization problems that are constructed to guarantee monotonic descent of the cost function. Many MM algorithms replace a computationally expensive Hessian matrix with another more computationally convenient majorizing matrix. These majorizing matrices are often generated using various matrix inequalities, and consequently the set of available majorizers is limited to structures for which these matrix inequalities can be efficiently applied. In this paper, we present a technique to algorithmically design matrix majorizers with wide varieties of structures. We use a novel duality-based approach to avoid the high computational and memory costs of standard semidefinite programming techniques. We present some preliminary results for 2D X-ray CT reconstruction that indicate these more exotic regularizers may significantly accelerate MM algorithms.
An approach for finding fully Bayesian optimal designs using normal-based approximations to loss functions

Antony M. Overstall,James M. McGree,Christopher C. Drovandi

DOI: https://doi.org/10.48550/arXiv.1608.05815

2017-02-06

Abstract:The generation of decision-theoretic Bayesian optimal designs is complicated by the significant computational challenge of minimising an analytically intractable expected loss function over a, potentially, high-dimensional design space. A new general approach for approximately finding Bayesian optimal designs is proposed which uses computationally efficient normal-based approximations to posterior summaries to aid in approximating the expected loss. This new approach is demonstrated on illustrative, yet challenging, examples including hierarchical models for blocked experiments, and experimental aims of parameter estimation and model discrimination. Where possible, the results of the proposed methodology are compared, both in terms of performance and computing time, to results from using computationally more expensive, but potentially more accurate, Monte Carlo approximations. Moreover the methodology is also applied to problems where the use of Monte Carlo approximations is computationally infeasible.

Methodology
Relaxing the Additivity Constraints in Decentralized No-Regret High-Dimensional Bayesian Optimization

Anthony Bardou,Patrick Thiran,Thomas Begin

2024-02-09

Abstract:Bayesian Optimization (BO) is typically used to optimize an unknown function $f$ that is noisy and costly to evaluate, by exploiting an acquisition function that must be maximized at each optimization step. Even if provably asymptotically optimal BO algorithms are efficient at optimizing low-dimensional functions, scaling them to high-dimensional spaces remains an open problem, often tackled by assuming an additive structure for $f$. By doing so, BO algorithms typically introduce additional restrictive assumptions on the additive structure that reduce their applicability domain. This paper contains two main contributions: (i) we relax the restrictive assumptions on the additive structure of $f$ without weakening the maximization guarantees of the acquisition function, and (ii) we address the over-exploration problem for decentralized BO algorithms. To these ends, we propose DuMBO, an asymptotically optimal decentralized BO algorithm that achieves very competitive performance against state-of-the-art BO algorithms, especially when the additive structure of $f$ comprises high-dimensional factors.

Machine Learning
Efficient construction of Bayes optimal designs for stochastic process models

Colin S. Gillespie,Richard J. Boys

DOI: https://doi.org/10.48550/arXiv.1803.04254

2018-09-17

Abstract:Stochastic process models are now commonly used to analyse complex biological, ecological and industrial systems. Increasingly there is a need to deliver accurate estimates of model parameters and assess model fit by optimizing the timing of measurement of these processes. Standard methods to construct Bayes optimal designs, such as the well known \Muller algorithm, are computationally intensive even for relatively simple models. A key issue is that, in determining the merit of a design, the utility function typically requires summaries of many parameter posterior distributions, each determined via a computer-intensive scheme such as MCMC. This paper describes a fast and computationally efficient scheme to determine optimal designs for stochastic process models. The algorithm compares favourably with other methods for determining optimal designs and can require up to an order of magnitude fewer utility function evaluations for the same accuracy in the optimal design solution. It benefits from being embarrassingly parallel and is ideal for running on multi-core computers. The method is illustrated by determining different sized optimal designs for three problems of increasing complexity.

Computation
Extending the Extension: Deterministic Algorithm for Non-monotone Submodular Maximization

Niv Buchbinder,Moran Feldman

2024-09-22

Abstract:Maximization of submodular functions under various constraints is a fundamental problem that has been studied extensively. A powerful technique that has emerged and has been shown to be extremely effective for such problems is the following. First, a continues relaxation of the problem is obtained by relaxing the (discrete) set of feasible solutions to a convex body, and extending the discrete submodular function $f$ to a continuous function $F$ known as the multilinear extension. Then, two algorithmic steps are implemented. The first step approximately solves the relaxation by finding a fractional solution within the convex body that approximately maximizes $F$; and the second step rounds this fractional solution to a feasible integral solution. While this ``fractionally solve and then round'' approach has been a key technique for resolving many questions in the field, the main drawback of algorithms based on it is that evaluating the multilinear extension may require a number of value oracle queries to $f$ that is exponential in the size of $f$'s ground set. The only known way to tackle this issue is to approximate the value of $F$ via sampling, which makes all algorithms based on this approach inherently randomized and quite slow. In this work, we introduce a new tool, that we refer to as the extended multilinear extension, designed to derandomize submodular maximization algorithms that are based on the successful ``solve fractionally and then round'' approach. We demonstrate the effectiveness of this new tool on the fundamental problem of maximizing a submodular function subject to a matroid constraint, and show that it allows for a deterministic implementation of both the fractionally solving step and the rounding step of the above approach. As a bonus, we also get a randomized algorithm for the problem with an improved query complexity.

Data Structures and Algorithms,Discrete Mathematics
Majorized Bayesian Persuasion and Fair Selection

Siddhartha Banerjee,Kamesh Munagala,Yiheng Shen,Kangning Wang

2024-10-16

Abstract:We address the fundamental problem of selection under uncertainty by modeling it from the perspective of Bayesian persuasion. In our model, a decision maker with imperfect information always selects the option with the highest expected value. We seek to achieve fairness among the options by revealing additional information to the decision maker and hence influencing its subsequent selection. To measure fairness, we adopt the notion of majorization, aiming at simultaneously approximately maximizing all symmetric, monotone, concave functions over the utilities of the options. As our main result, we design a novel information revelation policy that achieves a logarithmic-approximation to majorization in polynomial time. On the other hand, no policy, regardless of its running time, can achieve a constant-approximation to majorization. Our work is the first non-trivial majorization result in the Bayesian persuasion literature with multi-dimensional information sets.

Computer Science and Game Theory
Practical and Parallelizable Algorithms for Non-Monotone Submodular Maximization with Size Constraint

Yixin Chen,Alan Kuhnle

DOI: https://doi.org/10.1613/jair.1.14323

2024-02-17

Journal of Artificial Intelligence Research

Abstract:We present combinatorial and parallelizable algorithms for the maximization of a submodular function, not necessarily monotone, with respect to a size constraint. We improve the best approximation factor achieved by an algorithm that has optimal adaptivity and nearly optimal query complexity to 1/6 − ε, and even further to 0.193 − ε by increasing the adaptivity by a factor of O(log(k)). The conference version of this work mistakenly employed a subroutine that does not work for non-monotone, submodular functions. In this version, we propose a fixed and improved subroutine to add a set with high average marginal gain, ThreshSeq, which returns a solution in O(log(n)) adaptive rounds with high probability. Moreover, we provide two approximation algorithms. The first has approximation ratio 1/6 − ε, adaptivity O(log(n)), and query complexity O(n log(k)), while the second has approximation ratio 0.193 − ε, adaptivity O(log(n) log(k)), and query complexity O(n log(k)). Our algorithms are empirically validated to use a low number of adaptive rounds and total queries while obtaining solutions with high objective value in comparison with state-of-the-art approximation algorithms, including continuous algorithms that use the multilinear extension.

computer science, artificial intelligence
A Maximin $Φ_{p}$-Efficient Design for Multivariate GLM

Yiou Li,Lulu Kang,Xinwei Deng

DOI: https://doi.org/10.5705/ss.202020.0278

2020-08-15

Abstract:Experimental designs for a generalized linear model (GLM) often depend on the specification of the model, including the link function, the predictors, and unknown parameters, such as the regression coefficients. To deal with uncertainties of these model specifications, it is important to construct optimal designs with high efficiency under such uncertainties. Existing methods such as Bayesian experimental designs often use prior distributions of model specifications to incorporate model uncertainties into the design criterion. Alternatively, one can obtain the design by optimizing the worst-case design efficiency with respect to uncertainties of model specifications. In this work, we propose a new Maximin $\Phi_p$-Efficient (or Mm-$\Phi_p$ for short) design which aims at maximizing the minimum $\Phi_p$-efficiency under model uncertainties. Based on the theoretical properties of the proposed criterion, we develop an efficient algorithm with sound convergence properties to construct the Mm-$\Phi_p$ design. The performance of the proposed Mm-$\Phi_p$ design is assessed through several numerical examples.

Methodology
D-optimal Designs for Multinomial Logistic Models

Xianwei Bu,Dibyen Majumdar,Jie Yang

DOI: https://doi.org/10.48550/arXiv.1707.03063

2019-02-16

Abstract:We consider optimal designs for general multinomial logistic models, which cover baseline-category, cumulative, adjacent-categories, and continuation-ratio logit models, with proportional odds, non-proportional odds, or partial proportional odds assumption. We derive the corresponding Fisher information matrices in three different forms to facilitate their calculations, determine the conditions for their positive definiteness, and search for optimal designs. We conclude that, unlike the designs for binary responses, a feasible design for a multinomial logistic model may contain less experimental settings than parameters, which is of practical significance. We also conclude that even for a minimally supported design, a uniform allocation, which is typically used in practice, is not optimal in general for a multinomial logistic model. We develop efficient algorithms for searching D-optimal designs. Using examples based on real experiments, we show that the efficiency of an experiment can be significantly improved if our designs are adopted.

Statistics Theory
Optimal Bayesian design for model discrimination via classification

Markus Hainy,David J. Price,Olivier Restif,Christopher Drovandi

DOI: https://doi.org/10.1007/s11222-022-10078-2

2022-04-06

Abstract:Performing optimal Bayesian design for discriminating between competing models is computationally intensive as it involves estimating posterior model probabilities for thousands of simulated datasets. This issue is compounded further when the likelihood functions for the rival models are computationally expensive. A new approach using supervised classification methods is developed to perform Bayesian optimal model discrimination design. This approach requires considerably fewer simulations from the candidate models than previous approaches using approximate Bayesian computation. Further, it is easy to assess the performance of the optimal design through the misclassification error rate. The approach is particularly useful in the presence of models with intractable likelihoods but can also provide computational advantages when the likelihoods are manageable.

Computation
A Bayesian Discrete Optimization Algorithm for Permutation Based Combinatorial Problems

Jianming Zhang,Xifan Yao,Min Liu,Yan Wang

DOI: https://doi.org/10.1109/ssci44817.2019.9002675

2019-01-01

Abstract:Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.
Efficient Computational Algorithm for Optimal Continuous Experimental Designs

Jiangtao Duan,Wei Gao,Hon Keung Tony Ng

DOI: https://doi.org/10.48550/arXiv.1804.02655

2018-04-08

Abstract:A simple yet efficient computational algorithm for computing the continuous optimal experimental design for linear models is proposed. An alternative proof the monotonic convergence for $D$-optimal criterion on continuous design spaces are provided. We further show that the proposed algorithm converges to the $D$-optimal design. We also provide an algorithm for the $A$-optimality and conjecture that the algorithm convergence monotonically on continuous design spaces. Different numerical examples are used to demonstrated the usefulness and performance of the proposed algorithms.

Computation
Discretely Beyond $1/e$: Guided Combinatorial Algorithms for Submodular Maximization

Yixin Chen,Ankur Nath,Chunli Peng,Alan Kuhnle

2024-05-23

Abstract:For constrained, not necessarily monotone submodular maximization, all known approximation algorithms with ratio greater than $1/e$ require continuous ideas, such as queries to the multilinear extension of a submodular function and its gradient, which are typically expensive to simulate with the original set function. For combinatorial algorithms, the best known approximation ratios for both size and matroid constraint are obtained by a simple randomized greedy algorithm of Buchbinder et al. [9]: $1/e \approx 0.367$ for size constraint and $0.281$ for the matroid constraint in $\mathcal O (kn)$ queries, where $k$ is the rank of the matroid. In this work, we develop the first combinatorial algorithms to break the $1/e$ barrier: we obtain approximation ratio of $0.385$ in $\mathcal O (kn)$ queries to the submodular set function for size constraint, and $0.305$ for a general matroid constraint. These are achieved by guiding the randomized greedy algorithm with a fast local search algorithm. Further, we develop deterministic versions of these algorithms, maintaining the same ratio and asymptotic time complexity. Finally, we develop a deterministic, nearly linear time algorithm with ratio $0.377$.

Data Structures and Algorithms,Discrete Mathematics,Machine Learning
Simple Bayesian Algorithms for Best Arm Identification

Daniel Russo

DOI: https://doi.org/10.48550/arXiv.1602.08448

2018-06-08

Abstract:This paper considers the optimal adaptive allocation of measurement effort for identifying the best among a finite set of options or designs. An experimenter sequentially chooses designs to measure and observes noisy signals of their quality with the goal of confidently identifying the best design after a small number of measurements. This paper proposes three simple and intuitive Bayesian algorithms for adaptively allocating measurement effort, and formalizes a sense in which these seemingly naive rules are the best possible. One proposal is top-two probability sampling, which computes the two designs with the highest posterior probability of being optimal, and then randomizes to select among these two. One is a variant of top-two sampling which considers not only the probability a design is optimal, but the expected amount by which its quality exceeds that of other designs. The final algorithm is a modified version of Thompson sampling that is tailored for identifying the best design. We prove that these simple algorithms satisfy a sharp optimality property. In a frequentist setting where the true quality of the designs is fixed, one hopes the posterior definitively identifies the optimal design, in the sense that that the posterior probability assigned to the event that some other design is optimal converges to zero as measurements are collected. We show that under the proposed algorithms this convergence occurs at an exponential rate, and the corresponding exponent is the best possible among all allocation

Machine Learning

On a Multiplicative Algorithm for Computing Bayesian D-optimal Designs

A Fast Algorithm For Maximizing A Non-Monotone Dr-Submodular Integer Lattice Function

Approximate Truthful Mechanism Design For Two-Dimensional Orthogonal Knapsack Problem

D-optimal designs via a cocktail algorithm

Multiplication Algorithms for Approximate Optimal Distributions with Cost Constraints

Efficient computation of Bayesian optimal discriminating designs

Algorithmic Design of Majorizers for Large-Scale Inverse Problems

An approach for finding fully Bayesian optimal designs using normal-based approximations to loss functions

Relaxing the Additivity Constraints in Decentralized No-Regret High-Dimensional Bayesian Optimization

Efficient construction of Bayes optimal designs for stochastic process models

Extending the Extension: Deterministic Algorithm for Non-monotone Submodular Maximization

Majorized Bayesian Persuasion and Fair Selection

Practical and Parallelizable Algorithms for Non-Monotone Submodular Maximization with Size Constraint

A Maximin $Φ_{p}$-Efficient Design for Multivariate GLM

D-optimal Designs for Multinomial Logistic Models

Optimal Bayesian design for model discrimination via classification

A Bayesian Discrete Optimization Algorithm for Permutation Based Combinatorial Problems

Efficient Computational Algorithm for Optimal Continuous Experimental Designs

Discretely Beyond $1/e$: Guided Combinatorial Algorithms for Submodular Maximization

Simple Bayesian Algorithms for Best Arm Identification