Longitudinal evaluation of patients with cancer in the oral tongue, tonsils, or base of tongue--does interstitial radiation dose affect quality of life?

K. Petruson,C. Mercke,L. Lundberg,E. Silander,E. Hammerlid

DOI: https://doi.org/10.1016/J.BRACHY.2005.06.001

IF: 2.441

Brachytherapy

Abstract:

What problem does this paper attempt to address?

Frequentist Model Averaging under Inequality Constraints

Rong Zhu,Xinyu Zhang,Alan T. K. Wand,Guohua Zou

DOI: https://doi.org/10.1016/j.jspi.2022.03.002

IF: 1.095

2022-01-01

Journal of Statistical Planning and Inference

Abstract:Motivated by the inability of the Akaike information criterion to distinguish between statistical models with inequality constraints on parameters, a series of papers have considered alternative model selection criteria for models under inequality constraints. Among the criteria proposed, the order-restricted information criterion and its generalised version have received the most attention. This paper considers the alternative approach of model averaging, where estimates are obtained based not on one single linear model, but a weighted ensemble of linear models each subject to a distinct set of linear inequality constraints. We develop a J-fold cross validation criterion for choosing model weights that reflect the support of the models by the data, and prove that the resultant model average estimator has an optimal asymptotic property. We further prove that for the special case of a finite number of regressors, the sum of the weights obtained by the proposed method for models with the correct inequality constraints converges to one in probability. In our limited simulation study and real data analysis, it is observed that the proposed model average estimator frequently outperforms the estimator that arises from the generalised order-restricted information-based model selection criterion in terms of average MSE and median MSE.
Improved information criteria for Bayesian model averaging in lattice field theory

Ethan T. Neil,Jacob W. Sitison

DOI: https://doi.org/10.1103/PhysRevD.109.014510

2024-02-03

Abstract:Bayesian model averaging is a practical method for dealing with uncertainty due to model specification. Use of this technique requires the estimation of model probability weights. In this work, we revisit the derivation of estimators for these model weights. Use of the Kullback-Leibler divergence as a starting point leads naturally to a number of alternative information criteria suitable for Bayesian model weight estimation. We explore three such criteria, known to the statistics literature before, in detail: a Bayesian analogue of the Akaike information criterion which we call the BAIC, the Bayesian predictive information criterion (BPIC), and the posterior predictive information criterion (PPIC). We compare the use of these information criteria in numerical analysis problems common in lattice field theory calculations. We find that the PPIC has the most appealing theoretical properties and can give the best performance in terms of model-averaging uncertainty, particularly in the presence of noisy data, while the BAIC is a simple and reliable alternative.

Methodology,High Energy Physics - Lattice
The Focused Information Criterion for Stochastic Model Selection Problems Using $M$-Estimators

S.C.Pandhare,T.V.Ramanathan

DOI: https://doi.org/10.48550/arXiv.1807.08386

2018-07-23

Abstract:Claeskens and Hjort (2003) constructed the focused information criterion (FIC) and developed frequentist model averaging methods using maximum likelihood estimators assuming the observations to be independent and identically distributed. Towards the immediate extensions and generalizations of these results, the present article is aimed at providing the focused model selection and model averaging methods using general maximum likelihood type estimators, popularly known as $M$-estimators. The necessary asymptotic theory is derived in a setup of stationary and strong mixing stochastic processes employing von Mises functional calculus of empirical processes and Le Cam's contiguity lemmas. We illustrate the proposed focused stochastic modeling methods using three well-known spacial cases of $M$-estimators, namely, conditional maximum likelihood estimators, conditional least square estimators and estimators based on method of moments. For the sake of simulation exercises, we consider two simple applications of FIC. The first application discusses the simultaneous selection of order of autoregression and symmetry of innovations in asymmetric Laplace autoregressive models. The second application demonstrates the FIC based choice between general scale-shape Gamma density and exponential density with shape being unity. We observe that in terms of the correct selections, FIC outperforms classical Akaike's information criterion AIC and performs at par with Bayesian information criterion BIC.

Statistics Theory
Geostatistical model averaging based on conditional information criteria

Chun-Shu Chen,Hsin-Cheng Huang

DOI: https://doi.org/10.1007/s10651-011-0171-2

2011-04-11

Environmental and Ecological Statistics

Abstract:Variable selection in geostatistical regression is an important problem, but has not been well studied in the literature. In this paper, we focus on spatial prediction and consider a class of conditional information criteria indexed by a penalty parameter. Instead of applying a fixed criterion, which leads to an unstable predictor in the sense that it is discontinuous with respect to the response variables due to that a small change in the response may cause a different model to be selected, we further stabilize the predictor by local model averaging, resulting in a predictor that is not only continuous but also differentiable even after plugging-in estimated model parameters. Then Stein’s unbiased risk estimate is applied to select the penalty parameter, leading to a data-dependent penalty that is adaptive to the underlying model. Some numerical experiments show superiority of the proposed model averaging method over some commonly used variable selection methods. In addition, the proposed method is applied to a mercury data set for lakes in Maine.

environmental sciences,statistics & probability,mathematics, interdisciplinary applications
A Bayesian Information Criterion for Portfolio Selection

Wei Lan,Hansheng Wang,Chih-Ling Tsai

DOI: https://doi.org/10.1016/j.csda.2011.06.012

IF: 2.035

2012-01-01

Computational Statistics & Data Analysis

Abstract:The mean-variance theory of Markowitz (1952) indicates that large investment portfolios naturally provide better risk diversification than small ones. However, due to parameter estimation errors, one may find ambiguous results in practice. Hence, it is essential to identify relevant stocks to alleviate the impact of estimation error in portfolio selection. To this end, we propose a linkage condition to link the relevant and irrelevant stock returns via their conditional regression relationship. Subsequently, we obtain a BIC selection criterion that enables us to identify relevant stocks consistently. Numerical studies indicate that BIC outperforms commonly used portfolio strategies in the literature.
Kullback-Leibler Divergence and Akaike Information Criterion in General Hidden Markov Models

Cheng-Der Fuh,Chu-Lan Michael Kao,Tianxiao Pang

2023-03-14

Abstract:To characterize the Kullback-Leibler divergence and Fisher information in general parametrized hidden Markov models, in this paper, we first show that the log likelihood and its derivatives can be represented as an additive functional of a Markovian iterated function system, and then provide explicit characterizations of these two quantities through this representation. Moreover, we show that Kullback-Leibler divergence can be locally approximated by a quadratic function determined by the Fisher information. Results relating to the Cramér-Rao lower bound and the Hájek-Le Cam local asymptotic minimax theorem are also given. As an application of our results, we provide a theoretical justification of using Akaike information criterion (AIC) model selection in general hidden Markov models. Last, we study three concrete models: a Gaussian vector autoregressive-moving average model of order $(p,q)$, recurrent neural networks, and temporal restricted Boltzmann machine, to illustrate our theory.

Statistics Theory
Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

Scott I. Vrieze

DOI: https://doi.org/10.1037/a0027127

IF: 10.929

2012-01-01

Psychological Methods

Abstract:This article reviews the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) in model selection and the appraisal of psychological theory. The focus is on latent variable models, given their growing use in theory testing and construction. Theoretical statistical results in regression are discussed, and more important issues are illustrated with novel simulations involving latent variable models including factor analysis, latent profile analysis, and factor mixture models. Asymptotically, the BIC is consistent, in that it will select the true model if, among other assumptions, the true model is among the candidate models considered. The AIC is not consistent under these circumstances. When the true model is not in the candidate model set the AIC is efficient, in that it will asymptotically choose whichever model minimizes the mean squared error of prediction/estimation. The BIC is not efficient under these circumstances. Unlike the BIC, the AIC also has a minimax property, in that it can minimize the maximum possible risk in finite sample sizes. In sum, the AIC and BIC have quite different properties that require different assumptions, and applied researchers and methodologists alike will benefit from improved understanding of the asymptotic and finite-sample behavior of these criteria. The ultimate decision to use the AIC or BIC depends on many factors, including the loss function employed, the study's methodological design, the substantive research question, and the notion of a true model and its applicability to the study at hand.

psychology, multidisciplinary
Consistent Model Selection Procedure for Random Coefficient INAR Models

Kaizhi Yu,Tielai Tao

DOI: https://doi.org/10.3390/e25081220

2023-08-16

Abstract:In the realm of time series data analysis, information criteria constructed on the basis of likelihood functions serve as crucial instruments for determining the appropriate lag order. However, the intricate structure of random coefficient integer-valued time series models, which are founded on thinning operators, complicates the establishment of likelihood functions. Consequently, employing information criteria such as AIC and BIC for model selection becomes problematic. This study introduces an innovative methodology that formulates a penalized criterion by utilizing the estimation equation within conditional least squares estimation, effectively addressing the aforementioned challenge. Initially, the asymptotic properties of the penalized criterion are derived, followed by a numerical simulation study and a comparative analysis. The findings from both theoretical examinations and simulation investigations reveal that this novel approach consistently selects variables under relatively relaxed conditions. Lastly, the applications of this method to infectious disease data and seismic frequency data produce satisfactory outcomes.
A General Framework For Frequentist Model Averaging

Priyam Mitra,Heng Lian,Ritwik Mitra,Hua Liang,Min-ge Xie

DOI: https://doi.org/10.48550/arXiv.1802.03511

2018-02-10

Abstract:Model selection strategies have been routinely employed to determine a model for data analysis in statistics, and further study and inference then often proceed as though the selected model were the true model that were known a priori. This practice does not account for the uncertainty introduced by the selection process and the fact that the selected model can possibly be a wrong one. Model averaging approaches try to remedy this issue by combining estimators for a set of candidate models. Specifically, instead of deciding which model is the 'right' one, a model averaging approach suggests to fit a set of candidate models and average over the estimators using certain data adaptive weights. In this paper we establish a general frequentist model averaging framework that does not set any restrictions on the set of candidate models. It greatly broadens the scope of the existing methodologies under the frequentist model averaging development. Assuming the data is from an unknown model, we derive the model averaging estimator and study its limiting distributions and related predictions while taking possible modeling biases into account. We propose a set of optimal weights to combine the individual estimators so that the expected mean squared error of the average estimator is minimized. Simulation studies are conducted to compare the performance of the estimator with that of the existing methods. The results show the benefits of the proposed approach over traditional model selection approaches as well as existing model averaging methods.

Methodology
Parsimonious Model Averaging With a Diverging Number of Parameters

Xinyu Zhang,Guohua Zou,Hua Liang,Raymond J. Carroll

DOI: https://doi.org/10.1080/01621459.2019.1604363

IF: 4.369

2019-06-19

Journal of the American Statistical Association

Abstract:Model averaging generally provides better predictions than model selection, but the existing model averaging methods cannot lead to parsimonious models. Parsimony is an especially important property when the number of parameters is large. To achieve a parsimonious model averaging coefficient estimator, we suggest a novel criterion for choosing weights. Asymptotic properties are derived in two practical scenarios: (i) one or more correct models exist in the candidate model set and (ii) all candidate models are misspecified. Under the former scenario, it is proved that our method can put the weight one to the smallest correct model and the resulting model averaging estimators of coefficients have many zeros and thus lead to a parsimonious model. The asymptotic distribution of the estimators is also provided. Under the latter scenario, prediction is mainly focused on and we prove that the proposed procedure is asymptotically optimal in the sense that its squared prediction loss and risk are asymptotically identical to those of the best—but infeasible—model averaging estimator. Numerical analysis shows the promise of the proposed procedure over existing model averaging and selection methods.

statistics & probability
Model averaging approaches to data subset selection

Ethan T. Neil,Jacob W. Sitison

DOI: https://doi.org/10.1103/PhysRevE.108.045308

2023-10-25

Abstract:Model averaging is a useful and robust method for dealing with model uncertainty in statistical analysis. Often, it is useful to consider data subset selection at the same time, in which model selection criteria are used to compare models across different subsets of the data. Two different criteria have been proposed in the literature for how the data subsets should be weighted. We compare the two criteria closely in a unified treatment based on the Kullback-Leibler divergence, and conclude that one of them is subtly flawed and will tend to yield larger uncertainties due to loss of information. Analytical and numerical examples are provided.

Methodology,High Energy Physics - Lattice
A Scalable Frequentist Model Averaging Method

Rong Zhu,Haiying Wang,Xinyu Zhang,Hua Liang

DOI: https://doi.org/10.1080/07350015.2022.2116442

2022-01-01

Abstract:Frequentist model averaging is an effective technique to handle model uncertainty. However, calculation of the weights for averaging is extremely difficult, if not impossible, even when the dimension of the predictor vector, p, is moderate, because we may have 2(p) candidate models. The exponential size of the candidate model set makes it difficult to estimate all candidate models, and brings additional numeric errors when calculating the weights. This article proposes a scalable frequentist model averaging method, which is statistically and computationally efficient, to overcome this problem by transforming the original model using the singular value decomposition. The method enables us to find the optimal weights by considering at most p candidate models. We prove that the minimum loss of the scalable model averaging estimator is asymptotically equal to that of the traditional model averaging estimator. We apply the Mallows and Jackknife criteria to the scalable model averaging estimator and prove that they are asymptotically optimal estimators. We further extend the method to the high-dimensional case (i.e., p >= n). Numerical studies illustrate the superiority of the proposed method in terms of both statistical efficiency and computational cost.
A new class of information criteria for improved prediction in the presence of training/validation data heterogeneity

Javier E. Flores,Joseph E. Cavanaugh,Andrew A. Neath

DOI: https://doi.org/10.1007/s00180-024-01559-1

IF: 1.4049

2024-10-03

Computational Statistics

Abstract:Information criteria provide a cogent approach for identifying models that provide an optimal balance between the competing objectives of goodness-of-fit and parsimony. Models that better conform to a dataset are often more complex, yet such models are plagued by greater variability in estimation and prediction. Conversely, overly simplistic models reduce variability at the cost of increases in bias. Asymptotically efficient criteria are those that, for large samples, select the fitted candidate model whose predictors minimize the mean squared prediction error, optimizing between prediction bias and variability. In the context of prediction, asymptotically efficient criteria are thus a preferred tool for model selection, with the Akaike information criterion (AIC) being among the most widely used. However, asymptotic efficiency relies upon the assumption of a panel of validation data generated independently from, but identically to, the set of training data. We argue that assuming identically distributed training and validation data is misaligned with the premise of prediction and often violated in practice. This is most apparent in a regression context, where assuming training/validation data homogeneity requires identical panels of regressors. We therefore develop a new class of predictive information criteria (PIC) that do not assume training/validation data homogeneity and are shown to generalize AIC to the more practically relevant setting of training/validation data heterogeneity. The analytic properties and predictive performance of these new criteria are explored within the traditional regression framework. We consider both simulated and real-data settings. Software for implementing these methods is provided in the R package, picR , available through CRAN.

statistics & probability
An automatic adaptive method to combine summary statistics in approximate Bayesian computation

Jonathan U Harrison,Ruth E Baker

DOI: https://doi.org/10.48550/arXiv.1703.02341

2018-08-02

Abstract:To infer the parameters of mechanistic models with intractable likelihoods, techniques such as approximate Bayesian computation (ABC) are increasingly being adopted. One of the main disadvantages of ABC in practical situations, however, is that parameter inference must generally rely on summary statistics of the data. This is particularly the case for problems involving high-dimensional data, such as biological imaging experiments. However, some summary statistics contain more information about parameters of interest than others, and it is not always clear how to weight their contributions within the ABC framework. We address this problem by developing an automatic, adaptive algorithm that chooses weights for each summary statistic. Our algorithm aims to maximize the distance between the prior and the approximate posterior by automatically adapting the weights within the ABC distance function. Computationally, we use a nearest neighbour estimator of the distance between distributions. We justify the algorithm theoretically based on properties of the nearest neighbour distance estimator. To demonstrate the effectiveness of our algorithm, we apply it to a variety of test problems, including several stochastic models of biochemical reaction networks, and a spatial model of diffusion, and compare our results with existing algorithms.

Computation
AIC under the framework of least squares estimation

H.T. Banks,Michele L. Joyner

DOI: https://doi.org/10.1016/j.aml.2017.05.005

IF: 4.294

2017-12-01

Applied Mathematics Letters

Abstract:In this note we explain the use of the Akiake Information Criterion and its related model comparison indices (usually derived for maximum likelihood estimator inverse problem formulations) in the context of least squares (ordinary, weighted, iterative weighted or “generalized”, etc.) based inverse problem formulations. The ideas are illustrated with several examples of interest in biology.

mathematics, applied
A Conceptual Introduction to Bayesian Model Averaging

Max Hinne,Quentin F. Gronau,Don van den Bergh,Eric-Jan Wagenmakers

DOI: https://doi.org/10.1177/2515245919898657

2020-06-01

Advances in Methods and Practices in Psychological Science

Abstract:Many statistical scenarios initially involve several candidate models that describe the data-generating process. Analysis often proceeds by first selecting the best model according to some criterion and then learning about the parameters of this selected model. Crucially, however, in this approach the parameter estimates are conditioned on the selected model, and any uncertainty about the model-selection process is ignored. An alternative is to learn the parameters for all candidate models and then combine the estimates according to the posterior probabilities of the associated models. This approach is known as Bayesian model averaging (BMA). BMA has several important advantages over all-or-none selection methods, but has been used only sparingly in the social sciences. In this conceptual introduction, we explain the principles of BMA, describe its advantages over all-or-none model selection, and showcase its utility in three examples: analysis of covariance, meta-analysis, and network analysis.
Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood

Kohei Hayashi,Shin-ichi Maeda,Ryohei Fujimaki

DOI: https://doi.org/10.48550/arXiv.1504.05665

2015-04-22

Abstract:Factorized information criterion (FIC) is a recently developed approximation technique for the marginal log-likelihood, which provides an automatic model selection framework for a few latent variable models (LVMs) with tractable inference algorithms. This paper reconsiders FIC and fills theoretical gaps of previous FIC studies. First, we reveal the core idea of FIC that allows generalization for a broader class of LVMs, including continuous LVMs, in contrast to previous FICs, which are applicable only to binary LVMs. Second, we investigate the model selection mechanism of the generalized FIC. Our analysis provides a formal justification of FIC as a model selection criterion for LVMs and also a systematic procedure for pruning redundant latent variables that have been removed heuristically in previous studies. Third, we provide an interpretation of FIC as a variational free energy and uncover a few previously-unknown their relationships. A demonstrative study on Bayesian principal component analysis is provided and numerical experiments support our theoretical results.

Machine Learning
A Model-Averaging Approach for High-Dimensional Regression

Tomohiro Ando,Ker-Chau Li

DOI: https://doi.org/10.1080/01621459.2013.838168

IF: 4.369

2014-01-02

Journal of the American Statistical Association

Abstract:This article considers high-dimensional regression problems in which the number of predictors p exceeds the sample size n. We develop a model-averaging procedure for high-dimensional regression problems. Unlike most variable selection studies featuring the identification of true predictors, our focus here is on the prediction accuracy for the true conditional mean of y given the p predictors. Our method consists of two steps. The first step is to construct a class of regression models, each with a smaller number of regressors, to avoid the degeneracy of the information matrix. The second step is to find suitable model weights for averaging. To minimize the prediction error, we estimate the model weights using a delete-one cross-validation procedure. Departing from the literature of model averaging that requires the weights always sum to one, an important improvement we introduce is to remove this constraint. We derive some theoretical results to justify our procedure. A theorem is proved, showing that delete-one cross-validation achieves the lowest possible prediction loss asymptotically. This optimality result requires a condition that unravels an important feature of high-dimensional regression. The prediction error of any individual model in the class for averaging is required to be higher than the classic root n rate under the traditional parametric regression. This condition reflects the difficulty of high-dimensional regression and it depicts a situation especially meaningful for p > n. We also conduct a simulation study to illustrate the merits of the proposed approach over several existing methods, including lasso, group lasso, forward regression, Phase Coupled (PC)-simple algorithm, Akaike information criterion (AIC) model-averaging, Bayesian information criterion (BIC) model-averaging methods, and SCAD (smoothly clipped absolute deviation). This approach uses quadratic programming to overcome the computing time issue commonly encountered in the cross-validation literature. Supplementary materials for this article are available online.

statistics & probability
On the overestimation of widely applicable Bayesian information criterion

Toru Imai

DOI: https://doi.org/10.48550/arXiv.1908.10572

2019-08-28

Abstract:A widely applicable Bayesian information criterion (Watanabe, 2013) is applicable for both regular and singular models in the model selection problem. This criterion tends to overestimate the log marginal likelihood. We identify an overestimating term of a widely applicable Bayesian information criterion. Adjustment of the term gives an asymptotically unbiased estimator of the leading two terms of asymptotic expansion of the log marginal likelihood. In numerical experiments on regular and singular models, the adjustment resulted in smaller bias than the original criterion.

Methodology,Statistics Theory,Machine Learning

Longitudinal evaluation of patients with cancer in the oral tongue, tonsils, or base of tongue--does interstitial radiation dose affect quality of life?

Frequentist Model Averaging under Inequality Constraints

Improved information criteria for Bayesian model averaging in lattice field theory

The Focused Information Criterion for Stochastic Model Selection Problems Using $M$-Estimators

Geostatistical model averaging based on conditional information criteria

A Bayesian Information Criterion for Portfolio Selection

Kullback-Leibler Divergence and Akaike Information Criterion in General Hidden Markov Models

Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

Consistent Model Selection Procedure for Random Coefficient INAR Models

A General Framework For Frequentist Model Averaging

Parsimonious Model Averaging With a Diverging Number of Parameters

Model averaging approaches to data subset selection

A Scalable Frequentist Model Averaging Method

A new class of information criteria for improved prediction in the presence of training/validation data heterogeneity

An automatic adaptive method to combine summary statistics in approximate Bayesian computation

AIC under the framework of least squares estimation

A Conceptual Introduction to Bayesian Model Averaging

Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood

A Model-Averaging Approach for High-Dimensional Regression

On the overestimation of widely applicable Bayesian information criterion