Supplement to 'Comparison of Misspecified Calibrated Models'

Viktoria V. Hnatkovska,Vadim Marmer,Yao Tang

DOI: https://doi.org/10.2139/ssrn.1514081

2011-01-01

SSRN Electronic Journal

Abstract:This paper contains supplemental material to Hnatkovska, Marmer, and Tang (2009) "Comparison of Misspecified Calibrated Models: The Minimum Distance Approach".

What problem does this paper attempt to address?

Comparison of Misspecified Calibrated Models: the Minimum Distance Approach

Viktoria Hnatkovska,Vadim Marmer,Yao Tang

DOI: https://doi.org/10.1016/j.jeconom.2012.01.007

IF: 3.363

2012-01-01

Journal of Econometrics

Abstract:This paper proposes several testing procedures for comparison of misspecified calibrated models. The proposed tests are of the Vuong-type (Vuong, 1989, Rivers and Vuong, 2002). In our framework, the econometrician selects values for model’s parameters in order to match some characteristics of data with those implied by the theoretical model. We assume that all competing models are misspecified, and suggest a test for the null hypothesis that they provide equivalent fit to data characteristics, against the alternative that one of the models is a better approximation. We consider both nested and non-nested cases. We also relax the dependence of models’ ranking on the choice of a weight matrix by suggesting averaged and sup-norm procedures. The methods are illustrated by comparing the cash-in-advance and portfolio adjustment cost models in their ability to match the impulse responses of output and inflation to money growth shocks.
Broadening the Scope of Food Quality Control: a Calibration Transfer Strategy for Minimizing External Influences on Spectral Analysis

Penghui Liu,Yihang Qian,Changqing An,Yang Yao,Yingjie Zheng,Huirong Xu,Lijuan Xie

DOI: https://doi.org/10.1016/j.foodcont.2024.110899

IF: 6

2025-01-01

Food Control

Abstract:Calibration transfer (CT) is an effective tool to address spectral inconsistencies and the decline in model prediction capabilities caused by various influences. This study introduced a modified semi-supervised parameterfree calibration enhancement (MSS-PFCE) strategy to enable calibration models extensively applicable across varying temperatures, instruments and orientations. Investigations conducted on three datasets of watermelon juice, corn, and apples were utilized to validate the efficacy of MSS-PFCE. The results showed that MSS-PFCE decreased the average RMSEP for slave spectrum predictions by 96.51%, 75.96%, and 23.04%, respectively. MSS-PFCE significantly outperformed the original SS-PFCE approach and three classical standardization methods (direct standardization (DS), piecewise direct standardization (PDS) and slope and bias correction (SBC)). Furthermore, MSS-PFCE required no standards and needed only 5%-10% of slave samples for CT, maintaining robust reliability across varying sample sizes and cost thresholds (cth). The conclusions were further examined in three additional datasets with these variations. Notably, MSS-PFCE exhibited high accuracy, fitting stability, low sample dependency, and versatile transferability, positioning it as a promising candidate for widespread implementation in visible/near-infrared (Vis/NIR) spectroscopy. This study has significant potential to enhance the applicability of quality analysis, broaden the scope of food quality control, and promote the sound development of the food industry.
A Comparison Study of Mathematical Models for Tolerance Analysis

Jian Xin Yang,Zhen Tao Liu,Ben Zhao

DOI: https://doi.org/10.4028/www.scientific.net/amr.765-767.759

2013-01-01

Advanced Materials Research

Abstract:This paper reviews two major models (Small Displacement Torsor, Deviation and Clearance Domain) for 3D functional tolerance analysis and compares them. The underlying mathematical representation of geometric tolerances can be classified as inequalities and multi-variate region. The corresponding algebraic or geometric tolerance propagation mechanism of each model is briefly introduced for worst-case and statistical tolerancing. Through a comprehensive comparison of these models, this paper gives some suggestions for choosing the appropriate method for a given tolerancing problem.
A Review for Model Plant Mismatch Measures in Process Monitoring

Hong WANG,Lei XIE,Zhihuan SONG

DOI: https://doi.org/10.1016/S1004-9541(12)60584-9

IF: 3.8

2012-01-01

Chinese Journal of Chemical Engineering

Abstract:Model is usually necessary for the design of a control loop. Due to simplification and unknown dynamics, model plant mismatch is inevitable in the control loop. In process monitoring, detection of mismatch and evaluation of its influences are demanded. In this paper several mismatch measures are presented based on different model descriptions. They are categorized into different groups from different perspectives and their potential in detection and diagnosis is evaluated. Two case studies on mixing process and distillation process demonstrate the efficacy of the framework of mismatch monitoring.
The Distance Between: An Algorithmic Approach to Comparing Stochastic Models to Time-Series Data

Brock D Sherlock,Marko A A Boon,Maria Vlasiou,Adelle C F Coster

DOI: https://doi.org/10.1007/s11538-024-01331-y

2024-07-26

Abstract:While mean-field models of cellular operations have identified dominant processes at the macroscopic scale, stochastic models may provide further insight into mechanisms at the molecular scale. In order to identify plausible stochastic models, quantitative comparisons between the models and the experimental data are required. The data for these systems have small sample sizes and time-evolving distributions. The aim of this study is to identify appropriate distance metrics for the quantitative comparison of stochastic model outputs and time-evolving stochastic measurements of a system. We identify distance metrics with features suitable for driving parameter inference, model comparison, and model validation, constrained by data from multiple experimental protocols. In this study, stochastic model outputs are compared to synthetic data across three scales: that of the data at the points the system is sampled during the time course of each type of experiment; a combined distance across the time course of each experiment; and a combined distance across all the experiments. Two broad categories of comparators at each point were considered, based on the empirical cumulative distribution function (ECDF) of the data and of the model outputs: discrete based measures such as the Kolmogorov-Smirnov distance, and integrated measures such as the Wasserstein-1 distance between the ECDFs. It was found that the discrete based measures were highly sensitive to parameter changes near the synthetic data parameters, but were largely insensitive otherwise, whereas the integrated distances had smoother transitions as the parameters approached the true values. The integrated measures were also found to be robust to noise added to the synthetic data, replicating experimental error. The characteristics of the identified distances provides the basis for the design of an algorithm suitable for fitting stochastic models to real world stochastic data.
On Multivariate Calibration Problems

Guo-zheng Li,Jie Yang,Jun Lu,Wen-cong Lu,Nian-yi Chen

DOI: https://doi.org/10.1007/978-3-540-28647-9_65

2004-01-01

Abstract:Multivariate calibration is a classic problem in the analytical chemistry field and frequently solved by partial least squares method in the previous work. Unfortunately there are so many redundant features in the problem, that feature selection are often performed before modeling by partial least squares method and the features not selected are usually discarded. In this paper, the redundant information is, however, reused in the learning of partial least squares method within the frame of multitask learning. Results on three multivariate calibration data sets show that multitask learning can greatly improve the accuracy of partial least squares method.
Calibrated Model Criticism Using Split Predictive Checks

Jiawei Li,Jonathan H. Huggins

2024-05-23

Abstract:Checking how well a fitted model explains the data is one of the most fundamental parts of a Bayesian data analysis. However, existing model checking methods suffer from trade-offs between being well-calibrated, automated, and computationally efficient. To overcome these limitations, we propose split predictive checks (SPCs), which combine the ease-of-use and speed of posterior predictive checks with the good calibration properties of predictive checks that rely on model-specific derivations or inference schemes. We develop an asymptotic theory for two types of SPCs: single SPCs and the divided SPCs. Our results demonstrate that they offer complementary strengths. Single SPCs work well with smaller datasets and provide excellent power when there is substantial misspecification, such as when the estimate uncertainty in the test statistic is significantly underestimated. When the sample size is large, divided SPCs can provide better power and are able to detect more subtle form of misspecification. We validate the finite-sample utility of SPCs through extensive simulation experiments in exponential family and hierarchical models, and provide three real-data examples where SPCs offer novel insights and additional flexibility beyond what is available when using posterior predictive checks.

Methodology,Statistics Theory
Towards Automatic Model Comparison: An Adaptive Sequential Monte Carlo Approach

Yan Zhou,Adam M Johansen,John A D Aston

DOI: https://doi.org/10.48550/arXiv.1303.3123

2015-06-05

Abstract:Model comparison for the purposes of selection, averaging and validation is a problem found throughout statistics. Within the Bayesian paradigm, these problems all require the calculation of the posterior probabilities of models within a particular class. Substantial progress has been made in recent years, but difficulties remain in the implementation of existing schemes. This paper presents adaptive sequential Monte Carlo (\smc) sampling strategies to characterise the posterior distribution of a collection of models, as well as the parameters of those models. Both a simple product estimator and a combination of \smc and a path sampling estimator are considered and existing theoretical results are extended to include the path sampling variant. A novel approach to the automatic specification of distributions within \smc algorithms is presented and shown to outperform the state of the art in this area. The performance of the proposed strategies is demonstrated via an extensive empirical study. Comparisons with state of the art algorithms show that the proposed algorithms are always competitive, and often substantially superior to alternative techniques, at equal computational cost and considerably less application-specific implementation effort.

Methodology
Sensitivity and Uncertainty Analysis of Nonideal Calibration Standards for 16-Term Error Model

Jiefeng Zhou,Ling Zhang,Junhui Chen,Da Li,Er-Ping Li

DOI: https://doi.org/10.1109/tmtt.2023.3263596

IF: 4.3

2023-01-01

IEEE Transactions on Microwave Theory and Techniques

Abstract:This article presents an uncertainty analysis method for the 16-term error model, and applies the methodology in determining the best calibration standard combinations and their associated parasitic parameters. This article proposes a sensitivity analysis method using the complex-valued matrix differentiation to tackle the complexity of the 16-term error network, and the uncertainty propagation is further derived by the properties of the covariance matrix. A novel methodology is additionally proposed for calibration standard selection by combining the uncertainty analysis method with a parasitic-parameter extraction approach. The newly proposed methodology jointly determines the best combinations of calibration standards and their corresponding parasitic parameters and enhances the calibration accuracy and reliability of the 16-term error model. The effectiveness of the technique is validated by on-wafer measurements.
ModelDiff: A Framework for Comparing Learning Algorithms

Harshay Shah,Sung Min Park,Andrew Ilyas,Aleksander Madry

DOI: https://doi.org/10.48550/arXiv.2211.12491

2022-11-23

Abstract:We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at <a class="link-external link-https" href="https://github.com/MadryLab/modeldiff" rel="external noopener nofollow">this https URL</a> .

Machine Learning,Computer Vision and Pattern Recognition
Hyper-differential sensitivity analysis with respect to model discrepancy: Optimal solution updating

Joseph Hart,Bart van Bloemen Waanders

DOI: https://doi.org/10.1016/j.cma.2023.116082

IF: 6.588

2023-05-22

Computer Methods in Applied Mechanics and Engineering

Abstract:A common goal throughout science and engineering is to solve optimization problems constrained by computational models. However, in many cases a high-fidelity numerical emulation of systems cannot be optimized due to code complexity and computational costs which prohibit the use of intrusive and many query algorithms. Rather, lower-fidelity models are constructed to enable intrusive algorithms for large-scale optimization. As a result of the discrepancy between high and low-fidelity models, optimal solutions determined using low-fidelity models are frequently far from true optimality. In this article we introduce a novel approach that uses post-optimality sensitivities with respect to model discrepancy to update the optimization solution. Limited high-fidelity data is used to calibrate the model discrepancy in a Bayesian framework which in turn is propagated through post-optimality sensitivities of the low-fidelity optimization problem. Our formulation exploits structure in the post-optimality sensitivity operator to achieve computational scalability. Numerical results demonstrate how an optimal solution computed using a low-fidelity model may be significantly improved with limited evaluations of a high-fidelity model.

mechanics,engineering, multidisciplinary,mathematics, interdisciplinary applications
Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

Dongfang Li,Baotian Hu,Qingcai Chen

DOI: https://doi.org/10.48550/arxiv.2211.03041

2022-01-01

Abstract:Calibration strengthens the trustworthiness of black-box models by producing better accurate confidence estimates on given examples. However, little is known about if model explanations can help confidence calibration. Intuitively, humans look at important features attributions and decide whether the model is trustworthy. Similarly, the explanations can tell us when the model may or may not know. Inspired by this, we propose a method named CME that leverages model explanations to make the model less confident with non-inductive attributions. The idea is that when the model is not highly confident, it is difficult to identify strong indications of any class, and the tokens accordingly do not have high attribution scores for any class and vice versa. We conduct extensive experiments on six datasets with two popular pre-trained language models in the in-domain and out-of-domain settings. The results show that CME improves calibration performance in all settings. The expected calibration errors are further reduced when combined with temperature scaling. Our findings highlight that model explanations can help calibrate posterior estimates.
A Comparative Study of Representative Subset Selection for NIR Model Updating

Xiaping Fu,Yibin Ying,Danting Yang

DOI: https://doi.org/10.13031/2013.37411

2011-01-01

Abstract:Because of the uncertain changes (instrumental and non-instrumental), near infrared (NIR) calibration models for fruit quality determination tend to become invalid because of non-modeled variations in the new data. In order to avoid repetition of the whole calibration procedure and solve the model updating problem in NIR spectroscopy, several algorithms related to subset partitioning (Duplex algorithm and Rank-selection method) and calibration updating (Kennard-Stone (KS) algorithm and random selection) were tested with the objective of comparing them in terms of their performance. The issue of pear soluble solid content (SSC) model robustness over time (within different years), and the use of model updating procedure using small number of samples from the new population were examined in this study. The results showed that models established using subsets partitioned by Duplex algorithm performed better than those using Rank-selection method. KS algorithm is more effective for update sample selection. With the addition of approximately 15 samples, models can be successfully updated to improve prediction of new samples.
Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding

Lingdong Kong,Xiang Xu,Jun Cen,Wenwei Zhang,Liang Pan,Kai Chen,Ziwei Liu

2024-03-26

Abstract:Safety-critical 3D scene understanding tasks necessitate not only accurate but also confident predictions from 3D perception models. This study introduces Calib3D, a pioneering effort to benchmark and scrutinize the reliability of 3D scene understanding models from an uncertainty estimation viewpoint. We comprehensively evaluate 28 state-of-the-art models across 10 diverse 3D datasets, uncovering insightful phenomena that cope with both the aleatoric and epistemic uncertainties in 3D scene understanding. We discover that despite achieving impressive levels of accuracy, existing models frequently fail to provide reliable uncertainty estimates -- a pitfall that critically undermines their applicability in safety-sensitive contexts. Through extensive analysis of key factors such as network capacity, LiDAR representations, rasterization resolutions, and 3D data augmentation techniques, we correlate these aspects directly with the model calibration efficacy. Furthermore, we introduce DeptS, a novel depth-aware scaling approach aimed at enhancing 3D model calibration. Extensive experiments across a wide range of configurations validate the superiority of our method. We hope this work could serve as a cornerstone for fostering reliable 3D scene understanding. Code and benchmark toolkits are publicly available.

Computer Vision and Pattern Recognition,Machine Learning,Robotics
Additive Distortion Measurement Errors Regression Models with Exponential Calibration

Xuehu Zhu,Jun Zhang,Yiping Yang

DOI: https://doi.org/10.1080/00949655.2022.2055028

IF: 1.225

2022-01-01

Journal of Statistical Computation and Simulation

Abstract:In this paper, we used the newly proposed exponential calibration for the additive distortion measurement errors models, where neither the response variable nor the covariates can be directly observed but are distorted in additive fashions by an observed confounding variable. By using the exponential calibrated variables, three estimators of parameters and empirical likelihood-based confidence intervals are proposed, and we studied the asymptotic properties of the proposed estimators. For the hypothesis testing of model checking, an adaptive Neyman test statistic restricted is proposed. Simulation studies demonstrate the performance of the proposed estimators and the test statistic. A real example is analysed to illustrate its practical usage.
Calibration of imperfect geophysical models by multiple satellite interferograms with measurement bias

Mengyang Gu,Kyle Anderson,Erika McPhillips

DOI: https://doi.org/10.1080/00401706.2023.2182365

2023-02-26

Abstract:Model calibration consists of using experimental or field data to estimate the unknown parameters of a mathematical model. The presence of model discrepancy and measurement bias in the data complicates this task. Satellite interferograms, for instance, are widely used for calibrating geophysical models in geological hazard quantification. In this work, we used satellite interferograms to relate ground deformation observations to the properties of the magma chamber at K\=ılauea Volcano in Hawai`i. We derived closed-form marginal likelihoods and implemented posterior sampling procedures that simultaneously estimate the model discrepancy of physical models, and the measurement bias from the atmospheric error in satellite interferograms. We found that model calibration by aggregating multiple interferograms and downsampling the pixels in the interferograms can reduce the computation complexity compared to calibration approaches based on multiple data sets. The conditions that lead to no loss of information from data aggregation and downsampling are studied. Simulation illustrates that both discrepancy and measurement bias can be estimated, and real applications demonstrate that modeling both effects helps obtain a reliable estimation of a physical model's unobserved parameters and enhance its predictive accuracy. We implement the computational tools in the RobustCalibration package available on CRAN.

Methodology
Two Error Models for Calibrating SCARA Robots Based on the MDH Model

Xiaolong Li,Hong Hu,Wenhua Ding

DOI: https://doi.org/10.1051/matecconf/20179508008

2017-01-01

MATEC Web of Conferences

Abstract:This paper describes the process of using two error models for calibrating Selective Compliance Assembly Robot Arm (SCARA) robots based on the modified Denavit-Hartenberg(MDH) model, with the aim of improving the robot's accuracy. One of the error models is the position error model, which uses robot position errors with respect to an accurate robot base frame built before the measurement commenced. The other model is the distance error model, which uses only the robot moving distance to calculate errors. Because calibration requires the end-effector to be accurately measured, a laser tracker was used to measure the robot position and distance errors. After calibrating the robot and, the end-effector locations were measured again compensating the error models' parameters obtained from the calibration. The finding is that the robot's accuracy improved greatly after compensating the calibrated parameters.
Analyzing method comparison data with skew-normal measurement error models: incorporating generalized scale mixtures and varying degrees of freedom

Jeevana Duwarahan,Lakshika S. Nawarathna

DOI: https://doi.org/10.1080/03610918.2024.2366984

2024-06-25

Communications in Statistics - Simulation and Computation

Abstract:Comparing two measurement methods is vital in various fields, such as medical research, epidemiology, economics, and environmental studies, to determine whether a new measurement method can be used interchangeably with an existing one. Measurement error models (MEMs) are commonly used for this purpose, where the methods have different measuring scales. However, these models often assume normality, which can be problematic when dealing with skewed and heavy-tailed data. To address this issue, we propose the replicated measurement error model (RMEM) under scale mixtures of skew-normal (SMSN) distributions with different levels of skewness and heavy tails of the true covariate and error distributions. Our primary aim is to assess the extent of similarity and agreement between two measurement methods using this model. The expectation conditional maximization (ECM) approach is applied to fit the model. A simulation study is conducted to evaluate the effectiveness and robustness of the proposed methodology and is illustrated by analyzing systolic blood pressure data. The probability of agreement is used further to assess the agreement between the two measurement methods. The findings indicate that the proposed model effectively analyses replicated method comparison data with measurement errors, even when there are outliers, skewness, and heavy-tailedness.

statistics & probability
Relative Closeness Ranking of Kalman Filtering with Multiple Mismatched Measurement Noise Covariances

Teng Shao,Quanbo Ge,Zhansheng Duan,Junzhi Yu

DOI: https://doi.org/10.1049/iet-cta.2017.1088

IF: 2.67

2018-01-01

IET Control Theory and Applications

Abstract:Due to the complexity of practical systems, the used models generally mismatch the practical ones. For the Kalman filtering with a single mismatched measurement noise covariance, the ranking of the ideal MSE (IMSE), the filter calculated MSE (FMSE), and the true MSE (TMSE) has been established. This study considers the ranking of the relative closenesses from the FMSE and TMSE to the IMSE when multiple mismatched measurement noise covariances are used. It is found that for the case with two deviations of the same signs, the larger the absolute deviation is, the farther away the FMSE or TMSE is from the IMSE. It is also found that for the case with two deviations of different signs, if the positive deviations are less than the absolute value of the negative deviation, then the FMSE or TMSE with positive deviation is relatively closer to the IMSE. Otherwise the ranking of the two corresponding relative closenesses depends on the parallel sum of two deviations with a non-zero threshold. Then the pair-wised ranking of the relative closenesses corresponding to two arbitrary deviations is further extended to the case with more than two deviations. Numerical examples are provided to validate the results.
Multi-model mimicry for model selection according to generalised goodness-of-fit criteria

Lachlann McArthur,Melissa A. Humphries

DOI: https://doi.org/10.48550/arXiv.1911.09779

2019-12-16

Abstract:Multi-model mimicry (MMM) is a flexible model selection technique for comparison of multiple, non-nested models on any desired goodness-of-fit criteria. Applicable to any set of candidate models that are 1) able to be fit to observed data, 2) can simulate new sets of data under the models, and 3) have a metric by which a dataset's goodness-of-fit to the model can be calculated, MMM has a much broader range of applicability than many standard model selection techniques. This manuscript highlights the previous literature whilst presenting the theoretical framework underpinning MMM. The scope of applicability is broadened through presentation of generalised criteria for comparison and the effectiveness of the method is demonstrated. Clear instruction for the application of MMM and the classification techniques required for model selection are also included.

Methodology,Statistics Theory

Supplement to 'Comparison of Misspecified Calibrated Models'

Comparison of Misspecified Calibrated Models: the Minimum Distance Approach

Broadening the Scope of Food Quality Control: a Calibration Transfer Strategy for Minimizing External Influences on Spectral Analysis

A Comparison Study of Mathematical Models for Tolerance Analysis

A Review for Model Plant Mismatch Measures in Process Monitoring

The Distance Between: An Algorithmic Approach to Comparing Stochastic Models to Time-Series Data

On Multivariate Calibration Problems

Calibrated Model Criticism Using Split Predictive Checks

Towards Automatic Model Comparison: An Adaptive Sequential Monte Carlo Approach

Sensitivity and Uncertainty Analysis of Nonideal Calibration Standards for 16-Term Error Model

ModelDiff: A Framework for Comparing Learning Algorithms

Hyper-differential sensitivity analysis with respect to model discrepancy: Optimal solution updating

Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

A Comparative Study of Representative Subset Selection for NIR Model Updating

Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding

Additive Distortion Measurement Errors Regression Models with Exponential Calibration

Calibration of imperfect geophysical models by multiple satellite interferograms with measurement bias

Two Error Models for Calibrating SCARA Robots Based on the MDH Model

Analyzing method comparison data with skew-normal measurement error models: incorporating generalized scale mixtures and varying degrees of freedom

Relative Closeness Ranking of Kalman Filtering with Multiple Mismatched Measurement Noise Covariances

Multi-model mimicry for model selection according to generalised goodness-of-fit criteria