Abstract:Advancements in computer hardware have made it possible to utilize low- and mixed-precision arithmetic for enhanced computational efficiency. In practical predictive modeling, however, it is vital to quantify uncertainty due to rounding along other sources like measurement, sampling, and numerical discretization. Traditional deterministic rounding uncertainty analysis (DBEA) assumes that the rounding errors equal the unit roundoff $u$. However, despite providing strong guarantees, DBEA severely overestimates rounding uncertainty. This work presents a novel probabilistic rounding uncertainty analysis called VIBEA. By treating rounding errors as i.i.d. random variables and leveraging concentration inequalities, VIBEA provides high-confidence estimates for rounding uncertainty using higher-order rounding error statistics. The presented framework is valid for all problem sizes $n$, unlike DBEA, which necessitates $nu<1$. Further, it can account for the potential cancellation of rounding errors, resulting in rounding uncertainty estimates that grow slowly with $n$. We show that for $n>n_c(u)$, VIBEA produces tighter estimates for rounding uncertainty than DBEA. We also show that VIBEA improves existing probabilistic rounding uncertainty analysis techniques for $n\ge3$ by using higher-order rounding error statistics. We conduct numerical experiments on random vector dot products, a linear system solution, and a stochastic boundary value problem. We show that quantifying rounding uncertainty along with traditional sources (numerical discretization, sampling, parameters) enables a more efficient allocation of computational resources, thereby balancing computational efficiency with predictive accuracy. This study is a step towards a comprehensive mixed-precision approach that improves model reliability and enables budgeting of computational resources in predictive modeling and decision-making.

Exploring the impact of post-training rounding in regression models

Efficient Adaptive Activation Rounding for Post-Training Quantization

Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective

Attention Round for post-training quantization

On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization

Regression Discontinuity Applications with Rounding Errors in the Running Variable

A Multi-Precision Quadratic Regularization Method for Unconstrained Optimization with Rounding Error Analysis

Towards Accurate Post-training Quantization for Reparameterized Models

What You Will Gain By Rounding: Theory and Algorithms for Rounding Rank

Post-training quantization for re-parameterization via coarse & fine weight splitting

Estimating Counts Through an Average Rounded to the Nearest Non-negative Integer and its Theoretical & Practical Effects

Variance-informed Rounding Uncertainty Analysis for Floating-point Statistical Models

Post-training Quantization or Quantization-aware Training? That is the Question

Probabilistic Calibration by Design for Neural Network Regression

A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models

Low-Precision Floating-Point Schemes for Neural Network Training

Understanding the difficulty of low-precision post-training quantization of large language models

CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression