Abstract:Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce \texttt{Lightning UQ Box}: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: <a class="link-external link-https" href="https://github.com/lightning-uq-box/lightning-uq-box" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of the lack of uncertainty quantification (UQ) in deep learning (DL) models in practical applications. Specifically: 1. **Provide confidence assessment**: Uncertainty quantification (UQ) is a crucial tool for applying deep neural networks (DNNs) to real - world tasks because it can attach a confidence level to the output of a DNN. However, UQ is often excluded from the standard DNN workflow because additional technical knowledge is required to apply and evaluate existing UQ methods. 2. **Simplify UQ integration**: To enable users to easily integrate UQ into their modeling workflows without significant overhead, this paper introduces a unified interface called Lightning UQ Box for applying and evaluating various UQ methods. 3. **Handle complex tasks**: The paper verifies the effectiveness of its framework through two challenging visual tasks: - **Estimate tropical cyclone wind speed**: Estimate the maximum sustained wind speed of a tropical cyclone from infrared satellite images. - **Predict solar panel power output**: Estimate the power - voltage output of a solar panel from RGB images of the sky. 4. **Provide comprehensive comparison**: The paper provides extensive theoretical and quantitative comparisons of state - of - the - art UQ methods implemented in their toolbox. By highlighting the differences between different methods, the research results demonstrate the need for a comprehensive and accessible experimental framework for UQ methods, which is helpful for benchmarking UQ methods. ### Main contributions - **Comprehensive end - to - end UQ toolbox**: Lightning UQ Box enables practitioners to efficiently iterate ideas without re - implementing the provided UQ methods. It provides implementations of various UQ methods and their corresponding evaluation schemes that are independent of the backbone architecture and dataset, covering regression, classification, semantic segmentation, and pixel - level regression tasks. - **Adaptability and extensibility**: Using the Lightning modular implementation encourages practitioners and the community to make personalized adjustments and continuously expand and improve the toolbox. In addition, the implementation adapts to vector or visual data and supports partial randomness (when applicable). This supports any larger visual architecture, and the "freeze" function allows only a few layers to be retrained. - **Practical and theoretical introduction**: The toolbox contains a comprehensive practical and theoretical introduction to the UQ field, as well as tutorials and case studies for applying the toolbox to design downstream tasks to compare various UQ methods. A comprehensive theoretical guide introducing the methodological background of the implemented methods is also provided. Through these contributions, Lightning UQ Box aims to bridge the gap between theoretical researchers and practical practitioners in the field of deep learning uncertainty quantification, making the application of UQ methods more convenient and efficient.

Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning