Abstract:Recently, uncertainty-aware deep learning methods for multiclass labeling problems have been developed that provide calibrated class prediction probabilities and out-of-distribution (OOD) indicators, letting machine learning (ML) consumers and engineers gauge a model's confidence in its predictions. However, this extra neural network prediction information is challenging to scalably convey visually for arbitrary data sources under multiple uncertainty contexts. To address these challenges, we present ScatterUQ, an interactive system that provides targeted visualizations to allow users to better understand model performance in context-driven uncertainty settings. ScatterUQ leverages recent advances in distance-aware neural networks, together with dimensionality reduction techniques, to construct robust, 2-D scatter plots explaining why a model predicts a test example to be (1) in-distribution and of a particular class, (2) in-distribution but unsure of the class, and (3) out-of-distribution. ML consumers and engineers can visually compare the salient features of test samples with training examples through the use of a ``hover callback'' to understand model uncertainty performance and decide follow up courses of action. We demonstrate the effectiveness of ScatterUQ to explain model uncertainty for a multiclass image classification on a distance-aware neural network trained on Fashion-MNIST and tested on Fashion-MNIST (in distribution) and MNIST digits (out of distribution), as well as a deep learning model for a cyber dataset. We quantitatively evaluate dimensionality reduction techniques to optimize our contextually driven UQ visualizations. Our results indicate that the ScatterUQ system should scale to arbitrary, multiclass datasets. Our code is available at <a class="link-external link-https" href="https://github.com/mit-ll-responsible-ai/equine-webapp" rel="external noopener nofollow">this https URL</a>

UADAPy: An Uncertainty-Aware Visualization and Analysis Toolbox

A workflow to systematically design uncertainty-aware visual analytics applications

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

A Survey on Uncertainty Visualization

Visualizing Flow of Uncertainty Through Analytical Processes.

Visualizing Large-Scale Uncertainty in Astrophysical Data

The Role of Uncertainty, Awareness, and Trust in Visual Analytics

A Statistical Direct Volume Rendering Framework for Visualization of Uncertain Data

Uncertainty-aware Visual Analytics for Exploring Human Behaviors from Heterogeneous Spatial Temporal Data.

ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems

Sounding Spider: An Efficient Way for Representing Uncertainties in High Dimensions

Uncertainty-Aware Deep Neural Representations for Visual Analysis of Vector Field Data

Evaluating the Use of Uncertainty Visualisations for Imputations of Data Missing At Random in Scatterplots

UNMAT: Visual Comparison and Exploration of Uncertainty in Large Graph Sampling.

Uncertainty-Oriented Ensemble Data Visualization and Exploration using Variable Spatial Spreading

Data visualization for decision making under deep uncertainty: Current challenges and opportunities

Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

Maybe, Maybe Not: A Survey on Uncertainty in Visualization

Enhancing Uncertainty Communication in Time Series Predictions: Insights and Recommendations

Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI

Uncertainty-Aware Principal Component Analysis