Abstract:Recently, uncertainty-aware deep learning methods for multiclass labeling problems have been developed that provide calibrated class prediction probabilities and out-of-distribution (OOD) indicators, letting machine learning (ML) consumers and engineers gauge a model's confidence in its predictions. However, this extra neural network prediction information is challenging to scalably convey visually for arbitrary data sources under multiple uncertainty contexts. To address these challenges, we present ScatterUQ, an interactive system that provides targeted visualizations to allow users to better understand model performance in context-driven uncertainty settings. ScatterUQ leverages recent advances in distance-aware neural networks, together with dimensionality reduction techniques, to construct robust, 2-D scatter plots explaining why a model predicts a test example to be (1) in-distribution and of a particular class, (2) in-distribution but unsure of the class, and (3) out-of-distribution. ML consumers and engineers can visually compare the salient features of test samples with training examples through the use of a ``hover callback'' to understand model uncertainty performance and decide follow up courses of action. We demonstrate the effectiveness of ScatterUQ to explain model uncertainty for a multiclass image classification on a distance-aware neural network trained on Fashion-MNIST and tested on Fashion-MNIST (in distribution) and MNIST digits (out of distribution), as well as a deep learning model for a cyber dataset. We quantitatively evaluate dimensionality reduction techniques to optimize our contextually driven UQ visualizations. Our results indicate that the ScatterUQ system should scale to arbitrary, multiclass datasets. Our code is available at <a class="link-external link-https" href="https://github.com/mit-ll-responsible-ai/equine-webapp" rel="external noopener nofollow">this https URL</a>

Interactive Ranking Uncertain Multivariate Ordinal Time Series

RankBrushers: Interactive Analysis of Temporal Ranking Ensembles

Uncertainty-Oriented Ensemble Data Visualization and Exploration using Variable Spatial Spreading

RankExplorer: Visualization of Ranking Changes in Large Time Series Data

The Interval Grey Numbers Ranking Based on Risk Preferences

SRVis: Towards Better Spatial Integration in Ranking Visualization

RankAxis: Towards a Systematic Combination of Projection and Ranking in Multi-Attribute Data Exploration

Visual Analytics of Economic Features for Multivariate Spatio-Temporal GDP Data

The Risks of Ranking: Revisiting Graphical Perception to Model Individual Differences in Visualization Performance

Data-driven Ranking and Selection under Input Uncertainty

Multivariate Time Series Anomaly Detection and Interpretation Using Hierarchical Inter-Metric and Temporal Embedding

Capturing Variation and Uncertainty in Human Judgment

Ranked from Within: Ranking Large Multimodal Models for Visual Question Answering Without Labels

ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems

Visual Entropy and the Visualization of Uncertainty

Auditing the Sensitivity of Graph-based Ranking with Visual Analytics

Bayesian Rank-Clustering

Uncertainty-aware Multidimensional Ensemble Data Visualization and Exploration

Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking

Uncertainty-aware Visual Analytics for Exploring Human Behaviors from Heterogeneous Spatial Temporal Data.

A Survey on Uncertainty Visualization