Abstract:Understanding how the predictions of deep learning models are formed during the training process is crucial to improve model performance and fix model defects, especially when we need to investigate nontrivial training strategies such as active learning, and track the root cause of unexpected training results such as performance degeneration. In this work, we propose a time-travelling visual solution DeepVisualInsight (DVI), aiming to manifest the spatio-temporal causality while training a deep learning image classifier. The spatio-temporal causality demonstrates how the gradient-descent algorithm and various training data sampling techniques can influence and reshape the layout of learnt input representation and the classification boundaries in consecutive epochs. Such causality allows us to observe and analyze the whole learning process in the visible low dimensional space. Technically, we propose four spatial and temporal properties and design our visualization solution to satisfy them. These properties preserve the most important information when inverse-)projecting input samples between the visible low-dimensional and the invisible high-dimensional space, for causal analyses. Our extensive experiments show that, comparing to baseline approaches, we achieve the best visualization performance regarding the spatial/temporal properties and visualization efficiency. Moreover, our case study shows that our visual solution can well reflect the characteristics of various training scenarios, showing good potential of DVI as a debugging tool for analyzing deep learning training processes.

Illustrative Visualization of Time-Varying Features in Spatio-Temporal Data.

Visual Analytics of Economic Features for Multivariate Spatio-Temporal GDP Data

TieVis: visual analytics of evolution of interpersonal ties

Visual Analytics of the Spatio-temporal Multidimensional Air Monitoring Data

FeatureFlow: Exploring Feature Evolution for Time-Varying Volume Data.

Visualizing Large-Scale Spatial Time Series with GeoChron.

Illustrating Changes in Time-Series Data With Data Video

Visual Features Extraction Through Spatiotemporal Slice Analysis

DeepVisualInsight: Time-Travelling Visualization for Spatio-Temporal Causality of Deep Classification Training

A Visual Analytics Approach to Understanding Spatiotemporal Hotspots

Optimizing temporal topic segmentation for intelligent text visualization.

SalienTime: User-driven Selection of Salient Time Steps for Large-Scale Geospatial Data Visualization

Visual Analysis of GDP with Spatiotemporal Multidimensional Features

Transforming Multidimensional Time Series into Interpretable Event Sequences for Advanced Data Mining

Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations

Methods for Space-Time Analysis and Modeling

EasyXplorer: A Flexible Visual Exploration Approach for Multivariate Spatial Data.

Window Query and Analysis on Massive Spatio-temporal Data

Variable-Based Spatiotemporal Trajectory Data Visualization Illustrated

Topic- and Time-Oriented Visual Text Analysis.

Encoding Temporal Markov Dynamics in Graph for Visualizing and Mining Time Series