Abstract:In this paper, we design novel interactive deep learning methods to improve semantic interactions in visual analytics applications. The ability of semantic interaction to infer analysts' precise intents during sensemaking is dependent on the quality of the underlying data representation. We propose the $\text{DeepSI}_{\text{finetune}}$ framework that integrates deep learning into the human-in-the-loop interactive sensemaking pipeline, with two important properties. First, deep learning extracts meaningful representations from raw data, which improves semantic interaction inference. Second, semantic interactions are exploited to fine-tune the deep learning representations, which then further improves semantic interaction inference. This feedback loop between human interaction and deep learning enables efficient learning of user- and task-specific representations. To evaluate the advantage of embedding the deep learning within the semantic interaction loop, we compare $\text{DeepSI}_{\text{finetune}}$ against a state-of-the-art but more basic use of deep learning as only a feature extractor pre-processed outside of the interactive loop. Results of two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment, show that $\text{DeepSI}_{\text{finetune}}$ more accurately captures users' complex mental models with fewer interactions.

What problem does this paper attempt to address?

The paper aims to improve semantic interaction (SI) in visual analytics (VA) systems by designing a novel interactive deep learning method to enhance the efficiency and accuracy of the data analysis process. Specifically, the paper proposes a new framework called DeepSIfinetune, which integrates deep learning (DL) models into the human-computer interaction data analysis workflow, allowing adaptive adjustment of data representation based on the user's specific task requirements. ### Problems the Paper Attempts to Solve: 1. **How to use semantic interaction to accurately adjust pre-trained models to fit the current task?** - By leveraging user-system interaction feedback, the pre-trained model is adjusted to better understand user intent and generate data representations more suitable for the current task. 2. **How to efficiently perform such adjustments so that a small amount of interaction is sufficient for the user to express their intent?** - A fine-tuning method is proposed, requiring only a small number of parameter updates to achieve efficient model adjustment, thereby reducing the number of interactions needed by the user. ### Method Overview - **DeepSIfinetune Framework**: Embeds deep learning models (such as BERT) into the semantic interaction loop, dynamically adjusting model parameters through user interactions to generate data representations more suitable for the current task. - **Experimental Validation**: Evaluated the effectiveness of DeepSIfinetune through two complementary research methods (qualitative case studies and quantitative simulation experiments). The results show that this method not only captures user intent more accurately but also significantly reduces the number of interactions required. ### Experimental Section - **Case Study**: Evaluated the performance of DeepSIfinetune in real tasks by analyzing academic literature related to COVID-19. - **Quantitative Analysis**: Conducted algorithm-centric quantitative analysis using three commonly used text corpora (Stanford Sentiment Treebank, Vispubdata, and 20 Newsgroups) to further validate the effectiveness and superiority of DeepSIfinetune. Through these methods, the paper demonstrates the advantages of DeepSIfinetune in improving the accuracy and efficiency of semantic interaction, providing new ideas and technical support for future visual analytics systems.

DeepSI: Interactive Deep Learning for Semantic Interaction

NeuralSI: Neural Design of Semantic Interaction for Interactive Deep Learning

Evaluating Semantic Interaction on Word Embeddings via Simulation

ImageSI: Semantic Interaction for Deep Learning Image Projections

Interpretable Artificial Intelligence through the Lens of Feature Interaction

Improving Interpretability of Deep Neural Networks with Semantic Information

DeepInteraction++: Multi-Modality Interaction for Autonomous Driving

DeepInteraction: 3D Object Detection via Modality Interaction

Mixed Multi-Model Semantic Interaction for Graph-based Narrative Visualizations

An In-Situ Visual Analytics Framework for Deep Neural Networks

DEAR: a novel deep-level semantics feature reinforce framework for Infrared Small Object Segmentation

SDIF-DA: A Shallow-to-Deep Interaction Framework with Data Augmentation for Multi-modal Intent Detection

xDeepInt: a hybrid architecture for modeling the vector-wise and bit-wise feature interactions

Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies

DeepEye: Towards Automatic Data Visualization

$\textit{Revelio}$: Interpreting and leveraging semantic information in diffusion models

Hierarchical visual-semantic interaction for scene text recognition

Asynchronous Deep Interaction Network for Natural Language Inference.

Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers

A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding

Semantic-Enhanced Explainable Finetuning for Open-Domain Dialogues