ImageSI: Semantic Interaction for Deep Learning Image Projections

Jiayue Lin,Rebecca Faust,Chris North

2024-08-07

Abstract:Semantic interaction (SI) in Dimension Reduction (DR) of images allows users to incorporate feedback through direct manipulation of the 2D positions of images. Through interaction, users specify a set of pairwise relationships that the DR should aim to capture. Existing methods for images incorporate feedback into the DR through feature weights on abstract embedding features. However, if the original embedding features do not suitably capture the users' task then the DR cannot either. We propose ImageSI, an SI method for image DR that incorporates user feedback directly into the image model to update the underlying embeddings, rather than weighting them. In doing so, ImageSI ensures that the embeddings suitably capture the features necessary for the task so that the DR can subsequently organize images using those features. We present two variations of ImageSI using different loss functions - ImageSI_MDS_Inverse, which prioritizes the explicit pairwise relationships from the interaction and ImageSI_Triplet, which prioritizes clustering, using the interaction to define groups of images. Finally, we present a usage scenario and a simulation based evaluation to demonstrate the utility of ImageSI and compare it to current methods.

Human-Computer Interaction

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively incorporate user feedback in the process of image dimensionality reduction (DR) to generate an image organization that better meets the user's task requirements and prior knowledge. Existing methods typically incorporate user feedback into the DR model by adjusting feature weights, but this approach relies on whether the original embedded features can adequately capture the user's task requirements. If the original embedded features are not suitable, adjusting the weights will not achieve the desired effect.

To overcome this limitation, the authors propose the **ImageSI** framework, which incorporates user feedback by directly updating the embedded features in the image model rather than simply adjusting feature weights. This ensures that the embedded features can better capture the features relevant to the user's task, thereby generating dimensionality reduction results that better meet user needs.

Specifically, ImageSI provides two different loss functions to achieve this goal:
1. **ImageSIMDS−1**: Prioritizes the explicit pairwise relationships specified in the interaction by optimizing the model to minimize the difference between the distances in the embedding space and the distances specified by the user in the 2D DR space.
2. **ImageSI Triplet**: Prioritizes clustering based on user feedback by defining groups of images to optimize the model.

Through these methods, ImageSI can better capture the complex information in user feedback and generate an image organization that better meets user needs. The paper also validates the effectiveness and superiority of ImageSI through a use case and simulation-based evaluation.