Abstract:AI-driven models are increasingly deployed in operational analytics solutions, for instance, in investigative journalism or the intelligence community. Current approaches face two primary challenges: ethical and privacy concerns, as well as difficulties in efficiently combining heterogeneous data sources for multimodal analytics. To tackle the challenge of multimodal analytics, we present MULTI-CASE, a holistic visual analytics framework tailored towards ethics-aware and multimodal intelligence exploration, designed in collaboration with domain experts. It leverages an equal joint agency between human and AI to explore and assess heterogeneous information spaces, checking and balancing automation through Visual Analytics. MULTI-CASE operates on a fully-integrated data model and features type-specific analysis with multiple linked components, including a combined search, annotated text view, and graph-based analysis. Parts of the underlying entity detection are based on a RoBERTa-based language model, which we tailored towards user requirements through fine-tuning. An overarching knowledge exploration graph combines all information streams, provides in-situ explanations, transparent source attribution, and facilitates effective exploration. To assess our approach, we conducted a comprehensive set of evaluations: We benchmarked the underlying language model on relevant NER tasks, achieving state-of-the-art performance. The demonstrator was assessed according to intelligence capability assessments, while the methodology was evaluated according to ethics design guidelines. As a case study, we present our framework in an investigative journalism setting, supporting war crime investigations. Finally, we conduct a formative user evaluation with domain experts in law enforcement. Our evaluations confirm that our framework facilitates human agency and steering in security-sensitive applications.

Shared Model of Sense-making for Human-Machine Collaboration

A Framework for Developing and Using Shared Mental Models in Human-Agent Teams

Iceberg Sensemaking: A Process Model for Critical Data Analysis and Visualization

Extending Sense-Making Models with Ideas from Cognition and Learning Theories.

What Did My AI Learn? How Data Scientists Make Sense of Model Behavior

Making Sense of the World: Models for Reliable Sensor-Driven Systems

Flexible and Inherently Comprehensible Knowledge Representation for Data-Efficient Learning and Trustworthy Human-Machine Teaming in Manufacturing Environments

Towards a Comprehensive Model of the Cognitive Process and Mechanisms of Individual Sensemaking

Vital Insight: Assisting Experts' Sensemaking Process of Multi-modal Personal Tracking Data Using Visualization and LLM

Human-Computer Collaboration for Visual Analytics: an Agent-based Framework

MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework

Algorithms for the Greater Good! On Mental Modeling and Acceptable Symbiosis in Human-AI Collaboration

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

The Future Ethics of Artificial Intelligence in Medicine: Making Sense of Collaborative Models

Use Cases for Prospective Sensemaking of Human-AI-Collaboration

Sabotage Evaluations for Frontier Models

Designing for Collaborative Sensemaking: Leveraging Human Cognition For Complex Tasks

A Logical Model for Supporting Social Commonsense Knowledge Acquisition

Co-creating a globally interpretable model with human input

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

A Model for Intelligible Interaction Between Agents That Predict and Explain