HealthPrism: A Visual Analytics System for Exploring Children's Physical and Mental Health Profiles with Multimodal Data

Zhihan Jiang,Handi Chen,Rui Zhou,Jing Deng,Xinchen Zhang,Running Zhao,Cong Xie,Yifang Wang,Edith C.H. Ngai
2024-04-10
Abstract:The correlation between children's personal and family characteristics (e.g., demographics and socioeconomic status) and their physical and mental health status has been extensively studied across various research domains, such as public health, medicine, and data science. Such studies can provide insights into the underlying factors affecting children's health and aid in the development of targeted interventions to improve their health outcomes. However, with the availability of multiple data sources, including context data (i.e., the background information of children) and motion data (i.e., sensor data measuring activities of children), new challenges have arisen due to the large-scale, heterogeneous, and multimodal nature of the data. Existing statistical hypothesis-based and learning model-based approaches have been inadequate for comprehensively analyzing the complex correlation between multimodal features and multi-dimensional health outcomes due to the limited information revealed. In this work, we first distill a set of design requirements from multiple levels through conducting a literature review and iteratively interviewing 11 experts from multiple domains (e.g., public health and medicine). Then, we propose HealthPrism, an interactive visual and analytics system for assisting researchers in exploring the importance and influence of various context and motion features on children's health status from multi-level perspectives. Within HealthPrism, a multimodal learning model with a gate mechanism is proposed for health profiling and cross-modality feature importance comparison. A set of visualization components is designed for experts to explore and understand multimodal data freely. We demonstrate the effectiveness and usability of HealthPrism through quantitative evaluation of the model performance, case studies, and expert interviews in associated domains.
Human-Computer Interaction
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: How to effectively analyze the complex associations between children's personal and family background characteristics (such as demographics and socioeconomic status) and their physical and mental health states. With the popularization of wearable devices, large - scale continuous sensor signals can be collected from children, but the existing methods based on statistical assumptions and learning models are insufficient when dealing with large - scale, heterogeneous and multi - modal data. Specific challenges include: 1. **Visualization of multi - modal data**: How to visualize a large amount of multi - modal data in a comprehensive and easy - to - understand way. 2. **Association between input and health outcomes**: How to depict the associations between different types of input (i.e., context and motion features) and multiple health outcomes. 3. **Comparison of the importance of different modal features**: How to compare the importance of different modal features. 4. **Multi - level analysis and support**: How to support the analysis and reasoning of multi - modal data from the overall, group and individual levels. To address these challenges, researchers proposed **HealthPrism**, an interactive visual analysis system, aiming to help researchers explore multi - modal features in children's health records. The system solves the above problems through the following aspects: - **Multi - modal learning model**: A multi - modal learning model with a gating mechanism is proposed, which can automatically adjust feature weights and infer health indicators. - **Visualization components**: A series of visualization components are designed to enable experts to freely explore and understand multi - modal data. - **Multi - level views**: The system includes a Summary View, a Group View and an Individual View, which show data from the overall, group and individual levels respectively. Through these methods, HealthPrism can effectively analyze multi - modal data and reveal the importance of each feature in children's health records and its impact on health outcomes.