Abstract:Effectively modeling multimodal longitudinal data is a pressing need in various application areas, especially biomedicine. Despite this, few approaches exist in the literature for this problem, with most not adequately taking into account the multimodality of the data. In this study, we developed multiple configurations of a novel multimodal and longitudinal learning framework, Longitudinal Ensemble Integration (LEI), for sequential classification. We evaluated LEI's performance, and compared it against existing approaches, for the early detection of dementia, which is among the most studied multimodal sequential classification tasks. LEI outperformed these approaches due to its use of intermediate base predictions arising from the individual data modalities, which enabled their better integration over time. LEI's design also enabled the identification of features that were consistently important across time for the effective prediction of dementia-related diagnoses. Overall, our work demonstrates the potential of LEI for sequential classification from longitudinal multimodal data.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to effectively model multimodal longitudinal data, especially in the biomedical field. Specifically, the paper proposes a new framework - **Longitudinal Ensemble Integration (LEI)** for sequential classification tasks. This problem has not been fully addressed in the existing literature, and most methods fail to fully consider the multimodal characteristics of the data. #### Research Background and Challenges 1. **Importance of multimodal longitudinal data**: - In fields such as biomedicine, more and more studies are beginning to use datasets that have both time - series characteristics and contain multiple data modalities. - These data modalities may include different feature types, such as imaging, genomics, clinical test results, etc. 2. **Limitations of existing methods**: - Most existing methods are insufficient when dealing with multimodal longitudinal data, especially in terms of early fusion, and may confound signals from different modalities. - Existing methods usually consider only a limited number of modalities, samples or time points, and cannot fully utilize the time dimension and multimodal characteristics of the data. 3. **Advantages of the LEI framework**: - By introducing intermediate base predictions, LEI can better integrate data from different modalities and optimize over time. - This framework not only improves classification performance, but also can identify the features that are most important for predicting dementia diagnosis at different time points. #### Specific Applications The paper takes **early dementia detection** as an example to demonstrate the application value of the LEI framework. The specific task is to use the multimodal longitudinal data in the TADPOLE dataset to predict whether patients will progress from cognitively normal (CN), mild cognitive impairment (MCI) to dementia in future visits. The research results show that LEI performs better than other existing methods in this task. #### Method Overview 1. **Base prediction generation**: - Train base prediction models (such as KNN, Logistic Regression, SVM, etc.) for different - modality data at each time point respectively. 2. **LSTM stacking**: - Use the Long - Short - Term Memory network (LSTM) to stack these base prediction results to form the final prediction model. 3. **Loss function design**: - Propose a double - weighted cross - entropy loss function (DWCCE) to deal with the problems of class imbalance and ordered labels. 4. **Configuration evaluation**: - Evaluate four different LEI configurations, involving combinations of time - dependence, time - distribution and longitudinal modeling strategies. #### Summary By developing and evaluating the LEI framework, this paper solves the key challenges in multimodal longitudinal data modeling, especially for tasks that require prediction over time, such as early dementia detection. LEI not only improves prediction accuracy, but also provides the ability to interpret important features, providing new ideas and technical means for similar tasks in the future.

Longitudinal Ensemble Integration for sequential classification with multimodal data

Multi-modal Sequence Learning for Alzheimer's Disease Progression Prediction with Incomplete Variable-Length Longitudinal Data.

Multi-modal latent space inducing ensemble SVM classifier for early dementia diagnosis with neuroimaging data

Multi-modal sequence learning for Alzheimer’s disease progression prediction with incomplete variable-length longitudinal data

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Prediction of future dementia among patients with mild cognitive impairment (MCI) by integrating multimodal clinical data

Enhanced Multimodal Low-rank Embedding based Feature Selection Model for Multimodal Alzheimer's Disease Diagnosis

On self-supervised multi-modal representation learning: An application to Alzheimer's disease

Self-supervised multimodal neuroimaging yields predictive representations for a spectrum of Alzheimer's phenotypes

Alzheimer’s Disease Diagnosis from Multi-Modal Data Via Feature Inductive Learning and Dual Multilevel Graph Neural Network

Alzheimer's disease diagnosis from multi-modal data via feature inductive learning and dual multilevel graph neural network

Towards Multimodal Longitudinal Analysis for Predicting Cognitive Decline Using Neuroimaging Biomarkers, Cognitive Assessments, and Demographic Data

Leveraging Coupled Interaction for Multimodal Alzheimer’s Disease Diagnosis

WISE 2014 Challenge: Multi-label Classification of Print Media Articles to Topics

Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning

Poem: lichenplanus.

Supervised Multi-Modal Fission Learning

Enhancing Early Alzheimer's Disease Classification Accuracy Through the Fusion of Smri and Rsmeg Data: a Deep Learning Approach

Multimodal functional deep learning for multiomics data

High-Order Laplacian Regularized Low-Rank Representation for Multimodal Dementia Diagnosis

Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer's Disease