Abstract:In this study, we propose a scientific framework to detect capability among biomedical large language models (LLMs) for organizing expressions of comorbid disease and temporal progression. We hypothesize that biomedical LLMs pretrained on next-token prediction produce latent spaces that implicitly capture "disease states" and disease progression, i.e., the transitions over disease states over time. We describe how foundation models may capture and transfer knowledge from explicit pretraining tasks to specific clinical applications. A scoring function based on Kullback-Leibler divergence was developed to measure "surprise" in seeing specialization when subsetting admissions along 13 biomedical LLM latent spaces. By detecting implicit ordering of longitudinal data, we aim to understand how these models self-organize clinical information and support tasks such as phenotypic classification and mortality prediction. We test our hypothesis along a case study for obstructive sleep apnea (OSA) in the publicly available MIMIC-IV dataset, finding ordering of phenotypic clusters and temporality within latent spaces. Our quantitative findings suggest that increased compute, conformance with compute-optimal training, and widening contexts promote better implicit ordering of clinical admissions by disease states, explaining 60.3% of the variance in our proposed implicit task. Preliminary qualitative findings suggest LLMs' latent spaces trace patient trajectories through different phenotypic clusters, terminating at end-of-life phenotypes. This approach highlights the potential of biomedical LLMs in modeling disease progression, identifying new patterns in disease pathways and interventions, and evaluating clinical hypotheses related to drivers of severe illness. We underscore the need for larger, high-resolution longitudinal datasets to further validate and enhance understanding of the utility of LLMs in modeling patient trajectories along clinical text and advancing precision medicine.

Semi-Supervised Generative Models for Disease Trajectories: A Case Study on Systemic Sclerosis

Modeling Complex Disease Trajectories using Deep Generative Models with Semi-Supervised Latent Processes

Deep Semisupervised Multitask Learning Model and Its Interpretability for Survival Analysis.

A Framework for Individualizing Predictions of Disease Trajectories by Exploiting Multi-Resolution Structure

Time-dependent Probabilistic Generative Models for Disease Progression

Reconstructing Disease Dynamics for Mechanistic Insights and Clinical Benefit

Disease Trajectory Maps

Probabilistic Temporal Prediction of Continuous Disease Trajectories and Treatment Effects Using Neural SDEs

Longitudinal modeling of MS patient trajectories improves predictions of disability progression

Modelling and Analysing the Dynamics of Disease Progression from Cross-Sectional Studies

A temporal model for multiple sclerosis course evolution

Describing complex disease progression using joint latent class models for multivariate longitudinal markers and clinical endpoints

A probabilistic generative model to discover the treatments of coexisting diseases with missing data

Uncovering Disease Regions Using Pseudo Time-Series Trajectories on Clinical Trial Data

Towards Maps of Disease Progression: Biomedical Large Language Model Latent Spaces For Representing Disease Phenotypes And Pseudotime

Disease Progression Modelling and Stratification for detecting sub-trajectories in the natural history of pathologies: application to Parkinson's Disease trajectory modelling

Scalable Modeling of Multivariate Longitudinal Data for Prediction of Chronic Kidney Disease Progression

Modeling Disease Progression Trajectories from Longitudinal Observational Data

Forecasting Individualized Disease Trajectories using Interpretable Deep Learning

Learning transition times in event sequences: the Event-Based Hidden Markov Model of disease progression

Deep Ensemble Tensor Factorization for Longitudinal Patient Trajectories Classification