Abstract:Recurrent neural networks (RNNs) are commonly applied to clinical time-series data with the goal of learning patient risk stratification models. Their effectiveness is due, in part, to their use of parameter sharing over time (i.e., cells are repeated hence the name recurrent). We hypothesize, however, that this trait also contributes to the increased difficulty such models have with learning relationships that change over time. Conditional shift, i.e., changes in the relationship between the input X and the output y, arises when risk factors associated with the event of interest change over the course of a patient admission. While in theory, RNNs and gated RNNs (e.g., LSTMs) in particular should be capable of learning time-varying relationships, when training data are limited, such models often fail to accurately capture these dynamics. We illustrate the advantages and disadvantages of complete parameter sharing (RNNs) by comparing an LSTM with shared parameters to a sequential architecture with time-varying parameters on prediction tasks involving three clinically-relevant outcomes: acute respiratory failure (ARF), shock, and in-hospital mortality. In experiments using synthetic data, we demonstrate how parameter sharing in LSTMs leads to worse performance in the presence of conditional shift. To improve upon the dichotomy between complete parameter sharing and no parameter sharing, we propose a novel RNN formulation based on a mixture model in which we relax parameter sharing over time. The proposed method outperforms standard LSTMs and other state-of-the-art baselines across all tasks. In settings with limited data, relaxed parameter sharing can lead to improved patient risk stratification performance.

Hierarchical Parameter Sharing In Recursive Neural Networks With Long Short-Term Memory

When Are Tree Structures Necessary for Deep Learning of Representations?

Relaxed Parameter Sharing: Effectively Modeling Time-Varying Relationships in Clinical Time-Series

Parameters Sharing in Residual Neural Networks

Shuttlenet: A Biologically-Inspired RNN with Loop Connection and Parameter Sharing.

A New Hybrid-Parameter Recurrent Neural Network for Online Handwritten Chinese Character Recognition

Long Short-Term Memory with Quadratic Connections in Recursive Neural Networks for Representing Compositional Semantics

Understanding Parameter Sharing in Transformers

Adaptive Multi-Compositionality for Recursive Neural Network Models

DCRNN: A Deep Cross approach based on RNN for Partial Parameter Sharing in Multi-task Learning

Restricted Recurrent Neural Networks

MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

Learning Sparse Sharing Architectures for Multiple Tasks.

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network.

Highway State Gating for Recurrent Highway Networks: improving information flow through time

Going Wider: Recurrent Neural Network with Parallel Cells

Loop Neural Networks for Parameter Sharing

Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models