Abstract:Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is capturing (1) the spatial correlations at the same time, (2) the spatio-temporal relationships at different times, and (3) long-term dependency of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent and learn the dynamic spatio-temporal relationships between exogenous series and target series, but they only perform well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-II respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Then, we employ multiple attentions on target series to boost the long-term dependency. Finally, we study the performance of deep spatial attention mechanism and provide interpretation. Experimental results demonstrate that the present work can be successfully used to develop expert or intelligent systems for a wide range of applications, with state-of-the-art performances superior to nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively. Overall, the present work carries a significant value not merely in the domain of machine intelligence and deep learning, but also in the fields of many applications. (C) 2019 Elsevier Ltd. All rights reserved.

Recurrent Attentive Neural Process for Sequential Data

Residual Recurrent Neural Networks for Learning Sequential Representations.

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Learning Time Series Associated Event Sequences with Recurrent Point Process Networks

Probabilistic Trajectory Prediction for Autonomous Vehicles with Attentive Recurrent Neural Process

Learning The Sequential Temporal Information with Recurrent Neural Networks

Probabilistic Sequential Network for Deep Learning of Complex Process Data and Soft Sensor Application

Modeling The Intensity Function Of Point Process Via Recurrent Neural Networks

Machine Learning on Sequential Data Using a Recurrent Weighted Average

Recurrent Spiking Neural Network with Dynamic Presynaptic Currents Based on Backpropagation

Joint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural Networks

Temporal Pyramid Recurrent Neural Network.

On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes

Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit

DSTP-RNN: A Dual-Stage Two-Phase Attention-Based Recurrent Neural Network for Long-Term and Multivariate Time Series Prediction

Relation-Aware Attentive Neural Processes Model for Remaining Useful Life Prediction

Can recurrent neural networks learn process model structure?

Neural Processes with Stochastic Attention: Paying more attention to the context dataset

Feature-Aware Attentive Convolutional Neural Network for Sequence Processing.

An Adaptive Growing and Pruning Algorithm for Designing Recurrent Neural Network.