Abstract:Existing convolution recurrent neural networks (ConvRNNs)-based memory cells majorly take advantage of gated structures and attention mechanisms to extract discontinuous latent associations for spatial-temporal sequence forecast (STSF) problems, which may lead to serious over-fitting and spurious relationships with correlated noise. It is a consensus that incorporating cause-effect relationships in modeling can alleviate these problems. In this paper, we propose a Causality Attention Unit (CAU) to assist ConvRNNs by complementing the causal inference ability in a plug-and-play way. Specifically, CAU serially consists of the attention module and causality module. The former is constructed by a spatial-channel attention layer, which preliminarily generates the correlated future with the correlations between historical memories and the current state. The latter borrows the idea of transfer entropy ( TE ) to detect the latent cause-effect relationships and precisely correct the correlated future. A space-time exchange strategy for accelerating the calculation of TE in CAU is also designed. CAU can be easily combined with the existing ConvRNN cells, and we construct a simple general model to predict long-term spatial-temporal series, which consists of encoder/decoder and stacked CAU paralleled to stacked ConvRNN cells. After determining the optimal model structure, we carry out a series of experiments to evaluate model performance, including comparisons with other advanced models, training loss analysis, and multiple ablation and sensitivity studies. Experimental results show that our proposed model can effectively improve the performances of existing ConvRNNs to the state-of-the-are level on representative public datasets, including Moving MNIST, KTH, BAIR, and WeatherBench. The ablation and sensitivity studies verify the superiority of CAU. The learned causal maps precisely distinguish the pixel attributions and motion characteristics in sophisticated entangled scenarios.

Causal augmented ConvNet: A temporal memory dilated convolution model for long-sequence time series prediction

CAU: A Causality Attention Unit for Spatial-temporal Sequence Forecast

Dual Attention-Based Temporal Convolutional Network for Fault Prognosis under Time-Varying Operating Conditions.

A Multivariate Time Series Prediction Method Based on Convolution-Residual Gated Recurrent Neural Network and Double-Layer Attention

Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting

Probabilistic Forecasting with Temporal Convolutional Neural Network

Temporal Self-Attention-based Conv-LSTM Network for Multivariate Time Series Prediction

Time Series Prediction Based on Temporal Convolutional Network

Adaptive Convolutional Forecasting Network Based on Time Series Feature-Driven

EffiCANet: Efficient Time Series Forecasting with Convolutional Attention

Machine remaining life prediction based on multi-layer self-attention and temporal convolution network

DDTCN: Decomposed dimension time-domain convolutional neural network along spatial dimensions for multiple long-term series forecasting

Multi-task spatio-temporal augmented net for industry equipment remaining useful life prediction

CDA-LSTM: an Evolutionary Convolution-Based Dual-Attention LSTM for Univariate Time Series Prediction

ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis

CTFNet: Long-Sequence Time-Series Forecasting Based on Convolution and Time–Frequency Analysis

Mobile Virtual Reality Rail Traffic Congestion Prediction Algorithm Based on Convolutional Neural Network

MCA-DTCN: A Novel Dual-Task Temporal Convolutional Network with Multi-Channel Attention for First Prediction Time Detection and Remaining Useful Life Prediction

TSCND: Temporal Subsequence-Based Convolutional Network with Difference for Time Series Forecasting

Remaining Useful Life Prediction of Aero-engine Via Temporal Convolutional Network with Gated Convolution and Channel Selection Unit

Causal-Transformer: Spatial-temporal Causal Attention-Based Transformer for Time Series Prediction