Abstract:Real-time monitoring and accurate prediction of key variables are indispensable to ensure industrial production activities proceed as expected. With the increase in measurement data volume and the improvement of hardware computing power, the Transformer and its variants, due to their excellent capability in extracting global dependencies, are playing an increasingly important role among deep learning-based multidimensional time series prediction models. In addition, from the perspective of causality, cause variables contain parts of information in effect variables and can reduce the uncertainty of effect variables, which is beneficial for prediction. However, there has been relatively limited research on combining the Transformer and causal feature analysis. To fully use both advantages, this paper introduces the Causal-Transformer (CT) model, which utilizes semi-orthogonal projection to extract causal features from multiple input variables. A multi-head spatial-temporal causal attention mechanism is designed in the encoder block based on the classical Transformer model to simultaneously reduce feature dimensions and extract implicit causal features in both the temporal and spatial dimensions. The CT also utilizes the Granger causality analysis to select the causal teaching indicators of target variables to provide stable assistance by injecting explicit causality into the inputs of the decoder block. By leveraging more condensed and independent causal features, the CT possesses inherent advantages in predicting time series variables. Case study results show that the CT model outperforms the other models on the diesel refinery dataset, especially with a reduction of 46.0% and 30.4% in MSE towards the classic Transformer and informer in five-step prediction. Copyright (C)2024 The Authors. This is an open access article under the CC BY-NC-ND license (htips://creativecommons.org/licenses/by-nc-nd/4.0/)

Causal-Transformer: Spatial-temporal Causal Attention-Based Transformer for Time Series Prediction

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

CausalFormer: An Interpretable Transformer for Temporal Causal Discovery

Causal augmented ConvNet: A temporal memory dilated convolution model for long-sequence time series prediction

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Transformers with Sparse Attention for Granger Causality

Itransformer: Inverted Transformers Are Effective for Time Series Forecasting

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

Dual Cross-Attention Transformer Networks for Temporal Predictive Modeling of Industrial Process

DAG-aware Transformer for Causal Effect Estimation

TCCT: Tightly-coupled convolutional transformer on time series forecasting

NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting

Attention for Causal Relationship Discovery from Biological Neural Dynamics

Spatial-Temporal Convolutional Transformer Network for Multivariate Time Series Forecasting

Causal inference of multivariate time series in complex industrial systems

Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

Multivariate Time Series Forecasting with Causal-Temporal Attention Network.

Short-Term and Long-Term Causal Effect Estimation with Double-Head Transformer

Are Self-Attentions Effective for Time Series Forecasting?

Large-scale Chemical Process Causal Discovery from Big Data with Transformer-Based Deep Learning