Abstract:Many real-world applications require precise and fast time-series forecasting. Recent trends in time-series forecasting models are shifting from LSTM-based models to Transformer-based models. However, the Transformer-based model has a limited ability to represent sequential relationships in time-series data. In addition, the transformer-based model suffers from slow training and inference speed due to the bottleneck incurred by a deep encoder and step-by-step decoder inference. To address these problems, we propose a time-series forecasting optimized Transformer model, called TS-Fastformer. TS-Fastformer introduces three new optimizations: First, we propose a Sub Window Tokenizer for compressing input in a simple manner. The Sub Window Tokenizer reduces the length of input sequences to mitigate the complexity of self-attention and enables both single and multi-sequence learning. Second, we propose Time-series Pre-trained Encoder to extract effective representations through pre-training. This optimization enables TS-Fastformer to capture both seasonal and trend representations as well as to mitigate bottlenecks of conventional transformer models. Third, we propose the Past Attention Decoder to forecast target by incorporating past long short-term dependency patterns. Furthermore, Past Attention Decoder achieves high performance improvement by removing a trend distribution that changes over a long period. We evaluate the efficiency of our model with extensive experiments using seven real-world datasets and compare our model to six representative time-series forecasting approaches. The results show that the proposed TS-Fastformer reduces MSE by 10.1% compared to state-of-the-art model, and demonstrates 21.6% faster training time compared to the existing fastest transformer respectively.

Fastformer: Transformer-Based Fast Reasoning Framework

DFAformer: A Dual Filtering Auxiliary Transformer for Efficient Online Action Detection in Streaming Videos.

ViTframe: Vision Transformer Acceleration Via Informative Frame Selection for Video Recognition

Fastformer: Additive Attention Can Be All You Need

FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization

Optimal Topology of Vision Transformer for Real-Time Video Action Recognition in an End-To-End Cloud Solution

Dynamic Inference: A New Approach Toward Efficient Video Action Recognition

A Dynamic Frame Selection Framework for Fast Video Recognition.

Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling

Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition

Efficient Video Transformers via Spatial-Temporal Token Merging for Action Recognition

Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

TALLFormer: Temporal Action Localization with a Long-memory Transformer

FASTER Recurrent Networks for Efficient Video Classification

Factorization Vision Transformer: Modeling Long Range Dependency with Local Window Cost

SlowFast Networks for Video Recognition

Efficient Action Recognition with Introducing R(2+1)D Convolution to Improved Transformer

TP-VIT: A Two-Pathway Vision Transformer for Video Action Recognition

TS-Fastformer: Fast Transformer for Time-Series Forecasting

Lightweight Vision Transformer with Bidirectional Interaction

LS-VIT: Vision Transformer for action recognition based on long and short-term temporal difference