Abstract:Video prediction presents a formidable challenge, requiring effectively processing spatial and temporal information embedded in videos. While recurrent neural network (RNN) and transformer-based models have been extensively explored to address spatial changes over time, recent advancements in convolutional neural networks (CNNs) have yielded high-performance video prediction models. CNN-based models offer advantages over RNN and transformer-based models due to their ease of parallel processing and lower computational complexity, highlighting their significance in practical applications. However, existing CNN-based video prediction models typically treat the spatiotemporal channels of videos similarly to the channel axis of static images. They stack frames in temporal order to construct a spatiotemporal axis and employ standard convolution operations. Nevertheless, this approach has its limitations. Applying convolution directly to the spatiotemporal axis results in a mixture of temporal and spatial information, which may lead to computational inefficiencies and reduced accuracy. Additionally, this operation needs to improve in processing temporal data. This study introduces a CNN-based time series decomposition model for video prediction. The proposed model first divides the convolution operation within the channel aggregation module to independently process the temporal and spatial dimensions. To capture evolving features, the temporal axis is segregated into trend and residual components, followed by applying a time series decomposition forecasting method. To assess the performance of the proposed technique, experiments were conducted using the moving MNIST, KTH, and KITTI-Caltech benchmark datasets. In the experiments on moving MNIST, despite a reduction of approximately 55% in the number of parameters and 37% in computational cost, the proposed method improved accuracy by up to 7% compared to the previous approach.

Robust Temporal-Spatial Decomposition and Its Applications in Video Processing

Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Efficiently Exploiting Spatially Variant Knowledge for Video Deblurring

Spatiotemporal segmentation for compact video representation

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

Robust Super Resolution of Compressed Video

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Robust moving object segmentation in the compressed domain for H.264/AVC video stream

Real-time spatiotemporal segmentation of video objects in the H.264 compressed domain

Working memory inspired hierarchical video decomposition with transformative representations

Low-Light Video Enhancement via Spatial-Temporal Consistent Illumination and Reflection Decomposition

Intrinsic video and applications

Spatiotemporal Super-Resolution Reconstruction Based on Robust Optical Flow and Zernike Moment for Video Sequences

A Robust Video Super-Resolution Algorithm.

A time-slice optimization based weak feature association algorithm for video condensation

CNN-Based Time Series Decomposition Model for Video Prediction

Dynamic Mode Decomposition For Robust Pca With Applications To Foreground/Background Subtraction In Video Streams And Multi-Resolution Analysis

Depth prior-based stable tensor decomposition for video snow removal

Video Denoising Using Shape-Adaptive Sparse Representation over Similar Spatio-Temporal Patches

Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time