Abstract:Video super-resolution (VSR) is important in video processing for reconstructing high-definition image sequences from corresponding continuous and highly-related video frames. However, existing VSR methods have limitations in fusing spatial-temporal information. Some methods only fuse spatial-temporal information on a limited range of total input sequences, while others adopt a recurrent strategy that gradually attenuates the spatial information. While recent advances in VSR utilize Transformer-based methods to improve the quality of the upscaled videos, these methods require significant computational resources to model the long-range dependencies, which dramatically increases the model complexity. To address these issues, we propose a Collaborative Transformer for Video Super-Resolution (CTVSR). The proposed method integrates the strengths of Transformer-based and recurrent-based models by concurrently assimilating the spatial information derived from multi-scale receptive fields and the temporal information acquired from temporal trajectories. In particular, we propose a Spatial Enhanced Network (SEN) with two key components: Token Dropout Attention (TDA) and Deformable Multi-head Cross Attention (DMCA). TDA focuses on the key regions to extract more informative features, and DMCA employs deformable cross attention to gather information from adjacent frames. Moreover, we introduce a Temporal-trajectory Enhanced Network (TEN) that computes the similarity of a given token with temporally-related tokens in the temporal trajectory, which is different from previous methods that evaluate all tokens within the temporal dimension. With comprehensive quantitative and qualitative experiments on four widely-used VSR benchmarks, the proposed CTVSR achieves competitive performance with relatively low computational consumption and high forward speed.

CVSformer: Cross-View Synthesis Transformer for Semantic Scene Completion

SemiCVT: Semi-Supervised Convolutional Vision Transformer for Semantic Segmentation

Context and Geometry Aware Voxel Transformer for Semantic Scene Completion

Semantic scene completion with point cloud representation and transformer-based feature fusion

CVIformer: Cross-View Interactive Transformer for Efficient Stereoscopic Image Super-Resolution

Multi-View Clustering with Semantic Fusion and Contrastive Learning

Structure-Aware Cross-Modal Transformer for Depth Completion

CASSC: Context‐aware Method for Depth Guided Semantic Scene Completion

Anisotropic Convolutional Networks for 3D Semantic Scene Completion.

Multiview Fusion Driven 3-D Point Cloud Semantic Segmentation Based on Hierarchical Transformer

2D Semantic-Guided Semantic Scene Completion

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

Resolution-switchable 3D Semantic Scene Completion

LiDAR-Camera Continuous Fusion in Voxelized Grid for Semantic Scene Completion

PCSformer: Pair-wise Cross-scale Sub-prototypes Mining with CNN-transformers for Weakly Supervised Semantic Segmentation

Semantic Scene Completion with Cleaner Self

CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection

Unsupervised Semantic Scene Reconstruction via Transformer-Based Quantized Vector Reconstruction and Autoregressive Completion

CTVSR: Collaborative Spatial-Temporal Transformer for Video Super-Resolution

Multi-view Convolutional Vision Transformer for 3D Object Recognition.

Semantic Scene Completion Through Multi-Level Feature Fusion