Abstract:This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Temporal Coherent Video Segmentation with Support Vector Machine and Graph Cut

Motionobject segmentation algorithm based on spatial-temporal information

Detection and Segmentation of Moving Objects Using Temporal and Spatial Cues

Adaptive Video Segmentation Algorithm Using Hidden Conditional Random Fields

Vibration-resisted video segmentation algorithm based on hierarchical Markov random field model

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

An algorithm for automatic segmentation of video objects based on intra-frame image partition

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

Object Segmentation Using Graph Cuts for the H.264 Compressed Video with Moving Background

Sandwich Cut: an Algorithm for Temporally-Coherent Video Bilayer Segmentation

Real-time spatiotemporal segmentation of video objects in the H.264 compressed domain

Spatiotemporal segmentation for stereoscopic video

Spatiotemporal segmentation for compact video representation

Objective Performance Evaluation of Video Segmentation Algorithms with Ground-Truth

Semantic Video Segmentation Based on Region Boundary Motion Information

Dual Temporal Memory Network for Efficient Video Object Segmentation

Streaming Video Object Segmentation with the Adaptive Coherence Factor

User-Guided Clustering for Video Segmentation on Coarse-Grained Feature Extraction

Improved Video Object Segmentation Algorithm Based on Temporal-Spatial Information

Dual temporal memory network with high-order spatio-temporal graph learning for video object segmentation