Abstract:In this paper, we present a novel end-to-end learning neural network, i.e., MATNet, for zero-shot video object segmentation (ZVOS). Motivated by the human visual attention behavior, MATNet leverages motion cues as a bottom-up signal to guide the perception of object appearance. To achieve this, an asymmetric attention block, named Motion-Attentive Transition (MAT), is proposed within a two-stream encoder network to firstly identify moving regions and then attend appearance learning to capture the full extent of objects. Putting MATs in different convolutional layers, our encoder becomes deeply interleaved, allowing for close hierarchical interactions between object apperance and motion. Such a biologically-inspired design is proven to be superb to conventional two-stream structures, which treat motion and appearance independently in separate streams and often suffer severe overfitting to object appearance. Moreover, we introduce a bridge network to modulate multi-scale spatiotemporal features into more compact, discriminative and scale-sensitive representations, which are subsequently fed into a boundary-aware decoder network to produce accurate segmentation with crisp boundaries. We perform extensive quantitative and qualitative experiments on four challenging public benchmarks, i.e., DAVIS 16 , DAVIS 17 , FBMS and YouTube-Objects. Results show that our method achieves compelling performance against current state-of-the-art ZVOS methods. To further demonstrate the generalization ability of our spatiotemporal learning framework, we extend MATNet to another relevant task: dynamic visual attention prediction (DVAP). The experiments on two popular datasets (i.e., Hollywood-2 and UCF-Sports) further verify the superiority of our model (our code is available at https://github.com/tfzhou/MATNet).

Learning Motion-Appearance Co-Attention for Zero-Shot Video Object Segmentation.

Motion-Attentive Transition for Zero-Shot Video Object Segmentation

MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation.

Co-attention Propagation Network for Zero-Shot Video Object Segmentation

Hierarchical Co-Attention Propagation Network for Zero-Shot Video Object Segmentation.

Zero-Shot Video Object Segmentation with Co-Attention Siamese Networks

Visual Attention Based Video Object Segmentation in MPEG Compressed Domain

Motion-Guided Spatial Time Attention for Video Object Segmentation.

See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks

Zero-Shot Monocular Motion Segmentation in the Wild by Combining Deep Learning with Geometric Motion Model Fusion

Efficient Unsupervised Video Object Segmentation Network Based on Motion Guidance

MoNet: Deep Motion Exploitation for Video Object Segmentation

Attention Embedded Spatio-Temporal Network for Video Salient Object Detection

Motion-Appearance Interactive Encoding for Object Segmentation in Unconstrained Videos

Enhanced Memory Network for Video Segmentation

Multiscale Visual-Attribute Co-Attention for Zero-Shot Image Recognition

Motion Guided Attention for Video Salient Object Detection.

Joint Attention Mechanism for Unsupervised Video Object Segmentation

MPASNET: Motion Prior-Aware Siamese Network for Unsupervised Deep Crowd Segmentation in Video Scenes

Multi-Content Interaction Network for Few-Shot Segmentation

Multi-motion Segmentation Via Co-attention-induced Heterogeneous Model Fitting