Abstract:Generic Boundary Detection (GBD) aims at locating the general boundaries that divide videos into semantically coherent and taxonomy-free units, and could serve as an important pre-processing step for long-form video understanding. Previous works often separately handle these different types of generic boundaries with specific designs of deep networks from simple CNN to LSTM. Instead, in this paper, we present Temporal Perceiver , a general architecture with Transformer, offering a unified solution to the detection of arbitrary generic boundaries, ranging from shot-level, event-level, to scene-level GBDs. Our core design is to introduce a small set of latent feature queries as anchors to compress the redundant video input into a fixed dimension via cross-attention blocks. Thanks to this fixed number of latent units, it reduces the quadratic complexity of attention operation to a linear form of input frames. Specifically, to explicitly leverage the temporal structure of videos, we construct two types of latent feature queries: boundary queries and context queries, which handle the semantic incoherence and coherence accordingly. Moreover, to guide the learning of latent feature queries, we propose an alignment loss on the cross-attention maps to explicitly encourage the boundary queries to attend on the top boundary candidates. Finally, we present a sparse detection head on the compressed representation, and directly output the final boundary detection results without any post-processing module. We test our Temporal Perceiver on a variety of GBD benchmarks. Our method obtains the state-of-the-art results on all benchmarks with RGB single-stream features: SoccerNet-v2 (81.9 percent average-mAP), Kinetics-GEBD (86.0 percent average-f1), TAPOS (73.2 percent average-f1), MovieScenes (51.9 percent AP and 53.1 percent $M_{iou}$ ) and MovieNet (53.3 percent AP and 53.2 percent $M_{iou}$ ), demonstrating the generalization ability of our Temporal Perceiver. To further pursue a general GBD model, we combine various tasks to train a class-agnostic Temporal perceiver and evaluate its performance across all benchmarks. Results show that the class-agnostic Perceiver achieves comparable detection accuracy but better generalization ability compared to dataset-specific counterparts.

Boundary-sensitive Pre-training for Temporal Localization in Videos

Temporal Perceiving Video-Language Pre-training

Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos

Unsupervised Pre-training for Temporal Action Localization Tasks

Boundary Proposal Network for Two-Stage Natural Language Video Localization

Boundary-aware Self-supervised Learning for Video Scene Segmentation

Contrastive Language-Action Pre-training for Temporal Localization

Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization

Boundary-Aware Proposal Generation Method for Temporal Action Localization

Advancing Temporal Action Localization with a Boundary Awareness Network

Context-aware Biaffine Localizing Network for Temporal Sentence Grounding

Training-free Video Temporal Grounding using Large-scale Pre-trained Models

Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection.

BID: Boundary-Interior Decoding for Unsupervised Temporal Action Localization Pre-Trainin

No-frills Temporal Video Grounding: Multi-Scale Neighboring Attention and Zoom-in Boundary Detection

Improving Weakly Supervised Temporal Action Localization by Bridging Train-Test Gap in Pseudo Labels

Temporal Textual Localization in Video Via Adversarial Bi-Directional Interaction Networks

SCALE MATTERS: TEMPORAL SCALE AGGREGATION NETWORK FOR PRECISE ACTION LOCALIZATION IN UNTRIMMED VIDEOS

Boundary-Denoising for Video Activity Localization

Learning Salient Boundary Feature for Anchor-free Temporal Action Localization

Dual Masked Modeling for Weakly-Supervised Temporal Boundary Discovery