Abstract:Although deep-learning methods can achieve human-level performance in boundary detection, their improvements mostly rely on larger models and specific datasets, leading to significant computational power consumption. As a fundamental low-level vision task, a single model with fewer parameters to achieve cross-dataset boundary detection merits further investigation. In this study, a lightweight universal boundary detection method was developed based on convolution and a transformer. The network is called a "transformer with difference convolutional network" (TDCN), which implies the introduction of a difference convolutional network rather than a pure transformer. The TDCN structure consists of three parts: convolution, transformer, and head function. First, a convolution network fused with edge operators is used to extract multiscale difference features. These pixel difference features are then fed to the hierarchical transformer as tokens. Considering the intrinsic characteristics of the boundary detection task, a new boundary-aware self-attention structure was designed in the transformer to provide inductive bias. By incorporating the proposed attention loss function, it introduces the direction of the boundary as strongly supervised information to improve the detection ability of the model. Finally, several head functions with multiscale feature inputs were trained using a bidirectional additive strategy. In the experiments, the proposed method achieved competitive performance on multiple public datasets with fewer model parameters. A single model was obtained to realize universal prediction even for different datasets without retraining, demonstrating the effectiveness of the method. The code is available at https://github.com/neulmc/TDCN.

ConvTransformer Attention Network for temporal action detection

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection

Non-Local Temporal Difference Network for Temporal Action Detection

Introducing Gating and Context into Temporal Action Detection

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Temporal Action Localization with Enhanced Instant Discriminability

Convolutional transformer network for fine-grained action recognition

Temporal Transformer Networks with Self-Supervision for Action Recognition.

A Multi-Modal Transformer Network for Action Detection

Harnessing Temporal Causality for Advanced Temporal Action Detection

ContextDet: Temporal Action Detection with Adaptive Context Aggregation

TDN: Temporal Difference Networks for Efficient Action Recognition

Faster-TAD: Towards Temporal Action Detection with Proposal Generation and Classification in a Unified Network

Transformer with difference convolutional network for lightweight universal boundary detection

LGAFormer: transformer with local and global attention for action detection

Boundary-Recovering Network for Temporal Action Detection

Adapting Short-Term Transformers for Action Detection in Untrimmed Videos

Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation

A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection

Temporal Context Aggregation Network for Temporal Action Proposal Refinement