Abstract:Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on simple yet effective spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of current VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 - 5; mAP while running at around 30 FPS on a single V100 GPU device. Code and models are available at https://github.com/SJTU-LuHe/TransVOD.

Convolutional LSTM Based Video Object Detection.

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Plug & Play Convolutional Regression Tracker for Video Object Detection

Object Detection from Video Tubelets with Convolutional Neural Networks

Real-Time and Accurate Object Detection in Compressed Video by Long Short-term Feature Aggregation

Tracking Assisted Faster Video Object Detection

Detect to Track and Track to Detect

Compression Neuritis of Both Median Nerves in the Carpal Tunnel: Surgical Decompression. Klippel-Feil Deformity of Neck. Sprengel Shoulder

TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers

Convolutional LSTM networks for video-based person re-identification

Video Object Tracking Via Online Convolutional Network

Object-Adaptive LSTM Network for Real-time Visual Tracking with Adversarial Data Augmentation

Fast Object Detection in Compressed Video

Pyramid Dilated Deeper Convlstm For Video Salient Object Detection

Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos

Video Object Segmentation with 3D Convolution Network

Context-LSTM: a robust classifier for video detection on UCF101

Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Robust Visual Tracking via Convolutional Networks

Motion-Aware Memory Network for Fast Video Salient Object Detection

Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM