Abstract:The interactions between human and objects are important for recognizing object-centric actions. Existing methods usually adopt a two-stage pipeline, where object proposals are first detected using a pretrained detector, and then are fed to an action recognition model for extracting video features and learning the object relations for action recognition. However, since the action prior is unknown in the object detection stage, important objects could be easily overlooked, leading to inferior action recognition performance. In this paper, we propose an end-to-end object-centric action recognition framework that simultaneously performs Detection And Interaction Reasoning in one stage. Particularly, after extracting video features with a base network, we create three modules for concurrent object detection and interaction reasoning. First, a Patch-based Object Decoder generates proposals from video patch tokens. Then, an Interactive Object Refining and Aggregation identifies important objects for action recognition, adjusts proposal scores based on position and appearance, and aggregates object-level info into a global video representation. Lastly, an Object Relation Modeling module encodes object relations. These three modules together with the video feature extractor can be trained jointly in an end-to-end fashion, thus avoiding the heavy reliance on an off-the-shelf object detector, and reducing the multi-stage training burden. We conduct experiments on two datasets, Something-Else and Ikea-Assembly, to evaluate the performance of our proposed approach on conventional, compositional, and few-shot action recognition tasks. Through in-depth experimental analysis, we show the crucial role of interactive objects in learning for action recognition, and we can outperform state-of-the-art methods on both datasets.

Simultaneous Object Detection, Tracking, and Event Recognition

Event-based Moving Object Detection and Tracking

Structured Partial Least Squares for Simultaneous Object Tracking and Segmentation.

ADA-Track: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association

Scalable Detection and Tracking of Geometric Extended Objects

IOD-CNN: Integrating object detection networks for event recognition

Multi-object Tracking by Detection and Query: an efficient end-to-end manner

Tracking-Assisted Object Detection with Event Cameras

CORT: Class-Oriented Real-time Tracking for Embedded Systems

High-temporal-resolution event-based vehicle detection and tracking

Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognition

Multi-strategy object tracking in complex situation for video surveillance

Towards Unified 3D Object Detection via Algorithm and Data Unification

Detect to Track and Track to Detect

Real-time Multi-Object Tracking Based on Bi-directional Matching

Real-Time Online Multi-Object Tracking

Dual Integration of Multi‐model with Spatial‐temporal Occlusion‐awareness for Visual Object Tracking

Object Tracking over Multiple Uncalibrated Cameras Using Visual, Spatial and Temporal Similarities.

Robust Event-Based Object Tracking Combining Correlation Filter and CNN Representation

Detection-Free Multiobject Tracking by Reconfigurable Inference With Bundle Representations.

Tracking as A Whole: Multi-Target Tracking by Modeling Group Behavior with Sequential Detection