Abstract:Joint detection and tracking, which solves two fundamental vision challenges in a unified manner, is a challenging topic in computer vision. In this area, the proper use of spatial-temporal information in videos can help reduce local defects and improve the quality of feature representations. Although modeling low-level (usually pixel-wise) spatial-temporal information has been studied, instance-level spatial-temporal correlations (i.e., relations between semantic regions in which instances have occurred) have not been fully exploited. In comparison, modeling instance-level correlation is a more flexible and reasonable way to enhance feature representations. However, we have found that conventional instance-level relation learning that works for the separate tasks of detection or tracking is not effective in joint tasks in which a variety of scenarios may be presented. To try to resolve this problem, in this study, we effectively exploited instance-level spatial-temporal semantic information for joint detection and tracking via a joint relation learning pipeline with a novel relation learning mechanism called Similarity- and Quality-Guided Attention (SQGA). Specifically, we added task-specific SQGA relation modules before the corresponding task prediction heads to refine the instance feature representation using features of other reference instances in the neighboring frames; these features are aggregated on the basis of relational affinities. In particular, in SQGA, relational affinities were factorized to similarity and quality terms so that fine-grained supervision rules could be applied. Then we added task-specific attention losses for each SQGA relation module, resulting in a better feature aggregation for the corresponding task. Quantitative experiments based on several challenging multi-object tracking benchmarks showed that our approach was more effective than the baselines and provided competitive results compared with recent state-of-the-art methods.

Interactive Activity Learning from Trajectories with Qualitative Spatio-Temporal Relation

Qualitative Prediction of Multi-Agent Spatial Interactions

Spatio-Temporal Triangular-Chain Crf For Activity Recognition

Spatial-temporal activity interactions detection in video survalliance

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Activity Recognition Using Dense Long-Duration Trajectories

Modeling Scene-Agent Interaction for Pedestrian Trajectory Prediction

Deep Representation Learning of Activity Trajectory Similarity Computation.

Representation Learning With Multi-Level Attention for Activity Trajectory Similarity Computation

Recognizing Pair-Activities by Causality Analysis

Learning Spatiotemporal Interactions for User-Generated Video Quality Assessment

A Study of Relative Motion Point Trajectories for Action Recognition

Learning event representation: As sparse as possible, but not sparser

SSL-Interactions: Pretext Tasks for Interactive Trajectory Prediction

Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction

Similarity- and Quality-Guided Relation Learning for Joint Detection and Tracking

Spatio-Temporal-Interaction Graph Neural Networks for Multi-Agent Trajectory Prediction

A Trajectory Calculus for Qualitative Spatial Reasoning Using Answer Set Programming

Inferring Human Interaction from Motion Trajectories in Aerial Videos.

LG-LSTM: Modeling LSTM-Based Interactions for Multi-Agent Trajectory Prediction

VISUALIZATION OF SPATIO-TEMPORAL RELATIONS IN MOVEMENT EVENT USING MULTI-VIEW