Abstract:Recently, with the vigorous development of deep learning and multimedia technology, intelligent urban computing has received more and more extensive attention from academia and industry. Unfortunately, most of the related technologies are black-box paradigms that lack interpretability. Among them, video event recognition is a basic technology. Event contains multiple concepts and their rich interactions, which can assist us to construct explainable event recognition methods. However, the crucial concepts needed to recognize events have various temporal existing patterns, and the relationship between events and the temporal characteristics of concepts has not been fully exploited. This brings great challenges for concept-based event categorization. To address the above issues, we introduce the temporal concept receptive field, which is the length of the temporal window size required to capture key concepts for concept-based event recognition methods. Accordingly, we introduce the temporal dynamic convolution (TDC) to model the temporal concept receptive field dynamically according to different events. Its core idea is to combine the results of multiple convolution layers with the learned coefficients from two complementary perspectives. These convolution layers contain a variety of kernel sizes, which can provide temporal concept receptive fields of different lengths. Similarly, we also propose the cross-domain temporal dynamic convolution (CrTDC) with the help of the rich relationship between different concepts. Different coefficients can help us to capture suitable temporal concept receptive field sizes and highlight crucial concepts to obtain accurate and complete concept representations for event analysis. Based on the TDC and CrTDC, we introduce the temporal dynamic concept modeling network (TDCMN) for explainable video event recognition. We evaluate TDCMN on large-scale and challenging datasets FCVID, ActivityNet, and CCV. Experimental results show that TDCMN significantly improves the event recognition performance of concept-based methods, and the explainability of our method inspires us to construct more explainable models from the perspective of the temporal concept receptive field.

Temporal-Spatial Processing of Event Camera Data via Delay-Loop Reservoir Neural Network

Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis

TAM: Temporal Adaptive Module for Video Recognition

TEINet: Towards an Efficient Architecture for Video Recognition.

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Temporal-Spatial Mapping for Action Recognition

A novel spatio-temporal memory network for video anomaly detection

Temporal Dynamic Concept Modeling Network for Explainable Video Event Recognition

Spatio-temporal Transformers for Action Unit Classification with Event Cameras

TMAV: Temporal Motionless Analysis of Video using CNN in MPSoC

A Sparse Coding Multi-Scale Precise-Timing Machine Learning Algorithm for Neuromorphic Event-Based Sensors

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

SPADE-E2VID: Spatially-Adaptive Denormalization for Event-Based Video Reconstruction.

Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution

Bidirectional Multirate Reconstruction for Temporal Modeling in Videos

Multi-scale Spatial-temporal Interaction Network for Video Anomaly Detection

Temporal Modulation Network for Controllable Space-Time Video Super-Resolution

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Towards Neuro-Symbolic Video Understanding