Abstract:The Special type of videos refers to violent videos in this paper. Violent video dissemination has become one of the hidden dangers facing the Internet environment, and intelligent recognition technology for violent videos is of great significance for maintaining Internet content security. Due to the diversity of collection sources, the distribution of violent videos usually shows large intra-class variance and small inter-class variance, and it is difficult for common violence recognition frameworks to adapt to complex and variable violent scenarios. Meanwhile, the word violence itself has highly abstract semantics, and it becomes a major difficulty to learn a generic semantic representation of violence from limited data. In response to these problems, we present a novel multimodal violent video recognition model based on semantic embedding learning. The model mainly consists of the following three parts:（1） multimodal feature extraction. Considering that videos have multimodal properties, we use three different deep neural networks to extract feature representations of three modalities, i. e., appearance, motion, and audio.（2） multimodal feature fusion. To obtain a robust universal video representation, a lightweight multimodal feature fusion module, referred to as MEFM（Multimodal Efficient Fusion Module）, is designed in this paper. The module includes two parts: common space mapping and multimodal feature interaction, which can effectively suppress the interference between different modal information while fully interacting with multimodal features.（3） Semantic embedding learning. To accommodate violence datasets from different sources, we propose a multi-task learning method based on semantic embedding, which computes the semantic center of violence by introducing a center loss and uses cosine embedding loss to aggregate violent samples toward the center while discrete with non-violent samples to form a semantic discriminative feature representation, thus enhancing the generalization ability of the model and reducing the noise interference. Experiments on three publicly available datasets, VSD2015, Violent Flows, and RWF-2000, demonstrate that the violence video recognition framework proposed in this paper achieves competitive results by improving 4.79%, 0.81%, and 1.5% respectively, over the state of the arts.

Real-Time Recognition of Explosion Scenes Based on Audio-Visual Hierarchical Model

Vibration-resisted video segmentation algorithm based on hierarchical Markov random field model

Vision Based Flame Detection Using Compressed Domain Motion Prediction and Multi-Feature Fusion.

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Real-time multi-feature based fire flame detection in video.

Explosion Detection Using Smartphones: Ensemble Learning with the Smartphone High-Explosive Audio Recordings Dataset and the ESC-50 Dataset

Special Video Recognition Based on Semantic Embedding Learning Chinese Full Text

Multi-scale Harmonic Mean Time Surfaces for Event-based Object Classification

Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

The Shanghai-Hongkong Team at MediaEval2012: Violent Scene Detection Using Trajectory-based Features.

Insulator detection and recognition of explosion based on convolutional neural networks

Audio–visual representation learning for anomaly events detection in crowds

Sound Identification Method for Gas and Coal Dust Explosions Based on MLP

Classification of Hyperspectral Images of Explosive Fragments Based on Spatial–Spectral Combination

Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space

Detecting Violent Scenes in Movies by Auditory and Visual Cues.

Multimodal Variational Auto-encoder based Audio-Visual Segmentation

Audio-Visual Keyword Spotting Based on Multidimensional Convolutional Neural Network

Classification method of infrasound events based on the MVIDA algorithm and MS-SE-ResNet

Enhance audio-visual segmentation with hierarchical encoder and audio guidance

Robust Audiovisual Speech Recognition Models with Mixture-of-Experts