Abstract:Many micro-video related applications, such as personalized location recommendation and micro-video verification, can be benefited greatly from the venue information. Most existing works focus on integrating the information from multi-modal for exact venue category recognition. It is important to make full use of the information from different modalities. However, the performance may be limited by the lacked acoustic modality or textual descriptions in uploaded micro-videos. Therefore, in this paper visual modality is explored as the only modality according to its rich and indispensable semantic information. To this end, a hybrid-attention and frame difference enhanced network (HAFDN) is proposed to generate the comprehensive venue representation. Such network mainly contains two parallel branches: content and motion branches. Specifically, in the content branch, a domain-adaptive CNN model combined with temporal shift module (TSM) is employed to extract discriminative visual features. Then, a novel hybrid attention module (HAM) is introduced to enhance extracted features via three attention mechanisms. In HAM, channel attention, local and global spatial attention mechanisms are used to capture salient visual information from different views. In addition, convolutional Long Short-Term Memory (convLSTM) is enforced after HAM to better encode the long spatial-temporal dependency. A difference-enhanced module parallel with HAM is devised to learn the content variations among adjacent frames, which is usually ignored in prior works. Moreover, in the motion branch, 3D-CNNs and LSTM are used to capture movement variation as a supplement of content branch in a different form. Finally, the features from two branches are fused to generate robust video-level representations for predicting venue categories. Extensive experimental results on public datasets verify the effectiveness of the proposed micro-video venue recognition scheme. The source code is available at https://github.com/hs8945/HAFDN.

Joint Learning of NNeXtVLAD, CNN and Context Gating for Micro-Video Venue Classification.

Attention-enhanced Joint Learning Network for Micro-Video Venue Classification

Global and Compact Video Context Embedding for Video Semantic Segmentation

Hybrid-attention and frame difference enhanced network for micro-video venue recognition

Context-aware focal alignment network for micro-video multi-label classification

Enhancing Micro-video Understanding by Harnessing External Sounds

Towards Micro-video Understanding by Joint Sequential-Sparse Modeling

Attention-enhanced and trusted multimodal learning for micro-video venue recognition

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Enhancing Micro-Video Venue Recognition via Multi-Modal and Multi-Granularity Object Relations

Group Contextualization for Video Recognition

Non-local NetVLAD Encoding for Video Classification

Neural Multimodal Cooperative Learning Toward Micro-Video Understanding

Video Abnormal Event Detection by Learning to Complete Visual Cloze Tests

VMV-GCN: Volumetric Multi-View Based Graph CNN for Event Stream Classification

Online Video Instance Segmentation via Robust Context Fusion

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Context-aware network with foreground recalibration for grounding natural language in video

LEARNING-BASED MULTI-FRAME VIDEO QUALITY ENHANCEMENT

MC-Net: Integrating Multi-level Geometric Context for Two-view Correspondence Learning

Multi-Granularity Context Network for Efficient Video Semantic Segmentation