Abstract:Many micro-video related applications, such as personalized location recommendation and micro-video verification, can be benefited greatly from the venue information. Most existing works focus on integrating the information from multi-modal for exact venue category recognition. It is important to make full use of the information from different modalities. However, the performance may be limited by the lacked acoustic modality or textual descriptions in uploaded micro-videos. Therefore, in this paper visual modality is explored as the only modality according to its rich and indispensable semantic information. To this end, a hybrid-attention and frame difference enhanced network (HAFDN) is proposed to generate the comprehensive venue representation. Such network mainly contains two parallel branches: content and motion branches. Specifically, in the content branch, a domain-adaptive CNN model combined with temporal shift module (TSM) is employed to extract discriminative visual features. Then, a novel hybrid attention module (HAM) is introduced to enhance extracted features via three attention mechanisms. In HAM, channel attention, local and global spatial attention mechanisms are used to capture salient visual information from different views. In addition, convolutional Long Short-Term Memory (convLSTM) is enforced after HAM to better encode the long spatial-temporal dependency. A difference-enhanced module parallel with HAM is devised to learn the content variations among adjacent frames, which is usually ignored in prior works. Moreover, in the motion branch, 3D-CNNs and LSTM are used to capture movement variation as a supplement of content branch in a different form. Finally, the features from two branches are fused to generate robust video-level representations for predicting venue categories. Extensive experimental results on public datasets verify the effectiveness of the proposed micro-video venue recognition scheme. The source code is available at https://github.com/hs8945/HAFDN.

Context-aware focal alignment network for micro-video multi-label classification

Joint Learning of NNeXtVLAD, CNN and Context Gating for Micro-Video Venue Classification.

Dual-domain Aligned Deep Hierarchical Matrix Factorization Method for Micro-video Multi-label Classification

SADCMF: Self-Attentive Deep Consistent Matrix Factorization for Micro-Video Multi-Label Classification

Fine-grained Cross-modal Alignment Network for Text-Video Retrieval

Hybrid-attention and frame difference enhanced network for micro-video venue recognition

Context Attention Fusion Network for crowd counting

Joint learning of video scene detection and annotation via multi-modal adaptive context network

Context-aware network with foreground recalibration for grounding natural language in video

Multimodal Deep Hierarchical Semantic-Aligned Matrix Factorization Method for Micro-Video Multi-Label Classification

Attention-enhanced Joint Learning Network for Micro-Video Venue Classification

Multimodal Attentive Representation Learning for Micro-video Multi-label Classification

Context-Based Adaptive Multimodal Fusion Network for Continuous Frame-Level Sentiment Prediction

Context Alignment Network for Video Moment Retrieval.

Group Contextualization for Video Recognition

Multi-scale Fusion Visual Attention Network for Facial Micro-Expression Recognition.

A Multimodal Aggregation Network with Serial Self-Attention Mechanism for Micro-Video Multi-Label Classification

Context and Spatial Feature Calibration for Real-Time Semantic Segmentation

User-Video Co-Attention Network for Personalized Micro-video Recommendation

Multi-attention Networks for Temporal Localization of Video-level Labels

Attention in Attention: Modeling Context Correlation for Efficient Video Classification