Abstract:Visual and audio signals are concurrent and complementary types of modality in some video actions. A single visual modality limits the performance of video action recognition due to the similar appearance with subtle movements, such as tapping guitar and playing guitar. However, it is challenging to fuse audio-visual modalities since the heterogeneity gap caused by inconsistent distribution and representation of multi-modal data. In this paper, we propose a local-to-global multi-modal interaction network (LGMI-Net) that integrates RGB, optical flow with sound information. First, for the local multi-modal interaction, we propose a novel inter-modal channel recalibration (IMCR) block to learn a joint representation from different input modalities by recalibrating the channel information distribution of one modality according to another modality. Besides, we also propose a novel RGB modality aggregation (RMA) block to obtain the more robust appearance features by mixing optical flow and sound information. Second, for the global multi-modal interaction, we propose three distinct encoder modes: unitary, parallel and triplet encoders, to capture the global multi-modal representation. The unitary encoder has the lowest computational complexity. The parallel encoder utilizes RGB-guided attention to improve accuracy while maintaining the lightweight. The triplet encoder aggregates the self-attentions of different modalities and achieves the best recognition performance. We implement extensive experiments on three public datasets: UCF101 subset, Kinetics-Sounds and EPIC-Kitchens-55. The results demonstrate the effectiveness of audio-visual complementation. Compared with the state-of-the-art multi-modal methods with sound (i.e. MM-ViT, AdaMML and G-Blend), the proposed LGMI-Net achieves superior accuracies of 96.05%, 88.37% and 50.3% with the 4.49×, 1.40× and 6.02× lower giga floating point operations (GFLOPs) respectively.

An Analysis of the Combination of Global and Local Models for Activity Recognition in Videos

Activity Recognition Using A Combination of Category Components And Local Models for Video Surveillance.

Hierarchical Complex Activity Representation and Recognition Using Topic Model and Classifier Level Fusion.

A Method of Simultaneously Action Recognition and Video Segmentation of Video Streams.

Global2Local: Efficient Structure Search for Video Action Segmentation

Global and Local Discriminative Patches Exploiting for Action Recognition

Local-Global Associative Frame Assemble in Video Re-ID

A Hybride Active Contour Model Driven by Global and Local Image Information

Localizing Activity Groups in Videos.

Learning Visual Context for Group Activity Recognition.

Localizing and recognizing action unit using position information of local feature

GLOCAL: A self-supervised learning framework for global and local motion estimation

Mpadal: a Joint Local-and-global Multi-View Feature Selection Method for Activity Recognition

Fusing $${\mathcal {R}}$$R Features and Local Features with Context-Aware Kernels for Action Recognition

Multi-scale residual network model combined with Global Average Pooling for action recognition

See, Move and Hear: a Local-to-global Multi-Modal Interaction Network for Video Action Recognition.

A Spatio-Temporal Integrated Model Based on Local and Global Features for Video Expression Recognition

Human Activity Recognition by Multi-Modal Context Fusion

Local and Global Alignments for Generalizable Sensor-Based Human Activity Recognition

AMIR

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition