Abstract:Action quality assessment (AQA) is to assess how well an action is performed. Previous works perform modelling by only the use of visual information, ignoring audio information. We argue that although AQA is highly dependent on visual information, the audio is useful complementary information for improving the score regression accuracy, especially for sports with background music, such as figure skating and rhythmic gymnastics. To leverage multimodal information for AQA, i.e., RGB, optical flow and audio information, we propose a Progressive Adaptive Multimodal Fusion Network (PAMFN) that separately models modality-specific information and mixed-modality information. Our model consists of with three modality-specific branches that independently explore modality-specific information and a mixed-modality branch that progressively aggregates the modality-specific information from the modality-specific branches. To build the bridge between modality-specific branches and the mixed-modality branch, three novel modules are proposed. First, a Modality-specific Feature Decoder module is designed to selectively transfer modality-specific information to the mixed-modality branch. Second, when exploring the interaction between modality-specific information, we argue that using an invariant multimodal fusion policy may lead to suboptimal results, so as to take the potential diversity in different parts of an action into consideration. Therefore, an Adaptive Fusion Module is proposed to learn adaptive multimodal fusion policies in different parts of an action. This module consists of several FusionNets for exploring different multimodal fusion strategies and a PolicyNet for deciding which FusionNets are enabled. Third, a module called Cross-modal Feature Decoder is designed to transfer cross-modal features generated by Adaptive Fusion Module to the mixed-modality branch.

Multi-Stage Contrastive Regression for Action Quality Assessment

Group-aware Contrastive Regression for Action Quality Assessment

Fine-Grained Spatio-Temporal Parsing Network for Action Quality Assessment

MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment

Deep Video Quality Assessment Using Constrained Multi-Task Regression and Spatio-temporal Feature Fusion.

Hierarchical Graph Convolutional Networks for Action Quality Assessment

Interpretable Long-term Action Quality Assessment

2M-AF: A Strong Multi-Modality Framework for Human Action Quality Assessment with Self-supervised Representation Learning

Learning and fusing multiple hidden substages for action quality assessment

Dual-referenced assistive network for action quality assessment

Assessing action quality with semantic-sequence performance regression and densely distributed sample weighting

Multimodal Action Quality Assessment

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Gaussian guided frame sequence encoder network for action quality assessment

End-to-end Action Quality Assessment with Action Parsing Transformer

CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment

A multi-scale self-supervised hypergraph contrastive learning framework for video question answering

Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling

Auto-Encoding Score Distribution Regression for Action Quality Assessment.

Disjoint Contrastive Regression Learning for Multi-Sourced Annotations

Contrastive Video Question Answering via Video Graph Transformer