RICA2: Rubric-Informed, Calibrated Assessment of Actions

Abrar Majeedi,Viswanatha Reddy Gajjala,Satya Sai Srinath Namburi GNVV,Yin Li

2024-08-07

Abstract:The ability to quantify how well an action is carried out, also known as action quality assessment (AQA), has attracted recent interest in the vision community. Unfortunately, prior methods often ignore the score rubric used by human experts and fall short of quantifying the uncertainty of the model prediction. To bridge the gap, we present RICA^2 - a deep probabilistic model that integrates score rubric and accounts for prediction uncertainty for AQA. Central to our method lies in stochastic embeddings of action steps, defined on a graph structure that encodes the score rubric. The embeddings spread probabilistic density in the latent space and allow our method to represent model uncertainty. The graph encodes the scoring criteria, based on which the quality scores can be decoded. We demonstrate that our method establishes new state of the art on public benchmarks, including FineDiving, MTL-AQA, and JIGSAWS, with superior performance in score prediction and uncertainty calibration. Our code is available at <a class="link-external link-https" href="https://abrarmajeedi.github.io/rica2_aqa/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to address two main issues in Action Quality Assessment (AQA):

1. **Ignoring the Scoring Rubric**: Existing methods often ignore the scoring rubric used by human experts, leading to insufficient accuracy in model predictions.
2. **Prediction Uncertainty**: Existing methods fail to effectively quantify the uncertainty of model predictions, i.e., the confidence of the model's predictions. This is particularly important in high-risk scenarios (such as sports competitions or surgical procedures) because understanding prediction uncertainty can help decide whether further evaluation by human experts is needed.

To address these shortcomings, the authors propose RICA2 (Rubric-Informed, Calibrated Assessment of Actions), a deeply integrated probabilistic model that incorporates the scoring rubric and quantifies prediction uncertainty. RICA2 encodes the scoring rubric through a graph structure and defines random embeddings on the graph to propagate probability densities in the latent space, representing the model's uncertainty. This approach not only improves prediction accuracy but also provides calibrated uncertainty estimates.

### Main Contributions

1. **Proposing RICA2**: A novel deep probabilistic method that combines scoring rubrics and uncertainty modeling to achieve accurate action scoring and calibrated uncertainty estimation.
2. **Technical Innovations**:
- Using Graph Neural Networks (GNN) to model the scoring rubric and defining random embeddings on the graph to quantify prediction uncertainty.
- Designing a training scheme under the variational information bottleneck framework.
3. **Experimental Validation**: Conducting experiments on multiple public AQA datasets (such as FineDiving, MTL-AQA, and JIGSAWS), showing that RICA2 significantly outperforms existing methods in both prediction accuracy and uncertainty calibration.

### Experimental Results

- **FineDiving Dataset**: RICA2 improves the Spearman Rank Correlation Coefficient (SRCC) and relative L2 distance (Rℓ2) by 0.7% and 1.4%, respectively, compared to the latest method TPT, and by 4.9% and 29.8%, respectively, compared to non-example methods (such as MUSDL and DAE-MT).
- **MTL-AQA Dataset**: RICA2 also significantly outperforms existing methods in SRCC and Kendall Tau, particularly excelling in uncertainty calibration.

### Conclusion

By integrating the scoring rubric and quantifying prediction uncertainty, RICA2 significantly improves the accuracy and reliability of action quality assessment, providing strong support for decision-making in high-risk scenarios.