Abstract:The rapid growth of online video resources has significantly promoted the development of video retrieval methods. As a standard evaluation metric for video retrieval, Average Precision (AP) assesses the overall rankings of relevant videos at the top list, making the predicted scores a reliable reference for users. However, recent video retrieval methods utilize pair-wise losses that treat all sample pairs equally, leading to an evident gap between the training objective and evaluation metric. To effectively bridge this gap, in this work, we aim to address two primary challenges: a) The current similarity measure and AP-based loss are suboptimal for video retrieval; b) The noticeable noise from frame-to-frame matching introduces ambiguity in estimating the AP loss. In response to these challenges, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR). For the former challenge, we develop the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities in terms of AP. For the latter challenge, we suggest constraining the frame-level similarities to achieve an accurate AP loss estimation. Experimental results present that HAP-VR outperforms existing methods on several benchmark datasets, providing a feasible solution for video retrieval tasks and thus offering potential benefits for the multi-media application.

What problem does this paper attempt to address?

This paper attempts to solve two main problems in video retrieval: 1. **The current similarity measures and AP (Average Precision) loss functions are sub - optimal in video retrieval**: Existing methods usually use pairwise loss functions to train models. These loss functions treat all sample pairs equally and ignore the relative rankings between different sample pairs. This leads to a significant gap between the training objective and the evaluation metric. Specifically, as a standard evaluation metric, Average Precision (AP) focuses on the overall ranking of relevant videos in the top - list, while the existing pairwise loss functions only focus on bringing positive samples closer and pushing negative samples away, and cannot optimize AP well. 2. **Noise in frame - to - frame matching introduces ambiguity in AP loss estimation**: Due to the complex and redundant inter - frame relationships in videos, two related videos may not be completely related between all frame pairs. This inconsistency leads to false positives in frame - level matching easily without fine - grained annotations, thus affecting the accurate estimation of AP loss. To solve these problems, the authors propose a hierarchical learning framework HAP - VR (Hierarchical learning framework for Average - Precision - oriented Video Retrieval), which contains the following improvement measures: - **TopK - Chamfer Similarity**: It is used to measure and optimize video - level similarity, especially in terms of AP. Compared with the traditional max/mean aggregation methods, TopK - Chamfer Similarity retains more fine - grained information while filtering out false correlations. - **QuadLinear - AP loss function**: This is a new AP loss function that can provide sufficient gradients in the low - AP region, avoid the problem of vanishing gradients, and thus more effectively correct mis - ranked positive and negative sample pairs. - **Frame - level constraints**: Generate pseudo - labels through pre - trained visual models and distill frame - level information to reduce noise in frame - to - frame matching, thereby achieving more accurate AP loss estimation. Through these improvements, the HAP - VR framework can better bridge the gap between the training objective and the evaluation metric and improve the performance of video retrieval tasks. Experimental results show that HAP - VR outperforms existing methods on multiple benchmark datasets, especially in terms of the AP metric.

Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval

Learning User Interest with Improved Triplet Deep Ranking and Web-Image Priors for Topic-Related Video Summarization.

Hierarchical Average Precision Training for Pertinent Image Retrieval

A Supervised Video Hashing Method Based on a Deep 3D Convolutional Neural Network for Large-Scale Video Retrieval

Hierarchical Indexing Structure for Efficient Similarity Search in Video Retrieval

CLIPRerank: An Extremely Simple Method for Improving Ad-hoc Video Search

Effective Multiple Feature Hashing for Large-Scale Near-Duplicate Video Retrieval

Scalable Multimedia Retrieval By Deep Learning Hashing With Relative Similarity Learning

Are Binary Annotations Sufficient? Video Moment Retrieval Via Hierarchical Uncertainty-Based Active Learning

HFAG: Hierarchical Frame Affinity Group for Video Retrieval on Very Large Video Dataset

Multiple Feature Hashing for Real-Time Large Scale Near-Duplicate Video Retrieval

UATVR: Uncertainty-Adaptive Text-Video Retrieval

Mining Near-Duplicate Graph for Cluster-Based Reranking of Web Video Search Results

Robust Video-Text Retrieval via Noisy Pair Calibration

Clip-based Similarity Measure for Hierarchical Video Retrieval

Video quality assessment with dense features and ranking pooling

Joint Multi-View Hashing for Large-Scale Near-Duplicate Video Retrieval

Learn from Unlabeled Videos for Near-duplicate Video Retrieval

Neighborhood Preserving Hashing for Scalable Video Retrieval.

Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition