Abstract:Cross-modal noise-robust learning is a challenging task since noisy correspondence is hard to recognize and rectify. Due to the cumulative and unavoidable negative impact of unresolved noise, existing methods cannot maintain a stable performance when the noise increases. In this paper, we present a novel Negative Pre-aware Cross-modal (NPC) matching solution for large visual-language model fine-tuning on noisy downstream tasks. It is featured in two aspects: (1) For noise recognition and resistance, previous methods usually directly filter out a noise subset, we propose to estimate the negative impact of each sample. It does not need additional correction mechanisms that may predict unreliable correction results, leading to self-reinforcing error. We assign a confidence weight to each sample according to its negative impact in the training process. This adaptively adjusts the contribution of each sample to avoid noisy accumulation. (2) For maintaining stable performance with increasing noise, we utilize the memorization effect of DNNs by maintaining a memory bank. Specifically, we apply GMM to select high-confident clean samples as the memory entry, where the memory entry is used to estimate the negative impact of each sample. Since clean samples are easier distinguished by GMM with increasing noise, the memory bank can still maintain high quality at a high noise ratio. Compared to the correction mechanism focusing on noise samples, memory bank-based estimation is more robust, which makes the model performance stable on noisy datasets. Extensive experiments demonstrate that our method significantly improves matching accuracy and performance stability at increasing noise ratio. Our approach also surpasses the state-of-the-art methods by a large margin. The code is available at: https://github.com/ZhangXu0963/NPC.

Learning with Noisy Correspondence

Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation

NAC: Mitigating Noisy Correspondence in Cross-Modal Matching Via Neighbor Auxiliary Corrector.

Neighborhood Collective Estimation for Noisy Label Identification and Correction

Cross-modal Active Complementary Learning with Self-refining Correspondence

Deep Evidential Learning with Noisy Correspondence for Cross-modal Retrieval

Noisy Correspondence Learning with Meta Similarity Correction

Partially View-aligned Representation Learning with Noise-robust Contrastive Loss

Cross-Modal Retrieval With Noisy Correspondence via Consistency Refining and Mining

Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning

Learning with Neighbor Consistency for Noisy Labels

Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning

Disentangled Noisy Correspondence Learning

Learning From Noisy Correspondence With Tri-Partition for Cross-Modal Matching

Negative Pre-aware for Noisy Cross-modal Matching

UGNCL: Uncertainty-Guided Noisy Correspondence Learning for Efficient Cross-Modal Matching

Learning With Noisy Labels Over Imbalanced Subpopulations

A Joint Training Framework for Learning with Noisy Labels

Label Correction Using Contrastive Prototypical Classifier for Noisy Label Learning.

Robust Object Re-identification with Coupled Noisy Labels

Robust Noisy Correspondence Learning with Equivariant Similarity Consistency