Abstract:Coreference resolution, an essential task in natural language processing, is particularly challenging in multi-modal scenarios where data comes in various forms and modalities. Despite advancements, limitations due to scarce labeled data and underleveraged unlabeled data persist. We address these issues with a self-adaptive fine-grained multi-modal data augmentation framework for semi-supervised MCR, focusing on enriching training data from labeled datasets and tapping into the untapped potential of unlabeled data. Regarding the former issue, we first leverage text coreference resolution datasets and diffusion models,to perform fine-grained text-to-image generation with aligned text entities and image bounding boxes. We then introduce a self-adaptive selection strategy, meticulously curating the augmented data to enhance the diversity and volume of the training set without compromising its quality. For the latter issue, we design a self-adaptive threshold strategy that dynamically adjusts the confidence threshold based on the model's learning status and performance, enabling effective utilization of valuable information from unlabeled data. Additionally, we incorporate a distance smoothing term, which smooths distances between positive and negative samples, enhancing discriminative power of the model?s feature representations and addressing noise and uncertainty in the unlabeled data. Our experiments on the widely-used CIN dataset show that our framework significantly outperforms state-of-the-art baselines by at least 9.57% on MUC F1 score and 4.92% on CoNLL F1 score. Remarkably, against weakly-supervised baselines, our framework achieves a staggering 22.24% enhancement in MUC F1 score. These results, underpinned by in-depth analyses, underscore the effectiveness and potential of our approach for advancing MCR tasks.

Fine-grained Adaptive Contrastive Learning for Unsupervised Feature Extraction

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Unified Feature Extraction Framework Based on Contrastive Learning

Fairness-aware Contrastive Learning with Partially Annotated Sensitive Attributes

Supervised Contrastive Attentive Learning for Facial Expression Recognition in the Wild

Feature Extraction Framework based on Contrastive Learning with Adaptive Positive and Negative Samples

Self-Supervised Global–Local Contrastive Learning for Fine-Grained Change Detection in VHR Images

Unified Framework for Feature Extraction based on Contrastive Learning

Self-supervised extracted contrast network for facial expression recognition

Self-supervised Contrastive Feature Refinement for Few-Shot Class-Incremental Learning.

An Efficient Method of Supervised Contrastive Learning for Natural Language Understanding

SCL-FExR: supervised contrastive learning approach for facial expression Recognition

Supervised contrastive learning with multi-scale interaction and integrity learning for salient object detection

Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples

Adversarial Supervised Contrastive Learning

CLAF: Contrastive Learning with Augmented Features for Imbalanced Semi-Supervised Learning

Combating Medical Label Noise via Robust Semi-supervised Contrastive Learning

Integrating Foreground–background Feature Distillation and Contrastive Feature Learning for Ultra-Fine-grained Visual Classification

Pose-disentangled Contrastive Learning for Self-supervised Facial Representation

Self-Adaptive Fine-grained Multi-modal Data Augmentation for Semi-supervised Muti-modal Coreference Resolution

Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding