Abstract:Contrastive learning has been demonstrated effective in unsupervised sentence representation learning. Given one sentence, positive pairs are obtained by passing the sentence to the encoder twice using the different dropout masks, and negative pairs are obtained by taking another sentence in the same mini-batch. However, the method suffers from the surface structure bias, i.e., sentences with similar surface structures will be regarded as close in semantics while sentences with dissimilar surface structures will be viewed as distinct in semantics. This leads to the result that paraphrasing a sentence that is dissimilar in surface structure will receive a lower semantic similarity score than inserting a negative word into the sentence. In this paper, we first verify the bias by collecting a sentence transformation testset. Then we systematically probe the existing models by proposing novel splits based on benchmark datasets in accordance with semantic and surface structure similarity. We tackle the bias in two aspects: balancing the learning target by augmenting with data that counters the bias, and meanwhile preserving word semantics by leveraging recall loss to prevent catastrophic forgetting. We evaluate our model on standard semantic textual similarity (STS) tasks using different pre-trained backbones and achieve state-of-the-art averaged performance across the STS benchmarks. Particularly, our models that are fine-tuned with RoBERTa base and RoBERTa large achieve significantly better performance on most benchmark datasets.

Mix Before Align: Towards Zero-shot Cross-lingual Sentiment Analysis Via Soft-Mix and Multi-View Learning

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Code-Switching Can be Better Aligners: Advancing Cross-Lingual SLU through Representation-Level and Prediction-Level Alignment

Enhancing Code-Switching for Cross-lingual SLU: A Unified View of Semantic and Grammatical Coherence

Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment

Cross-Lingual Propagation for Deep Sentiment Analysis

Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup

Improving Multi-lingual Alignment Through Soft Contrastive Learning

Learning Bilingual Embedding Model for Cross-Language Sentiment Classification

HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

AudioVSR: Enhancing Video Speech Recognition with Audio Data

MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition.

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

Cross-Modal Semantic Alignment before Fusion for Two-Pass End-to-End Spoken Language Understanding

Multi-label Masked Language Modeling on Zero-shot Code-switched Sentiment Analysis

Cross-Lingual Mixture Model for Sentiment Classification.

Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon