Abstract:Multilingual pre-trained language models have achieved impressive results on most natural language processing tasks. However, the performance is inhibited due to capacity limitations and their under-representation of pre-training data, especially for languages with limited resources. This has led to the creation of tailored pre-trained language models, in which the models are pre-trained on large amounts of monolingual data or domain specific corpus. Nevertheless, compared to relying on multiple monolingual models, utilizing multilingual models offers the advantage of multilinguality, such as generalization on cross-lingual resources. To combine the advantages of both multilingual and monolingual models, we propose KDDA - a framework that leverages monolingual models to a single multilingual model with the aim to improve sentence representation for Vietnamese. KDDA employs teacher-student framework and cross-lingual transfer that aims to adopt knowledge from two monolingual models (teachers) and transfers them into a unified multilingual model (student). Since the representations from the teachers and the student lie on disparate semantic spaces, we measure discrepancy between their distributions by using Sinkhorn Divergence - an optimal transport distance. We conduct experiments on two Vietnamese natural language understanding tasks, including machine reading comprehension and natural language inference. Experimental results show that our model outperforms other state-of-the-art models and yields competitive performances.

Vietnamese Sentiment Analysis: An Overview and Comparative Study of Fine-tuning Pretrained Language Models

Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

Investigating Monolingual and Multilingual BERTModels for Vietnamese Aspect Category Detection

ViDeBERTa: A powerful pre-trained language model for Vietnamese

Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

A Systematic Literature Review on Vietnamese Aspect-based Sentiment Analysis

ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation

An Efficient Model for Sentiment Analysis of Electronic Product Reviews in Vietnamese

From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection

Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese

PhoBERT: Pre-trained language models for Vietnamese

Improving sentence representation for vietnamese natural language understanding using optimal transport

Building a Vietnamese Dataset for Natural Language Inference Models

ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

Multi-dimensional data refining strategy for effective fine-tuning LLMs

A Novel Pretrained General-Purpose Vision Language Model for the Vietnamese Language

Integrating Semantic Information into Sketchy Reading Module of Retro-Reader for Vietnamese Machine Reading Comprehension

BERT-VBD: Vietnamese Multi-Document Summarization Framework

ViANLI: Adversarial Natural Language Inference for Vietnamese

Augmenting Part-of-speech Tagging with Syntactic Information for Vietnamese and Chinese