Abstract:Unsupervised domain adaptation (UDA) aims to learn transferable representation across domains. Recently a few UDA works have successfully applied Transformer-based methods and achieved state-of-the-art (SOTA) results. However, it remains challenging when there exists a large domain gap between the source and target domain. Inspired by humans' exceptional transferability abilities to adapt knowledge from familiar to uncharted domains, we try to apply the universally existing organizational structure in the human functional brain networks, i.e., the core-periphery principle to design the Transformer and improve its UDA performance. In this paper, we propose a novel brain-inspired robust core-periphery constrained transformer (RCCT) for unsupervised domain adaptation, which brings a large margin of performance improvement on various datasets. Specifically, in RCCT, the self-attention operation across image patches is rescheduled by an adaptively learned weighted graph with the Core-Periphery structure (CP graph), where the information communication and exchange between images patches are manipulated and controlled by the connection strength, i.e., edge weight of the learned weighted CP graph. Besides, since the data in domain adaptation tasks can be noisy, to improve the model robustness, we intentionally add perturbations to the patches in the latent space to ensure generating robust learned weighted core-periphery graphs. Extensive evaluations are conducted on several widely tested UDA benchmarks. Our proposed RCCT consistently performs best compared to existing works, including 88.3\% on Office-Home, 95.0\% on Office-31, 90.7\% on VisDA-2017, and 46.0\% on DomainNet.

Domain Adaptive Multi-Task Transformer for Low-Resource Machine Reading Comprehension

Feeding What You Need by Understanding What You Learned

A novel multi-domain machine reading comprehension model with domain interference mitigation

Adversarial Domain Adaptation for Machine Reading Comprehension

Effective Strategies for Low-Resource Reading Comprehension

Robust Core-Periphery Constrained Transformer for Domain Adaptation

Curriculum Learning Driven Domain Adaptation for Low-Resource Machine Reading Comprehension

Exploiting Diverse Information in Pre-Trained Language Model for Multi-Choice Machine Reading Comprehension

R-Trans: RNN Transformer Network for Chinese Machine Reading Comprehension.

CAT-BERT: A Context-Aware Transferable BERT Model for Multi-turn Machine Reading Comprehension.

MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension

Multi-Task Deep Neural Networks for Multi-Document Reading Comprehension

A Multi-Task Learning Machine Reading Comprehension Model for Noisy Document (student Abstract)

Continual Domain Adaptation for Machine Reading Comprehension

An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension

Multi-Stage Pre-training for Low-Resource Domain Adaptation

Enhancing Transformer-based language models with commonsense representations for knowledge-driven machine comprehension

DUMA: Reading Comprehension with Transposition Thinking

NER-to-MRC: Named-Entity Recognition Completely Solving as Machine Reading Comprehension

A Multi-task Approach for Machine Reading Comprehension Form Named Entity Recognition Tasks

Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination