Abstract:With breakthroughs in pretrained language models, a large number of finetuned models specialized in distinct domains have surfaced online. Yet, when faced with a fresh dataset covering multiple (sub)domains, their performance might degrade. Reusing these available finetuned models to train a new model is a more feasible solution than the finetuning method that demands extensive manual labeling. Knowledge Amalgamation (KA) is such a model reusing technique, which derives a new model (termed student model) by amalgamating those trained models (termed teacher models) tailored for distinct domains, bypassing the need for manual labeling. However, when the domains of text samples are unknown, selecting a number of appropriate teacher models (simply called a combination) for reuse becomes complicated. To learn an accurate student model, the classical KA method resorts to manual selections, a process both tedious and inefficient. Our study pioneers the automation of this combination selection process for KA in the fundamental text classification task, an area previously unexplored. In this paper, we introduce BoKA : an automatic knowledge amalgamation framework for identifying a combination that can learn a superior student model without human labor. Through the lens of Bayesian optimization, BoKA iteratively samples a subset of possible combinations for amalgamation instead of manual selections. Furthermore, we introduce a novel KA method tailored for text classification, which guides the student model using both soft and pseudo-hard labels from the teacher models when their predictions are closely aligned; in cases of significant disagreement, it uses randomly generated labels. Experiments on two public multi-domain datasets show that BoKA achieves remarkable efficiency by sampling only up to 5.5% of all potential combinations. Moreover, BoKA is capable of matching or even surpassing leading zero-shot large language models, despite having dozens of times fewer parameters.

DA-BERT: Enhancing Knowledge Selection in Dialog via Domain Adapted BERT with Dynamic Masking Probability

An Improved Mask Approach Based on Pointer Network for Domain Adaptation of BERT

Improving Dialog System Grounded with Unstructured Knowledge by Domain Adaptive Pre-Training and Post-Ranking

BoKA: Bayesian Optimization Based Knowledge Amalgamation for Multi-unknown-domain Text Classification

K-BERT: Enabling Language Representation with Knowledge Graph

A Multi Domain Knowledge Enhanced Matching Network for Response Selection in Retrieval-Based Dialogue Systems

Knowledge Augmented Dialogue Generation with Divergent Facts Selection

Generative Knowledge Selection for Knowledge-Grounded Dialogues

Learning to Select Knowledge for Response Generation in Dialog Systems

Using Selective Masking as a Bridge between Pre-training and Fine-tuning

Knowledge As A Bridge: Improving Cross-domain Answer Selection with External Knowledge.

Bridging the Gap Between Prior and Posterior Knowledge Selection for Knowledge-Grounded Dialogue Generation

Improving knowledge-based dialogue generation through two-stage knowledge selection and knowledge selection-guided pointer network

Knowledgeable Salient Span Mask for Enhancing Language Models As Knowledge Base

Learning Multi-turn Response Selection in Grounded Dialogues with Reinforced Knowledge and Context Distillation

Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots

Policy Learning for Domain Selection in an Extensible Multi-domain Spoken Dialogue System

AdaDS: Adaptive Data Selection for Accelerating Pre-Trained Language Model Knowledge Distillation

E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

A Knowledge-Grounded Task-Oriented Dialogue System with Hierarchical Structure for Enhancing Knowledge Selection

Learning to share by masking the non-shared for multi-domain sentiment classification