Abstract:With breakthroughs in pretrained language models, a large number of finetuned models specialized in distinct domains have surfaced online. Yet, when faced with a fresh dataset covering multiple (sub)domains, their performance might degrade. Reusing these available finetuned models to train a new model is a more feasible solution than the finetuning method that demands extensive manual labeling. Knowledge Amalgamation (KA) is such a model reusing technique, which derives a new model (termed student model) by amalgamating those trained models (termed teacher models) tailored for distinct domains, bypassing the need for manual labeling. However, when the domains of text samples are unknown, selecting a number of appropriate teacher models (simply called a combination) for reuse becomes complicated. To learn an accurate student model, the classical KA method resorts to manual selections, a process both tedious and inefficient. Our study pioneers the automation of this combination selection process for KA in the fundamental text classification task, an area previously unexplored. In this paper, we introduce BoKA : an automatic knowledge amalgamation framework for identifying a combination that can learn a superior student model without human labor. Through the lens of Bayesian optimization, BoKA iteratively samples a subset of possible combinations for amalgamation instead of manual selections. Furthermore, we introduce a novel KA method tailored for text classification, which guides the student model using both soft and pseudo-hard labels from the teacher models when their predictions are closely aligned; in cases of significant disagreement, it uses randomly generated labels. Experiments on two public multi-domain datasets show that BoKA achieves remarkable efficiency by sampling only up to 5.5% of all potential combinations. Moreover, BoKA is capable of matching or even surpassing leading zero-shot large language models, despite having dozens of times fewer parameters.

An Approach Based on Domain Knowledge to Text Categorization

Text Categorization Based on Domain Ontology

Knowledge-based Document Embedding for Cross-Domain Text Classification

BoKA: Bayesian Optimization Based Knowledge Amalgamation for Multi-unknown-domain Text Classification

An Approach to Automatic Text Categorization Based on Combination of Rule and Bayesian Method

Hybrid Chinese Text Classification Approach Using General Knowledge from Baidu Baike

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

Chinese Text Categorization Without Word Segmentation Using String Kernel

Dimensionality Reduction With Category Information Fusion And Non-Negative Matrix Factorization For Text Categorization

Chinese Short-Text Categorization Based on the Key Classification Dictionary Words

A Rough Set-Based Hybrid Method to Text Categorization

A Feature Selection Method Based on Class Feature Domains for Text Categorization

Incorporating Domain Knowledge into Text Classification Diagnosis in Customer Service Dialogue Field

A New Approach of Feature Selection for Text Categorization

An Enhanced Text Categorization Method Based on Improved Text Frequency Approach and Mutual Information Algorithm

Improved KNN Text Categorization

Knowledge Transfer Based on Feature Representation Mapping for Text Classification

Context Based Feature Description Model in Chinese Text Categorization

Associative Rule-Based Text Categorization Method Using Category Similarity

A Text Categorization Method Based on Features Clustering

Domain and Language Independent Feature Extraction for Statistical Text Categorization