AdapterEM: Pre-trained Language Model Adaptation for Generalized Entity Matching using Adapter-tuning

John Bosco Mugeni,Steven Lynden,Toshiyuki Amagasa,Akiyoshi Matono

DOI: https://doi.org/10.1145/3589462.3589498

2023-05-30

Abstract:Entity Matching (EM) involves identifying different data representations referring to the same entity from multiple data sources and is typically formulated as a binary classification problem. It is a challenging problem in data integration due to the heterogeneity of data representations. State-of-the-art solutions have adopted NLP techniques based on pre-trained language models (PrLMs) via the fine-tuning paradigm, however, sequential fine-tuning of overparameterized PrLMs can lead to catastrophic forgetting, especially in low-resource scenarios. In this study, we propose a parameter-efficient paradigm for fine-tuning PrLMs based on adapters, small neural networks encapsulated between layers of a PrLM, by optimizing only the adapter and classifier weights while the PrLMs parameters are frozen. Adapter-based methods have been successfully applied to multilingual speech problems achieving promising results, however, the effectiveness of these methods when applied to EM is not yet well understood, particularly for generalized EM with heterogeneous data. Furthermore, we explore using (i) pre-trained adapters and (ii) invertible adapters to capture token-level language representations and demonstrate their benefits for transfer learning on the generalized EM benchmark. Our results show that our solution achieves comparable or superior performance to full-scale PrLM fine-tuning and prompt-tuning baselines while utilizing a significantly smaller computational footprint $\approx 13\%$ of the PrLM parameters.

Computation and Language,Databases

What problem does this paper attempt to address?

The paper aims to address several key issues in Entity Matching (EM). Specifically:

1. **Data Representation Heterogeneity**: In the real world, data from different sources come in various formats (such as structured, semi-structured, or unstructured data), making it difficult for traditional entity matching methods to handle.

2. **Catastrophic Forgetting**: Pre-trained Language Models (PrLMs) tend to forget previously learned knowledge during fine-tuning, especially in resource-limited scenarios.

3. **High Storage Costs**: Full fine-tuning for each new task leads to significant storage overhead, as a complete model checkpoint needs to be saved for each task.

To address these issues, the authors propose an adapter-based method called AdapterEM. This method leverages small neural networks (i.e., adapters) to optimize specific parts of the pre-trained language model instead of fine-tuning the entire model. This approach not only reduces computational resource consumption but also effectively mitigates the problem of catastrophic forgetting and performs well in multiple benchmark tests. Experimental results show that under both low-resource and fully-resourced settings, this method outperforms or matches full fine-tuning in most tasks while significantly reducing computational overhead.