SS-CRE: A Continual Relation Extraction Method Through SimCSE-BERT and Static Relation Prototypes

Jinguang Chen,Suyue Wang,Lili Ma,Bo Yang,Kaibing Zhang

DOI: https://doi.org/10.1007/s11063-024-11647-4

IF: 2.565

2024-06-21

Neural Processing Letters

Abstract:Continual relation extraction aims to learn new relations from a continuous stream of data while avoiding forgetting old relations. Existing methods typically use the BERT encoder to obtain semantic embeddings, ignoring the fact that the vector representations suffer from anisotropy and uneven distribution. Furthermore, the relation prototypes are usually computed by memory samples directly, resulting in the model being overly sensitive to memory samples. To solve these problems, we propose a new continual relation extraction method. Firstly, we modified the basic structure of the sample encoder to generate uniformly distributed semantic embeddings using the supervised SimCSE-BERT to obtain richer sample information. Secondly, we introduced static relation prototypes and dynamically adjust their proportion with dynamic relation prototypes to adapt to the feature space. Lastly, through experimental analysis on the widely used FewRel and TACRED datasets, the results demonstrate that the proposed method effectively enhances semantic embeddings and relation prototypes, resulting in a further alleviation of catastrophic forgetting in the model. The code will be soon released at https://github.com/SuyueW/SS-CRE.

computer science, artificial intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of how to learn new relations continuously from a new data stream while avoiding forgetting old relations in the Continual Relation Extraction (CRE) task. Existing methods typically use BERT encoders to obtain semantic embeddings, but these embeddings have issues with anisotropy and uneven distribution. Additionally, relation prototypes are usually calculated directly from memory samples, making the model overly sensitive to these samples. To overcome these problems, the authors propose a new continual relation extraction method—SS-CRE (Supervised SimCSE-BERT and Static Relation Prototypes for Continual Relation Extraction). This method generates uniformly distributed semantic embeddings by improving the basic structure of the sample encoder and introduces static relation prototypes to adapt to the feature space, effectively alleviating the model's catastrophic forgetting problem.

Specifically, the main contributions of this method include:
1. Using supervised SimCSE-BERT as the basic structure of the encoder, the contrastive learning framework makes semantically similar embeddings closer and semantically dissimilar embeddings farther apart, addressing the issues of anisotropy and uneven distribution of semantic embeddings.
2. Introducing static relation prototypes and adjusting their ratio with dynamic relation prototypes to adapt to the feature space, reducing the dependence of relation prototypes on memory samples and alleviating the model's over-sensitivity to memory samples.
3. Conducting experimental analysis on the widely used FewRel and TACRED datasets, the results show that this method effectively improves semantic embeddings and relation prototypes, further alleviating the model's catastrophic forgetting problem.