Distant Supervised Relation Extraction on Pre-train Model with Improved Multi-label Attention Mechanism

Qiming Zhao,Chuantao Yin,Xin Fan,Hui Chen,Yanmei Chai,Ouyang Yuanxin

DOI: https://doi.org/10.1007/978-981-97-5492-2_24

2024-01-01

Abstract:Relation extraction serves as the cornerstone for numerous natural language processing tasks. Supervised methods necessitate manual data labeling, incurring significant costs, while unsupervised approaches often suffer from low precision. Consequently, distant supervised relation extraction has emerged as a hot spot. However, the underlying assumption of distant supervision relation extraction introduces considerable noise into data, markedly impairing performances. Therefore, this paper aims to mitigate the impact of noisy sentences and enhance overall performance. Herein, we proposed a modification to the conventional approach by employing a model integrating a piecewise convolutional neural network (PCNN) with a sentence-level attention mechanism as the baseline. This model comprises two components: a sentence encoder and an attention layer. For the sentence encoder, we substitute the PCNN with a pre-trained GPT model, leveraging its superior ability to capture sentence features. Additionally, we enhance the attention layer by utilizing all possible relations as queries to compute attention weights for sentences. Subsequently, we aggregate these weighted representations to obtain a comprehensive feature representation for various relations. Experimental results confirm the superior performance of the proposed model, which combines pre-trained Transformer-combined encoders with the refined sentence-level attention mechanism. Specially, employing Transformer as a sentence encoder yields significant precision improvements, particularly at high recall levels. Meanwhile, the enhanced multi-label sentence-level attention mechanism enhances precision, particularly in scenarios with low recall.