Abstract:Pre-trained language models (PLMs) have achieved noticeable success on a variety of natural language processing tasks, such as sequence labeling. In particular, the existing sequence labeling methods fine-tune PLMs on large-scale labeled data, which can avoid training the sequence labeling models from scratch. The fine-tuning process still requires large amounts of labeled training data so as to be effective. However, obtaining rich annotated data for sequence labeling is a time-consuming and expensive process, creating a substantial barrier for directly applying the PLMs trained on general-purpose large-scale text data to sequence labeling. In this paper, we investigate sequence labeling tasks from a novel perspective and propose a general framework that uses labeled clue sentences to mitigate the problem of insufficient annotation data for sequence labeling. Specifically, we first retrieve the labeled clue sentences for each original sentence in the training set based on the semantic (or syntactic) relevance. Here, the number of annotated clue sentences determines the expansion degree of the training set. Then, we modify the transformer’s self-attention mechanism to not only exploit the contextual information of the original sentence but also leverage the contextual and label information of the labeled clue sentences. In addition, we devise a mask label strategy to further avoid over-fitting by randomly masking out the labels of certain tokens in the clue sentence and then predicting these mask labels based on the context of the tokens corresponding to the mask labels. We verify the effectiveness and generalizability of the proposed framework on three sequence labeling tasks, including Chinese Named Entity Recognition, English Named Entity Recognition, and Aspect Term Extraction. Extensive experimental results show that our method can yield state-of-the-art or competitive results on the three tasks. • A general framework uses labeled clues to mitigate labeled data shortages. • Two retrieved ways of labeled clue sentences are designed. • Mask label strategy is devised to avoid over-fitting. • Transformer’s self-attention is modified to exploit original and clue sentences. • We verify effectiveness of the proposed framework on three sequence labeling tasks.

An Effective Incorporating Heterogeneous Knowledge Curriculum Learning for Sequence Labeling

Neural Coupled Sequence Labeling for Heterogeneous Annotation Conversion

Coupled Sequence Labeling On Heterogeneous Annotations: Pos Tagging As A Case Study

A Semi-shared Hierarchical Joint Model for Sequence Labeling

Fast Coupled Sequence Labeling on Heterogeneous Annotations Via Context-aware Pruning.

Incorporating Deep Syntactic and Semantic Knowledge for Chinese Sequence Labeling with GCN

A Deep Reinforced Sequence-to-Set Model for Multi-Label Classification

Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning

Enhancing Multi-Label Classification Via Dynamic Label-Order Learning

Coarse-to-Fine Curriculum Learning

Piecewise Large Margin Learning for Partially Annotated Sequences

Sequence Labeling with Meta-Learning

Joint Segmentation and Tagging with Coupled Sequences Labeling

Learning with Different Amounts of Annotation: From Zero to Many Labels

Co-Learning Meets Stitch-Up for Noisy Multi-label Visual Recognition

Coupled POS Tagging on Heterogeneous Annotations

Complementary Labels Learning with Augmented Classes

Improving Sequence Labeling with Labeled Clue Sentences

Unsupervised Lifelong Learning with Curricula

Enhancing Label Sharing Efficiency in Complementary-Label Learning with Label Augmentation

Complementary label learning based on knowledge distillation