HugNLP: A Unified and Comprehensive Library for Natural Language Processing

Jianing Wang,Nuo Chen,Qiushi Sun,Wenkang Huang,Chengyu Wang,Ming Gao
DOI: https://doi.org/10.1145/3583780.3614742
2023-01-01
Abstract:In this paper, we introduce HugNLP, a unified and comprehensive library for natural language processing (NLP) with the prevalent backend of Hugging Face Transformers, which is designed for NLP researchers to easily utilize off-the-shelf algorithms and develop novel methods with user-defined models and tasks in real-world scenarios. HugNLP consists of a hierarchical structure including models, processors and applications that unifies the learning process of pre-trained language models (PLMs) on different NLP tasks. Additionally, we present some featured NLP applications to show the effectiveness of HugNLP, such as knowledge-enhanced PLMs, universal information extraction, low-resource mining, and code understanding and generation, etc. The source code will be released on GitHub (https://github.com/HugAILab/HugNLP).
What problem does this paper attempt to address?