FudanNLP: A Toolkit for Chinese Natural Language Processing with Online Learning Algorithms

Feng Ji,Wenjun Gao,Xipeng Qiu,Xuanjing Huang

2013-01-01

Abstract:In this paper, we describe a Chinese natural language processing (NLP) toolkit FudanNLP, which we employed in CIPS-PARSEVAL 2009. In all five closed test, our system achieved the above-average ranks and was slightly weaker than the best system. 1

What problem does this paper attempt to address?

FudanNLP: A Toolkit for Chinese Natural Language Processing.

Xipeng Qiu,Qi Zhang,Xuanjing Huang

2013-01-01

Abstract:The growing need for Chinese natural language processing (NLP) is largely in a range of research and commercial applications. However, most of the currently Chinese NLP tools or components still have a wide range of issues need to be further improved and developed. FudanNLP is an open source toolkit for Chinese natural language processing (NLP), which uses statistics-based and rule-based methods to deal with Chinese NLP tasks, such as word segmentation, part-ofspeech tagging, named entity recognition, dependency parsing, time phrase recognition, anaphora resolution and so on.
Development and Evaluation of Task-Specific NLP Framework in China.

Caixia Ge,Yinsheng Zhang,Zhenzhen Huang,Zheng Jia,Meizhi Ju,Huilong Duan,Haomin Li

DOI: https://doi.org/10.3233/978-1-61499-564-7-1031

2015-01-01

Abstract:Natural language processing (NLP) has been designed to convert narrative text into structured data. Although some general NLP architectures have been developed, a task-specific NLP framework to facilitate the effective use of data is still a challenge in lexical resource limited regions, such as China. The purpose of this study is to design and develop a task-specific NLP framework to extract targeted information from particular documents by adopting dedicated algorithms on current limited lexical resources. In this framework, a shared and evolving ontology mechanism was designed. The result has shown that such a free text driven platform will accelerate the NLP technology acceptance in China.
N-LTP: an Open-source Neural Language Technology Platform for Chinese

Wanxiang Che,Yunlong Feng,Libo Qin,Ting Liu

DOI: https://doi.org/10.48550/arxiv.2009.11616

2021-01-01

Abstract:We introduce N-LTP, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: lexical analysis (Chinese word segmentation, pan-of-speech tagging, and named entity recognition), syntactic parsing (dependency parsing), and semantic parsing (semantic dependency parsing and semantic role labeling). Unlike the existing state-of-the-art toolkits, such as Stanza, that adopt an independent model for each task, N-LTP adopts the multi-task framework by using a shared pre-trained model, which has the advantage of capturing the shared knowledge across relevant Chinese tasks. In addition, a knowledge distillation method (Clark et al., 2019) where the single-task model teaches the multi-task model is further introduced to encourage the multi-task model to surpass its single-task teacher. Finally, we provide a collection of easy-to-use APIs and a visualization tool to make users to use and view the processing results more easily and directly. To the best of our knowledge, this is the first toolkit to support six Chinese NLP fundamental tasks. Source code, documentation, and pre-trained models are available at https : //github.com/HIT-SCIR/ltp.
N-LTP: A Open-source Neural Chinese Language Technology Platform with Pretrained Models.

Wanxiang Che,Yunlong Feng,Libo Qin,Ting Liu

2020-01-01

Abstract:We introduce N-LTP, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: lexical analysis (Chinese word segmentation, part-of-speech tagging, and named entity recognition), {syntactic parsing} (dependency parsing), and {semantic parsing} (semantic dependency parsing and semantic role labeling). Unlike the existing state-of-the-art toolkits, such as Stanza, that adopt an independent model for each task, N-LTP adopts the multi-task framework by using a shared pre-trained model, which has the advantage of capturing the shared knowledge across relevant Chinese tasks. In addition, knowledge distillation where the single-task model teaches the multi-task model is further introduced to encourage the multi-task model to surpass its single-task teacher. Finally, we provide a collection of easy-to-use APIs and a visualization tool to make users easier to use and view the processing results directly. To the best of our knowledge, this is the first toolkit to support six Chinese NLP fundamental tasks. Source code, documentation, and pre-trained models are available at \url{this https URL}.
Fasthan: A BERT-based Multi-Task Toolkit for Chinese NLP.

Zhichao Geng,Hang Yan,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.18653/v1/2021.acl-demo.12

2021-01-01

Abstract:We present fastHan, an open-source toolkit for four basic tasks in Chinese natural language processing: Chinese word segmentation (CWS), Part-of-Speech (POS) tagging, named entity recognition (NER), and dependency parsing. The backbone of fastHan is a multi-task model based on a pruned BERT, which uses the first 8 layers in BERT. We also provide a 4-layer base model compressed from the 8-layer model. The joint-model is trained and evaluated on 13 corpora of four tasks, yielding near state-of-the-art (SOTA) performance in dependency parsing and NER, achieving SOTA performance in CWS and POS. Besides, fastHan’s transferability is also strong, performing much better than popular segmentation tools on a non-training corpus. To better meet the need of practical application, we allow users to use their own labeled data to further fine-tune fastHan. In addition to its small size and excellent performance, fastHan is user-friendly. Implemented as a python package, fastHan isolates users from the internal technical details and is convenient to use. The project is released on Github.
FudanNLP at RITE 2011: a Shallow Semantic Approach to Textual Entailment.

Ling Cao,Xipeng Qiu,Xuanjing Huang

2011-01-01

Abstract:RITE is a task recognizing logic relations between texts. This paper presents FDCS’s approach on NTCIR9-RITE Chinese simplified BC & MC subtasks. Our system is built on a machine learning architecture with features selected on shallow semantic methods, including named entity recognition, date & time expression extraction, words overlap and negation concept recognition. FudanNLP is wildly used by our system on NLP procession and feature extraction. System gets accuracy as 76% on BC subtask and 58.5% on MC subtask, separately.
Fasthan: A BERT-based Joint Many-Task Toolkit for Chinese NLP

Zhichao Geng,Hang Yan,Xipeng Qiu,Xuanjing Huang

2020-01-01

Abstract:We present fastHan, an open-source toolkit for four basic tasks in Chinese natural language processing: Chinese word segmentation, Part-of-Speech tagging, named entity recognition, and dependency parsing. The kernel of fastHan is a joint many-task model based on a pruned BERT, which uses the first 8 layers in BERT. We also provide a 4-layer base version of model compressed from the 8-layer model. The joint-model is trained and evaluated in 13 corpora of four tasks, yielding near state-of-the-art (SOTA) performance in the dependency parsing task and SOTA performance in the other three tasks. In addition to its small size and excellent performance, fastHan is also very user-friendly. Implemented as a python package, fastHan allows users to easily download and use it. Users can get what they want with one line of code, even if they have little knowledge of deep learning. The project is released on Github.
EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

Chengyu Wang,Minghui Qiu,Chen Shi,Taolin Zhang,Tingting Liu,Lei Li,Jianing Wang,Ming Wang,Jun Huang,Wei Lin

DOI: https://doi.org/10.48550/arXiv.2205.00258

2023-03-13

Abstract:The success of Pre-Trained Models (PTMs) has reshaped the development of Natural Language Processing (NLP). Yet, it is not easy to obtain high-performing models and deploy them online for industrial practitioners. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale PTMs, and provides a unified framework of model training, inference and deployment for real-world applications. Currently, EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. The source code of our EasyNLP toolkit is released at GitHub (<a class="link-external link-https" href="https://github.com/alibaba/EasyNLP" rel="external noopener nofollow">this https URL</a>).

Computation and Language
A Novel Sentiment Polarity Detection Framework for Chinese

Tinghuai Ma,Huan Rong,Yongsheng Hao,Jie Cao,Yuan Tian,Mznah Al-Rodhaan

DOI: https://doi.org/10.1109/taffc.2019.2932061

IF: 13.99

2022-01-01

IEEE Transactions on Affective Computing

Abstract:Nowadays, mining opinions or sentiment from online user-generated text has become a research hot spot. Although a large amount of lexicon-based Chinese polarity detection works have been done, the existing methods have one common flaw: that even the same word can have opposite polarities among different seed lexicons. This is known as polarity fuzziness. To enhance the performance of Chinese sentiment polarity detection, we start from a two-aspect lexicon expansion so that the polarity fuzziness can be avoided. Specifically, we detect sentiment polarity for new words and revise sentiment polarity for words already defined in seed lexicons. Then, we formulate a novel sentiment polarity detection framework for Chinese (SPDFC) with more attention to fine-grained sentiment processing, which is involved in symmetrical mapping, sentiment feature pruning and text representation. In this way, words' polarity can be directly taken as features, penetrating further in the polarity detection phase. According to our experimental results, the proposed SPDFC framework can achieve the best overall performance from the perspective of Chinese polarity detection, sentiment feature pruning, and text representation compared to other classical and state-of-the-art methods.

computer science, cybernetics, artificial intelligence
WYWEB: A NLP Evaluation Benchmark For Classical Chinese

Bo Zhou,Qianglong Chen,Tianyu Wang,Xiaomi Zhong,Yin Zhang

DOI: https://doi.org/10.48550/arXiv.2305.14150

2023-05-23

Computation and Language

Abstract:To fully evaluate the overall performance of different NLP models in a given domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and CLUE. The fi eld of natural language understanding has traditionally focused on benchmarks for various tasks in languages such as Chinese, English, and multilingua, however, there has been a lack of attention given to the area of classical Chinese, also known as "wen yan wen", which has a rich history spanning thousands of years and holds signifi cant cultural and academic value. For the prosperity of the NLP community, in this paper, we introduce the WYWEB evaluation benchmark, which consists of nine NLP tasks in classical Chinese, implementing sentence classifi cation, sequence labeling, reading comprehension, and machine translation. We evaluate the existing pre-trained language models, which are all struggling with this benchmark. We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on classical Chinese NLU. The github repository is https://github.com/baudzhou/WYWEB.
A Simple Yet Effective Joint Training Method for Cross-Lingual Universal Dependency Parsing.

Danlu Chen,Mengxiao Lin,Zhifeng Hu,Xipeng Qiu

DOI: https://doi.org/10.18653/v1/k18-2026

2018-01-01

Abstract:This paper describes Fudan’s submission to CoNLL 2018’s shared task Universal Dependency Parsing. We jointly train models when two languages are similar according to linguistic typology and then ensemble the models using a simple re-parse algorithm. We outperform the baseline method by 4.4% (2.1%) on average on development (test) set in CoNLL 2018 UD Shared Task.
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Yuzhen Huang,Yuzhuo Bai,Zhihao Zhu,Junlei Zhang,Jinghan Zhang,Tangjun Su,Junteng Liu,Chuancheng Lv,Yikai Zhang,Jiayi Lei,Yao Fu,Maosong Sun,Junxian He,Fanchao Qi

DOI: https://doi.org/10.48550/arXiv.2305.08322

2023-05-15

Computation and Language

Abstract:New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.
FDU at TREC-10: Filtering, QA, Web and Video Tasks

Lide Wu,Xuanjing Huang,Junyu Niu,Yikun Guo,Yingju Xia,Zhe Feng

DOI: https://doi.org/10.6028/nist.sp.500-250.filtering-fudan

2001-01-01

Abstract:This year Fudan University takes part in the TREC conference for the second time. We have participated in four tracks of Filtering, Q&A, Web and Video. For filtering, we participate in the sub-task of adaptive and batch filtering. Vector representation and computation are heavily applied in filtering procedure. Four runs have been submitted, which includes one T10SU and one T10F run for adpative filtering, as well as another one T10SU and one T10F run for batch filtering. We have tried many natural language processing techniques in our QA system, including statistical sentence breaking, POS tagging, parsing, name entity tagging, chunking and semantic verification. Various sources of world knowledge are also incorporated, such as WordNet and geographic information. For web retrieval, relevant document set is first created by an extended Boolean retrieval engine, and then reordered according to link information. Four runs with different combination of topic coverage and link information are submitted. On video track, We take part in both of the sub-tasks. In the task of shot boundary detection, we have submitted two runs with different parameters. In the task of video retrieval, we have submitted the results of 17 topics among all the topics.
Fudan University at TRECVID 2007

Xiangyang Xue,Hui Yu,Hong Lu,Youguang Guo,Yuejie Zhang,Shile Zhang,Bin Li,Bolan Su,Yingbin Zheng,Wenjian Zhou,Lei Cen,Jie Zhang,Yu Jiang,Jiahui Qi,Jiaojiao Lu,Qian Diao,Zhenzhen Shi,Zichen Sun

2007-01-01

Abstract:In this notebook paper we describe our participation in the NIST TRECVID 2007 evaluation. We took part in two tasks of benchmark this year including high-level feature extraction and interactive search. For high-level feature extraction, we submitted 4 runs. FD_SVM: using SVM. FD_SVM_BN: using SVM and ontology. FD_MTL: using multi-task learning. FD_MTL_BN: using multi-task learning and ontology. Evaluation results illustrate that there are both advantages and disadvantages exist in all methods. For search, we submitted 5 interactive runs. Fudan_P: using multi-model and AP-based fusion. Fudan_R: using multi-model and MGR fusion. Fudan_C: cross system retrieval. Fudan_T: textual retrieval Fudan_I: image retrieval Evaluation results illustrate that the AP-based fusion method yields higher precision while the MGR fusion method finds more positive shots than other runs. We also experimented with simple cross system interactive retrieval to estimate the impact of manual browsing on the results.
LTP: a Chinese Language Technology Platform

Wanxiang Che,Zhenghua Li,Ting Liu

2010-01-01

Abstract:LTP (Language Technology Platform) is an integrated Chinese processing platform which includes a suite of high performance natural language processing (NLP) modules and relevant corpora. Especially for the syntactic and semantic parsing modules, we achieved good results in some relevant evaluations, such as CoNLL and SemEval. Based on XML internal data representation, users can easily use these modules and corpora by invoking DLL (Dynamic Link Library) or Web service APIs (Application Program Interface), and view the processing results directly by the visualization tool.
An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

Xihong Wu,Xiaojun Lin,Xinhao Wang,Chunyao Wu,Yaozhong Zhang,Dianhai Yu

2008-01-01

Abstract:This paper describes three systems: the Chinese word segmentation (WS) system, the named entity recognition (NER) system and the Part-of-Speech tagging (POS) system, which are submitted to the Fourth International Chinese Language Processing Bakeoff. Here, Conditional Random Fields (CRFs) are employed as the primary models. For the WS and NER tracks, the ngram language model is incorporated in our CRFs based systems in order to take into account the higher level language information. Furthermore, to improve the performances of our submitted systems, a transformationbased learning (TBL) technique is adopted for post-processing.
Extraction of Sentiment Targets of Chinese Micro-blog Based on Dependency Parsing and Derogatory and Commendatory Direction Tendency(DCDT)

Lei Gao,Bin Li,Xinyu Dai,Shujian Huang,Jiajun Chen

2012-01-01

Abstract:This paper describes our (NJU-NLP) participation for the evaluation of the 1 CCF Conference on Natural Language Processing & Chinese Computing. We submit results for all three subtasks of the sentiment analysis of Chinese micro-blog task, including opinion sentences recognition, sentiment tendency analysis, and sentiment targets extraction. In opinion sentences recognition and sentiment tendency analysis, we use tokenization and emotional directories to recognize the opinion sentences and judge the sentiment tendency of each opinion sentence. In sentiment targets extraction, we use the rules of Derogatory and Commendatory Direction Tendency(DCDT) of sentiment words to acquire the sentiment targets. This paper makes a detailed description of our system, as well as the evaluation results.
Research on Chinese Lexical Analysis System by Fusing Multiple Knowledge Sources

Wei JIANG,Xiao-Long WANG,Yi GUAN,Jian ZHAO

DOI: https://doi.org/10.3321/j.issn:0254-4164.2007.01.016

2007-01-01

Jisuanji Xuebao/Chinese Journal of Computers

Abstract:Chinese lexical analysis is the foundation task for most Chinese natural language processing. In this paper, word segmentation, POS tagging, named entity recognition and their relation are well discussed. Moreover, a pragmatic lexical analysis system based on mixed language models is presented, which adopts many models, such as n-gram, hidden Markov model, maximum entropy model, support vector machine and conditional random fields, they have good performance in the special sub-tasks. The Word Segmenter participated in the Second International Chinese Word Segmentation Bakeoff in 2005, and achieved 97.2% and 96.7% in terms of F-measure in MSR and PKU open test respectively. While the POS tagging and named entity recognition modules achieved 96.1% in precision and 88.6% in F-measure respectively in open test with the corpus that came from six-month corpora of Chinese Peoples' Daily.
Natural Language Processing and Chinese Computing - 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part I.

Xiaodan Zhu,Min Zhang,Yu Hong,Ruifang He

DOI: https://doi.org/10.1007/978-3-030-60450-9

2020-01-01

Abstract:ive Summarization via Discourse Relation and Graph Convolutional Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 Wenjie Wei, Hongling Wang, and Zhongqing Wang Chinese Question Classification Based on ERNIE and Feature Fusion . . . . . . 343 Gaojun Liu, Qiuxia Yuan, Jianyong Duan, Jie Kou, and Hao Wang An Abstractive Summarization Method Based on Global Gated Dual Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 Lu Peng, Qun Liu, Lebin Lv, Weibin Deng, and Chongyu Wang Rumor Detection on Hierarchical Attention Network with User and Sentiment Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Sujun Dong, Zhong Qian, Peifeng Li, Xiaoxu Zhu, and Qiaoming Zhu Measuring the Semantic Stability of Word Embedding. . . . . . . . . . . . . . . . . 378 Zhenhao Huang and Chenxu Wang Task-to-Task Transfer Learning with Parameter-Efficient Adapter . . . . . . . . . 391 Haiou Zhang, Hanjun Zhao, Chunhua Liu, and Dong Yu Key-Elements Graph Constructed with Evidence Sentence Extraction for Gaokao Chinese. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Xiaoyue Wang, Yu Ji, and Ru Li
A Chinese Framework of Semantic Taxonomy and Description: Preliminary Experimental Evaluation Using Web Information Extraction.

Liangjun Zang,Weimin Wang,Ya Wang,Fang Fang,Cong Cao,Xiaolong Wu,Qi Zhou,Ye Lu,Tingyu Li,Cungen Cao

DOI: https://doi.org/10.1007/978-3-319-25159-2_25

2015-01-01

Abstract:The Chinese Framework of Semantic Taxonomy and Description FSTD is a linguistic resource that stores lexical and predicate-argument semantics about events or states in Chinese text, developed with the application of knowledge acquisition from Chinese text in mind. In this paper we build a web information extraction system, called NkiExtractor, to evaluate FSTD experimentally. We use two metrics: grammar coverage measures whether there is a semantic category of FSTD that corresponds to an event description in text, and extraction precision measures whether the correct predicate-argument structure can be extracted from text. Experimental results show that FSTD is a fairly comprehensive and effective resource for knowledge acquisition. We also discuss future work for expanding FSTD and improving extraction precision of NkiExtractor.

FudanNLP: A Toolkit for Chinese Natural Language Processing with Online Learning Algorithms

FudanNLP: A Toolkit for Chinese Natural Language Processing.

Development and Evaluation of Task-Specific NLP Framework in China.

N-LTP: an Open-source Neural Language Technology Platform for Chinese

N-LTP: A Open-source Neural Chinese Language Technology Platform with Pretrained Models.

Fasthan: A BERT-based Multi-Task Toolkit for Chinese NLP.

FudanNLP at RITE 2011: a Shallow Semantic Approach to Textual Entailment.

Fasthan: A BERT-based Joint Many-Task Toolkit for Chinese NLP

EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

A Novel Sentiment Polarity Detection Framework for Chinese

WYWEB: A NLP Evaluation Benchmark For Classical Chinese

A Simple Yet Effective Joint Training Method for Cross-Lingual Universal Dependency Parsing.

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

FDU at TREC-10: Filtering, QA, Web and Video Tasks

Fudan University at TRECVID 2007

LTP: a Chinese Language Technology Platform

An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

Extraction of Sentiment Targets of Chinese Micro-blog Based on Dependency Parsing and Derogatory and Commendatory Direction Tendency(DCDT)

Research on Chinese Lexical Analysis System by Fusing Multiple Knowledge Sources

Natural Language Processing and Chinese Computing - 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part I.

A Chinese Framework of Semantic Taxonomy and Description: Preliminary Experimental Evaluation Using Web Information Extraction.