Evaluation Reportof the fourth Chinese Parsing Evaluation: CIPS-SIGHAN-ParsEval-2014

qiang zhou

DOI: https://doi.org/10.3115/v1/w14-6823

2014-01-01

Abstract:This paper gives the overview of the third Chinese parsing evaluation: CIPS-SIGHANParsEval-2012, including its parsing sub-tasks, evaluation metrics, training and test data. The detailed evaluation results and simple discussions will be given to show the difficulties in Chinese syntactic parsing.

What problem does this paper attempt to address?

Evaluation Reportof the Third Chinese Parsing Evaluation: CIPS-SIGHAN-ParsEval-2012

Qiang Zhou

2012-01-01

Abstract:This paper gives the overview of the third Chinese parsing evaluation: CIPS-SIGHANParsEval-2012, including its parsing sub-tasks, evaluation metrics, training and test data. The detailed evaluation results and simple discussions will be given to show the difficulties in Chinese syntactic parsing.
Chinese Chunk Parsing Evaluation Tasks

周强,李玉梅

DOI: https://doi.org/10.3969/j.issn.1003-0077.2010.01.022

2010-01-01

Abstract:The paper introduces three chunk parsing tasks carried in current CIPS parsing evaluation workshop(CIPS-ParsEval-2009),which is organized by Tsinghua University and Northeastern University.They are base chunk parsing,functional chunk parsing and event description clause recognition tasks.The designing motivation and the classification standards of these three chunks are discussed in the paper.Based on the detailed syntactic annotations in Tsinghua Chinese Treebank(TCT),three benchmark chunk banks automatically extracted from TCT are built.The evaluation results of top-5 participating systems are also given.The data analysis from their statistics and the comparison with current chunk schemes show some characteristics of these three chunk parsing tasks.
Chinese Question Answering System-Oriented Chinese Parsing

ZHANG Liang,WANG Shu-mei,HUANG He-yan,ZHANG Xiao-fei

DOI: https://doi.org/10.3969/j.issn.1671-9352.2006.03.008

2006-01-01

Abstract:Chinese syntactic parsing is a key of Chinese study and Chinese information processing,as well as a difficulty.The characteristics of Chinese question sentence are discussed.In Chinese question answering system-oriented Chinese parsing,cor-pus-based syntactic processes are applied.Chinese question sentence short,question word,question structure etc are employed.The preliminary experiment result shows that its performance can achieve design goal.
SemEval-2012 Task 5: Chinese Semantic Dependency Parsing

Wanxiang Che,Meishan Zhang,Yanqiu Shao,Ting Liu

DOI: https://doi.org/10.18653/v1/s16-1167

2012-01-01

Abstract:The paper presents the SemEval-2012 Shared Task 5: Chinese Semantic Dependency Parsing. The goal of this task is to identify the dependency structure of Chinese sentences from the semantic view. We firstly introduce the motivation of providing Chinese semantic dependency parsing task, and then describe the task in detail including data preparation, data format, task evaluation, and so on. Over ten thousand sentences were labeled for participants to train and evaluate their systems. At last, we briefly describe the submitted systems and analyze these results.
NJU-Parser: Achievements on Semantic Dependency Parsing.

Guangchao Tang,Bin Li,Shuaishuai Xu,Xinyu Dai,Jiajun Chen

2012-01-01

Abstract:In this paper, we introduce our work on SemEval-2012 task 5: Chinese Semantic Dependency Parsing. Our system is based on MSTParser and two effective methods are proposed: splitting sentence by punctuations and extracting last character of word as lemma. The experiments show that, with a combination of the two proposed methods, our system can improve LAS about one percent and finally get the second prize out of nine participating systems. We also try to handle the multilevel labels, but with no improvement.
A Survey of Semantic Parsing Techniques

Peng Jiang,Xiaodong Cai

DOI: https://doi.org/10.3390/sym16091201

2024-09-13

Symmetry

Abstract:In the information age, semantic parsing technology drives efficiency improvement and accelerates the process of intelligence. However, it faces complex understanding, data inflation, inappropriate evaluation, and difficult application of advanced large models. This study analyses the current challenges and looks forward to the development trend of the technology. Specific approaches include: this study adopts a systematic review method and strictly follows the PRISMA framework, deeply analyzes the key ideas, methods, problems, and solutions of traditional and neural network methods, and explores the model performance, API application, dataset, and evaluation mechanism. Through literature analysis, the technology is classified according to its application scenarios. Then, the practical application contributions are summarized, current limitations such as data size, model performance, and resource requirements are analyzed, and future directions such as dataset expansion, real-time performance enhancement, and industrial applications are envisioned. The results of the study show significant advances in semantic parsing technology with far-reaching impacts. Traditional and neural network methods complement each other to promote theoretical and practical innovation. In the future, with the continuous progress and in-depth application of machine learning technology, semantic parsing technology needs to further deepen the research on logical reasoning and evaluation, to better cope with technical challenges and lead the new development of natural language processing and AI.

multidisciplinary sciences
An Evaluation of Chinese Human-Computer Dialogue Technology

Zixian Feng,Caihai Zhu,Weinan Zhang,Zhigang Chen,Wanxiang Che,Minlie Huang,Linlin Li

DOI: https://doi.org/10.1162/dint_a_00090

2021-01-01

Data Intelligence

Abstract:There is a growing interest in developing human-computer dialogue systems which is an important branch in the field of artificial intelligence (AI). However, the evaluation of large-scale Chinese human-computer dialogues is still a challenging task. To attract more attention to dialogue evaluation work, we held the fourth Evaluation of Chinese Human-Computer Dialogue Technology (ECDT). It consists of few-shot learning in spoken language understanding (SLU) (Task 1) and knowledge-driven multi-turn dialogue competition (Task 2), the data sets of which are provided by Harbin Institute of Technology and Tsinghua University. In this paper, we will introduce the evaluation tasks and data sets in detail. Meanwhile, we will also analyze the evaluation results and the existing problems in the evaluation.
Building a comprehensive syntactic and semantic corpus of Chinese clinical texts

Bin He,Bin Dong,Yi Guan,Jinfeng Yang,Zhipeng Jiang,Qiubin Yu,Jianyi Cheng,Chunyan Qu

DOI: https://doi.org/10.1016/j.jbi.2017.04.006

IF: 8

2017-05-01

Journal of Biomedical Informatics

Abstract:OBJECTIVE: To build a comprehensive corpus covering syntactic and semantic annotations of Chinese clinical texts with corresponding annotation guidelines and methods as well as to develop tools trained on the annotated corpus, which supplies baselines for research on Chinese texts in the clinical domain.MATERIALS AND METHODS: An iterative annotation method was proposed to train annotators and to develop annotation guidelines. Then, by using annotation quality assurance measures, a comprehensive corpus was built, containing annotations of part-of-speech (POS) tags, syntactic tags, entities, assertions, and relations. Inter-annotator agreement (IAA) was calculated to evaluate the annotation quality and a Chinese clinical text processing and information extraction system (CCTPIES) was developed based on our annotated corpus.RESULTS: The syntactic corpus consists of 138 Chinese clinical documents with 47,426 tokens and 2612 full parsing trees, while the semantic corpus includes 992 documents that annotated 39,511 entities with their assertions and 7693 relations. IAA evaluation shows that this comprehensive corpus is of good quality, and the system modules are effective.DISCUSSION: The annotated corpus makes a considerable contribution to natural language processing (NLP) research into Chinese texts in the clinical domain. However, this corpus has a number of limitations. Some additional types of clinical text should be introduced to improve corpus coverage and active learning methods should be utilized to promote annotation efficiency.CONCLUSIONS: In this study, several annotation guidelines and an annotation method for Chinese clinical texts were proposed, and a comprehensive corpus with its NLP modules were constructed, providing a foundation for further study of applying NLP techniques to Chinese texts in the clinical domain.

medical informatics,computer science, interdisciplinary applications
Overview of the NLPCC 2015 Shared Task: Chinese Word Segmentation and POS Tagging for Micro-blog Texts

Xipeng Qiu,Peng Qian,Liusong Yin,Shiyu Wu,Xuanjing Huang

DOI: https://doi.org/10.1007/978-3-319-25207-0_50

2015-01-01

Abstract: In this paper, we give an overview for the shared task at the 4th CCF Conference on Natural Language Processing \& Chinese Computing (NLPCC 2015): Chinese word segmentation and part-of-speech (POS) tagging for micro-blog texts. Different with the popular used newswire datasets, the dataset of this shared task consists of the relatively informal micro-texts. The shared task has two sub-tasks: (1) individual Chinese word segmentation and (2) joint Chinese word segmentation and POS Tagging. Each subtask has three tracks to distinguish the systems with different resources. We first introduce the dataset and task, then we characterize the different approaches of the participating systems, report the test results, and provide a overview analysis of these results. An online system is available for open registration and evaluation at http://nlp.fudan.edu.cn/nlpcc2015.
Improve Chinese Semantic Dependency Parsing Via Syntactic Dependency Parsing

Meishan Zhang,Wanxiang Che,Yanqiu Shao,Ting Liu

DOI: https://doi.org/10.1109/ialp.2012.42

2012-01-01

Abstract:We address the problem of Chinese semantic dependency parsing. Dependency parsing is traditionally oriented to syntax analysis, which we denote by syntactic dependency parsing to distinguish it from semantic dependency parsing. In this paper, firstly we compare Chinese semantic dependency parsing and syntactic dependency parsing systematically, showing that syntactic dependency parsing can potentially improve the performance of semantic dependency parsing. Thus then we suggest an approach based on quasi-synchronous grammar to incorporate the auto-parsed syntactic dependency tree into semantic dependency parsing. We conduct experiments on the Chinese semantic dependency parsing corpus of SemEval-2012. Finally we achieve 65.25% LAS on test corpus, gaining increases of 2.45% compared to the top result of 62.80% in SemEval-2012.
Special Perceptual Parsing for Chinese Landscape Painting Scene Understanding: a Semantic Segmentation Approach.

Rui Yang,Honghong Yang,Min Zhao,Ru Jia,Xiaojun Wu,Yumei Zhang

DOI: https://doi.org/10.1007/s00521-023-09343-w

2023-01-01

Abstract:The automatic and precise perceptual parsing of Chinese landscape paintings (CLP) significantly aids in the digitization and recreation of artworks. Manual extraction and analysis of objects in CLPs is challenging, even for expert painters with professional knowledge and sharp discernment. Two main key reasons restricted the development of CLP parsing: (1) a lack of pixel-level labeled data used to supervise model training, and (2) the inherent complexity of CLP images compared to real scenes, characterized by varied scales, diverse textures, and intricate empty spaces. To address these challenges, we first construct a pixel-level annotated CLP segmentation datasets to advance perceptual parsing. Then, a novel CLP Perceptual Parsing (CLPPP) model is designed to fully utilize the intrinsic features of CLP images. To dynamically and adaptively capture context information, we introduced a set of learnable kernels into the CLPPP model based on the multiscale features of objects within CLPs. This enabled the model to learn an appropriate receptive field for context information extraction. Additionally, a positional attention head is devised to effectively eliminate noise from the intergroup and help the kernel gain inter-object position information. This iterative optimization process is helpful to learn powerful feature representations for different textures in CLPs. The experiment results demonstrate that the proposed CLPPP model outperforms state-of-the-art methods with mIoU, aAcc, and mAcc scores of 55.45, 75.08, and 71.15, respectively, achieving a large margin on the CLP dataset under consistent conditions.
The CIPS-SIGHAN CLP 2014 Chinese Word Segmentation Bake-off.

Huiming Duan,Zhifang Sui,Tao Ge

DOI: https://doi.org/10.3115/v1/w14-6814

2014-01-01

Abstract:This paper summarizes the SIGHAN 2014 Chinese Word Segmentation bakeoff in several aspects such as dataset, evaluation results.In addition, we analyze errors of segmentation by instance and make a suggestion for improving segmentation systems.
Discourse Representation Structure Parsing for Chinese

Chunliu Wang,Xiao Zhang,Johan Bos

2023-06-16

Abstract:Previous work has predominantly focused on monolingual English semantic parsing. We, instead, explore the feasibility of Chinese semantic parsing in the absence of labeled data for Chinese meaning representations. We describe the pipeline of automatically collecting the linearized Chinese meaning representation data for sequential-to sequential neural networks. We further propose a test suite designed explicitly for Chinese semantic parsing, which provides fine-grained evaluation for parsing performance, where we aim to study Chinese parsing difficulties. Our experimental results show that the difficulty of Chinese semantic parsing is mainly caused by adverbs. Realizing Chinese parsing through machine translation and an English parser yields slightly lower performance than training a model directly on Chinese data.

Computation and Language
Supervised Approaches and Dependency Parsing for Chinese Opinion Analysis at NTCIR-8.

Bin Lu,Benjamin K. Tsou,Tao Jiang

2010-01-01

Abstract:In this paper, we describe our participating system, which is based on supervised approaches and dependency parsing, for opinion analysis on traditional Chinese texts at NTCIR-8. For opinionated sentence recognition, the supervised lexicon-based approach, SVM and Maximum Entropy are combined together. For polarity classification, we use only the supervised lexicon-based approach. For opinion holder and target identification, we, on the basis of dependency parsing, identify opinion holders by means of reporting verbs and identify opinion targets by considering both opinion holders and opinion-bearing words. The results show that among all the teams participating in the traditional Chinese task, our system achieve: 1) the highest F-measure on the opinionated sentence recognition task, 2) the second highest F-measure on the identification of both opinion holders and targets, 3) the middle ranking for opinion polarity classification.
Dependency-Gated Cascade Biaffine Network for Chinese Semantic Dependency Graph Parsing

Zizhuo Shen,Huayong Li,Dianqing Liu,Yanqiu Shao

DOI: https://doi.org/10.1007/978-3-030-32233-5_65

2019-01-01

Abstract:The Chinese Semantic Dependency Graph (CSDG) parsing breaks the limitation of the syntactic or semantic tree structure dependency system with a richer representation ability to express more complex language phenomena and semantic relationships. Most of the existing CSDG parsing systems used transition-based approach. It needs to define a complex transition system and its performance depends heavily on whether the model can properly represent the transition state. In this paper, we adopt neural graph-based approach which using Biaffine network to solve the CSDG parsing task. Furthermore, considering that dependency edge and label have the strong relationship, we design an effective dependency-gated cascade mechanism to improve the accuracy of dependency label prediction. We test our system on the SemEval-2016 Task 9 dataset. Experiment result shows that our model achieves state-of-the-art performance with 7.48% and 6.36% labeled F1-score improvement compared to the previous best model in TEXTBOOKS and NEWS domain respectively.
Overview of the NLPCC 2015 Shared Task: Entity Recognition and Linking in Search Queries.

Yansong Feng,Zhe Han,Kun Zhang

DOI: https://doi.org/10.1007/978-3-319-25207-0_51

2015-01-01

Abstract:This paper provides an overview of the Shared Task at the 4th CCF Conference on Natural Language Processing and Chinese Computing NLPCC 2015: Entity Recognition and Linking in Search Queries, where participant systems are required to recognize entity mentions from short search queries in Chinese, and further link them into a given structured knowledge base. In this paper, we introduce how the task is defined, how we collect the datasets and last we report the evaluation results with a brief analysis.
System Report for CCL24-Eval Task 7: Multi-Error Modeling and Fluency-Targeted Pre-training for Chinese Essay Evaluation

Jingshen Zhang,Xiangyu Yang,Xinkai Su,Xinglu Chen,Tianyou Huang,Xinying Qiu

2024-07-11

Abstract:This system report presents our approaches and results for the Chinese Essay Fluency Evaluation (CEFE) task at CCL-2024. For Track 1, we optimized predictions for challenging fine-grained error types using binary classification models and trained coarse-grained models on the Chinese Learner 4W corpus. In Track 2, we enhanced performance by constructing a pseudo-dataset with multiple error types per sentence. For Track 3, where we achieved first place, we generated fluency-rated pseudo-data via back-translation for pre-training and used an NSP-based strategy with Symmetric Cross Entropy loss to capture context and mitigate long dependencies. Our methods effectively address key challenges in Chinese Essay Fluency Evaluation.

Computation and Language
Calibrated Parsing Items Evaluation: a step towards objectifying the translation assessment

Alireza Akbari,Mohammadtaghi Shahnazari

DOI: https://doi.org/10.1186/s40468-019-0083-x

2019-05-31

Language Testing in Asia

Abstract:The present research paper introduces a translation evaluation method called Calibrated Parsing Items Evaluation (CPIE hereafter). This evaluation method maximizes translators’ performance through identifying the parsing items with an optimal p-docimology and d-index (item discrimination). This method checks all the possible parses (annotations) in a source text by means of the Brat Visualization Stanford CoreNLP software. CPIE takes a step towards the objectification of translation assessment by allowing evaluators to assess values (impacts) of the items in source texts via docimologically justified parsing items. For this paper, 16 evaluators were recruited to score translation drafts by means of the holistic, analytic, Preselected Items Evaluation (PIE) methods and CPIE method. For the present research paper, “F-Statistics,” “Probability Plot,” “Spearman rho,” and “Regression Variable Plot” were applied to the evaluators’ translation assessments to ensure the degree of validity and reliability of CPIE compared to the holistic, analytic, and PIE methods, respectively. The results indicated that the CPIE method was more consistent and valid in terms of docimologically justified parsing items. The limitations and the possibilities of the CPIE method in web-based platforms were also discussed.

education & educational research,linguistics
A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

Zhihan Jiang,Jinyang Liu,Junjie Huang,Yichen Li,Yintong Huo,Jiazhen Gu,Zhuangbin Chen,Jieming Zhu,Michael R. Lyu

2024-03-23

Abstract:Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.

Software Engineering
Peking: Profiling Syntactic Tree Parsing Techniques for Semantic Graph Parsing

Yantao Du,Fan Zhang,Weiwei Sun,Xiaojun Wan

DOI: https://doi.org/10.3115/v1/s14-2080

2014-01-01

Abstract:Using the SemEval-2014 Task 8 data, we profile the syntactic tree parsing techniques for semantic graph parsing. In particular, we implement different transitionbased and graph-based models, as well as a parser ensembler, and evaluate their effectiveness for semantic dependency parsing. Evaluation gauges how successful data-driven dependency graph parsing can be by applying existing techniques.

Evaluation Reportof the fourth Chinese Parsing Evaluation: CIPS-SIGHAN-ParsEval-2014

Evaluation Reportof the Third Chinese Parsing Evaluation: CIPS-SIGHAN-ParsEval-2012

Chinese Chunk Parsing Evaluation Tasks

Chinese Question Answering System-Oriented Chinese Parsing

SemEval-2012 Task 5: Chinese Semantic Dependency Parsing

NJU-Parser: Achievements on Semantic Dependency Parsing.

A Survey of Semantic Parsing Techniques

An Evaluation of Chinese Human-Computer Dialogue Technology

Building a comprehensive syntactic and semantic corpus of Chinese clinical texts

Overview of the NLPCC 2015 Shared Task: Chinese Word Segmentation and POS Tagging for Micro-blog Texts

Improve Chinese Semantic Dependency Parsing Via Syntactic Dependency Parsing

Special Perceptual Parsing for Chinese Landscape Painting Scene Understanding: a Semantic Segmentation Approach.

The CIPS-SIGHAN CLP 2014 Chinese Word Segmentation Bake-off.

Discourse Representation Structure Parsing for Chinese

Supervised Approaches and Dependency Parsing for Chinese Opinion Analysis at NTCIR-8.

Dependency-Gated Cascade Biaffine Network for Chinese Semantic Dependency Graph Parsing

Overview of the NLPCC 2015 Shared Task: Entity Recognition and Linking in Search Queries.

System Report for CCL24-Eval Task 7: Multi-Error Modeling and Fluency-Targeted Pre-training for Chinese Essay Evaluation

Calibrated Parsing Items Evaluation: a step towards objectifying the translation assessment

A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

Peking: Profiling Syntactic Tree Parsing Techniques for Semantic Graph Parsing