Abstract:We describe an annotation initiative to capture the scholarly contributions in natural language processing (NLP) articles, particularly, for the articles that discuss machine learning (ML) approaches for various information extraction tasks. We develop the annotation task based on a pilot annotation exercise on 50 NLP-ML scholarly articles presenting contributions to five information extraction tasks 1. machine translation, 2. named entity recognition, 3. question answering, 4. relation classification, and 5. text classification. In this article, we describe the outcomes of this pilot annotation phase. Through the exercise we have obtained an annotation methodology; and found ten core information units that reflect the contribution of the NLP-ML scholarly investigations. The resulting annotation scheme we developed based on these information units is called NLPContributions. The overarching goal of our endeavor is four-fold: 1) to find a systematic set of patterns of subject-predicate-object statements for the semantic structuring of scholarly contributions that are more or less generically applicable for NLP-ML research articles; 2) to apply the discovered patterns in the creation of a larger annotated dataset for training machine readers of research contributions; 3) to ingest the dataset into the Open Research Knowledge Graph (ORKG) infrastructure as a showcase for creating user-friendly state-of-the-art overviews; 4) to integrate the machine readers into the ORKG to assist users in the manual curation of their respective article contributions. We envision that the NLPContributions methodology engenders a wider discussion on the topic toward its further refinement and development. Our pilot annotated dataset of 50 NLP-ML scholarly articles according to the NLPContributions scheme is openly available to the research community at <a class="link-external link-https" href="https://doi.org/10.25835/0019761" rel="external noopener nofollow">this https URL</a>.

An Annotated Corpus for Machine Reading of Instructions in Wet Lab Protocols

Transcription Between Human-Readable Synthetic Descriptions and Machine-Executable Instructions: an Application of the Latest Pre-Training Technology

The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures

Corpus Annotation for Parser Evaluation

ProtoCode: Leveraging Large Language Models for Automated Generation of Machine-Readable Protocols from Scientific Publications

A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

Large language model enhanced corpus of CO 2 reduction electrocatalysts and synthesis procedures

REInstruct: Building Instruction Data from Unlabeled Corpus

Towards Constructing a Corpus for Studying the Effects of Treatments and Substances Reported in PubMed Abstracts

Annotated Chemical Patent Corpus: A Gold Standard for Text Mining

NLPContributions: An Annotation Scheme for Machine Reading of Scholarly Contributions in Natural Language Processing Literature

TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation

A corpus for plant-chemical relationships in the biomedical domain

A large annotated corpus for learning natural language inference

Annotating the Contemporary Chinese Corpus

Pipelines for Procedural Information Extraction from Scientific Literature: Towards Recipes using Machine Learning and Data Science

A Manually Annotated Chinese Corpus for Non-task-oriented Dialogue Systems

FunctionAnnotator, a versatile and efficient web tool for non-model organism annotation

Corpus Conversion Service: A machine learning platform to ingest documents at scale [Poster abstract]