Abstract:To support nanocrystal device development, we have been working on a computational framework to utilize information in research papers on nanocrystal devices. We developed an annotated corpus called " NaDev" (Nanocrystal Device Development) for this purpose. We also proposed an automatic information extraction system called "NaDevEx" (Nanocrystal Device Automatic Information Extraction Framework). NaDevEx aims at extracting information from research papers on nanocrystal devices using the NaDev corpus and machine-learning techniques. However, the characteristics of NaDevEx were not examined in detail. In this paper, we conduct system evaluation experiments for NaDevEx using the NaDev corpus. We discuss three main issues: system performance, compared with human annotators; the effect of paper type (synthesis or characterization) on system performance; and the effects of domain knowledge features (e.g., a chemical named entity recognition system and list of names of physical quantities) on system performance. We found that overall system performance was 89% in precision and 69% in recall. If we consider identification of terms that intersect with correct terms for the same information category as the correct identification, i.e., loose agreement (in many cases, we can find that appropriate head nouns such as temperature or pressure loosely match between two terms), the overall performance is 95% in precision and 74% in recall. The system performance is almost comparable with results of human annotators for information categories with rich domain knowledge information (source material). However, for other information categories, given the relatively large number of terms that exist only in one paper, recall of individual information categories is not high (39-73%); however, precision is better (75-97%). The average performance for synthesis papers is better than that for characterization papers because of the lack of training examples for characterization papers. Based on these results, we discuss future research plans for improving the performance of the system.

CRYSTAL: Inducing a Conceptual Dictionary

KICE: A Knowledge Consolidation and Expansion Framework for Relation Extraction.

Computer system and method for establishing concept knowledge according to machine readable dictionary

To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models

Bootstrapping Information Extraction Via Conceptualization

Discovering Chinese Concept-In-Corpus

Guided Attention Network for Concept Extraction

Knowledge graphs for empirical concept retrieval

Framework for automatic information extraction from research papers on nanocrystal devices

Three-loop renormalization group analysis of a complex model with stable fixed point: Critical exponents up to $ε^3$ and $ε^4$

Automated concept-level information extraction to reduce the need for custom software and rules development

CrysText: A Generative AI Approach for Text-Conditioned Crystal Structure Generation using LLM

Unsupervised Extraction of Representative Concepts from Scientific Literature

Learning to Understand Phrases by Embedding the Dictionary

Using WordNet in Conceptual Query Expansion

Con-CDVAE: A method for the conditional generation of crystal structures

Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

Introducing an Interpretable Deep Learning Approach to Domain-Specific Dictionary Creation: A Use Case for Conflict Prediction

Vocab-Expander: A System for Creating Domain-Specific Vocabularies Based on Word Embeddings

CliNER 2.0: Accessible and Accurate Clinical Concept Extraction

Development of a Conceptual Structure for a Domain-Specific Corpus