Abstract:The most important way of conveying new findings in biomedical research is scientific publication. Extraction of protein-protein interactions (PPIs) reported in scientific publications is one of the core topics of text mining in the life sciences. Recently, a new class of such methods has been proposed - convolution kernels that identify PPIs using deep parses of sentences. However, comparing published results of different PPI extraction methods is impossible due to the use of different evaluation corpora, different evaluation metrics, different tuning procedures, etc. In this paper, we study whether the reported performance metrics are robust across different corpora and learning settings and whether the use of deep parsing actually leads to an increase in extraction quality. Our ultimate goal is to identify the one method that performs best in real-life scenarios, where information extraction is performed on unseen text and not on specifically prepared evaluation data. We performed a comprehensive benchmarking of nine different methods for PPI extraction that use convolution kernels on rich linguistic information. Methods were evaluated on five different public corpora using cross-validation, cross-learning, and cross-corpus evaluation. Our study confirms that kernels using dependency trees generally outperform kernels based on syntax trees. However, our study also shows that only the best kernel methods can compete with a simple rule-based approach when the evaluation prevents information leakage between training and test corpora. Our results further reveal that the F-score of many approaches drops significantly if no corpus-specific parameter optimization is applied and that methods reaching a good AUC score often perform much worse in terms of F-score. We conclude that for most kernels no sensible estimation of PPI extraction performance on new text is possible, given the current heterogeneity in evaluation data. Nevertheless, our study shows that three kernels are clearly superior to the other methods.

Improving Kernel-based Protein-Protein Interaction Extraction by Unsupervised Word Representation.

Feature generation and representations for protein-protein interaction classification.

Extracting Protein-Protein Interactions (PPIs) from Biomedical Literature using Attention-based Relational Context Information

Protein-protein interaction extraction from bio-literature with compact features and data sampling strategy

A hybrid method for extraction of protein-protein interactions from literature

A Deep Learning Approach with Deep Contextualized Word Representations for Chemical–Protein Interaction Extraction from Biomedical Literature

Improving Neural Protein-Protein Interaction Extraction with Knowledge Selection

Protein-protein Interaction Extraction Based on Self-Training

Using Maximum Entropy Model To Extract Protein-Protein Interaction Information From Biomedical Literature

Automatic Extraction of Protein Interaction in Literature

Classifying protein-protein interaction articles using word and syntactic features

Protein-protein Interaction Relation Extraction Based on Multigranularity Semantic Fusion.

Feature Assisted bi-directional LSTM Model for Protein-Protein Interaction Identification from Biomedical Texts

PPI-IRO: a Two-Stage Method for Protein-Protein Interaction Extraction Based on Interaction Relation Ontology.

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

An Empirical Study of Features Fusion Techniques for Protein-Protein Interaction Prediction

The Role of Dependency Information for Protein-Protein Interaction Extraction

Uncovering protein interaction in abstracts and text using a novel linear model and word proximity networks

An Active Transfer Learning Framework For Protein-Protein Interaction Extraction

Advances in Computational Methods for Protein–Protein Interaction Prediction

A comprehensive benchmark of kernel methods to extract protein-protein interactions from literature