Abstract:Background: Transformer is an attention-based architecture proven the state-of-the-art model in natural language processing (NLP). To reduce the difficulty of beginning to use transformer-based models in medical language understanding and expand the capability of the scikit-learn toolkit in deep learning, we proposed an easy to learn Python toolkit named transformers-sklearn. By wrapping the interfaces of transformers in only three functions (i.e., fit, score, and predict), transformers-sklearn combines the advantages of the transformers and scikit-learn toolkits. Methods: In transformers-sklearn, three Python classes were implemented, namely, BERTologyClassifier for the classification task, BERTologyNERClassifier for the named entity recognition (NER) task, and BERTologyRegressor for the regression task. Each class contains three methods, i.e., fit for fine-tuning transformer-based models with the training dataset, score for evaluating the performance of the fine-tuned model, and predict for predicting the labels of the test dataset. transformers-sklearn is a user-friendly toolkit that (1) Is customizable via a few parameters (e.g., model_name_or_path and model_type), (2) Supports multilingual NLP tasks, and (3) Requires less coding. The input data format is automatically generated by transformers-sklearn with the annotated corpus. Newcomers only need to prepare the dataset. The model framework and training methods are predefined in transformers-sklearn. Results: We collected four open-source medical language datasets, including TrialClassification for Chinese medical trial text multi label classification, BC5CDR for English biomedical text name entity recognition, DiabetesNER for Chinese diabetes entity recognition and BIOSSES for English biomedical sentence similarity estimation. In the four medical NLP tasks, the average code size of our script is 45 lines/task, which is one-sixth the size of transformers' script. The experimental results show that transformers-sklearn based on pretrained BERT models achieved macro F1 scores of 0.8225, 0.8703 and 0.6908, respectively, on the TrialClassification, BC5CDR and DiabetesNER tasks and a Pearson correlation of 0.8260 on the BIOSSES task, which is consistent with the results of transformers. Conclusions: The proposed toolkit could help newcomers address medical language understanding tasks using the scikit-learn coding style easily. The code and tutorials of transformers-sklearn are available at https://doi.org/10.5281/zenodo.4453803 . In future, more medical language understanding tasks will be supported to improve the applications of transformers_sklearn.

TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks

Transformers-sklearn: a toolkit for medical language understanding with transformer-based models

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis

rerankers: A Lightweight Python Library to Unify Ranking Methods

Rank over Class: The Untapped Potential of Ranking in Natural Language Processing

LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models

Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval

Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Transformers4NewsRec: A Transformer-based News Recommendation Framework

KronA: Parameter Efficient Tuning with Kronecker Adapter

RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers

Better Explain Transformers by Illuminating Important Information

Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval

Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings

A concise analysis of low-rank adaptation

PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance

Pretrained Transformers for Text Ranking: BERT and Beyond

LaoPLM: Pre-trained Language Models for Lao

Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models