Abstract:Since the emergence of the Transformer architecture, language model development has increased, driven by their promising potential. However, releasing these models into production requires properly understanding their behavior, particularly in sensitive domains such as medicine. Despite this need, the medical literature still lacks technical assessments of pre-trained language models, which are especially valuable in resource-constrained settings in terms of computational power or limited budget. To address this gap, we provide a comprehensive survey of language models in the medical domain. In addition, we selected a subset of these models for thorough evaluation, focusing on classification and text generation tasks. Our subset encompasses 53 models, ranging from 110 million to 13 billion parameters, spanning the three families of Transformer-based models and from diverse knowledge domains. This study employs a series of approaches for text classification together with zero-shot prompting instead of model training or fine-tuning, which closely resembles the limited resource setting in which many users of language models find themselves. Encouragingly, our findings reveal remarkable performance across various tasks and datasets, underscoring the latent potential of certain models to contain medical knowledge, even without domain specialization. Consequently, our study advocates for further exploration of model applications in medical contexts, particularly in resource-constrained settings. The code is available on <a class="link-external link-https" href="https://github.com/anpoc/Language-models-in-medicine" rel="external noopener nofollow">this https URL</a>.

Pre-training Polish Transformer-based Language Models at Scale

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish

KLEJ: Comprehensive Benchmark for Polish Language Understanding

Universal Language Model Fine-Tuning with Subword Tokenization for Polish

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

Multilingual Transformers for Product Matching -- Experiments and a New Benchmark in Polish

Larger-Scale Transformers for Multilingual Masked Language Modeling

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Opt: Open pre-trained transformer language models

A Family of Pretrained Transformer Language Models for Russian

Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching

The Dark Side of the Language: Pre-trained Transformers in the DarkNet

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings

Evaluation of Sentence Representations in Polish

Real-Time Sentiment Analysis for Polish Dialog Systems Using MT as Pivot

Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining