Abstract:Background: In response to the intricate language, specialized terminology outside everyday life, and the frequent presence of abbreviations and acronyms inherent in health care text data, domain adaptation techniques have emerged as crucial to transformer-based models. This refinement in the knowledge of the language models (LMs) allows for a better understanding of the medical textual data, which results in an improvement in medical downstream tasks, such as information extraction (IE). We have identified a gap in the literature regarding health care LMs. Therefore, this study presents a scoping literature review investigating domain adaptation methods for transformers in health care, differentiating between English and non-English languages, focusing on Portuguese. Most specifically, we investigated the development of health care LMs, with the aim of comparing Portuguese with other more developed languages to guide the path of a non–English-language with fewer resources. Objective: This study aimed to research health care IE models, regardless of language, to understand the efficacy of transformers and what are the medical entities most commonly extracted. Methods: This scoping review was conducted using the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) methodology on Scopus and Web of Science Core Collection databases. Only studies that mentioned the creation of health care LMs or health care IE models were included, while large language models (LLMs) were excluded. The latest were not included since we wanted to research LMs and not LLMs, which are architecturally different and have distinct purposes. Results: Our search query retrieved 137 studies, 60 of which met the inclusion criteria, and none of them were systematic literature reviews. English and Chinese are the languages with the most health care LMs developed. These languages already have disease-specific LMs, while others only have general–health care LMs. European Portuguese does not have any public health care LM and should take examples from other languages to develop, first, general-health care LMs and then, in an advanced phase, disease-specific LMs. Regarding IE models, transformers were the most commonly used method, and named entity recognition was the most popular topic, with only a few studies mentioning Assertion Status or addressing medical lexical problems. The most extracted entities were diagnosis, posology, and symptoms. Conclusions: The findings indicate that domain adaptation is beneficial, achieving better results in downstream tasks. Our analysis allowed us to understand that the use of transformers is more developed for the English and Chinese languages. European Portuguese lacks relevant studies and should draw examples from other non-English languages to develop these models and drive progress in AI. Health care professionals could benefit from highlighting medically relevant information and optimizing the reading of the textual data, or this information could be used to create patient medical timelines, allowing for profiling.

When Is Knowledge Ripe for Primary Care?

Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain

Advancing Italian biomedical information extraction with transformers-based models: Methodological insights and multicenter practical application

A Review of Recent Work in Transfer Learning and Domain Adaptation for Natural Language Processing of Electronic Health Records

Domain Adaptation for Medical Text Translation Using Web Resources

Who is still using lip implants?

Machine Translation in the Covid domain: an English-Irish case study for LoResMT 2021

Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings

LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation

Translating Terminological Expressions in Knowledge Bases with Neural Machine Translation

Finding the Right Recipe for Low Resource Domain Adaptation in Neural Machine Translation

Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

Health Care Language Models and Their Fine-Tuning for Information Extraction: Scoping Review

LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language

From Priest to Doctor: Domain Adaptaion for Low-Resource Neural Machine Translation

Early Rehabilitation in Elderly after Arthroplasty versus Internal Fixation for Unstable Intertrochanteric Fractures of Femur: Systematic Review and Meta-Analysis

Development of a Natural Language Processing (NLP) model to automatically extract clinical data from electronic health records: results from an Italian comprehensive stroke center

Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning