Simple Natural Language Processing Tools for Danish

Leon Derczynski

DOI: https://doi.org/10.48550/arXiv.1906.11608

2019-06-27

Computation and Language

Abstract:This technical note describes a set of baseline tools for automatic processing of Danish text. The tools are machine-learning based, using natural language processing models trained over previously annotated documents. They are maintained at ITU Copenhagen and will always be freely available.

What problem does this paper attempt to address?

DaCy: A Unified Framework for Danish NLP

Kenneth Enevoldsen,Lasse Hansen,Kristoffer Nielbo

DOI: https://doi.org/10.48550/arXiv.2107.05295

2021-07-12

Abstract:Danish natural language processing (NLP) has in recent years obtained considerable improvements with the addition of multiple new datasets and models. However, at present, there is no coherent framework for applying state-of-the-art models for Danish. We present DaCy: a unified framework for Danish NLP built on SpaCy. DaCy uses efficient multitask models which obtain state-of-the-art performance on named entity recognition, part-of-speech tagging, and dependency parsing. DaCy contains tools for easy integration of existing models such as for polarity, emotion, or subjectivity detection. In addition, we conduct a series of tests for biases and robustness of Danish NLP pipelines through augmentation of the test set of DaNE. DaCy large compares favorably and is especially robust to long input lengths and spelling variations and errors. All models except DaCy large display significant biases related to ethnicity while only Polyglot shows a significant gender bias. We argue that for languages with limited benchmark sets, data augmentation can be particularly useful for obtaining more realistic and fine-grained performance estimates. We provide a series of augmenters as a first step towards a more thorough evaluation of language models for low and medium resource languages and encourage further development.

Computation and Language,Machine Learning
DANSK and DaCy 2.6.0: Domain Generalization of Danish Named Entity Recognition

Kenneth Enevoldsen,Emil Trenckner Jessen,Rebekah Baglini

2024-02-28

Abstract:Named entity recognition is one of the cornerstones of Danish NLP, essential for language technology applications within both industry and research. However, Danish NER is inhibited by a lack of available datasets. As a consequence, no current models are capable of fine-grained named entity recognition, nor have they been evaluated for potential generalizability issues across datasets and domains. To alleviate these limitations, this paper introduces: 1) DANSK: a named entity dataset providing for high-granularity tagging as well as within-domain evaluation of models across a diverse set of domains; 2) DaCy 2.6.0 that includes three generalizable models with fine-grained annotation; and 3) an evaluation of current state-of-the-art models' ability to generalize across domains. The evaluation of existing and new models revealed notable performance discrepancies across domains, which should be addressed within the field. Shortcomings of the annotation quality of the dataset and its impact on model training and evaluation are also discussed. Despite these limitations, we advocate for the use of the new dataset DANSK alongside further work on the generalizability within Danish NER.

Computation and Language
Danish Foundation Models

Kenneth Enevoldsen,Lasse Hansen,Dan S. Nielsen,Rasmus A. F. Egebæk,Søren V. Holm,Martin C. Nielsen,Martin Bernstorff,Rasmus Larsen,Peter B. Jørgensen,Malte Højmark-Bertelsen,Peter B. Vahlstrup,Per Møldrup-Dalum,Kristoffer Nielbo

DOI: https://doi.org/10.48550/arXiv.2311.07264

2023-11-13

Computation and Language

Abstract:Large language models, sometimes referred to as foundation models, have transformed multiple fields of research. However, smaller languages risk falling behind due to high training costs and small incentives for large companies to train these models. To combat this, the Danish Foundation Models project seeks to provide and maintain open, well-documented, and high-quality foundation models for the Danish language. This is achieved through broad cooperation with public and private institutions, to ensure high data quality and applicability of the trained models. We present the motivation of the project, the current status, and future perspectives.
ARTIST: ARTificial Intelligence for Simplified Text

Lorenzo Corti,Jie Yang

2023-08-26

Abstract:Complex text is a major barrier for many citizens when accessing public information and knowledge. While often done manually, Text Simplification is a key Natural Language Processing task that aims for reducing the linguistic complexity of a text while preserving the original meaning. Recent advances in Generative Artificial Intelligence (AI) have enabled automatic text simplification both on the lexical and syntactical levels. However, as applications often focus on English, little is understood about the effectiveness of Generative AI techniques on low-resource languages such as Dutch. For this reason, we carry out empirical studies to understand the benefits and limitations of applying generative technologies for text simplification and provide the following outcomes: 1) the design and implementation for a configurable text simplification pipeline that orchestrates state-of-the-art generative text simplification models, domain and reader adaptation, and visualisation modules; 2) insights and lessons learned, showing the strengths of automatic text simplification while exposing the challenges in handling cultural and commonsense knowledge. These outcomes represent a first step in the exploration of Dutch text simplification and shed light on future endeavours both for research and practice.

Computation and Language
Natural Language Processing (Almost) from Scratch

Michael Karlen,K. Kavukcuoglu,J. Weston,L. Bottou,R. Collobert,Pavel P. Kuksa

DOI: https://doi.org/10.5555/1953048.2078186

2011-02-01

Journal of Machine Learning Research

Abstract:We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

Computer Science,Linguistics
Neural Cross-Lingual Transfer and Limited Annotated Data for Named Entity Recognition in Danish

Barbara Plank

DOI: https://doi.org/10.48550/arXiv.2003.02931

2020-03-05

Computation and Language

Abstract:Named Entity Recognition (NER) has greatly advanced by the introduction of deep neural architectures. However, the success of these methods depends on large amounts of training data. The scarcity of publicly-available human-labeled datasets has resulted in limited evaluation of existing NER systems, as is the case for Danish. This paper studies the effectiveness of cross-lingual transfer for Danish, evaluates its complementarity to limited gold data, and sheds light on performance of Danish NER.
CMULAB: An Open-Source Framework for Training and Deployment of Natural Language Processing Models

Zaid Sheikh,Antonios Anastasopoulos,Shruti Rijhwani,Lindia Tjuatja,Robbie Jimerson,Graham Neubig

2024-04-03

Abstract:Effectively using Natural Language Processing (NLP) tools in under-resourced languages requires a thorough understanding of the language itself, familiarity with the latest models and training methodologies, and technical expertise to deploy these models. This could present a significant obstacle for language community members and linguists to use NLP tools. This paper introduces the CMU Linguistic Annotation Backend, an open-source framework that simplifies model deployment and continuous human-in-the-loop fine-tuning of NLP models. CMULAB enables users to leverage the power of multilingual models to quickly adapt and extend existing tools for speech recognition, OCR, translation, and syntactic analysis to new languages, even with limited training data. We describe various tools and APIs that are currently available and how developers can easily add new models/functionality to the framework. Code is available at

Computation and Language
Building Low-Resource NER Models Using Non-Speaker Annotation

Tatiana Tsygankova,Francesca Marini,Stephen Mayhew,Dan Roth

DOI: https://doi.org/10.48550/arXiv.2006.09627

2021-04-27

Abstract:In low-resource natural language processing (NLP), the key problems are a lack of target language training data, and a lack of native speakers to create it. Cross-lingual methods have had notable success in addressing these concerns, but in certain common circumstances, such as insufficient pre-training corpora or languages far from the source language, their performance suffers. In this work we propose a complementary approach to building low-resource Named Entity Recognition (NER) models using ``non-speaker'' (NS) annotations, provided by annotators with no prior experience in the target language. We recruit 30 participants in a carefully controlled annotation experiment with Indonesian, Russian, and Hindi. We show that use of NS annotators produces results that are consistently on par or better than cross-lingual methods built on modern contextual representations, and have the potential to outperform with additional effort. We conclude with observations of common annotation patterns and recommended implementation practices, and motivate how NS annotations can be used in addition to prior methods for improved performance. For more details, <a class="link-external link-http" href="http://cogcomp.org/page/publication_view/941" rel="external noopener nofollow">this http URL</a>

Computation and Language
Natural Language or Not (NLoN) - A Package for Software Engineering Text Analysis Pipeline

Mika V. Mäntylä,Fabio Calefato,Maelick Claes

DOI: https://doi.org/10.1145/3196398.3196444

2018-03-20

Abstract:The use of natural language processing (NLP) is gaining popularity in software engineering. In order to correctly perform NLP, we must pre-process the textual information to separate natural language from other information, such as log messages, that are often part of the communication in software engineering. We present a simple approach for classifying whether some textual input is natural language or not. Although our NLoN package relies on only 11 language features and character tri-grams, we are able to achieve an area under the ROC curve performances between 0.976-0.987 on three different data sources, with Lasso regression from Glmnet as our learner and two human raters for providing ground truth. Cross-source prediction performance is lower and has more fluctuation with top ROC performances from 0.913 to 0.980. Compared with prior work, our approach offers similar performance but is considerably more lightweight, making it easier to apply in software engineering text mining pipelines. Our source code and data are provided as an R-package for further improvements.

Software Engineering
ScandEval: A Benchmark for Scandinavian Natural Language Processing

Dan Saattrup Nielsen

2023-04-03

Abstract:This paper introduces a Scandinavian benchmarking platform, ScandEval, which can benchmark any pretrained model on four different tasks in the Scandinavian languages. The datasets used in two of the tasks, linguistic acceptability and question answering, are new. We develop and release a Python package and command-line interface, scandeval, which can benchmark any model that has been uploaded to the Hugging Face Hub, with reproducible results. Using this package, we benchmark more than 100 Scandinavian or multilingual models and present the results of these in an interactive online leaderboard, as well as provide an analysis of the results. The analysis shows that there is substantial cross-lingual transfer among the Mainland Scandinavian languages (Danish, Swedish and Norwegian), with limited cross-lingual transfer between the group of Mainland Scandinavian languages and the group of Insular Scandinavian languages (Icelandic and Faroese). The benchmarking results also show that the investment in language technology in Norway, Sweden and Denmark has led to language models that outperform massively multilingual models such as XLM-RoBERTa and mDeBERTaV3. We release the source code for both the package and leaderboard.

Computation and Language,Machine Learning
Making and Using AI in the Library: Creating a BERT Model at the National Library of Sweden

Chris Haffenden,Elena Fano,Martin Malmsten,Love Börjeson

DOI: https://doi.org/10.31235/osf.io/k9duq

2022-01-12

Abstract:How can novel AI techniques be made and put to use in the library? Combining methods from data and library science, this article focuses on Natural Language Processing technologies in especially national libraries. It explains how the National Library of Sweden’s collections enabled the development of a new BERT language model for Swedish. It also outlines specific use cases for the model in the context of academic libraries, detailing strategies for how such a model could make digital collections available for new forms of research: from automated classification to enhanced searchability and improved OCR cohesion. Highlighting the potential for cross-fertilizing AI with libraries, the conclusion suggests that while AI may transform the workings of the library, libraries can also have a key role to play in the future development of AI.
The Current State of Finnish NLP

Mika Hämäläinen,Khalid Alnajjar

DOI: https://doi.org/10.48550/arXiv.2109.11326

2021-09-23

Computation and Language

Abstract:There are a lot of tools and resources available for processing Finnish. In this paper, we survey recent papers focusing on Finnish NLP related to many different subcategories of NLP such as parsing, generation, semantics and speech. NLP research is conducted in many different research groups in Finland, and it is frequently the case that NLP tools and models resulting from academic research are made available for others to use on platforms such as Github.
Improved Text Extraction from PDF Documents for Large-Scale Natural Language Processing

Jörg Tiedemann

DOI: https://doi.org/10.1007/978-3-642-54906-9_9

2014-01-01

Abstract:The inability of reliable text extraction from arbitrary documents is often an obstacle for large scale NLP based on resources crawled from the Web. One of the largest problems in the conversion of PDF documents is the detection of the boundaries of common textual units such as paragraphs, sentences and words. PDF is a file format optimized for printing and encapsulates a complete description of the layout of a document including text, fonts, graphics and so on. This paper describes a tool for extracting texts from arbitrary PDF files for the support of large-scale data-driven natural language processing. Our approach combines the benefits of several existing solutions for the conversion of PDF documents to plain text and adds a language-independent post-processing procedure that cleans the output for further linguistic processing. In particular, we use the PDF-rendering libraries pdfXtk, Apache Tika and Poppler in various configurations. From the output of these tools we recover proper boundaries using on-the-fly language models and language-independent extraction heuristics. In our research, we looked especially at publications from the European Union, which constitute a valuable multilingual resource, for example, for training statistical machine translation models. We use our tool for the conversion of a large multilingual database crawled from the EU bookshop with the aim of building parallel corpora. Our experiments show that our conversion software is capable of fixing various common issues leading to cleaner data sets in the end.
Playing with Words at the National Library of Sweden -- Making a Swedish BERT

Martin Malmsten,Love Börjeson,Chris Haffenden

DOI: https://doi.org/10.48550/arXiv.2007.01658

2020-07-03

Abstract:This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsförmedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: <a class="link-external link-https" href="https://github.com/Kungbib/swedish-bert-models" rel="external noopener nofollow">this https URL</a> .

Computation and Language
Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model

Per E Kummervold,Javier de la Rosa,Freddy Wetjen,Svein Arne Brygfjeld

DOI: https://doi.org/10.48550/arXiv.2104.09617

2021-04-19

Computation and Language

Abstract:In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokm{\aa}l and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.
The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change

Dominik Schlechtweg,Shafqat Mumtaz Virk,Pauline Sander,Emma Sköldberg,Lukas Theuer Linke,Tuo Zhang,Nina Tahmasebi,Jonas Kuhn,Sabine Schulte im Walde

2024-02-05

Abstract:We present the DURel tool that implements the annotation of semantic proximity between uses of words into an online, open source interface. The tool supports standardized human annotation as well as computational annotation, building on recent advances with Word-in-Context models. Annotator judgments are clustered with automatic graph clustering techniques and visualized for analysis. This allows to measure word senses with simple and intuitive micro-task judgments between use pairs, requiring minimal preparation efforts. The tool offers additional functionalities to compare the agreement between annotators to guarantee the inter-subjectivity of the obtained judgments and to calculate summary statistics giving insights into sense frequency distributions, semantic variation or changes of senses over time.

Computation and Language,Artificial Intelligence
Natural Language Processing Chains Inside a Cross-lingual Event-Centric Knowledge Pipeline for European Union Under-resourced Languages

Diego Alves,Gaurish Thakkar,Marko Tadić

DOI: https://doi.org/10.48550/arXiv.2010.12433

2020-10-23

Computation and Language

Abstract:This article presents the strategy for developing a platform containing Language Processing Chains for European Union languages, consisting of Tokenization to Parsing, also including Named Entity recognition andwith addition ofSentiment Analysis. These chains are part of the first step of an event-centric knowledge processing pipeline whose aim is to process multilingual media information about major events that can cause an impactin Europe and the rest of the world. Due to the differences in terms of availability of language resources for each language, we have built this strategy in three steps, starting with processing chains for the well-resourced languages and finishing with the development of new modules for the under-resourced ones. In order to classify all European Union official languages in terms of resources, we have analysed the size of annotated corpora as well as the existence of pre-trained models in mainstream Language Processing tools, and we have combined this information with the proposed classification published at META-NETwhitepaper series.
A Survey of Resources and Methods for Natural Language Processing of Serbian Language

Ulfeta A. Marovac,Aldina R. Avdić,Nikola Lj. Milošević

DOI: https://doi.org/10.48550/arXiv.2304.05468

2023-04-11

Computation and Language

Abstract:The Serbian language is a Slavic language spoken by over 12 million speakers and well understood by over 15 million people. In the area of natural language processing, it can be considered a low-resourced language. Also, Serbian is considered a high-inflectional language. The combination of many word inflections and low availability of language resources makes natural language processing of Serbian challenging. Nevertheless, over the past three decades, there have been a number of initiatives to develop resources and methods for natural language processing of Serbian, ranging from developing a corpus of free text from books and the internet, annotated corpora for classification and named entity recognition tasks to various methods and models performing these tasks. In this paper, we review the initiatives, resources, methods, and their availability.
Natural Language Processing: Chances and Challenges in Dentistry

Martha Büttner,Ulf Leser,Lisa Schneider,Falk Schwendicke

DOI: https://doi.org/10.1016/j.jdent.2023.104796

Abstract:Introduction: Natural language processing (NLP) is an intersection between Computer Science and Linguistic which aims to enable machines to process and understand human language. We here summarized applications and limitations of NLP in dentistry. Data and sources: Narrative review. Findings: NLP has evolved increasingly fast. For the dental domain, relevant NLP applications are text classification (e.g., symptom classification) and natural language generation and understanding (e.g., clinical chatbots assisting professionals in office work and patient communication). Analyzing large quantities of text will allow understanding diseases and their trajectories and support a more precise and personalized care. Speech recognition systems may serve as virtual assistants and facilitate automated documentation. However, to date, NLP has rarely been applied in dentistry. Existing research focuses mainly on rule-based solutions for narrow tasks. Technologies such as Recurrent Neural Networks and Transformers have been shown to surpass the language processing capabilities of such rule-based solutions in many fields, but are data-hungry (i.e., rely on large amounts of training data), which limits their application in the dental domain at present. Technologies such as federated or transfer learning or data sharing concepts may allow to overcome this limitation, while challenges in terms of explainability, reproducibility, generalizability and evaluation of NLP in dentistry remain to be resolved for enabling approval of such technologies in medical devices and services. Conclusions: NLP will become a cornerstone of a number of applications in dentistry. The community is called to action to improve the current limitations and foster reliable, high-quality dental NLP. Clinical significance: NLP for text classification (e.g., dental symptom classification) and language generation and understanding (e.g., clinical chatbots, speech recognition) will support administrative tasks in dentistry, provide deeper insights for clinicians and support research and education.
Mapping SNOMED CT Codes to Semi-Structured Texts via an NLP Pipeline

Sebastian Kunz,Cyril Zgraggen,Murat Sariyar

DOI: https://doi.org/10.3233/SHTI220747

2022-06-29

Abstract:In the project presented here, we used NLP tools for annotating German medical trainings documents with SNOMED CT codes. Following research question was addressed: Is it possible to automate the annotation of training documents with an NLP pipeline especially designed for this task but requiring translation into English? The goal of our stakeholder, an institution responsible for the continuing education of physicians, was to facilitate the switch between different medical trainings programs by coding the same requirement with the same SNOMED CT code, even if the wording is different. We first describe how we chose the concrete NLP tools, after which the concrete steps for implementing our prototype are outlined: the NLP pipeline construction, the implementation, and the validation. We infer three important lessons from our results: (i) self-supervision is no free lunch and should be based on a sophisticated task, (ii) the translation via DeepL can be too context-dependent for a peculiar use case, and (iii) ontology extraction can increase efficiency as well as accuracy.

Simple Natural Language Processing Tools for Danish

DaCy: A Unified Framework for Danish NLP

DANSK and DaCy 2.6.0: Domain Generalization of Danish Named Entity Recognition

Danish Foundation Models

ARTIST: ARTificial Intelligence for Simplified Text

Natural Language Processing (Almost) from Scratch

Neural Cross-Lingual Transfer and Limited Annotated Data for Named Entity Recognition in Danish

CMULAB: An Open-Source Framework for Training and Deployment of Natural Language Processing Models

Building Low-Resource NER Models Using Non-Speaker Annotation

Natural Language or Not (NLoN) - A Package for Software Engineering Text Analysis Pipeline

ScandEval: A Benchmark for Scandinavian Natural Language Processing

Making and Using AI in the Library: Creating a BERT Model at the National Library of Sweden

The Current State of Finnish NLP

Improved Text Extraction from PDF Documents for Large-Scale Natural Language Processing

Playing with Words at the National Library of Sweden -- Making a Swedish BERT

Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model

The DURel Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change

Natural Language Processing Chains Inside a Cross-lingual Event-Centric Knowledge Pipeline for European Union Under-resourced Languages

A Survey of Resources and Methods for Natural Language Processing of Serbian Language

Natural Language Processing: Chances and Challenges in Dentistry

Mapping SNOMED CT Codes to Semi-Structured Texts via an NLP Pipeline