RuBia: A Russian Language Bias Detection Dataset

Veronika Grigoreva,Anastasiia Ivanova,Ilseyar Alimova,Ekaterina Artemova

2024-03-26

Abstract:Warning: this work contains upsetting or disturbing content.

Computation and Language

What problem does this paper attempt to address?

RuBQ: A Russian Dataset for Question Answering over Wikidata

Vladislav Korablinov,Pavel Braslavski

DOI: https://doi.org/10.48550/arXiv.2005.10659

2020-05-21

Computation and Language

Abstract:The paper presents RuBQ, the first Russian knowledge base question answering (KBQA) dataset. The high-quality dataset consists of 1,500 Russian questions of varying complexity, their English machine translations, SPARQL queries to Wikidata, reference answers, as well as a Wikidata sample of triples containing entities with Russian labels. The dataset creation started with a large collection of question-answer pairs from online quizzes. The data underwent automatic filtering, crowd-assisted entity linking, automatic generation of SPARQL queries, and their subsequent in-house verification.
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian

Tatiana Shamardina,Vladislav Mikhailov,Daniil Chernianskii,Alena Fenogenova,Marat Saidov,Anastasiya Valeeva,Tatiana Shavrina,Ivan Smurov,Elena Tutubalina,Ekaterina Artemova

DOI: https://doi.org/10.28995/2075-7182-2022-21-497-511

2022-06-03

Abstract:We present the shared task on artificial text detection in Russian, which is organized as a part of the Dialogue Evaluation initiative, held in 2022. The shared task dataset includes texts from 14 text generators, i.e., one human writer and 13 text generative models fine-tuned for one or more of the following generation tasks: machine translation, paraphrase generation, text summarization, text simplification. We also consider back-translation and zero-shot generation approaches. The human-written texts are collected from publicly available resources across multiple domains. The shared task consists of two sub-tasks: (i) to determine if a given text is automatically generated or written by a human; (ii) to identify the author of a given text. The first task is framed as a binary classification problem. The second task is a multi-class classification problem. We provide count-based and BERT-based baselines, along with the human evaluation on the first sub-task. A total of 30 and 8 systems have been submitted to the binary and multi-class sub-tasks, correspondingly. Most teams outperform the baselines by a wide margin. We publicly release our codebase, human evaluation results, and other materials in our GitHub repository (<a class="link-external link-https" href="https://github.com/dialogue-evaluation/RuATD" rel="external noopener nofollow">this https URL</a>).

Computation and Language
RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Sergey Smetanin

DOI: https://doi.org/10.7717/peerj-cs.1039

2022-07-19

PeerJ Computer Science

Abstract:The Russian language is still not as well-resourced as English, especially in the field of sentiment analysis of Twitter content. Though several sentiment analysis datasets of tweets in Russia exist, they all are either automatically annotated or manually annotated by one annotator. Thus, there is no inter-annotator agreement, or annotation may be focused on a specific domain. In this article, we present RuSentiTweet, a new sentiment analysis dataset of general domain tweets in Russian. RuSentiTweet is currently the largest in its class for Russian, with 13,392 tweets manually annotated with moderate inter-rater agreement into five classes: Positive, Neutral, Negative, Speech Act, and Skip. As a source of data, we used Twitter Stream Grab, a historical collection of tweets obtained from the general Twitter API stream, which provides a 1% sample of the public tweets. Additionally, we released a RuBERT-based sentiment classification model that achieved F1 = 0.6594 on the test subset.

computer science, information systems, artificial intelligence, theory & methods
Reducing Unintended Identity Bias in Russian Hate Speech Detection

Nadezhda Zueva,Madina Kabirova,Pavel Kalaidin

DOI: https://doi.org/10.48550/arXiv.2010.11666

2020-10-22

Abstract:Toxicity has become a grave problem for many online communities and has been growing across many languages, including Russian. Hate speech creates an environment of intimidation, discrimination, and may even incite some real-world violence. Both researchers and social platforms have been focused on developing models to detect toxicity in online communication for a while now. A common problem of these models is the presence of bias towards some words (e.g. woman, black, jew) that are not toxic, but serve as triggers for the classifier due to model caveats. In this paper, we describe our efforts towards classifying hate speech in Russian, and propose simple techniques of reducing unintended bias, such as generating training data with language models using terms and words related to protected identities as context and applying word dropout to such words.

Computation and Language,Artificial Intelligence
RuSemShift: a dataset of historical lexical semantic change in Russian

Julia Rodina,Andrey Kutuzov

DOI: https://doi.org/10.48550/arXiv.2010.06436

2020-10-13

Abstract:We present RuSemShift, a large-scale manually annotated test set for the task of semantic change modeling in Russian for two long-term time period pairs: from the pre-Soviet through the Soviet times and from the Soviet through the post-Soviet times. Target words were annotated by multiple crowd-source workers. The annotation process was organized following the DURel framework and was based on sentence contexts extracted from the Russian National Corpus. Additionally, we report the performance of several distributional approaches on RuSemShift, achieving promising results, which at the same time leave room for other researchers to improve.

Computation and Language
BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service

Anna Kołos,Inez Okulska,Kinga Głąbińska,Agnieszka Karlińska,Emilia Wiśnios,Paweł Ellerik,Andrzej Prałat

2024-03-26

Abstract:Since the Internet is flooded with hate, it is one of the main tasks for NLP experts to master automated online content moderation. However, advancements in this field require improved access to publicly available accurate and non-synthetic datasets of social media content. For the Polish language, such resources are very limited. In this paper, we address this gap by presenting a new open dataset of offensive social media content for the Polish language. The dataset comprises content from <a class="link-external link-http" href="http://Wykop.pl" rel="external noopener nofollow">this http URL</a>, a popular online service often referred to as the "Polish Reddit", reported by users and banned in the internal moderation process. It contains a total of 691,662 posts and comments, evenly divided into two categories: "harmful" and "neutral" ("non-harmful"). The anonymized subset of the BAN-PL dataset consisting on 24,000 pieces (12,000 for each class), along with preprocessing scripts have been made publicly available. Furthermore the paper offers valuable insights into real-life content moderation processes and delves into an analysis of linguistic features and content characteristics of the dataset. Moreover, a comprehensive anonymization procedure has been meticulously described and applied. The prevalent biases encountered in similar datasets, including post-moderation and pre-selection biases, are also discussed.

Computation and Language
Detecting Human Rights Violations on Social Media during Russia-Ukraine War

Poli Nemkova,Solomon Ubani,Suleyman Olcay Polat,Nayeon Kim,Rodney D. Nielsen

DOI: https://doi.org/10.48550/arXiv.2306.05370

2023-06-06

Computers and Society

Abstract:The present-day Russia-Ukraine military conflict has exposed the pivotal role of social media in enabling the transparent and unbridled sharing of information directly from the frontlines. In conflict zones where freedom of expression is constrained and information warfare is pervasive, social media has emerged as an indispensable lifeline. Anonymous social media platforms, as publicly available sources for disseminating war-related information, have the potential to serve as effective instruments for monitoring and documenting Human Rights Violations (HRV). Our research focuses on the analysis of data from Telegram, the leading social media platform for reading independent news in post-Soviet regions. We gathered a dataset of posts sampled from 95 public Telegram channels that cover politics and war news, which we have utilized to identify potential occurrences of HRV. Employing a mBERT-based text classifier, we have conducted an analysis to detect any mentions of HRV in the Telegram data. Our final approach yielded an $F_2$ score of 0.71 for HRV detection, representing an improvement of 0.38 over the multilingual BERT base model. We release two datasets that contains Telegram posts: (1) large corpus with over 2.3 millions posts and (2) annotated at the sentence-level dataset to indicate HRVs. The Telegram posts are in the context of the Russia-Ukraine war. We posit that our findings hold significant implications for NGOs, governments, and researchers by providing a means to detect and document possible human rights violations.
CoRAL: a Context-aware Croatian Abusive Language Dataset

Ravi Shekhar,Mladen Karan,Matthew Purver

DOI: https://doi.org/10.48550/arXiv.2211.06053

2022-11-11

Abstract:In light of unprecedented increases in the popularity of the internet and social media, comment moderation has never been a more relevant task. Semi-automated comment moderation systems greatly aid human moderators by either automatically classifying the examples or allowing the moderators to prioritize which comments to consider first. However, the concept of inappropriate content is often subjective, and such content can be conveyed in many subtle and indirect ways. In this work, we propose CoRAL -- a language and culturally aware Croatian Abusive dataset covering phenomena of implicitness and reliance on local and global context. We show experimentally that current models degrade when comments are not explicit and further degrade when language skill and context knowledge are required to interpret the comment.

Computation and Language
Uncovering and Categorizing Social Biases in Text-to-SQL.

Yan Liu,Yan Gao,Zhe Su,Xiaokang Chen,Elliott Ash,Jian-Guang Lou

DOI: https://doi.org/10.18653/v1/2023.acl-long.759

2023-01-01

Abstract:Content Warning: This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups. Large pre-trained language models are acknowledged to carry social biases towards different demographics, which can further amplify existing stereotypes in our society and cause even more harm. Text-to-SQL is an important task, models of which are mainly adopted by administrative industries, where unfair decisions may lead to catastrophic consequences. However, existing Text-to-SQL models are trained on clean, neutral datasets, such as Spider and WikiSQL. This, to some extent, cover up social bias in models under ideal conditions, which nevertheless may emerge in real application scenarios. In this work, we aim to uncover and categorize social biases in Text-to-SQL models. We summarize the categories of social biases that may occur in structured data for Text-to-SQL models. We build test benchmarks and reveal that models with similar task accuracy can contain social biases at very different rates. We show how to take advantage of our methodology to uncover and assess social biases in the downstream Text-to-SQL task. We will release our code and data.
RuCoLA: Russian Corpus of Linguistic Acceptability

Vladislav Mikhailov,Tatiana Shamardina,Max Ryabinin,Alena Pestova,Ivan Smurov,Ekaterina Artemova

DOI: https://doi.org/10.18653/v1/2022.emnlp-main.348

2022-10-24

Abstract:Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers. However, the application scope of LA in languages other than English is limited due to the lack of high-quality resources. To this end, we introduce the Russian Corpus of Linguistic Acceptability (RuCoLA), built from the ground up under the well-established binary LA approach. RuCoLA consists of $9.8$k in-domain sentences from linguistic publications and $3.6$k out-of-domain sentences produced by generative models. The out-of-domain set is created to facilitate the practical use of acceptability for improving language generation. Our paper describes the data collection protocol and presents a fine-grained analysis of acceptability classification experiments with a range of baseline approaches. In particular, we demonstrate that the most widely used language models still fall behind humans by a large margin, especially when detecting morphological and semantic errors. We release RuCoLA, the code of experiments, and a public leaderboard (<a class="link-external link-http" href="http://rucola-benchmark.com" rel="external noopener nofollow">this http URL</a>) to assess the linguistic competence of language models for Russian.

Computation and Language
OLID-BR: offensive language identification dataset for Brazilian Portuguese

Douglas Trajano,Rafael H. Bordini,Renata Vieira

DOI: https://doi.org/10.1007/s10579-023-09657-0

2023-05-04

Language Resources and Evaluation

Abstract:Social media has revolutionized the manner in which our society is interconnected. While this extensive connectivity offers numerous benefits, it is also accompanied by significant drawbacks, particularly in terms of the proliferation of fake news and the vast dissemination of hate speech. Identifying offensive comments is a critical task for ensuring the safety of users, which is why industry and academia have been working on developing solutions to this problem. Prior research on hate speech detection has predominantly focused on the English language, with few studies devoted to other languages such as Portuguese. This paper introduces the Offensive Language Identification Dataset for Brazilian Portuguese (OLID-BR), a high-quality NLP dataset for offensive language detection, which we make publicly available. The dataset contains 6,354 (extendable to 13,538) comments labeled using a fine-grained three-layer annotation schema compatible with datasets in other languages, which allows the training of multilingual/cross-lingual models. The five NLP tasks available in OLID-BR allow the detection of offensive comments, the classification of the types of offenses such as racism, LGBTQphobia, sexism, xenophobia, and so on, the identification of the type and the target of offensive comments, and the extraction of toxic spans of offensive comments. All those tasks can enhance the capabilities of content moderation systems by providing deep contextual analysis or highlighting the spans that make a text toxic. We further experiment with and evaluate the dataset using state-of-the-art BERT-based and NER models, which demonstrates the usefulness of OLID-BR for the development of toxicity detection systems for Portuguese texts.

computer science, interdisciplinary applications
RuMedBench: A Russian Medical Language Understanding Benchmark

Pavel Blinov,Arina Reshetnikova,Aleksandr Nesterov,Galina Zubkova,Vladimir Kokh

DOI: https://doi.org/10.48550/arXiv.2201.06499

2022-01-17

Computation and Language

Abstract:The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural language inference, named entity recognition) on a number of novel text sets. Given the sensitive nature of the data in healthcare, such a benchmark partially closes the problem of Russian medical dataset absence. We prepare the unified format labeling, data split, and evaluation metrics for new tasks. The remaining tasks are from existing datasets with a few modifications. A single-number metric expresses a model's ability to cope with the benchmark. Moreover, we implement several baseline models, from simple ones to neural networks with transformer architecture, and release the code. Expectedly, the more advanced models yield better performance, but even a simple model is enough for a decent result in some tasks. Furthermore, for all tasks, we provide a human evaluation. Interestingly the models outperform humans in the large-scale classification tasks. However, the advantage of natural intelligence remains in the tasks requiring more knowledge and reasoning.
Dataset for Automatic Summarization of Russian News

Ilya Gusev

DOI: https://doi.org/10.1007/978-3-030-59082-6_9

2021-10-05

Abstract:Automatic text summarization has been studied in a variety of domains and languages. However, this does not hold for the Russian language. To overcome this issue, we present Gazeta, the first dataset for summarization of Russian news. We describe the properties of this dataset and benchmark several extractive and abstractive models. We demonstrate that the dataset is a valid task for methods of text summarization for Russian. Additionally, we prove the pretrained mBART model to be useful for Russian text summarization.

Computation and Language
A Study on Bias Detection and Classification in Natural Language Processing

Ana Sofia Evans,Helena Moniz,Luísa Coheur

2024-08-14

Abstract:Human biases have been shown to influence the performance of models and algorithms in various fields, including Natural Language Processing. While the study of this phenomenon is garnering focus in recent years, the available resources are still relatively scarce, often focusing on different forms or manifestations of biases. The aim of our work is twofold: 1) gather publicly-available datasets and determine how to better combine them to effectively train models in the task of hate speech detection and classification; 2) analyse the main issues with these datasets, such as scarcity, skewed resources, and reliance on non-persistent data. We discuss these issues in tandem with the development of our experiments, in which we show that the combinations of different datasets greatly impact the models' performance.

Computation and Language,Artificial Intelligence
RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

Ekaterina Taktasheva,Maxim Bazhukov,Kirill Koncha,Alena Fenogenova,Ekaterina Artemova,Vladislav Mikhailov

2024-10-02

Abstract:Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and carefully curating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used language models for Russian are sensitive to morphological and agreement-oriented contrasts but fall behind humans on phenomena requiring understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available.

Computation and Language
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

Vera Neplenbroek,Arianna Bisazza,Raquel Fernández

2024-07-17

Abstract:Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at <a class="link-external link-https" href="https://github.com/Veranep/MBBQ" rel="external noopener nofollow">this https URL</a>.

Computation and Language
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation

Aida Mostafazadeh Davani,Mark Díaz,Dylan Baker,Vinodkumar Prabhakaran

2024-04-17

Abstract:While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examined this issue are often situated in the Western context, and solely document differences across age, gender, or racial groups. As a result, NLP research on subjectivity have overlooked the fact that individuals within demographic groups may hold diverse values, which can influence their perceptions beyond their group norms. To effectively incorporate these considerations into NLP pipelines, we need datasets with extensive parallel annotations from various social and cultural groups. In this paper we introduce the \dataset dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K sentences annotated by a pool of over 4k annotators, balanced across gender and age, from across 21 countries, representing eight geo-cultural regions. The dataset contains annotators' moral values captured along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators' perceptions that are shaped by individual moral values, offering crucial insights for building pluralistic, culturally sensitive NLP models.

Computation and Language
BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented Generation

Bryan Li,Samar Haider,Fiona Luo,Adwait Agashe,Chris Callison-Burch

2024-10-02

Abstract:Large language models excel at creative generation but continue to struggle with the issues of hallucination and bias. While retrieval-augmented generation (RAG) provides a framework for grounding LLMs' responses in accurate and up-to-date information, it still raises the question of bias: which sources should be selected for inclusion in the context? And how should their importance be weighted? In this paper, we study the challenge of cross-lingual RAG and present a dataset to investigate the robustness of existing systems at answering queries about geopolitical disputes, which exist at the intersection of linguistic, cultural, and political boundaries. Our dataset is sourced from Wikipedia pages containing information relevant to the given queries and we investigate the impact of including additional context, as well as the composition of this context in terms of language and source, on an LLM's response. Our results show that existing RAG systems continue to be challenged by cross-lingual use cases and suffer from a lack of consistency when they are provided with competing information in multiple languages. We present case studies to illustrate these issues and outline steps for future research to address these challenges. We make our dataset and code publicly available at <a class="link-external link-https" href="https://github.com/manestay/bordIRlines" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Measuring Gender Bias in West Slavic Language Models

Sandra Martinková,Karolina Stańczak,Isabelle Augenstein

DOI: https://doi.org/10.48550/arXiv.2304.05783

2023-04-12

Computation and Language

Abstract:Pre-trained language models have been known to perpetuate biases from the underlying datasets to downstream tasks. However, these findings are predominantly based on monolingual language models for English, whereas there are few investigative studies of biases encoded in language models for languages beyond English. In this paper, we fill this gap by analysing gender bias in West Slavic language models. We introduce the first template-based dataset in Czech, Polish, and Slovak for measuring gender bias towards male, female and non-binary subjects. We complete the sentences using both mono- and multilingual language models and assess their suitability for the masked language modelling objective. Next, we measure gender bias encoded in West Slavic language models by quantifying the toxicity and genderness of the generated words. We find that these language models produce hurtful completions that depend on the subject's gender. Perhaps surprisingly, Czech, Slovak, and Polish language models produce more hurtful completions with men as subjects, which, upon inspection, we find is due to completions being related to violence, death, and sickness.
When a Language Question Is at Stake. A Revisited Approach to Label Sensitive Content

Stetsenko Daria

2023-11-17

Abstract:Many under-resourced languages require high-quality datasets for specific tasks such as offensive language detection, disinformation, or misinformation identification. However, the intricacies of the content may have a detrimental effect on the annotators. The article aims to revisit an approach of pseudo-labeling sensitive data on the example of Ukrainian tweets covering the Russian-Ukrainian war. Nowadays, this acute topic is in the spotlight of various language manipulations that cause numerous disinformation and profanity on social media platforms. The conducted experiment highlights three main stages of data annotation and underlines the main obstacles during machine annotation. Ultimately, we provide a fundamental statistical analysis of the obtained data, evaluation of models used for pseudo-labelling, and set further guidelines on how the scientists can leverage the corpus to execute more advanced research and extend the existing data samples without annotators' engagement.

Computation and Language

RuBia: A Russian Language Bias Detection Dataset

RuBQ: A Russian Dataset for Question Answering over Wikidata

Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian

RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Reducing Unintended Identity Bias in Russian Hate Speech Detection

RuSemShift: a dataset of historical lexical semantic change in Russian

BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service

Detecting Human Rights Violations on Social Media during Russia-Ukraine War

CoRAL: a Context-aware Croatian Abusive Language Dataset

Uncovering and Categorizing Social Biases in Text-to-SQL.

RuCoLA: Russian Corpus of Linguistic Acceptability

OLID-BR: offensive language identification dataset for Brazilian Portuguese

RuMedBench: A Russian Medical Language Understanding Benchmark

Dataset for Automatic Summarization of Russian News

A Study on Bias Detection and Classification in Natural Language Processing

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation

BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented Generation

Measuring Gender Bias in West Slavic Language Models

When a Language Question Is at Stake. A Revisited Approach to Label Sensitive Content