Abstract:The rapid advancement of large language models (LLMs) has enabled natural language processing capabilities similar to those of humans, and LLMs are being widely utilized across various societal domains such as education and healthcare. While the versatility of these models has increased, they have the potential to generate subjective and normative language, leading to discriminatory treatment or outcomes among social groups, especially due to online offensive language. In this paper, we define such harm as societal bias and assess ethnic, gender, and racial biases in a model fine-tuned with Korean comments using Bidirectional Encoder Representations from Transformers (KcBERT) and KOLD data through template-based Masked Language Modeling (MLM). To quantitatively evaluate biases, we employ LPBS and CBS metrics. Compared to KcBERT, the fine-tuned model shows a reduction in ethnic bias but demonstrates significant changes in gender and racial biases. Based on these results, we propose two methods to mitigate societal bias. Firstly, a data balancing approach during the pre-training phase adjusts the uniformity of data by aligning the distribution of the occurrences of specific words and converting surrounding harmful words into non-harmful words. Secondly, during the in-training phase, we apply Debiasing Regularization by adjusting dropout and regularization, confirming a decrease in training loss. Our contribution lies in demonstrating that societal bias exists in Korean language models due to language-dependent characteristics.

How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models

An Analysis of Social Biases Present in BERT Variants Across Multiple Languages

Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models

Mitigating Language-Dependent Ethnic Bias in BERT

Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias

This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models

Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models

Unmasking the Stereotypes: Evaluating Social Biases in Chinese BERT

Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Evaluating Biased Attitude Associations of Language Models in an Intersectional Context

Linguistically Grounded Analysis of Language Models using Shapley Head Values

Measuring and Mitigating Gender Bias in Legal Contextualized Language Models

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance

Detecting Bias in Large Language Models: Fine-tuned KcBERT

Latin BERT: A Contextual Language Model for Classical Philology

Characterizing English Variation across Social Media Communities with BERT

BERTScore is Unfair: on Social Bias in Language Model-Based Metrics for Text Generation