Abstract:Word segmentation in agglutinative languages presents significant challenges due to morphological complexity and variability of linguistic structure. Although practical, traditional rule-based and statistical model-based approaches show limitations in handling these complexities. This study investigates the effectiveness of neural network models, specifically LSTM, Bi-LSTM with CRF, and BERT, in comparison to these traditional methods, using datasets from several agglutinative languages such as Turkish, Finnish, Hungarian, Nahuatl, and Swahili. The methodology includes preprocessing and data augmentation to improve data quality and consistency, followed by training and evaluating the selected models. The results reveal that the neural network models significantly outperform rule-based and statistical model-based approaches on all metrics assessed. Specifically, for the rule-based models, the BERT model achieved 92% accuracy and 91% F1-score in Turkish, compared to 70% and 67%, respectively. Moreover, the Bi-LSTM with CRF showed 86% recall in Finnish, significantly outperforming traditional models. Implementing advanced preprocessing and data augmentation techniques allows for optimizing the performance of the models. This study confirms the effectiveness of neural network models in word segmentation and provides a valuable framework for future research in natural language processing in complex linguistic contexts.

More than Just Statistical Recurrence: Human and Machine Unsupervised Learning of Māori Word Segmentation across Morphological Processes

Rule-Based and Word-Level Statistics-Based Processing of Language: Insights from Neuroscience

Controlling Utterance Length in NMT-based Word Segmentation with Attention

Morphological and Language-Agnostic Word Segmentation for NMT

Evaluating Neural Network Models for Word Segmentation in Agglutinative Languages: Comparison With Rule-Based Approaches and Statistical Models

The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation

Using Contextual Information for Sentence-level Morpheme Segmentation

A Masked Segmental Language Model for Unsupervised Natural Language Segmentation

Mostly-Unsupervised Statistical Segmentation of Japanese Kanji Sequences

TAMS: Translation-Assisted Morphological Segmentation

A study of universal morphological analysis using morpheme-based, holistic, and neural approaches under various data size conditions

Subword Representations Successfully Decode Brain Responses to Morphologically Complex Written Words

Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation

Cognate-aware morphological segmentation for multilingual neural translation

Unsupervised Multimodal Word Discovery Based on Double Articulation Analysis With Co-Occurrence Cues

Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text

Finding Better Subword Segmentation for Neural Machine Translation

Awakening the Proto‐Lexicon: A Proto‐Lexicon Gives Learning Advantages for Intentionally Learning a Language

LLMs' morphological analyses of complex FST-generated Finnish words

Multi Task Deep Morphological Analyzer: Context Aware Joint Morphological Tagging and Lemma Prediction

Human-Computer Interactive Chinese Word Segmentation: an Adaptive Dirichlet Process Mixture Model Approach.