Abstract:Morphological segmentation and stemming are foundational tasks in natural language processing. They have become effective ways to alleviate data sparsity in agglutinative languages because of the nature of agglutinative language word formation. Uyghur and Kazakh, as typical agglutinative languages, have made significant progress in morphological segmentation and stemming in recent years. However, the evaluation metrics used in previous work are character-level based, which may not comprehensively reflect the performance of models in morphological segmentation or stemming. Moreover, existing methods avoid manual feature extraction, but the model’s ability to learn features is inadequate in complex scenarios, and the correlation between different features has not been considered. Consequently, these models lack representation in complex contexts, affecting their effective generalization in practical scenarios. To address these issues, this paper redefines the morphological-level evaluation metrics: F1-score and accuracy (ACC) for morphological segmentation and stemming tasks. In addition, two models are proposed for morpheme segmentation and stem extraction tasks: supervised model and unsupervised model. The supervised model learns character and contextual features simultaneously, then feature embeddings are input into a Transformer encoder to study the correlation between character and context embeddings. The last layer of the model uses a CRF or softmax layer to determine morphological boundaries. In unsupervised learning, an encoder–decoder structure introduces n-gram correlation assumptions and masked attention mechanisms, enhancing the correlation between characters within n-grams and reducing the impact of characters outside n-grams on boundaries. Finally, comprehensive comparative analyses of the performance of different models are conducted from various points of view. Experimental results demonstrate that: (1) The proposed evaluation method effectively reflects the differences in morphological segmentation and stemming for Uyghur and Kazakh; (2) Learning different features and their correlation can enhance the model’s generalization ability in complex contexts. The proposed models achieve state-of-the-art performance on Uyghur and Kazakh datasets.

Unsupervised Learning and Linguistic Rule Based Algorithm for Uyghur Word Segmentation.

Uyghur Word Segmentation Using a Combination of Rules and Statistics

Uyghur sentiment analysis based on supervised word segmentation method

Uyghur Morphological Segmentation with Bidirectional GRU Neural Networks

Polygon-Location Method Based on Uyghur Text Regional Rules

Research on Uyghur Morphological Segmentation Based on Long Sequence Labeling Method.

Automatic Phonetic Segmentation Using HMM Model in Uyghur Language

A Benchmark for Morphological Segmentation in Uyghur and Kazakh

Log-linear Models for Uyghur Segmentation in Spoken Language Translation

Improved Spoken Uyghur Segmentation for Neural Machine Translation

Learning Subword Embedding to Improve Uyghur Named-Entity Recognition.

Research on the Uyghur Morphological Segmentation Model with an Attention Mechanism

Uighur Syllable Automatic Segment Algorithm

Text Representation and Similarity Measure for Text Clustering Based on Semantic Strings: A Case Study on Uyghur Language

Learning Distributed Representations Of Uyghur Words And Morphemes

Sentence Boundary Detection of Uyghur Based on Rules and Statistics

Toward Better Loanword Identification in Uyghur Using Cross-lingual Word Embeddings.

Bidirectional Long Short-Term Memory Network with a Conditional Random Field Layer for Uyghur Part-Of-Speech Tagging

Sentiment classification of Uyghur text based on BLSTM

Unsupervised Query Segmentation Using Monolingual Word Alignment Method.

Text Filtering through Multi-Pattern Matching: A Case Study of Wu–Manber–Uy on the Language of Uyghur