Abstract:Text stemming is a natural language processing technique that is used to reduce words to their base form, also known as the root form. The use of stemming in IR has been shown to often improve the effectiveness of keyword-matching models such as BM25. However, traditional stemming methods, focusing solely on individual terms, overlook the richness of contextual information. Recognizing this gap, in this paper, we investigate the promising idea of using large language models (LLMs) to stem words by leveraging its capability of context understanding. With this respect, we identify three avenues, each characterised by different trade-offs in terms of computational cost, effectiveness and robustness : (1) use LLMs to stem the vocabulary for a collection, i.e., the set of unique words that appear in the collection (vocabulary stemming), (2) use LLMs to stem each document separately (contextual stemming), and (3) use LLMs to extract from each document entities that should not be stemmed, then use vocabulary stemming to stem the rest of the terms (entity-based contextual stemming). Through a series of empirical experiments, we compare the use of LLMs for stemming with that of traditional lexical stemmers such as Porter and Krovetz for English text. We find that while vocabulary stemming and contextual stemming fail to achieve higher effectiveness than traditional stemmers, entity-based contextual stemming can achieve a higher effectiveness than using Porter stemmer alone, under specific conditions.

Overview of Stemming Algorithms for Indian and Non-Indian Languages

A Literature Review: Stemming Algorithms for Indian Languages

An improved Urdu stemming algorithm for text mining based on multi-step hybrid approach

Stemming via distribution-based word segregation for classification and retrieval

Stemmers for Tamil Language: Performance Analysis

Rule Based Stemmer in Urdu

N-gram Statistical Stemmer for Bangla Corpus

Building a Multilevel Inflection Handling Stemmer to Improve Search Effectiveness for Urdu Language

Unsupervised Stemming based Language Model for Telugu Broadcast News Transcription

A new hybrid stemming algorithm for Persian

Effect of stemming on text similarity for Arabic language at sentence level

XSTEM: An exemplar-based stemming algorithm

Large Language Models for Stemming: Promises, Pitfalls and Failures

Recent advancements in computational morphology : A comprehensive survey

A comparative study of root-based and stem-based approaches for measuring the similarity between arabic words for arabic text mining applications

Quality Estimation Of Machine Translation Outputs Through Stemming

Clustering of Spell Variations for Proper Nouns Transliterated from the other languages

CBAS: context based arabic stemmer

Automatic Extraction of Bengali Root Verbs using Paninian Grammar

SUSTEM: An Improved Rule-Based Sundanese Stemmer

Rational Kernels for Arabic Stemming and Text Classification