Abstract:Text clustering is an important method for organising the increasing volume of digital content, aiding in the structuring and discovery of hidden patterns in uncategorised data. The effectiveness of text clustering largely depends on the selection of textual embeddings and clustering algorithms. This study argues that recent advancements in large language models (LLMs) have the potential to enhance this task. The research investigates how different textual embeddings, particularly those utilised in LLMs, and various clustering algorithms influence the clustering of text datasets. A series of experiments were conducted to evaluate the impact of embeddings on clustering results, the role of dimensionality reduction through summarisation, and the adjustment of model size. The findings indicate that LLM embeddings are superior at capturing subtleties in structured language. OpenAI's GPT-3.5 Turbo model yields better results in three out of five clustering metrics across most tested datasets. Most LLM embeddings show improvements in cluster purity and provide a more informative silhouette score, reflecting a refined structural understanding of text data compared to traditional methods. Among the more lightweight models, BERT demonstrates leading performance. Additionally, it was observed that increasing model dimensionality and employing summarisation techniques do not consistently enhance clustering efficiency, suggesting that these strategies require careful consideration for practical application. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by integrating embeddings from LLMs, offering improved methodologies and suggesting new avenues for future research in various types of textual analysis.

Textual Factors: A Scalable, Interpretable, and Data-driven Approach to Analyzing Unstructured Information

Incorporating Knowledge into Neural Network for Text Representation.

Scalable Text Mining with Sparse Generative Models

Textual analysis and machine leaning: Crack unstructured data in finance and accounting ☆

Topic Analysis for Text with Side Data

A Survey of Text Representation Methods and Their Genealogy

Latent Topic Text Representation Learning on Statistical Manifolds.

Sequential Text-Term Selection in Vector Space Models

NeuralFactors: A Novel Factor Learning Approach to Generative Modeling of Equities

Scalable Community Discovery on Textual Data with Relations

Discovering influential text using convolutional neural networks

Exploiting textual and relationship information for fine-grained financial sentiment analysis

Parallel Non-negative Matrix Tri-Factorization for Text Data Co-clustering

Author Clustering and Topic Estimation for Short Texts

Modeling Dynamic User Interests: A Neural Matrix Factorization Approach

A Case Study in Text Mining: Interpreting Twitter Data From World Cup Tweets

Textiverse: A Scalable Visual Analytics System for Exploring Geotagged and Timestamped Text Corpora

Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs

Matching Natural Language Sentences with Hierarchical Sentence Factorization

Text Clustering with Large Language Model Embeddings

A novel text-based framework for forecasting agricultural futures using massive online news headlines