Abstract:Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns. Moreover, we find that the performance of general-purpose embedding models on MTEB is not correlated with their performance on FinMTEB, indicating the need for domain-specific embedding benchmarks for domain-specific embedding models. This study sheds light on developing domain-specific embedding models in the LLM era. FinMTEB comes with open-source code at <a class="link-external link-https" href="https://github.com/yixuantt/FinMTEB" rel="external noopener nofollow">this https URL</a>

Financial Topic Modeling Based on the BERT-LDA Embedding

Document Clustering Based on Probabilistic Topic Model

Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data

Deep Learning based Topic Analysis on Financial Emerging Event Tweets

An integrated clustering and BERT framework for improved topic modeling

Topic Modelling on Consumer Financial Protection Bureau Data: An Approach Using BERT Based Embeddings

Topics Modeling Based on Selective Zipf Distribution

Topic Modeling over Short Texts by Incorporating Word Embeddings

Topic Modeling in Embedding Spaces

Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts

BERTopic-Driven Stock Market Predictions: Unraveling Sentiment Insights

Data-Centric Financial Large Language Models

Topic2Vec: Learning distributed representations of topics

Design and Development of Partitional Topic Model

A Topic Model for Hierarchical Documents

Do We Need Domain-Specific Embedding Models? An Empirical Investigation

Multi-Label Topic Model for Financial Textual Data

A Joint Model Of Extended Lda And Ibtm Over Streaming Chinese Short Texts

A Novel Topic Model for Documents by Incorporating Semantic Relations Between Words