Abstract:With the explosion of online communication and publication, texts become obtainable via forums, chat messages, blogs, book reviews and movie reviews. Usually, these texts are much short and noisy without sufficient statistical signals and enough information for a good semantic analysis. Traditional natural language processing methods such as Bow-of-Word (BOW) based probabilistic latent semantic models fail to achieve high performance due to the short text environment. Recent researches have focused on the correlations between words, i.e., term dependencies, which could be helpful for mining latent semantics hidden in short texts and help people to understand them. Long short-term memory (LSTM) network can capture term dependencies and is able to remember the information for long periods of time. LSTM has been widely used and has obtained promising results in variants of problems of understanding latent semantics of texts. At the same time, by analyzing the texts, we find that a number of keywords contribute greatly to the semantics of the texts. In this paper, we establish a keyword vocabulary and propose an LSTM-based model that is sensitive to the words in the vocabulary; hence, the keywords leverage the semantics of the full document. The proposed model is evaluated in a short-text sentiment analysis task on two datasets: IMDB and SemEval-2016, respectively. Experimental results demonstrate that our model outperforms the baseline LSTM by 1%~2% in terms of accuracy and is effective with significant performance enhancement over several non-recurrent neural network latent semantic models (especially in dealing with short texts). We also incorporate the idea into a variant of LSTM named the gated recurrent unit (GRU) model and achieve good performance, which proves that our method is general enough to improve different deep learning models.

SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

Short Text Topic Modeling With Flexible Word Patterns

Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

A Unified Framework for Learned Sparse Retrieval

Topic-Grained Text Representation-based Model for Document Retrieval

LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval

Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval

Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Modeling over Short Texts

FecTek: Enhancing Term Weight in Lexicon-Based Retrieval with Feature Context and Term-level Knowledge

Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval

Emphasizing Essential Words for Sentiment Classification Based on Recurrent Neural Networks

SPAR: Personalized Content-Based Recommendation via Long Engagement Attention

Short Text Model Based on Strong Feature Thesaurus

Mistral-SPLADE: LLMs for better Learned Sparse Retrieval

Exploiting Global Semantic Similarity Biterms for Short-Text Topic Discovery

Adapting Learned Sparse Retrieval for Long Documents