Abstract:In many natural language processing (NLP) tasks, a document is commonly modeled as a bag of words using the term frequency-inverse document frequency (TF-IDF) vector. One major shortcoming of the frequency-based TF-IDF feature vector is that it ignores word orders that carry syntactic and semantic relationships among the words in a document, and they can be important in some NLP tasks such as genre classification. This paper proposes a novel distributed vector representation of a document: a simple recurrent-neural-network language model (RNN-LM) or a long short-term memory RNN language model (LSTM-LM) is first created from all documents in a task; some of the LM parameters are then adapted by each document, and the adapted parameters are vectorized to represent the document. The new document vectors are labeled as DV-RNN and DV-LSTM respectively. We believe that our new document vectors can capture some high-level sequential information in the documents, which other current document representations fail to capture. The new document vectors were evaluated in the genre classification of documents in three corpora: the Brown Corpus, the BNC Baby Corpus and an artificially created Penn Treebank dataset. Their classification performances are compared with the performance of TF-IDF vector and the state-of-the-art distributed memory model of paragraph vector (PV-DM). The results show that DV-LSTM significantly outperforms TF-IDF and PV-DM in most cases, and combinations of the proposed document vectors with TF-IDF or PV-DM may further improve performance.

Adaptive Multi-Compositionality for Recursive Neural Network Models

Adaptive Multi-Compositionality For Recursive Neural Models With Applications To Sentiment Analysis

When Are Tree Structures Necessary for Deep Learning of Representations?

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network.

Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification.

Dynamic Compositional Neural Networks over Tree Structure

Meta Multi-Task Learning for Sequence Modeling.

Learning to Compose over Tree Structures via POS Tags

Adaptive Semantic Compositionality for Sentence Modelling.

Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

Model Composition for Multimodal Large Language Models

An Adaptive Hierarchical Compositional Model for Phrase Embedding

An Adaptive Sentence Representation Learning Model Based on Multi-gram CNN

Recursive Subtree Composition in LSTM-Based Dependency Parsing

Long Short-Term Memory with Quadratic Connections in Recursive Neural Networks for Representing Compositional Semantics

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network

Recurrent Neural Network Language Model Adaptation Derived Document Vector

Obtaining Faithful Interpretations from Compositional Neural Networks

Recursive neural programs: A differentiable framework for learning compositional part-whole hierarchies and image grammars