Abstract:In vector space model (VSM), text representation is the task of transforming the content of a textual document into a vector in the term space so that the document could be recognized and classified by a computer or a classifier. Different terms (i.e. words, phrases, or any other indexing units used to identify the contents of a text) have different importance in a text. The term weighting methods assign appropriate weights to the terms to improve the performance of text categorization. In this study, we investigate several widely-used unsupervised (traditional) and supervised term weighting methods on benchmark data collections in combination with SVM and kNN algorithms. In consideration of the distribution of relevant documents in the collection, we propose a new simple supervised term weighting method, i.e. tf.rf, to improve the terms' discriminating power for text categorization task. From the controlled experimental results, these supervised term weighting methods have mixed performance. Specifically, our proposed supervised term weighting method, tf.rf, has a consistently better performance than other term weighting methods while other supervised term weighting methods based on information theory or statistical metric perform the worst in all experiments. On the other hand, the popularly used tf.idf method has not shown a uniformly good performance in terms of different data sets.

Research on Algorithm of Text Feature Selection and Weighting Based on Category

CLDA: Feature Selection for Text Categorization Based on Constrained LDA

A Novel Feature Selection Approach and Feature Weight Adjustment Technique in Text Classification

A Study On Feature Weighting In Chinese Text Categorization

Improving Short Text Classification Through Better Feature Space Selection

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

New Feature Selection and Weighting Methods Based on Category Information.

Feature Selection Method on Imbalanced Text

A Feature Selection Method Based on Class Feature Domains for Text Categorization

A Comparative Study on Feature Weight in Text Categorization

An Effective Feature Selection Method For Text Categorization

Learning Effective Features for Chinese Text Categorization

Relative Term-Frequency Based Feature Selection for Text Categorization

A Method of Feature Selection Based on Word2Vec in Text Categorization

A Discriminative and Semantic Feature Selection Method for Text Categorization

A New Approach of Feature Selection for Text Categorization

A Novel Term Weighting Scheme for Automated Text Categorization

An Improved Method of Feature Selection Based on Concept Attributes in Text Classification

A General Framework of Feature Selection for Text Categorization

Feature Selection for Support Vector Machines in Text Categorization

Supervised and Traditional Term Weighting Methods for Automatic Text Categorization