Abstract:Feature selection techniques play an important role in text categorization (TC), especially for the largescale TC tasks. Many new and improved methods have been proposed, and most of them are based on document frequency, such as the famous Chi-square statistic and information gain etc. These methods based on document frequency, however, have two shortcomings: (1) they are not reliable for low-frequency terms, that is, low-frequency terms will be filtered because of their smaller weights; and (2) they only count whether one term occurs within a document and ignore term frequency. Actually, high-frequency term (except stop words) occurred in few documents is often regards as a discriminators in the real-life corpus.Aimed at solving the above drawbacks, the paper focuses on how to construct a feature selection function based on term frequency, and proposes a new approach using student t-test. The t-test function is used to measure the diversity of the distributions of a term frequency between the specific category and the entire corpus. Extensive comparative experiments on two text corpora using three classifiers show that the proposed approach is comparable to the state-of-the-art feature selection methods in terms of macro-F1 and micro-F1. Especially on micro-F1, our method achieves slightly better performance on Reuters with kNN and SVMs classifiers, compared to x(2), and IG. (C) 2014 Elsevier B.V. All rights reserved.

Chi-square Statistics Feature Selection Based on Term Frequency and Distribution for Text Categorization

CLDA: Feature Selection for Text Categorization Based on Constrained LDA

Feature Selection Based on Term Frequency and T-test for Text Categorization

T-Test Feature Selection Approach Based on Term Frequency for Text Categorization

Improved method for text feature selection based on CHI

Relative Term-Frequency Based Feature Selection for Text Categorization

An Evaluation on Feature Selection for Text Clustering

Efficient Method for Feature Selection in Text Classification

Novel Feature Selection Algorithm For Chinese Text Categorization Based On Chi

Select Strong Information Features to Improve Text Categorization Effectiveness

An Improved Feature Selection Algorithm Utilizing the Within Category Variance

Learning Effective Features for Chinese Text Categorization

A Discriminative and Semantic Feature Selection Method for Text Categorization

Study on Feature Selection Methods in Chinese Text Categorization

A New Feature Selection Based on Comprehensive Measurement Both in Inter-Category and Intra-Category for Text Categorization

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

A New Approach of Feature Selection for Text Categorization

An Effective Feature Selection Method For Text Categorization

A Feature Selection Method Based on Class Feature Domains for Text Categorization

A New Approach To Feature Selection For Text Categorization

A General Framework of Feature Selection for Text Categorization