Abstract:Feature selection techniques play an important role in text categorization (TC), especially for the largescale TC tasks. Many new and improved methods have been proposed, and most of them are based on document frequency, such as the famous Chi-square statistic and information gain etc. These methods based on document frequency, however, have two shortcomings: (1) they are not reliable for low-frequency terms, that is, low-frequency terms will be filtered because of their smaller weights; and (2) they only count whether one term occurs within a document and ignore term frequency. Actually, high-frequency term (except stop words) occurred in few documents is often regards as a discriminators in the real-life corpus.Aimed at solving the above drawbacks, the paper focuses on how to construct a feature selection function based on term frequency, and proposes a new approach using student t-test. The t-test function is used to measure the diversity of the distributions of a term frequency between the specific category and the entire corpus. Extensive comparative experiments on two text corpora using three classifiers show that the proposed approach is comparable to the state-of-the-art feature selection methods in terms of macro-F1 and micro-F1. Especially on micro-F1, our method achieves slightly better performance on Reuters with kNN and SVMs classifiers, compared to x(2), and IG. (C) 2014 Elsevier B.V. All rights reserved.

Improved Comprehensive Measurement Feature Selection Method for Text Categorization

A New Feature Selection Based on Comprehensive Measurement Both in Inter-Category and Intra-Category for Text Categorization

Feature Selection Algorithm Based on Comprehensive Measurement for Text Categorization

An Enhanced Text Categorization Method Based on Improved Text Frequency Approach and Mutual Information Algorithm

Improving Short Text Classification Through Better Feature Space Selection

FIVE NEW FEATURE SELECTION METRICS IN TEXT CATEGORIZATION

A New Approach of Feature Selection for Text Categorization

A Framework of Feature Selection Methods for Text Categorization.

A Discriminative and Semantic Feature Selection Method for Text Categorization

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

Text Categorization Method Based on Improved Mutual Information and Characteristic Weights Evaluation Algorithms

A New Feature Selection Method in Text Categorization

Relative Term-Frequency Based Feature Selection for Text Categorization

Comparison and Improvement of Feature Selection Method for Text Categorization

Study on Mutual Information-Based Feature Selection in Text Categorization

Application of Feature Selection Method to Text Categorization Based on Feature Contribution Degree

Study on an Improved Mutual Information for Feature Selection in Chinese Text Categorization

An Improved Feature Selection Method in Chinese Text Categorization

A Two-Stage Feature Selection Method for Text Categorization

A Comparative Study on Feature Weight in Text Categorization

T-Test Feature Selection Approach Based on Term Frequency for Text Categorization