Abstract:Text categorization (TC) becomes the key technology to find relevant and timely information from a volume of digital documents, and feature selection techniques are proposed to overcome the high dimensionality which causes the high computational complexity and low accuracy in TC tasks. Chi-square statistics (CHI) is one of the most efficient feature selection methods; however, it has two weaknesses. (1) It is document frequency based, and only counts whether the term occurs or not. Actually, high-frequency term occurring in few documents is often regarded as a discriminator in corpus. (2) It does not consider the term distribution. A term has more discriminating power for a specific category when its difference in degree of distribution is lower. In this paper, we propose a modified CHI feature selection approach which is called term frequency and distribution based CHI to overcome these weaknesses. We use sample variance to calculate the term distribution, and improve the classic CHI with maximum term frequency. Extensive and comparative experiments on three corpora show that the proposed approach is comparable to the classic feature selection methods in terms of macro-F1 and micro-F1.

Learning Effective Features for Chinese Text Categorization

CLDA: Feature Selection for Text Categorization Based on Constrained LDA

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

Novel Feature Selection Algorithm For Chinese Text Categorization Based On Chi

Select Strong Information Features to Improve Text Categorization Effectiveness

Feature selection method based on category discriminability

Study on Feature Selection Methods in Chinese Text Categorization

An Effective Feature Selection Method For Text Categorization

Study on Improved CHI for Feature Selection in Chinese Text Categorization

A Discriminative and Semantic Feature Selection Method for Text Categorization

Relative Term-Frequency Based Feature Selection for Text Categorization

An Improved Feature Selection Method in Chinese Text Categorization

A Feature Selection Method Based on Class Feature Domains for Text Categorization

Study on Information Gain-Based Feature Selection in Chinese Text Categorization

A Combined Feature Selection Method for Chinese Text Categorization

Dimensionality Reduction With Category Information Fusion And Non-Negative Matrix Factorization For Text Categorization

Chi-square Statistics Feature Selection Based on Term Frequency and Distribution for Text Categorization

Context Based Feature Description Model in Chinese Text Categorization

Improved method for text feature selection based on CHI

A Method of the Feature Selection in Hierarchical Text Classification Based on the Category Discrimination and Position Information

A Study On Feature Weighting In Chinese Text Categorization