Abstract:Text processing tasks commonly grapple with the challenge of high dimensionality. One of the most effective solutions to this challenge is to preprocess text data through feature selection methods. Feature selection can select the most advantageous features for subsequent operations (e.g., classification) from the native feature space of the text. This process effectively trims the feature space's dimensionality, enhancing subsequent operations' efficiency and accuracy. This paper proposes a straightforward and efficient filter feature selection method based on document-term matrix unitization (DTMU) for text processing. Diverging from previous filter feature selection methods that concentrate on scoring criteria definition, our method achieves more optimal feature selection by unitizing each column of the document-term matrix. This approach mitigates feature-to-feature influence and reinforces the role of the weighting proportion within the features. Subsequently, our scoring criterion subtracts the sum of weights for negative samples from positive samples and takes the absolute value. We conduct numerical experiments to compare DTMU with four advanced filter feature selection methods: max–min ratio metric, proportional rough feature selector, least loss, and relative discrimination criterion, along with two classical filter feature selection methods: Chi-square and information gain. The experiments are performed on four ten-thousand-dimensional feature space datasets: book , dvd , music , movie and two thousand-dimensional feature space datasets: imdb , amazon_cells , sourced from Amazon product reviews and movie reviews. Experimental findings demonstrate that DTMU selects more advantageous features for subsequent operations and achieves a higher dimensionality reduction rate than those of the other six methods used for comparison. Moreover, DTMU exhibits robust generalization capabilities across various classifiers and dimensional datasets. Notably, the average CPU time for a single run of DTMU is measured at 1.455 s.

Document Feature Selection Based on the Minimum Term Frequency Threshold

CLDA: Feature Selection for Text Categorization Based on Constrained LDA

Relative Term-Frequency Based Feature Selection for Text Categorization

Text Classification Method Based on Normalized Document Frequency Feature Selection

A New Methal Unsupervised Feature Selection for Text Mining

Feature Selection Method Based on Document Frequency

Feature Selection Method on Imbalanced Text

Efficient Method for Feature Selection in Text Classification

Feature Selection Based on Term Frequency and T-test for Text Categorization

T-Test Feature Selection Approach Based on Term Frequency for Text Categorization

Chi-square Statistics Feature Selection Based on Term Frequency and Distribution for Text Categorization

Improved Document Feature Selection with Categorical Parameter for Text Classification.

A New Feature Selection Based on Comprehensive Measurement Both in Inter-Category and Intra-Category for Text Categorization

An Enhanced Text Categorization Method Based on Improved Text Frequency Approach and Mutual Information Algorithm

A New Feature Selection Method for Text Classification Based on Independent Feature Space Search

A Feature Selection Method Based on Class Feature Domains for Text Categorization

Research on the Feature Selection Techniques Used in Text Classification.

Joint Feature Selection Method Based on Relevance and Redundancy

A simple and efficient filter feature selection method via document-term matrix unitization

A New Approach of Feature Selection for Text Categorization

New Feature Selection Approach(cdf) for Text Categorization