Abstract:In today's business environment, enterprises are increasingly under pressure to process the vast amount of data produced everyday within enterprises. One method is to focus on the business intelligence BI applications and increasing the commercial added-value through such business analytics activities. Term weighting scheme, which has been used to convert the documents as vectors in the term space, is a vital task in enterprise Information Retrieval IR, text categorisation, text analytics, etc. When determining term weight in a document, the traditional TF-IDF scheme sets weight value for the term considering only its occurrence frequency within the document and in the entire set of documents, which leads to some meaningful terms that cannot get the appropriate weight. In this article, we propose a new term weighting scheme called Term Frequency – Function of Document Frequency TF-FDF to address this issue. Instead of using monotonically decreasing function such as Inverse Document Frequency, FDF presents a convex function that dynamically adjusts weights according to the significance of the words in a document set. This function can be manually tuned based on the distribution of the most meaningful words which semantically represent the document set. Our experiments show that the TF-FDF can achieve higher value of Normalised Discounted Cumulative Gain in IR than that of TF-IDF and its variants, and improving the accuracy of relevance ranking of the IR results.

A Learning-Based Term-Weighting Approach for Information Retrieval

Exploiting Category Information and Document Information to Improve Term Weighting for Text Categorization

Learning the weight of the query term from the relevance feedback.

An Improved Term Weighting Scheme for Text Classification

Improve Precategorized Collection Retrieval by Using Supervised Term Weighting Schemes

Class-indexing-based term weighting for automatic text classification

A Novel Term Weighting Scheme for Automated Text Categorization

Term Selection and Weighting Approach Based on Key Words in Text Categorization

A Simple Probability Based Term Weighting Scheme For Automated Text Classification

Several alternative term weighting methods for text representation and classification

Term frequency – function of document frequency: a new term weighting scheme for enterprise information retrieval

Turning from TF-IDF to TF-IGM for Term Weighting in Text Classification

Inverse-Category-Frequency based Supervised Term Weighting Schemes for Text Categorization.

An improvement to TF-IDF: Term Distribution based Term Weight Algorithm.

An improved supervised term weighting scheme for text representation and classification

A study of supervised term weighting scheme for sentiment analysis

Weigh Matrix Based Solution Framework for Term Proximity Information Retrieval

Supervised and Traditional Term Weighting Methods for Automatic Text Categorization

Hidden markov model for term weighting in verbose queries

An Improved TF-IDF Weights Function Based on Information Theory

A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency