Abstract:Data preprocessing is an essential task to prepare suitable target datasets to apply statistical and data mining algorithms. It has become one of the complex segment in Web usage mining due to the massive and unstructured nature of Web server log. The data preprocessing segment in Web usage mining is divided into several phases such as data fusion, data cleaning, user identification, session identification, path completion, and data formatting. This paper focuses on the initial phases of the Web usage mining process, such as data cleaning, user identification, and session identification. Due to the growing size of log data at terabyte and petabyte scale, traditional data preprocessing algorithms fail at scalability points and possess Big Data issues. During the previous few years, the MapReduce framework has evolved as one of the most used parallel programming frameworks for processing Big Data on a cluster of nodes. In this paper, a MapReduce-based data preprocessing algorithm is developed. This algorithm comprises data preprocessing subphases such as data cleaning, user identification, and session identification. Various efficient heuristics are incorporated into existing MapReduce-based data preprocessing algorithm to detect ethical and unethical robots. Further several experiments are performed on a cluster of nodes and found that the proposed MapReduce-based data preprocessing algorithm is efficient and scalable for larger datasets. Moreover, we have also analyzed the impact of robots' requests on sessions generated in the session identification phase to measure the effectiveness of the proposed approach.

Research on the Technologies of Data Preprocessing in Web Usage Mining

A Survey on Preprocessing Methods for Web Usage Data

An integration approach of data mining with web cache pre-fetching

Research on Web Mining Technique Facing Electronic Business and Application

Effective E-Learning Environment Personalization Using Web Usage Mining Technology

Preprocessing: A Prerequisite for Discovering Patterns in Web Usage Mining Process

A Combined Data Preprocessing Method Based on K-means Clustering and Singular Spectrum Analysis

Catching potential customers: an example of Web-mining-aided e-commerce decision-making

Data Preprocessing Technique in Web Log Mining

A Research of Data Mining Technologies on Web Search

Application of Web Text Mining in Study Assistance

A Preprocessing Framework and Approach for Web Applications

Performance Evaluation of the MapReduce-based Parallel Data Preprocessing Algorithm in Web Usage Mining with Robot Detection Approaches

The Research of Web Users' Behavior Mining Based on Association Rules

Preprocessing: A Prerequisite for Discovering Patterns in WUM Process

Research on Data Preprocessing Methods for Big Data

Web Usage mining framework for Data Cleaning and IP address Identification

Incorporating Caching with Prefetching for Wireless Web Access

Research on the Methods of Data Reduction in Data Preprocessing

Mining Web Logs to Reduce the Time Cost of Information Foraging

Chinese Web-page Classification Study