Abstract:AbstractWeb mining is an emerging research area due to the rapid growth of websites. Web mining is classified into Web Content Mining (WCM), Web Usage Mining and Web Structure Mining. Extraction of required information from web page content available on World Wide Web (WWW) is WCM. The WCM is further classified into two categories first category is to directly mine the content on documents and second category is to mine the content using search engine. The mining method focuses on the information extraction and integration. The content of Web may be text, image, audio, video. Web pages typically contain a large amount of information that is not part of the main contents of the pages, like banner advertisements, navigation bars, copyright notices, etc. Such noises on Web pages usually lead to poor results in Web mining. This paper focuses on the problem of Noise free Information retrieval on web pages, which means the pre-processing of Web pages automatically to detect and eliminate noises. This paper proposes an approach for eliminating noises from web pages for the purpose of improving the accuracy and efficiency of web content mining. The main objective of removing noise from a Web Page is to improve the performance of the search. It is very essential to differentiate important information from noisy content that may misguide users’ interest. This approach mainly concentrates on removing the following noises in stages: (1) Primary noises-Navigation bars, Panels and Frames, Page Headers and Footers, Copyright and Privacy Notices, Advertisements and other Uninteresting Data such as audio, video, multiple links. (2) Duplicate Contents and (3) Noise Contents according to block importance. The removal of these noises is done by performing three operations. Firstly, using the Block Splitting operation, primary noises are removed and only the useful text contents are partitioned into blocks. Secondly, using simhash algorithm, the duplicate blocks are removed to obtain the distinct blocks. For each block, three parameters namely Keyword Redundancy (KR), Linkword Percentage (LP) and Titleword Relevancy (TR) calculated. Using these three parameters block importance value (BI) is calculated, which is called Simhash algorithm. The importance of the block is then calculated using simhash algorithm. Based on a threshold value the important blocks are selected using sketching algorithm and the keywords are extracted from those important blocks.

Duplicate Web Page Elimination Based on Bloom Filter

Duplicate Web Page Elimination Based on HTML and Extraction of Long Sentence

Improved fuzzy set information retrieval approach on duplicate webpage detection

Sizespotsigs: An Effective Deduplicate Algorithm Considering The Size Of Page Content

A binary-tree based algorithm for online duplicate documents detection

Research on method to detect reduplicative Chinese short texts

A Query-Dependent Duplicate Detection Approach for Large Scale Search Engines

Research and Evaluation of Near-replicas of Web Pages Detection Algorithms

Research on Parallel Duplicated Webpage Deletion Based on MapReduce Model

Efficiently Filtering Duplicates over Distributed Data Streams

A HTML Parser to Improve Chinese Search Engines

Effectual Web Content Mining using Noise Removal from Web Pages

Optimized Focused Web Crawler with Natural Language Processing Based Relevance Measure in Bioinformatics Web Sources

Parallelized Near-Duplicate Document Detection Algorithm for Large Scale Chinese Web Pages

Efficient Partial-Duplicate Detection Based on Sequence Matching

False Negative Problem of Counting Bloom Filter

Large-Scale Duplicate Detection for Web Image Search

An Improved Algorithm Based on Bloom Filter and Its Application in Bar Code Recognition and Processing

Reducing the number of Bloom filters

A Modified Algorithm Based on the Bloom Filter

Role of Bloom Filter in Big Data Research: A Survey