Abstract:Understanding what kinds of Web pages are the most useful for Web search engine users is a critical task in Web information retrieval (IR). Most previous works used hyperlink analysis algorithms to solve this problem. However, little research has been focused on query-independent Web data cleansing for Web IR. In this paper, we first provide analysis of the differences between retrieval target pages and ordinary ones based on more than 30 million Web pages obtained from both the Text Retrieval Conference (TREC) and a widely used Chinese search engine, SOGOU (www.sogou.com). We further propose a learning-based data cleansing algorithm for reducing Web pages that are unlikely to be useful for user requests. We found that there exists a large proportion of low-quality Web pages in both the English and the Chinese Web page corpus, and retrieval target pages can be identified using query-independent features and cleansing algorithms. The experimental results showed that our algorithm is effective in reducing a large portion of Web pages with a small loss in retrieval target pages. It makes it possible for Web IR tools to meet a large fraction of users' needs with only a small part of pages on the Web. These results may help Web search engines make better use of their limited storage and computation resources to improve search performance. © 2007 Wiley Periodicals, Inc.

Data Cleansing for Web Information Retrieval Using Query Independent Features

Web Data Cleansing for Effective Information Retrieval

Learning-based Web Data Cleansing for Information Retrieval

Web data cleansing for information retrieval using key resource page selection.

Extracting Web Content by Exploiting Multi-Category Characteristics

Improving short text classification using public search engines

Extracting Novel Features for E-Commerce Page Quality Classification.

Extracting Novel Features for E-Commerce Page Quality Classification

Topic-independent web high-quality page selection based on k-means clustering

A HTML Parser to Improve Chinese Search Engines

Web Page Quality Estimation Based on Linear Discriminant Function1

Improved fuzzy set information retrieval approach on duplicate webpage detection

Focused Crawler Research for Business Intelligence Acquisition

Unhealthy Webpage Detection Based on Topic-focused Web Crawler

Research on Indexing Page Collection Selection Method for Search Engine.

An Intelligent Platform For Information Retrieval

DF or IDF? On the Use of Primary Feature Model for Web Information Retrieval

Research of Web Information Quality Control Based on Informatics Theory Applied in Water Resource Search Engine

Automated Quality Assessment of Web Pages from Textual Content

Quality Estimation Based on Linear Discriminant Function

Helping identify when users find useful documents: examination of query reformulation intervals.