Web Site Partition Scheme for Vertical Search Engine

李学凯,许笑,孙春奇,张伟哲,李斌
DOI: https://doi.org/10.3969/j.issn.1000-3428.2010.08.096
2010-01-01
Abstract:In allusion to the problem of traditional search engines' task allocating methods,a new fine-grained method called Web site partition is presented,which is as an effective optimization of the traditional method adopted by vertical search engines.This method divides large-scale Web sites into a number of smaller subsets,so that several crawlers can parallel crawl each subset in order to accelerate the overall downloading progress.The proposed algorithm is proved to be effective against the sample data sets.
What problem does this paper attempt to address?