Abstract:Despite recent effort to estimate topology characteristics of large graphs (e.g., online social networks and peer-to-peer networks), little attention has been given to develop a formal crawling methodology to characterize the vast amount of content distributed over these networks. Due to the large-scale nature of these networks and a limited query rate imposed by network service providers, exhaustively crawling and enumerating content maintained by each vertex is computationally prohibitive. In this paper, we show how one can obtain content properties by crawling only a small fraction of vertices and collecting their content. We first show that when sampling is naively applied, this can produce a huge bias in content statistics (i.e., average number of content replicas). To remove this bias, one may use maximum likelihood estimation to estimate content characteristics. However, our experimental results show that this straightforward method requires to sample most vertices to obtain accurate estimates. To address this challenge, we propose two efficient estimators: special copy estimator (SCE) and weighted copy estimator (WCE) to estimate content characteristics using available information in sampled content. SCE uses the special content copy indicator to compute the estimate, while WCE derives the estimate based on meta-information in sampled vertices. We conduct experiments on a variety of real-word and synthetic datasets, and the results show that WCE and SCE are cost effective and also “asymptotically unbiased”. Our methodology provides a new tool for researchers to efficiently query content distributed in large-scale networks.

Algorithm of Estimating Index Sizes of Resource Collections in Distributed Search

Estimating Collection Size in Distributed Search

A LDA Topic Model Based Collection Selection Method for Distributed Information Retrieval

Estimating collection size with logistic regression.

Research on Indexing Page Collection Selection Method for Search Engine.

Estimation of Distribution Algorithm with Scatter Search for Dynamic Optimisation Problems

Towards A Quality-Oriented Real-Time Web Crawler

On Distributed Web Crawler: Architecture, Algorithms and Strategy

Learning to Rank Collections.

The Ranking of Deep Web Sources Based on Data Quality

Ssrw: A Scalable Algorithm For Estimating Graphlet Statistics Based On Random Walk

Fast Crawling Methods of Exploring Content Distributed over Large Graphs

Research on Optimization Algorithm for Deep Web Query

Scale-Adaptable Recrawl Strategies For Dht-Based Distributed Web Crawling System

Sampling Content Distributed Over Graphs

Biased Random Walk-Based Replication and Search

A hunger-based scheduling strategy for distributed crawler

Random walk spread and search in unstructured p2p

Web Site Partition Scheme for Vertical Search Engine

Allocating images and selecting image collections for distributed visual search

Large-Scale Duplicate Detection for Web Image Search