Abstract:Despite recent effort to estimate topology characteristics of large graphs (e.g., online social networks and peer-to-peer networks), little attention has been given to develop a formal crawling methodology to characterize the vast amount of content distributed over these networks. Due to the large-scale nature of these networks and a limited query rate imposed by network service providers, exhaustively crawling and enumerating content maintained by each vertex is computationally prohibitive. In this paper, we show how one can obtain content properties by crawling only a small fraction of vertices and collecting their content. We first show that when sampling is naively applied, this can produce a huge bias in content statistics (i.e., average number of content replicas). To remove this bias, one may use maximum likelihood estimation to estimate content characteristics. However, our experimental results show that this straightforward method requires to sample most vertices to obtain accurate estimates. To address this challenge, we propose two efficient estimators: special copy estimator (SCE) and weighted copy estimator (WCE) to estimate content characteristics using available information in sampled content. SCE uses the special content copy indicator to compute the estimate, while WCE derives the estimate based on meta-information in sampled vertices. We conduct experiments on a variety of real-word and synthetic datasets, and the results show that WCE and SCE are cost effective and also “asymptotically unbiased”. Our methodology provides a new tool for researchers to efficiently query content distributed in large-scale networks.

Sampling on Bipartite Networks: a Comparative Analysis of Eight Crawling Methods

Unbiased Sampling Method Analysis on Online Social Network

Sampling Online Social Networks by Random Walk with Indirect Jumps

Fast Crawling Methods of Exploring Content Distributed over Large Graphs

Towards Cost-efficient Sampling Methods

Toward Cost-Efficient Sampling Methods

Graph sub-sampling for divide-and-conquer algorithms in large networks

Understanding Graph Sampling Algorithms for Social Network Analysis

An Improved Sampling Method of Complex Network

Preserving the topological properties of complex networks in network sampling

Hub search method based on sampling

Design of Efficient Sampling Methods on Hybrid Social-Affiliation Networks.

On Random Walk Based Graph Sampling

Albatross sampling: robust and effective hybrid vertex sampling for social graphs.

Empirical comparison of network sampling techniques

Sampling Algorithms for Butterfly Counting on Temporal Bipartite Graphs

Sampling Content Distributed Over Graphs

Sampling for Approximate Bipartite Network Projection

SGP: a Social Network Sampling Method Based on Graph Partition

A Community-Based Sampling Method Using DPL for Online Social Network

A Survey of Sampling Method for Social Media Embeddedness Relationship