Abstract:Capturing accurate snapshots of peer-to-peer (P2P) networks, especially those with millions of users, is essential to many P2P-based applications, including those monitoring and analyzing P2P networks. The large scale and dynamic nature of P2P networks, however, make this task very challenging. Existent crawlers of P2P networks, for example, often miss a substantial portion of the ID space while unnecessarily crawling numerous nodes repeatedly. In this paper, we design and evaluate a new crawler called Splider. Unlike traditional crawling algorithms that adopt an iterative approach, Splider recursively splits the ID space of P2P nodes to crawl even tiny corners of the ID space, while avoiding crawling repeated nodes. We further implement a Splider prototype for BT-DHT, a Kademlia-based distributed hash table (DHT) P2P network, that exploits the structure of routing tables at BT-DHT nodes. Experiments show that Splider is able to gather more than 16 million nodes with a 100% recall ratio, whereas a traditional iterative crawler can at best capture only about 8 million nodes with a 66% recall ratio while its traffic-cost effectiveness is 50% less than Splider. Splider can further support distributed deployment; without any synchronization overhead, it reduces the time of capturing a full snapshot to be only about 3 minutes. We finally report and analyze the captured BT-DHT snapshots, including the spatial and temporal distribution of BT-DHT nodes and the existence of sybil and eclipse attacks in BT-DHT.

A Scalable and Robust DHT Protocol for Structured P2P Network

Secure P2P Topology Based on a Multidimensional DHT Space Mapping

A Hybrid Peer-to-Peer Lookup Service Algorithm on Distributed Hash Table

An effective single-hop distributed hash table with high lookup performance and low traffic overhead

LDHT: Locality-aware Distributed Hash Tables

Facilitating Effective Resource Publishing and Searching in DHT Networks

A Multidimensional Mapping Mechanism Based Secure Routing Method for DHT

A Peer-to-Peer DHT Algorithm Based on Small-World Network

Building Locality-Aware P2P Topology Formation Protocol for Heterogeneous Systems

A Novel Content Distribution Mechanism in DHT Networks

Design of Obscure Inquiry in DHT-based P2P Systems

A peer-to-peer approach to large-scale content-based publish-subscribe

Design of Hybrid Peer-to-Peer Lookup Service Algorithm for DHT

A topology-aware hierarchical structured overlay network based on locality sensitive hashing scheme.

Scalable and Robust Structured Peer-to-Peer Based on Balanced Kautz Tree

Experiences Deploying Peer-To-Peer Network For A Distributed File System

Efficient File Search in Non-Dht P2P Networks

On Maximum Stability with Enhanced Scalability in High-Churn DHT Deployment

LHT: A Low-Maintenance Indexing Scheme over DHTs

An Implementation and Optimization for Scalable DHT Crawler

Splider: A split-based crawler of the BT-DHT network and its applications