Abstract:In complex network, the links among the nodes fabricate thousands of polygons, where triangles are the most foundamental ones. As more links are added to the network, more polygons are split to be triangles. Discovering the triangles can help understand the current status of the network. Due to the limitations of internal memory, discovering all the triangles in a large-scale real-world network, e.g., online social network (OSN), is challenging and has intrigued researchers for years. Currently, parallel computation frameworks, e.g., MapReduce, have been employed to address this issue. The last reducer is required to load all the neighbors of each node into internal memory, which causes “the curse of the last reducer.” The scale-free characteristic in real-world networks further exacerbate the applicability of the parallel triangle discovery algorithms. The current efforts on this issue mainly focus on dividing a large network into several small subnetworks and later employ internal memory algorithms. In this paper, we address this issue from a different perspective by employing the power-law degree distribution in scale-free networks. We theoretically and empirically prove that one node in scale-free network has a limited number of neighbors with a higher degree. Employing this finding, an novel parallel triangle discovery algorithm called degree partition (DePart) is designed. Since only the portion of neighbors with higher degrees are loaded into internal memory in the reducers, DePart solves “the curse of the last reducer” problem and works efficiently in clusters with limited internal memory. We develop DePart using MapReduce framework and examine the performance of DePart on five large real-world networks. The empirical testing results demonstrate that DePart has a favorable tradeoff between internal memory and total disk space and outperforms the current state-of-the-art MapReduce-based triangle discovery algorithms.

A New Data Streaming Method for Locating Hosts with Large Connection Degree

A Data Streaming Method for Monitoring Host Connection Degrees of High-Speed Links

A New Virtual Indexing Method for Measuring Host Connection Degrees

A New Sketch Method for Measuring Host Connection Degree Distribution

A new connection degree calculation and measurement method for large scale network monitoring.

Virtual indexing based methods for estimating node connection degrees

Detecting Stealthy Spreaders Using Online Outdegree Histograms

Characteristic measurement of the connection degree for network monitoring

Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams

Utilizing Dynamic Properties of Sharing Bits and Registers to Estimate User Cardinalities over Time

A novel spread estimation based abnormal flow detection in high-speed networks

Reverse Hashing for Sketch-based Change Detection on High-speed Networks

Reverse Hashing For High-Speed Network Monitoring: Algorithms, Evaluation, And Applications

A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams

Hashing Pursuit for Online Identification of Heavy-Hitters in High-Speed Network Streams

A Fast Sampling Method of Exploring Graphlet Degrees of Large Directed and Undirected Graphs.

Scalable Triangle Discovery Algorithm for Large Scale-Free Network with Limited Internal Memory

Monitoring Flow-level High-speed Data Streams with Reversible Sketches

Anomaly Detection of Large Scale Network Based on Data Streams

Locality-Sensitive Sketching for Resilient Network Flow Monitoring

Fine-grained Probability Counting for Cardinality Estimation of Data Streams.