Bandwidth-Greedy Hashing for Massive-Scale Concurrent Flows

Tian Pan,Bin Liu,Xiaoyu Guo,Yang Li,Haoyu Song

DOI: https://doi.org/10.1109/icdcs.2016.23

2016-01-01

Abstract:The explosion of network bandwidth poses greatchallenges to data-plane flow processing. Due to the variable andpoor worst-case performance, naive hash table is incapable ofwire-speed processing. State-of-the-art schemes rely on multiplehash functions for enhanced load balancing to improve the worst-case performance. These schemes exploit the memory hierarchyand allocate compact on-chip data structures as the off-chiphash table summaries. However, when the flow number inflates, they fail to scale the on-chip memory consumption gracefully. This work is inspired by modern DRAM's burst-transfer feature. Specifically, we propose bandwidth-greedy hashing which resolveshash collisions with just one DRAM burst. Besides, load balancingefforts are made in an "on-demand" fashion. This radicaldesign surmounts the major obstacle of mapping multiple choicehashing schemes to real-world hierarchical memory systems formassive-scale items. Essentially, this solution follows a designpattern of on-demand load balancing and can be regarded asa generalization of closed hashing. To establish its theoreticalbase, we analyze it via Poisson distribution approximation. Theevaluation on DRAMSim2 reports that our scheme requires onlyone DRAM burst access (in 99.999% cases) and minuscule on-chip memory (less than 16MB, or 1% of the previous) to supportlookups for 100M flows at a throughput of 122.82Mpps.