A Scalable Real-Time Burst Detection System based on Kleinberg Algorithm

Min Luo,Yamanaka Akihiro,Masato Sawada

Abstract:Real-time burst detection is an important topic in data stream analysis. An effective way to find burst events is using a sliding time-window to record the frequency of evens that happened in each slide interval. Although many burst detection methods have been conducted based on this idea, only a handful of them are suitable for real-time detection. In this paper, we propose a real-time burst detection system based on Kleinberg algorithm and Jubatus. It is able to detect 100 events’ burst whose arriving frequency is of several milliseconds. By deploying our proposal on Jubatus framework, 10x times larger numbers of events could be detected at the same accuracy with the same regular specs of PCs. According to our knowledge, this is the first system for scalable real-time burst detection. Experimental evaluations of our prototype system on tweets and financial data analysis demonstrate the feasibility and scalability of our proposal.