Abstract:We introduce Cloud DIKW as an analysis environment supporting scientific discovery through integrated parallel batch and streaming processing, and apply it to one representative domain application: social media data stream clustering. Recent work demonstrated that high-quality clusters can be generated by representing the data points using high-dimensional vectors that reflect textual content and social network information. Due to the high cost of similarity computation, sequential implementations of even single-pass algorithms cannot keep up with the speed of real-world streams. This paper presents our efforts to meet the constraints of real-time social stream clustering through parallelization. We focus on two system-level issues. Most stream processing engines like Apache Storm organize distributed workers in the form of a directed acyclic graph, making it difficult to dynamically synchronize the state of parallel workers. We tackle this challenge by creating a separate synchronization channel using a pub-sub messaging system. Due to the sparsity of the high-dimensional vectors, the size of centroids grows quickly as new data points are assigned to the clusters. Traditional synchronization that directly broadcasts cluster centroids becomes too expensive and limits the scalability of the parallel algorithm. We address this problem by communicating only dynamic changes of the clusters rather than the whole centroid vectors. Our algorithm under Cloud DIKW can process the Twitter 10% data stream in real-time with 96-way parallelism. By natural improvements to Cloud DIKW, including advanced collective communication techniques developed in our Harp project, we will be able to process the full Twitter stream in real-time with 1000-way parallelism. Our use of powerful general software subsystems will enable many other applications that need integration of streaming and batch data analytics.

HTwitt: a hadoop-based platform for analysis and visualization of streaming Twitter data

Socialanalysis: A Real-Time Query And Mining System From Social Media Data Streams

Towards Large-scale Twitter Mining for Drug-related Adverse Events.

Tweet Insights: A Visualization Platform to Extract Temporal Insights from Twitter

Real-time Twitter data analysis using Hadoop ecosystem

Personalized Knowledge Visualization In Twitter

Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture

The Unified Logging Infrastructure for Data Analytics at Twitter

Identifying disaster-related tweets and their semantic, spatial and temporal context using deep learning, natural language processing and spatial analysis: a case study of Hurricane Irma

A framework to simplify pre-processing location-based social media big data for sustainable urban planning and management

Early Warning Signals of Social Instabilities in Twitter Data

Real Time Big Data Sentiment Analysis and Classification of Facebook

On Identifying Disaster-Related Tweets: Matching-based or Learning-based?

Sub-story detection in Twitter with hierarchical Dirichlet processes

TiDeH: Time-Dependent Hawkes Process for Predicting Retweet Dynamics

Parallel clustering of high-dimensional social media data streams

Identification and Classification of Informative Tweets During Disasters

A Twitter Tale of Three Hurricanes: Harvey, Irma, and Maria

Visualizing Streaming Text Data with Dynamic Maps

Hashing it Out: Predicting Unhealthy Conversations on Twitter

Descriptive and visual summaries of disaster events using artificial intelligence techniques: case studies of Hurricanes Harvey, Irma, and Maria