Abstract:Abstract Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools insufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems: (1) traditional data processing techniques are inadequate to handle big data; (2) data preprocessing needs substantial manual effort; (3) domain knowledge is required before the classification; (4) semantic explanation is ignored. In this work, these challenges are overcome by using different algorithms combined with a Naïve Bayes classifier to ensure reliability and highly precise recommendations in virtualization and cloud environments. These features make HTwitt different from others in terms of having an effective and practical design for text classification in big data analytics. The main contribution of the paper is to propose a framework for building landslide early warning systems by pinpointing useful tweets and visualizing them along with the processed information. We demonstrate the results of the experiments which quantify the levels of overfitting in the training stage of the model using different sizes of real-world datasets in machine learning phases. Our results demonstrate that the proposed system provides high-quality results with a score of nearly 95% and meets the requirement of a Hadoop-based classification system.

Sub-story detection in Twitter with hierarchical Dirichlet processes

Online Neural Sequence Detection with Hierarchical Dirichlet Point Process

Hierarchical Topic Modeling with Nested Hierarchical Dirichlet Process

A Knowledge-Based Semisupervised Hierarchical Online Topic Detection Framework.

Evolutionary Hierarchical Dirichlet Process for Timeline Summarization.

Hierarchical Dirichlet Processes with Social Influence

Retweet Behavior Prediction Using Hierarchical Dirichlet Process

A Semi-automatic Method for Efficient Detection of Stories on Social Media

Tracking and Connecting Topics Via Incremental Hierarchical Dirichlet Processes

Uncovering the Spatiotemporal Patterns of Collective Social Activity

Discovering Interesting Subgraphs in Social Media Networks

Detecting Abnormal Events Via Hierarchical Dirichlet Processes.

Learning-based topic detection using multiple features

A Fast Hierarchical Topic Detection Method

A Novel Hot Topic Detection Framework With Integration of Image and Short Text Information From Twitter

Real time event detection in twitter

SCStory: Self-supervised and Continual Online Story Discovery

The early bird catches the term: combining twitter and news data for event detection and situational awareness

Mining Multi-aspect Reflection of News Events in Twitter: Discovery, Linking and Presentation

HTwitt: a hadoop-based platform for analysis and visualization of streaming Twitter data

Hierarchical Dirichlet process for tracking complex topical structure evolution and its application to autism research literature