Abstract:Data analysis is an important functionality in cloud computing which allows a huge amount of data to be processed over very large clusters. MapReduce is recognized as a popular way to handle data in the cloud environment due to its excellent scalability and good fault tolerance. However, compared to parallel databases, the performance of MapReduce is slower when it is adopted to perform complex data analysis tasks that require the joining of multiple data sets in order to compute certain aggregates. A common concern is whether MapReduce can be improved to produce a system with both scalability and efficiency. In this paper, we introduce Map-Join-Reduce, a system that extends and improves MapReduce runtime framework to efficiently process complex data analysis tasks on large clusters. We first propose a filtering-join-aggregation programming model, a natural extension of MapReduce's filtering-aggregation programming model. Then, we present a new data processing strategy which performs filtering-join-aggregation tasks in two successive MapReduce jobs. The first job applies filtering logic to all the data sets in parallel, joins the qualified tuples, and pushes the join results to the reducers for partial aggregation. The second job combines all partial aggregation results and produces the final answer. The advantage of our approach is that we join multiple data sets in one go and thus avoid frequent checkpointing and shuffling of intermediate results, a major performance bottleneck in most of the current MapReduce-based systems. We benchmark our system against Hive, a state-of-the-art MapReduce-based data warehouse on a 100-node cluster on Amazon EC2 using TPC-H benchmark. The results show that our approach significantly boosts the performance of complex analysis queries.

MapReduce for Large-Scale Monitor Data Analyses

The Performance of MapReduce

The performance of MapReduce: an in-depth study

Distributed data management using MapReduce

An Analytical Performance Model of MapReduce

Distributed Parallel PCA for Modeling and Monitoring of Large-Scale Plant-Wide Processes with Big Data.

Plant-Wide Industrial Process Monitoring: A Distributed Modeling Framework

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Predicting Heart Diseases from Large Scale IoT Data Using a Map-Reduce Paradigm

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

A Big Data Management Architecture for Standardized IoT Based on Smart Scalable SNMP.

MAP-JOIN-REDUCE: Toward Scalable and Efficient Data Analysis on Large Clusters

Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads

Large-Scale Social Network Analysis Based on MapReduce

IoTDeM: An IoT Big Data-oriented MapReduce performance prediction extended model in multiple edge clouds.

Online Internet Traffic Measurement and Monitoring Using Spark Streaming

Continuous, Online Monitoring and Analysis in Large Water Distribution Networks

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Research of a MapReduce Model to Process the Traffic Big Data

Online Internet Traffic Monitoring System Using Spark Streaming