Abstract:Most of the popular Big Data analytics tools evolved to adapt their working environment to extract valuable information from a vast amount of unstructured data. The ability of data mining techniques to filter this helpful information from Big Data led to the term Big Data Mining. Shifting the scope of data from small-size, structured, and stable data to huge volume, unstructured, and quickly changing data brings many data management challenges. Different tools cope with these challenges in their own way due to their architectural limitations. There are numerous parameters to take into consideration when choosing the right data management framework based on the task at hand. In this paper, we present a comprehensive benchmark for two widely used Big Data analytics tools, namely Apache Spark and Hadoop MapReduce, on a common data mining task, i.e., classification. We employ several evaluation metrics to compare the performance of the benchmarked frameworks, such as execution time, accuracy, and scalability. These metrics are specialized to measure the performance for classification task. To the best of our knowledge, there is no previous study in the literature that employs all these metrics while taking into consideration task-specific concerns. We show that Spark is 5 times faster than MapReduce on training the model. Nevertheless, the performance of Spark degrades when the input workload gets larger. Scaling the environment by additional clusters significantly improves the performance of Spark. However, similar enhancement is not observed in Hadoop. Machine learning utility of MapReduce tend to have better accuracy scores than that of Spark, like around 3%, even in small size data sets.

Big Data analytics. Three use cases with R, Python and Spark

Big Data Analysis in Stock Market Prediction

Benchmarking Apache Spark and Hadoop MapReduce on Big Data Classification

Usages of Spark Framework with Different Machine Learning Algorithms

Evaluating New Approaches of Big Data Analytics Frameworks

Technical Report: On the Usability of Hadoop MapReduce, Apache Spark & Apache Flink for Data Science

A Big Data Analysis Platform for Healthcare on Apache Spark

Emerging Trends and Challenges in Data Science and Big Data Analytics

Performance evaluation of Map-reduce jar pig hive and spark with machine learning using big data

Computational Performance Analysis of Cluster-based Technologies for Big Data Analytics

Predicting the ratings of Amazon products using Big Data

Basketball Data Analysis Using Spark Framework and K-Means Algorithm

Scalable Econometrics on Big Data -- The Logistic Regression on Spark

Big Data Processing Using Spark in Cloud

Run Time Prediction for Big Data Iterative ML Algorithms: a KMeans case study

Application of Improved Recommendation System Based on Spark Platform in Big Data Analysis

Mobile Big Data Analytics Using Deep Learning and Apache Spark

Random Forests for Big Data

Framing Apache Spark in life sciences

Predictive Analytics Accomplished by the Utilization of Social Big Data and Machine Learning

A Review of Data Mining, Big Data Analytics and Machine Learning Approaches