Abstract:With the Internet and data growth increasing trends, big data is becoming an extremely important and challenging problem for Data Centers. Many platforms and frameworks are working to bring a cutting edge technology to this problem. Apache Hadoop is a software framework addressing the big-data processing and storing on clusters, providing reliability, scala-bility and distributed computing. Hadoop has a distributed file system to store vast amount of data in distributed environments, and uses Map reduce algorithm to perform the computations and process large amount of data, by parallelizing the workload and storage. In comparison to other relational database systems, Hadoop works well with unstructured data. Our work is focused on performance evaluation of benchmarks of Hadoop, which are crucial for testing the infrastructure of the clusters. Taking into consideration the sensitiveness and importance of data, it’s inevitable testing the clusters and distributed systems before deploying. The benchmark results can lead to optimizing the parameters for an enhanced performance tuning of the cluster. In this paper, we are motivated to study and evaluate the performance of Hadoop and a comprehensive listing of bench-marks used to test Hadoop, while providing detailed information for their appliance and procedures to run them. We construct a distributed hadoop cluster simulation based on VmWare Workstation consisted of multiple nodes running hadoop. we have conducted a measurement study on the performance evaluation of hadoop cluster simulation under multiple scenarios. Our results demonstrates the trade-off between performance and flexibility. This evaluation study is focused on the throughput performance comparisons under different scenarios. The Hadoop performance has been evaluated for different size of data sets in this simulation. To measure the performance we set up a Hadoop cluster with many nodes and use the fileTestDFSIO.java of the Hadoop version 1.2.1 which gives us the data throughput, average I/O rate and I/O rate standard deviation. Our results demonstrates that the HDFS writing performance scales well on both small and big data set. The average HDFS reading performance scales well on big data set where it is, however, lower than on the small data set. The more nodes a writing/reading operation is run on, the faster its performance is.

Performance Evaluation of Apache Kafka – A Modern Platform for Real Time Data Streaming

Efficient topic partitioning of Apache Kafka for high-reliability real-time data streaming applications

A Survey on Networked Data Streaming With Apache Kafka

A study on Modern Messaging Systems- Kafka, RabbitMQ and NATS Streaming

Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison for Enterprise and Scientific Computing

Exploring Real-Time Data Processing Using Big Data Frameworks

On Efficiently Partitioning a Topic in Apache Kafka

Benchmarking Distributed Stream Data Processing Systems

Streaming Technologies and Serialization Protocols: Empirical Performance Analysis

Implementation and Performance Analysis of Apache Hadoop

Revolutionizing Supply Chain Management: Real-time Data Processing and Concurrency

Real-time Text Analytics Pipeline Using Open-source Big Data Tools

Real-time Event Joining in Practice With Kafka and Flink

Reproducible Experiments for Comparing Apache Flink and Apache Spark on Public Clouds

Kafka Consumer Group Autoscaler

High-Performance Real-Time Data Processing: Managing Data Using Debezium, Postgres, Kafka, and Redis

Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

The orchestration of Machine Learning frameworks with data streams and GPU acceleration in Kafka‐ML: A deep‐learning performance comparative

Comparative Evaluation of Java Virtual Machine-Based Message Queue Services: A Study on Kafka, Artemis, Pulsar, and RocketMQ

An efficient architecture for processing real-time traffic data streams using apache flink