Abstract:Hadoop, especially Hadoop 2.0, has been a dominant framework for real-time big data processing. However, Hadoop is not optimized for energy efficiency. Aiming to solve this problem, in this paper, we propose a new framework to improve the energy efficiency of Hadoop 2.0. We focus on the resource manager in Hadoop 2.0, namely YARN, and propose energy-efficient task scheduling mechanisms on YARN. Particularly, we focus on CPU-intensive streaming jobs and classify streaming jobs into two types, namely batch streaming jobs ( i.e., a set of jobs are submitted simultaneously) and online streaming jobs ( i.e., jobs are continuously submitted one by one). We devise different energy-efficient task scheduling algorithms for each kind of streaming jobs. Specially, we first propose to abstractly model performance and energy consumption by considering the characteristics of tasks as well as the computational resources in YARN. Based on this model, we study the energy efficiency of streaming tasks which consist of the performance model and energy consumption model of task. We propose two key principles for improving energy efficiency: 1) CPU usage aware task allocation, partitions tasks to NMs based on the task characteristic in term of CPU usage; and 2) resource efficient task allocation, reduce idle resource. Then, we propose a D-based binning algorithm for the batch task scheduling and K-based binning algorithm for the online task scheduling that can adapt to continuously arriving tasks. We conduct extensive experiments on a real Hadoop 2.0 cluster and use two kinds of workloads to evaluate the performance and energy efficiency of our proposal. Compared with Storm ( the streaming data processing tool in Hadoop 2.0) and other approaches including TAPA and DVFS-MR, our proposal is more energy efficient. The batch task scheduling algorithm reduces up to 10 percent of energy consumption and keeps comparable performance. In addition, the online task scheduling algorithm reduces up to 7 percent over the existing algorithms.

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

Adaptive Scheduling Parallel Jobs with Dynamic Batching in Spark Streaming

Adaptive Scheduling of Parallel Jobs in Spark Streaming.

TATA: Throughput-Aware TAsk Placement in Heterogeneous Stream Processing with Deep Reinforcement Learning

A Cost-Efficient Scheduling Algorithm for Streaming Processing Applications on Cloud

Parameter and Resource Coordination Adjustment Strategy for Spark Streaming

Runtime-Aware Adaptive Scheduling in Stream Processing

An On-the-Fly Scheduling Strategy for Distributed Stream Processing Platform.

Communication-Efficient Task Scheduling for Real-Time Distributed Computing.

An Efficient Scheduling Algorithm for Stream Computing.

Energy-Efficient Task Scheduling for CPU-Intensive Streaming Jobs on Hadoop.

Elastic Allocator: An Adaptive Task Scheduler for Streaming Query in the Cloud

An enforcement of real time scheduling in Spark Streaming.

Comparative Analysis of Energy-Efficient Scheduling Algorithms for Big Data Applications.

Pec: Proactive Elastic Collaborative Resource Scheduling in Data Stream Processing

A Dynamic Two-Phase Schedule Strategy for Streaming Media Edge Cloud

Towards Low-Latency Batched Stream Processing by Pre-Scheduling

E-AHRW: An Energy-Efficient Adaptive Hash Scheduler for Stream Processing on Multi-core Servers.

Performance Prediction Method for Stream Computing Platform Based on Time Series

Efficient and Extensible Scheduling of Streaming Applications on Communication-Aware Heterogeneous Platforms

A Power-Conserving Online Scheduling Scheme for Video Streaming Services.