Abstract:Currently, in addition to the performance, the energy consumption (hereinafter EC) of jobs running in a big data processing systemis also of interest to academia and industry because it grows rapidly as an increasing amount of data is processed. Many studies focus on the EC optimization of jobs from the perspective of computation, which is specific to the algorithms in each job. However, the part of EC involved in I/O operations, which is general and universal, is mostly ignored in optimization. In this paper, we concentrate on the EC optimization of jobs from the perspective of I/O operations. To save energy, we argue that data compression could be exploited. On one hand, energy is saved by processing compressed data with less I/O cost. On the other hand, extra EC is incurred from the necessary data compression/decompression process, which may offset the saved energy. Therefore, there are tradeoffs to consider when determining whether to compress data for these jobs. In this paper, such tradeoffs and boundary conditions are studied. We first abstract a paradigm for the runtime environment of big data processing jobs. Then, we establish the power, jobs, compression, and I/O models in detail. Based on these models, we discuss the compression tradeoffs and derive the boundary conditions for EC optimization. Finally, we design and conduct experiments to validate our proposition. The experimental results confirm that the tradeoffs and boundary conditions exist for typical jobs in MapReduce and Spark. As explained, first, the EC of a job is reduced using data compression. Second, whether or not such optimization occurs is related to the specification of both the compression algorithm and the job and is determined by corresponding boundary conditions. Third, for a compression algorithm, the larger its compression/decompression speed and the better its compression ratio, the more likely it is to achieve EC optimization.

IC-Data: Improving Compressed Data Processing in Hadoop.

Data Compression and Storage under High Speed Network

Benefit of Compression in Hadoop: A Case Study of Improving IO Performance on Hadoop

Improving I/O Throughput of Scientific Applications Using Transparent Parallel Compression

Towards Higher Efficiency in a Distributed Memory Storage System Using Data Compression

Efficient Finer-Grained Incremental Processing with MapReduce for Big Data

Improving the I/O Throughput for Data-Intensive Scientific Applications with Efficient Compression Mechanisms

A self-aware data compression system on FPGA in Hadoop

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Compress Blocks or Not: Tradeoffs for Energy Consumption of a Big Data Processing System

An Optimized Iterative Semantic Compression Algorithm And Parallel Processing for Large Scale Data.

Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications

Parallel Data Compression Techniques

An Uncoupled Data Process and Transfer Model for MapReduce.

FASTA/Q Data Compressors for MapReduce-Hadoop Genomics:Space and Time Savings Made Easy -- Version 1

Homomorphic Compression: Making Text Processing on Compression Unlimited.

Compressed Data Direct Computing for Databases

CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

Overview of Caching Mechanisms to Improve Hadoop Performance

Analysis and Optimization of Massive Data Processing on High Performance Computing Architecture

DEC: An Efficient Deduplication-Enhanced Compression Approach