Research Advance on MapReduce Based Big Data Processing Platforms and Algorithms
Jie SONG,Zong-Zhe SUN,Ke-Ming MAO,Yu-Bin BAO,Ge YU
DOI: https://doi.org/10.13328/j.cnki.jos.005169
2017-01-01
Abstract:This paper introduces the research advance on MapReduce based big data processing platforms.Frist,twelve typical MapReduce based data processing platforms are descripted,their implementation principles and application areas are compared,and their commonalities are concluded.Second,the MapReduce based big data processing algorithms,including search algorithms,data cleansing/transformation algorithms,aggregation algorithms,join algorithms,sorting algorithms,optimization algorithms,preference query algorithms,graph algorithms,and data mining algorithms,are studied.These algorithms are classified by their MapReduce implementations,and the factors that affect their performance are analyzed.Finally,big data processing algorithms are abstracted as the out-of-core algorithms whose performance features are well analyzed.The considerations,ideas and challenges of universal optimizations on the performance of out-of-core algorithms are proposed as references for researchers.These optimizations include optimizing algorithms' I/O cost and locality,and designing incremental iterative algorithms.Comparing the current topics,such as resource allocation and task scheduling based dynamic optimizations on platform,parallelization for specific algorithms,and performance optimizations on iterative algorithms,the proposed static optimizations serve as complements that highlight new areas for the researchers.