The Evolvement of Big Data Systems

- ChenGang,WuSai,WangYuan

DOI: https://doi.org/10.5555/2991325.2991435

IF: 3.3

2015-01-01

Big Data Research

Abstract:Recently, Google revealed that it has replaced the 10-year old MapReduce with its new systems (e.g., DataFlow) which can provide better performances and support more sophisticated applications. Sim...

What problem does this paper attempt to address?

An Overview of Big Data Industry in China

Liu Yue,He Jia,Guo Minjie,Yang Qing,Zhang Xinsheng

DOI: https://doi.org/10.1109/CC.2014.7019834

2014-01-01

China Communications

Abstract:The year of 2011 is considered the first year of big data market in China. Compared with the global scale, China's big data growth will be faster than the global average growth rate, and China will usher in the rapid expansion of big data market in the next few years. This paper presents the overall big data development in China in terms of market scale and development stages, enterprise development in the industry chain, the technology standards, and industrial applications. The paper points out the issues and challenges facing big data development in China and proposes to make polices and create support approaches for big data transactions and personal privacy protection.
Networks for Large Data Flows - Revolution or Evolutions?

weiqiang sun,weisheng hu

DOI: https://doi.org/10.1364/ps.2014.pw1b.1

2014-01-01

Abstract:We highlight the characteristics of traffic flows in the big data era - the flows are bulk in size but delay tolerant. We discuss the challenges of big data delivery, by examining the past and ongoing research activities. We also show possible directions of future work in both control and forward planes.
The performance of MapReduce: an in-depth study

Dawei Jiang,Beng Chin Ooi,Lei Shi,Sai Wu

DOI: https://doi.org/10.14778/1920841.1920903

2010-01-01

Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.
A novel agent-based parallel ETL system for massive data

Gang Chen,Baoran An,Yan Liu

DOI: https://doi.org/10.1109/CCDC.2016.7531673

2016-01-01

Abstract:Business Intelligence (BI) systems are crucial for enterprise improvement. They consolidate heterogeneous data from distributed sources and input high-quality data into strategic indicators. An essential component of the data consolidation is Extraction, Transformation and Loading (ETL) which are responsible for extracting data from heterogeneous sources, transforming, restructuring and integrating them into homogenous data warehouse. Due to the deficiency of traditional ETL, the entire ETL component for massive data has decreased performance. Aiming at this challenge, we propose a novel workflow framework for parallel ETL execution based on multi-agent system. The purpose of the system is to utilize a parallel strategy to improve the efficiency of ETL process. Through research, we find that some ETL activities are often executed on the same priority or using the same input data. Based on this discovery, this paper presents a parallel ETL framework based on agent theory and multi-thread techniques. The experimental results show that the proposed approach can greatly improve the efficiency of ETL process.
Traffic Data Processing At Age Of Big Data

Hong Zhang,Xiaoming Wang,Changsheng Zhu

2015-01-01

Abstract:Big data technique has brought a substantial change in the development of intelligent traffic and its activatory application is a new topic in intelligent traffic research. Based on strong real-time characteristics and high efficiency of traffic big data processing, a traffic data processing platform based on big data technique is built up. And the architecture of traffic big data processing system is proposed for this platform from methodological views. The main investigation topic of this architecture and its concerned theory, method and technique are discussed in connection with data source, data processing, knowledge database and application of the system, laying a foundation for detailed analysis and processing of traffic data.
A Survey on Emerging Computing Paradigms for Big Data

Yaoxue Zhang,Ju Ren,Jiagang Liu,Chugui Xu,Hui Guo,Yaping Liu

DOI: https://doi.org/10.1049/cje.2016.11.016

IF: 1.019

2017-01-01

Chinese Journal of Electronics

Abstract:The explosive growth of data volume and the ever-increasing demands of data value extraction have driven us into the era of big data. The “5V” (Variety, Velocity, Volume, Value, and Veracity) characteristics of big data pose great challenges to traditional computing paradigms and motivate the emergence of new solutions. Cloud computing is one of the representative technologies that can perform mas...
A Survey of Big Data Research

Hua Fang,Zhaoyang Zhang,Chanpaul Jin Wang,Mahmoud Daneshmand,Chonggang Wang,Honggang Wang

DOI: https://doi.org/10.1109/mnet.2015.7293298

IF: 10.294

2015-01-01

IEEE Network

Abstract:Big data create values for business and research, but pose significant challenges in terms of networking, storage, management, analytics, and ethics. Multidisciplinary collaborations from engineers, computer scientists, statisticians, and social scientists are needed to tackle, discover, and understand big data. This survey presents an overview of big data initiatives, technologies, and research in industries and academia, and discusses challenges and potential solutions.
Progressive online aggregation in a distributed stream system

Dingyu Yang,Jian Cao,Sai Wu,Jie Wang

DOI: https://doi.org/10.1016/j.jss.2014.11.027

IF: 3.5

2015-01-01

Journal of Systems and Software

Abstract:distributed weighted random sampling algorithm in distributed stream systems is novel.Multi-level query processing is presented to solve complex tasks and queries in distributed system.A synthetic processing topology strategy is provided to merge streams on partial repeated computing of overlap data.Early results with statistical estimations are continuously sent to users. Interactive query processing aims at generating approximate results with minimum response time. However, it is quite difficult for a batch-oriented processing system to progressively provide cumulatively accurate results in the context of a distributed environment. MapReduce Online extends the MapReduce framework to support online aggregation, but it is hindered by its processing speed in keeping up with ongoing real-time data events. We deploy the online aggregation algorithm over S4, a scalable stream processing system that is inspired by the combined functionalities of MapReduce and Actor model. Our system applies an asynchronous message communication mechanism from actor model to support online aggregation. It can process large scale data stream with high concurrency in a short response time. In this system, we adopt a distributed weighted random sampling algorithm to solve biased distribution between different streams. Furthermore, a multi-level query processing topology is developed to reduce overlapped processing for multiple queries. Our system can provide continuous window aggregation with a confidence interval and error bound. We have implemented our system and conducted plentiful experiments over the TPC-H benchmark. A large number of experiments are carried out to demonstrate that by using our system, high-quality query results can be generated within a short response time and that the approach outperforms MapReduce Online on data streams.
Visualization Analysis on the Hotspots and Frontier Evolution of International Big Data Research

Wang Yucan,Li Yifei,Yuan Qinjian

DOI: https://doi.org/10.3724/sp.j.1224.2014.00282

2014-01-01

JOURNAL OF ENGINEERING STUDIES

Abstract:With the comprehensive collection of“big data”relevant literature from international core jour-nals database of WoS, we employs co-word analysis as the basic method, together with some other tools such as social network analysis, clustering and strategic diagram, to reveal the hotspots and frontier evolution of big data research. The results show that the current big data research has the following five main aspects:algorithm design and development, platform and framework for practice, data analysis methods and statistical practice, semantics and ontology research and big data applications. The related research has continued to expand and deepen, and has gradually reflected the inheritance, while overall is still in its infancy. Thus, we also give our forecast about the next research frontier:1) data processing techniques and methods research in big data environment, including ap-plication research on data processing and statistical methods in big data context;2) semantic analysis and ontology research;3) the mining of commercial values of big data.
The anatomy of big data computing

Raghavendra Kune,Pramod Kumar Konugurthi,Arun Agarwal,Raghavendra Rao Chillarige,Rajkumar Buyya

DOI: https://doi.org/10.1002/spe.2374

2015-10-09

Abstract:Summary Advances in information technology and its widespread growth in several areas of business, engineering, medical, and scientific studies are resulting in information/data explosion. Knowledge discovery and decision‐making from such rapidly growing voluminous data are a challenging task in terms of data organization and processing, which is an emerging trend known as big data computing , a new paradigm that combines large‐scale compute, new data‐intensive techniques, and mathematical models to build data analytics. Big data computing demands a huge storage and computing for data curation and processing that could be delivered from on‐premise or clouds infrastructures. This paper discusses the evolution of big data computing, differences between traditional data warehousing and big data, taxonomy of big data computing and underpinning technologies, integrated platform of big data and clouds known as big data clouds, layered architecture and components of big data cloud, and finally open‐technical challenges and future directions. Copyright © 2015 John Wiley & Sons, Ltd.

computer science, software engineering
E3: an Elastic Execution Engine for Scalable Data Processing.

Gang Chen,Ke Chen,Dawei Jiang,Beng Chin Ooi,Lei Shi,Hoang Tam Vo,Sai Wu

DOI: https://doi.org/10.2197/ipsjjip.20.65

2012-01-01

Abstract:With the unprecedented growth of data generated by mankind nowadays, it has become critical to develop efficient techniques for processing these massive data sets. To tackle such challenges, analytical data processing systems must be extremely efficient, scalable, and flexible as well as economically effective. Recently, Hadoop, an open-source implementation of MapReduce, has gained interests as a promising big data processing system. Although Hadoop offers the desired flexibility and scalability, its performance has been noted to be suboptimal when it is used to process complex analytical tasks. This paper presents E3, an elastic and efficient execution engine for scalable data processing. E3 adopts a “middle” approach between MapReduce and Dryad in that E3 has a simpler communication model than Dryad yet it can support multi-stages job better than MapReduce. E3 avoids reprocessing intermediate results by adopting a stage-based evaluation strategy and collocating data and user-defined (map or reduce) functions into independent processing units for parallel execution. Furthermore, E3 supports block-level indexes, and built-in functions for specifying and optimizing data processing flows. Benchmarking on an in-house cluster shows that E3 achieves significantly better performance than Hadoop, or put it another way, building an elastically scalable and efficient data processing system is possible.
The Performance of MapReduce

Dawei Jiang,Beng Chin Ooi,Lei Shi,Sai Wu

DOI: https://doi.org/10.14778/1920841.1920903

IF: 2.5

2010-01-01

Proceedings of the VLDB Endowment

Abstract:MapReduce has been widely used for large-scale data analysis in the Cloud. The system is well recognized for its elastic scalability and fine-grained fault tolerance although its performance has been noted to be suboptimal in the database context. According to a recent study [19], Hadoop, an open source implementation of MapReduce, is slower than two state-of-the-art parallel database systems in performing a variety of analytical tasks by a factor of 3.1 to 6.5. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation; however, this approach of "renting more nodes" is not cost effective in a pay-as-you-go environment. Users desire an economical elastically scalable data processing system, and therefore, are interested in whether MapReduce can offer both elastic scalability and efficiency. In this paper, we conduct a performance study of MapReduce (Hadoop) on a 100-node cluster of Amazon EC2 with various levels of parallelism. We identify five design factors that affect the performance of Hadoop, and investigate alternative but known methods for each factor. We show that by carefully tuning these factors, the overall performance of Hadoop can be improved by a factor of 2.5 to 3.5 for the same benchmark used in [19], and is thus more comparable to that of parallel database systems. Our results show that it is therefore possible to build a cloud data processing system that is both elastically scalable and efficient.
Big Data Meet Green Challenges: Big Data Toward Green Applications.

Jinsong Wu,Song Guo,Jie Li,Deze Zeng

DOI: https://doi.org/10.1109/jsyst.2016.2550530

IF: 4.802

2016-01-01

IEEE Systems Journal

Abstract:Big data are widely recognized as being one of the most powerful drivers to promote productivity, improve efficiency, and support innovation. It is highly expected to explore the power of big data and turn big data into big values. To answer the interesting question whether there are inherent correlations between the two tendencies of big data and green challenges, a recent study has investigated the issues on greening the whole life cycle of big data systems. This paper would like to discover the relations between the trend of big data era and that of the new generation green revolution through a comprehensive and panoramic literature survey in big data technologies toward various green objectives and a discussion on relevant challenges and future directions.
Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Yun-Cong Zhang,Xiao-Yang Wang,Cong Wang,Yao Xu,Jian-Wei Zhang,Xiao-Dong Lin,Guang-Yu Sun,Gong-Lin Zheng,Shan-Hui Yin,Xian-Jin Ye,Li,Zhan Song,Dong-Dong Miao

DOI: https://doi.org/10.1007/s11390-020-9702-3

2020-01-01

Journal of Computational Science and Technology

Abstract:As data volumes grow rapidly, distributed computations are widely employed in data-centers to provide cheap and efficient methods to process large-scale parallel datasets. Various computation models have been proposed to improve the abstraction of distributed datasets and hide the details of parallelism. However, most of them follow the single-layer partitioning method, which limits developers to express a multi-level partitioning operation succinctly. To overcome the problem, we present the NDD (Nested Distributed Dataset) data model. It is a more compact and expressive extension of Spark RDD (Resilient Distributed Dataset), in order to remove the burden on developers to manually write the logic for multi-level partitioning cases. Base on the NDD model, we develop an open-source framework called Bigflow, which serves as an optimization layer over computation engines from most widely used processing frameworks. With the help of Bigflow, some advanced optimization techniques, which may only be applied by experienced programmers manually, are enabled automatically in a distributed data processing job. Currently, Bigflow is processing about 3 PB data volumes daily in the data-centers of Baidu. According to customer experience, it can significantly save code length and improve performance over the intuitive programming style.
Comparative Study on MapReduce and Spark for Big Data Analytics

Xin-Dong WU,Sheng-Wei JI

DOI: https://doi.org/10.13328/j.cnki.jos.005557

2018-01-01

Abstract:This paper reviews two state-of-the-art algorithmic architectures,MapReduce and Spark,and compares them from their backgrounds,principles and application scenarios.The advantages and their corresponding limitations of these two algorithms are summarized.When dealing with non-iterative problems,MapReduce,by virtue of its task scheduling strategy and shuffle mechanisms,performs better than Spark in terms of intermediate data transfers and number of files.Spark can be used to deal with iterative problems and low latency issues,as it divides a computing task according to the dependencies between the data and the task.Compared with MapReduce,Spark can effectively reduce the number of intermediate data transmissions and the number of synchronizations,and improve the running efficiency of computing systems.
Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

Chen Kang

DOI: https://doi.org/10.3969/j.issn.1008-5599.2009.11.004

2009-01-01

Abstract:Data processing technologies used in servers are the critical foundations for the cloud computing systems.This paper introduces the large scale data processing technologies used in the current cloud computing implementations.Such technologies are quite important for the data collections as well as processing.There are three key goals for processing the large scale data i.e.reliability,scalability and easy to program.Based on the Map Reduce programming paradigm from Google and Dryad model from Microsoft,how to achieve the goals in practice are discussed in detail.
Rethinking Abstractions for Big Data: Why, Where, How, and What

Mary Hall,Robert M. Kirby,Feifei Li,Miriah Meyer,Valerio Pascucci,Jeff M. Phillips,Rob Ricci,Jacobus Van der Merwe,Suresh Venkatasubramanian

DOI: https://doi.org/10.48550/arXiv.1306.3295

2013-06-14

Abstract:Big data refers to large and complex data sets that, under existing approaches, exceed the capacity and capability of current compute platforms, systems software, analytical tools and human understanding. Numerous lessons on the scalability of big data can already be found in asymptotic analysis of algorithms and from the high-performance computing (HPC) and applications communities. However, scale is only one aspect of current big data trends; fundamentally, current and emerging problems in big data are a result of unprecedented complexity--in the structure of the data and how to analyze it, in dealing with unreliability and redundancy, in addressing the human factors of comprehending complex data sets, in formulating meaningful analyses, and in managing the dense, power-hungry data centers that house big data. The computer science solution to complexity is finding the right abstractions, those that hide as much triviality as possible while revealing the essence of the problem that is being addressed. The "big data challenge" has disrupted computer science by stressing to the very limits the familiar abstractions which define the relevant subfields in data analysis, data management and the underlying parallel systems. As a result, not enough of these challenges are revealed by isolating abstractions in a traditional software stack or standard algorithmic and analytical techniques, and attempts to address complexity either oversimplify or require low-level management of details. The authors believe that the abstractions for big data need to be rethought, and this reorganization needs to evolve and be sustained through continued cross-disciplinary collaboration.

General Literature,Distributed, Parallel, and Cluster Computing
Data Deluge Dynamics: Tracing the Evolution and Ramifications of Big Data Phenomenon

Mohammed Yaseer Nazeer,Mohammad Tarik Nadir

DOI: https://doi.org/10.47772/ijriss.2024.805157

2024-01-01

International Journal of Research and Innovation in Social Science

Abstract:This paper presents a comprehensive review of the evolution, methodologies, challenges, and implications of Big Data in various domains. Big Data has emerged as a critical resource, offering unprecedented opportunities for decision-making, innovation, and societal advancement. The analysis delves into the historical trajectory of Big Data, examining its evolution from the early 2000s to its current status as a cornerstone of contemporary data-driven practices. Drawing on seminal works by Chen et al. (2014), Manyika et al. (2011), and Kitchin (2014), the review highlights the fundamental characteristics of Big Data, encapsulated by the “three Vs” – Volume, Velocity, and Variety – along with the emerging dimensions of Veracity and Value. Methodologically, the paper surveys the diverse approaches and technologies employed in Big Data analytics, ranging from descriptive and predictive analytics to advanced machine learning algorithms. Provost and Fawcett (2013) and Zikopoulos et al. (2011) provide valuable insights into the practical applications of these methodologies across sectors such as healthcare, finance, marketing, and governance. However, amidst the promise of Big Data lies a myriad of challenges, including data quality issues, scalability constraints, and ethical dilemmas. Davenport and Harris (2007) discuss the imperative of organizations to compete on analytics while navigating the complexities of managing large and heterogeneous datasets. Moreover, the paper examines the ethical, legal, and social considerations inherent in Big Data practices, emphasizing the importance of privacy, consent, fairness, transparency, and accountability. These concerns are further underscored by recent controversies surrounding data privacy breaches and algorithmic biases, prompting calls for enhanced regulatory frameworks and ethical guidelines. Looking ahead, the paper outlines future research directions in Big Data, including the development of ethical frameworks for governance, the integration of diverse data sources, and the exploration of emerging applications in smart cities, precision agriculture, and autonomous vehicles. In conclusion, while Big Data holds immense potential for driving innovation and progress, its responsible and ethical utilization is paramount to ensuring equitable and sustainable societal outcomes.
A Survey on Spark Ecosystem for Big Data Processing

Shanjiang Tang,Bingsheng He,Ce Yu,Yusen Li,Kun Li

DOI: https://doi.org/10.1109/TKDE.2020.2975652

2018-11-18

Abstract:With the explosive increase of big data in industry and academic fields, it is necessary to apply large-scale data processing systems to analysis Big Data. Arguably, Spark is state of the art in large-scale data computing systems nowadays, due to its good properties including generality, fault tolerance, high performance of in-memory data processing, and scalability. Spark adopts a flexible Resident Distributed Dataset (RDD) programming model with a set of provided transformation and action operators whose operating functions can be customized by users according to their applications. It is originally positioned as a fast and general data processing system. A large body of research efforts have been made to make it more efficient (faster) and general by considering various circumstances since its introduction. In this survey, we aim to have a thorough review of various kinds of optimization techniques on the generality and performance improvement of Spark. We introduce Spark programming model and computing system, discuss the pros and cons of Spark, and have an investigation and classification of various solving techniques in the literature. Moreover, we also introduce various data management and processing systems, machine learning algorithms and applications supported by Spark. Finally, we make a discussion on the open issues and challenges for large-scale in-memory data processing with Spark.

Distributed, Parallel, and Cluster Computing
CHAPTER BIG DATA MANAGEMENT ON WIRELESS SENSOR NETWORKS 5

Chih-Chieh Hung,Chu-Cheng Hsieh

Abstract:Recent development of various areas of information and communication technologies has contributed to an explosive growth in the volume of data. These data can be analyzed for insights that lead to better decisions and strategic business moves. Amazon Inc. shows us an example that uses data insights to improve business intelligence. The company gained a patent for what it calls anticipatory shipping, a method to start delivering packages even before customers click “buy.” This method takes previous orders, product searches, wish lists, shopping-cart contents, returns, and even how long an Internet user’s cursor hovers over an item. Analyzing such large-scale, rapid-generated, and various types of data helps Amazon Inc. shorten the delivery time from their hubs to customers, thereby earning great customer satisfactory. The data sets used in the above example are so-called big data. This term refers to data sets that so large or complex that traditional data processing applications are inadequate. Big data usually has three V characteristics: volume (the quantity of generated and stored data may not be easily handled by conventional databases), velocity (the speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development), and variety (data generated are from multiple sources and in multiple formats). These characteristics of big data bring not only huge opportunities but also huge challenges, including analysis, capture, data curation, search, sharing, storage, transfer, visualization, querying, updating, and so forth. Thanks to advanced development of big data techniques, these techniques include file systems for big data [e.g., Hadoop distributed file system (HDFS)], noSQL databases (e.g., HBase), data processing models for big data (e.g., MapReduce), streaming techniques for big data (e.g., Storm), query engines

Computer Science,Engineering

The Evolvement of Big Data Systems

An Overview of Big Data Industry in China

Networks for Large Data Flows - Revolution or Evolutions?

The performance of MapReduce: an in-depth study

A novel agent-based parallel ETL system for massive data

Traffic Data Processing At Age Of Big Data

A Survey on Emerging Computing Paradigms for Big Data

A Survey of Big Data Research

Progressive online aggregation in a distributed stream system

Visualization Analysis on the Hotspots and Frontier Evolution of International Big Data Research

The anatomy of big data computing

E3: an Elastic Execution Engine for Scalable Data Processing.

The Performance of MapReduce

Big Data Meet Green Challenges: Big Data Toward Green Applications.

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Comparative Study on MapReduce and Spark for Big Data Analytics

Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

Rethinking Abstractions for Big Data: Why, Where, How, and What

Data Deluge Dynamics: Tracing the Evolution and Ramifications of Big Data Phenomenon

A Survey on Spark Ecosystem for Big Data Processing

CHAPTER BIG DATA MANAGEMENT ON WIRELESS SENSOR NETWORKS 5