Introduction to Big data Technology

Bilal Abu-Salih,Pornpit Wongthongtham,Dengya Zhu,Kit Yan Chan,Amit Rudra
DOI: https://doi.org/10.1007/978-981-33-6652-7_2
2021-04-15
Abstract:Big data is no more "all just hype" but widely applied in nearly all aspects of our business, governments, and organizations with the technology stack of AI. Its influences are far beyond a simple technique innovation but involves all rears in the world. This chapter will first have historical review of big data; followed by discussion of characteristics of big data, i.e. from the 3V's to up 10V's of big data. The chapter then introduces technology stacks for an or-ganization to build a big data application, from infrastruc-ture/platform/ecosystem to constructional units and components. Finally, we provide some big data online resources for reference.
Other Computer Science
What problem does this paper attempt to address?
The paper attempts to address issues primarily focused on big data technology and its applications. Specifically: 1. **Characteristics and Challenges of Big Data**: With the explosive growth of data volume, traditional methods of data processing, analysis, retrieval, storage, and visualization are no longer applicable. This data comes from a variety of sources, such as sensor-generated data, social media, transaction records, etc. The paper explores various characteristics of big data (from 3V to 10V), including Volume, Velocity, Variety, and emphasizes Veracity, Variability, Validity, Vulnerability, Volatility, Visualization, and Value. 2. **Information Overload in Business Decision-Making**: Although many enterprises strive to become data-driven companies, the proportion of truly successful ones is not high. Information overload makes decision-making more difficult, thus necessitating a re-examination of internal business processes and a review of the tools used to collect, transmit, store, and analyze large amounts of data. 3. **Cloud Computing Solutions**: To address the challenges brought by big data, the paper introduces various service models (SaaS, PaaS, IaaS) and deployment models (public cloud, private cloud, community cloud, hybrid cloud) of cloud computing. These solutions aim to improve the flexibility, cost-effectiveness, and security of data processing. In summary, this paper aims to help readers better understand and cope with the challenges brought by big data by reviewing its historical development, exploring its characteristics, and introducing corresponding technical solutions.