Abstract:Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six iss ues. In order to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processi ng (RTDP) architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarc hical computing model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA, GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the dat a and then upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system design.

Exploring Real-Time Data Processing Using Big Data Frameworks

An efficient architecture for processing real-time traffic data streams using apache flink

Beyond Batch Processing: Towards Real-Time and Streaming Big Data

Data Processing Framework Using Apache and Spark Technologies in Big Data

Evaluating New Approaches of Big Data Analytics Frameworks

Real-time Intelligent Big Data Processing:Technology, Platform, and Applications

Real-time intelligent big data processing: technology, platform, and applications

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Typical Big Data Computing Frameworks

Research and Design of Real Time Big Data Visualization Analysis Platform Based on Flink

Real-Time Big Data Processing Framework: Challenges and Solutions

A review on big data real-time stream processing and its scheduling techniques

Real-time Text Analytics Pipeline Using Open-source Big Data Tools

Benchmarking Distributed Stream Data Processing Systems

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

Framing Apache Spark in life sciences

Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison for Enterprise and Scientific Computing

A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning

Large-scale Real-time Data-driven Scientific Applications

Architectural Impact on Performance of In-memory Data Analytics: Apache Spark Case Study

Revolutionizing Supply Chain Management: Real-time Data Processing and Concurrency