Abstract:Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six iss ues. In order to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processi ng (RTDP) architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarc hical computing model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA, GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the dat a and then upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system design.

A Kind of Cloud Storage Model Research Based on Massive Information Processing

Cloud Storage As the Infrastructure of Cloud Computing

An Efficient Massive Data Processing Model in the Cloud -- A Preliminary Report

Cloud Computing:Concept and Platform

Cloud Storage of Massive Remote Sensing Data Based on Distributed File System

Massive Electronic Records Processing for Digital Archives in Cloud

Efficient B-tree Based Indexing for Cloud Data Processing.

Towards a New Model of Storage and Access to Data in Big Data and Cloud Computing

A distributed storage method of remote sensing data based on image blocks organization

Research on Distributed Storage Technology of Database Big Data Based on Cloud Computing

Cloud Computing and Big Data: A Review of Current Service Models and Hardware Perspectives

Research on Cloud Computing Security & Privacy Protection of Massive Unstructured Multi-Modal Data

Research on Semantic++ Computing Based on Big Data Environment

Real-Time Big Data Processing Framework: Challenges and Solutions

DataCloud: an Efficient Massive Data Mining and Analysis Framework on Large Clusters.

The Design and Implementation of Geographic Information Storage System Based on the Cloud Platform.

Big Data Storage Index Mechanism Based on Spatiotemporal Information Cloud Platform

Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

A Cloud Computing Platform Technology Based on Space-based Intelligent Information Network

An Open, Flexible and Multilevel Data Storing and Processing Platform for Very Large Scale Sensor Network

Cloud Network and Mathematical Model Calculation Scheme for Dynamic Big Data