Abstract:Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six iss ues. In order to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processi ng (RTDP) architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarc hical computing model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA, GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the dat a and then upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system design.

DataCloud: an Efficient Massive Data Mining and Analysis Framework on Large Clusters.

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

An Efficient Massive Data Processing Model in the Cloud -- A Preliminary Report

Visual Analysis of Cloud Computing Performance Using Behavioral Lines

Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

Efficient B-tree Based Indexing for Cloud Data Processing.

Mass Data Analysis and Forecasting Based on Cloud Computing

Building a Productive Domain-Specific Cloud for Big Data Processing and Analytics Service

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

A System for Exploratory Analysis in Cloud

Cloud Computing Service Framework for Bioinformatics Tools

A Kind of Cloud Storage Model Research Based on Massive Information Processing

Cloud Storage of Massive Remote Sensing Data Based on Distributed File System

The performance of MapReduce: an in-depth study

Power Big Data Analysis Platform Design Based on Hadoop

Research on Service-Oriented Data Mining Engine Based on Cloud Computing

Moving big data to the cloud

Analysis of Big Data Platform with OpenStack and Hadoop.

Real-Time Big Data Processing Framework: Challenges and Solutions

A System for Parallel Data Mining Service on Cloud

Adaptive priority-based data placement and multi-task scheduling in geo-distributed cloud systems