Abstract:Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six iss ues. In order to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processi ng (RTDP) architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarc hical computing model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA, GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the dat a and then upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system design.

Private Cloud Computing System Based on Dynamic Service Adaptable to Large-Scale Data Processing

Visual Analysis of Cloud Computing Performance Using Behavioral Lines

A System for Exploratory Analysis in Cloud

The Public Service Framework Supported by High Performance Computing Based on Cloud Computing and Facing Intelligent Traffic Massive Data

A Policy of Task Allocation Base on Distributed Cluster Computing Towards Cloud

Resource Monitoring and Optimization System on Private Cloud Platform

The performance of MapReduce: an in-depth study

A Scheduler System for Large-Scale Distributed Data Computing in Cloud

Studies on the Large Scale Data Processing Technologies Used in Servers for Cloud Computing

Deployment of Batch Mode Scientific Workflow on a Computation-as-a-Service Private Cloud

Building a Productive Domain-Specific Cloud for Big Data Processing and Analytics Service

A Cloud-Based System for Spatial Analysis Service

Private Cloud System Based on BOINC with Support for Parallel and Distributed Simulation

Cloud Computing Service Framework for Bioinformatics Tools

The Performance of MapReduce

Energy-efficient Multi-task Scheduling Based on MapReduce for Cloud Computing

Efficient B-tree Based Indexing for Cloud Data Processing.

Study and Implementation of Elastic Stream Computing in the Cloud

Providing Scalable Database Services On The Cloud

Real-Time Big Data Processing Framework: Challenges and Solutions

A Kind of Cloud Storage Model Research Based on Massive Information Processing