Abstract:MapReduce has been widely used in Hadoop for parallel processing larger-scale data for the last decade. However, remote-sensing (RS) algorithms based on the programming model are trapped in dense disk I/O operations and unconstrained network communication, and thus inappropriate for timely processing and analyzing massive, heterogeneous RS data. In this paper, a novel in-memory computing framework called Apache Spark (Spark) is introduced. Through its merits of transferring transformation to in-memory datasets of Spark, the shortages are eliminated. To facilitate implementation and assure high performance of Spark-based algorithms in a complex cloud computing environment, a strip-oriented parallel programming model is proposed. By incorporating strips of RS data with resilient distributed datasets (RDDs) of Spark, all-level parallel RS algorithms can be easily expressed with coarse-grained transformation primitives and BitTorrent-enabled broadcast variables. Additionally, a generic image partition method for Spark-based RS algorithms to efficiently generate differentiable key/value strips from a Hadoop distributed file system (HDFS) is implemented for concealing the heterogeneousness of RS data. Data-intensive multitasking algorithms and iteration-intensive algorithms were evaluated on a Hadoop yet another resource negotiator (YARN) platform. Experiments indicated that our Spark-based parallel algorithms are of great efficiency, a multitasking algorithm took less than 4 h to process more than half a terabyte of RS data on a small YARN cluster, and 9*9 convolution operations against a 909-MB image took less than 260 s. Further, the efficiency of iteration-intensive algorithms is insensitive to image size.

Parallelization of Classification Algorithms Based on SparkR

Parallelization of Machine Learning Algorithms Respectively on Single Machine and Spark

Parallel naive Bayes algorithm for large-scale Chinese text classification based on spark

Parallel Implementation Of Classification Algorithms Based On Mapreduce

The Parallelization of Back Propagation Neural Network in MapReduce and Spark

A Fast Parallel Random Forest Algorithm Based on Spark

Study of ELM Algorithm Parallelization Based on Spark

Design and Implementation of Parallel DBSCAN Algorithm Based on Spark

A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment

A Performance Evaluation of Classification Algorithms for Big Data.

Parallel multi-label K-nearest neighbor algorithm based on Spark

The parallel algorithms for LIBSVM parameter optimization based on Spark

A kind of hyperspectral image classification parallel method based on Spark

Microblog Sentiment Classification Using Parallel SVM in Apache Spark

A Fast Parallel Attribute Reduction Algorithm Using Apache Spark.

In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model

A Parallel DBSCAN Algorithm Based on Spark

Multiple Submodels Parallel Support Vector Machine on Spark

Research on Parallelisation of Collaborative Filtering Recommendation Algorithm Based on Spark

The Parallel Implementation of Deep Neural Decision Tree Based on Spark