Abstract:Remotely-sensed satellite image fusion is indispensable for the generation of long-term gap-free Earth observation data. While cloud computing (CC) provides the big picture for RS big data (RSBD), the fundamental question of the efficient fusion of RSBD on CC platforms has not yet been settled. To this end, we propose a lightweight cloud-native framework for the elastic processing of RSBD in this study. With the scaling mechanisms provided by both the Infrastructure as a Service (IaaS) and Platform as a Services (PaaS) of CC, the Spark-on-Kubernetes operator model running in the framework can enhance the efficiency of Spark-based algorithms without considering bottlenecks such as task latency caused by an unbalanced workload, and can ease the burden to tune the performance parameters for their parallel algorithms. Internally, we propose a task scheduling mechanism (TSM) to dynamically change the Spark executor pods’ affinities to the computing hosts. The TSM learns the workload of a computing host. Learning from the ratio between the number of completed and failed tasks on a computing host, the TSM dispatches Spark executor pods to newer and less-overwhelmed computing hosts. In order to illustrate the advantage, we implement a parallel enhanced spatial and temporal adaptive reflectance fusion model (PESTARFM) to enable the efficient fusion of big RS images with a Spark aggregation function. We construct an OpenStack cloud computing environment to test the usability of the framework. According to the experiments, TSM can improve the performance of the PESTARFM using only PaaS scaling to about 11.7%. When using both the IaaS and PaaS scaling, the maximum performance gain with the TSM can be even greater than 13.6%. The fusion of such big Sentinel and PlanetScope images requires less than 4 min in the experimental environment.

A generic framework for the development of geospatial processing pipelines on clusters

A new architecture paradigm for image processing pipeline applied to massive remote sensing data production

CLUSTER-BASED OCEAN REMOTE SENSING IMAGE FUSION PARALLEL COMPUTING STRATEGY

A new framework of cluster-based parallel processing system for high-performance geo-computing

A Mechanism of Remote Sensing Image for Parallel Processing Base on Splitting Blocks

OpenCluster: A Flexible Distributed Computing Framework for Astronomical Data Processing

An On-Demand Processing Framework for Faster Remote Sensing Big Data Analysis

Recent Developments in Parallel and Distributed Computing for Remotely Sensed Big Data Processing

A MapReduce Computing Framework Based on GPU Cluster.

Study of High Performance Processing Platform Architecture in Remote Sensing

On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes

A Parallel Computing System Of Remote Sensing Images Based On Global Subdivision Model

A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

A framework for remote sensing images processing using deep learning technique

Analyzing large-scale Data Cubes with user-defined algorithms: A cloud-native approach

Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

RSIMS: Large-Scale Heterogeneous Remote Sensing Images Management System

On-Demand Processing for Remote Sensing Big Data Analysis

Cloud Computing Cloud Computing in Remote Sensing : High Performance Remote Sensing Data Processing in a Big data Environment

A Novel Approach For Geospatial Computational Task Processing In Grid Environment