Abstract:Remotely-sensed satellite image fusion is indispensable for the generation of long-term gap-free Earth observation data. While cloud computing (CC) provides the big picture for RS big data (RSBD), the fundamental question of the efficient fusion of RSBD on CC platforms has not yet been settled. To this end, we propose a lightweight cloud-native framework for the elastic processing of RSBD in this study. With the scaling mechanisms provided by both the Infrastructure as a Service (IaaS) and Platform as a Services (PaaS) of CC, the Spark-on-Kubernetes operator model running in the framework can enhance the efficiency of Spark-based algorithms without considering bottlenecks such as task latency caused by an unbalanced workload, and can ease the burden to tune the performance parameters for their parallel algorithms. Internally, we propose a task scheduling mechanism (TSM) to dynamically change the Spark executor pods’ affinities to the computing hosts. The TSM learns the workload of a computing host. Learning from the ratio between the number of completed and failed tasks on a computing host, the TSM dispatches Spark executor pods to newer and less-overwhelmed computing hosts. In order to illustrate the advantage, we implement a parallel enhanced spatial and temporal adaptive reflectance fusion model (PESTARFM) to enable the efficient fusion of big RS images with a Spark aggregation function. We construct an OpenStack cloud computing environment to test the usability of the framework. According to the experiments, TSM can improve the performance of the PESTARFM using only PaaS scaling to about 11.7%. When using both the IaaS and PaaS scaling, the maximum performance gain with the TSM can be even greater than 13.6%. The fusion of such big Sentinel and PlanetScope images requires less than 4 min in the experimental environment.

SCASA: A Spark-Based Parallel Approach for Net Primary Productivity Calculation with CASA Model

Enhancing Terrestrial Net Primary Productivity Estimation with EXP-CASA: A Novel Light Use Efficiency Model Approach

Improved CASA model based on satellite remote sensing data: simulating net primary productivity of Qinghai Lake basin alpine grassland

Fine‐grained forest net primary productivity monitoring: Software system integrating multisource data and smart optimization

Assimilating multi-source remotely sensed data into a light use efficiency model for net primary productivity estimation

SciAP: A Programmable, High-Performance Platform for Large-Scale Scientific Data

Monitoring of Net Primary Production in California Rangelands Using Landsat and MODIS Satellite Remote Sensing

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

ParSA: High-throughput Scientific Data Analysis Framework with Distributed File System

On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes

Research Progress on Spatial-Temporal Dynamic Simulation Model of Net Primary Productivity of Terrestrial Ecosystems

A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

CPSA: A Coordinated Process Scheduling Algorithm for Coupled Earth System Model

Research On The Parallelization Of The Dbscan Clustering Algorithm For Spatial Data Mining Based On The Spark Platform

Developing an online estimation and query system of net primary productivity of China based on RIA technology

A Benchmarking Study to Evaluate Apache Spark on Large-Scale Supercomputers

Mining Area Skyline Objects from Map-based Big Data using Apache Spark Framework

Spark Cloud-Based Parallel Computing for Traffic Network Flow Predictive Control Using Non-Analytical Predictive Model

A Learning-Based Method for Efﬁcient Large-Scale Sensitivity Analysis and Tuning of Single Column Atmosphere Model (SCAM)

In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model

Sdac: Porting Scientific Data To Spark Rdds