Abstract:With the booming of high-resolution Earth observation and open-data efforts, petabyte-scale Earth observation data have been available for free access. Due to the unprecedented availability of big data deluge, regional to global spatio-temporal analysis has been significantly challenged with the huge computational barriers, the tedious cycles of download-preprocess-store-analyze leading to excessive data downloading overhead, and the acquisition-oriented 2-D file-based structure, which is not fit for spatio-temporal analysis. The Earth observation data cube (EODC) paradigm revolutionizes the traditional way of storing, managing, and analyzing spatio-temporal RS data, and solves problems of easy-to-use of RS data to a certain extent. However, different EODC solutions are becoming information silos. Therefore, the sharing and joint use of remote sensing (RS) data across EODCs have become extremely challenging. To address the abovementioned challenges, we proposed a method of in-memory distributed data cube autodiscovery and retrieval across clouds. We construct a distributed in-memory data orchestration across clouds to shield the heterogeneity of the EODC storage solutions, solving information silos problems, and we put forward a larger-sites-first and spatio-temporal aware RS data discovery strategy, which can automatically discover data across clouds for requirements. Based on the data cube paradigm, this article proposes a quality-first data filtering strategy, which can help users to filter out high-quality data covering the target spatio-temporal range from the huge amount of data, and solve the problem of data cube joint retrieval and efficient use across clouds. In addition, we have confirmed that our method is effective and efficient through comparative experiments.

A collaborative analysis framework for distributed gridded environmental data

A Novel and Extensible Remote Sensing Collaboration Platform: Architecture Design and Prototype Implementation

A Tile-Based Framework with a Spatial-Aware Feature for Easy Access and Efficient Analysis of Marine Remote Sensing Data

CAFE: Carbon-Aware Federated Learning in Geographically Distributed Data Centers

An On-Demand Processing Framework for Faster Remote Sensing Big Data Analysis

A Tile Service-Driven Architecture for Online Climate Analysis with an Application to Estimation of Ocean Carbon Flux

Analyzing large-scale Data Cubes with user-defined algorithms: A cloud-native approach

CM-DataONE: A Framework for collaborative analysis of climate model output

A Scalable Framework for Large-Scale Distributed Collaboration

Research on a Kind of Web Based Distributed Forest Remote Sensing Parallel Processing Service

A2CI: A Cloud-based, Service-oriented Geospatial Cyberinfrastructure to Support Atmospheric Research

The design of a streaming analytical workflow for processing massive transit feeds

Towards A Framework For Offering Remote Sensing Data In An Analysis-Ready Format

An In-memory Data-Cube Aware Distributed Data Discovery Across Clouds for Remote Sensing Big Data

A Kind of Lidar Application Grid Based on eScience’s View

A Distributed Parallel Computing Environment for Bioinformatics Problems

A System for Exploratory Analysis in Cloud

A framework on task configuration and execution for distributed geographical simulation

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

CAFE: An Integrated Web App for High-Dimensional Analysis and Visualization in Spectral Flow Cytometry

Accelerating Remote Sensing Data Analysis Workflows by On-demand Processing