Abstract:Very High Resolution satellite and aerial imagery are used to monitor and conduct large scale surveys of ecological systems. Convolutional Neural Networks have successfully been employed to analyze such imagery to detect large animals and salient features. As the datasets increase in volume and number of images, utilizing High Performance Computing resources becomes necessary. In this paper, we investigate three task-parallel, data-driven workflow designs to support imagery analysis pipelines with heterogeneous tasks on HPC. We analyze the capabilities of each design when processing datasets from two use cases for a total of 4,672 satellite and aerial images, and 8.35 TB of data. We experimentally model the execution time of the tasks of the image processing pipelines. We perform experiments to characterize the resource utilization, total time to completion, and overheads of each design. Based on the model, overhead and utilization analysis, we show which design is best suited to scientific pipelines with similar characteristics.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to design and implement an efficient task - parallel, data - driven workflow when dealing with high - resolution satellite image analysis, in order to support the execution of heterogeneous tasks on high - performance computing (HPC) resources. As the scale and quantity of data sets keep increasing, it becomes necessary to utilize high - performance computing resources. By studying three different task - parallel workflow designs, this paper aims to solve the following specific problems: 1. **Support for heterogeneous tasks**: Different tasks may require different numbers of CPUs and GPUs, implement different functions, and also have different running times. There are data - dependency relationships among these tasks, so effective scheduling, correct resource binding, and efficient data management are required. 2. **Challenges in large - scale data processing**: With the growth of data sets, the need to process a large number of images makes parallel processing necessary. However, the existing high - performance computing infrastructure is more inclined to execute a single long - term task, which poses challenges to large - scale data processing. 3. **Performance optimization**: When dealing with large - scale image data sets, how to optimize resource utilization, reduce completion time, and lower overhead are the focuses of this paper. By experimentally modeling task execution time, the author experimentally analyzes the resource utilization, total completion time, and overhead of each design. 4. **Guidance on design choices**: Due to the lack of architecture and performance analysis, it becomes difficult to choose functionally equivalent implementation schemes. By comparing the performance of the three designs, this paper provides a basis for choosing the best design scheme, especially for scientific workflows with similar characteristics. In summary, the main objective of this paper is to provide guidance on performance optimization and design choices for processing large - scale high - resolution satellite image analysis by comparing three different workflow designs.

Comparing Workflow Application Designs for High Resolution Satellite Image Analysis

Workflow Design Analysis for High Resolution Satellite Image Analysis

High-throughput Analysis of Large Microscopy Image Datasets on CPU-GPU Cluster Platforms

A Mechanism of Remote Sensing Image for Parallel Processing Base on Splitting Blocks

A new architecture paradigm for image processing pipeline applied to massive remote sensing data production

Apache Spark Accelerated Deep Learning Inference for Large Scale Satellite Image Analytics

High Performance Hyperspectral Image Classification using Graphics Processing Units

Improving the Performance of Hyperspectral Image and Signal Processing Algorithms Using Parallel, Distributed and Specialized Hardware-Based Systems

Parallel Optimization for Large Scale Interferometric Synthetic Aperture Radar Data Processing

A Workflow-Serialized Parallel Spatial Idw Interpolation on Windows Hpc

Benchmarking the Processing of Aircraft Tracks with Triples Mode and Self-Scheduling

A review of parallel computing for large-scale remote sensing image mosaicking

Parallel programing templates for remote sensing image processing on GPU architectures: design and implementation

Analyzing the HCP Datasets using GPUs: The Anatomy of a Science Engagement

Parallel Geospatial Analysis on Windows HPC Platform

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

Parallel Versus Distributed Data Access for Gigapixel-Resolution Histology Images: Challenges and Opportunities

A Parallel Computing System Of Remote Sensing Images Based On Global Subdivision Model

The Design of Dynamic Grid Workflow Web Portal for Remote Sensing Information Service

A Workflow for Automated Satellite Image Processing: from Raw VHSR Data to Object-Based Spectral Information for Smallholder Agriculture

Workflows to driving high-performance interactive supercomputing for urgent decision making