Abstract:High-performance computing (HPC) environment is the core information infrastructure supporting national scientific and technological innovation, economic development and national defense construction.High-performance computing powers around the world have been building wide-area HPC environments based on multi-supercomputing center resources.However, in the high-performance computing environment, there are many kinds of resources and wide geographical distribution, which cannot effectively exert the aggregation effect of resources, and it is difficult to meet the requirements of large-scale applications for unified management and efficient access to wide-area distributed data.To this end, a complete set of technologies were proposed, which could be used to build wide-area global virtual data space, including virtual data space model, cross-domain virtual data space constructing, efficiently migrating data in a wide-area environment, co-scheduling of storage resources and computing job and cross-domain high concurrency data aggregation processing, etc.Based on the above, a virtual data space system has been developed for the national high-performance computing environment (NHPCE), which can effectively support the unified and efficient access to the wide area distributed heterogeneous storage resources, and the distributed data in the wide-area environment can be shared and cooperative processed in a cross-domain manner.At present, the system was experimental deployed in NHPCE and three typical large-scale applications, such as molecular docking, genome-wide association study and weather forecasting model, have been verified.The verification results show that the developed technology and software system can effectively aggregate the wide area distributed storage resources and meet the data space requirements of large-scale applications.

A Wide-Area Collaborative Scheduling System Oriented to Big Data Processing Applications

Performance-Driven Task and Data Co-scheduling Algorithms for Data-Intensive Applications in Grid Computing

GCSS: a Global Collaborative Scheduling Strategy for Wide-Area High-Performance Computing

Virtual Data Space System for National High-Performance Computing Environment

Joint Online Optimization of Task Rescheduling and Data Redistribution

GVDS:a Global Virtual Data Space for Wide-Area High-Performance Computing Environments

Unified Data Management System for Large Scale Task Parallel Computing

A Scheduling System for Big Data Hybrid Computing Workflow

Efficient Data and Task Co-Scheduling for Scientific Workflow in Geo-Distributed Datacenters

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Hypergraph-partitioning-based online joint scheduling of tasks and data

A Scheduler System for Large-Scale Distributed Data Computing in Cloud

Joint Scheduling of Data and Computation in Geo-Distributed Cloud Systems

A Novel Architecture Of Scheduling System For Big Data Framework

Research on Scheduling Strategy of Information Coordination Mechanism for Big Data Storage and Computing

Optimization of Big Data Parallel Scheduling Based on Dynamic Clustering Scheduling Algorithm

OPTIMAL ASSIGNMENT STRATEGY FOR DYNAMIC WORKFLOW OF REMOTE SENSING BIG DATA PROCESSING

Design And Implementation Of Task Scheduling Strategies For Massive Remote Sensing Data Processing Across Multiple Data Centers

Storage-Aware Task Scheduling for Performance Optimization of Big Data Workflows.

Scheduling Framework and Algorithms for Large-Scale Divisible Load Processing with Multi-Dimensional QoS Constraints

Dynamic Workflow Engine of Atmospheric Big Remote Sensing Data Processing Powered by Heterogenous Platform for High Performance Computing