SymWorkManager: A Novel HPC Multi-Thread Programming Model

Xiaohui Wei,Hongliang Li,Xing Shaocheng,Li Ji,Zhenhua H. Zane,Yuan Shutao

DOI: https://doi.org/10.1109/FGCNS.2008.146

2009-01-01

Abstract:The resources in a grid are distributed in multiple autonomic domains. Hence, it’s always a time consuming process to perform the resource co-allocation in grids especially for parallel jobs due to the diversity of local domain policies and the ...

What problem does this paper attempt to address?

Multi-discipline computations and collaborative visualization on the campus grid

Yingjie Xia,Guanghua Song,Yao Zheng

DOI: https://doi.org/10.3321/j.issn:1671-4512.2007.z2.029

2007-01-01

Abstract:A campus grid is developed and deployed to integrate high performance computing resources in Zhejiang University.The architecture is based on a hierarchical combination of two grid middleware products,Globus Toolkit and Sun Grid Engine,in order to make use of their advantages.The campus grid is applied for scientific computation,bioinformatics,computational fluid dynamics,etc.Some test cases using Message Passing Interface(MPI) show that the campus grid can provide powerful and stable computing cycles.The campus grid also provides two collaborative visualization services,the local rendering and the server rendering,which can be customized by the user.Finally,the collaborative visualization service facilitates users to research and analyze cooperatively.
A Flexible Architecture for Job Management in A Grid Environment

Luan Cui-ju,Song Guang-hua,Zheng Yao,Zhang Ji-fa

DOI: https://doi.org/10.1631/jzus.2007.a0095

2006-01-01

Journal of Zhejiang University SCIENCE A

Abstract:Job management is a key issue in computational grids, and normally involves job definition, scheduling, executing and monitoring. However, job management in the existing grid middleware needs to be improved in terms of efficiency and flexibility. This paper addresses a flexible architecture for job management with detailed design and implementation. Frameworks for job scheduling and monitoring, as two important aspects, are also presented. The proposed job management has the advantages of reusability of job definition, flexible and automatic file operation, visual steering of file transfer and job execution, and adaptive application job scheduler. A job management wizard is designed to implement each step. Therefore, what the grid user needs to do is only to define the job by constructing necessary information at runtime. In addition, the job space is adopted to ensure the security of the job management. Experimental results showed that this approach is user-friendly and system efficient.
A Public Grid Computing Framework Based on a Hierarchical Combination of Middleware

Yingjie Xia,Yao Zheng,Yudang Li

DOI: https://doi.org/10.1007/11610496_92

2006-01-01

Abstract:Grids, as one implementation of public computing, can be deployed by some middleware to integrate all kinds of resources across institutional boundaries, to tackle complex problems in scientific and engineering computation, data storage, scientific visualization, etc. In this paper, we propose a public computing grid framework built upon the hierarchical combination of two popular middleware products, Globus Toolkit (GT) and Sun Grid Engine (SGE). The communication between them can be conducted by Transfer-queue Over Globus (TOG). We discuss issues related to the deployment and the usage of this kind of hybrid grids. We examine it using serial programs and parallel program codes, and compare it with the grids deployed by only GT or SGE, respectively. The case studies imply that this hybrid grid is suitable for loosely coupled applications with high computational complexity.
A Deadline And Budget Constrained Cost-Time Optimization Algorithm For Scheduling Dependent Tasks In Grid Computing

Haolin Feng,Guanghua Song,Yao Zheng,Jun Xia

DOI: https://doi.org/10.1007/978-3-540-24680-0_15

2004-01-01

Abstract:Computational grid has a promising future in large-scale computing, because it enables the sharing of widely distributed computing resources. Good managements with excellent scheduling algorithms are in great demand to take full advantage of it. Many scheduling algorithms in grid computing are for independent tasks. However, communications are very common in scientific computing programs. In this paper, we will propose an easy-implemented algorithm to schedule the tasks with some communications. Our algorithm is suitable for a large proportion of scientific computing programs, and is based on Binary Integer Programming. It is able to meet the users' quality of service (QoS) requirements, and to minimize the combination of costs and time consumed by the users' programs. We will give an example of scheduling a typical scientific computing task to show the power of our algorithm. In our experiment, the grid resource consists of an SGI Onyx 3900 supercomputer, four SGI Octane workstations, four Intel P4-2.OGHz PCs and four Intel P4-1.8GHz PCs.
Application-adaptive Resource Scheduling in a Computational Grid

Luan Cui-ju,Song Guang-hua,Zheng Yao

DOI: https://doi.org/10.1631/jzus.2006.a1634

2006-01-01

Abstract:Selecting appropriate resources for running a job efficiently is one of the common objectives in a computational grid. Resource scheduling should consider the specific characteristics of the application, and decide the metrics to be used accordingly. This paper presents a distributed resource scheduling framework mainly consisting of a job scheduler and a local scheduler. In order to meet the requirements of different applications, we adopt HGSA, a Heuristic-based Greedy Scheduling Algorithm, to schedule jobs in the grid, where the heuristic knowledge is the metric weights of the computing resources and the metric workload impact factors. The metric weight is used to control the effect of the metric on the application. For different applications, only metric weights and the metric workload impact factors need to be changed, while the scheduling algorithm remains the same. Experimental results are presented to demonstrate the adaptability of the HGSA.
A Novel WSRF and Multi-Agent based Distributed System Resource Management Scheme

ZhiHui Lv,Shiyong Zhang,Jie Wu,Weiming Fu,YiPing Zhong

DOI: https://doi.org/10.1109/GCC.2007.21

2007-01-01

Abstract:Wide-area distributed systems like computational grids are emergent infrastructures for high-performance parallel applications. On these systems, communication mechanisms have to deal with many issues, including: private networks, heterogeneity, dynamic ...
Multisite Task Scheduling On Distributed Computing Grid

Weizhe Zhang,Hongli Zhang,Hui He,Mingzeng Hu

DOI: https://doi.org/10.1007/978-3-540-24680-0_8

2004-01-01

Abstract:Multisite task scheduling plays more and more important role in the grid computing as the WAN becomes faster and faster. Through the development of a three-level architecture of the distributed computing grid model and a grid schedule model, a scalable environment for multisite task scheduling is put forward. Then, a multisite Distributed Scheduling Server is designed and its prototype is implemented. A heuristic strategy, Clustering-based Grid Resource Selection algorithm, is described. Experiment indicates the scheduler and the algorithm are effective.
Multisite job scheduling models in computaional grid environment

Weizhe Zhang,Mingzeng Hu,Bin Li,Lei Gao

DOI: https://doi.org/10.13245/j.hust.2005.s1.014

2005-01-01

Abstract:Cooperation of massive parallel processor systems in different management-domains lies on the design of multi-domain job scheduling models. Three multisite grid job scheduling models, named local, sharing, multisite job scheduling models, are presented to overcome the heterogeneity and low bandwidth in computational grid. The job performance models on different scheduling models are analyzed in theory. The analysis results show that when the sites are homogeneous, multisite job scheduling should prefer the local and single site resources to multisite; when the sites are heterogeneous, these heuristics are invalid so that new heuristics should be designed to achieve the sub-optimum solutions.
GEO-WMS: an Improved Approach to Geoscientific Workflow Management System on HPC

Jiaxu Guo,Yidan Xu,Haohuan Fu,Wei Xue,Lin Gan,Mengxuan Tan,Tingye Wu,Yutong Shen,Xianwei Wu,Liang Hu,Xilong Che

DOI: https://doi.org/10.1007/s42514-022-00131-x

2022-01-01

Abstract:With the increasing complexity of scientific computing, it is imperative to enhance the efficiency and ease of High Performance Computing (HPC) utilization. Scientific workflow is introduced to that aim, but the current infrastructure still needs optimization. In this paper, we discuss the current problems based on scientific computing scenarios and design a more user-friendly workflow system solution targeting HPC services. In the proposed solution, we introduce a structured method to describe the workflow and employ a more user-friendly interface for scientific workflows to bring a better experience than traditional command line approaches. We have integrated a variety of methods to enhance the user experience during geoscience experiments. Data analytics are being used to make more intelligent recommendations to users. Runtime predictions help users to better plan their schedules for research. The statistics of the testing period and user feedback show that the proposed workflow management system can effectively save the operating time and complexity of the scientists, while saving computing resources. Our proposed system has a variety of advantageous features, including the ease of use, uniform specification with scalability, improved utilization of computing resources, and exemplary significance.
Multisite Co-Allocation Algorithms for Computational Grid

Weizhe Zhang,Albert M. K. Cheng,Mingzeng Hu

DOI: https://doi.org/10.1109/ipdps.2006.1639652

2006-01-01

Abstract:Efficient multisite job scheduling facilitates the cooperation of multi-domain massively parallel processor systems in a computing grid environment. However, co-allocation, heterogeneity, adaptability, and scalability emerge as tough challenges for the design of multisite job scheduling models and algorithms. This paper presents a new multisite job scheduling schema based on the multisite job scheduling model and the performance model for a heterogeneous grid environment. There are three key components: resource selection, reservation, and backfilling. The optimal and greedy-heuristic adaptive resource selection strategies are introduced. The conservative and easy backfilling are incorporated into the backfilling procedure. Experiments indicate that the scheduler and the algorithm are effective and perform better than a non-adaptive algorithm.
Adaptive Parallelism Load Balancing Strategy Design and Realize for Desktop Grid

Gui Xiaolin

2005-01-01

Abstract:Through the study of different computing system architecture, a load balancing strategy with fault-tolerence and adaptive parallelism is designed for desktop grid master-worker programming model. Using Java technology, prototype systems with this load balancing strategy has been setup on campus network. Different complexity Monte Carlo simulation applications as benchmark, ideal speedup is achieved. The total computing capacity which is used exceeds 90%.
Resource Management and Scheduling for High Performance Computing Application Based on WSRF

Chuliang Weng,Minglu Li,Xinda Lu

DOI: https://doi.org/10.1007/978-3-540-31849-1_69

2005-01-01

Abstract:In this paper, we challenge the issue of resource management and scheduling in the grid context, which is compliant with WS-Resource Framework. Firstly, we focus on the high performance application, and model the large-scale scientific computing problem that can be decomposed into multiple sub-problems, which evolves to be represented by a DAG. Then, a hierarchical infrastructure for resource management in the grid context is proposed in accordance with the specifications of WS-Resource Framework. Thirdly, we discuss the scheduling issue in the presented scenario and present a modification of the DLS algorithm. At last we analyze the presented modified algorithm with simulation experiments.
Hybrid Workload Scheduling on HPC Systems

Yuping Fan,Paul Rich,William Allcock,Michael Papka,Zhiling Lan

DOI: https://doi.org/10.48550/arXiv.2109.05412

2021-09-12

Abstract:Traditionally, on-demand, rigid, and malleable applications have been scheduled and executed on separate systems. The ever-growing workload demands and rapidly developing HPC infrastructure trigger the interest of converging these applications on a single HPC system. Although allocating the hybrid workloads within one system could potentially improve system efficiency, it is difficult to balance the tradeoff between the responsiveness of on-demand requests, the incentive for malleable jobs, and the performance of rigid applications. In this study, we present several scheduling mechanisms to address the issues involved in co-scheduling on-demand, rigid, and malleable jobs on a single HPC system. We extensively evaluate and compare their performance under various configurations and workloads. Our experimental results show that our proposed mechanisms are capable of serving on-demand workloads with minimal delay, offering incentives for declaring malleability, and improving system performance.

Distributed, Parallel, and Cluster Computing
How OpenMP Applications Get More Benefit from Many-Core Era

Jianian Yan,Jiangzhou He,Wentao Han,Wenguang Chen,Weimin Zheng

DOI: https://doi.org/10.1007/978-3-642-13217-9_7

2010-01-01

Abstract:With the approaching of the many-core era, it becomes more and more difficult for a single OpenMP application to efficiently utilize all the available processor cores. On the other hand, the available cores become more than necessary for some applications. We believe executing multiple OpenMP applications concurrently will be a common usage model in the future. In this model, how threads are scheduled on the cores are important as cores are asymmetric. We have designed and implemented a prototype scheduler, SWOMPS, to help schedule the threads of all the concurrent applications system-widely. The scheduler makes its decision based on underlying hardware configuration as well as the hints of scheduling preference of each application provided by users. Experiment evaluation shows SWOMPS is quite efficient in improving the performance. With the help of SWOMPS, we compared exclusive running one application and concurrent running multiple applications in term of system throughput and individual application performance. In various experimental comparisons, concurrent execution outperforms in throughput, meanwhile the performance slowdown of individual applications in concurrent execution is reasonable.
Interconnection Network Energy-Aware Workflow Scheduling Algorithm on Heterogeneous Systems

Xiaoyong Tang,Weiqiang Shi,Fan Wu

DOI: https://doi.org/10.1109/tii.2019.2962531

IF: 12.3

2019-01-01

IEEE Transactions on Industrial Informatics

Abstract:Heterogeneous systems based on multicore (CPU) and manycore (GPU) processors have been regarded as an important computing infrastructure in recent years. Large-scale computationally intensive scientific workflow applications have recently been deployed on such systems. However, improving the system performance and reducing the energy consumption under user deadline constraints remain challenging problems. In this article, we first investigate the computing node network energy consumption problem of fat-tree interconnection networks for a low communication-to-computation ratio workflow application. We then propose a heuristic list-based network energy-efficient workflow scheduling (NEEWS) algorithm including top-level task computing, task subdeadline initialization, a dynamic adjustment, and an edge data optimization communication method. Extensive simulations were conducted based on randomly generated workflow applications and two real-world scientific applications. The experiment results clearly demonstrate that our proposed workflow scheduling strategy outperforms three other algorithms in terms of energy consumption. In particular, NEEWS is extremely suitable owing to its high parallelism and low communication in large-scale scientific applications.
Standardizing Resource Selection and Access on Computational Grids

Yu Huashan,Xu Zhuoqun

DOI: https://doi.org/10.1109/gcc.2006.88

2006-01-01

Abstract:Standardization of resource interfaces and behaviors is critical for realizing the Grid vision. Most works up to now mainly focus on technologies of accessing remote and heterogeneous resources. A twolayer framework for selecting and accessing resources on computational Grids is presented. It groups resources across organizations in an application oriented manner, and virtualizes each group of resources to be a set of stateful WEB services. These WEB services provide the capabilities for combining resources within, as well as across, organizations to prepare customized job execution environments as required. For every dynamically created execution environment, its lifecycle is allowed to span as long as desired to execute multiple jobs, and its interface and behaviors are standardized in an application oriented way. The framework has been implemented to virtualize resources in Harmonia, which is a software platform for coordinative computation and is utilized for developing the campus Grid of Peking University.
Multisite Co-Allocation Scheduling Algorithms for Parallel Jobs in Computing Grid Environments

Zhang Weizhe,Fang Binxing,Hu Mingzeng,Liu Xinran,Zhang Hongli,Gao Lei

DOI: https://doi.org/10.1007/s11432-006-2034-2

2006-01-01

Science in China Series F Information Sciences

Abstract:Cooperation of multi-domain massively parallel processor systems in computing grid environment provides new opportunities for multisite job scheduling. At the same time, in the area of co-allocation, heterogeneity, network adaptability and scalability raise the challenge for the international design of multisite job scheduling models and algorithms. It presents multisite job scheduling schema through the introduction of multisite job scheduling model and the performance model under the grid environment. It introduces two job multisite and cooperative scheduling models and algorithms with the core of the optimal and greedy-heuristic resource selection strategies. Meanwhile, compared with single and multisite cooperative scheduling models and algorithms introduced by Sabin, Yahyapour and other persons, the validity and advance of the scheduling model and the performance model herein are proved.
A QoS-Aware and Fault-Tolerant Workflow Composition for Grid

Jiang Bian,Chuliang Weng,Jie Du,Minglu Li

DOI: https://doi.org/10.1109/gcc.2008.95

2008-01-01

Abstract:The continuous increasing of computing power in electric system computing places a threshold to the single host use and suggests an approach based on distributed computing. An emerging solution is grid technology, which allows organization to make better use of existing computing resources by providing them with a single, transparent, aggregated source of computing power. New generation of grid infrastructure, where Web services are building blocks, allow management of a Web services workflow. In this paper, a novel strategy - QoS-aware and fault-tolerant workflow composition - has been applied to utilize Web services and business process execution language for overcoming the issues about task assignment, security, flexibility and workflow management. This new strategy is lightweight Web services based computing power-sharing architecture, and it is not only suitable for executing computing works which are able to run in batches, but also be able to solve current issues in Web services based computing application such as system resilience, fault tolerance, efficiency of job scheduling and the instability in congested network environment.
Coordinating Distributed Resources for Complex Scientific Computation

Huashan Yu,Zhuoqun Xu,Wenkui Ding

DOI: https://doi.org/10.1007/978-3-540-30141-7_19

2004-01-01

Abstract:There exist a large number of computational resources in the domain of scientific and engineering computation. They are distributed, heterogeneous and often too restricted in computability for oneself to satisfy the requirement of modem scientific problems. To address this challenge, this paper proposes a component-based architecture for managing and accessing legacy applications on the computational grid. It automatically schedules legacies with domain expertise, and coordinates them to serve large-scale scientific computation. A prototype has been implemented to evaluate the architecture.
Union: An Automatic Workload Manager for Accelerating Network Simulation

Xin Wang,Misbah Mubarak,Yao Kang,Robert B. Ross,Zhiling Lan

DOI: https://doi.org/10.1109/IPDPS47924.2020.00089

2024-04-04

Abstract:With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time.

Distributed, Parallel, and Cluster Computing

SymWorkManager: A Novel HPC Multi-Thread Programming Model

Multi-discipline computations and collaborative visualization on the campus grid

A Flexible Architecture for Job Management in A Grid Environment

A Public Grid Computing Framework Based on a Hierarchical Combination of Middleware

A Deadline And Budget Constrained Cost-Time Optimization Algorithm For Scheduling Dependent Tasks In Grid Computing

Application-adaptive Resource Scheduling in a Computational Grid

A Novel WSRF and Multi-Agent based Distributed System Resource Management Scheme

Multisite Task Scheduling On Distributed Computing Grid

Multisite job scheduling models in computaional grid environment

GEO-WMS: an Improved Approach to Geoscientific Workflow Management System on HPC

Multisite Co-Allocation Algorithms for Computational Grid

Adaptive Parallelism Load Balancing Strategy Design and Realize for Desktop Grid

Resource Management and Scheduling for High Performance Computing Application Based on WSRF

Hybrid Workload Scheduling on HPC Systems

How OpenMP Applications Get More Benefit from Many-Core Era

Interconnection Network Energy-Aware Workflow Scheduling Algorithm on Heterogeneous Systems

Standardizing Resource Selection and Access on Computational Grids

Multisite Co-Allocation Scheduling Algorithms for Parallel Jobs in Computing Grid Environments

A QoS-Aware and Fault-Tolerant Workflow Composition for Grid

Coordinating Distributed Resources for Complex Scientific Computation

Union: An Automatic Workload Manager for Accelerating Network Simulation