Challenges and Reflections on Exascale Computing

Jinfu Yang,Xuejun

2014-01-01

Abstract:This paper introduces the development of the exascale （10^18） computing. Though exascalc computing is a hot research direction worldwide, we are facing many challenges in the areas of memory wall, communica- tion wall, reliability wall, power wall and scalability of parallel computing. According to these challenges, some thoughts and strategies are proposed.

What problem does this paper attempt to address?

Application software beyond exascale: challenges and possible trends

Guang-Wen Yang,Hao-Huan Fu

DOI: https://doi.org/10.1631/FITEE.1800459

2018-01-01

Abstract:With various exascale systems in different countries planned over the next three to five years, developing application software for such unprecedented computing capabilities and parallel scaling becomes a major challenge. In this study, we start our discussion with the current 125-Pflops Sunway TaihuLight system in China and its related application challenges and solutions. Based on our current experience with Sunway TaihuLight, we provide a projection into the next decade and discuss potential challenges and possible trends we would probably observe in future high performance computing software.
A Vision of Post-Exascale Programming.

Ji-Dong Zhai,Wen-Guang Chen

DOI: https://doi.org/10.1631/fitee.1800442

2018-01-01

Abstract:Exascale systems have been under development for quite some time and will be available for use in a few years. It is time to think about future post-exascale systems. There are many main challenges with regard to future post-exascale systems, such as processor architecture, programming, storage, and interconnect. In this study, we discuss three significant programming challenges for future post-exascale systems: heterogeneity, parallelism, and fault tolerance. Based on our experience of programming on current large-scale systems, we propose several potential solutions for these challenges. Nevertheless, more research efforts are needed to solve these problems.
Cluster and Grid Superservers: the Dawning Experiences in China

Zhiwei Xu,Ninghui Sun,Dan Meng,Wei Li

DOI: https://doi.org/10.1109/clustr.2001.959996

2001-01-01

Abstract:This paper summarizes recent activities at Institute of Computing Technology, Chinese Academy of Sciences, in developing superservers for cluster and grid computing. We first identify market and technical trends observed from a Chinese perspective. Then we describe the research work in developing the Dawning series high performance computers and the China computational grid. We also highlight some on-going research work in developing grid-oriented superserver systems.
Architecture, Challenges and Applications of Edge Computing

LI Linzhe,ZHOU Peilei,CHENG Peng,SHI Zhiguo

DOI: https://doi.org/10.11959/j.issn.2096-0271.2019010

2019-01-01

Abstract:Edge computing is a new type of computing models that performs computing tasks at the edge of the network.Compared with cloud computing,it can respond to user’s needs more quickly and reliably.Starting from the shortcomings of cloud computing,the concept and general architecture of edge computing were illustrated,and then two reference frameworks proposed by industry alliances were elaborated.Four challenges of edge computing and their latest research progress were summarized.With the development of theory and technology related to edge computing,it will become a key technology to promote the upgrade of Internet of things (IoT) services.For this reason,two applications of edge computing in manufacturing and security monitoring were introduced.
Programming at Exascale: Challenges and Innovations

Jalal Abdulbaqi

DOI: https://doi.org/10.48550/arXiv.1809.10023

2018-09-25

Abstract:Supercomputers become faster as hardware and software technologies continue to evolve. Current supercomputers are capable of 1015 floating point operations per second (FLOPS) that called Petascale system. The High Performance Computer (HPC) community is Looking forward to the system with capability of 1018 (FLOPS) that is called Exascale. Having a system to thousand times faster than the previous one produces challenges to the high performance computer (HPC) community. These challenges require innovation in software and hardware. In this paper, the challenges posed for programming at Exascale systems are reviewed and the developments in the main programming models and systems are surveyed.

Distributed, Parallel, and Cluster Computing
Challenges Towards Deploying Data Intensive Scientific Applications on Extreme Heterogeneity Supercomputers

Hang Liu,Yufei Ding,Da Zheng,Seung Woo Son,Da Yan

DOI: https://doi.org/10.48550/arXiv.1804.09738

2018-04-26

Abstract:Shrinking transistors, which powered the advancement of computing in the past half century, has stalled due to power wall; now extreme heterogeneity is promised to be the next driving force to feed the needs of ever-increasingly diverse scientific domains. To unlock the potentials of such supercomputers, we identify eight potential challenges in three categories: First, one needs fast data movement since extreme heterogeneity will inevitably complicate the communication circuits -- thus hampering the data movement. Second, we need to intelligently schedule suitable hardware for corresponding applications/stages. Third, we have to lower the programming complexity in order to encourage the adoption of heterogeneous computing.

Distributed, Parallel, and Cluster Computing
Editorial for the Special Issue on HPC Algorithms and Applications

Haohuan Fu,Wei Xue,Guangming Tan,Xiaowen Xu

DOI: https://doi.org/10.1007/s42514-020-00036-7

2020-01-01

CCF Transactions on High Performance Computing

Abstract:With the HPC systems gradually evolving into the era of exascale, the algorithms and applications are facing challenges of both architectural changes, significantly increased level of parallelism and heterogeneity, as well as new domain requirements from big data analytics and machine learning.This issue focuses on the novel ideas, methods, as well as efforts of software development for resolving the above challenges, and to fill the gap between applications and hardware systems in the coming exascale era.We have nine invited papers selected for this special issue based on a peer-review procedure, which cover a number of different aspects that relate to the algorithm and application challenges mentioned above.The first part of the special issue focuses on the algorithmic improvements for sparse matrix solvers, which become a major design challenge for the continuously increasing compute density of recent high-end hardware platforms.We have four papers that discuss algorithm innovations in different aspects of parallel sparse matrix solvers.
Conceptual and Technical Challenges for High Performance Computing

Claude Tadonki

DOI: https://doi.org/10.48550/arXiv.2010.02769

2020-10-06

Abstract:High Performance Computing (HPC) aims at providing reasonably fast computing solutions to scientific and real life problems. The advent of multicore architectures is noticeable in the HPC history, because it has brought the underlying parallel programming concept into common considerations. At a larger scale, there is a keen interest in building or hosting frontline supercomputers; the Top500 ranking is a nice illustration of this (implicit) racing. Supercomputers, as well as ordinary computers, have fallen in price for years while gaining processing power. We clearly see that, what commonly springs up in mind when it comes to HPC is computer capability. However, when going deeper into the topic, especially on large-scale problems, it appears that the processing speed by itself is no longer sufficient. Indeed, the real concern of HPC users is the time-to-output. Thus, we need to study each important aspect in the critical path between inputs and outputs. The first step is clearly the method, which is a conjunction of modelling with specific considerations (hypothesis, simplifications, constraints, to name a few) and a corresponding algorithm, which could be numerical and/or non numerical. Then comes the topic of programming, which should yield a skillful mapping of the algorithm onto HPC machines. Based on multicore processors, probably enhanced with acceleration units, current generation of supercomputers is rated to deliver an increasing peak performance, the Exascale era being the current horizon. However, getting a high fraction of the available peak performance is more and more difficult. The Design of an efficient code that scales well on a supercomputer is a non-trivial task. The present note will discuss the aforementioned points, interleaved with commented contributions from the literature and our personal views.

Distributed, Parallel, and Cluster Computing
Wafer-scale Computing: Advancements, Challenges, and Future Perspectives

Yang Hu,Xinhan Lin,Huizheng Wang,Zhen He,Xingmao Yu,Jiahao Zhang,Qize Yang,Zheng Xu,Sihan Guan,Jiahao Fang,Haoran Shang,Xinru Tang,Xu Dai,Shaojun Wei,Shouyi Yin

2023-10-14

Abstract:Nowadays, artificial intelligence (AI) technology with large models plays an increasingly important role in both academia and industry. It also brings a rapidly increasing demand for the computing power of the hardware. As the computing demand for AI continues to grow, the growth of hardware computing power has failed to keep up. This has become a significant factor restricting the development of AI. The augmentation of hardware computing power is mainly propelled by the escalation of transistor density and chip area. However, the former is impeded by the termination of the Moore's Law and Dennard scaling, and the latter is significantly restricted by the challenge of disrupting the legacy fabrication equipment and process. In recent years, advanced packaging technologies that have gradually matured are increasingly used to implement bigger chips that integrate multiple chiplets, while still providing interconnections with chip-level density and bandwidth. Compared to conventional high-performance computing paradigms such as multi-accelerator and datacenter-scale computing, Wafer-scale Computing shows remarkable advantages in communication bandwidth, integration density, and programmability potential. Not surprisingly, disruptive Wafer-scale Computing also brings unprecedented design challenges for hardware architecture, design-system-technology co-optimization, power and cooling systems, and compiler tool chain. At present, there are no comprehensive surveys summarizing the current state and design insights of Wafer-scale Computing. This paper aims to take the first step to help academia and industry review existing wafer-scale chips and essential technologies in a one-stop manner. So that people can conveniently grasp the basic knowledge and key points, understand the achievements and shortcomings of existing research, and contribute to this promising research direction.

Hardware Architecture
Virtual Reactors Towards the Exascale Supercomputing Era

Xianmeng Wang,Changjun Hu,Wen Yang,Tiancai Liu,Zhaoshun Wang,Xuesong Wang,Hongwei Yang,Mingyu Wu,Jue Wang,An Wang,Zhifeng Zhou

DOI: https://doi.org/10.1007/978-981-32-9987-0_9

2019-01-01

Abstract:Virtual reactor is an interplanetary research field, which leverages the leading supercomputers to provide a profound understanding of existing reactors and design guidance for future reactors. Virtual reactor is at the cutting-edge of high performance computing and nuclear engineering research fields. The United States, European Union and China have put great effort into virtual reactor research. The research areas of virtual reactors include integrated platform, safety analysis, system analysis, core physics components, database, uncertainty quantification, verification and validation. Typically, core physics consist of thermal-hydraulics, mechanics, neutronics, fuel, and material performance software. Virtual reactor requires enormous computing and storage resource. The computing capabilities of Exascale supercomputers enables realistic models and high-resolution numerical methods come into play. Compared to traditional way, virtual reactor makes simulation more accurate (high-fidelity models are used, such as LES, multi-group transport theory), larger scaled (such as more atoms in molecular dynamics) and tighter coupled. Meanwhile, great challenges have been posed to models, numerical methods, algorithms and programing, especially considering the complicated modern architectures of supercomputers. The paper focuses on the research areas, computing and storage requirements, challenges of virtual reactors.
High Performance Optimization at the Door of the Exascale

Claude Tadonki

DOI: https://doi.org/10.48550/arXiv.2106.11819

2021-06-22

Abstract:quest for processing speed potential. In fact, we always get a fraction of the technically available computing power (so-called {\em theoretical peak}), and the gap is likely to go hand-to-hand with the hardware complexity of the target system. Among the key aspects of this complexity, we have: the {\em heterogeneity} of the computing units, the {\em memory hierarchy and partitioning} including the non-uniform memory access (NUMA) configuration, and the {\em interconnect} for data exchanges among the computing nodes. Scientific investigations and cutting-edge technical activities should ideally scale-up with respect to sustained performance. The special case of quantitative approaches for solving (large-scale) problems deserves a special focus. Indeed, most of common real-life problems, even when considering the artificial intelligence paradigm, rely on optimization techniques for the main kernels of algorithmic solutions. Mathematical programming and pure combinatorial methods are not easy to implement efficiently on large-scale supercomputers because of {\em irregular control flow}, {\em complex memory access patterns}, {\em heterogeneous kernels}, {\em numerical issues}, to name a few. We describe and examine our thoughts from the standpoint of large-scale supercomputers.

Distributed, Parallel, and Cluster Computing
The Scientific Impact of the Exascale Computing Project

Erik W. Draeger,Andrew Siegel,Steven Gottlieb

DOI: https://doi.org/10.1109/mcse.2024.3437608

2024-10-09

Computing in Science & Engineering

Abstract:The recent arrival of the Frontier Supercomputer at Oak Ridge National Laboratory officially marked the dawn of the exascale computing era. Its successful deployment coincided with the culmination of the U.S. Department of Energy Exascale Computing Project (ECP), an ambitious, complex, and risky research and development effort that integrated contributions from a broad and diverse subset of the high-performance computing community. The success of ECP will ultimately be judged by the scientific and engineering advances that it enabled. This Special Issue is focused on showcasing early successes in the use of exascale resources to enable breakthroughs in key areas of science in engineering.

computer science, interdisciplinary applications
Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]

Yang Hu,Xinhan Lin,Huizheng Wang,Zhen He,Xingmao Yu,Jiahao Zhang,Qize Yang,Zheng Xu,Sihan Guan,Jiahao Fang,Haoran Shang,Xinru Tang,Xu Dai,Shaojun Wei,Shouyi Yin

DOI: https://doi.org/10.1109/mcas.2024.3349669

IF: 4.04

2024-01-01

IEEE Circuits and Systems Magazine

Abstract:Nowadays, artificial intelligence (AI) technology with large models plays an increasingly important role in both academia and industry. It also brings a rapidly increasing demand for the computing power of the hardware. As the computing demand for AI continues to grow, the growth of hardware computing power has failed to keep up. This has become a significant factor restricting the development of AI. The augmentation of hardware computing power is mainly propelled by the escalation of transistor density and chip area. However, the former is impeded by the termination of the Moore's Law and Dennard scaling, and the latter is significantly restricted by the challenge of disrupting the legacy fabrication equipment and process. In recent years, advanced packaging technologies that have gradually matured are increasingly used to implement bigger chips that integrate multiple chiplets, while still providing interconnections with chip-level density and bandwidth. This technique points out a new path of continuing the increase of computing power while leveraging the current fabrication process without significant disruption. Enabled by this technique, a chip can extend to a size of wafer-scale provisioning orders of magnitude more computing capabilities (several POPS within just one monolithic chip) and die-to-die bandwidth density (over 15 GB/s/mm) than regular chips, and emerges a new Wafer-scale Computing paradigm. Compared to conventional high-performance computing paradigms such as multi-accelerator and datacenter-scale computing, Wafer-scale Computing shows remarkable advantages in communication bandwidth, integration density, and programmability potential. Not surprisingly, disruptive Wafer-scale Computing also brings unprecedented design challenges for hardware architecture, design- $\backslash $ system- technology co-optimization, power and cooling systems, and compiler tool chain. At present, there are no comprehensive surveys summarizing the current state and design insights of Wafer-scale Computing. This article aims to take the first step to help academia and industry review existing wafer-scale chips and essential technologies in a one-stop manner. So that people can conveniently grasp the basic knowledge and key points, understand the achievements and shortcomings of existing research, and contribute to this promising research direction.
Big Data Challenges: A Program Optimization Perspective

Arun Kejariwal

DOI: https://doi.org/10.1109/cgc.2012.17

2012-11-01

Abstract:Big Data is characterized by the increasing volume (of the order of zeta bytes) and velocity of data generation. It is projected that the market size of Big Data shall climb up to $53.7 billion by 2017 from the current market size of $5.1 billion. Big Data in conjunction with emerging applications such as RMS applications and others has sown the seeds of exascale computing. In a similar vein, In [12], Sexton argued that applications from domains such as materials science, energy, environment and life sciences will require exascale computing. Recent studies directed towards challenges in building exascale systems and charting the roadmap of exascale computing conjecture that exascale systems would support 10-to 100-way concurrency per core and hundreds of cores per die. In [15], HPC Advisory Council predicts that the first exaflop system will be built between 2018 – 2020. In this paper present a program optimization perspective to the challenges posed by Big Data.
Research on Performance Optimization for Large-Scale Sparse Computation over Many-Core Heterogenous Supercomputer

HU Zhengding,XUE Wei

DOI: https://doi.org/10.11959/j.issn.2096-271.2020032

2020-01-01

Abstract:With development of supercomputer technique,it is possible to solve extra-scale sparse problems in big data applications.However,irregular feature in computation and memory access of sparse problems brings challenges to implementation and optimization of applications.Many-core heterogenous architecture is popular in supercomputer design,which advances a higher requirement for application developers.How to utilize its extraordinary computing ability becomes a very difficult problem.Challenges in optimizing sparse computing problems were analyzed,and three cases of implementation and optimization based on typical many-core heterogenous computer system were introduced,which of all achieve very high performance.Experiences in those successful cases were summed up,to better solve extra-scale sparse computing problems on many-core heterogenous system of new generation.
Reinventing High Performance Computing: Challenges and Opportunities

Daniel Reed,Dennis Gannon,Jack Dongarra

DOI: https://doi.org/10.48550/arXiv.2203.02544

2022-03-05

Abstract:The world of computing is in rapid transition, now dominated by a world of smartphones and cloud services, with profound implications for the future of advanced scientific computing. Simply put, high-performance computing (HPC) is at an important inflection point. For the last 60 years, the world's fastest supercomputers were almost exclusively produced in the United States on behalf of scientific research in the national laboratories. Change is now in the wind. While costs now stretch the limits of U.S. government funding for advanced computing, Japan and China are now leaders in the bespoke HPC systems funded by government mandates. Meanwhile, the global semiconductor shortage and political battles surrounding fabrication facilities affect everyone. However, another, perhaps even deeper, fundamental change has occurred. The major cloud vendors have invested in global networks of massive scale systems that dwarf today's HPC systems. Driven by the computing demands of AI, these cloud systems are increasingly built using custom semiconductors, reducing the financial leverage of traditional computing vendors. These cloud systems are now breaking barriers in game playing and computer vision, reshaping how we think about the nature of scientific computation. Building the next generation of leading edge HPC systems will require rethinking many fundamentals and historical approaches by embracing end-to-end co-design; custom hardware configurations and packaging; large-scale prototyping, as was common thirty years ago; and collaborative partnerships with the dominant computing ecosystem companies, smartphone, and cloud computing vendors.

Distributed, Parallel, and Cluster Computing
Opportunities and Challenges for Next Generation Computing

Gregory D. Hager,Mark D. Hill,Katherine Yelick

DOI: https://doi.org/10.48550/arXiv.2008.00023

2020-07-31

Computers and Society

Abstract:Computing has dramatically changed nearly every aspect of our lives, from business and agriculture to communication and entertainment. As a nation, we rely on computing in the design of systems for energy, transportation and defense; and computing fuels scientific discoveries that will improve our fundamental understanding of the world and help develop solutions to major challenges in health and the environment. Computing has changed our world, in part, because our innovations can run on computers whose performance and cost-performance has improved a million-fold over the last few decades. A driving force behind this has been a repeated doubling of the transistors per chip, dubbed Moore's Law. A concomitant enabler has been Dennard Scaling that has permitted these performance doublings at roughly constant power, but, as we will see, both trends face challenges. Consider for a moment the impact of these two trends over the past 30 years. A 1980's supercomputer (e.g. a Cray 2) was rated at nearly 2 Gflops and consumed nearly 200 KW of power. At the time, it was used for high performance and national-scale applications ranging from weather forecasting to nuclear weapons research. A computer of similar performance now fits in our pocket and consumes less than 10 watts. What would be the implications of a similar computing/power reduction over the next 30 years - that is, taking a petaflop-scale machine (e.g. the Cray XK7 which requires about 500 KW for 1 Pflop (=1015 operations/sec) performance) and repeating that process? What is possible with such a computer in your pocket? How would it change the landscape of high capacity computing? In the remainder of this paper, we articulate some opportunities and challenges for dramatic performance improvements of both personal to national scale computing, and discuss some "out of the box" possibilities for achieving computing at this scale.
Design and Implementation of the Tianhe-2 Data Storage and Management System

Yu-Tong Lu,Peng Cheng,Zhi-Guang Chen

DOI: https://doi.org/10.1007/s11390-020-9799-4

IF: 1.871

2020-01-01

Journal of Computer Science and Technology

Abstract:With the convergence of high-performance computing (HPC), big data and artificial intelligence (AI), the HPC community is pushing for "triple use" systems to expedite scientific discoveries. However, supporting these converged applications on HPC systems presents formidable challenges in terms of storage and data management due to the explosive growth of scientific data and the fundamental differences in I/O characteristics among HPC, big data and AI workloads. In this paper, we discuss the driving force behind the converging trend, highlight three data management challenges, and summarize our efforts in addressing these data management challenges on a typical HPC system at the parallel file system, data management middleware, and user application levels. As HPC systems are approaching the border of exascale computing, this paper sheds light on how to enable application-driven data management as a preliminary step toward the deep convergence of exascale computing ecosystems, big data, and AI.

computer science, software engineering, hardware & architecture
Exascale Quantum Mechanical Simulations: Navigating the Shifting Sands of Hardware and Software

Ravindra Shinde,Claudia Filippi,Anthony Scemama,William Jalby

DOI: https://doi.org/10.26434/chemrxiv-2024-qcgfd-v2

2024-09-19

Abstract:The era of exascale computing presents both exciting opportunities and unique challenges for quantum mechanical simulations. While the transition from petaflops to exascale computing has been marked by a steady increase in computational power, the shift towards heterogeneous architectures, particularly the dominant role of graphical processing units (GPUs), demands a fundamental shift in software development strategies. This review examines the changing landscape of hardware and software for exascale computing, highlighting the limitations of traditional algorithms and software implementations in light of the increasing use of heterogeneous architectures in high-end systems. We discuss the challenges of adapting quantum chemistry software to these new architectures, including the fragmentation of the software stack, the need for more efficient algorithms (including reduced precision versions) tailored for GPUs, and the importance of developing standardized libraries and programming models.

Chemistry

Challenges and Reflections on Exascale Computing

Application software beyond exascale: challenges and possible trends

A Vision of Post-Exascale Programming.

Cluster and Grid Superservers: the Dawning Experiences in China

Architecture, Challenges and Applications of Edge Computing

Programming at Exascale: Challenges and Innovations

Challenges Towards Deploying Data Intensive Scientific Applications on Extreme Heterogeneity Supercomputers

Editorial for the Special Issue on HPC Algorithms and Applications

Conceptual and Technical Challenges for High Performance Computing

Wafer-scale Computing: Advancements, Challenges, and Future Perspectives

Virtual Reactors Towards the Exascale Supercomputing Era

High Performance Optimization at the Door of the Exascale

The Scientific Impact of the Exascale Computing Project

Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]

Big Data Challenges: A Program Optimization Perspective

Research on Performance Optimization for Large-Scale Sparse Computation over Many-Core Heterogenous Supercomputer

Reinventing High Performance Computing: Challenges and Opportunities

Opportunities and Challenges for Next Generation Computing

Design and Implementation of the Tianhe-2 Data Storage and Management System

Exascale Quantum Mechanical Simulations: Navigating the Shifting Sands of Hardware and Software