Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers

Julia Mullen,Albert Reuther,William Arcand,Bill Bergeron,David Bestor,Chansup Byun,Vijay Gadepally,Michael Houle,Matthew Hubbell,Michael Jones,Anna Klein,Peter Michaleas,Lauren Milechin,Andrew Prout,Antonio Rosa,Siddharth Samsi,Charles Yee,Jeremy Kepner

DOI: https://doi.org/10.1007/978-3-030-02465-9_47

2019-03-06

Abstract:For decades, the use of HPC systems was limited to those in the physical sciences who had mastered their domain in conjunction with a deep understanding of HPC architectures and algorithms. During these same decades, consumer computing device advances produced tablets and smartphones that allow millions of children to interactively develop and share code projects across the globe. As the HPC community faces the challenges associated with guiding researchers from disciplines using high productivity interactive tools to effective use of HPC systems, it seems appropriate to revisit the assumptions surrounding the necessary skills required for access to large computational systems. For over a decade, MIT Lincoln Laboratory has been supporting interactive, on-demand high performance computing by seamlessly integrating familiar high productivity tools to provide users with an increased number of design turns, rapid prototyping capability, and faster time to insight. In this paper, we discuss the lessons learned while supporting interactive, on-demand high performance computing from the perspectives of the users and the team supporting the users and the system. Building on these lessons, we present an overview of current needs and the technical solutions we are building to lower the barrier to entry for new users from the humanities, social, and biological sciences.

Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that over the past few decades, the use of high - performance computing (HPC) systems has been mainly limited to researchers who have in - depth expertise in the physical sciences and a deep understanding of HPC architectures and algorithms. With the development of consumer computing devices, such as tablets and smartphones, millions of children around the world are able to develop and share code projects interactively. However, when the HPC community is faced with guiding researchers from different disciplines to shift from using highly productive interactive tools to effectively using HPC systems, it becomes particularly important to re - examine the assumptions about the skills required to access large - scale computing systems.

Specifically, the paper explores how to support interactive, on - demand high - performance computing by seamlessly integrating familiar high - productivity tools, providing users with more design iterations, rapid prototyping capabilities, and faster insight times. The paper summarizes the lessons learned from the perspectives of users and support teams in the process of supporting this model, and based on these lessons, outlines the current requirements and technical solutions, aiming to lower the entry barrier for new users in the humanities, social, and biological sciences.

The core issue of the paper is to explore how to break the single path of traditional HPC use, which requires obtaining the necessary skills through postgraduate education, and instead provide diverse learning paths for a broader user group, especially for those disciplines that have already adopted computational solution strategies but lack an HPC background. In this way, the paper aims to promote the popularization of HPC technology, making it not only limited to the physical sciences and engineering fields, but also serving a wider range of research and industrial applications.